System for voice data processing and method for operation thereof

Abstract

This system for voice data processing comprises: an analog voice processing unit that receives an input of an analog voice signal and generates a digital voice signal; and a feature extraction unit that, when the digital voice signal is input, operates as an infinite impulse response (IIR) filter to filter the digital voice signal, and operate as an RNN to extract a voice feature from the filtered digital voice signal.

Description

Voice data processing system and its operation method

[0001] The present invention relates to a voice data processing system for processing voice data with low power and an operating method thereof.

[0002] Command recognition technology, now widely used, allows voice recognition systems to detect and identify specific commands. Because it's difficult to predict when a command will be issued, voice recognition systems must remain on at all times.

[0003] Therefore, in order to use a voice recognition system in environments such as mobile devices and edge devices with limited battery life, the power consumption resulting from command recognition must be minimized.

[0004] At this time, the feature extractor in the speech recognition system processes the speech signal and converts it into a value that can be understood by the speech recognition algorithm (neural network).

[0005] Feature extractors in typical speech recognition systems use the Mel Frequency Cepstral Coefficient (MFCC) method. MFCC is a widely used feature extraction method in speech signal processing and is suitable for software implementation. However, because it requires multiple signal processing steps, it requires complex hardware that requires significant computational effort.

[0006] That is, complex dedicated hardware is required to perform Mel scaling, Fast Fourier Transform (FFT), logarithmic transformation, and Discrete Cosine Transform (DCT) on MFCCs. Furthermore, the memory requirements for storing intermediate data structures and calculation results required to calculate MFCCs also increase, leading to the disadvantage of requiring a large hardware footprint.

[0007] Accordingly, the present invention provides a voice data processing system and an operating method thereof that not only reduces hardware area but also enables efficient recognition of commands through hardware compatible with a feature extractor using RNN (Recurrent Neural Network) hardware specialized for voice processing.

[0008] A voice data processing system, which is one feature of the present invention for achieving the technical problem of the present invention, includes an analog voice processing unit that receives an analog voice signal as input and generates a digital voice signal, and a feature extraction unit that operates as an IIR (Infinite Impulse Response) filter when the digital voice signal is input to filter the digital voice signal and operates as an RNN to extract voice features from the filtered digital voice signal.

[0009] The above feature extraction unit may include an input / output memory that stores a value obtained by filtering the digital voice signal with an IIR filter, the voice feature extracted by RNN processing the filtered digital voice signal, and a weight and coefficient memory that stores a first IIR coefficient, a second IIR coefficient, a first RNN weight, and a second RNN weight for processing the digital voice signal.

[0010] The input / output memory may include an IIR input memory that stores a first value accumulated by multiplying the digital voice signal by the first IIR coefficient according to the order of the IIR filter, a second value accumulated by multiplying the first value by the second IIR coefficient according to the order of the IIR filter, and an IIR output memory that stores a voice feature obtained by accumulating a value obtained by multiplying a third value obtained based on the second value by the first RNN weight according to the size of the first RNN weight, an RNN input memory that stores the second value, and an RNN output memory that stores the third value accumulated by multiplying the second value by the second RNN weight according to the size of the second RNN weight.

[0011] The above weight and coefficient memory may include a first IIR coefficient memory in which the feed forward coefficient, which is the first IIR coefficient, is stored, a second IIR coefficient memory in which the feedback coefficient, which is the second IIR coefficient, is stored, a first RNN weight memory in which the RNN h weight, which is the first RNN weight, is stored, and a second RNN weight memory in which the RNN x weight, which is the second RNN weight, is stored.

[0012] The above feature extraction unit further includes a nonlinear transformation unit that nonlinearly transforms the second value, and the nonlinear transformation unit can be implemented as a nonlinear function.

[0013] The above feature extraction unit may further include the weight and coefficient memory, the input / output memory, and a controller that generates a control signal for controlling the feature extraction unit.

[0014] Another feature of the present invention for achieving the technical problem of the present invention is a method of operating a voice data processing system operated by at least one processor, comprising the steps of: when receiving an analog voice signal, generating the analog voice signal into a digital voice signal; and filtering the digital voice signal through an IIR filter having a circular structure to extract voice features from the digital voice signal.

[0015] The step of extracting the above voice feature may include a step of performing a multiplication operation on the digital voice signal and a first IIR coefficient, and a step of accumulating the result of the multiplication operation according to the order of the IIR filter to obtain a first value.

[0016] After the step of obtaining the first value, the step of performing a multiplication operation on the first value and the second IIR coefficient and accumulating the result according to the order of the IIR filter to obtain the second value may be included.

[0017] After the step of obtaining the second value, the step of receiving the second value as an input of the RNN, performing a multiplication operation with the second value and the first RNN weight, and accumulating the result according to the size of the first RNN weight to obtain a third value may be included.

[0018] After the step of obtaining the third value, a step of applying the third value to a nonlinear function to obtain a nonlinearly transformed result may be included.

[0019] After the step of obtaining the nonlinearly transformed result, the step of performing a multiplication operation on the nonlinearly transformed result and the second RNN weight and accumulating the result according to the size of the second RNN weight to output the voice feature may be included.

[0020] The first IIR coefficient may be a feedforward coefficient, the second IIR coefficient may be a feedback coefficient, the first RNN weight may be an RNN x weight, and the second RNN weight may be an RNN h weight.

[0021] According to the present invention, the performance of a command recognizer can be improved by using an RNN, and the area of ​​hardware can be saved by utilizing an RNN layer as a feature extractor using an IIR (Infinite Impulse Response).

[0022] Additionally, by reducing the hardware area, it supports the voice data processing system to operate at low power.

[0023] Figures 1 and 2 are examples of a general command recognition system.

[0024] Figure 3 is an exemplary diagram of an environment in which a voice data processing system according to an embodiment of the present invention is applied.

[0025] Figure 4 is an example diagram of a typical IIR filter.

[0026] Figures 5 to 8 are exemplary diagrams of a feature extraction unit according to an embodiment of the present invention.

[0027] Figure 9 is a flowchart of an operation method of a voice data processing system according to an embodiment of the present invention.

[0028] Figure 10 is a structural diagram of a computing system according to an embodiment of the present invention.

[0029] Below, with reference to the attached drawings, embodiments of the present invention are described in detail so that those skilled in the art can easily implement the invention. However, the present invention can be implemented in various different forms and is not limited to the embodiments described herein. In addition, in the drawings, parts irrelevant to the description are omitted for clarity of description, and similar parts are designated with similar reference numerals throughout the specification.

[0030] Throughout the specification, whenever a part is said to "include" a certain configuration, this does not mean that it excludes other configurations, but rather that it may include other configurations, unless otherwise specifically stated.

[0031] Hereinafter, a voice data processing system and its operating method according to an embodiment of the present invention will be described with reference to the drawings. Before describing the embodiment of the present invention, a general command recognition system will first be described with reference to FIGS. 1 and 2.

[0032] Figures 1 and 2 are examples of a general command recognition system.

[0033] As illustrated in Fig. 1, a general command recognition system (10) is composed of an analog circuit (11) and a feature extractor (12). In addition, the general command recognition system (10) may additionally include a neural network (13).

[0034] The analog circuit (11) processes an analog voice signal and generates a digital voice signal.

[0035] The feature extractor (12) extracts features of a voice signal in the frequency domain using techniques such as FFT. However, implementing FFT in hardware requires a large area and a large amount of computation, resulting in high power consumption.

[0036] Therefore, instead of a feature extractor (12), a neural network (13) can be used to extract features of a voice signal. Using a neural network (13) can reduce the area, but there is a problem that the amount of computation increases further.

[0037] Here, referring to Fig. 2, an example of extracting features of a voice signal using a feature extractor (12) will be described. MFCC, which is a method of extracting features by processing a voice signal in a feature extractor (12), is a feature extraction method that is currently widely used.

[0038] MFCCs are widely used in various audio analysis fields, including speech signal processing, speech recognition, and acoustic analysis. Because MFCCs require multiple signal processing steps, they are suitable for software implementation, but hardware implementations are more complex and computationally intensive.

[0039] In other words, complex hardware is required to perform Mel scaling, FFT, log transformation, DCT, etc. on MFCC, and a large amount of memory is required to store the intermediate data structures and calculation results required to calculate MFCC. Therefore, a large hardware area is required.

[0040] At this time, one of the methods for utilizing the function of the feature extractor (12) by utilizing a neural network is a method using the Sinc function. The result of performing a multiplication operation with a rectangular function in the frequency domain is equivalent to applying a convolution operation with the Sinc function in the time domain.

[0041] When performing a convolution operation using a sink function, a neural network convolution layer can be utilized, thereby reducing the area of ​​the feature extractor (12). However, due to the nature of the convolutional neural network, it is specialized for extracting features from images, making it difficult to achieve high accuracy in extracting audio features during speech processing.

[0042] Therefore, in an embodiment of the present invention, a voice data processing system compatible with a feature extractor is proposed by utilizing RNN (Recurrent Neural Network) hardware specialized for voice processing.

[0043] Figure 3 is a structural diagram of a voice data processing system according to an embodiment of the present invention.

[0044] As illustrated in FIG. 3, the voice data processing system (100) includes an analog voice processing unit (110) and a feature extraction unit (120).

[0045] In the embodiment of the present invention, the analog voice processing unit (110) is implemented as an example of an LNA (Low Noise Amplifier) ​​(111) that minimizes noise in an analog voice signal, a VGA (Variable Gain Amplifier) ​​(112) that performs the role of more precisely controlling the gain of the signal whose gain is roughly controlled by the LNA (111) of the analog voice signal with minimized noise, and an ADC (Analog Digital Converter) (113) that converts the analog voice signal whose gain is controlled into a digital voice signal. However, the present invention is not limited thereto. Here, the functions of the LNA (111), the VGA (112), and the ADC (113) are already known, and thus, a detailed description thereof is omitted in the embodiment of the present invention.

[0046] The feature extraction unit (120) is implemented as hardware based on a recurrent neural network (RNN). That is, the feature extraction unit (120) according to the embodiment of the present invention operates as an IIR (Infinite Impulse Response) filter having a recurrent structure to filter a digital voice signal, and also operates as an RNN to extract voice features from the filtered digital voice signal.

[0047] This reduces the area occupied by the voice data processing system (100) and enables low-power operation. Here, the IIR filter is a recursive filter that supports a relatively simple calculation process and has good area and power efficiency. The feature extraction unit (120) implemented as an IIR filter will be described with reference to FIGS. 4 to 8.

[0048] Fig. 4 is an example diagram of a general IIR filter, and Figs. 5 to 8 are examples diagrams of a feature extraction unit according to an embodiment of the present invention.

[0049] As shown in Fig. 4, in general, the IIR filter transmits the input signal (x[n]) through the feedforward path (①). In the feedforward path, the input signal is fed into the feedforward coefficient (b x ) is multiplied, and the value resulting from the multiplication is passed to the feedback path (②) together with the previous output values ​​(y[n-1], y[n-2], …).

[0050] In the feedback path (①), the previous output value is the feedback coefficient (a x ) is multiplied. The multiplication results of the feedforward path (①) and the feedback path (②) are added together to produce the final output value (y[n]).

[0051] Because of this simple computational process, IIR filters can reduce area and improve power efficiency.

[0052] That is, as illustrated in FIG. 5, the embodiment of the present invention proposes a structure capable of performing both the operation of the feature extraction unit (120) and the artificial neural network operation by utilizing the common recursion of the IIR filter and the RNN. This is different from the general feature extraction unit (12) and the artificial neural network dedicated hardware (13) that exist separately as illustrated in FIG. 1 described above.

[0053] RNNs have a recurrent structure, using the output of the previous time step along with the input of the current time step. RNNs have weights for the output and hidden state of the previous time step, and the weights used at each time step of the RNN share the same weight matrix.

[0054] Additionally, the IIR filter reflects the previous output to the current output through a cyclic structure. Since the IIR filter is effective in controlling the frequency response and filtering signals, it can improve the accuracy of the voice data processing system (100).

[0055] That is, the feature extraction unit (120) according to the embodiment of the present invention performs the control operations necessary for the IIR filter and RNN operation operations. To this end, the feature extraction unit (120) is described as being implemented by a controller (121), an input / output memory (122), a weight and coefficient memory (123), and a nonlinear transformation unit (124) as an example. In addition, the feature extraction unit (120) is described as further including a multiplier (125) and an adder (126) as an example.

[0056] The controller (121) generates a control signal that controls the input / output memory (122), the weight and coefficient memory (123), and the nonlinear transformation unit (124).

[0057] When an input signal corresponding to an IIR input, i.e., a digital voice signal, is output from the input / output memory (122), the weight and coefficient memory (123) outputs the IIR b coefficient registered in the IIR b coefficient memory, as illustrated in FIG. 6. In the embodiment of the present invention, the implementation of a second-order IIR filter as a feature extraction unit (120) is described as an example, and therefore the IIR b coefficient is described as a feedforward coefficient.

[0058] The multiplier (125) performs a multiplication operation on the input signal output from the input / output memory (122) and the IIR b coefficient output from the weight and coefficient memory (123) and outputs the result.

[0059] The adder (126) accumulates the output from the multiplier (125) according to the order of the IIR filter. For example, assuming that the order of the IIR filter is n, the adder (126) accumulates the output of the multiplier (126) n-1 times and transmits it to the nonlinear transformation unit (124).

[0060] And, the nonlinear transformation unit (124) outputs the received input value as it is, and the output value is transmitted to the input / output memory (122) and stored in the IIR output memory as illustrated in FIG. 7. The value stored in the IIR output memory of the input / output memory (122) is output again, and as illustrated in FIG. 6, it is multiplied in a multiplier (125) together with the feedback coefficient, which is the IIR a coefficient previously stored in the IIR a coefficient memory of the weight and coefficient memory (123).

[0061] The value multiplied N-1 times according to the order of the IIR filter is accumulated through the adder (126), and the final output value is stored in the IIR output memory of the input / output memory (122) shown in Fig. 7.

[0062] After the feature extraction unit (120) operates as an IIR filter through the above-described procedure, it operates as an RNN through the following procedure. That is, when the RNN input registered in the RNN input memory shown in FIG. 7 in the input / output memory (122), i.e., the output of the IIR filter, is output as an RNN input, the RNN x weight previously registered in the RNN x weight memory shown in FIG. 6 among the weight and coefficient memories (123) is output.

[0063] Here, in the embodiment of the present invention, the IIR output memory and the RNN input memory of the input / output memory (122) are illustrated as being separated from each other, but the IIR output memory may also be used as the RNN input memory.

[0064] Meanwhile, the multiplier (125) performs a multiplication operation on the RNN input and the RNN x weight. At this time, the adder (126) accumulates the result of the multiplication operation on the RNN x weight according to the size of the RNN x weight registered in the RNN x weight memory.

[0065] As shown in Fig. 8, the nonlinear transformation unit (124) applies an activation function, and a nonlinear transformation is performed to store the result of the nonlinear transformation in the RNN output memory of the input / output memory (122) shown in Fig. 7. At this time, the function applied to the nonlinear transformation unit (124) may be a nonlinear function of various forms (e.g., hyperbolic tangent (tanh), sigmoid (sigmoid), ReLU, etc.).

[0066] The values ​​stored in the RNN output memory of the input / output memory (122) are output again, and at the same time, the RNN h weights previously stored in the RNN h weight memory of the weight and coefficient memory (123) are also output as shown in FIG. 6.

[0067] The multiplier (125) performs a multiplication operation on the value stored in the RNN output memory and the RNN h weight. The adder (126) adds the value accumulated in the RNN output memory and the value obtained by multiplying the RNN h weight according to the size of the RNN h weight, and the finally calculated value is stored in the input / output memory (122) as a voice feature. At this time, the feature extraction unit (120) repeatedly performs the RNN operation according to the number of RNN layers.

[0068] In the embodiment of the present invention, an example is described in which the feature extraction unit (120) performs an IIR operation and then an RNN operation. Accordingly, the output of the IIR can become the input of the RNN.

[0069] The operation method of the voice data processing system (100) including the above-described feature extraction unit (120) is described with reference to FIG. 9.

[0070] Figure 9 is a flowchart of an operation method of a voice data processing system according to an embodiment of the present invention.

[0071] As illustrated in FIG. 9, when the voice data processing system (100) receives an analog voice signal (S100), the analog voice processing unit (110) of the voice data processing system (100) generates the analog voice signal into a digital voice signal (S110). The method by which the analog voice processing unit (110) generates the analog voice signal into a digital voice signal is a known technology, and the embodiment of the present invention is not limited to any one method.

[0072] When the analog voice processing unit (110) of the voice data processing system (100) generates a digital voice signal through step S110, the feature extraction unit (120), which is RNN hardware, operates as an IIR filter when receiving the digital voice signal as input (S120). When the feature extraction unit (120) operating as an IIR filter extracts a value from the digital voice signal, the feature extraction unit (120) operates as an RNN and outputs a voice feature using the value (S130).

[0073] Steps S120 and S130 will be described in detail. When an input signal corresponding to a digital voice signal is output, the feature extraction unit (120) outputs an IIR b coefficient. In the embodiment of the present invention, a second-order IIR filter is implemented as the feature extraction unit (120) as an example, and therefore, the IIR b coefficient is described as a feedforward coefficient as an example.

[0074] The multiplier (125) multiplies the digital voice signal and the IIR b coefficient to output a first value. The adder (126) accumulates the first value, which is the output from the multiplier (125), according to the order of the IIR filter to obtain a second value. For example, assuming that the order of the IIR filter is n, the adder (126) accumulates the output of the multiplier (126) n-1 times and transmits the second value to the nonlinear converter (124).

[0075] And, the nonlinear transformation unit (124) outputs the received second value as it is, and the output second value is stored in the IIR input memory. The second value stored in the IIR input memory is multiplied by the feedback coefficient, which is the IIR a coefficient, in the multiplier (125) to calculate the third value. The third value multiplied n-1 times according to the order of the IIR filter is accumulated through the adder (126), and the fourth value calculated by accumulation is stored in the IIR output memory.

[0076] The feature extraction unit (120) that operates as an IIR filter through the above-described procedure operates as an RNN. That is, when the fourth value, which is the output of the IIR filter, is input as an RNN input, the feature extraction unit (120) performs a multiplication operation with the previously stored RNN x weight and the fourth value. At this time, the feature extraction unit (120) accumulates the result of the multiplication operation of the RNN x weight according to the RNN x weight size to obtain the fifth value.

[0077] The feature extraction unit (120) applies the fifth value to the activation function, which is a nonlinear function, and stores the result of the nonlinear transformation. The feature extraction unit (120) multiplies the result of the nonlinear transformation and the previously stored RNN h weights, and then adds them, and the final calculated value is output as a voice feature.

[0078] Figure 10 is a structural diagram of a computing system according to an embodiment of the present invention.

[0079] Referring to FIG. 10, a voice data processing system (100) operated by at least one processor may be implemented as a computing system (200), and executes a program including instructions described to execute the operations of the present invention in the computing system (200). The program may be stored in a computer-readable storage medium and distributed.

[0080] The hardware of the computing system (200) may include at least one processor (210), memory (220), storage (230), and a communication interface (240), and may be connected via a bus. In addition, hardware such as input devices and output devices may be included. The computing system (200) may be equipped with various software, including an operating system capable of running programs.

[0081] The processor (210) is a device that controls the operation of the computing system (200), and may be a processor of various types that processes commands included in a program, for example, a CPU (Central Processing Unit), an MPU (Micro Processor Unit), an MCU (Micro Controller Unit), a GPU (Graphics Processing Unit), etc.

[0082] The memory (220) loads the program so that the commands described to execute the operations of the present invention are processed by the processor (210). The memory (220) may be, for example, a ROM (read only memory), a RAM (random access memory), etc. The storage (230) stores various data, programs, etc. required to execute the operations of the present invention. The communication interface (240) may be a wired / wireless communication module.

[0083] Although the embodiments of the present invention have been described in detail above, the scope of the present invention is not limited thereto, and various modifications and improvements made by those skilled in the art using the basic concept of the present invention defined in the following claims also fall within the scope of the present invention.

Claims

1. As a voice data processing system, An analog voice processing unit that receives an analog voice signal as input and generates a digital voice signal, and When the above digital voice signal is input, a feature extraction unit that operates as an IIR (Infinite Impulse Response) filter to filter the digital voice signal and operates as an RNN to extract voice features from the filtered digital voice signal. A voice data processing system comprising:

2. In paragraph 1, The above feature extraction unit, An input / output memory for storing the value of the digital voice signal filtered by an IIR filter, the voice features extracted by RNN processing the filtered digital voice signal, and A weight and coefficient memory storing a first IIR coefficient, a second IIR coefficient, a first RNN weight, and a second RNN weight for processing the above digital voice signal. A voice data processing system comprising:

3. In paragraph 2, The above input / output memory is, An IIR input memory that stores the product of the digital voice signal and the first IIR coefficient as a first accumulated value according to the order of the IIR filter. An IIR output memory that stores a voice feature obtained by accumulating a second value obtained by multiplying the first value by the second IIR coefficient according to the order of the IIR filter, and a third value obtained by multiplying the third value by the first RNN weight according to the size of the first RNN weight. RNN input memory storing the second value, and An RNN output memory that stores the third value accumulated by multiplying the second value by the second RNN weight according to the size of the second RNN weight. A voice data processing system including 4. In paragraph 3, The above weight and coefficient memory, A first IIR coefficient memory in which the feed forward coefficient, which is the first IIR coefficient, is stored, A second IIR coefficient memory in which the feedback coefficient, which is the second IIR coefficient, is stored, A first RNN weight memory in which the first RNN weight, RNN h weight, is stored, and A second RNN weight memory in which the second RNN weight, RNN x weight, is stored. A voice data processing system comprising:

5. In paragraph 4, The above feature extraction unit, A nonlinear transformation unit that nonlinearly transforms the above second value Including more, A voice data processing system, wherein the above nonlinear transformation unit is implemented as a nonlinear function.

6. In paragraph 2, The above feature extraction unit, A controller that generates a control signal that controls the above weight and coefficient memory, the above input / output memory, and the above feature extraction unit. A voice data processing system further comprising:

7. A method of operating a voice data processing system operated by at least one processor, When receiving an analog voice signal, a step of generating the analog voice signal into a digital voice signal, and A step of extracting voice features from the digital voice signal by filtering the digital voice signal through an IIR filter having a circular structure. A method of operation, comprising:

8. In paragraph 7, The step of extracting the above voice features is: A step of multiplying the above digital voice signal and the first IIR coefficient, and A step of accumulating the result of the above multiplication operation according to the order of the IIR filter to obtain the first value. A method of operation, comprising:

9. In paragraph 8, After the step of obtaining the above first value, A step of multiplying the first value and the second IIR coefficient and accumulating them according to the order of the IIR filter to obtain the second value. A method of operation, comprising:

10. In paragraph 9, After the step of obtaining the above second value, A step of receiving the second value as an input of the RNN, performing a multiplication operation with the second value and the first RNN weight, and accumulating the second value according to the size of the first RNN weight to obtain a third value. A method of operation, comprising:

11. In paragraph 10, After the step of obtaining the third value, A step of applying the above third value to a nonlinear function to obtain a nonlinearly transformed result. A method of operation, comprising:

12. In paragraph 11, After the step of obtaining the above nonlinearly transformed result, A step of multiplying the nonlinearly transformed result and the second RNN weight, accumulating the result according to the size of the second RNN weight, and outputting the voice feature. A method of operation, comprising:

13. In paragraph 12, A method of operation, wherein the first IIR coefficient is a feedforward coefficient, the second IIR coefficient is a feedback coefficient, the first RNN weight is an RNN x weight, and the second RNN weight is an RNN h weight.

Images