Speech recognition device and method

Abstract

The present disclosure relates to an electronic device for recognizing user voice and a method for recognizing user voice by the electronic device. According to one embodiment, the method for recognizing user voice includes the steps of: obtaining an audio signal segmented into a plurality of frame units; determining an energy component for each filter group by applying a filter group distributed according to a preset scale to the spectrum of the audio signal segmented in frame units; smoothing the determined energy component for each filter group; extracting a feature vector of the audio signal based on the smoothed energy component for each filter group; and recognizing the user voice in the audio signal by inputting the extracted feature vector into a speech recognition model.

Description

Technical Field

[0001] The present disclosure relates to a speech recognition apparatus and method. More specifically, the present disclosure relates to a method for processing audio data for speech recognition. Background Art

[0002] Unlike traditional rule-based intelligent systems, artificial intelligence (AI) systems are intelligent systems that learn and determine on their own. The more an AI system is used, the higher its recognition rate becomes, and therefore, the more accurately it can understand user preferences. As a result, traditional rule-based intelligent systems are gradually being replaced by deep learning-based AI systems.

[0003] AI technology includes machine learning (deep learning) and elemental technology using machine learning. Machine learning is an algorithmic technology that classifies / learns the features of input data, and elemental technology is a technology that utilizes machine learning algorithms such as deep learning, and consists of technical fields including language understanding, visual understanding, reasoning / prediction, knowledge expression, motion control, etc.

[0004] Artificial intelligence technology is applied to various fields as follows. Language understanding is the technology of recognizing and applying / processing human language / text, and includes natural language processing, machine translation, dialogue systems, question and answer, speech recognition / synthesis, etc. Visual understanding is the technology of recognizing and processing objects such as human vision, and includes object recognition, object tracking, image search, human recognition, scene understanding, spatial understanding, image improvement, etc. Inference and prediction are the technologies of logically inferring and predicting information by determining information, and include knowledge / probability-based inference, optimization prediction, preference-based planning, recommendation, etc. Knowledge expression is the technology of processing human experience information into knowledge data through automation, and includes knowledge construction (data generation / classification), knowledge management (data utilization), etc. Motion control is the technology of controlling the autonomous driving of vehicles and the movement of robots, and includes mobile control (navigation, collision, driving), operation control (behavior control), etc.

[0005] Recently, in the field of speech recognition, various technologies for recognizing user voices are being studied, and speech recognition methods based on artificial neural networks using an end-to-end learning method are being actively studied. In particular, in the field of speech recognition, technology development is necessary to improve the accuracy of speech recognition. Summary of the invention

[0006] Technical issues

[0007] According to an embodiment, a speech recognition apparatus and method for recognizing a user's speech using a speech recognition model are provided.

[0008] In addition, according to an embodiment, a device and method for learning a speech recognition model based on an audio signal are provided. Specifically, an audio data processing method for improving the accuracy of speech recognition is provided.

[0009] Technical Solution

[0010] According to one embodiment, a method for recognizing user voice includes: obtaining an audio signal segmented into multiple frame units, determining an energy component for each filter group by applying a filter group distributed according to a preset scale to the frequency spectrum of the audio signal segmented into frame units, smoothing the determined energy component for each filter group, extracting a feature vector of the audio signal based on the smoothed energy component for each filter group, and recognizing the user voice in the audio signal by inputting the extracted feature vector into a speech recognition model.

[0011] In addition, according to an embodiment, an electronic device for recognizing user voice includes: a memory storing one or more instructions; and a processor configured to execute the one or more instructions, wherein the processor is further configured to perform the following operations by executing the one or more instructions: obtaining an audio signal segmented into multiple frame units, determining an energy component for each filter group by applying a filter group distributed according to a preset scale to the frequency spectrum of the audio signal segmented into frame units, smoothing the determined energy component for each filter group, extracting a feature vector of the audio signal based on the smoothed energy component for each filter group, and recognizing the user voice in the audio signal by inputting the extracted feature vector into a speech recognition model.

[0012] In addition, according to an embodiment, a method for processing an audio signal, by which an electronic device trains a speech recognition model, includes: obtaining a first audio signal divided into multiple frame units for training the speech recognition model, obtaining a spectrum of the first audio signal divided into frame units, transforming the frequency axis of the spectrum of the first audio signal to represent changes in different vocal tract lengths of multiple speakers, generating a second audio signal by using the spectrum of the first audio signal with the frequency axis transformed, and extracting a feature vector of the generated second audio signal.

[0013] The method may further include applying a room impulse filter to the second audio signal, and extracting a feature vector from the second audio signal to which the room impulse filter is applied, wherein the room impulse filter indicates an acoustic feature of the second audio signal for each transmission path in the room.

[0014] The transforming of the frequency axis may include determining a warp coefficient randomly generated for each frame, determining a warp function for transforming the frequency axis of the spectrum of the first audio signal based on the determined warp coefficient, and transforming the frequency axis of the spectrum of the first audio signal by using the determined warp function.

[0015] The generating of the second audio signal may include performing inverse fast Fourier transform on a spectrum of the frequency axis transformed first audio signal, and generating the second audio signal by overlapping the spectrum of the inverse fast Fourier transformed first audio signal on the time axis.

[0016] The step of extracting the feature vector of the second audio signal may include: determining an energy component for each filter group by applying a filter group distributed according to a preset scale to the frequency spectrum of the second audio signal divided into frame units, smoothing the determined energy component for each filter group, and extracting the feature vector of the second audio signal based on the smoothed energy component for each filter group.

[0017] In addition, according to an embodiment, a method for training a speech recognition model by an electronic device includes: obtaining a first audio signal divided into multiple frame units for training the speech recognition model, obtaining a spectrum of the first audio signal divided into frame units, transforming the frequency axis of the spectrum of the first audio signal to represent changes in different vocal tract lengths of multiple speakers, generating a second audio signal by using the spectrum of the first audio signal with the frequency axis transformed, extracting a feature vector of the generated second audio signal, and training the speech recognition model by using the extracted feature vector.

[0018] In addition, according to an embodiment, a computer-readable recording medium is provided, on which a program for executing a method for recognizing user voice on a computer is recorded, the method comprising: obtaining an audio signal segmented into multiple frame units, determining an energy component for each filter group by applying a filter group distributed according to a preset scale to the frequency spectrum of the audio signal segmented into frame units, smoothing the determined energy component for each filter group, extracting a feature vector of the audio signal based on the smoothed energy component for each filter group, and recognizing the user voice in the audio signal by inputting the extracted feature vector into a speech recognition model.

[0019] Technical Effects

[0020] Based on the speech recognition apparatus and method according to the present disclosure, the user's speech can be effectively recognized by using a small amount of data, and the user's speech can be accurately recognized based on the smoothed energy component of the audio signal for each filter group. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Figure 1 is a schematic diagram of a method for recognizing a user's voice by an electronic device and a method for processing an audio signal for voice recognition according to an embodiment;

[0022] Figure 2 is a flow chart of a method for recognizing a user's voice by an electronic device according to an embodiment;

[0023] Figure 3 is a flowchart of a method for determining, by an electronic device, an energy component for each filter bank according to an embodiment;

[0024] Figure 4 is a flowchart of a method for smoothing, by an electronic device, an energy component for each filter bank by using a pre-trained smoothing coefficient according to an embodiment;

[0025] Figure 5 is a diagram for describing the smoothing coefficients trained for each filter bank channel.

[0026] Figure 6 is a flowchart of a method for smoothing an energy component for each filter bank by an electronic device by mapping a target histogram using a mapping function according to an embodiment;

[0027] Figure 7 is a diagram of energy components of a filter bank corresponding to a particular channel index according to an embodiment;

[0028] Figure 8 is a diagram of nonlinear outputs regarding filter bank energy according to an embodiment;

[0029] Fig. 9 is a flowchart of a method for processing an audio signal for speech recognition model training by an electronic device according to an embodiment;

[0030] Fig.10 is a schematic diagram of a process of processing an audio signal for speech recognition model training by an electronic device according to an embodiment;

[0031] Fig.11 is a flow chart of a method for transforming a frequency axis by an electronic device according to an embodiment;

[0032] Fig.12 is a diagram for describing a process of warping an input frequency according to a warping coefficient;

[0033] Fig.13 is a diagram of a method of resynthesizing, by an electronic device, a second audio signal by using a first audio signal whose frequency axis is transformed according to an embodiment;

[0034] Fig.14 is a block diagram of an electronic device according to an embodiment;

[0035] Fig.15 is a block diagram of an electronic device according to another embodiment;

[0036] Fig.16 is a diagram of a structure of an electronic device for training a speech recognition model according to an embodiment.

[0037] Fig.17 is a diagram of a process of training a speech recognition model by an electronic device according to an embodiment;

[0038] Fig.18 is a diagram of a process in which an electronic device according to an embodiment performs speech recognition by inputting a feature vector having smoothed energy components for each filter bank into a speech recognition model;

[0039] Fig.19 is a block diagram of a server according to an embodiment;

[0040] Fig. 20 is a diagram of a method of recognizing a user voice when an electronic device and a server are associated with each other according to an embodiment.

[0041] Fig.21 is a diagram of a method of recognizing a user voice when an electronic device and a server are associated with each other according to an embodiment.

[0042] Fig. 22 is a diagram of a method of recognizing a user's voice when an electronic device and a server are associated with each other according to an embodiment; and

[0043] Fig.23 is a diagram of a process of training a speech recognition model by an electronic device by using both a target speech signal and a noise signal according to an embodiment. DETAILED DESCRIPTION

[0044] Optimal Mode

[0045] According to one embodiment, a method for recognizing user voice includes: obtaining an audio signal segmented into multiple frame units, determining an energy component for each filter group by applying a filter group distributed according to a preset scale to the frequency spectrum of the audio signal segmented into frame units, smoothing the determined energy component for each filter group, extracting a feature vector of the audio signal based on the smoothed energy component for each filter group, and recognizing the user voice in the audio signal by inputting the extracted feature vector into a speech recognition model.

[0046] In addition, according to an embodiment, an electronic device for recognizing user voice includes: a memory storing one or more instructions; and a processor configured to execute the one or more instructions, wherein the processor is further configured to perform the following operations by executing the one or more instructions: obtaining an audio signal segmented into multiple frame units, determining an energy component for each filter group by applying a filter group distributed according to a preset scale to the frequency spectrum of the audio signal segmented into frame units, smoothing the determined energy component for each filter group, extracting a feature vector of the audio signal based on the smoothed energy component for each filter group, and recognizing the user voice in the audio signal by inputting the extracted feature vector into a speech recognition model.

[0047] In addition, according to an embodiment, a method for processing an audio signal, by which an electronic device trains a speech recognition model, includes: obtaining a first audio signal divided into multiple frame units for training the speech recognition model, obtaining a spectrum of the first audio signal divided into frame units, transforming the frequency axis of the spectrum of the first audio signal to represent changes in different vocal tract lengths of multiple speakers, generating a second audio signal by using the spectrum of the first audio signal with the frequency axis transformed, and extracting a feature vector of the generated second audio signal.

[0048] In addition, according to an embodiment, a method for training a speech recognition model by an electronic device includes: obtaining a first audio signal divided into multiple frame units for training the speech recognition model, obtaining a spectrum of the first audio signal divided into frame units, transforming the frequency axis of the spectrum of the first audio signal to represent changes in different vocal tract lengths of multiple speakers, generating a second audio signal by using the spectrum of the first audio signal with the frequency axis transformed, extracting a feature vector of the generated second audio signal, and training the speech recognition model by using the extracted feature vector.

[0049] Public Mode

[0050] The terms used in the specification are briefly described, and the present disclosure is described in detail.

[0051] Considering the functions in the present disclosure, the terms used in the present disclosure are selected from the common terms currently widely used. However, these terms may change according to the intentions of ordinary technicians in the field, case precedents, and the emergence of new technologies. In addition, for special cases, the meanings of the terms selected by the applicant are described in detail in the specification section. Therefore, the terms used in the present disclosure are defined based on their meanings related to the contents discussed throughout the specification, rather than being defined by their simple meanings.

[0052] When a component may "include" a specific constituent element, unless otherwise specified, it may not be interpreted as excluding another constituent element, but may be interpreted as further including other constituent elements. Terms such as "-part", "-unit", "-module" and the like stated in the specification may represent a unit for processing at least one function or operation, and the unit may be implemented by hardware, software, or a combination of hardware and software.

[0053] The embodiments are provided to further fully explain the present disclosure to those skilled in the art to which the present disclosure belongs. However, the present disclosure is not limited thereto, and it should be understood that various changes in form and detail may be made to the present disclosure without departing from the spirit and scope of the appended claims. In the accompanying drawings, parts not related to the description are omitted to clearly describe the present disclosure, and similar parts are represented by similar reference numerals throughout the specification.

[0054] Figure 1 2 is a schematic diagram of a method for recognizing a user's voice by an electronic device 1000 and a method for processing an audio signal for voice recognition according to an embodiment.

[0055] According to an embodiment, the electronic device 1000 may include a speech recognition model 3000. For example, the electronic device 1000 may obtain an audio signal 122, and may recognize a user's speech in the audio signal 122 by inputting a feature vector 132 extracted from the obtained audio signal 122 into the speech recognition model 3000. The speech recognition model 3000 used by the electronic device 1000 to recognize the user's speech may be a model based on a neural network for automatic speech recognition.

[0056] The electronic device 1000 according to an embodiment may include a smart phone, a tablet PC, a PC, a smart TV, a mobile phone, a personal digital assistant (PDA), a laptop computer, a media player, a server, a microserver, a global positioning system (GPS) device, an e-book terminal, a digital broadcast terminal, a navigation, an information kiosk, an MP3 player, a digital camera, a speaker, and other mobile or non-mobile computing devices including an AI program and a voice recognition function installed thereon, but the present disclosure is not limited thereto.

[0057] According to an embodiment, the electronic device 1000 may recognize a user voice in the audio signal 122 in association with a server 2000 connected to the electronic device via a network. According to an embodiment, the network may include a local area network (LAN), a wide area network (WAN), a value-added network (VAN), a mobile radio communication network, a satellite communication network, and a combination thereof. The server 2000 connected to the electronic device 1000 via a network may include at least one of different electronic devices capable of communicating with the electronic device 1000 according to the present disclosure.

[0058] According to an embodiment, the electronic device 1000 may pre-train the speech recognition model 3000. For example, the electronic device 1000 may enhance the audio signal used to train the speech recognition model according to at least one data enhancement technology, and train the speech recognition model based on the enhanced audio signal. In addition, the electronic device 1000 may smooth the energy component of the spectrum of the audio signal including the user's voice for each filter group, and input the feature vector extracted from the spectrum of the audio signal including the smoothed energy component for each filter group to the speech recognition model, thereby accurately recognizing the user's voice.

[0059] In the following description, a method of training a speech recognition model based on an enhanced audio signal by the electronic device 1000 and a method of recognizing a user's voice by inputting a feature vector extracted from a spectrum of an audio signal including a smoothed energy component for each filter group into the trained speech recognition model are briefly described.

[0060] For example, the electronic device 1000 may obtain a first audio signal 102 from a user, segment the first audio signal 102, and overlap the segmented first audio signals at predetermined intervals, thereby obtaining the first audio signal 102 of frame units overlapping adjacent frames at predetermined intervals. The first audio signal 102 may be a discrete signal obtained by sampling a continuous signal at a preset sampling frequency.

[0061] The electronic device 1000 can enhance the first audio signal 102 by using the first data enhancement (104) technique. According to an embodiment, the electronic device 1000 can enhance the audio signal by applying vocal tract length perturbation (VTLP) to the spectrum of the audio signal instead of the feature vector of the audio signal. In other words, even when a small amount of audio data is obtained, the electronic device 1000 can obtain audio data of speakers of various vocal tract lengths with enhancement based on a small amount of audio data by applying VTLP to the spectrum of the first audio signal 102.

[0062] In detail, the electronic device 1000 can obtain a spectrum by performing a fast Fourier transform (FFT) on the first audio signal 102 divided into frame units, and transform the frequency axis of the spectrum (frequency distortion) by using a distortion function defined based on a distortion coefficient to represent changes in different vocal tract lengths of multiple speakers.

[0063] According to an embodiment, the electronic device 1000 may obtain a frequency spectrum by applying various interpolation algorithms between FFT coefficients after performing a fast Fourier transform on the first audio signal 102. However, according to another embodiment, the electronic device 1000 may obtain a frequency spectrum by performing a super large FFT on the first audio signal 102 based on an FFT size coefficient preset according to the number of samples to be upsampled. The frequency spectrum may indicate the size of a complex number of each frequency component of the fast Fourier transformed first audio signal 102.

[0064] The electronic device 1000 may obtain a signal in the time domain by performing inverse FFT on a spectrum obtained from the first audio signal 102 whose frequency axis is transformed, and obtain a resynthesized signal 112 by connecting the obtained signals in the time domain. According to an embodiment, the electronic device 1000 may perform inverse FFT on the spectrum of the first audio signal whose frequency axis is transformed, and overlap and add the spectrum of the first audio signal subjected to inverse fast Fourier transformation on the time axis (overlap addition, 112), thereby obtaining the resynthesized signal 112. However, the method of connecting the inverse fast Fourier transformed time domain signals by the electronic device 1000 is not limited to the above description, and various other methods of connecting signals in the time domain may be used.

[0065] According to another embodiment, the electronic device 1000 may enhance the resynthesized signal 112 by using a second data enhancement (114) technique. According to an embodiment, the electronic device 1000 may apply a room impulse filter (e.g., a transfer function indicating the acoustic characteristics of each transmission path as a room impulse response) to the second audio signal 116 in a room in which the second audio signal 116 is transmitted to indicate the acoustic characteristics of each transmission path of the audio signal, and pre-train a speech recognition model based on a feature vector 118 extracted from the spectrum of the second audio signal 116 to which the room impulse filter is applied. In other words, the electronic device 1000 may obtain enhanced data representing changes in vocal tract lengths of multiple speakers by applying VTLP to the spectrum of the first audio signal, apply a room impulse response (RIR) filter to the enhanced data, and obtain a plurality of further enhanced data representing the characteristics of each transmission path of the audio signal in the rooms of various environments.

[0066] Therefore, even when a small amount of audio data is obtained, the electronic device 1000 can train a speech recognition model based on enhanced data of the small amount of data, thereby improving the accuracy of user speech recognition.

[0067] Conventionally, general electronic devices adjust the position of the Mel filter bank based on a warp function without applying VTLP to the spectrum, and extract Mel features based on the energy components of each Mel filter bank whose position is adjusted. Therefore, due to the unidirectionality of the energy conversion of each Mel filter bank, it is difficult for general electronic devices to apply the RIR filter after extracting the Mel features. However, according to an embodiment, when the electronic device 1000 applies VTLP to the spectrum before extracting the features of the audio signal, the RIP filter can be further applied to the resynthesized audio signal after applying VTLP, thereby effectively enhancing the audio data.

[0068] In addition, when the electronic device 1000 according to the present disclosure enhances the audio signal to represent the variation in the vocal tract length of different speakers, the order of applying VTLP to the spectrum of the audio signal and applying the RIR filter to the resynthesized audio signal further matches a series of transmission orders of the audio signal until the microphone receives the speech signals of speakers with different vocal tract lengths in the actual room. Therefore, the electronic device 1000 according to the present disclosure can obtain the feature vector 118 by applying both VTLP and RIR techniques to the first audio signal 102 according to the transmission order of the audio signal required for actual speech recognition, and train the speech recognition model based on the obtained feature vector, thereby further improving the accuracy of speech recognition of the speech recognition model.

[0069] In addition, despite Figure 1 Although not shown in the figure, the electronic device 1000 may obtain energy components for each filter group from the spectrum of the second audio signal 116 obtained by enhancing the first audio signal, smooth the obtained energy components for each filter group, and then train the speech recognition model 3000 based on feature vectors, wherein the feature vectors are extracted based on the smoothed energy components for each filter group. The electronic device 1000 may smooth the energy components of the Mel filter group that are generally concentrated in a specific frequency region, and may train the speech recognition model 3000 based on feature vectors extracted from the spectrum including uniformly distributed energy components, thereby improving the accuracy of speech recognition of the speech recognition model 3000.

[0070] When the electronic device 1000 smoothes the energy components of the second audio signal 116 for each filter group, and trains the speech recognition model 3000 by using the feature vector extracted based on the smoothed energy components for each filter group, as described below, the energy components of each Mel filter group of the newly obtained audio signal 122 are smoothed, and the feature vector 132 extracted based on the smoothed energy components for each filter group is input into the speech recognition model 3000, thereby recognizing the user voice in the audio signal 122.

[0071] The electronic device 1000 may obtain an audio signal 122, and recognize a user voice in the audio signal 122 obtained by using the pre-trained speech recognition model 3000. For example, the electronic device 1000 may obtain an audio signal 122 including a user voice, and segment the obtained audio signal 122 into frame units, thereby obtaining the audio signal 122 segmented into frame units.

[0072] In addition, the electronic device 1000 can extract the Mel feature 126 by applying a Mel filter group distributed according to a preset scale to the frequency spectrum of the audio signal segmented into frame units, and input the audio feature vector including the extracted Mel feature to the speech recognition model 3000, thereby recognizing the user voice in the audio signal.

[0073] In addition, despite Figure 1 Not shown, but when the electronic device 1000 smoothes the energy components of the second audio signal 116 for each filter group and uses the speech recognition model 3000 trained by using the feature vectors extracted based on the smoothed energy components for each filter group, the electronic device 1000 can smooth the energy components of each Mel filter group of the newly obtained audio signal 122, and input the feature vectors 132 extracted based on the smoothed energy components for each filter group into the speech recognition model 3000, thereby recognizing the user voice in the audio signal 122.

[0074] Figure 2 is a flowchart of a method for recognizing a user's voice by the electronic device 1000 according to an embodiment.

[0075] At S210, the electronic device 1000 may obtain an audio signal divided into a plurality of frame units. For example, the electronic device 1000 may determine a window length of a specific window, and divide the audio signal into a plurality of frame units by using a window having the determined window length. In addition, the electronic device 1000 may overlap windows at a predetermined window interval, and divide the audio signal into a plurality of frame units by using windows overlapped at a predetermined interval.

[0076] According to an embodiment, the window used by the electronic device 1000 may be used as a bandpass filter to remove noise included in the frequency component at each of the two side edges in the frame. According to an embodiment, the window may include a Hanning window or a Hamming window, but the present disclosure is not limited thereto. In addition, the window length may be about 30ms to about 50ms, but the present disclosure is not limited thereto, and the window length may vary according to the resolution of the target spectrum.

[0077] In addition, according to an embodiment, the length of the window used when the electronic device 1000 applies VTLP to the spectrum of the first audio signal, the length of the window used when the RIR filter is adopted, and the length of the window used when the user voice actually obtained in the audio signal is recognized after the training of the speech recognition model may be different from each other. According to another embodiment, the length of the window used when the electronic device 1000 applies VTLP to the spectrum of the first audio signal may be different from the length of the window used when the RIR filter is adopted and the length of the window used when the user voice actually obtained in the audio signal is recognized after the training of the speech recognition model.

[0078] At S220, the electronic device 1000 may determine the energy component for each filter bank by applying the filter banks distributed according to the preset scale to the frequency spectrum of the audio signal. Although the preset scale used by the electronic device 1000 to arrange the filter banks may include a logarithmic scale or a Mel scale, the present disclosure is not limited thereto, and the preset scale may also include other nonlinear scales to nonlinearly distribute the filter banks based on the human auditory characteristics that better distinguish low frequencies.

[0079] According to an embodiment, the filter bank distributed according to a preset scale may represent a plurality of bandpass filter arrays for dividing the spectrum of the audio signal into a plurality of sub-band spectrums. In addition, the energy component for each filter bank may represent the energy obtained by applying the filter bank to the spectrum of the audio signal or a probability density function (PDF) of the energy.

[0080] At S230, the electronic device 1000 may smooth the energy components for each filter group determined by applying the filter group distributed according to the preset scale to the spectrum. The electronic device 1000 may also accurately recognize the user's voice by inputting a feature vector including uniformly distributed energy components into a voice recognition model. Figure 4 and Figure 5 A method of smoothing the energy component for each filter bank by the electronic device 1000 is described in detail.

[0081] At S240, the electronic device 1000 may extract a feature vector of the audio signal based on the smoothed energy component for each filter bank. According to an embodiment, the electronic device 1000 may determine a DCT coefficient by performing a discrete cosine transform (DCT) on the smoothed energy component for each filter bank, and extract a feature vector having at least one DCT coefficient among the determined DCT coefficients as an element.

[0082] When the electronic device 1000 uses all determined DCT coefficients according to DCT, the performance of the speech recognition model may deteriorate due to the rapid change of the filter bank energy. Therefore, the electronic device 1000 can improve the accuracy of speech recognition by removing some DCT coefficients in the determined DCT coefficients and inputting a feature vector including the remaining DCT coefficients. According to an embodiment, the electronic device 1000 can determine the feature vector based on the 2nd to 13th DCT coefficients, but the present disclosure is not limited to this. However, the process of extracting the audio feature vector from the smoothed energy component for each filter bank by the electronic device 1000 is not limited to the method using the DCT coefficients, and the process can be performed based on other audio feature extraction methods.

[0083] For example, when the filter groups applied to the frequency spectrum of the audio signal by the electronic device 1000 are in a state of overlapping each other at a predetermined interval in the frequency domain, the energy components for each filter group may be correlated with each other. Therefore, the electronic device 1000 can separate the correlation between the energy for each filter group by performing discrete cosine transform on the energy components for each filter group, and input the extracted feature vector into the speech recognition model based on the separated independent energy for each filter group (the frequency band energy corresponding to the frequency band of the filter group), thereby improving the accuracy of speech recognition.

[0084] At S250, the electronic device 1000 may recognize the user's voice in the audio signal by inputting the extracted feature vector into a voice recognition model. Fig.18 A method of recognizing a user's voice based on a probability value of a tag including specific text information when the electronic device 1000 inputs the extracted feature vector into a voice recognition model is described in detail.

[0085] Figure 3 is a flowchart of a method of determining, by the electronic device 1000 , an energy component for each filter bank according to an embodiment.

[0086] At S320, the electronic device 1000 may apply a distributed filter bank to the frequency spectrum of the audio signal. For example, an operation of multiplying a function of each channel of the filter bank distributed according to a preset scale by a function of a frequency spectrum corresponding to a frequency band of the filter bank by the electronic device 1000 in the frequency domain may correspond to an operation of applying the distributed filter bank to the frequency spectrum of the audio signal by the electronic device 1000.

[0087] At S340, the electronic device 1000 may convert the values ​​of the spectrum to which the filter bank is applied into a logarithmic scale. The electronic device 1000 may apply a logarithmic function to each spectrum to represent human auditory characteristics (non-linear scale), thereby converting the values ​​of the spectrum to which the filter bank is applied into a logarithmic scale.

[0088] At S360, the electronic device 1000 may determine the energy component for each filter bank by using the value of the spectrum transformed to the logarithmic scale. For example, as at S320, the electronic device 1000 may square the product of the function of each channel of the filter bank distributed according to the preset scale and the function of the spectrum corresponding to the frequency band of the filter bank, and apply the logarithmic function to the square result, thereby determining the energy component for each filter bank.

[0089] Figure 4 is a flowchart of a method of smoothing, by the electronic device 1000 , an energy component for each filter bank by using a pre-trained smoothing coefficient according to an embodiment.

[0090] At S420, the electronic device 1000 may train a smoothing coefficient for smoothing the energy component for each filter group for each filter group. According to an embodiment, the electronic device 1000 may extract a feature vector (e.g., a feature according to logarithmic Mel or MFCC) based on the energy component for each filter group generated by applying a filter group distributed according to a logarithmic scale or a Mel scale to the spectrum of the audio signal to represent the nonlinear characteristics of the intensity between the energies of the filter groups perceived by humans.

[0091] According to another embodiment, in order to represent the relationship between the intensity of the energies of the filter groups perceived by humans, the electronic device 1000 may smooth all energy values ​​for each filter group by using a power function having a smoothing coefficient with a value of 1 / 15 as a power exponent, and extract a feature vector based on the smoothed energy components for each filter group, for example, based on the features of the power normalized cepstral coefficients (PNCC).

[0092] However, the electronic device 1000 according to the present disclosure can smooth the energy value for each filter group by training the smoothing coefficient for each filter group and using a power function with the trained smoothing coefficient as the exponent of the power to represent the nonlinear characteristics of the intensity between the energy of the filter group perceived by a person. The smoothing coefficient according to the present disclosure may include a power coefficient as the exponent of the power of the power function. In the following description, the process of training the smoothing coefficient for each filter group by the electronic device 1000 is described in detail. First, a variable Y of uniform distribution is defined based on the following mathematical expression 1.

[0093] [Mathematical expression 1]

[0094] Y=σ p (X) = (Xx min ) α

[0095] Y is the target uniformly distributed variable, X is x max and x minThe random variable between p (X) is a nonlinear function with variable X as input, and α is a smoothing coefficient. The interval of the uniformly distributed variable Y can be defined as I based on the minimum and maximum values ​​of the variable X. Y =[0,(X max -x min ) α ] IY is the interval of the uniformly distributed variable Y, x max and x min are the maximum and minimum values ​​of the interval of the variable X, and α is the smoothing coefficient. According to the interval of the uniform distribution variable Y, the uniform distribution variable Y is defined again as Mathematical Expression 2 by using the following uniform function.

[0096] [Mathematical expression 2]

[0097] Y~μ(0,(Xx min ) α )

[0098] As shown in Mathematical Expression 2, the uniformly distributed variable Y can be obtained by using an input (Xx min ) α The PDF of the uniformly distributed variable Y is defined by the uniform function μ(). Assuming that the uniformly distributed variable Y follows a uniform distribution, the PDF of the uniformly distributed variable Y can be expressed by the following mathematical expression 3.

[0099] [Mathematical expression 3]

[0100]

[0101] pY(y) is the PDF of a uniformly distributed variable Y, and the PDF of a uniformly distributed variable Y is in the range 0≤y≤(x max -x min ) α Uniform distribution within the range value, and may have a probability density of 0 in other ranges. Regarding the PDF of the uniformly distributed variable Y defined in Mathematical Expression 3, at x max and x min The PDF of a random variable X chosen between is defined as follows.

[0102] [Mathematical expression 4]

[0103]

[0104] To obtain the PDF pX(x) of variable x relative to the PDF of uniformly distributed variable Y, pY(y) is differentiated with respect to x. By differentiating pY(y) with respect to x, the PDF of variable x can be found at x. min ≤x≤x maxValues ​​in the range And may indicate a value of 0 within other ranges. In Mathematical Expression 4, a variable X may be defined as expressed in Mathematical Expression 5 below.

[0105] [Mathematical expression 5]

[0106] X={x 0 , x 1 , ...X N-1}

[0107] In other words, the variable X may be represented by a vector including a total of N random variables from 0 to N-1. In order to determine the smoothing coefficient α (maximum likelihood estimate (MLE)) that maximizes the log likelihood of the PDF of the variable X, the likelihood function of the variable X is defined as follows.

[0108] [Mathematical expression 6]

[0109]

[0110] L(α|X) is the log-likelihood of the smoothing coefficient α of the variable X, and px(x i ) is the PDF of the i-th variable X. L(α|X) can be represented by three logarithmic terms as in Mathematical Expression 6, and the last logarithmic term ln(x i -x min ) can not follow x i =x min Therefore, x can be expressed by using the lower limit coefficient δ i =x min L(α|X).

[0111] [Mathematical expression 7]

[0112]

[0113] L(α|X) is the log-likelihood of the smoothing coefficient α of the variable x, and when the variable x is x i =x min When To represent the logarithmic term. According to an embodiment, the lower limit coefficient may be 10^-100, but the present disclosure is not limited thereto. In order to determine the maximum likelihood of the smoothing coefficient α, L(α|X) is differentiated with respect to the smoothing coefficient α. Therefore, the smoothing coefficient of the variable X is The smoothing coefficient α that maximizes the log-likelihood of is as follows.

[0114] [Mathematical expression 8]

[0115]

[0116] As the smoothing coefficient that maximizes the log-likelihood of the smoothing coefficient α of the variable X, the maximum value x of the interval of the variable X can be used. max , minimum value x min The sum of the lower limit coefficient δ is expressed by Mathematical Expression 8.

[0117] In other words, as in mathematical expressions 1 to 8, assuming that the target variable Y is uniformly distributed, the electronic device 1000 may determine the logarithmic likelihood of the current variable X and the target variable Y, and determine the smoothing coefficient according to whether the determined logarithmic likelihood is maximized. In mathematical expressions 1 to 8, the target variable Y may correspond to the uniformly distributed energy component for each filter group, and the variable X may correspond to the energy component for each filter group of the currently received audio signal.

[0118] For example, the electronic device 1000 may determine the logarithmic likelihood of the energy component for each filter group (e.g., the PDF of the energy for each filter group) of the currently received audio signal relative to the uniformly distributed energy component for each filter group (e.g., the PDF of the energy for each filter group), and determine the smoothing coefficient based on whether the logarithmic likelihood is maximized. In other words, the electronic device 1000 may receive an audio training signal, and pre-train the smoothing coefficient for each filter group based on whether the logarithmic likelihood of the energy component for each filter group of the audio training signal is maximized relative to the uniformly distributed energy component for each filter group. In other words, the smoothing coefficient trained by the electronic device 1000 for each filter group may have different values ​​for each filter group, which is different from the PNCC feature extraction method, in which the same smoothing coefficient of 1 / 15 is applied to each filter group in all filter groups to represent the nonlinear characteristics of the intensity between the energies of the filter groups perceived by humans. The following refers to Figure 5 The smoothing coefficients trained to be different from each other for each filter bank are described.

[0119] Figure 5 is a diagram for describing the smoothing coefficients trained for each filter bank channel.

[0120] Unlike the PNCC in which the smoothing coefficient 606 of the same value (eg, 1 / 15) is applied to all of each filter group, the electronic device 1000 according to the present disclosure may train different smoothing coefficients for each filter group having different channels.

[0121] For example, the electronic device 1000 may receive an audio training signal that does not include a noise signal, and train the smoothing coefficient 602 to maximize the log likelihood of energy components of the audio training signal for each filter bank relative to uniformly distributed energy components for each filter bank.

[0122] Furthermore, the electronic device 1000 may receive an audio training signal including a noise signal, and may train the smoothing coefficient 604 to maximize the log likelihood of energy components of the audio training signal for each filter bank relative to uniformly distributed energy components for each filter bank.

[0123] In other words, despite Figure 5 As shown, the electronic device 1000 may determine the same smoothing coefficient (e.g., 1 / 15) for each filter bank channel to represent the nonlinear characteristics of the intensity between the energy of the filter bank perceived by humans, but the electronic device 1000 may train different smoothing coefficients for each filter bank based on an audio training signal that does not include a noise signal or an audio training signal that includes a noise signal. The electronic device 1000 according to the present disclosure may more accurately recognize the user's voice by smoothing the energy component of the currently received audio signal for each filter bank using the smoothing coefficients that are differently trained for each filter bank.

[0124] Figure 6 is a flowchart of a method of smoothing, by the electronic device 1000 , an energy component for each filter bank by mapping a target histogram using a mapping function according to an embodiment.

[0125] and Figures 4 to 5 Differently, the electronic device 1000 can obtain energy components for each filter group by applying a distributed filter group to the frequency spectrum of the audio signal, generate a histogram related to the size of the energy components for each filter group obtained, and smooth the energy components for each filter group of the audio signal based on a mapping function that maps the generated histogram to a target histogram.

[0126] In detail, at S520, the electronic device 1000 can obtain the energy component for each filter group by applying the filter group to the frequency spectrum of the audio signal, and generate a histogram related to the size of the energy component for each filter group obtained. For example, the electronic device 1000 can divide the size of the energy component for each filter group of the audio signal into a specific frequency domain (interval), and generate a histogram based on the size of the filter group energy component of each divided frequency domain. The frequency component may include a unit frequency domain or a frequency interval.

[0127] According to another embodiment, the electronic device 1000 may generate a histogram based on the frequency of each frequency component of the energy component of each filter group. For example, the electronic device 1000 may segment the spectrum of the energy component of each filter group obtained by applying the filter group to the audio signal (e.g., the PDF of the energy of each filter group) into a specific frequency domain (interval), and generate a histogram based on the frequency of the spectrum values ​​appearing in each segmented frequency domain.

[0128] At S540, the electronic device 1000 may determine a mapping function for mapping the generated histogram to a target histogram with a uniform distribution of the size of each frequency component. The target histogram may indicate the size of each frequency component in the energy component for each filter group of the audio signal, assuming that the value of the energy for each filter group of the audio signal with a uniform distribution of the energy component for each filter group is obtained. The mapping function may be determined from the following mathematical expression 9.

[0129] [Mathematical expression 9]

[0130]

[0131] σ np (X) is a nonlinear function having as input the histogram of variable X to map the histogram of variable X (e.g., the PDF of the energy component for each filter bank) to the histogram of variable Y, F u (·) is the probability density function of the uniformly distributed target histogram and The cumulative distribution function (CDF) of u -1 is the mapping function used to map the histogram of variable X to the target histogram.

[0132] At S560, the electronic device 1000 may convert the energy components for each filter group of the audio signal by using a determined mapping function. For example, the electronic device 1000 may input the filter group energy value of each frequency component in the filter group energy components of the currently received audio signal into the mapping function, and obtain the filter group energy value based on the output result of the mapping function to allow the size or frequency of each frequency component to indicate the target histogram, thereby converting the energy components for each filter group of the currently received audio signal. The electronic device 1000 may smooth the energy components for each filter group of the audio signal by converting the energy components for each filter group of the audio signal using a mapping function. The electronic device 1000 may extract a feature vector of the audio signal based on the smoothed energy component for each filter group, and recognize the user voice in the audio signal by using the extracted audio feature vector, thereby accurately recognizing the user voice. Since S560 can be combined with Figure 2S220 to S230 correspond to each other, and thus the detailed description of S560 is omitted.

[0133] Figure 7 is a diagram of energy components of a filter bank corresponding to a specific channel index according to an embodiment.

[0134] Reference Figure 7 , among the filter banks distributed according to the Mel scale by the electronic device 1000, shows the PDF of the Mel filter bank energy obtained by applying a filter bank with a channel index of 3 to a frequency spectrum.

[0135] For example, when the electronic device 1000 is not as described above Figure 4 and Figure 6 When the PDF of the Mel filter bank energy is smoothed as described in (702), the electronic device 1000 can obtain the PDF of the Mel filter bank energy concentrated in the frequency domain near 0 Hz. When the electronic device 1000 uses the feature vector extracted from the spectrum including the unevenly distributed energy components, the accuracy of the speech recognition of the electronic device 1000 may deteriorate.

[0136] However, according to an embodiment, when the electronic device 1000 smoothes the PDF of the filter bank energy with channel index I=3 by using a pre-trained smoothing coefficient (704), a PDF of the filter bank energy similar to a normal distribution shape from -1 to 1 can be obtained. According to another embodiment, when the electronic device 1000 smoothes the PDF of the filter bank energy with channel index I=3 by using a histogram-based mapping function (706), a PDF of the filter bank energy with uniform distribution can be obtained. The electronic device 1000 can accurately recognize the user's voice in the audio signal by smoothing the energy of each filter bank PDF of the audio signal.

[0137] Figure 8 is a diagram of nonlinear output with respect to filter bank energy according to an embodiment.

[0138] Although the electronic device 1000 may apply the same smoothing coefficient 1 / 15 to the energy component (708) for each filter group of the audio signal to represent the auditory characteristics of a person indicating a nonlinear feature, the electronic device 1000 may apply the smoothing coefficient trained differently for each filter group to the energy component (704) for each filter group of the audio signal by training, for each filter group, a smoothing coefficient that maximizes the log likelihood of the energy component for each filter group of the currently received audio signal relative to the uniformly distributed energy component for each filter group. In addition, the electronic device 1000 may generate a histogram related to the size of the frequency spectrum of the audio signal for each frequency component, and smooth the energy component (706) for each filter group of the audio signal based on a mapping function for mapping the generated histogram to a target histogram.

[0139] When the electronic device 1000 smoothes the energy component for each filter group of the audio signal by using the smoothing coefficient trained for each filter group (704), as the energy component of the filter group decreases, the nonlinear output can be sharply reduced (708) compared to the case where the energy component for each filter group is smoothed by using the same smoothing coefficient 1 / 15. However, when the electronic device 1000 smoothes the energy component for each filter group by using a histogram-based mapping function (706), the nonlinear output can remain uniform regardless of the change in the energy component of the filter group. The nonlinear output can indicate the degree of nonlinear characteristics of the energy for each filter group to represent the auditory characteristics of a person indicating the nonlinear characteristics.

[0140] Fig. 9 is a flowchart of a method for processing, by the electronic device 1000 , an audio signal for speech recognition model training according to an embodiment.

[0141] The electronic device 1000 may obtain an audio signal and input a feature vector extracted from the audio signal to a pre-trained speech recognition model, thereby recognizing a user's speech in the audio signal. In the following description, a method of processing an audio signal for a pre-trained speech recognition model by the electronic device 1000 is described.

[0142] At S910, the electronic device 1000 may obtain a first audio signal divided into a plurality of frame units for training a speech recognition model. For example, the electronic device 1000 may arrange Hanning windows having a preset window length to overlap at a predetermined interval, and obtain the first audio signal divided into a plurality of frame units by using the Hanning windows arranged to overlap each other. According to an embodiment, the length of the Hanning window may be about 50 ms, but the present disclosure is not limited thereto, and the length of the Hanning window may vary according to the target frequency resolution. Since S910 may be performed by Figure 2The electronic device 1000 obtains the audio signal segmented into a plurality of frame units, and thus a detailed description of S910 is omitted.

[0143] At S920, the electronic device 1000 may obtain a spectrum of the first audio signal divided into a plurality of frame units. For example, the electronic device 1000 may obtain a first audio signal of a plurality of frame units sampled at a frequency of 16 kHz with a sampling number K=1024. In order to obtain better frequency resolution during the transformation of the frequency axis (frequency distortion), the electronic device 1000 may set the ultra-large FFT coefficient U to 16, upsample the first audio signal according to the set ultra-large FFT coefficient U, and perform FFT on the upsampled first audio signal to obtain a spectrum. When the frequency axis of the spectrum having the FFT size coefficient K is transformed (frequency distorted), the audio signal The spectrum of the original signal with FFT size coefficients UK before transforming the frequency axis is The relationship is expressed by mathematical expression 10.

[0144] [Mathematical expression 10]

[0145]

[0146] m represents a frame index of the audio signal obtained by the electronic device 1000, represents the distortion function, and the discrete time frequency is The discrete time frequency is U represents an ultra-large FFT size coefficient, and K represents an FFT size coefficient.

[0147] Considering the symmetry of the frequency spectrum of the audio signal whose frequency axis is transformed, the frequency spectrum of the audio signal whose frequency axis is transformed can be defined as within the range In obtaining the mathematical expression 10 Since the spectrum of the original signal before the frequency axis transformation is can be defined only with respect to a discrete value k, so it can be defined by using a specific integer k 0 Relative to v k0 instruct Because v k0 is defined as The warp function can be defined as shown in Mathematical Expression 11:

[0148] [Mathematical expression 11]

[0149]

[0150] U represents the extra large FFT coefficient, K represents the FFT size coefficient, and represents the distortion function. Therefore, in Mathematical Expression 11, k is defined as shown in Mathematical Expression 12 0 .

[0151] [Mathematical expression 12]

[0152]

[0153] In mathematical expression 12, k 0 The value can be rounded up using the following mathematical expression 13.

[0154] [Mathematical expression 13]

[0155]

[0156] U represents the ultra-large FFT coefficient, K represents the FFT size coefficient, represents the warp function, and is a floor operator for obtaining the largest integer less than or equal to the input value in the function. 0.5 is a constant added after the decimal point for rounding. By using Mathematical Expressions 10 and 13, an audio signal whose frequency axis is transformed (frequency distorted) can be defined as shown in the following Mathematical Expression 14

[0157] [Mathematical expression 14]

[0158]

[0159] represents the frequency spectrum of the audio signal with the frequency axis transformed, represents the spectrum of the original signal before transforming the frequency axis. The discrete time frequency is and the discrete time frequency is

[0160] In S930, in order to represent the changes in the lengths of different vocal tracts of multiple speakers, the electronic device 1000 may use the warping function defined in mathematical expressions 10 and 14. To transform the frequency axis of the spectrum of the first audio signal. For example, the electronic device 1000 can transform the frequency axis of the spectrum of the first audio signal by using a warp coefficient and a warp function defined based on the warp coefficient (108). Fig.11 and Fig.12 A method of transforming the frequency axis of the frequency spectrum of the first audio signal performed by the electronic device 1000 is described in detail.

[0161] At S940, the electronic device 1000 may generate a second audio signal by using the frequency spectrum of the first audio signal whose frequency axis is transformed. For example, the electronic device 1000 may perform inverse FFT on the frequency spectrum of the first audio signal whose frequency axis is transformed, and overlap and add the frequency spectrum (112) of the first audio signal that is inversely fast Fourier transformed on the time axis, thereby obtaining a resynthesized second audio signal 116.

[0162] At S950, the electronic device 1000 may apply a room impulse filter indicating acoustic characteristics of the audio signal of each transmission path in the room to the second audio signal. For example, the room impulse filter as a transfer function may be a room impulse response. For example, the room impulse filter may indicate different acoustic characteristics of the audio signal of each transmission path in the room.

[0163] The acoustic characteristics indicated by the room impulse filter may include the reverberation time (wide distribution of reverberation) of each transmission path in the room, the room size of the audio signal of each transmission path, and the signal-to-noise ratio of each transmission path in the room, but the present disclosure is not limited thereto. In addition, the transmission path of the audio signal in the room may be defined based on the position of the sound source for generating the audio signal and the microphone for receiving the audio signal.

[0164] The operation of the electronic device 1000 applying the room impulse filter to the second audio signal may correspond to an operation of convolving the room impulse response with the second audio signal in the time domain or an operation of multiplying the second audio signal by the room impulse response in the frequency domain. The electronic device 1000 can simulate the situation of transmitting the audio signal in a room under various environments by applying the room impulse filter to the resynthesized second audio signal, and can obtain an audio signal representing the acoustic characteristics of a transmission path through which the audio signal is not actually transmitted. In other words, the electronic device 1000 can obtain an enhanced audio signal that is not actually obtained by applying the room impulse (RIR) filter to the second audio signal.

[0165] Fig.10 is a schematic diagram of a process of processing an audio signal for speech recognition model training by the electronic device 1000 according to an embodiment.

[0166] The electronic device 1000 may obtain an audio signal and enhance the obtained audio signal by using a specific data enhancement technique. For example, the electronic device 1000 may obtain a first audio signal 1002 including a user voice from a user and noise signals 1004, 1006, and 1008 from the first noise source to the (I-1)th noise source.

[0167] The electronic device 1000 may obtain a spectrum of a first audio signal 1002, and transform a frequency axis of the spectrum of the first audio signal 1002 by using a warp coefficient and a warp function defined based on the warp coefficient (frequency warping, 1010). The electronic device 1000 may multiply in the frequency domain the spectrum of the first audio signal 1002 whose frequency axis is transformed by the room impulse filter 1020. The room impulse filter 1020 is a room impulse response indicating an acoustic characteristic of a transmission path through which an audio signal is transmitted from a 0th sound source in a room to a jth microphone.

[0168] In addition, the electronic device 1000 may multiply the noise signals 1004, 1006, and 1008 obtained from the 1st to (I-1)th noise sources by a gain factor g to adjust the relative strengths between the noise signals, and multiply each noise signal multiplied by the gain factor by a room impulse filter in the frequency domain. The electronic device 1000 may obtain an enhanced audio signal y by adding both a result of applying the room impulse filter to the first audio signal whose frequency axis is transformed and a result of applying the room impulse filter 1020 to the result of multiplying the noise signals 1004, 1006, and 1008 by the gain factor g. j [n]. Enhanced audio signal j [n] can be defined by the following mathematical expression.

[0169] [Mathematical expression 15]

[0170]

[0171] y j [n] represents the audio signal obtained from the jth microphone, x 0 [n] represents the user voice signal to be recognized, h 0j [n] represents the room impulse response from the 0th noise source to the jth noise source, x i [n] represents the noise signal obtained from the i-th noise source in the range of 1≤i≤I-1, and h ij [n] indicates the noise signal x i [n] The room impulse response of the acoustic characteristics of the path through which it is transmitted from the i-th noise source to the j-th microphone. In addition, F(·|α) represents a warp function (frequency warp), and α represents a warp coefficient.

[0172] Typically, for VTLP for modeling variations in vocal tract length for different speakers, variations in vocal tract length for different speakers are modeled by adjusting the scale of a filter bank in the spectrum of an audio signal in the frequency domain rather than the time domain, but after the filter bank is applied during the VTLP signal processing process, it is difficult to apply an RIR filter through another data enhancement technique.

[0173] In the electronic device 1000 according to the present disclosure, before obtaining a first audio signal and extracting a feature vector from the obtained first audio signal, first, in order to model the variation in vocal tract length of different speakers, the frequency axis of the spectrum of the first audio signal is transformed, and the second audio signal is resynthesized by using the spectrum of the first audio signal whose frequency axis is transformed, so that RIR can be simultaneously applied through VTLP and acoustic simulation.

[0174] In addition, when the audio signal is enhanced by the electronic device 1000 according to the present disclosure, the order of resynthesizing the audio signal using the warp coefficient and the warp function to represent the variation of the vocal tract length of different speakers and applying the RIR filter to the resynthesized audio signal can represent the transmission order of a series of audio signals, that is, the speech signals of speakers with different vocal tract lengths are actually received by the microphone in the room. Therefore, the electronic device 1000 according to the present disclosure can train a speech recognition model with improved accuracy of speech recognition by resynthesizing the audio signal using the warp coefficient and the warp function to represent the variation of the vocal tract length of different speakers and then applying the RIR filter to the resynthesized audio signal.

[0175] Fig.11 is a flowchart of a method of transforming a frequency axis performed by the electronic device 1000 according to an embodiment.

[0176] Refer to the following Fig.11 The process of transforming the frequency axis of the spectrum of the audio signal by the electronic device 1000 to model the variation of the vocal tract length of different speakers is described in detail. At S1112, the electronic device 1000 determines a warp coefficient. For example, the electronic device 1000 may randomly determine the warp coefficient of each frame of the first audio signal.

[0177] At S1114, the electronic device 1000 may determine a warp function for transforming the frequency axis of the spectrum of the first audio signal based on the determined warp coefficient. The electronic device 1000 may determine the warp function based on a split linear rule or a bilinear rule to transform the frequency axis of the spectrum. First, the warp function according to the split linear rule may be defined by the following mathematical expression.

[0178] [Mathematical expression 16]

[0179]

[0180] α represents the warping coefficient, and the warping function defines the input frequency W and the frequency-warped W' as the output frequency k For example, when the input frequency W satisfies When the frequency distortion output frequency W' kis Wα, and the frequency distortion output frequency W' in other ranges k can be The electronic device 1000 can determine the input frequency W as the input and the output frequency W' according to the segmentation linear rule and based on the mathematical expression 16. k The warp function as output.

[0181] According to another embodiment, the electronic device 1000 may determine a warp function based on a bilinear rule, and transform a frequency axis of the spectrum of the first audio signal according to the determined warp function.

[0182] [Mathematical expression 17]

[0183]

[0184] W k represents the input frequency, and W' k The electronic device 1000 can determine the output frequency with frequency W according to the bilinear rule and based on the mathematical expression 17. k As input and output frequency W' k Warp function as output. At S1116, the electronic device 1000 may determine a warp function based on Mathematical Expression 16 or 17, and transform a frequency axis of a frequency spectrum of the first audio signal by using the determined warp function.

[0185] Fig.12 is a diagram for describing a process of warping an input frequency according to a warping coefficient.

[0186] The electronic device 1000 may determine the warp coefficient and transform the frequency axis of the spectrum of the first audio signal by using a warp function defined based on the determined warp coefficient. Fig.12 , the electronic device 1000 determines the warping coefficient to be between 0.8 and 1.2, and performs frequency warping on the input frequency by using a warping function according to a bilinear rule and based on the determined warping coefficient.

[0187] For example, when the electronic device 1000 sets the warp coefficient to 0.8, the distribution of the output frequency (e.g., the warp frequency) output from the warp function according to the bilinear rule may indicate a frequency distribution that is convex downward relative to the input frequency. According to another embodiment, when the electronic device 1000 sets the warp coefficient to 1, the distribution of the output frequency output from the warp function may indicate a linear distribution.

[0188] In addition, when the electronic device 1000 sets the warp coefficient to 1.2, the distribution of the output frequency output from the warp function may indicate a frequency distribution in an upward convex shape. In other words, the electronic device 1000 may define a warp function according to a bilinear rule or a split linear rule based on the warp coefficient to represent different vocal tract lengths of the speaker, and transform the frequency axis of the spectrum of the first audio signal by using the defined warp function, thereby modeling the change in the vocal tract length of the speaker.

[0189] Fig.13 is a diagram of a method of resynthesizing, by the electronic device 1000, a second audio signal by using a first audio signal whose frequency axis is transformed according to an embodiment.

[0190] At S1302, the electronic device 1000 may perform inverse FFT (IFFT) on the frequency axis transformed (frequency distorted) spectrum of the first audio signal. At S1304, the electronic device 1000 may generate a second audio signal as a resynthesized first audio signal by overlapping and adding the spectrum of the first audio signal subjected to inverse fast Fourier transformation on the time axis.

[0191] For example, the electronic device 1000 obtains an audio signal divided into frame units by dividing the audio signal using a window having a specific window length, and transforms the audio signal for each frame in the frequency domain. In other words, when the electronic device 1000 transforms the audio signal of each frame in the frequency domain, when the spectrum of the audio signal transformed in the frequency domain is inversely transformed in the time domain, a resynthesized audio signal can be obtained by overlapping the spectrum of the audio signal transformed in the time domain and adding the overlapped audio signal in the time domain.

[0192] Fig.14 is a block diagram of an electronic device according to an embodiment. Fig.15 is a block diagram of an electronic device according to another embodiment.

[0193] like Fig.14 As shown, the electronic device 1000 according to the embodiment may include a processor 1400 and a memory 1402. However, all the components shown are not necessary components. The electronic device 1000 may be implemented by more or less components than the components shown. For example, Fig.15 As shown, the electronic device 1000 according to the embodiment may further include a user input unit 1502 , a communication unit 1504 , a microphone 1506 , a display 1507 , and a speaker 1508 .

[0194] The processor 1400 may control the overall operation of the electronic device 1000 by executing one or more instructions in the memory 1402. For example, the processor 1400 may generally control the user input unit 1502, the communication unit 1504, the microphone 1506, the display 1507, the speaker 1508, etc. by executing one or more instructions stored in the memory 1402. In addition, the processor 1400 may execute one or more instructions stored in the memory 1402 to perform Figures 1 to 12 Functions of the electronic device 1000.

[0195] In addition, the processor 1400 may include one or more processors, and the one or more processors may include a general-purpose processor (such as a CPU, an AP, a digital signal processor (DSP), etc.), a graphics-specific processor (such as a GPU or a visual processing unit (VPU)), or an artificial intelligence (AI)-specific processor (such as an NPU). According to an embodiment, when the processor 1400 is implemented by a plurality of processors, a graphics-specific processor, or an AI-specific processor such as an NPU, at least some of the plurality of processors, graphics-specific processors, or AI-specific processors such as an NPU may be installed on the electronic device 1000 and other electronic devices or a plurality of servers connected to the electronic device 1000.

[0196] For example, the processor 1400 may obtain an audio signal segmented into a plurality of frame units by executing one or more instructions, determine an energy component for each filter group by applying a filter group distributed according to a preset scale to the frequency spectrum of the audio signal segmented into frame units, smooth the determined energy component for each filter group, extract a feature vector of the audio signal based on the smoothed energy component for each filter group, and input the extracted feature vector into a speech recognition model, thereby recognizing the user voice in the audio signal.

[0197] For example, the processor 1400 may determine a window length of a window used to segment an audio signal into multiple frame units by executing one or more instructions, overlap windows of the determined window length at a predetermined window interval, and segment the audio signal into multiple frame units by using the overlapping windows.

[0198] Furthermore, the processor 1400 may apply a distributed filter bank to the spectrum of the audio signal, convert values ​​of the spectrum to which the filter bank is applied into a logarithmic scale, and determine an energy component for each filter bank by using the values ​​of the spectrum converted into the logarithmic scale.

[0199] In addition, the processor 1400 may train a smoothing coefficient for smoothing the energy component for each filter group based on a uniformly distributed target energy component for each filter group, and smooth the energy component for each filter group by using the smoothing coefficient trained for each filter group.

[0200] In addition, the processor 1400 can generate a histogram related to the size of the spectrum of the audio signal for each frequency component, determine a mapping function for mapping the generated histogram to a target histogram with a uniform distribution of the size of each frequency component, transform the spectrum of the audio signal by using the determined mapping function, and apply a filter group distributed according to a preset scale to the transformed spectrum of the audio signal, thereby smoothing the energy component determined for each filter group.

[0201] According to another embodiment, the processor 1400 may process an audio signal for training a speech recognition model. For example, in order to train a speech recognition model, the processor 1400 may obtain a first audio signal divided into a plurality of frame units, obtain a spectrum of the first audio signal divided into frame units to represent a change in different vocal tract lengths of a plurality of speakers, transform a frequency axis of the spectrum of the first audio signal, generate a second audio signal by using the spectrum of the first audio signal whose frequency axis is transformed, and extract a feature vector from the generated second audio signal.

[0202] According to another embodiment, the processor 1400 may train a speech recognition model for recognizing a user's voice by using a feature vector extracted from the resynthesized second audio signal.

[0203] According to an embodiment, the processor 1400 may apply a room impulse filter indicating acoustic characteristics of the second audio signal of each transmission path in the room to the second audio signal, and extract a feature vector from the second audio signal to which the room impulse filter is applied.

[0204] In addition, the processor 1400 may determine a warp coefficient randomly generated for each frame, determine a warp function for transforming a frequency axis of a spectrum of the first audio signal based on the determined warp coefficient, and transform the frequency axis of the spectrum of the first audio signal by using the determined warp function.

[0205] Also, the processor 1400 may perform inverse FFT on the frequency axis-transformed spectrum of the first audio signal and overlap the inverse fast Fourier transformed spectrum of the first audio signal on the time axis, thereby generating the second audio signal.

[0206] In addition, the processor 1400 can determine the energy component for each filter group by applying the filter group distributed according to a preset scale to the frequency spectrum of the second audio signal divided into multiple frame units, smooth the determined energy component for each filter group, and extract the feature vector of the second audio signal based on the smoothed energy component for each filter group.

[0207] According to an embodiment, in order to represent the changes in different vocal tract lengths of multiple speakers, a speech recognition model pre-trained by the processor 1400 may be pre-trained based on a feature vector of an audio training signal resynthesized by using the spectrum of the audio training signal, wherein the frequency axis of the spectrum of the audio training signal obtained for each frame unit is transformed in the audio training signal.

[0208] In addition, the speech recognition model may be pre-trained based on a resynthesized audio training signal to which a room impulse filter is applied, the room impulse filter indicating the acoustic characteristics of the audio signal of each transmission path in the room for transmitting the audio signal. Fig.17 and 18 The structure of the speech recognition model and the training method thereof used by the electronic device 1000 according to the present disclosure are described in detail.

[0209] The memory 1402 may include one or more instructions for controlling the operation of the electronic device 1000. In addition, the memory 1402 may include one or more speech recognition models for recognizing the user's voice in the audio signal. In addition, the programs stored in the memory 1402 may be classified into a UI module, a touch screen module, an alarm module, etc. according to their functions.

[0210] According to an embodiment, the memory 1402 may include at least one type of storage medium, such as a flash memory type, a hard disk type, a multimedia card micro, a card type memory (e.g., an SD or XD memory, etc.), a random access memory (RAM), a static RAM (SRAM), a read-only memory (ROM), an electrically erasable programmable ROM (EEPROM), a programmable ROM (PROM), a magnetic memory, a magnetic disk, an optical disk, etc., but the present disclosure is not limited thereto.

[0211] The user input unit 1502 may receive a user input for controlling the operation of the electronic device 1000. For example, the user input unit 1502 may include a keyboard, a dome switch, a touch pad (such as a contact capacitance method, a pressure-resistant film method, an infrared detection method, a surface ultrasonic conduction method, an overall tension measurement method, a piezoelectric effect method, etc.), a scroll wheel, a micro switch, etc., but the present disclosure is not limited thereto.

[0212] The communication unit 1504 may include one or more communication modules for communicating with the server 2000. For example, the communication unit 1504 may include at least one of a short-range wireless communication unit or a mobile communication unit.

[0213] The short-range wireless communication unit may include a Bluetooth communication unit, a Bluetooth low energy (BLE) communication unit, a near field communication unit, a WLAN (Wi-Fi) communication unit, a Zigbee communication unit, an infrared data association (IrDA) communication unit, a Wi-Fi Direct (WFD) communication unit, an ultra-wideband (UWB) communication unit, an Ant+ communication unit, etc., but the present disclosure is not limited thereto.

[0214] The mobile communication unit can transmit / receive wireless signals with at least one of a base station, an external terminal and a server in a mobile communication network. The wireless signals may include various types of data transmitted / received according to voice call signals, video call signals or text / multimedia message transmission / reception.

[0215] Microphone 1506 may receive an audio signal including a user's voice. In addition, microphone 1506 may receive an audio signal including a noise signal generated from a plurality of noise sources other than the user's voice. Microphone 1506 may send the obtained audio signal to processor 1400 so that voice recognition through a voice recognition model is performed.

[0216] The speaker 1508 may output the recognition result of the user voice in the audio signal as an audio signal. For example, the speaker 1508 may output a signal related to a function performed in the electronic device 1000 (eg, a call signal reception sound, a message reception sound, and an alarm sound) as a sound.

[0217] The display 1507 may output and display information processed in the electronic device 1000. For example, the display 1507 may display the recognition result of the user's voice in the audio signal in the form of text. The display 1507 may display an interface for controlling the electronic device 1000, an interface for displaying the status of the electronic device 1000, and the like.

[0218] Fig.16 is a diagram of a structure of an electronic device for training a speech recognition model according to an embodiment.

[0219] According to an embodiment, the electronic device 1000 may train a speech recognition model by using a CPU cluster 1640 including a plurality of central processing units (CPUs) 1642 and 1644 and a GPU cluster 1620 including a plurality of graphic processing units (GPUs) 1622, 1624, and 1626. Generally, a CPU consumes a large amount of computing resources to apply VTLP and acoustic simulation to an audio signal and extract features from an audio signal resynthesized by the VTLP and acoustic simulation, and the electronic device 1000 may efficiently perform VTLP and acoustic simulation by using a cluster including a plurality of CPUs, and extract feature vectors from an audio signal resynthesized by the VTLP and acoustic simulation.

[0220] In addition, the electronic device 1000 may send the feature vector output from the CPU and the text information corresponding to the feature vector to the GPU node 1612, so that each GPU cluster trains the speech recognition model in parallel, wherein the GPU node 1612 includes a plurality of GPUs and a plurality of CPUs for controlling the GPUs. In the following description, the process of training the speech recognition model by using a plurality of CPUs and GPUs is described in detail below.

[0221] First, the electronic device 1000 may receive multiple audio signals by using the CPU cluster 1640. The audio signal received by the electronic device 1000 may be an audio signal in a time domain that is randomly divided. For example, the SharedTFreCords 1652 received by the electronic device 1000 may include an audio signal that is randomly selected rather than selected in chronological order. In addition, the ShardedTFRecords 1652 may include audio signals received from multiple speakers and divided (sliced) into specific frame units. For example, the ShardedTFRecords 1652 may include an audio signal such as "Hello" received from speaker A and an audio signal such as "Hi, Bixby" received from speaker B.

[0222] The electronic device 1000 may randomly split an audio signal such as “Hello” received from speaker A and an audio signal such as “Hi, Bixby” received from speaker B, and temporarily store the split audio signals in the example queue 1654. The electronic device 1000 may send the split audio signals stored in the example queue 1654 to a plurality of CPUs in the CPU cluster 1640. The electronic device 1000 may sequentially send the split audio signals stored in the example queue 1654 to the CPUs, or may send the split audio signals stacked in the queue to the CPUs in a random order.

[0223] Each CPU in the CPU cluster 1640 may receive an audio signal, transform a frequency axis of a spectrum of the received audio signal by using a warp function, and perform inverse transformation on the spectrum including the transformed frequency axis, thereby obtaining a resynthesized audio signal (VTLP). In addition, each CPU in the CPU cluster 1640 may apply acoustic simulation by applying a room impulse filter indicating an acoustic characteristic of the audio signal of each transmission path in the room (e.g., indicating the acoustic characteristic of each transmission path as a transfer function of a room impulse response) to the resynthesized audio signal.

[0224] In addition, each CPU in the CPU cluster 1640 may determine an energy component for each filter group by applying a filter group distributed according to a preset scale to a spectrum of an audio signal obtained by acoustic simulation, and extract a feature vector from a spectrum in which the determined energy component for each filter group is smoothed (MUD based on a power function). Each CPU in the CPU cluster 1640 may obtain text information corresponding to the feature vector from tags 1662 and 1664 including text information corresponding to the extracted feature vector, and send the obtained text information together with the extracted feature vector to the CPU 1628 in the GPU node 1612.

[0225] The CPU 1628 in the GPU node 1612 may send the feature vectors and text information corresponding to the feature vectors received from the CPU cluster 1640 to the plurality of GPUs, and the speech recognition model may be trained by each GPU. According to an embodiment, the CPU 1628 in the GPU node 1612 may determine priority information for sending the feature vectors and text information corresponding to the feature vectors received from the CPU cluster 1640 to the GPU, and send the feature vectors and text information corresponding to the feature vectors to the GPU based on the determined priority information.

[0226] The GPU in GPU node 1612 may train a speech recognition model based on the received feature vector and text information. For example, GPU 1622, GPU 1624, and GPU 1626 may determine an inference output about the speech recognition model based on the feature vector and text information, and determine a loss by comparing an output value of the inference output with a correct value, thereby training the speech recognition model. The process of determining an inference based on the feature vector and text information may be similar to the following reference to Fig.17 and Fig.18 The decoding process in the speech recognition model described above corresponds to the following. Fig.17 and 18 The method of training a speech recognition model by the electronic device 1000 is described in detail.

[0227] According to an embodiment, the CPU 1628 in the GPU node 1612 may determine the current status of the plurality of GPUs in the GPU node 1612 and send information about the status of the GPUs as feedback information to the CPU cluster 1640. The CPU cluster 1640 may adjust the transmission speed of the feature vector and text information output by the CPU 1628 by using the feedback information received from the GPU cluster 1620.

[0228] In other words, the electronic device 1000 according to the present disclosure may include a GPU that is relatively faster than a CPU, and determine a device ratio between the CPU and the GPU, thereby effectively training a speech recognition model. The electronic device 1000 may extract a feature vector from an audio signal that has not performed VTLP and acoustic simulation, and determine a device ratio between the CPU and the GPU based on a speed value of training a speech recognition model by using the extracted feature vector.

[0229] Fig.17 is a diagram of a process of training a speech recognition model by an electronic device according to an embodiment.

[0230] According to an embodiment, the electronic device 1000 may pre-train a speech recognition model for recognizing a user's voice in an audio signal. The speech recognition model used by the electronic device 1000 is an automatic speech recognition (ASR) model, and may be an end-to-end attention-based neural network model. However, the speech recognition model used by the electronic device 1000 is not limited to an end-to-end attention-based neural network model, and may include all other neural network-based models required for training the speech recognition model. The speech recognition model (an attention-based neural network model) used by the electronic device 1000 may include: an encoder including a plurality of long short-term memory (LSTM) layers and a maximum pooling layer for encoding a feature vector of an audio signal; a decoder including an LSTM layer for outputting text by decoding the encoded feature vector; an attention layer for setting attention to a feature vector output from the encoder based on a correlation between the encoder and the decoder; and a soft maximum layer for outputting a probability that a feature vector weighted by attention indicates a label about a specific keyword. However, as described above, the speech recognition model used by the electronic device 1000 is not limited to Fig.17 The structure of the speech recognition model and may include all other neural network based models for speech recognition.

[0231] For example, the electronic device 1000 may obtain an audio signal divided into a plurality of frame units and a distortion coefficient. Although the electronic device 1000 may obtain the distortion coefficient from a user or an external device connected to the electronic device 1000, the electronic device 1000 itself may generate the distortion coefficient for each frame of the audio signal. The electronic device 1000 may determine a distortion function by using the distortion coefficient generated for each frame, transform the frequency axis of the spectrum of the audio signal by using the determined distortion function, and overlap and add the frequency-distorted spectrum again in the time domain, thereby generating a resynthesized audio signal.

[0232] In other words, the electronic device 1000 may perform VTLP processing (1702) for representing the vocal tract lengths of multiple speakers relative to the frequency spectrum of the audio signal. According to another embodiment, the electronic device 1000 may ensure various data for training the speech recognition model by further performing acoustic simulation on the resynthesized audio signal through the VTLP processing.

[0233] The electronic device 1000 can extract a feature vector from the resynthesized audio signal through VTLP. For example, the electronic device 1000 can determine the energy component for each filter group by applying the filter group distributed according to the Mel scale to the resynthesized spectrum of the audio signal, and smooth the energy component for each filter group by applying a power function having a smoothing coefficient of 1 / 15 as an exponent of the power to the determined energy component for each filter group. In addition, the electronic device 1000 can extract a feature vector based on the smoothed energy component for each filter group by using a power function having an exponent of the smoothing coefficient of 1 / 15 as an exponent of the power (1704).

[0234] The feature vector extracted by the electronic device 1000 includes m-dimensional frequency features and can be formed in units of frames. In other words, one frame 1706 can be represented by a feature vector including m-dimensional frequency features. The electronic device 1000 can sequentially input the feature vectors extracted in units of frames into the speech recognition model.

[0235] The encoder 1720 of the speech recognition model may include a plurality of LSTM layers and a maximum pooling layer. The encoder 1720 may transform the dimensions of the frames by applying nonlinearity to a plurality of frames input sequentially, and output only some of the input frames by selecting frames having the transformed dimensions among the frames at a predetermined ratio.

[0236] For example, it is assumed that the encoder 1720 includes four pairs of LSTM layers 1721 and maximum pooling layers 1722. The LSTM layers 1721 and the maximum pooling layers 1722 may form a single layer in the encoder 1720, and thus the encoder 1720 may include a total of four single layers. For example, when eight frames having m dimensions are sequentially input to the encoder 1720, each frame may be transformed into a dimension different from the m dimension by applying nonlinearity to each frame by the LSTM layer 1721, and the maximum pooling layer 1722 receiving the dimension-transformed frame may select a frame having a larger audio feature between the audio features of the two frames.

[0237] Therefore, the encoder 1720 including a total of four layers may receive eight frames in m dimensions and output a frame having a dimension different from the m dimensions of the eight frames of m dimensions received. Because the dimension 1732 of the frame output from the encoder 1720 receiving the input of the eight frames of m dimensions is unknown due to the nonlinear characteristics of the LSTM layer 1721, the dimension 1732 may be in a hidden dimension state. According to another embodiment, when an audio signal of 1000 frames segmented into 40 dimensions is input to the encoder 1720, since eight poolings are obtained through four maximum pooling layers, the encoder 1720 may output 1000 / 8 frames of unknown dimension.

[0238] The attention layer 1740 may set attention to the feature vector output from the encoder 1720 based on the correlation between the encoder 1720 and the decoder 1760. For example, the attention layer 1740 may set attention between the feature vectors output from the encoder 1720 by using the feature vector 1732 in the hidden dimension state output from the encoder 1720 and the feature vector 1734 in the hidden dimension state output from the decoder 1760 at the previous time and the attention weight feedback determined in the attention layer 1740.

[0239] For example, the attention layer 1740 may determine the attention weight feedback based on the feature vector of the previous time output from the encoder 1720 and the feature vector 1734 of the previous time output from the decoder 1760. The attention layer 1740 may apply attention to the feature vector output from the encoder 1720 by using the attention weight feedback, and determine the context feature vector 1744 by weighted summing the feature vectors to which the attention is applied. In other words, the context feature vector 1744 output by the attention layer 1740 may be a weighted sum of the frames to which the attention weight is applied, which are output from the encoder 1720. Alternatively, the context feature vector 1744 may be some frames selected from the frames output from the encoder 1720 based on the attention weight determined by the attention layer 1740.

[0240] The decoder 1760 may decode the weighted sum feature vector output from the attention layer 1740. For example, the decoder 1760 may include one LSTM layer. The decoder 1760 may transform the dimension of the context feature vector 1742 at the previous time by applying nonlinearity to the context feature vector 1742 at the previous time based on the context feature vector 1744 at the current time, the weighted sum feature vector output from the attention layer 1740, and the recurrent state 1762 of the LSTM in the decoder 1760. Since the dimension of the feature vector output from the decoder 1760 is unknown due to the nonlinear characteristics of the LSTM, the dimension may be in a hidden dimension state.

[0241] The soft maximum layer 1780 may determine a probability value that a feature vector in a hidden dimension state output from the decoder 1760 corresponds to a specific text label. For example, the soft maximum layer 1780 may receive an input of a feature vector in a hidden dimension state output from the decoder 1760 at a previous time, a context feature vector 1744 at a current time output from the attention layer 1740, and text information output from the soft maximum layer 1780 at a previous time, and determine a probability value that a feature vector output from the decoder 1760 corresponds to a specific text label such as "Hi, Bixby" or "Hello".

[0242] According to an embodiment, the soft maximum layer 1780 may determine a probability value corresponding to a specific text label of a feature vector of a hidden dimension state output from the decoder 1760 without receiving an input of a context feature vector 1744 of the current time output from the attention layer 1740 and text information output from the soft maximum layer 1780 at a previous time. The soft maximum layer 1780 may output text information 1782 of a text label corresponding to a highest probability value among probability values ​​of the text labels. A speech recognition model having the above structure may be pre-trained by modifying and refining attention regarding connection strengths between the encoder 1720, the decoder 1760, the attention layer 1740, and the soft maximum layer 1780 based on a feature vector extracted from a resynthesized audio signal through VTLP processing.

[0243] Fig.18 2 is a diagram of a process in which the electronic device 1000 performs speech recognition by inputting a feature vector having smoothed energy components for each filter bank to a speech recognition model according to an embodiment.

[0244] The electronic device 1000 can accurately recognize the user's voice by smoothing the spectrum of the energy component of the audio signal for each filter group, and inputting the feature vector extracted from the spectrum with the smoothed energy component for each filter group into the pre-trained voice recognition model.

[0245] For example, in order to apply the power nonlinearity 1804 to the spectrum of the energy component for each filter group of the audio signal, the electronic device 1000 may determine a smoothing coefficient for each filter group, and smooth the energy component for each filter group by using a power function having an exponent of the smoothing coefficient as a power. According to another embodiment, the electronic device 1000 may smooth the energy component for each filter group of the audio signal by using a mapping function determined based on a histogram related to the size or frequency of the spectrum of the audio signal for each frequency component. The processing of the electronic device 1000 applying the power nonlinearity 1804 to the energy component for each filter group of the spectrum may be similar to Figures 4 to 7 The processing is accordingly.

[0246] The electronic device 1000 may extract a feature vector 1806 from the spectrum having the smoothed energy component for each filter group, and input the extracted feature vector 1806 into the pre-trained speech recognition model. Fig.17 The structure and operation method of the speech recognition model correspond to those of the embodiment, so their detailed description is omitted.

[0247] Fig.19 is a block diagram of a server 2000 according to an embodiment.

[0248] According to an embodiment, the server 2000 may include a communication unit 2100 , a database (DB) 2200 , and a processor 2300 .

[0249] The communication unit 2100 can communicate with Fig.15 The communication unit 1504 of the electronic device 1000 corresponds. For example, the communication unit 2100 may receive an audio signal from the electronic device 1000, or transmit information about a speech recognition result for an audio signal received from the electronic device 1000 to the electronic device 1000. In addition, the communication unit 2100 may transmit / receive an audio signal received from the electronic device 1000 or information about a speech recognition result for a shared audio signal to other servers by communicating with a communication unit in another server.

[0250] DB 2200 can be used with Fig.15 The memory 1402 corresponds to the DB 2200. For example, the DB 2200 may store a speech recognition model. In addition, the DB 2200 may provide a speech recognition service using the speech recognition model, and may also store a data enhancement algorithm and a feature extraction algorithm required for training the speech recognition model.

[0251] The processor 2300 may generally control the overall operation of the server 2000. For example, the processor 2300 may generally control the DB 2200, the communication unit 2100, etc. by executing a program stored in the DB 2200 of the server 2000. In addition, the processor 2300 may execute a program stored in the DB 2200. Figures 1 to 18 part of the operation of the electronic device 1000.

[0252] For example, in order to represent the change of different channel lengths of multiple speakers, the processor 2300 can obtain an audio signal from the electronic device 1000, transform the frequency axis of the spectrum of the audio signal, and generate a resynthesized audio signal by using the spectrum of the audio signal whose frequency axis is transformed. In addition, the processor 2300 can represent different acoustic characteristics of the audio signal of each transmission path in the room by applying RIR to the resynthesized audio signal.

[0253] In addition, in order to improve the accuracy of speech recognition of the speech recognition model, the processor 2300 can smooth the energy component of the audio signal resynthesized for each filter group, and extract feature vectors from the spectrum including the smoothed energy component for each filter group. For example, the processor 2300 can pre-train the smoothing coefficient for each filter group, and smooth the energy of each filter group of the audio signal by using a trained power function with a smoothing coefficient as an exponent of the power. According to another embodiment, the processor 2300 can smooth the energy component of each filter group of the audio signal by using a histogram related to the size or frequency of the spectrum of the audio signal of each frequency component and a mapping function of the histogram.

[0254] Fig. 20 is a diagram of a method of recognizing a user voice when an electronic device and a server are associated with each other according to an embodiment.

[0255] At S2002, the electronic device 1000 may obtain a first audio signal. At S2004, the electronic device 1000 may send the obtained first audio signal to the server 2000. According to an embodiment, the electronic device 1000 may segment the first audio signal into a plurality of frame units by using a window having a preset window length, and send the first audio signal segmented into frame units to the server 2000.

[0256] At S2006, the server 2000 may transform the frequency axis of the spectrum of the received first audio signal (perform frequency distortion). For example, the server 2000 may receive a predetermined distortion coefficient from the electronic device 1000, and determine a distortion function based on the distortion coefficient according to Mathematical Expression 15 or 16. The server 2000 may represent changes in the vocal tract lengths of multiple speakers by transforming the frequency axis of the spectrum of the first audio signal using the distortion coefficient and the distortion function.

[0257] At S2008, the server 2000 may resynthesize the first audio signal by using the spectrum of the first audio signal whose frequency axis is transformed. For example, the server 2000 may generate a resynthesized audio signal (VTLP) by performing inverse FFT on the spectrum of the first audio signal whose frequency axis is transformed and overlapping and adding the spectrum of the first audio signal that is inversely fast Fourier transformed on the time axis. According to an embodiment, the server 2000 may also generate an audio signal indicating different acoustic characteristics of the audio signal of each transmission path in the room by applying an RIR filter to the resynthesized audio signal.

[0258] At S2010, the server 2000 may extract a feature vector from the resynthesized audio signal. According to an embodiment, the server 2000 may extract a feature vector from the resynthesized audio signal to which the RIR filter is applied. In addition, according to an embodiment, at S2010, the server 2000 may smooth the energy component of the resynthesized audio signal for each filter group, and extract a feature vector based on the smoothed energy component for each filter group. The method of extracting the resynthesized feature vector by the server 2000 may be similar to the method of extracting the resynthesized feature vector by the electronic device 1000 from Figures 2 to 7 The process of extracting the feature vector of the re-synthesized audio signal is accordingly omitted.

[0259] At S2012, the server 2000 may train the speech recognition model by using the extracted feature vector. According to an embodiment, when the server 2000 smoothes the energy component for each filter group of the resynthesized audio signal and extracts the feature vector based on the smoothed energy component for each filter group, the server 2000 may extract the feature vector from the audio signal having the smoothed energy component for each filter group and train the speech recognition model by using the extracted feature vector.

[0260] When the server 2000 smoothes the energy components for each filter group by using the feature vector extracted based on the smoothed energy components for each filter group and trains the speech recognition model 3000, in the recognition of the user's voice in the newly obtained audio signal, the server 2000 can smooth the energy components of each Mel filter group of the newly obtained audio signal 122, and input the feature vector 132 extracted based on the smoothed energy components for each filter group into the speech recognition model, thereby recognizing the user's voice. The process of training the speech recognition model by the server 2000 using the extracted feature vector can be similar to the process of training the speech recognition model by the electronic device 1000. Fig.17 and Fig.18 The processing of the speech recognition model in .

[0261] At S2014, the electronic device 1000 may obtain a second audio signal from the user. At S2016, the electronic device 1000 may transmit the obtained second audio signal to the server 2000. At S2020, the server 2000 may extract a feature vector of a spectrum of the second audio signal, wherein the second audio signal has smoothed energy components for each filter group. The operation of extracting the feature vector by the server 2000 may correspond to the operation of Figure 2 In S240, the electronic device 1000 performs an operation of extracting a feature vector.

[0262] In addition, according to an embodiment, when the server 2000 smoothes the energy component for each filter group of the first audio signal and uses the speech recognition model trained by using the feature vector extracted based on the smoothed energy component for each filter group, at S2020, the server 2000 may smooth the energy component for each filter group of the spectrum of the second audio signal and extract the feature vector from the second audio signal including the smoothed energy component for each filter group. The operation of smoothing the energy component for each filter group of the spectrum of the second audio signal by the server 2000 may be the same as Figures 4 to 7 The electronic device 1000 smoothes the energy components of the audio signal for each filter bank accordingly.

[0263] At S2022, the server 2000 may recognize the user's voice in the second audio signal by inputting the extracted feature vector into the pre-trained voice recognition model. At S2024, the server 2000 may send the voice recognition result output by the voice recognition model to the electronic device 1000. At S2026, the electronic device 1000 may output the voice recognition result received from the server 2000.

[0264] Fig.21 is a diagram of a method of recognizing a user voice when an electronic device and a server are associated with each other according to an embodiment.

[0265] According to an embodiment of the present disclosure, a method of recognizing a user's voice and a method of processing an audio signal for training a voice recognition model may be performed by an electronic device and a plurality of servers associated with the electronic device.

[0266] At S2102, the electronic device 1000 may obtain a first audio signal. At S2104, the electronic device 1000 may send the first audio signal to the first server 2500. According to another embodiment, the electronic device 1000 may send the first audio signal to the second server 2600 so that the first audio signal may be sent to the first server 2500 via the second server 2600.

[0267] At S2106, the first server 2500 may transform the frequency axis of the spectrum of the first audio signal (perform frequency warping) by using a predetermined warping coefficient and a warping function according to Mathematical Expression 15 or 16. At S2108, the first server 2500 may resynthesize the audio signal by using the spectrum of the audio signal whose frequency axis is transformed, thereby performing VTLP processing on the first audio signal. At S2110, the first server 2500 may transmit the resynthesized first audio signal to the second server 2600. In other words, according to an embodiment, Fig.21 The first server 2500 may be a dedicated server on the network for enhancing audio data.

[0268] At S2112, the second server 2600 may extract a feature vector from the resynthesized first audio signal received from the first server 2500. According to an embodiment, the second server 2600 may smooth the energy component of the first audio signal resynthesized for each filter group, and extract the feature vector based on the smoothed energy component for each filter group. The operation of extracting the feature vector from the resynthesized first audio signal by the second server 2600 may be the same as that by the first server 2500 at S2010. Fig. 20 The first server extracts a feature vector from the resynthesized audio signal in response.

[0269] At S2114, the second server 2600 may train the speech recognition model by using the feature vector. According to an embodiment, when the second server 2600 smoothes the resynthesized energy component of the first audio signal for each filter group and extracts the feature vector based on the smoothed energy component for each filter group, the second server 2600 may train the speech recognition model by using the feature vector extracted from the first audio signal having the smoothed energy component for each filter group. The operation of training the speech recognition model by the second server 2600 by using the feature vector may be the same as the operation of training the speech recognition model by the second server 2600 in S2112. Figure 2The server 2000 trains the speech recognition model accordingly.

[0270] At S2116, the electronic device 1000 may obtain a second audio signal. At S2118, the electronic device 1000 may send the second audio signal to the second server 2600. According to an embodiment, the electronic device 1000 may send the second audio signal to the second server 2600 via the first server 2500. At S2120, the second server 2600 may smooth the energy component of the second audio signal for each filter group. S2120 is a process of smoothing the energy component of the audio signal for each filter group, wherein, in S2120, Figures 4 to 7 The electronic device 1000 may determine whether the log-likelihood of the energy component for each filter group of the audio signal is maximized, and S2120 may correspond to a process of smoothing the energy component for each filter group by using a power function having a determined smoothing coefficient as an exponent of the power.

[0271] At S2122, the second server 2600 may extract a feature vector from a second audio signal having a smoothed energy component for each filter group. According to an embodiment, when the second server 2600 smoothes the energy component for each filter group of the first audio signal and uses a speech recognition model trained by using a feature vector extracted based on the smoothed energy component for each filter group, the second server 2600 may recognize the user voice in the second audio signal by smoothing the energy component of each Mel filter group of the newly obtained second audio signal and inputting the feature vector extracted based on the smoothed energy component for each filter group into the speech recognition model. At S2124, the second server 2600 may recognize the user voice in the second audio signal by inputting the extracted feature vector into the speech recognition model. At S2126, the second server 2600 may send the speech recognition result output by the speech recognition model to the electronic device 1000. At S2128, the electronic device 1000 may output the speech recognition result received from the second server 2600.

[0272] As described above, the method for recognizing a user voice can be performed by using a plurality of servers including a first server 2500 and a second server 2600, wherein the first server 2500 enhances the obtained audio signal, the second server 2600 receives the enhanced audio signal from the first server 2500, trains a voice recognition model based on the enhanced audio signal, and recognizes the user voice in the second audio signal by using the trained voice recognition model.

[0273] Fig. 22 is a diagram of a method of recognizing a user voice when an electronic device and a server are associated with each other according to an embodiment.

[0274] According to an embodiment of the present disclosure, the electronic device 1000 may accurately recognize voices of a plurality of speakers associated with a plurality of servers.

[0275] For example, at S2202, the electronic device 1000 may obtain a first audio signal including a child's voice. At S2204, the electronic device 1000 may send the first audio signal including the child's voice to the first server 2500. At S2206, the first server 2500 may perform VTLP processing on the first audio signal including the child's voice. Fig.11 and Fig.12 , an operation of performing VTLP processing on the first audio signal including a child's voice by the first server 2500 may correspond to an operation of transforming a frequency axis of a spectrum of the audio signal by the electronic device 1000 by using a warp coefficient and a warp function including the warp coefficient.

[0276] According to an embodiment, the first server 2500 may also apply the RIR filter to the resynthesized first audio signal through VTLP processing. At S2208, the first server 2500 may extract a feature vector from the resynthesized audio signal through VTLP processing. According to an embodiment, the first server 2500 may smooth the energy component of each filter group of the resynthesized audio signal through VTLP processing, and extract a feature vector based on the smoothed energy component for each filter group.

[0277] At S2210, the first server 2500 may train the first speech recognition model by using the extracted feature vector. According to an embodiment, the first server 2500 may extract the feature vector from the resynthesized audio signal by VTLP processing to which an RIR filter is applied, and train the first speech recognition model by using the extracted feature vector. According to another embodiment, when the first server 2500 smoothes the energy component of the resynthesized audio signal for each filter group and extracts the feature vector based on the smoothed energy component for each filter group, the first server 2500 may train the speech recognition model by using the feature vector extracted from the resynthesized audio signal having the smoothed energy component for each filter group.

[0278] At S2212, the first server 2500 may obtain a second audio signal including a child's voice. At S2214, the first server 2500 may transmit the second audio signal including the child's voice to the first server 2500. At S2216, in order to allow energy components for each filter group of the spectrum of the second audio signal to be distributed according to maximum uniformity, the first server 2500 may smooth the energy components for each filter group of the second audio signal by using a power function having a smoothing coefficient pre-trained for each filter group as an exponent of the power.

[0279] At S2218, the first server 2500 may recognize a child's voice in a second audio signal including a child's voice by inputting the extracted feature vector into a first speech recognition model. According to an embodiment, when the first server 2500 smoothes the energy component for each filter group of the first audio signal including the child's voice and uses a speech recognition model trained by using a feature vector extracted based on the smoothed energy component for each filter group, the first server 2500 may recognize a child's voice in the second audio signal by smoothing the energy component for each Mel filter group of the second audio signal including the newly obtained child's voice and inputting the feature vector extracted based on the smoothed energy component for each filter group into the speech recognition model. At S2220, the first server 2500 may send a speech recognition result output from the speech recognition model to the electronic device 1000. In S2221, the electronic device 1000 may output the speech recognition result received from the first server 2500.

[0280] At S2222, the electronic device 1000 may obtain a first audio signal including an adult voice. At S2224, the electronic device 1000 may transmit the first audio signal including an adult voice to the second server 2600. According to an embodiment, the electronic device 1000 may transmit the first audio signal including an adult voice to the second server 2600 through the first server 2500. At S2226, the second server 2600 may perform VTLP processing on the first audio signal including an adult voice. The operation of performing VTLP processing on the first audio signal including an adult voice by the second server 2600 may be similar to the operation of performing VTLP processing on the first audio signal including an adult voice by the second server 2600. Fig.11 and Fig.12 The electronic device 1000 may transform a frequency axis of a frequency spectrum of an audio signal by using a warp coefficient and a warp function including the warp coefficient.

[0281] According to an embodiment, the second server 2600 may further apply the RIR filter to the first audio signal including the adult voice resynthesized by the VTLP process. At S2228, the second server 2600 may extract a feature vector from the first audio signal including the adult voice resynthesized by the VTLP process. According to an embodiment, the second server 2600 may extract a feature vector from the first audio signal including the adult voice resynthesized by the VTLP process. Fig. 22 , but at S2228, the second server 2600 may smooth the energy components for each filter group of the second audio signal including the adult voice resynthesized by the VTLP process, and extract the feature vector based on the smoothed energy components for each filter group. In other words, in order to allow the energy components for each filter group of the spectrum of the second audio signal including the voice to be distributed according to maximum uniformity, the second server 2600 may smooth the energy components for each filter group of the second audio signal by using a power function having a smoothing coefficient pre-trained for each filter group as an exponent of the power.

[0282] At S2230, the second server 2600 may train the second speech recognition model by using the extracted feature vector. According to an embodiment, the second server 2600 may extract a feature vector from the second audio signal including the adult speech having the smoothed energy component for each filter bank, and train the second speech recognition model by using the extracted feature vector.

[0283] At S2232, the electronic device 1000 may obtain a second audio signal including an adult voice. At S2234, the electronic device 1000 may send the second audio signal including the adult voice to the second server 2600. At S2236, the second server 2600 may extract a feature vector from the second audio signal including the adult voice. According to an embodiment, in order to allow the energy component for each filter group of the spectrum of the second audio signal to be distributed according to maximum uniformity, the second server 2600 may smooth the energy component for each filter group of the second audio signal including the adult voice, and extract a feature vector from the second audio signal including the adult voice by using a power function having a smoothing coefficient pre-trained for each filter group as an exponent of the power, wherein the adult voice has a smoothed energy component for each filter group.

[0284] At S2238, the second server 2600 may recognize the adult voice in the second audio signal including the adult voice by inputting the feature vector into the second speech recognition model. According to an embodiment, when the second server 2600 smoothes the energy component for each filter group of the second audio signal including the adult voice and uses the speech recognition model trained by using the feature vector extracted based on the smoothed energy component for each filter group, the second server 2600 may recognize the adult voice in the second audio signal by smoothing the energy component for each Mel filter group of the newly obtained second audio signal including the adult voice and inputting the feature vector extracted based on the smoothed energy component for each filter group into the speech recognition model.

[0285] At S2240, the second server 2600 may transmit the speech recognition result output from the second speech recognition model to the electronic device 1000. At S2242, the electronic device 1000 may output the speech recognition result received from the second server 2600. In other words, the electronic device 1000 according to the present disclosure may accurately recognize the speech of multiple speakers associated with multiple servers, each server including a speech recognition model for recognizing the speech of a different speaker.

[0286] Fig.23 is a diagram of a process of training a speech recognition model by using both a target speech signal and a noise signal by the electronic device 1000 according to an embodiment.

[0287] According to an embodiment, the electronic device 1000 may train a speech recognition model so that the speech recognition model recognizes the target speech signal and the noise signal by distinguishing the target speech signal and the noise signal. At S2302, the electronic device 1000 may obtain a small amount of short-range speech database (DB) about the target speech signal. The short-range speech database (DB) may include a small amount of speech signals related to the target speech signal represented by various speech features (such as pronunciation duration, speech frequency, pitch, tone, etc.).

[0288] At S2304, the electronic device 1000 may perform VTLP processing on a small number of target voice signals in a small number of short-range voice databases. The operation of performing VTLP processing on voice signals related to the target voice signals in a small number of short-range voice databases by the electronic device 1000 may be similar to the operation of performing VTLP processing on the voice signals related to the target voice signals in the small number of short-range voice databases. Fig.11 and Fig.12 The operation of transforming the frequency axis of the frequency spectrum of the audio signal by the electronic device 1000 by using the warp coefficient and the warp function including the warp coefficient corresponds to the above.

[0289] At S2306, the electronic device 1000 may also apply a RIR filter to a speech signal related to the target speech signal resynthesized by the VTLP process. In other words, the electronic device 1000 may model the target speech signal sent in the virtual room by applying the acoustic features of the audio signal for each transmission path in the virtual room including the remote environment to the speech signal related to the target speech signal resynthesized by the VTLP process. The electronic device 1000 may obtain a target speech 2308 for training by further applying a RIR filter to a speech signal related to the target speech signal resynthesized by the VTLP process.

[0290] The electronic device 1000 can train the speech recognition model 3000 of the electronic device 1000 by using the obtained target speech 2308 for training. Fig.23 As shown in the figure, the electronic device 1000 may smooth the energy components of the signal of the target speech 2308 for training for each filter group, and train the speech recognition model 3000 by using feature vectors extracted based on the smoothed energy components for each filter group.

[0291] At S2314, the electronic device 1000 may perform VTLP processing on the noise signal in the small amount of short-range voice database. The operation of performing VTLP processing on the noise signal in the small amount of short-range voice database by the electronic device 1000 may be similar to the operation of performing VTLP processing on the noise signal in the small amount of short-range voice database. Fig.11 and Fig.12 The electronic device 1000 may transform a frequency axis of a frequency spectrum of an audio signal by using a warp coefficient and a warp function including the warp coefficient.

[0292] At S2316, the electronic device 1000 may also apply a RIR filter to a speech signal related to the noise signal resynthesized by the VTLP process. In other words, the electronic device 1000 may model the noise signal sent in the virtual room by applying the acoustic features for each transmission path of the noise signal in the virtual room including the remote environment to the speech signal related to the noise signal resynthesized by the VTLP process. The electronic device 1000 may obtain a noisy speech 2318 for training by further applying a RIR filter to a speech signal related to the noise signal resynthesized by the VTLP process.

[0293] The electronic device 1000 can train the speech recognition model 3000 of the electronic device 1000 by using the obtained noise speech 2318 for training. Fig.23Although not shown, the electronic device 1000 may smooth the energy components of the signal of the noisy speech 2318 for training for each filter group, and train the speech recognition model 3000 by using feature vectors extracted based on the smoothed energy components for each filter group.

[0294] The electronic device 1000 can accurately recognize the user's voice by using the speech recognition model 3000 obtained based on the target speech 2308 for training and the noise speech 2318 for training, wherein the target speech 2308 for training and the noise speech 2318 for training are obtained by respectively enhancing a small amount of target speech signal and a small amount of noise signal. For example, even when the target speech signal 2322 is obtained from the user 2332 and the noise signal is obtained from the noise source 2334 (e.g., TV, speaker, etc.) at the same time, the electronic device 1000 can accurately recognize the user's voice by using the speech recognition model 3000 trained based on the target speech 2308 for training and the noise speech 2318 for training, by accurately distinguishing the target speech signal 2322 of the user 2332 from the noise signal.

[0295] The method for recognizing user voice by an electronic device, the method for processing an audio signal for training a voice recognition model, and the method for training a voice recognition model by an electronic device according to the above-mentioned embodiments can be implemented in the form of program instructions executed by various computer devices, and can be recorded on a computer-readable medium. The computer-readable medium may include program instructions, data files, data structures, etc., alone or by combining program instructions, data files, data structures, etc. The program commands recorded on the medium may be specially designed and configured for the present disclosure, or may be available program commands known to a person of ordinary skill in the art to which the present disclosure belongs, such as computer software. In addition, a computer program product including a recording medium may be provided by at least a portion of the content displayed on the electronic device, on which a program is stored that allows the electronic device to execute a method for providing a virtual device associated with the electronic device.

[0296] Computer-readable recording media may include magnetic media (such as hard disks, floppy disks, and magnetic tapes), optical media (such as CD-ROMs or DVDs), magneto-optical media (such as optical floppy disks), and hardware devices (such as ROM, RAM flash memory) that are specially configured to store and execute program commands. Examples of program commands may include not only machine codes created by a compiler, but also high-level programming languages ​​that can be executed by a computer using an interpreter.

[0297] In addition, the embodiments of the present disclosure may be embodied in the form of a recording medium including computer executable instructions (such as a program module executed by a computer). Computer readable storage media may be available media that can be accessed by a computer and may include all volatile and non-volatile media and detachable and non-detachable media.

[0298] For example, the device-readable storage medium may be provided in the form of a non-transitory storage medium. A "non-transitory storage medium" is a tangible device and simply means that there is no signal, such as an electromagnetic wave. Such a term does not distinguish between a case where data is semi-permanently stored in a storage medium and a case where data is temporarily stored in a storage medium. For example, a "non-transitory storage medium" may include a buffer that temporarily stores data.

[0299] According to an embodiment, the method according to various embodiments disclosed in this specification may be provided by being included in a computer program product. The computer program product as a commodity may be traded between a seller and a buyer. For example, the computer program product may be sold in the form of a device-readable storage medium (e.g., a compact disc read-only memory (CD-ROM)) through an application store (e.g., PlayStore). TM ) distribution, or directly online between two user devices (e.g., smartphones). For online distribution, at least a portion of the computer program product (e.g., a downloadable application) may be at least temporarily stored in or temporarily generated from a device-readable storage medium (such as a memory of a manufacturer's server, an application store's server, or a relay server).

[0300] Therefore, the embodiments disclosed in the present disclosure are intended to illustrate the scope of the technical concept of the present disclosure, and the scope of the technical concept of the present disclosure is not limited by the embodiments. The protection scope of the present disclosure should be interpreted based on the attached claims, and should be interpreted as all technical ideas included in the scope equivalent to the claims are included in the correct scope of the present disclosure.

Claims

1. A method for recognizing a user's voice, the method comprising: Obtaining an audio signal divided into a plurality of frame units; determining an energy component for each filter bank by applying a filter bank distributed according to a preset scale to a frequency spectrum of the audio signal segmented into the plurality of frame units; smoothing the determined energy components for each filter bank; extracting a feature vector of the audio signal based on the smoothed energy component for each filter bank; as well as The extracted feature vector is input into the speech recognition model to recognize the user's voice in the audio signal. The step of obtaining the audio signal divided into a plurality of frame units comprises: Determining the window length of the window for dividing the audio signal into the plurality of frame units according to the resolution of the frequency spectrum of the audio signal, Among them, the window length of the window used when the vocal tract length perturbation VTLP is applied to the spectrum of the audio signal used for training, the window length of the window used when the RIR filter is applied to the spectrum of the audio signal used for training, and the window length of the window used when the speech recognition model is used to recognize the user speech actually obtained in the audio signal are different from each other.

2. The method according to claim 1, wherein: The steps of obtaining an audio signal include: Overlapping windows having a determined window length at a predetermined window interval; and The audio signal is segmented into the plurality of frame units by using overlapping windows.

3. The method according to claim 1, wherein: The steps of determining the energy components for each filter bank include: Applying a distributed filter bank to the frequency spectrum of an audio signal; converting the values ​​of the spectrum to which the filter bank is applied to a logarithmic scale; and The energy component for each filter bank is determined by using the values ​​of the spectrum converted to a logarithmic scale.

4. The method according to claim 1, wherein: The step of smoothing the determined energy components for each filter bank comprises: For each filter bank, training a smoothing coefficient for smoothing the energy component for each filter bank based on a uniformly distributed target energy component; and The energy component for each filter bank is smoothed by using the smoothing coefficients trained for each filter bank.

5. The method according to claim 1, wherein: The step of smoothing the determined energy components for each filter bank comprises: generating a histogram associated with the magnitude of an energy component of the audio signal for each filter bank; determining a mapping function for mapping the generated histogram to a target histogram, wherein in the target histogram, the magnitudes of the energy components for each filter bank are uniformly distributed; and The energy component for each filter bank of the audio signal is smoothed by converting the energy component for each filter bank using a determined mapping function.

6. The method according to claim 1, wherein: The steps of extracting the feature vector of the audio signal include: Determining DCT coefficients by performing a discrete cosine transform (DCT) on the smoothed energy components for each filter bank; and A feature vector including at least one of the determined DCT coefficients as an element is extracted.

7. The method according to claim 1, wherein: The speech recognition model is pre-trained based on a feature vector of an audio training signal resynthesized by using a spectrum of the audio training signal, wherein in the spectrum of the audio training signal, a frequency axis of the spectrum of the audio training signal obtained for each frame unit is transformed to represent changes in different vocal tract lengths of multiple speakers.

8. The method according to claim 7, wherein: A frequency axis of a spectrum of the audio training signal is transformed based on a warp coefficient randomly generated for each frame and a warp function for transforming the frequency axis on the spectrum of the audio training signal based on the warp coefficient.

9. The method according to claim 7, wherein: A speech recognition model is pre-trained based on the resynthesized audio training signal to which a room impulse filter is applied, the room impulse filter indicating acoustic features of the audio signal for each transmission path in a room from which the audio signal originates.

10. The method according to claim 7, wherein: The resynthesized audio training signal is generated by performing inverse fast Fourier transform on the frequency axis transformed spectrum of the audio training signal and overlapping with the inverse fast Fourier transformed spectrum of the audio training signal on the time axis.

11. An electronic device for recognizing a user's voice, the electronic device comprising: a memory storing one or more instructions; as well as a processor configured to execute the one or more instructions, The processor is further configured to perform the following operations by executing the one or more instructions: Obtaining an audio signal divided into a plurality of frame units; determining an energy component for each filter bank by applying a filter bank distributed according to a preset scale to a frequency spectrum of the audio signal segmented into the plurality of frame units; smoothing the determined energy components for each filter bank; extracting a feature vector of the audio signal based on the smoothed energy component for each filter bank; and The extracted feature vector is input into the speech recognition model to recognize the user's voice in the audio signal. The step of obtaining the audio signal divided into a plurality of frame units comprises: Determining the window length of the window for dividing the audio signal into the plurality of frame units according to the resolution of the frequency spectrum of the audio signal, Among them, the window length of the window used when the vocal tract length perturbation VTLP is applied to the spectrum of the audio signal used for training, the window length of the window used when the RIR filter is applied to the spectrum of the audio signal used for training, and the window length of the window used when the speech recognition model is used to recognize the user speech actually obtained in the audio signal are different from each other.

12. The electronic device according to claim 11, wherein: The processor is further configured to perform the following operations by executing the one or more instructions: For each filter group, a smoothing coefficient for smoothing the energy component for each filter group is trained based on the uniformly distributed target energy component; as well as The energy component for each filter bank is smoothed by using the smoothing coefficients trained for each filter bank.

13. The electronic device according to claim 11, wherein: The processor is further configured to perform the following operations by executing the one or more instructions: generating a histogram associated with the magnitude of an energy component of the audio signal for each filter bank; Determining a mapping function for mapping the generated histogram to a target histogram, wherein in the target histogram, the magnitudes of the energy components for each filter bank are uniformly distributed; as well as The energy component for each filter bank of the audio signal is smoothed by converting the energy component for each filter bank using a determined mapping function.

14. The electronic device according to claim 11, wherein: A speech recognition model is pre-trained based on a feature vector of an audio training signal resynthesized by using a spectrum of the audio training signal, wherein a frequency axis of the spectrum of the audio training signal obtained for each frame unit is transformed to represent changes in different vocal tract lengths of multiple speakers.

15. A computer-readable recording medium having recorded thereon a program for executing the method defined in claim 1 on a computer.

Images