Device for recognizing multi-channel input speech independent of microphone array configuration and method for learning thereof

Abstract

A multi-channel input speech recognition device independent of the shape of a microphone array is provided. The device comprises: a time-frequency converter that receives a plurality of channel audio signals extracted from voice data recorded through a plurality of microphones in an unspecified shape of a microphone array and converts them into a plurality of time-frequency domain signals; a speaker and noise mask estimator that receives the plurality of time-frequency domain signals and estimates a time-frequency domain mask for the voices and noise of a plurality of speakers; a beamformer estimator that estimates a time-frequency domain signal for the voices of a plurality of speakers from which noise has been removed using the time-frequency domain mask; a time-frequency inverse converter that inversely converts the noise-removed time-frequency domain signal into a time domain signal; and a learner that learns the speaker and noise mask estimator based on a loss function obtained through a comparison result between the inversely converted time domain signal and a predefined correct signal.

Description

Technology Field

[0001] The present invention relates to a multi-channel input speech recognition device independent of the microphone array type and a learning method thereof. Background Technology

[0002] Speech recognition technology has achieved significant performance improvements recently due to advancements in deep learning technology and is being applied and used in various fields. However, recognition performance often deteriorates in complex real-world environments where various sounds occur simultaneously. In particular, in situations where multiple sounds occur at the same time, such as environments where multiple speakers speak, there is a significant decline in recognition performance, including the inability to accurately distinguish the voices of each speaker.

[0003] To overcome these limitations, deep learning-based multi-channel input speech recognition techniques utilizing microphone arrays composed of multiple microphones have recently been proposed. This technique is a technology that uses sounds recorded from each microphone simultaneously to consider and utilize spatial information of the real environment. In other words, by considering the location where the sound originated, it enables the extraction and recognition of only the voice signal desired by the user. Through this, it has become possible to extract and recognize the desired voice signal even in environments with ambient noise.

[0004] However, existing multi-channel input speech recognition technology has the problem of being dependent on a specific type of microphone array. For example, assuming the data used to train a multi-channel input speech recognition model is a 4-channel linear microphone array, other data not recorded by the 4-channel linear microphone array cannot be used during the training process. Furthermore, there is a problem in that it does not demonstrate sufficient performance even after training when using other types of microphone arrays.

[0005] Since various devices used in daily life (e.g., smartphones, tablet PCs, smart speakers, etc.) feature different microphone configurations with varying positions or numbers, additional work is required to integrate and utilize a multi-channel input speech recognition system in each device. Specifically, it is necessary to acquire a sufficient amount of new data using the desired microphone array and to retrain the multi-channel input speech recognition system based on this data.

[0006] Recently, multi-channel input speech recognition techniques capable of operating with various types of microphone arrays have been proposed, but there are constraints that the shape or number of microphones must be consistently maintained. For example, there are constraints such as the microphone array shape having to follow a circular form and the number of microphones having to be fixed.

[0007] Therefore, there is a need for technology that enables stable multi-channel input speech recognition regardless of the shape of the microphone array. Prior art literature

[0008] Published Patent Application No. 10-2021-0089347 (July 16, 2021) The problem to be solved

[0009] The problem that the present invention aims to solve is to provide a multi-channel input speech recognition device independent of the microphone array type and a learning method thereof, which enables the speech recognition device to operate stably in various types of microphone arrays and to stably perform speech recognition even in a desired microphone array type using a small amount of training data.

[0010] However, the problems that the present invention aims to solve are not limited to those described above, and other problems may exist. means of solving the problem

[0011] A multi-channel input speech recognition device independent of the shape of a microphone array according to the first aspect of the present invention for solving the above-described problem comprises: a time-frequency converter that receives a plurality of channel audio signals extracted from voice data recorded through a plurality of microphones in an unspecified shape of a microphone array and converts them into a plurality of time-frequency domain signals; a speaker and noise mask estimator that receives the plurality of time-frequency domain signals and estimates a time-frequency domain mask for the voices and noise of a plurality of speakers; a beamformer estimator that estimates a time-frequency domain signal for the voices of a plurality of speakers from which noise has been removed using the time-frequency domain mask; a time-frequency inverse converter that inversely converts the noise-removed time-frequency domain signal into a time domain signal; and a learner that learns the speaker and noise mask estimator based on a loss function obtained through a comparison result between the inversely converted time domain signal and a predefined correct signal.

[0012] Additionally, a method performed by a multi-channel input speech recognition device independent of the shape of a microphone array according to a second aspect of the present invention comprises: a step of extracting a plurality of channel audio signals from voice data recorded through a plurality of microphones having an unspecified shape of a microphone array and converting them into a plurality of time-frequency domain signals; a step of inputting the plurality of time-frequency domain signals into a speaker and noise mask estimator to estimate a time-frequency domain mask for the voices and noises of a plurality of speakers; a step of estimating a time-frequency domain signal for the voices of a plurality of speakers from which noise has been removed using the time-frequency domain mask; a step of inversely converting the noise-removed time-frequency domain signal into a time domain signal; and a step of training the speaker and noise mask estimator based on a loss function obtained by comparing the inversely converted time domain signal with a predefined correct signal.

[0013] A computer program according to another aspect of the present invention for solving the above-described problem executes a multi-channel input speech recognition learning method and is stored in a computer-readable recording medium.

[0014] Other specific details of the present invention are included in the detailed description and drawings. Effects of the invention

[0015] According to one embodiment of the present invention, unlike conventional multi-channel input speech recognition technology that relies on a specific type of microphone array for learning and operation, input can be received from various types of microphone arrays, enabling stable operation even in various environments.

[0016] Furthermore, by performing model training through meta-learning methods, stable speech recognition can be achieved even with microphone array configurations desired by the user. Consequently, this reduces the amount of data required for training and enables the provision of a speech recognition device capable of rapidly adapting to new environments.

[0017] In addition, after meta-learning, fine-tuning can be performed using a small amount of training data to provide more stable speech recognition performance in the desired microphone array shape.

[0018] The effects of the present invention are not limited to those mentioned above, and other unmentioned effects will be clearly understood by a person skilled in the art from the description below. Brief explanation of the drawing

[0019] FIG. 1 is a block diagram of a multi-channel input speech recognition device according to one embodiment of the present invention. FIG. 2 is a diagram illustrating the process of deriving optimal model parameters for an initial model in one embodiment of the present invention. FIG. 3 is a diagram illustrating a fine-tuning learning process in one embodiment of the present invention. FIG. 4 is a diagram illustrating a loss function for fine-tuning in one embodiment of the present invention. FIG. 5 is a diagram illustrating a beamformer estimator in one embodiment of the present invention. FIG. 6 is a diagram illustrating an application example of a multi-channel input speech recognition device that has completed learning in one embodiment of the present invention. FIG. 7 is a block diagram of a multi-channel input speech recognition device according to one embodiment of the present invention. FIG. 8 is a flowchart of a multi-channel input speech recognition learning method according to an embodiment of the present invention. Specific details for implementing the invention

[0020] The advantages and features of the present invention and the methods for achieving them will become clear by referring to the embodiments described below in detail together with the accompanying drawings. However, the present invention is not limited to the embodiments disclosed below but may be implemented in various different forms. These embodiments are provided merely to ensure that the disclosure of the present invention is complete and to fully inform those skilled in the art of the scope of the present invention, and the present invention is defined only by the scope of the claims.

[0021] The terms used in this specification are for describing embodiments and are not intended to limit the invention. In this specification, the singular form includes the plural form unless specifically stated otherwise in the text. The terms "comprises" and / or "comprising" used in this specification do not exclude the presence or addition of one or more other components in addition to the components mentioned. Throughout the specification, the same reference numerals refer to the same components, and "and / or" includes each of the mentioned components and all combinations of one or more. Although terms such as "first," "second," etc., are used to describe various components, these components are not limited by these terms. These terms are used merely to distinguish one component from another. Therefore, the first component mentioned below may be the second component within the technical scope of the invention.

[0022] Unless otherwise defined, all terms used herein (including technical and scientific terms) may be used in a meaning commonly understood by those skilled in the art to which the present invention pertains. Additionally, terms defined in commonly used dictionaries are not to be interpreted ideally or excessively unless explicitly and specifically defined otherwise.

[0023] Hereinafter, a multi-channel input speech recognition device (100) according to an embodiment of the present invention will be described with reference to FIGS. 1 to 6. At this time, FIGS. 1 to 5 describe a learning process for the multi-channel speech recognition device (100), and FIG. 6 describes a process of performing speech recognition using the multi-channel speech recognition device (100).

[0024] FIG. 1 is a block diagram of a multi-channel input speech recognition device (100) according to one embodiment of the present invention.

[0025] A multi-channel input speech recognition device (100) according to one embodiment of the present invention includes a microphone array selector (110), a time-frequency converter (120), a speaker and noise mask estimator (130), a beamformer estimator (140), a time-frequency inverse converter (150), and a learner (160).

[0026] Meanwhile, in one embodiment of the present invention, the multi-channel input speech recognition device (100) estimates initial model parameters based on meta-learning so as not to be constrained by the shape of the microphone array, and the learned initial model parameters enable stable operation in various shapes of microphone arrays. Subsequently, speech recognition is enabled by fine-tuning with a small amount of data recorded in the shape of the microphone array to be used.

[0027] A microphone array selector (110) selects and receives voice data recorded in one of the forms of microphone arrays among voice data recorded in multiple forms of microphone arrays mounted on various devices. For example, the various devices may be devices equipped with multiple array-type microphones, such as RGB-D cameras, tablet PCs, and smartphones, and are not limited to any specific type or form.

[0028] In Fig. 1 A multi-type device is shown, and the microphone array selector (110) Select a device having the form of a nth microphone array to receive voice data. The selected device The first microphone array type When composed of multiple microphones, the microphone array selector (110) is Channel audio signal Extract and transmit to the time-frequency converter (120). At this time, the number of channels in each microphone array type may differ from each other.

[0029] The time-frequency converter (120) is an STFT (Short-Time Fourier Transform) converter, and is a plurality of channel audio signals Receives multiple time-frequency domain signals It is converted into. At this time, each time-frequency domain signal consists of T time bin information and F frequency bin information.

[0030] The speaker and noise mask estimator (130) is a plurality of time-frequency domain signals It receives input and estimates time-frequency domain masks for the voices and noise of multiple speakers. That is, the speaker and noise mask estimator (130) Time-frequency domain mask for the voice of [Name] speakers Time-frequency domain mask for and noise Estimate the value. Here, the mask is a filter designed to have binary values ​​of 0 and 1, and is used to determine whether to extract only the target speaker's voice signal or noise in a specific time and frequency domain.

[0031] Meanwhile, in one embodiment of the present invention, the speaker and noise mask estimator (130) may be an array-agnostic estimator. Additionally, in one embodiment of the present invention, the speaker and noise mask estimator (130) may operate regardless of the microphone array type and number of channels of the input signal. That is, it demonstrates consistent performance even with various microphone array types and provides flexibility in that its operation is not restricted by the position or number of microphones.

[0032] The beamformer estimator (140) estimates a time-frequency domain signal for a voice signal of a plurality of speakers from which noise has been removed from a plurality of time-frequency domain signals using a time-frequency domain mask. That is, the beamformer estimator (140) Time-frequency domain mask for speakers and noise Using, Channel time-frequency domain signal from Time-frequency domain signal of a clear voice signal of a famous speaker Estimates.

[0033] Meanwhile, in one embodiment of the present invention, the beamformer estimator (140) may be a Wiener Filter-based MVDR (Minimum Variance Distortionless Response) beamformer estimator (140).

[0034] The inverse time-frequency converter is an ISTFT (Inverse Short-Time Fourier Transform) converter, a time-frequency domain signal with noise removed Receives time domain signal It performs an inverse transformation.

[0035] The learner (160) is inversely transformed Named speaker's time domain signal and the predefined correct answer signal The speaker and noise mask estimator (130) is trained based on the loss function defined through the comparison results. In one embodiment of the present invention, the time-frequency converter (120), the beamformer estimator (140), and the time-frequency inverse converter (150) are composed of elements that do not need to be trained because they perform matrix operations using the input signal. In contrast, the speaker and noise mask estimator (130) is a learnable parameter A model consisting of It consists of, so it requires learning.

[0036] Specifically, the learner (160) can obtain a total loss function by calculating the sum of reconstruction loss functions defined as the distance between the inversely transformed time domain signal and the correct signal for each speaker, and perform learning by adjusting model parameters that constitute the speaker and noise mask estimator (130) based on the total loss function.

[0037] [Equation 1]

[0038]

[0039] The total loss function in Equation 1 above is a model Calculated as Represents the reconstruction loss function in the form of the nth microphone array. Total loss function Is Reconstruction loss function for speakers It is calculated as the sum of.

[0040] also, is a model Calculated as Represents the reconstruction loss function of the i-th speaker. Reconstruction loss function of individual speakers is the estimated speaker's time-domain signal and the correct answer signal A function that measures the distance between It is calculated using. That is, This is a function that measures the distance between two features, and examples such as L1, L2, Cosine, and Euclidean Distance can be used.

[0041] FIG. 2 is a diagram illustrating the process of deriving optimal model parameters for an initial model in one embodiment of the present invention.

[0042] In one embodiment, the learner (160) corresponds to a variety of devices having different microphone array shapes. Total loss function After calculating each through Equation 1, the corresponding to various devices is obtained through the Gradient Descent algorithm Temporary model parameters for the shape of a microphone array It can be calculated through Equation 2.

[0043] [Equation 2]

[0044]

[0045] In Equation 2 Is Representing temporary model parameters in the form of the nth microphone array, represents the learning rate. Equation 2 is the current model parameter Subtract the value obtained by multiplying the learning rate by the gradient of the loss function to obtain temporary model parameters Calculates.

[0046] The learner (160) is a temporary model parameter calculated through Equation 2 After applying it to the speaker and noise mask estimator (130), the total loss function according to Equation 1 is calculated again, and the gradient descent algorithm is applied again to learn in a direction that minimizes the loss function to derive the optimal model parameters. The above speaker and noise mask estimator (130) can be updated with model parameters. This process is as shown in Equation 3.

[0047] [Equation 3]

[0048]

[0049] In Equation 3 represents the learning rate. The optimal model parameters are obtained by repeatedly performing the process of Equations 2 and 3. You can obtain . In the subsequent process, the model It is made possible to use a specific microphone array type by utilizing a small amount of data for fine-tuning.

[0050] FIG. 3 is a diagram illustrating a fine-tuning learning process in one embodiment of the present invention.

[0051] As described above, by estimating and applying optimal initial model parameters, the multi-channel input speech recognition device (100) can operate stably in various microphone array types. Furthermore, one embodiment of the present invention performs a fine-tuning process using a small amount of data to operate in the microphone array type to be used.

[0052] Generally, there are many situations where training data recorded in a specific array format desired by the user does not exist, and generating a sufficient amount of training data requires significant time and cost. Therefore, in one embodiment of the present invention, optimal model parameters By performing fine-tuning using, effective performance can be achieved even with a small amount of data.

[0053] To this end, the present invention, from a specific microphone array type to be used (e.g., a 4-channel microphone array mounted on a tablet PC) Channel time domain signal Extracts it. Then, it is passed as input through a time-frequency converter (120), a speaker and noise mask estimator (130), and a beamformer estimator (140). Time-frequency domain signal with noise removed ...is obtained. This process is the same as described in FIG. 1, and the speaker and noise mask estimator (130) obtains the optimal model parameters obtained through the process described in FIG. 2 and Equations 2 and 3. It is initialized to.

[0054] The end-to-end speech recognition model (170) is Time-frequency domain signal with noise removed Speech recognition is performed by receiving input. At this time, one embodiment of the present invention may define a loss function for fine-tuning as shown in Equation 4 by comparing the speech recognition result with the prepared correct answer information for fine-training of the end-to-end speech recognition model (170).

[0055] [Equation 4]

[0056]

[0057] In one embodiment, the loss function for fine-tuning is constructed by summing the speaker-specific connectionist time series classification loss function and the cross-entropy loss function. Referring to Equation 4, Is Represents the Connectionist Temporal Classification (CTC) loss function for the i-th speaker, and Is This represents the cross-entropy loss function of the i-th speaker.

[0058] FIG. 4 is a diagram illustrating a loss function for fine-tuning in one embodiment of the present invention.

[0059] In one embodiment, the end-to-end speech recognition model (170) may include a speech recognition encoder (171) and a speech recognition decoder (172). The speech recognition encoder (171) receives a noise-removed time-frequency domain signal of a specific speaker and embeds it into a vector that constitutes a hidden vector space. The speech recognition decoder (172) receives the vector that constitutes the hidden vector space and outputs a speech recognition result for a specific speaker.

[0060] Specifically When, the estimated The clear voice of the nth speaker (noise-removed time-frequency domain signal) A vector that forms a hidden vector space by inputting it into a speech recognition encoder (171). Embed it as. Then, of the hidden vector space input into the speech recognition decoder (172) Speech recognition result text of the nth speaker Prints.

[0061] At this point, speaker-specific connectionist time series classification loss function is of the hidden vector space It is defined using the CTC (Connectionist Temporal Classification) technique, and the cross-entropy loss function is the speech recognition result text It is defined by measuring the cross-entropy between the and the correct text.

[0062] FIG. 5 is a drawing for explaining a beamformer estimator (140) in one embodiment of the present invention.

[0063] In one embodiment, the beamformer estimator (140) is Time-frequency domain mask for the speaker and Multiple time-frequency domain signals of a channel It receives input. Through this input, the beamformer estimator (140) receives multiple speakers The Power Spectral Density (PSD) matrix for voice and noise can be calculated using Equation 5.

[0064] [Equation 5]

[0065]

[0066] In Equation 5 When, Is Represents the power spectral density matrix for the i-th speaker or noise. And, is speaker or noise's The i-th time bin and It represents the mask coefficients of the nth frequency bin. Is The th hour empty and All of the nth frequency bins Represents a column vector composed of channel time-frequency signals (i.e., ).

[0067] In addition, the filter coefficients of the beamformer estimator (140) Power spectral density matrix of speakers and noise It can be calculated through Equation 6 below based on .

[0068] [Equation 6]

[0069]

[0070] In Equation 6 When, Is It represents the filter coefficients of the i-th sound source. It calculates the sum of the diagonal elements of a matrix as a trace operator. is a one-hot vector representing the reference microphone.

[0071] Additionally, the beamformer estimator (140) can estimate the time-frequency domain signal for multiple speakers' voice signals from which noise has been removed by applying the filter coefficients to the time bin information and frequency bin information in the time-frequency domain signal for the speaker's voice and noise. That is, the filter coefficients calculated through Equation 6 is in the following Equation 7 It is used to extract the clear voice of a speaker.

[0072] [Equation 7]

[0073]

[0074] In Equation 7 When, is a speaker extracted by the beamformer estimator (140) It represents the signal from which only the clean voice has been extracted. And is the filter coefficient calculated through Equation 6, and Is The th hour empty and All of the nth frequency bins It represents a column vector composed of channel time-frequency signals (Equation 5).

[0075] FIG. 6 is a diagram illustrating an application example of a multi-channel input speech recognition device (100) that has completed learning in one embodiment of the present invention.

[0076] After the optimization process of model parameters and fine-tuning learning are completed, speech recognition can be performed using a microphone array to be used (e.g., a 5-channel linear microphone array).

[0077] In other words, from the specific microphone array you intend to use Channel's time domain signal When input is received, using a time-frequency converter (120), a speaker and noise mask estimator (130), a beamformer estimator (140), and an end-to-end speech recognition model (170). Speech recognition result text for the speakers Prints.

[0078] The application process of the speech recognition device (100) is the same as the method performed in the fine-tuning learning of FIGS. 3 and FIG. 4 described above, and the speaker and noise mask estimator (130) and the end-to-end speech recognition model (170) are applied after being initialized with model parameters obtained during the fine-tuning learning process.

[0079] FIG. 7 is a block diagram of a multi-channel input speech recognition device (100) according to one embodiment of the present invention.

[0080] Meanwhile, a multi-channel input voice recognition device (100) according to one embodiment of the present invention includes an input unit (210), a communication unit (220), a display unit (230), a memory (240), and a processor (250).

[0081] The input unit (210) generates input data in response to user input. User input may include user input regarding data that the multi-channel input speech recognition device (100) intends to process. The input unit (210) may include at least one input means. The input unit (210) may include a keyboard, a key pad, a dome switch, a touch panel, a touch key, a mouse, a menu button, etc. Meanwhile, the input unit (210) may be integrally formed with any one of the aforementioned types of devices, and in this case, it may include a specific microphone array.

[0082] The communication unit (220) transmits and receives data between internal components or performs communication with external devices such as external servers. Such a communication unit (220) may include both a wired communication module and a wireless communication module. The wired communication module may be implemented as a power line communication device, a telephone line communication device, a cable home (MoCA), Ethernet, IEEE1294, an integrated wired home network, and an RS-485 control device. Additionally, the wireless communication module may be composed of a module for implementing functions such as WLAN (wireless LAN), Bluetooth, HDR WPAN, UWB, ZigBee, Impulse Radio, 60GHz WPAN, Binary-CDMA, wireless USB technology and wireless HDMI technology, as well as 5G (5th generation communication), LTE-A (long term evolution-advanced), LTE (long term evolution), and Wi-Fi (wireless fidelity).

[0083] The display unit (230) displays display data according to the operation of the multi-channel input voice recognition device (100). The display unit (230) can display the output of the input voice itself, or voice recognition results converted into a string, etc. The display unit (230) includes a liquid crystal display (LCD), a light emitting diode (LED) display, an organic light emitting diode (OLED) display, a micro electro mechanical systems (MEMS) display, and an electronic paper display. The display unit (230) can be combined with the input unit (210) to be implemented as a touch screen.

[0084] The memory (240) stores programs for model learning and applying the learned model in the multi-channel input speech recognition device (100). Here, the memory (240) is a general term for non-volatile storage devices and volatile storage devices that retain stored information even when power is not supplied. For example, the memory (240) may include NAND flash memory such as compact flash (CF) cards, SD (secure digital) cards, memory sticks, solid-state drives (SSDs), and micro SD cards, magnetic computer memory devices such as hard disk drives (HDDs), and optical disc drives such as CD-ROMs and DVD-ROMs.

[0085] The processor (250) can execute software, such as a program, to control at least one other component (e.g., hardware or software component) of the voice recognition device (100) and can perform various data processing or operations.

[0086] Hereinafter, a multi-channel input speech recognition learning method performed by a multi-channel input speech recognition device (100) will be described with reference to FIG. 8.

[0087] FIG. 8 is a flowchart of a multi-channel input speech recognition learning method according to an embodiment of the present invention.

[0088] First, multiple channel audio signals are extracted from voice data recorded through multiple microphones having an unspecified microphone array shape and converted into multiple time-frequency domain signals (S110).

[0089] Next, multiple time-frequency domain signals are input into a speaker and noise mask estimator (130) to estimate time-frequency domain masks for the voices and noise of multiple speakers (S120).

[0090] Next, using a time-frequency domain mask, a time-frequency domain signal for a voice signal of a plurality of speakers from which noise has been removed from a plurality of time-frequency domain signals is estimated (S130).

[0091] Next, the noise-removed time-frequency domain signal is inversely converted into a time domain signal (S140).

[0092] Next, a speaker and noise mask estimator (130) is trained based on a loss function obtained by comparing the inverse transformed time domain signal with a predefined correct signal (S150).

[0093] Meanwhile, in the above description, steps S110 to S150 may be further divided into additional steps or combined into fewer steps according to an embodiment of the present invention. Also, some steps may be omitted as necessary, and the order between steps may be changed. Furthermore, even if other omitted details are included, the details described in FIGS. 1 to 7 and the details described in FIG. 8 may be mutually applicable.

[0094] The multi-channel input speech recognition learning method according to one embodiment of the present invention described above may be implemented as a program (or application) and stored on a medium to be executed in combination with a computer, which is hardware.

[0095] The aforementioned program may include code encoded in computer languages ​​such as C, C++, JAVA, Python, Ruby, and machine language, which can be read by the computer's processor (CPU) through the computer's device interface, in order for the computer to read the program and execute the methods implemented in the program. Such code may include functional code related to functions that define the necessary functions for executing the methods, and may include control code related to execution procedures necessary for the computer's processor to execute the functions according to a predetermined procedure. Additionally, such code may further include memory reference code regarding where (address) additional information or media necessary for the computer's processor to execute the functions should be referenced in the computer's internal or external memory. In addition, if the processor of the computer needs to communicate with any other computer or server located remotely in order to execute the above functions, the code may further include communication-related code regarding how to communicate with any other computer or server located remotely using the communication module of the computer, and what information or media to transmit or receive during communication.

[0096] The above-mentioned storage medium refers to a medium that stores data semi-permanently and is readable by a device, rather than a medium that stores data for a short period of time, such as a register, cache, or memory. Specifically, examples of the above-mentioned storage medium include, but are not limited to, ROM, RAM, CD-ROM, magnetic tape, floppy disk, and optical data storage device. That is, the above-mentioned program may be stored on various recording media on various servers that the computer can access, or on various recording media on the user's computer. Additionally, the above-mentioned medium may be distributed across networked computer systems, and computer-readable code may be stored in a distributed manner.

[0097] The foregoing description of the present invention is for illustrative purposes only, and those skilled in the art will understand that other specific forms can be easily modified without altering the technical spirit or essential features of the present invention. Therefore, the embodiments described above should be understood as illustrative in all respects and not restrictive. For example, each component described as a single unit may be implemented in a distributed manner, and components described as distributed may likewise be implemented in a combined form.

[0098] The scope of the present invention is defined by the claims set forth below rather than by the detailed description above, and all modifications or variations derived from the meaning and scope of the claims and equivalent concepts thereof should be interpreted as being included within the scope of the present invention. Explanation of the symbols

[0099] 100: Multi-channel input speech recognition device 110: Microphone Array Selector 120: Time-Frequency Converter 130: Speaker and Noise Mask Estimator 140: Beamformer Estimator 150: Time-Frequency Inverter 160: Learning device 170: End-to-end speech recognition model

Claims

Claim 1 A multi-channel input speech recognition device independent of the shape of a microphone array, comprising: a time-frequency converter that receives a plurality of channel audio signals extracted from speech data recorded through a plurality of microphones in an unspecified shape of a microphone array and converts them into a plurality of time-frequency domain signals; a speaker and noise mask estimator that receives the plurality of time-frequency domain signals and estimates a time-frequency domain mask for the voices and noise of a plurality of speakers; a beamformer estimator that estimates a time-frequency domain signal for the voices of a plurality of speakers from which noise has been removed from the plurality of time-frequency domain signals using the time-frequency domain mask; a time-frequency inverse converter that inversely converts the noise-removed time-frequency domain signal into a time domain signal; and a learner that learns the speaker and noise mask estimator based on a loss function obtained through a comparison result between the inversely converted time domain signal and a predefined correct signal, wherein the multi-channel input speech recognition device further comprises a microphone array selector that selects and receives speech data recorded in one of the shapes of microphone arrays among speech data recorded by a plurality of shapes of microphone arrays mounted on a plurality of devices. Claim 2 delete Claim 3 A multi-channel input speech recognition device according to claim 1, wherein the learner performs learning by adjusting model parameters constituting the speaker and noise mask estimator based on the loss function obtained by calculating the sum of reconstruction loss functions defined as the distance between the inverse transformed time domain signal and the correct answer signal for each speaker. Claim 4 A multi-channel input speech recognition device according to paragraph 3, wherein the learner acquires a number of loss functions corresponding to a number of different types of devices having different microphone array shapes, and updates temporary model parameters for a number of microphone array shapes corresponding to the number of different devices to the model parameters through a gradient descent algorithm. Claim 5 A multi-channel input speech recognition device according to claim 4, wherein the learner updates the model parameters of the speaker and noise mask estimators with the optimal model parameters derived by applying a gradient descent algorithm to the loss function obtained after updating with the temporary model parameters and learning in a direction that minimizes the loss function. Claim 6 A multi-channel input speech recognition device according to claim 1, further comprising an end-to-end speech recognition model that receives a noise-removed time-frequency domain signal obtained through the time-frequency converter, the speaker and noise mask estimator, and the beamformer estimator based on voice data recorded through a plurality of microphones having a specific microphone array shape, and outputs a speech recognition result. Claim 7 A multi-channel input speech recognition device according to claim 6, wherein the end-to-end speech recognition model defines a loss function for fine-tuning by comparing the speech recognition result with pre-prepared correct answer information, and the loss function for fine-tuning is constructed by summing a speaker-specific connectionist time-series classification loss function and a cross-entropy loss function. Claim 8 In claim 7, the end-to-end speech recognition model comprises a speech recognition encoder that receives a noise-removed time-frequency domain signal of a specific speaker and embeds it into a vector that constitutes a hidden vector space, and a speech recognition decoder that receives the vector that constitutes the hidden vector space and outputs a speech recognition result for the specific speaker, thereby forming a multi-channel input speech recognition device. Claim 9 A multi-channel input speech recognition device according to claim 1, wherein the beamformer estimator receives the time-frequency domain mask and the plurality of time-frequency domain signals to calculate a power spectral density matrix for the voices and noises of the plurality of speakers, calculates filter coefficients of the beamformer estimator from the power spectral density matrix, and applies the filter coefficients to time bin information and frequency bin information in the time-frequency domain signals for the voices and noises of the speakers to estimate the time-frequency domain signals for the voices of the plurality of speakers from which the noise has been removed. Claim 10 A multi-channel input speech recognition device according to claim 9, wherein the beamformer estimator calculates the power spectral density matrix based on a column vector composed of time bin information and frequency bin information in a time-frequency domain signal for the speaker's speech and noise, and time-frequency signals for all channels of the time bin information and frequency bin information. Claim 11 A method performed by a multi-channel input speech recognition device independent of the shape of a microphone array, comprising: a step of extracting a plurality of channel audio signals from voice data recorded through a plurality of microphones having an unspecified shape of a microphone array and converting them into a plurality of time-frequency domain signals; a step of inputting the plurality of time-frequency domain signals into a speaker and noise mask estimator to estimate a time-frequency domain mask for the voices of a plurality of speakers and noise; a step of estimating a time-frequency domain signal for the voices of a plurality of speakers from which noise has been removed from the plurality of time-frequency domain signals using the time-frequency domain mask; and a step of inversely converting the noise-removed time-frequency domain signal into a time domain signal. A multi-channel input speech recognition learning method comprising the step of learning the speaker and noise mask estimators based on a loss function obtained by comparing the inversely transformed time domain signal with a predefined correct signal, wherein the step of extracting a plurality of channel audio signals from speech data recorded through a plurality of microphones having an unspecified microphone array shape and converting them into a plurality of time-frequency domain signals comprises: the step of selecting and receiving speech data recorded through a plurality of microphones having an unspecified microphone array shape; the step of extracting a plurality of channel audio signals corresponding to the plurality of microphones from the speech data; and the step of converting the plurality of channel audio signals into a plurality of time-frequency domain signals. Claim 12 delete Claim 13 In claim 11, the step of selecting and receiving voice data recorded through a plurality of microphones having the above-mentioned unspecified microphone array shape is to select and receive the voice data recorded in a specific microphone array shape, which is one of the voice data recorded by a plurality of microphone array shapes mounted on a multi-type device. Claim 14 A multi-channel input speech recognition learning method according to claim 11, wherein the step of learning the speaker and noise mask estimator comprises: a step in which the learner calculates the sum of reconstruction loss functions defined as the distance between the inversely transformed time domain signal and the correct signal for each speaker; and a step of performing learning by adjusting model parameters constituting the speaker and noise mask estimator based on the loss function obtained from the sum of the reconstruction loss functions. Claim 15 A multi-channel input speech recognition learning method according to claim 14, wherein the step of performing learning by adjusting the above model parameters is to obtain a number of the above loss functions corresponding to a number of different types of devices having different microphone array shapes, and to update the model parameters with temporary model parameters for a number of microphone array shapes corresponding to the number of different devices through a gradient descent algorithm. Claim 16 A multi-channel input speech recognition learning method, wherein the step of performing learning by adjusting the above model parameters is to update the model parameters of the speaker and noise mask estimators with the optimal model parameters derived by applying a gradient descent algorithm to the loss function obtained after updating with the above temporary model parameters and learning in a direction that minimizes the above loss function. Claim 17 A multi-channel input speech recognition learning method according to claim 11, further comprising: a step of obtaining a noise-removed time-frequency domain signal based on voice data recorded through a plurality of microphones having a specific microphone array shape after the learning of the speaker and noise mask estimator is completed; and a step of inputting the noise-removed time-frequency domain signal into an end-to-end speech recognition model to output a speech recognition result. Claim 18 A multi-channel input speech recognition learning method according to claim 17, further comprising the step of fine-tuning the end-to-end speech recognition model through a loss function defined by comparing the speech recognition result with pre-prepared correct answer information, wherein the loss function for fine-tuning is constructed by summing a speaker-specific connectionist time-series classification loss function and a cross-entropy loss function. Claim 19 In claim 11, the step of estimating a time-frequency domain signal for a plurality of speaker voice signals from which noise has been removed comprises: receiving the time-frequency domain mask and the plurality of time-frequency domain signals as inputs and calculating a power spectral density matrix for the voices and noise of the plurality of speakers; calculating filter coefficients of a beamformer estimator from the power spectral density matrix; and applying the filter coefficients to time bin information and frequency bin information in the time-frequency domain signal for the voices and noise of the speakers to estimate the time-frequency domain signal for the voices and noise of the plurality of speakers, thereby forming a multi-channel input speech recognition learning method. Claim 20 A multi-channel input speech recognition learning method, wherein the step of calculating the power spectral density matrix in claim 19 is to calculate the power spectral density matrix based on a column vector composed of time bin information and frequency bin information in a time-frequency domain signal for the speaker's speech and noise, and time-frequency signals for all channels of the time bin information and frequency bin information.

Images