INFORMATION PROCESSING EQUIPMENT, PROGRAM AND INFORMATION PROCESSING PROCESS

Abstract

Information processing facility (130, 230, 330, 430), comprising: an acoustic component computation unit (138) configured to compute an acoustic component from mixed audio data using a predetermined function, wherein the mixed audio data includes target speech audio content and mixed noise, wherein the target speech audio content is a target to be enhanced, wherein the mixed noise is noise to be mixed with the target speech audio content, and the acoustic component is a component of the target speech audio content and the mixed noise; an acoustic feature detection unit (139) configured to detect an acoustic feature by inputting the acoustic component into a feature detection model trained to detect an acoustic feature of speech and mixed background noise; a noise component calculation unit (140) configured to calculate a noise component from noise data using the predetermined function, wherein the noise data does not contain the target speech audio content but does contain noise; a noise feature detection unit (141, 341, 441) configured to detect a noise feature by the background noise component is fed into a background noise detection model that is trained to identify an acoustic feature of the background noise; a correlation determination unit (142, 342, 442) that is set up to determine a correlation between the acoustic feature and the noise feature by inputting the acoustic feature and the noise feature into a correlation determination model that is trained to determine a correlation between the acoustic feature and the noise feature; a feature integration unit (143) that is set up to calculate an integrated feature by weighting the acoustic feature with the determined correlation; a mask detection unit (144) configured to determine a target language mask by inputting the integrated feature into a language enhancement model trained to determine a language enhancement mask; and a speech recovery unit (145) which is configured to recover speech audio content from the acoustic component and the target speech mask in which the target speech audio content is enhanced.

Description

[0001] The present disclosure relates to an information processing device, a program and an information processing method. TECHNICAL BACKGROUND

[0002] Traditionally, speech recognition is performed to identify speech. Generally, speech recognition processing is affected by background noise other than the target speech audio content, and the presence of background noise significantly reduces the accuracy of speech recognition. Therefore, it is necessary to extract the target speech audio content from noisy audio.

[0003] For example, NPL1 proposes a method that uses neural networks (NNs) to learn data pairs from mixed audio content and target language content, and to extract the target language audio content from the mixed audio content. REFERENCES ON THE STATE OF THE TECHNOLOGY PATENT REFERENCE

[0004] NPL 1: Felix Weniger, et al., “Discriminatively trained recurrent neural networks for single-channel speech separation,” IEEE Global Conference on Signal and Information Processing (GlobalSIP), Feb. 2015. SUMMARY OF THE INVENTION; PROBLEM TO BE SOLVED BY THE INVENTION

[0005] However, the conventional method for learning sounds, which are a mixture of speech and non-speech, has the problem that speech improvement performance decreases when non-speech is not learned.

[0006] Accordingly, one task of one or more aspects of the revelation is to improve or optimize speech with unknown background noise that is not included in the training data. MEANS OF SOLVING THE TASK

[0007] An information processing device according to one aspect of the disclosure comprises: an acoustic component computation unit configured to compute an acoustic component from mixed audio data using a predetermined function, wherein the mixed audio data contains target speech audio content and mixed noise, the target speech audio content being a target to be enhanced, the mixed noise being noise to be mixed with the target speech audio content, and the acoustic component being a component of the target speech audio content and the mixed noise; an acoustic feature detection unit configured to detect an acoustic feature by inputting the acoustic component into a feature detection model trained to detect an acoustic feature of speech and noise;a noise component computation unit configured to compute a noise component from noise data using a predetermined function, wherein the noise component is a component of noise, and the noise data does not contain the target speech audio content; a noise feature detection unit configured to detect a noise feature by inputting the noise component into a noise detection model trained to detect an acoustic feature of the noise;a correlation determination unit configured to determine a correlation between the acoustic feature and the noise feature by inputting the acoustic feature and the noise feature into a correlation determination model trained to determine a correlation between the acoustic feature of speech and noise and the acoustic feature of noise; a feature integration unit configured to compute an integrated feature by weighting the acoustic feature with the determined correlation; a mask determination unit configured to determine a target speech mask by inputting the integrated feature into a speech enhancement model trained to determine a speech enhancement mask;and a speech recovery unit that is configured to recover speech audio content in which the target speech audio content is enhanced from the acoustic component and the target speech mask.

[0008] A program according to one aspect of the disclosure causes a computer to operate as: an acoustic component computation unit configured to compute an acoustic component from mixed audio data using a predetermined function, wherein the mixed audio data includes target speech audio content and mixed noise, the target speech audio content being a target to be enhanced, the mixed noise being noise to be mixed with the target speech audio content, and the acoustic component being a component of the target speech audio content and the mixed noise; an acoustic feature detection unit configured to detect an acoustic feature by inputting the acoustic component into a feature detection model trained to detect an acoustic feature of speech and noise;a noise component computation unit configured to compute a noise component from noise data using a predetermined function, wherein the noise component is a component of noise, and the noise data does not contain the target speech audio content; a noise feature detection unit configured to detect a noise feature by inputting the noise component into a noise detection model trained to detect an acoustic feature of the noise;a correlation determination unit configured to determine a correlation between the acoustic feature and the noise feature by inputting the acoustic feature and the noise feature into a correlation determination model trained to determine a correlation between the acoustic feature of speech and noise and the acoustic feature of noise; a feature integration unit configured to compute an integrated feature by weighting the acoustic feature with the determined correlation; a mask determination unit configured to determine a target speech mask by inputting the integrated feature into a speech enhancement model trained to determine a speech enhancement mask;and a speech recovery unit that is configured to recover speech audio content in which the target speech audio content is enhanced from the acoustic component and the target speech mask.

[0009] An information processing method according to one aspect of the disclosure comprises: calculating an acoustic component from mixed audio data using a predetermined function, wherein the mixed audio data contains target speech audio content and mixed noise, the target speech audio content being a target to be enhanced, the mixed noise being noise to be mixed with the target speech audio content, and the acoustic component being a component of the target speech audio content and the mixed noise; determining an acoustic feature by inputting the acoustic component into a feature-determination model trained to determine an acoustic feature of speech and noise; calculating a noise component from noise data using the predetermined function, wherein the noise component is a component of noise, and the noise data does not contain the target speech audio content and contains noise.Determine a noise feature by inputting the noise component into a noise detection model trained to identify an acoustic feature of the noise; determine a correlation between the acoustic feature and the noise feature by inputting the acoustic feature and the noise feature into a correlation detection model trained to determine a correlation between the acoustic feature of speech and noise and the acoustic feature of noise; calculate an integrated feature by weighting the acoustic feature with the determined correlation; determine a target speech mask by inputting the integrated feature into a speech enhancement model trained to determine a speech enhancement mask; and restore speech audio content in which the target speech audio content is enhanced from the acoustic component and the target speech mask. EFFECTS OF THE INVENTION

[0010] According to one or more aspects of the revelation, speech can be improved even with unknown background noise that is not included in the training data. BRIEF DESCRIPTION OF THE DRAWINGS Fig. Figure 1 is a block diagram which schematically shows a configuration of a speech enhancement system according to a first to fourth embodiment. Fig. Figure 2 is a block diagram which schematically shows a configuration of a training facility according to the first embodiment. Fig. 3A and Fig. 3B are block diagrams that show examples of a hardware configuration. Fig. Figure 4 is a block diagram which schematically shows a configuration of a speech enhancement device according to the first embodiment. Fig. Figure 5 is a flowchart showing the operation of the training facility according to the first embodiment. Fig. Figure 6 is a flowchart showing the operation of the speech enhancement device according to the first embodiment. Fig. Figure 7 is a block diagram which schematically shows a configuration of a speech recognition device according to the second embodiment. Fig. Figure 8 is a flowchart showing the operation of the speech enhancement device according to the second embodiment. Fig. Figure 9 is a block diagram which schematically shows a configuration of a training facility according to the third embodiment. Fig. Figure 10 is a block diagram which schematically shows a configuration of the speech enhancement device according to the third embodiment. Fig. Figure 11 is a flowchart showing the operation of the training facility according to the first embodiment. Fig. Figure 12 is a flowchart showing the operation of the speech enhancement device according to the third embodiment. Fig. Figure 13 is a block diagram which schematically shows a configuration of a speech enhancement device according to the fourth embodiment. Fig. Figure 14 is a flowchart showing the operation of the speech enhancement device according to the fourth embodiment. EXECUTION FORM FOR IMPLEMENTING THE FIRST INVENTION FORM

[0011] Fig. Figure 1 is a block diagram which schematically represents a configuration of a speech enhancement system 100 according to the first embodiment.

[0012] The language improvement system 100 comprises a training facility 110 and a language improvement facility 130, which serve as an information processing facility.

[0013] The training facility 110 trains learning models that function as a feature-detection NN, which is a feature-detection model for determining an acoustic feature of speech and background noise; a background-detection NN, which is a background-detection model for determining an acoustic feature of background noise; a correlation-detection NN, which is a correlation-detection model for determining a correlation between the acoustic feature of speech and background noise and the acoustic feature of background noise; and a speech-enhancement NN, which is a speech-enhancement model for determining a mask for improving speech.

[0014] The speech enhancement device 130 acquires the trained feature detection NN, noise detection NN, correlation detection NN, and speech enhancement NN from the training device 110 and uses these learning models to enhance the target speech audio content in the mixed audio content.

[0015] During a training phase, the training facility 110 trains the learning models that are to be used by the language improvement facility 130.

[0016] In an inference phase, the language enhancement facility 130 uses the learning models trained by the training facility to enhance the target language audio content in the mixed audio content.

[0017] Fig. Figure 2 is a block diagram which schematically shows a configuration of the training facility 110 according to the first embodiment.

[0018] The training facility 110 comprises a speech data storage unit 111, a noise data storage unit 112, an audio content mixing unit 113, a component calculation unit 114, a teacher mask detection unit 115, a model training unit 116, a model storage unit 117 and a communication unit 118.

[0019] The speech data storage unit 111 stores training purpose speech data, which represents training purpose target speech audio content to be used for training.

[0020] The noise data storage unit 112 stores training purpose noise data, which represents training purpose noise to be used for training.

[0021] The audio content mixing unit 113 acquires the training purpose speech data from the speech data storage unit 111 and the training purpose noise data from the noise data storage unit 112, selects the training purpose target speech audio content represented by the training purpose speech data and the training purpose noise represented by the training purpose noise data, overlays these to create mixed audio content, and outputs the target speech audio content and the mixed audio content to the component processing unit 114.

[0022] Component Calculation Unit 114 calculates a target language component, which is a component of the target language audio content from Audio Content Mixing Unit 113, and calculates a mixed audio component, which is a component of the mixed audio content from Audio Content Mixing Unit 113. For example, Component Calculation Unit 114 determines an audio component as a time series of a power spectrum calculated from an audio signal by short-time Fourier transform (STFT). The target language component and the mixed audio component are passed to Teacher Mask Determination Unit 115.

[0023] The Teacher Mask Determination Unit 115 generates a teacher mask from the target language component and the mixed audio component from the Component Calculation Unit 114. For example, the performance spectrum of the target language audio content and the performance spectrum of the mixed audio content are determined from the target language component and the mixed audio component, respectively, and the ratio of the performance spectrum of the target language audio content to the performance spectrum of the mixed audio content is determined as a teacher mask. The teacher mask is then passed to the Model Training Unit 116.

[0024] The model training unit 116 receives the mixed audio content from the audio content mixing unit 113, the training purpose background noise data from the background noise data storage unit 112, and the teacher mask from the teacher mask determination unit 115, and trains the neural networks (NNs). Training an NN is a process for determining an input weighting coefficient, which is a parameter of the NN. During training, the feature-detection NN, the background noise-detection NN, the correlation-detection NN, and the speech enhancement NN are combined and, for example, fed into a loss function mentioned in the following reference, and an error is calculated based on the training purpose target speech audio content.Then, for example, an optimization method such as adaptive moment determination (Adam) can be used to learn the input weighting coefficient of each layer of the feature detection NN, noise detection NN, correlation detection NN, and speech enhancement NN based on, for example, backpropagation. The generated feature detection NN, noise detection NN, correlation detection NN, and speech enhancement NN are stored in model memory unit 117.

[0025] Reference: R. Aihara et al., “Deep clustering-based single-channel speech separation and recent advances,” Acoust. Sci. & Tech. February 41, 2020.

[0026] The model storage unit 117 stores the feature detection NN, the noise detection NN, the correlation detection NN and the speech enhancement NN, which were trained by the model training unit 116.

[0027] The communication unit 118 operates as a transmission unit that transmits the feature detection NN, the noise detection NN, the correlation detection NN and the speech enhancement NN, which are stored in the model storage unit 117, to the speech enhancement device 130.

[0028] Part or all of the audio content mixing unit 113, component calculation unit 114, teacher mask detection unit 115, and model training unit 116 described above can, for example, be implemented by a working memory 10 and a processor 11, such as a central processing unit (CPU), which executes the programs stored in the working memory 10, as in Fig. Figure 3A illustrates this. In other words, the training facility can be implemented using a generally known computer. Such programs can be provided over a network or can be recorded and provided on a recording medium. That is, such programs can be provided, for example, as a software product.

[0029] Part or all of the audio content mixing unit 113, the component calculation unit 114, the teacher mask determination unit 115, and the model training unit 116 can, for example, be implemented by a single circuit, a composite circuit, a program-driven processor, a program-driven parallel processor, a processing circuit 12 such as an application-specific integrated circuit (ASIC) or a field-programmable gate array (FPGA), as in Fig. 3B is shown, to be realized.

[0030] As explained above, the audio content mixing unit 113, the component calculation unit 114, the teacher mask determination unit 115 and the model training unit 116 can be implemented by a processing circuit.

[0031] The speech data storage unit 111, the noise data storage unit 112 and the model storage unit 117 can be implemented by a data storage device (not shown), such as a hard disk drive (HDD), a solid-state drive (SSD), or non-volatile memory.

[0032] The communication unit 118 can be implemented through a communication interface, such as a network interface card (NIC) or the like.

[0033] Fig. Figure 4 is a block diagram which schematically shows a configuration of the speech enhancement device 130 according to the first embodiment.

[0034] The speech enhancement device 130 comprises a communication unit 131, a feature detection NN storage unit 132, a noise detection NN storage unit 133, a correlation detection NN storage unit 134, a speech enhancement NN storage unit 135, a noise mixing audio content acquisition unit 136, a noise acquisition unit 137, an acoustic component calculation unit 138, an acoustic feature detection unit 139, a noise component calculation unit 140, a noise feature detection unit 141, a correlation detection unit 142, a feature integration unit 143, a mask detection unit 144, and a speech restoration unit 145.

[0035] The communication unit 131 functions as a receiving unit that receives the feature detection NN, the noise detection NN, the correlation detection NN and the speech enhancement NN from the training facility 110.

[0036] The feature detection NN storage unit 132 stores the feature detection NN received from the communication unit 131.

[0037] The noise detection NN storage unit 133 stores the noise detection NN received from the communication unit 131.

[0038] The correlation-determination NN storage unit 134 stores the correlation-determination NN received from the communication unit 131.

[0039] The speech enhancement NN storage unit 135 stores the speech enhancement NN received from the communication unit 131.

[0040] The noise-mixed audio acquisition unit 136 acquires mixed audio data, which represents mixed audio content containing target speech audio content and noise captured by a microphone (not shown) acting as a noise collection unit. The noise contained in the mixed audio data is also referred to as mixed noise. For example, the noise-mixed audio acquisition unit 136 can acquire mixed audio data via the communication unit 131 or from a microphone connected via a communication interface, such as a universal serial bus (USB). The mixed audio data is also referred to here as inference-purpose mixed audio data, and the mixed audio content represented by the inference-purpose mixed audio data is also referred to as inference-purpose mixed audio content.The communication unit 131 or the connection interface serves as an interface (input interface) which accepts input of data, or an interface unit (input interface unit).

[0041] The noise acquisition unit 137 acquires noise data representing background noise captured by a microphone and containing no target speech audio content. For example, the noise-mixed audio content acquisition unit 136 can acquire noise data via the communication unit 131 or acquire noise data from a microphone connected to the communication interface. The background noise might be, for example, noise captured by a microphone at a specific time before or after the input of mixed audio content containing target speech audio and noise. The noise data is also referred to here as inference-purpose noise data, and the background noise represented by the inference-purpose noise data is also referred to as inference-purpose noise.

[0042] The acoustic component calculation unit 138 calculates an acoustic component from mixed audio data, which contains the target speech audio content to be improved, and mixed noise, which is noise to be mixed with the target speech audio content, using a predefined function.

[0043] For example, the acoustic component calculation unit 138 receives the inference-purpose mixed audio data from the noise-mixed audio content acquisition unit 136 and calculates an acoustic component from the mixed audio content represented by the inference-purpose mixed audio data. For example, the acoustic component is a time series of a power spectrum calculated from an audio signal by short-time Fourier transform (STFT). The acoustic component is then passed to the acoustic feature detection unit 139 and the speech recovery unit 145.

[0044] The acoustic feature detection unit 139 passes the acoustic component from the acoustic component calculation unit 138 to the feature detection NN, which is a feature detection model trained to determine an acoustic feature of speech and background noise. For example, the acoustic feature detection unit 139 inputs the acoustic component received from the acoustic component calculation unit 138 into the feature detection NN, which is stored in the feature detection NN storage unit 132, and determines an acoustic feature. The feature detection NN is a neural network consisting of multiple layers, and for propagation between the layers, for example, a technique similar to long short-term memory (LSTM) or a technique combining one-dimensional convolution operations can be used, with no limit to the number of layers.The acoustic feature is forwarded to the correlation determination unit 142 and the feature integration unit 143.

[0045] The noise component calculation unit 140 calculates a noise component from noise data that contains noise but no target speech audio content by using a predefined function.

[0046] For example, the noise component calculation unit 140 receives the inference-purpose noise data from the noise determination unit 137 and calculates a noise component from the noise represented by the inference-purpose noise data. The noise component is, for example, a time series of a power spectrum calculated from an audio signal by short-time Fourier transform (STFT). The noise component is then passed on to the noise feature determination unit 141.

[0047] The noise feature detection unit 141 passes the noise component from the noise component calculation unit 140 to the noise detection NN, which is a noise detection model trained to determine an acoustic feature of noise.

[0048] For example, the noise feature detection unit 141 inputs the noise component from the noise component calculation unit 140 into the noise detection neural network, which is stored in the noise detection neural network storage unit 133, and determines the noise feature. In this case, the noise detection neural network is a multi-layered neural network, and for propagation between the layers, a technique similar to LSTM or a technique combining one-dimensional convolution operations can be used, for example, with no limit to the number of layers.

[0049] The correlation determination unit 142 determines a correlation between the acoustic feature and the noise feature by inputting the acoustic feature and the noise feature into the correlation determination NN, which is a correlation determination model trained to determine a correlation between the acoustic feature of speech and noise and the acoustic feature of noise.

[0050] For example, the correlation determination unit 142 inputs the acoustic feature determined by the acoustic feature determination unit 139 and the noise feature determined by the background noise feature determination unit 141 into the correlation determination neural network, which is stored in the correlation determination neural network storage unit 134, and determines a correlation between the two features. The correlation is then passed to the feature integration unit 143.

[0051] The correlation-determining NN could, for example, be an attention NN, as described in the following reference.

[0052] Ashish Vaswani et al., “Attention Is All You Need,” in Proc. NIPS, 2017.

[0053] The feature integration unit 143 calculates an integrated feature by weighting the acoustic feature with the determined correlation.

[0054] For example, feature integration unit 143 integrates the acoustic feature from acoustic feature determination unit 139 and the correlation from correlation determination unit 142. Integration means that these two matrix representations are transformed into a single matrix representation. For example, if the acoustic feature is expressed in a time-frequency representation of N × time, and the correlation is expressed in a time-frequency representation of M × time, these can be linked on a frequency axis other than the time axis to form a time-frequency representation of (N + M) × time, or the number of dimensions on the frequency axis of the acoustic feature and the frequency axis of the correlation can be unified by some kind of dimensionality transformation to obtain the element product of the two matrices.

[0055] The mask determination unit 144 determines a target language mask by inputting the integrated feature into the language enhancement NN, which is a language enhancement model trained to determine a mask for language enhancement.

[0056] For example, the mask determination unit 144 receives the integrated feature from the feature integration unit 143 as input and determines a mask using the speech enhancement NN stored in the speech enhancement NN storage unit 135. The speech enhancement NN is a neural network consisting of multiple layers, and propagation between the layers can be achieved using, for example, a technique similar to LSTM or a technique combining one-dimensional convolution operations, with no limit to the number of layers.

[0057] Here, the mask is a time-frequency representation of the same order of magnitude as the acoustic component, if the acoustic component is a time-frequency representation of N × time. The mask detection unit 144 can determine either a target speech mask that improves the target speech audio content within the mixed audio content, or a noise mask that improves the noise within the mixed audio content. The determined target speech mask, which is a mask that improves the target speech audio content, is passed to the speech recovery unit 145.

[0058] The speech recovery unit 145 restores speech audio content using the target speech audio content from the acoustic component and the target speech mask.

[0059] For example, the speech recovery unit 145 applies the target speech mask from the mask detection unit 144 to the acoustic component from the acoustic component calculation unit 138 and restores an audio signal by, for example, using an inverse short-time Fourier transform (iSTFT).

[0060] Some or all of the above-described noise-mixing audio content acquisition unit 136, noise acquisition unit 137, acoustic component calculation unit 138, acoustic feature detection unit 139, noise component calculation unit 140, noise feature detection unit 141, correlation detection unit 142, feature integration unit 143, mask detection unit 144, and speech recovery unit 145 may, for example, be implemented by the main memory 10 and the processor 11, such as a CPU executing programs stored in memory 10, as in Fig. Figure 3A illustrates this. Such programs can be delivered over a network or can be recorded and delivered on a recording medium. That is, such programs can be delivered, for example, as a single program product.

[0061] Some or all of the noise-mixing audio content acquisition unit 136, the noise acquisition unit 137, the acoustic component calculation unit 138, the acoustic feature detection unit 139, the noise component calculation unit 140, the noise feature detection unit 141, the correlation detection unit 142, the feature integration unit 143, the mask detection unit 144, and the speech recovery unit 145 can, for example, be implemented by a processing circuit 12, such as a single circuit, a composite circuit, a program-driven processor, a program-driven parallel processor, an ASIC, or an FPGA, as in Fig. 3B is shown.

[0062] As described above, the noise-mixing audio content acquisition unit 136, the noise acquisition unit 137, the acoustic component calculation unit 138, the acoustic feature detection unit 139, the noise component calculation unit 140, the noise feature detection unit 141, the correlation detection unit 142, the feature integration unit 143, the mask detection unit 144 and the speech recovery unit 145 can be implemented by a processing circuit.

[0063] The feature detection NN storage unit 132, the noise detection NN storage unit 133, the correlation detection NN storage unit 134 and the speech enhancement NN storage unit 135 can be implemented by a data storage device (not shown), such as an HDD, an SSD or a non-volatile memory.

[0064] The communication unit 131 can be implemented through a communication interface, such as a NIC.

[0065] Fig. Figure 5 is a flowchart showing the operation of the training facility 110 according to the first embodiment.

[0066] First, the audio content mixing unit 113 acquires the training purpose speech data from the speech data storage unit 111 and the training purpose noise data from the noise data storage unit 112, and generates mixed audio content by superimposing the training purpose speech audio content represented by the training purpose speech data and noise from the noise data (step S10).

[0067] Next, the component calculation unit 114 calculates a target language component, which is a mixed audio component from the target language audio content or the mixed audio content, from the audio content mixing unit 113 (step S11).

[0068] Next, the Teacher Mask Determination Unit 115 generates a Teacher Mask from the target language component and the mixed audio component from the Component Calculation Unit 114 (step S12).

[0069] Next, the model training unit 116 receives the mixed audio content from the audio content mixing unit 113, the training purpose noise data from the noise data storage unit 112, and the teacher mask from the teacher mask detection unit 115, and trains the neural networks to generate a feature detection neural network, a noise detection neural network, a correlation detection neural network, and a speech enhancement neural network (step S13). The feature detection neural network, the noise detection neural network, the correlation detection neural network, and the speech enhancement neural network generated by training the neural networks are stored in the model storage unit 117 and sent to the speech enhancement unit 130.

[0070] Fig. Figure 6 is a flowchart showing the operation of the speech enhancement device 130 according to the first embodiment.

[0071] First, the acoustic component computation unit 138 receives the inference purpose mix audio data from the noise mix audio content acquisition unit 136 and computes an acoustic component from the mix audio content specified by the inference purpose mix audio data (step S20).

[0072] Next, the acoustic feature determination unit 139 inputs the acoustic component received from the acoustic component calculation unit 138 into the feature determination NN, which is stored in the feature determination NN storage unit 132, and determines an acoustic feature (step S21).

[0073] The noise component calculation unit 140 receives the inference purpose noise data from the noise acquisition unit 137 and calculates a noise component from the inference purpose noise data (step S22).

[0074] Next, the noise feature detection unit 141 inputs the noise component from the noise component calculation unit 140 into the noise detection NN, which is stored in the noise detection NN storage unit 133, and determines a noise feature (step S23).

[0075] Next, the correlation determination unit 142 inputs the acoustic feature determined by the acoustic feature determination unit 139 and the noise feature determined by the background noise feature determination unit 141 into the correlation determination NN, which is stored in the correlation determination NN storage unit 134, and determines a correlation between the two features (step S24).

[0076] Next, the feature integration unit 143 integrates the acoustic feature from the acoustic feature determination unit 139 and the correlation from the correlation determination unit 142 (step S25). This creates an integrated feature.

[0077] The mask determination unit 144 receives the integrated feature from the feature integration unit 143 as input and determines a mask using the language enhancement NN stored in the language enhancement NN storage unit 135 (step S26).

[0078] Next, the speech recovery unit 145 applies the target speech mask from the mask detection unit 144 to the acoustic component from the acoustic component calculation unit 138, and furthermore restores an audio signal with the improved target speech audio content by, for example, using an inverse short-time Fourier transform (iSTFT) (step S27).

[0079] As described above, according to the first embodiment, speech can also be improved in unknown background noise not included in the training data by extracting features not only from data on mixed background noise audio content containing target speech audio content and background noise, but also from background noise that does not contain the target speech audio content and is assumed to be similar to the mixed background noise, by determining a correlation between the features extracted from the data and the background noise, and by inputting the correlation into a trained model.

[0080] In other words, since, according to the first embodiment, a feature is extracted from the background noise and a correlation between a feature extracted from the background noise and a feature extracted from the background noise superimposed audio content is determined by NNs, speech can be stably improved even with unknown background noise. SECOND VERSION

[0081] In the second embodiment, speech segments are detected in order to distinguish between mixed audio content and background noise.

[0082] As in Fig. Figure 1 shows a speech improvement system 200 according to the second embodiment comprising a training device 110 and a speech improvement device 230.

[0083] The training device 110 of the speech improvement system 200 according to the second embodiment is the same as the training device 110 of the speech improvement system 100 according to the first embodiment.

[0084] Fig. Figure 7 is a block diagram which schematically shows a configuration of the speech enhancement device 230 according to the second embodiment.

[0085] The speech enhancement device 230 comprises a communication unit 131, a feature detection NN storage unit 132, a noise detection NN storage unit 133, a correlation detection NN storage unit 134, a speech enhancement NN storage unit 135, an acoustic component calculation unit 138, an acoustic feature detection unit 139, a noise component calculation unit 140, a noise feature detection unit 141, a correlation detection unit 142, a feature integration unit 143, a mask detection unit 144, a speech recovery unit 145, and a speech segment acquisition unit 246.

[0086] The communication unit 131, the feature detection NN storage unit 132, the noise detection NN storage unit 133, the correlation detection NN storage unit 134, the speech enhancement NN storage unit 135, the acoustic component calculation unit 138, the acoustic feature detection unit 139, the noise component calculation unit 140, the noise feature detection unit 141, the feature integration unit 143, the mask detection unit 144, and the speech restoration unit 145 of the speech enhancement device 230 according to the second embodiment are each identical to the communication unit 131, the feature detection NN storage unit 132, the noise detection NN storage unit 133, the Correlation-determination NN storage unit 134, speech enhancement NN storage unit 135, acoustic component calculation unit 138, acoustic feature determination unit 139,the noise component calculation unit 140, the noise feature detection unit 141, the feature integration unit 143, the mask detection unit 144, and the speech restoration unit 145 of the speech enhancement device 130 according to the first embodiment.

[0087] However, the acoustic component calculation unit 138 uses speech segment data from the speech segment acquisition unit 246 as inference-purpose mixed audio data to calculate an acoustic component from the inference-purpose mixed audio data, and the noise component calculation unit 140 uses non-speech segment data from the speech segment acquisition unit 246 as inference-purpose noise data to calculate a noise component from the inference-purpose mixed noise data.

[0088] The speech segment acquisition unit 246 uses acoustic data which contains segments that contain target speech audio content and segments that do not contain target speech audio content to generate mixed audio data from data about the segments that contain target speech audio content and noise data from data about the segments that do not contain target speech audio content.

[0089] The speech segment detection unit 246, for example, detects speech segments containing speech and non-speech segments containing no speech from the noise represented by acoustic data acquired with a microphone (not shown) acting as a noise collection unit. The speech segment detection unit 246 then outputs the speech segment data (i.e., data about the speech segments in the acoustic data) to the acoustic component calculation unit 138 and the non-speech segment data (i.e., data about the non-speech segments) to the noise component calculation unit 140.

[0090] Here, speech segments can be captured using a known technique, such as the speech segmentation method described in international publication no. WO 2016 / 143125. A speech segment can be determined using a threshold value based on the intensity of the acoustic signal input to the microphone.

[0091] The speech segment acquisition unit 246, which has been described above, can, for example, be provided by a working memory 10 and a processor 11, such as a CPU, which executes the programs stored in the working memory 10, as in Fig. 3A is shown, and should be realized.

[0092] The speech segment acquisition unit 246 can, for example, also be implemented by a single circuit, a composite circuit, a program-driven processor, a program-driven parallel processor, a processing circuit 12, such as an application-specific integrated circuit (ASIC) or a field-programmable gate array (FPGA), as in Fig. 3B is shown, to be realized.

[0093] As described above, the speech segment acquisition unit 246 can be implemented by a processing circuit.

[0094] Fig. Figure 8 is a flowchart showing the operation of the speech enhancement device 230 according to the second embodiment.

[0095] In Fig. 8. The processing steps are the same as the processing in the Fig. The 6 steps shown in the flowchart are labeled with the same reference symbols as the corresponding steps in Fig. 6 is designated.

[0096] First, the speech segment acquisition unit 246 generates speech segment data and non-speech segment data from acoustic data acquired via a microphone, and passes the speech segment data to the acoustic component calculation unit 138 and the non-speech segment data to the noise component calculation unit 140 (step S30). The process then continues with steps S20 and S22.

[0097] The processing of steps S20 to S27 in Fig. 8 is equivalent to the processing in steps S20 to S27 in Fig. 6. However, the speech segment data is treated as inference-purpose mixed audio data, and the non-speech segment data is treated as inference-purpose mixed noise data.

[0098] As described above, according to the second embodiment, speech segment data and non-speech segment data are generated from acoustic data captured with a microphone and can be treated as inference-purpose mixed audio data and inference-purpose background noise data.

[0099] In other words, according to the second embodiment, the detection of speech segments enables the detection of noise-only segments and other segments in mixed audio content and speech, and speech can be stably improved even in unknown background noise without manual addition of background noise. THIRD VERSION

[0100] In the third embodiment, the inference-purpose mixed audio content is divided into blocks in order to process the block-divided inference-purpose mixed audio content, and the interference noise can be recovered.

[0101] As in Fig. Figure 1 shows a speech improvement system 300 according to the third embodiment comprising a training device 310 and a speech improvement device 330.

[0102] Fig. Figure 9 is a block diagram which schematically shows a configuration of the training facility 310 according to the third embodiment.

[0103] The training facility 310 comprises a speech data storage unit 111, a noise data storage unit 112, an audio content mixing unit 113, a component calculation unit 114, a teacher mask detection unit 315, a model training unit 316, a model storage unit 117, a communication unit 118 and a block subdivision unit 319.

[0104] The speech data storage unit 111, the noise data storage unit 112, the audio content mixing unit 113, the component calculation unit 114, the model training unit 116, the model storage unit 117 and the communication unit 118 of the training device 310 according to the third embodiment are each identical to the speech data storage unit 111, the noise data storage unit 112, the audio content mixing unit 113, the component calculation unit 114, the model training unit 116, the model storage unit 117 and the communication unit 118 of the training device 110 according to the first embodiment.

[0105] However, the audio content mixing unit 113 passes the target language audio content and the mixed audio content to the block subdivision unit 319.

[0106] The component calculation unit 114 calculates an acoustic component for each block from the block subdivision unit 319.

[0107] The block subdivision unit 319 subdivides the target language audio content and the mixed audio content from the audio content mixing unit 113 into blocks, each with a fixed time length, and passes the blocks to the component calculation unit 114 and the model training unit 116.

[0108] The teacher mask detection unit 315 performs the same processing as the teacher mask detection unit 115 of the first embodiment and also determines a noise mask that improves noise in the blocks from the block subdivision unit 319 as a teacher mask, and passes the determined teacher mask to the model training unit 316.

[0109] The model training unit 316 receives the blocks from the block subdivision unit 319, the training-purpose noise data from the noise data storage unit 112, and the teacher mask from the teacher mask detection unit 315, and trains a speech enhancement neural network. In the third embodiment, the model training unit 316 trains the speech enhancement neural network while noise is recovered from the blocks using the noise mask from the teacher mask detection unit 315.

[0110] The model training unit 316 receives the blocks from the block subdivision unit 319, the training purpose noise data from the noise data storage unit 112, and the teacher mask from the teacher mask detection unit 315, and retrains the speech enhancement neural network. Again, the model training unit 316 retrains the speech enhancement neural network while noise is recovered from the blocks using the noise mask from the teacher mask detection unit 315.

[0111] The block subdivision unit 319, which has been explained above, can also be, for example, a working memory 10 and a processor 11, such as a CPU, which executes a program stored in the working memory 10, as in Fig. 3A is shown, and should be realized.

[0112] The block subdivision unit 319 can, for example, also be a single circuit, a composite circuit, a program-driven processor, a program-driven parallel processor, a processing circuit 12, such as an application-specific integrated circuit (ASIC) or a field-programmable gate array (FPGA), as in Fig. 3B is shown, to be realized.

[0113] As described above, the block subdivision unit 319 can be implemented by a processing circuit.

[0114] Fig. Figure 10 is a block diagram which schematically shows a configuration of the speech enhancement device 330 according to the third embodiment.

[0115] The speech enhancement device 330 comprises a communication unit 131, a feature detection NN storage unit 132, a noise detection NN storage unit 133, a correlation detection NN storage unit 134, a speech enhancement NN storage unit 135, an acoustic component calculation unit 138, an acoustic feature detection unit 139, a noise component calculation unit 140, a noise feature detection unit 341, a correlation detection unit 342, a feature integration unit 143, a mask detection unit 144, a speech recovery unit 145, a block subdivision unit 347, and a noise recovery unit 348.

[0116] The communication unit 131, the feature detection NN storage unit 132, the noise detection NN storage unit 133, the correlation detection NN storage unit 134, the speech enhancement NN storage unit 135, the acoustic component calculation unit 138, the acoustic feature detection unit 139, the noise component calculation unit 140, the feature integration unit 143, the mask detection unit 144, and the speech restoration unit 145 of the speech enhancement device 330 according to the third embodiment are each equivalent to the communication unit 131, the feature detection NN storage unit 132, the noise detection NN storage unit 133, the correlation detection NN storage unit 134, the Speech enhancement NN storage unit 135, acoustic component calculation unit 138, acoustic feature determination unit 139, noise component calculation unit 140, feature integration unit 143,the mask detection unit 144, and the speech restoration unit 145 of the speech enhancement device 130 according to the first embodiment.

[0117] However, the acoustic component calculation unit 138 calculates an acoustic component from each of the blocks into which the inference purpose mixing audio data was subdivided by the block subdivision unit 347.

[0118] The mask detection unit 144 also passes the detected mask to the speech recovery unit 348. Here, it is sufficient to provide a noise mask, which is a mask for improving noise; however, if the mask detection unit 144 does not detect a noise mask, it generates a noise-improving mask based on a target speech mask and passes the generated noise mask to the noise recovery unit 348. For example, if a teacher mask is expressed as the ratio of a power spectrum of the target speech audio content to a power spectrum of the mixed audio content, a noise mask can be obtained by subtracting one for each element of the mask that improves the target speech audio content in the mixed audio content.

[0119] The block subdivision unit 347 divides the mixed audio data into blocks.

[0120] The block subdivision unit 347, for example, subdivides the inference-purpose mixing audio data from the noise-mixing audio content acquisition unit 136 into blocks, each with a fixed time length, and passes the blocks to the acoustic component calculation unit 138. The acoustic component calculation unit 138 according to the third embodiment calculates an acoustic component for each of the blocks.

[0121] The blocks are preferably subdivided in such a way that an overlay is included, for example as described in the above-mentioned reference “Deep clustering-based single-channel speech separation and recent advances”.

[0122] The noise recovery unit 348 calculated a recovered noise component by improving noise with the acoustic components and the noise mask.

[0123] For example, the speech recovery unit 348 applies the noise mask from the mask detection unit 144 to an acoustic component from the acoustic component calculation unit 138 to calculate a recovered noise component. The recovered noise component is then passed to the noise feature detection unit 341.

[0124] In addition to the processing performed by the noise feature detection unit 141 according to the first embodiment, the noise feature detection unit 341 determines a recovered noise feature by inputting the recovered noise components output by the noise recovery unit 348 into the noise detection NN. The recovered noise feature is combined with a previously determined noise feature in the time direction and output as a combined noise feature to the correlation detection unit 342.

[0125] When the combined noise feature is generated, the correlation determination unit 342 determines a correlation between the acoustic feature and the combined noise feature.

[0126] In addition to performing the processing by the correlation determination unit 142 according to the first embodiment, the correlation determination unit 342, for example, inputs the acoustic feature determined by the acoustic feature determination unit 139 and the combined noise feature determined by the background noise feature determination unit 341 into the correlation determination neural network, which is stored in the correlation determination neural network storage unit 134, in order to determine the correlation between the two features. The correlation is then passed to the feature integration unit 143.

[0127] In the third embodiment, since the noise feature determination unit 341 generates a combined noise feature by combining, in the time direction, the recovered noise feature with a noise feature that is in the block immediately following the block for which a recovered noise component was calculated, the correlation determination unit 342 also determines a correlation from the acoustic feature and the combined noise feature for the block immediately following the block for which the recovered noise component is calculated.

[0128] All or part of the block subdivision unit 347 and the noise recovery unit 348 described above can also be replaced, for example, by a memory 10 and a processor 11, such as a CPU, which executes a program stored in memory 10, as in Fig. 3A is shown, and should be realized.

[0129] Some or all of the block subdivision unit 347 and the noise recovery unit 348 can also be implemented, for example, by a single circuit, a composite circuit, a program-driven processor, a program-driven parallel processor, a processing circuit 12, such as an application-specific integrated circuit (ASIC) or a field-programmable gate array (FPGA), as in Fig. 3B is shown, to be realized.

[0130] As described above, some or all of the block subdivision unit 347 and the noise recovery unit 348 can be implemented by a processing circuit.

[0131] Fig. Figure 11 is a flowchart showing the operation of the training facility 310 according to the third embodiment.

[0132] First, the audio content mixing unit 113 acquires the training purpose speech data from the speech data storage unit 111 and the training purpose noise data from the noise data storage unit 112, and generates mixed audio content by superimposing the training purpose target speech audio content represented by the training purpose speech data and noise represented by the training purpose noise data (step S40).

[0133] Next, the block subdivision unit 319 divides the target language audio content and the mixed audio content from the audio content mixing unit 113 into blocks (step S41).

[0134] Next, the component calculation unit 114 calculates a target language component, which is a mixed audio component from the blocks of the target language audio content or the mixed audio content, from the block subdivision unit 319 (step S42).

[0135] Next, the Teacher Mask Determination Unit 115 generates, as Teacher Masks: a target language mask to highlight the target language audio content in the target language component and the mixed audio component from the Component Calculation Unit 114, and a noise mask to highlight noise (S43).

[0136] Next, the model training unit 316 receives the mixed audio content from the audio content mixing unit 113, the training purpose noise data from the noise data storage unit 112, and the teacher mask from the teacher mask detection unit 115, and trains the neural networks to generate a feature detection neural network, a noise detection neural network, a correlation detection neural network, and a speech enhancement neural network (step S44). The feature detection neural network, the noise detection neural network, the correlation detection neural network, and the speech enhancement neural network generated by training the neural networks are stored in the model storage unit 117.

[0137] Next, the model training unit 316 receives the mixed audio content from the audio content mixing unit 113, the training purpose noise data from the noise data storage unit 112, and the teacher masks from the teacher mask detection unit 115, and retrains the speech enhancement neural network to generate a feature detection neural network, a noise detection neural network, a correlation detection neural network, and a speech enhancement neural network (step S45). The feature detection neural network, the noise detection neural network, the correlation detection neural network, and the speech enhancement neural network generated by retraining the speech enhancement neural network are stored in the model storage unit 117 and sent to the speech enhancement unit 130.

[0138] Fig. Figure 12 is a flowchart showing the operation of the speech enhancement device 330 according to the third embodiment.

[0139] First, the block subdivision unit 347 receives the inference-purpose mixing audio data from the noise-mixing audio content acquisition unit 136 and subdivides the inference-purpose mixing audio data into blocks, each with a fixed time length (step S50). The block subdivision unit 347 then passes the blocks sequentially in chronological order to the acoustic component calculation unit 138.

[0140] Next, the acoustic component calculation unit 138 receives the blocks from the block subdivision unit 347 and calculates acoustic components from the mixed audio content specified in the blocks (step S51).

[0141] Next, the acoustic feature determination unit 139 inputs the acoustic components received from the acoustic component calculation unit 138 into the feature determination NN, which is stored in the feature determination NN storage unit 132, and determines an acoustic feature (step S52).

[0142] The noise component calculation unit 140 receives the inference purpose noise data from the noise acquisition unit 137 and calculates noise components from the inference purpose noise data (step S53).

[0143] Next, the noise feature detection unit 341 inputs the noise components from the noise component calculation unit 140 into the noise detection NN, which is stored in the noise detection NN storage unit 133, and determines a noise feature (step S54).

[0144] Next, the correlation determination unit 342 inputs the acoustic feature determined by the acoustic feature determination unit 139 and the noise feature determined by the background noise feature determination unit 141 into the correlation determination NN, which is stored in the correlation determination NN storage unit 134, and determines the correlation between the two features (step S55).

[0145] Next, the feature integration unit 143 integrates the acoustic feature from the acoustic feature determination unit 139 and the correlation from the correlation determination unit 342 (step S56).

[0146] The mask determination unit 144 receives the integrated feature from the feature integration unit 143 as input and determines a mask using the language enhancement NN stored in the language enhancement NN storage unit 135 (step S57).

[0147] Next, the speech recovery unit 145 applies the target speech mask from the mask detection unit 144 to the acoustic components from the acoustic component calculation unit 138, and furthermore restores an audio signal with the improved target speech audio content by, for example, using an inverse short-time Fourier transform (iSTFT) (step S58).

[0148] For example, the noise recovery unit 348 applies the noise mask from the mask detection unit 144 to the acoustic components from the acoustic component calculation unit 138 to calculate recovered noise components.

[0149] Block subdivision unit 347 then determines whether or not any blocks remain that have not been passed to acoustic component calculation unit 138 (step S60). If such a block remains (Yes in step S60), the process returns to steps S51 and S54, and if such a block does not remain (No in step S60), the process ends.

[0150] When the process goes back from step S60 to step S54, the noise feature detection unit 341 inputs the recovered noise components, which were recovered by the noise recovery unit 348, into the noise detection NN stored in the noise detection NN storage unit 133, and determines a recovered noise feature.

[0151] Then, in step S55, the correlation determination unit 342 inputs the acoustic feature determined by the acoustic feature determination unit 139 and the recovered noise feature determined by the background noise feature determination unit 141 into the correlation determination NN, which is stored in the correlation determination NN storage unit 134, and determines a correlation between the two features.

[0152] As described above, according to the third embodiment, it is possible to reduce the processing delay during speech improvement by block processing. Furthermore, it is possible to improve stability against unknown noise by using noise extracted from a block immediately preceding the block being improved. Additionally, the model can be trained efficiently during training by first training it using noise extracted from an immediately preceding block, employing a true mask determined by the teacher mask determination unit, and then training it again using noise extracted with the mask determined from an immediately preceding block, as in the case of inference. FOURTH VERSION

[0153] In the fourth embodiment, the likelihood of a recovered noise component is calculated to determine whether or not the recovered noise component should be used.

[0154] As in Fig. Figure 1 shows a speech improvement system 400 according to the fourth embodiment comprising a training device 310 and a speech improvement device 430.

[0155] The training device 310 of the speech enhancement system 400 according to the fourth embodiment is the same as the training device 310 of the speech enhancement system 300 according to the third embodiment.

[0156] Fig. Figure 13 is a block diagram which schematically shows a configuration of the speech enhancement device 430 according to the fourth embodiment.

[0157] The speech enhancement device 430 comprises a communication unit 131, a feature detection NN storage unit 132, a noise detection NN storage unit 133, a correlation detection NN storage unit 134, a speech enhancement NN storage unit 135, an acoustic component calculation unit 138, an acoustic feature detection unit 139, a noise component calculation unit 140, a noise feature detection unit 441, a correlation detection unit 142, a feature integration unit 143, a mask detection unit 144, a speech recovery unit 145, a block subdivision unit 347, a noise recovery unit 348, and the noise likelihood determination unit. 449.

[0158] The communication unit 131, the feature detection NN storage unit 132, the noise detection NN storage unit 133, the correlation detection NN storage unit 134, the speech enhancement NN storage unit 135, the acoustic component calculation unit 138, the acoustic feature detection unit 139, the noise component calculation unit 140, the feature integration unit 143, the mask detection unit 144, and the speech restoration unit 145 of the speech enhancement device 430 according to the fourth embodiment are each equivalent to the communication unit 131, the feature detection NN storage unit 132, the noise detection NN storage unit 133, the correlation detection NN storage unit 134, the Speech enhancement NN storage unit 135, acoustic component calculation unit 138, acoustic feature determination unit 139, noise component calculation unit 140, feature integration unit 143,the mask detection unit 144, and the speech restoration unit 145 of the speech enhancement device 130 according to the first embodiment.

[0159] However, the acoustic component calculation unit 138 calculates an acoustic component from each of the blocks into which the inference purpose mixing audio data was subdivided by the block subdivision unit 347.

[0160] The mask detection unit 144 also passes the detected mask to the speech recovery unit 348. Here, it is sufficient to provide a noise mask; however, if the mask detection unit 144 does not detect a noise mask, it generates a mask that enhances noise based on a target speech mask and passes the generated noise mask to the noise recovery unit 348. For example, if a teacher mask is expressed as the ratio of a power spectrum of the target speech audio content to a power spectrum of the mixed audio content, a noise mask can be obtained by subtracting one for each element of the mask that enhances the target speech audio content in the mixed audio content.

[0161] The block subdivision unit 347 and the noise recovery unit 348 of the speech enhancement device 430 according to the fourth embodiment are each identical to the block subdivision unit 347 and the noise recovery unit 348 of the speech enhancement device 330 according to the third embodiment.

[0162] However, the noise recovery unit 348 according to the fourth embodiment provides recovered noise components to the noise likelihood determination unit 449.

[0163] The noise likelihood determination unit 449 calculates a noise likelihood, which is the likelihood of a recovered noise component, and determines whether or not the noise likelihood is equal to or greater than a predetermined threshold.

[0164] For example, the noise likelihood determination unit 449 receives the recovered noise components from the noise recovery unit 348 and calculates their noise likelihood. The noise likelihood determination unit 449 then passes the recovered noise components corresponding to the blocks with a noise likelihood equal to or greater than the threshold to the noise feature determination unit 441.

[0165] If a recovered noise component is a time-frequency representation of N × time, the noise likelihood is calculated for each block corresponding to a time frame. The noise likelihood can be calculated using a neural network, as described, for example, in NPL 1. Here, the threshold can be determined experimentally, for instance.

[0166] For a block immediately following the block corresponding to the recovered noise component received by the noise likelihood determination unit 449, the noise feature determination unit 441 outputs the recovered noise component output by the noise likelihood determination unit 449 to the noise determination NN to determine a recovered noise feature and computes a combined noise feature by combining the recovered noise feature with the noise feature determined from the noise component received by the noise component calculation unit 140 in the time direction.

[0167] In contrast, for a block immediately following the block for which no recovered noise component was received by the noise likelihood determination unit 449, the noise feature determination unit 441 determines a noise feature from the noise component received by the noise component calculation unit 140.

[0168] The noise feature determination unit 441 then outputs the combined noise feature to the correlation determination unit 342 for the block immediately following the block corresponding to the recovered noise component received by the noise likelihood determination unit 449, and outputs the noise features for the other blocks to the correlation determination unit 342.

[0169] In other words, the noise feature detection unit 441 generates a combined noise feature when the noise likelihood is equal to or greater than the threshold.

[0170] For a block for which the noise feature detection unit 441 has received a noise feature, the correlation detection unit 442 inputs the acoustic feature determined by the acoustic feature detection unit 139 and the noise feature determined by the noise feature detection unit 341 into the correlation detection NN, which is stored in the correlation detection NN storage unit 134, and determines a correlation between the two features.

[0171] On the other hand, for a block for which the noise feature detection unit 441 has received a combined noise feature, the correlation detection unit 442 inputs the acoustic feature determined by the acoustic feature detection unit 139 and the noise feature determined by the noise feature detection unit 341 into the correlation detection network stored in the correlation detection network storage unit 134, and determines a correlation between the two features. The correlation is then passed to the feature integration unit 143.

[0172] The noise likelihood determination unit 449, which has been explained above, can also be implemented, for example, by a main memory 10 and a processor 11, such as a CPU, which executes a program stored in the main memory 10, as in Fig. 3A is shown, and should be realized.

[0173] The noise likelihood determination unit 449 can, for example, also be implemented by a single circuit, a composite circuit, a program-driven processor, a program-driven parallel processor, a processing circuit 12, such as an application-specific integrated circuit (ASIC) or a field-programmable gate array (FPGA), as in Fig. 3B is shown, to be realized.

[0174] As described above, the noise likelihood determination unit 449 can be implemented by a processing circuit.

[0175] Fig. Figure 14 is a flowchart showing the operation of the speech enhancement device 430 according to the fourth embodiment.

[0176] In Fig. 14. The process steps, which are identical to the process in the one described in Fig. The 12 steps shown in the flowchart are labeled with the same reference symbols as the corresponding steps in Fig. 12 is designated.

[0177] The processing of steps S50 to S53 in Fig. 14 is the same as the processing of steps S50 to S53 in Fig. 12. In Fig. 14 However, after processing step S53, the process continues with step S70.

[0178] In step S70, the noise feature detection unit 441 determines a noise feature from the noise component received by the noise component calculation unit 140. The process then continues with step S55.

[0179] The processing of steps S55 to S59 in Fig. 14 is the same as the processing of steps S55 to S59 in Fig. 12. In Fig. However, after processing step S59, the process continues with step S71.

[0180] In step S71, the noise likelihood determination unit 449 receives a recovered noise component from the noise recovery unit 348 and calculates the noise likelihood. If the calculated noise likelihood is equal to or greater than the threshold, the noise likelihood determination unit 449 passes the recovered noise component to the noise feature determination unit 441. The process then continues with step S60.

[0181] Then, in step S60, the block subdivision unit 347 determines whether or not there are any remaining blocks that have not been passed to the acoustic component calculation unit 138. If such a block remains (Yes in step S60), the process returns to steps S51 and S70, and if such a block does not remain (No in step S60), the process ends.

[0182] When the process returns from step S60 to step S70, and the noise feature determination unit 441 receives a recovered noise component from the noise likelihood determination unit 449, the noise feature determination unit 441 calculates a combined noise feature by inputting the recovered noise component, which was recovered by the noise recovery unit 348, to the noise determination NN stored in the noise determination NN storage unit 133, determining the recovered noise feature, and combining the recovered noise feature with the noise feature determined from the noise component received in the time direction by the noise component calculation unit 140.

[0183] In this case, in step S55, the correlation determination unit 342 inputs the acoustic feature determined by the acoustic feature determination unit 139 and the recovered noise feature determined by the background noise feature determination unit 141 into the correlation determination NN, which is stored in the correlation determination NN storage unit 134, and determines a correlation between the two features.

[0184] As described above, according to the fourth embodiment, it is possible to prevent the learning of noise containing a detection error by using only a portion of the noise detected from an immediately preceding block with a high noise likelihood. REFERENCE MARK LIST

[0185] 100, 200, 300, 400 Speech Enhancement System; 110, 310 Training Unit; 111 Speech Data Storage Unit; 112 Noise Data Storage Unit; 113 Audio Content Mixer Unit; 114 Component Computing Unit; 115, 315 Teacher Mask Detection Unit; 116, 316 Model Training Unit; 117 Model Storage Unit; 118 Communication Unit; 319 Block Subdivision Unit; 130, 230, 330, 430 Speech Enhancement Unit; 131 Communication Unit; 132 Feature Detection NN Storage Unit; 133 Noise Detection NN Storage Unit; 134 Correlation Detection NN Storage Unit; 135 Speech Enhancement NN Storage Unit; 136 Noise-mixed audio content acquisition unit; 137 Noise acquisition unit; 138 Acoustic component calculation unit; 139 Acoustic feature determination unit; 140 Noise component calculation unit; 141, 341, 441 Noise feature determination unit; 142, 342, 442 Correlation determination unit; 143 Feature integration unit;144 Mask detection unit; 145 Speech recovery unit; 246 Speech segment detection unit; 347 Block subdivision unit; 348 Noise recovery unit; 449 Noise likelihood determination unit.;

Claims

[1] Information processing equipment (130, 230, 330, 430), comprising: an acoustic component computation unit (138) configured to compute an acoustic component from mixed audio data using a predetermined function, wherein the mixed audio data includes target speech audio content and mixed noise, wherein the target speech audio content is a target to be enhanced, wherein the mixed noise is noise to be mixed with the target speech audio content, and the acoustic component is a component of the target speech audio content and the mixed noise; an acoustic feature detection unit (139) configured to detect an acoustic feature by inputting the acoustic component into a feature detection model trained to detect an acoustic feature of speech and mixed background noise; a noise component calculation unit (140) configured to calculate a noise component from noise data using the predetermined function, wherein the noise data does not contain the target speech audio content but does contain noise; a noise feature detection unit (141, 341, 441) configured to detect a noise feature by the background noise component is fed into a background noise detection model that is trained to identify an acoustic feature of the background noise; a correlation determination unit (142, 342, 442) that is set up to determine a correlation between the acoustic feature and the noise feature by inputting the acoustic feature and the noise feature into a correlation determination model that is trained to determine a correlation between the acoustic feature and the noise feature; a feature integration unit (143) that is set up to calculate an integrated feature by weighting the acoustic feature with the determined correlation; a mask detection unit (144) configured to determine a target language mask by inputting the integrated feature into a language enhancement model trained to determine a language enhancement mask; and a speech recovery unit (145) which is configured to recover speech audio content from the acoustic component and the target speech mask in which the target speech audio content is enhanced. [2] Information processing device (130) according to claim 1, further comprising: an interface unit (131) that is set up to accept input of data; a noise-mixing audio content acquisition unit (136) configured to acquire the mixed audio data via the interface unit (131); and a noise acquisition unit (137) which is configured to acquire noise data via the interface unit (131). [3] Information processing device (230) according to claim 1, further comprising: an interface unit (131) that is set up to accept input of data; a speech segment acquisition unit (246) which is configured to acquire acoustic data via the interface unit (131) with a segment containing the target speech audio content and a segment not containing the target speech audio content, to generate the mixed audio data from data about the segment containing the target speech audio content, and to generate the noise data from data about the segment not containing the target speech audio content. [4] Information processing device (330, 430) according to one of claims 1 to 3, further comprising: a block subdivision unit (347) configured to subdivide the mixed audio data into a multitude of blocks, where the acoustic component calculation unit (138) calculates the acoustic component for the blocks. [5] Information processing device (330, 430) according to claim 4, wherein the mask detection unit (144) also determined a noise mask for improving noise reduction, the information processing unit (330, 430) further comprises a noise recovery unit (348) which is configured to calculate a recovered noise component by highlighting the noise with the acoustic component and the noise mask, The noise feature detection unit (341, 441) determines a recovered noise feature by inputting the recovered noise component into the noise detection model, and creates a combined noise feature by combining the recovered noise feature with the noise feature in a time direction, and When the combined noise feature is generated, the correlation determination unit (342, 442) determines the correlation from the acoustic feature and the combined noise feature. [6] Information processing device (330) according to claim 5, wherein the noise feature detection unit (341) generates a combined noise feature by combining, in a temporal direction, the recovered noise feature with the noise feature that is calculated for a block immediately following the block for which the recovered noise component was calculated. [7] Information processing device (430) according to claim 5, further comprising: a noise likelihood determination unit (449) configured to calculate a noise likelihood and to determine whether or not the noise likelihood is equal to or greater than a predetermined threshold, wherein the noise likelihood is a likelihood of the recovered noise component, wherein the noise feature detection unit (441) generates the combined noise feature if the noise likelihood is equal to or greater than the threshold. [8] Program causing a computer to operate as: an acoustic component computation unit (138) configured to compute an acoustic component from mixed audio data using a predetermined function, wherein the mixed audio data includes target speech audio content and mixed noise, wherein the target speech audio content is a target to be enhanced, wherein the mixed noise is noise to be mixed with the target speech audio content, and the acoustic component is a component of the target speech audio content and the mixed noise; an acoustic feature detection unit (139) configured to detect an acoustic feature by inputting the acoustic component into a feature detection model trained to detect an acoustic feature of speech and mixed background noise; a noise component calculation unit (140) configured to calculate a noise component from noise data using the predetermined function, wherein the noise data does not contain the target speech audio content and contains noise; a noise feature detection unit (141, 341, 441) configured to detect a noise feature by the background noise component is fed into a background noise detection model that is trained to identify an acoustic feature of the background noise; a correlation determination unit (142, 342, 442) that is set up to determine a correlation between the acoustic feature and the noise feature by inputting the acoustic feature and the noise feature into a correlation determination model that is trained to determine a correlation between the acoustic feature and the noise feature; a feature integration unit (143) that is set up to calculate an integrated feature by weighting the acoustic feature with the determined correlation; a mask detection unit (144) configured to determine a target language mask by inputting the integrated feature into a language enhancement model trained to determine a language enhancement mask; and a speech recovery unit (145) which is configured to recover speech audio content from the acoustic component and the target speech mask in which the target speech audio content is enhanced. [9] Information processing techniques, comprehensive: Computation of an acoustic component from mixed audio data using a predetermined function, wherein the mixed audio data contains target speech audio content and mixed noise, where the target speech audio content is a target to be improved, where the mixed noise is noise to be mixed with the target speech audio content, and the acoustic component is a component of the target speech audio content and the mixed noise; Determining an acoustic feature by inputting the acoustic component into a feature detection model that is trained to determine an acoustic feature of speech and mixed background noise; Calculating a noise component from noise data using the predetermined function, where the noise data does not contain the target speech audio content and contains noise; Determining a noise characteristic by inputting the noise component into a noise detection model that is trained to determine an acoustic characteristic of the noise; Determining a correlation between the acoustic feature and the noise feature by inputting the acoustic feature and the noise feature into a correlation determination model that is trained to determine a correlation between the acoustic feature and the noise feature; Calculating an integrated feature by weighting the acoustic feature with the determined correlation; Determining a target language mask by inputting the integrated feature into a language enhancement model that is trained to determine a language enhancement mask; and Restore speech audio content from the acoustic component and the target language mask in which the target language audio content is enhanced.

Images