Information processing device, program, and information processing method

Abstract

A speech enhancement device (130) is provided with: a sound component calculation unit (138) that calculates sound components on the basis of mixed speech data; an acoustic feature amount estimation unit (139) that estimates an acoustic feature amount by inputting the acoustic component into the feature estimation model; a noise component calculation unit (140) that calculates a noise component from the noise data; a noise feature amount estimation unit (141) that estimates a noise feature amount by inputting the noise component into the noise estimation model; a correlation estimation unit (142) that estimates the correlation between the sound feature amount and the noise feature amount by inputting the sound feature amount and the noise feature amount into a correlation estimation model; a feature quantity integration unit (143) that calculates an integrated feature quantity by weighting the acoustic feature quantity using the estimated correlation; a mask estimation unit (144) that estimates a target voice mask by inputting the integrated feature quantity into a voice emphasis model; and a speech restoration unit (145) that restores speech in which the target speech is emphasized on the basis of the sound component and the target speech mask.

Description

Technical Field

[0001] The present disclosure relates to an information processing device, a program, and an information processing method. Background Art

[0002] In the past, speech recognition processing for recognizing speech has been performed. Generally, speech recognition processing is affected by noise other than the target speech, so when noise is included, the accuracy of speech recognition is greatly reduced. Therefore, it is necessary to extract the target speech from the speech mixed with noise.

[0003] For example, Non-Patent Document 1 proposes a method of using a neural network (NN) to learn paired data of a mixed speech and a target speech and extract the target speech from the mixed speech.

[0004] Prior art literature

[0005] Non-patent literature

[0006] Non-patent literature 1: Felix Weniger, 1st outsider, "Discriminatively trained recurrent neural networks for single-channel speech separation", IEEE Global Conference on Signal and Information Processing (GlobalSIP), February 2015 Summary of the invention

[0007] Problem that the invention aims to solve

[0008] However, the conventional method of learning a mixed sound of speech and non-speech has a problem that the speech emphasis performance is degraded for non-speech that has not been learned.

[0009] Therefore, an object of one or more aspects of the present disclosure is to enable speech emphasis even for unknown noise not included in learning data.

[0010] Means used to solve problems

[0011] An information processing device according to an aspect of the present disclosure is characterized in that it comprises: an acoustic component calculation unit that uses a predetermined function to calculate an acoustic component from mixed speech data, the mixed speech data including a target speech to be emphasized and noise mixed with the target speech, that is, mixed noise, the acoustic component being a component of the target speech and the mixed noise; an acoustic feature quantity estimation unit that estimates an acoustic feature quantity by inputting the acoustic component into a feature estimation model learned for estimating acoustic feature quantities of speech and noise; a noise component calculation unit that uses the predetermined function to calculate a noise component that is a component of noise from noise data that does not include the target speech but includes noise; and a noise feature quantity estimation unit that inputs the noise component into a feature estimation model learned for estimating acoustic feature quantities of speech and noise. a correlation estimating unit that estimates the correlation between the acoustic feature quantity and the noise feature quantity by inputting the acoustic feature quantity and the noise feature quantity into the correlation estimating model that has been learned in order to estimate the correlation between the acoustic feature quantity of speech and noise and the acoustic feature quantity of noise; a feature quantity integrating unit that calculates a comprehensive feature quantity by weighting the acoustic feature quantity using the estimated correlation; a masking estimating unit that estimates the target speech masking by inputting the comprehensive feature quantity into a speech emphasis model that has been learned in order to estimate the masking for emphasizing the speech; and a speech restoration unit that restores the speech with the target speech emphasized based on the acoustic component and the target speech masking.

[0012] A program according to one aspect of the present disclosure is characterized in that a computer is caused to function as the following units: an acoustic component calculation unit that uses a predetermined function to calculate an acoustic component from mixed speech data, the mixed speech data including a target speech to be emphasized and noise mixed with the target speech, that is, mixed noise, the acoustic component being a component of the target speech and the mixed noise; an acoustic feature quantity estimation unit that estimates an acoustic feature quantity by inputting the acoustic component into a feature estimation model learned for estimating acoustic feature quantities of speech and noise; a noise component calculation unit that uses the predetermined function to calculate a noise component that is a component of noise from noise data that does not include the target speech but includes noise; and a noise feature quantity estimation unit that estimates an acoustic feature quantity by inputting the noise component into a feature estimation model learned for estimating acoustic feature quantities of speech and noise. A noise estimation model learned for estimating an acoustic feature of noise is used to estimate a noise feature; a correlation estimation unit estimates the correlation between the acoustic feature and the noise feature by inputting the acoustic feature and the noise feature into a correlation estimation model learned for estimating the correlation between the acoustic feature of speech and noise and the acoustic feature of noise; a feature quantity synthesis unit calculates a synthesis feature by weighting the acoustic feature using the estimated correlation; a masking estimation unit estimates a target speech masking by inputting the synthesis feature into a speech emphasis model learned for estimating a masking for emphasizing speech; and a speech restoration unit restores the speech with the target speech emphasized based on the acoustic component and the target speech masking.

[0013] An information processing method according to an aspect of the present disclosure is characterized in that a sound component is calculated from mixed speech data using a predetermined function, the mixed speech data including a target speech to be emphasized and noise mixed with the target speech, namely, mixed noise, the sound component being a component of the target speech and the mixed noise, the sound feature quantity is estimated by inputting the sound component into a feature estimation model learned for estimating the sound feature quantity of speech and noise, the component of noise, namely, noise component is calculated from noise data not including the target speech but including noise, using the predetermined function, the component of noise, namely, noise component is calculated by inputting the noise component into a feature estimation model learned for estimating the sound feature quantity of noise, The method comprises the steps of: estimating the noise feature quantity by using a noise estimation model learned for estimating the acoustic feature quantity of speech and noise, estimating the correlation between the acoustic feature quantity of speech and noise and the acoustic feature quantity of noise by inputting the acoustic feature quantity and the noise feature quantity into a correlation estimation model learned for estimating the correlation between the acoustic feature quantity of speech and noise and the acoustic feature quantity of noise, weighting the acoustic feature quantity using the estimated correlation to calculate a comprehensive feature quantity, estimating the target speech masking by inputting the comprehensive feature quantity into a speech emphasis model learned for estimating the masking for emphasizing speech, and restoring the speech with the target speech emphasized according to the acoustic component and the target speech masking.

[0014] Effects of the Invention

[0015] According to one or more aspects of the present disclosure, speech emphasis can be performed even for unknown noise that is not included in learning data. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 This is a block diagram schematically showing the configuration of the speech emphasis system according to Embodiments 1 to 4.

[0017] Figure 2 This is a block diagram schematically showing the configuration of the learning device in the first embodiment.

[0018] Figure 3 (A) and (B) are block diagrams showing hardware configuration examples.

[0019] Figure 4 This is a block diagram schematically showing the configuration of the speech emphasis device in the first embodiment.

[0020] Figure 5 This is a flowchart showing the operation of the learning device in the first embodiment.

[0021] Figure 6 This is a flowchart showing the operation of the speech emphasis device in the first embodiment.

[0022] Figure 7This is a block diagram schematically showing the configuration of a speech emphasis device in the second embodiment.

[0023] Figure 8 This is a flowchart showing the operation of the speech emphasis device in the second embodiment.

[0024] Fig. 9 This is a block diagram schematically showing the configuration of a learning device in the third embodiment.

[0025] Fig.10 This is a block diagram schematically showing the configuration of a speech emphasis device in the third embodiment.

[0026] Fig.11 This is a flowchart showing the operation of the learning device in the first embodiment.

[0027] Fig.12 This is a flowchart showing the operation of the speech emphasis device in the third embodiment.

[0028] Fig.13 This is a block diagram schematically showing the configuration of a speech emphasis device in a fourth embodiment.

[0029] Fig.14 This is a flowchart showing the operation of the speech emphasis device in the fourth embodiment. DETAILED DESCRIPTION

[0030] Implementation method 1.

[0031] Figure 1 This is a block diagram schematically showing the configuration of the speech emphasis system 100 according to the first embodiment.

[0032] The speech emphasis system 100 includes a learning device 110 and a speech emphasis device 130 as an information processing device.

[0033] The learning device 110 learns learning models that function as a feature estimation NN, a noise estimation NN, a correlation estimation NN, and a speech emphasis NN, wherein the feature estimation NN is a feature estimation model for estimating acoustic feature quantities of speech and noise, the noise estimation NN is a noise estimation model for estimating acoustic feature quantities of noise, the correlation estimation NN is a correlation estimation model for estimating the correlation between acoustic feature quantities of speech and noise and acoustic feature quantities of noise, and the speech emphasis NN is a speech emphasis model for estimating a mask for emphasizing speech.

[0034] The speech emphasis device 130 obtains the learned feature estimation NN, noise estimation NN, correlation estimation NN, and speech emphasis NN from the learning device 110, and uses these learned models to speech emphasize the target speech from the mixed speech.

[0035] The learning phase is a phase in which the learning device 110 learns the learning model used by the speech emphasis device 130 .

[0036] The inference phase is a phase in which the speech emphasizing device 130 performs speech emphasis on the target speech from the mixed speech using the learning model learned by the learning device.

[0037] Figure 2 This is a block diagram schematically showing the configuration of the learning device 110 in the first embodiment.

[0038] The learning device 110 includes a speech data storage unit 111 , a noise data storage unit 112 , a speech mixing unit 113 , a component calculation unit 114 , a training mask estimation unit 115 , a model learning unit 116 , a model storage unit 117 , and a communication unit 118 .

[0039] The speech data storage unit 111 stores speech data for learning which represents a target speech for learning, that is, a target speech for learning.

[0040] The noise data storage unit 112 stores learning noise data indicating learning noise.

[0041] The speech mixing unit 113 obtains the learning speech data from the speech data storage unit 111 and the learning noise data from the noise data storage unit 112, selects the learning target speech shown in the learning speech data and the learning noise shown in the learning noise data, superimposes them to generate a mixed speech, and provides the target speech and the mixed speech to the component calculation unit 114.

[0042] The component calculation unit 114 calculates the target speech component, i.e., the target speech component, from the speech mixing unit 113, and calculates the mixed speech component, i.e., the mixed speech component, from the speech mixing unit 113. For example, the component calculation unit 114 uses the time series of the power spectrum calculated by the short-time Fourier transform (STFT) based on the speech signal as the speech component. The target speech component and the mixed speech component are provided to the training masking estimation unit 115.

[0043] The training mask estimation unit 115 generates a training mask based on the target speech component and the mixed speech component from the component calculation unit 114. For example, the training mask estimates the power spectra of the target speech and the mixed speech respectively based on the target speech component and the mixed speech component, and uses the ratio of the target speech to the power spectrum of the mixed speech as the training mask. The training mask is provided to the model learning unit 116.

[0044] The model learning unit 116 receives the mixed speech from the speech mixing unit 113, receives the learning noise data from the noise data storage unit 112, receives the training mask from the training mask estimation unit 115, and learns the NN. The learning of the NN is the process of determining the input weight coefficients as the parameters of the NN. During the learning, the feature estimation NN, the noise estimation NN, the correlation estimation NN and the speech emphasis NN are combined, for example, input into the loss function shown in the following literature, and the error is calculated based on the target speech for learning. Then, for example, an optimization method such as Adam (Adaptive Moment Estimation) is used, for example, based on the inverse error propagation method to learn the input weight coefficients of each layer of the feature estimation NN, the noise estimation NN, the correlation estimation NN and the speech emphasis NN. In addition, the generated feature estimation NN, noise estimation NN, correlation estimation NN and speech emphasis NN are stored in the model storage unit 117.

[0045] Literature: R.Aihara et al., "Deep clustering-based single-channel speech separation and recent advances", Acoust.Sci.&Tech.41.2, 2020

[0046] The model storage unit 117 stores the feature estimation NN, the noise estimation NN, the correlation estimation NN, and the speech emphasis NN learned by the model learning unit 116 .

[0047] The communication unit 118 functions as a transmission unit that transmits the feature estimation NN, the noise estimation NN, the correlation estimation NN, and the speech sound emphasis NN stored in the model storage unit 117 to the speech sound emphasis device 130 .

[0048] Part or all of the speech mixing unit 113, component calculation unit 114, training mask estimation unit 115, and model learning unit 116 described above can be, for example, Figure 3 As shown in (A), the learning device is composed of a memory 10 and a processor 11 such as a CPU (Central Processing Unit) that executes the program stored in the memory 10. In other words, the learning device can be implemented by a so-called computer. Such a program can be provided via a network, and can also be provided by recording in a recording medium. That is, such a program can also be provided as a program product, for example.

[0049] In addition, part or all of the speech mixing unit 113, the component calculation unit 114, the training mask estimation unit 115, and the model learning unit 116 may also be configured as follows. Figure 3As shown in (B), the processing circuit 12 is composed of a single circuit, a composite circuit, a processor operated by a program, a parallel processor operated by a program, an ASIC (Application Specific Integrated Circuit) or an FPGA (Field Programmable Gate Array).

[0050] As described above, the speech mixing unit 113 , the component calculation unit 114 , the training mask estimation unit 115 , and the model learning unit 116 can be realized by a processing circuit network.

[0051] The speech data storage unit 111 , the noise data storage unit 112 , and the model storage unit 117 can be implemented by a storage device (not shown) such as a HDD (Hard Disk Drive), an SSD (Solid State Drive), or a nonvolatile memory.

[0052] In addition, the communication unit 118 can be realized by a communication interface such as a NIC (Network Interface Card).

[0053] Figure 4 This is a block diagram schematically showing the configuration of the speech emphasis device 130 in the first embodiment.

[0054] The speech emphasis device 130 includes a communication unit 131, a feature estimation NN storage unit 132, a noise estimation NN storage unit 133, a correlation estimation NN storage unit 134, a speech emphasis NN storage unit 135, a noise mixed speech acquisition unit 136, a noise acquisition unit 137, an acoustic component calculation unit 138, an acoustic feature quantity estimation unit 139, a noise component calculation unit 140, a noise feature quantity estimation unit 141, a correlation estimation unit 142, a feature quantity integration unit 143, a masking estimation unit 144 and a speech restoration unit 145.

[0055] The communication unit 131 functions as a receiving unit that receives the feature estimation NN, the noise estimation NN, the correlation estimation NN, and the speech emphasis NN from the learning device 110 .

[0056] The feature estimated NN storage unit 132 stores the feature estimated NN received by the communication unit 131 .

[0057] The noise estimation NN storage unit 133 stores the noise estimation NN received by the communication unit 131 .

[0058] The correlation estimated NN storage unit 134 stores the correlation estimated NN received by the communication unit 131 .

[0059] The speech emphasis NN storage unit 135 stores the speech emphasis NN received by the communication unit 131 .

[0060] The noise mixed speech acquisition unit 136 acquires mixed speech data representing a mixed speech including a target speech and noise, which is recorded by a microphone (not shown) functioning as a speech collection unit. The noise contained in the mixed speech data is also referred to as mixed noise. For example, the noise mixed speech acquisition unit 136 can acquire the mixed speech data via the communication unit 131, and can also acquire the mixed speech data from a microphone connected to a connection interface such as a USB (Universal Serial Bus). The mixed speech data here is also referred to as mixed speech data for inference, and the mixed speech shown in the mixed speech data for inference is also referred to as mixed speech for inference. In addition, the communication unit 131 or the connection interface functions as an interface (input interface) or an interface unit (input interface unit) for receiving input of data.

[0061] The noise acquisition unit 137 acquires noise data representing noise that does not include the target voice recorded by the microphone. For example, the noise mixed voice acquisition unit 136 can acquire the noise data via the communication unit 131, and can also acquire the noise data from the microphone connected to the connection interface. Here, as the noise, for example, the sound before and after a fixed time of inputting the mixed voice including the target voice and noise into the microphone can be used. In addition, the noise data here is also referred to as inference noise data, and the noise shown in the inference noise data is also referred to as inference noise.

[0062] The acoustic component calculation unit 138 calculates the acoustic component from mixed speech data including a target speech to be emphasized and mixed noise which is noise mixed with the target speech, using a predetermined function.

[0063] For example, the acoustic component calculation unit 138 receives the mixed speech data for inference from the noise mixed speech acquisition unit 136, and calculates the acoustic component based on the mixed speech shown in the mixed speech data for inference. The acoustic component is, for example, a time series of a power spectrum calculated from a speech signal by short-time Fourier transform (STFT). The acoustic component is provided to the acoustic feature estimation unit 139 and the speech restoration unit 145.

[0064] The acoustic feature quantity estimation unit 139 estimates the acoustic feature quantity by inputting the acoustic component from the acoustic component calculation unit 138 to a feature estimation NN which is a feature estimation model learned to estimate the acoustic feature quantity of speech and noise.

[0065] For example, the acoustic feature estimation unit 139 inputs the acoustic component received from the acoustic component calculation unit 138 to the feature estimation NN stored in the feature estimation NN storage unit 132, and estimates the acoustic feature. The feature estimation NN is a neural network composed of multiple layers. For the propagation between layers, for example, a method similar to LSTM (Long Short Term Memory) or a method combining one-dimensional convolution operations can be used, and the number of layers is arbitrary. The acoustic feature is provided to the correlation estimation unit 142 and the feature synthesis unit 143.

[0066] The noise component calculation unit 140 calculates the noise component from the noise data that does not include the target speech but includes noise, using a predetermined function.

[0067] For example, the noise component calculation unit 140 receives the inference noise data from the noise acquisition unit 137, and calculates the noise component based on the noise shown in the inference noise data. The noise component is, for example, a time series of a power spectrum calculated by a short-time Fourier transform (STFT) based on a speech signal. The noise component is provided to the noise feature estimation unit 141.

[0068] The noise feature quantity estimation unit 141 estimates the noise feature quantity by inputting the noise component from the noise component calculation unit 140 to a noise estimation NN which is a noise estimation model learned to estimate the acoustic feature quantity of noise.

[0069] For example, the noise feature estimation unit 141 estimates the noise feature by inputting the noise component from the noise component calculation unit 140 into the noise estimation NN stored in the noise estimation NN storage unit 133. Here, the noise estimation NN is a neural network composed of a plurality of layers, and regarding the propagation between layers, for example, a method similar to LSTM or a method combining one-dimensional convolution operations can also be used, and the number of layers is arbitrary.

[0070] The correlation estimation unit 142 estimates the correlation between the acoustic feature quantity and the noise feature quantity by inputting the acoustic feature quantity and the noise feature quantity into a correlation estimation NN which is a correlation estimation model learned to estimate the correlation between the acoustic feature quantity of speech and noise and the acoustic feature quantity of noise.

[0071] For example, the correlation estimator 142 estimates the correlation between the two feature quantities by inputting the acoustic feature quantity estimated by the acoustic feature quantity estimator 139 and the noise feature quantity estimated by the noise feature quantity estimator 141 to the correlation estimation NN stored in the correlation estimation NN storage 134. The correlation is provided to the feature quantity integration unit 143.

[0072] Here, as the correlation estimation NN, for example, the Attention NN shown in the following document may be used.

[0073] Reference: A. Vaswani et al., "Attention Is All You Need", in Proc. NIPS, 2017

[0074] The feature quantity integration unit 143 calculates an integrated feature quantity by weighting the acoustic feature quantity using the estimated correlation.

[0075] For example, the feature quantity synthesis unit 143 synthesizes the acoustic feature quantity from the acoustic feature quantity estimation unit 139 and the correlation from the correlation estimation unit 142. Synthesis means converting these two matrix expressions into one matrix expression. For example, when the acoustic feature quantity is a time-frequency expression of N×time and the correlation is a time-frequency expression of M×time, they may be connected through a frequency axis different from time to become a time-frequency expression of (N+M)×time, or the dimensions of the frequency axes of the acoustic feature quantity and the correlation may be unified through some dimensional transformation as the element product of the two matrices.

[0076] The masking estimation unit 144 estimates the target speech masking by inputting the comprehensive feature quantity into the speech emphasis NN which is a speech emphasis model learned in order to estimate the masking for emphasizing the speech.

[0077] For example, the masking estimation unit 144 takes the integrated feature quantity from the feature quantity integration unit 143 as input, and estimates the masking using the speech emphasis NN stored in the speech emphasis NN storage unit 135. The speech emphasis NN is a neural network composed of a plurality of layers, and for the propagation between the layers, for example, a method similar to LSTM or a method combining one-dimensional convolution operations may be used, and the number of layers is arbitrary.

[0078] Here, masking is a time-frequency representation of the same magnitude when the acoustic component is a time-frequency representation of N×time. The masking estimation unit 144 may estimate only the target speech masking that emphasizes the target speech from the mixed speech, for example, or may estimate the noise masking that emphasizes the noise from the mixed speech inversely. The masking that emphasizes the target speech estimated here, i.e., the target speech masking, is provided to the speech restoration unit 145.

[0079] The speech restoration unit 145 restores the speech in which the target speech is emphasized based on the acoustic component and the target speech masking.

[0080] For example, the speech restoration unit 145 applies the target speech mask from the masking estimation unit 144 to the acoustic component from the acoustic component calculation unit 138 , and further restores the speech signal using, for example, inverse short-time Fourier transform (iSTFT).

[0081] Part or all of the noise-mixed speech acquisition unit 136, the noise acquisition unit 137, the acoustic component calculation unit 138, the acoustic feature quantity estimation unit 139, the noise component calculation unit 140, the noise feature quantity estimation unit 141, the correlation estimation unit 142, the feature quantity synthesis unit 143, the masking estimation unit 144, and the speech restoration unit 145 described above can be, for example, Figure 3 As shown in (A), the computer system comprises a memory 10 and a processor 11 such as a CPU that executes a program stored in the memory 10. Such a program may be provided via a network or recorded in a recording medium. That is, such a program may be provided as a program product, for example.

[0082] In addition, part or all of the noise-mixed speech acquisition unit 136, the noise acquisition unit 137, the acoustic component calculation unit 138, the acoustic feature quantity estimation unit 139, the noise component calculation unit 140, the noise feature quantity estimation unit 141, the correlation estimation unit 142, the feature quantity synthesis unit 143, the masking estimation unit 144, and the speech restoration unit 145 may also be configured as follows. Figure 3 As shown in (B), the processing circuit 12 is composed of a single circuit, a composite circuit, a processor operated by a program, a parallel processor operated by a program, an ASIC, or an FPGA.

[0083] As described above, the noise-mixed speech acquisition unit 136, the noise acquisition unit 137, the sound component calculation unit 138, the sound feature quantity estimation unit 139, the noise component calculation unit 140, the noise feature quantity estimation unit 141, the correlation estimation unit 142, the feature quantity synthesis unit 143, the masking estimation unit 144 and the speech restoration unit 145 can be realized by a processing circuit network.

[0084] The feature estimation NN storage unit 132 , the noise estimation NN storage unit 133 , the correlation estimation NN storage unit 134 , and the speech emphasis NN storage unit 135 can be realized by a storage device (not shown) such as a HDD, an SSD, or a nonvolatile memory.

[0085] In addition, the communication unit 131 can be realized by a communication interface such as a NIC.

[0086] Figure 5 This is a flowchart showing the operation of the learning device 110 in the first embodiment.

[0087] First, the speech mixing unit 113 obtains learning speech data from the speech data storage unit 111 and learning noise data from the noise data storage unit 112, and superimposes the learning target speech indicated in the learning speech data and the noise from the noise data to generate a mixed speech (S10).

[0088] Next, the component calculation unit 114 calculates a target speech component and a mixed speech component, respectively, based on the target speech and the mixed speech from the speech mixing unit 113 ( S11 ).

[0089] Next, the training mask estimation unit 115 generates a training mask based on the target speech component and the mixed speech component from the component calculation unit 114 ( S12 ).

[0090] Next, the model learning unit 116 receives the mixed speech from the speech mixing unit 113, receives the learning noise data from the noise data storage unit 112, receives the training mask from the training mask estimation unit 115, and learns the NN, thereby generating a feature estimation NN, a noise estimation NN, a correlation estimation NN, and a speech emphasis NN (S13). The feature estimation NN, the noise estimation NN, the correlation estimation NN, and the speech emphasis NN generated by learning the NN are stored in the model storage unit 117 and sent to the speech emphasis device 130.

[0091] Figure 6 1 is a flowchart showing the operation of the speech emphasis device 130 in the first embodiment.

[0092] First, the acoustic component calculation unit 138 receives the inference mixed speech data from the noise mixed speech acquisition unit 136, and calculates the acoustic component based on the mixed speech indicated in the inference mixed speech data (S20).

[0093] Next, the acoustic feature quantity estimation unit 139 inputs the acoustic component received from the acoustic component calculation unit 138 to the feature estimation NN stored in the feature estimation NN storage unit 132, and estimates the acoustic feature quantity (S21).

[0094] Furthermore, the noise component calculation unit 140 receives the inference noise data from the noise acquisition unit 137 and calculates the noise component based on the inference noise data ( S22 ).

[0095] Next, the noise feature quantity estimation unit 141 inputs the noise component from the noise component calculation unit 140 to the noise estimation NN stored in the noise estimation NN storage unit 133 to estimate the noise feature quantity ( S23 ).

[0096] Next, the correlation estimation unit 142 inputs the acoustic feature quantity estimated by the acoustic feature quantity estimation unit 139 and the noise feature quantity estimated by the noise feature quantity estimation unit 141 to the correlation estimation NN stored in the correlation estimation NN storage unit 134, and estimates the correlation between the two feature quantities (S24).

[0097] Next, the feature quantity integration unit 143 integrates the acoustic feature quantity from the acoustic feature quantity estimation unit 139 and the correlation from the correlation estimation unit 142 (S25). Thus, an integrated feature quantity is generated.

[0098] The masking estimation unit 144 receives the integrated feature amount from the feature amount integration unit 143 as an input, and estimates the masking using the speech emphasis NN stored in the speech emphasis NN storage unit 135 ( S26 ).

[0099] Next, the speech restoration unit 145 applies the target speech mask from the masking estimation unit 144 to the acoustic component from the acoustic component calculation unit 138, and further restores the speech signal in which the target speech is emphasized, for example, using inverse short-time Fourier transform (iSTFT) (S27).

[0100] As described above, according to implementation mode 1, not only data of noise mixed speech containing target speech and noise is used, but also noise that does not contain target speech and is considered to be similar to the mixed noise is used to estimate the correlation of feature quantities extracted from them and input them into the learned model, thereby making it possible to implement speech emphasis even for unknown noise that is not included in the learning data.

[0101] In other words, according to the first embodiment, feature quantities are extracted from noise, and the correlation between the feature quantities extracted from noise and the feature quantities extracted from noise-superimposed speech is estimated by NN, so that speech emphasis can be performed robustly even for unknown noise.

[0102] Implementation method 2.

[0103] In the second embodiment, mixed speech and noise can be distinguished by detecting the speech section.

[0104] like Figure 1 As shown, the speech emphasis system 200 according to the second embodiment includes the learning device 110 and the speech emphasis device 230 .

[0105] The learning device 110 of the speech emphasis system 200 according to the second embodiment is the same as the learning device 110 of the speech emphasis system 100 according to the first embodiment.

[0106] Figure 7 This is a block diagram schematically showing the configuration of speech emphasis device 230 in Embodiment 2.

[0107] The speech emphasis device 230 includes a communication unit 131, a feature estimation NN storage unit 132, a noise estimation NN storage unit 133, a correlation estimation NN storage unit 134, a speech emphasis NN storage unit 135, a sound component calculation unit 138, a sound feature quantity estimation unit 139, a noise component calculation unit 140, a noise feature quantity estimation unit 141, a correlation estimation unit 142, a feature quantity integration unit 143, a masking estimation unit 144, a speech restoration unit 145 and a speech interval detection unit 246.

[0108] The communication unit 131, feature estimation NN storage unit 132, noise estimation NN storage unit 133, correlation estimation NN storage unit 134, speech emphasis NN storage unit 135, acoustic component calculation unit 138, acoustic feature quantity estimation unit 139, noise component calculation unit 140, noise feature quantity estimation unit 141, feature quantity integration unit 143, masking estimation unit 144, and speech restoration unit 145 of the speech emphasis device 230 in the second embodiment are the same as the communication unit 131, feature estimation NN storage unit 132, noise estimation NN storage unit 133, correlation estimation NN storage unit 134, speech emphasis NN storage unit 135, acoustic component calculation unit 138, acoustic feature quantity estimation unit 139, noise component calculation unit 140, noise feature quantity estimation unit 141, feature quantity integration unit 143, masking estimation unit 144, and speech restoration unit 145 of the speech emphasis device 130 in the first embodiment.

[0109] However, the sound component calculation unit 138 uses the speech interval data from the speech interval detection unit 246 as mixed speech data for inference, and calculates the sound component based on the mixed speech data for inference, and the noise component calculation unit 140 uses the non-speech interval data from the speech interval detection unit 246 as noise data for inference, and calculates the noise component based on the noise data for inference.

[0110] The speech section detection unit 246 generates mixed speech data from the data of the section including the target speech among the acoustic data having the section including the target speech and the section not including the target speech, and generates noise data from the data of the section not including the target speech among the acoustic data.

[0111] For example, the speech section detection unit 246 detects a speech section including speech and a non-speech section not including speech based on the sound shown in the sound data collected by the microphone (not shown) functioning as the sound collection unit. Then, the speech section detection unit 246 provides the speech section data as the data of the speech section in the sound data to the sound component calculation unit 138, and provides the non-speech section data as the data of the non-speech section in the sound data to the noise component calculation unit 140.

[0112] Here, for example, the detection of the speech interval may be performed by using a known technique such as the speech interval detection method disclosed in International Publication No. 2016 / 143125. In addition, the speech interval may be determined according to a threshold value based on the power of the sound signal input to the microphone.

[0113] The speech section detection unit 246 described above can also be used as follows, for example Figure 3 As shown in (A), the system includes a memory 10 and a processor 11 such as a CPU that executes a program stored in the memory 10 .

[0114] In addition, the speech section detection unit 246 can also be used as follows, for example Figure 3 As shown in (B), the processing circuit 12 is composed of a single circuit, a composite circuit, a processor operated by a program, a parallel processor operated by a program, an ASIC, or an FPGA.

[0115] As described above, the speech section detection unit 246 can be realized by a processing circuit network.

[0116] Figure 8 2 is a flowchart showing the operation of the speech emphasis device 230 in the second embodiment.

[0117] In addition, Figure 8 In the Figure 6 The steps of the same processing as the steps included in the flowchart shown are marked with Figure 6 Same label.

[0118] First, the speech interval detection unit 246 generates speech interval data and non-speech interval data based on the sound data recorded by the microphone, and provides the speech interval data to the sound component calculation unit 138 and the non-speech interval data to the noise component calculation unit 140 (S30). Then, the process proceeds to steps S20 and S22.

[0119] Figure 8 The processing of steps S20 to S27 in Figure 6 The processing of steps S20 to S27 in is the same as that of steps S20 to S27 in . However, the speech section data is processed as mixed speech data for inference, and the non-speech section data is processed as noise data for inference.

[0120] As described above, according to the second embodiment, voice section data and non-voice section data can be generated from sound data recorded by a microphone and processed as mixed voice data for inference and noise data for inference, respectively.

[0121] In other words, according to the second embodiment, by detecting the speech section, it is possible to detect the noise-only section and other sections from the noise-mixed speech, and speech emphasis can be performed robustly for unknown noise without manually providing noise.

[0122] Implementation method 3.

[0123] In the third embodiment, the mixed speech for inference can be divided and processed for each block, and the noise can be restored.

[0124] like Figure 1 As shown, the speech emphasis system 300 according to the third embodiment includes a learning device 310 and a speech emphasis device 330 .

[0125] Fig. 9 This is a block diagram schematically showing the configuration of a learning device 310 in the third embodiment.

[0126] The learning device 310 includes a speech data storage unit 111 , a noise data storage unit 112 , a speech mixing unit 113 , a component calculation unit 114 , a training mask estimation unit 315 , a model learning unit 316 , a model storage unit 117 , a communication unit 118 , and a block division unit 319 .

[0127] The speech data storage unit 111, noise data storage unit 112, speech mixing unit 113, component calculation unit 114, model learning unit 116, model storage unit 117 and communication unit 118 of the learning device 310 in Implementation Example 3 are the same as the speech data storage unit 111, noise data storage unit 112, speech mixing unit 113, component calculation unit 114, model learning unit 116, model storage unit 117 and communication unit 118 of the learning device 110 in Implementation Example 1.

[0128] However, the speech mixing section 113 supplies the target speech and the mixed speech to the block dividing section 319 .

[0129] Furthermore, the component calculation unit 114 calculates the acoustic component for each block supplied from the block division unit 319 .

[0130] The block division unit 319 divides the target speech and the mixed speech from the speech mixing unit 113 into a plurality of blocks of a fixed time length, and supplies the divided blocks to the component calculation unit 114 and the model learning unit 116 .

[0131] The training mask estimation unit 315 performs the same processing as the training mask estimation unit 115 of the first embodiment, and estimates a noise mask for emphasizing noise as a training mask based on the blocks from the block division unit 319 , and supplies the estimated noise mask to the model learning unit 316 .

[0132] The model learning unit 316 receives the blocks from the block division unit 219, receives the learning noise data from the noise data storage unit 112, receives the training mask from the training mask estimation unit 315, and learns the speech emphasis NN. In the third embodiment, the model learning unit 316 restores the noise from the blocks by the noise mask from the training mask estimation unit 315 and learns the speech emphasis NN.

[0133] In addition, the model learning unit 316 receives the blocks from the block division unit 319, receives the learning noise data from the noise data storage unit 112, receives the training mask from the training mask estimation unit 315, and relearns the speech emphasis NN. Here, the model learning unit 316 also restores the noise from the blocks using the noise mask from the training mask estimation unit 315 and relearns the speech emphasis NN.

[0134] The block division unit 319 described above can also be used as follows, for example Figure 3 As shown in (A), the system includes a memory 10 and a processor 11 such as a CPU that executes a program stored in the memory 10 .

[0135] In addition, the block division unit 319 can also be as follows, for example Figure 3 As shown in (B), the processing circuit 12 is composed of a single circuit, a composite circuit, a processor operated by a program, a parallel processor operated by a program, an ASIC, or an FPGA.

[0136] As described above, the block division unit 319 can be realized by a processing circuit network.

[0137] Fig.10 This is a block diagram schematically showing the configuration of the speech emphasis device 330 in the third embodiment.

[0138] The speech emphasis device 330 includes a communication unit 131, a feature estimation NN storage unit 132, a noise estimation NN storage unit 133, a correlation estimation NN storage unit 134, a speech emphasis NN storage unit 135, a sound component calculation unit 138, a sound feature quantity estimation unit 139, a noise component calculation unit 140, a noise feature quantity estimation unit 341, a correlation estimation unit 342, a feature quantity integration unit 143, a masking estimation unit 144, a speech restoration unit 145, a block division unit 347 and a noise restoration unit 348.

[0139] The communication unit 131, feature estimation NN storage unit 132, noise estimation NN storage unit 133, correlation estimation NN storage unit 134, speech emphasis NN storage unit 135, acoustic component calculation unit 138, acoustic feature quantity estimation unit 139, noise component calculation unit 140, feature quantity integration unit 143, masking estimation unit 144 and speech restoration unit 145 of the speech emphasis device 330 in embodiment 3 are the same as the communication unit 131, feature estimation NN storage unit 132, noise estimation NN storage unit 133, correlation estimation NN storage unit 134, speech emphasis NN storage unit 135, acoustic component calculation unit 138, acoustic feature quantity estimation unit 139, noise component calculation unit 140, feature quantity integration unit 143, masking estimation unit 144 and speech restoration unit 145 of the speech emphasis device 130 in embodiment 1.

[0140] However, the acoustic component calculation unit 138 calculates the acoustic component based on the inference mixed speech data divided for each block by the block division unit 347 .

[0141] In addition, the masking estimation unit 144 also provides the estimated masking to the noise restoration unit 348. Here, it is sufficient to provide a masking for emphasizing noise, that is, a noise masking, but when the masking estimation unit 144 does not estimate the noise masking, the masking estimation unit 144 generates a masking for emphasizing noise based on the target speech masking, and provides the generated noise masking to the noise restoration unit 348. In addition, for example, when the training masking is represented by the ratio of the power spectrum of the target speech to the power spectrum of the mixed speech, the noise masking can be obtained by subtracting each element of the masking for emphasizing the target speech from the mixed speech from 1.

[0142] The block division unit 347 divides the mixed speech data into a plurality of blocks.

[0143] For example, the block division unit 347 divides the inference mixed speech data from the noise mixed speech acquisition unit 136 into blocks of a fixed time length and supplies the blocks to the acoustic component calculation unit 138. The acoustic component calculation unit 138 in the third embodiment calculates the acoustic component for each of the plurality of blocks.

[0144] For example, it is desirable to segment the blocks in a manner including superposition as shown in the above-mentioned document “Deep clustering-based single-channel speech separation and recent advances”.

[0145] The noise restoration unit 348 calculates a restored noise component by emphasizing the noise based on the acoustic component and the noise masking.

[0146] For example, the noise restoration unit 348 applies the noise mask output by the mask estimation unit 144 to the acoustic component from the acoustic component calculation unit 138 , and calculates the restored noise component. The restored noise component is supplied to the noise feature amount estimation unit 341 .

[0147] The noise feature estimation unit 341 inputs the restored noise component output by the noise restoration unit 348 to the noise estimation NN to estimate the restored noise feature based on the processing of the noise feature estimation unit 141 in Embodiment 1. The restored noise feature is combined with the already estimated noise feature in the time direction and output as a combined noise feature to the correlation estimation unit 342.

[0148] When the combined noise feature amount is generated, the correlation estimation unit 342 estimates the correlation from the acoustic feature amount and the combined noise feature amount.

[0149] For example, the correlation estimating unit 342 estimates the correlation between the two feature quantities by inputting the acoustic feature quantity estimated by the acoustic feature quantity estimating unit 139 and the combined noise feature quantity estimated by the noise feature quantity estimating unit 341 to the correlation estimation NN stored in the correlation estimation NN storage unit 134, based on the processing of the correlation estimating unit 142 in Embodiment 1. The correlation is provided to the feature quantity integrating unit 143.

[0150] In implementation mode 3, the noise feature estimation unit 341 combines the restored noise feature with the noise feature estimated in the next block of the block in which the restored noise component is calculated in the time direction, thereby generating a combined noise feature. Therefore, the correlation estimation unit 342 also estimates the correlation based on the acoustic feature and the combined noise feature in the next block of the block in which the restored noise component is calculated.

[0151] Part or all of the block division unit 347 and the noise restoration unit 348 described above can be, for example, Figure 3 As shown in (A), the system is composed of a memory 10 and a processor 11 such as a CPU that executes a program stored in the memory 10.

[0152] In addition, part or all of the block division unit 347 and the noise restoration unit 348 may also be configured as follows, for example. Figure 3 As shown in (B), the processing circuit 12 is composed of a single circuit, a composite circuit, a processor operated by a program, a parallel processor operated by a program, an ASIC, or an FPGA.

[0153] As described above, part or all of the block division unit 347 and the noise restoration unit 348 can be realized by a processing circuit network.

[0154] Fig.11This is a flowchart showing the operation of the learning device 110 in the first embodiment.

[0155] First, the speech mixing unit 113 obtains the learning speech data from the speech data storage unit 111 and the learning noise data from the noise data storage unit 112, and generates a mixed speech by superimposing the learning target speech indicated in the learning speech data and the noise indicated in the learning noise data (S40).

[0156] Next, the block division unit 319 divides the target speech and the mixed speech from the speech mixing unit 113 into blocks ( S41 ).

[0157] Next, the component calculation unit 114 calculates the target speech component and the mixed speech component, respectively, based on the blocks of the target speech and the mixed speech from the block division unit 319 ( S42 ).

[0158] Next, the training mask estimation section 115 generates a target speech mask for emphasizing the target speech and a noise mask for emphasizing noise from the target speech component and the mixed speech component from the component calculation section 114 as training masks (S43).

[0159] Next, the model learning unit 316 receives the mixed speech from the speech mixing unit 113, receives the learning noise data from the noise data storage unit 112, receives the training mask from the training mask estimation unit 115, and learns the NN, thereby generating a feature estimation NN, a noise estimation NN, a correlation estimation NN, and a speech emphasis NN (S44). The feature estimation NN, the noise estimation NN, the correlation estimation NN, and the speech emphasis NN generated by learning the NN are stored in the model storage unit 117.

[0160] In addition, the model learning unit 316 receives the mixed speech from the speech mixing unit 113, receives the learning noise data from the noise data storage unit 112, receives the training mask from the training mask estimation unit 115, and relearns the speech emphasis NN, thereby generating a feature estimation NN, a noise estimation NN, a correlation estimation NN, and a speech emphasis NN (S45). The feature estimation NN, the noise estimation NN, the correlation estimation NN, and the speech emphasis NN generated by relearning the speech emphasis NN are stored in the model storage unit 117 and sent to the speech emphasis device 130.

[0161] Fig.12 : is a flowchart showing the operation of the speech emphasis device 330 in the third embodiment.

[0162] First, the block division unit 347 receives the mixed speech data for inference from the noise mixed speech acquisition unit 136, and divides the mixed speech data for inference into blocks of a fixed time length (S50). Then, the block division unit 347 provides the divided blocks to the acoustic component calculation unit 138 one by one, starting from the oldest block.

[0163] Next, the acoustic component calculation unit 138 receives the block from the block division unit 347, and calculates the acoustic component based on the mixed speech represented by the block (S51).

[0164] Next, the acoustic feature quantity estimation unit 139 inputs the acoustic component received from the acoustic component calculation unit 138 to the feature estimation NN stored in the feature estimation NN storage unit 132 to estimate the acoustic feature quantity ( S52 ).

[0165] Furthermore, the noise component calculation unit 140 receives the inference noise data from the noise acquisition unit 137 and calculates the noise component based on the inference noise data ( S53 ).

[0166] Next, the noise feature amount estimation unit 341 inputs the noise component from the noise component calculation unit 140 to the noise estimation NN stored in the noise estimation NN storage unit 133 to estimate the noise feature amount ( S54 ).

[0167] Next, the correlation estimation unit 342 inputs the acoustic feature quantity estimated by the acoustic feature quantity estimation unit 139 and the noise feature quantity estimated by the noise feature quantity estimation unit 141 to the correlation estimation NN stored in the correlation estimation NN storage unit 134 to estimate the correlation between the two feature quantities (S55).

[0168] Next, the feature quantity integration unit 143 integrates the acoustic feature quantity from the acoustic feature quantity estimation unit 139 and the correlation from the correlation estimation unit 342 ( S56 ).

[0169] The masking estimation unit 144 receives the integrated feature amount from the feature amount integration unit 143 as an input, and estimates the masking using the speech emphasis NN stored in the speech emphasis NN storage unit 135 ( S57 ).

[0170] Next, the speech restoration unit 145 applies the target speech mask from the masking estimation unit 144 to the acoustic component from the acoustic component calculation unit 138, and further restores the speech signal in which the target speech is emphasized, for example, using inverse short-time Fourier transform (iSTFT) (S58).

[0171] Furthermore, the noise restoration unit 348 applies the noise mask output by the masking estimation unit 144 to the acoustic component from the acoustic component calculation unit 138 , and calculates a restored noise component.

[0172] Then, the block division unit 347 determines whether there are any blocks that have not been supplied to the acoustic component calculation unit 138 (S60). If such blocks remain (Yes in S60), the process returns to step S51 and step S54, and if such blocks do not remain (No in S60), the process ends.

[0173] When returning from step S60 to step S54 , the noise feature amount estimation unit 341 inputs the restored noise component restored by the noise restoration unit 348 to the noise estimation NN stored in the noise estimation NN storage unit 133 , and estimates the restored noise feature amount.

[0174] Then, in step S55, the correlation estimation unit 342 inputs the acoustic feature estimated by the acoustic feature estimation unit 139 and the restored noise feature estimated by the noise feature estimation unit 141 into the correlation estimation NN stored in the correlation estimation NN storage unit 134 to estimate the correlation between the two feature quantities.

[0175] As described above, according to Embodiment 3, by performing speech emphasis in block processing, the processing delay of speech emphasis can be reduced, and by using the noise estimated from the previous block of the block currently performing speech emphasis, the robustness against unknown noise can be improved. In addition, during learning, after learning using the noise extracted from the previous block using the true mask estimated by the training mask estimation unit, learning using the noise extracted from the previous block using the estimated mask is performed in the same manner as during inference, thereby enabling efficient learning of the model.

[0176] Implementation method 4.

[0177] In the fourth embodiment, the likelihood of the restored noise component is calculated, and it is determined whether to use the restored noise component.

[0178] like Figure 1 As shown, the speech emphasis system 400 according to the fourth embodiment includes the learning device 310 and a speech emphasis device 430 .

[0179] The learning device 310 of the speech emphasis system 400 according to the fourth embodiment is the same as the learning device 310 of the speech emphasis system 300 according to the third embodiment.

[0180] Fig.13 This is a block diagram schematically showing the configuration of a speech emphasis device 430 in the fourth embodiment.

[0181] The speech emphasis device 430 includes a communication unit 131, a feature estimation NN storage unit 132, a noise estimation NN storage unit 133, a correlation estimation NN storage unit 134, a speech emphasis NN storage unit 135, a sound component calculation unit 138, a sound feature quantity estimation unit 139, a noise component calculation unit 140, a noise feature quantity estimation unit 441, a correlation estimation unit 142, a feature quantity integration unit 143, a masking estimation unit 144, a speech restoration unit 145, a block division unit 347, a noise restoration unit 348 and a noise likelihood determination unit 449.

[0182] The communication unit 131, feature estimation NN storage unit 132, noise estimation NN storage unit 133, correlation estimation NN storage unit 134, speech emphasis NN storage unit 135, acoustic component calculation unit 138, acoustic feature quantity estimation unit 139, noise component calculation unit 140, feature quantity integration unit 143, masking estimation unit 144 and speech restoration unit 145 of the speech emphasis device 430 in the fourth embodiment are the same as the communication unit 131, feature estimation NN storage unit 132, noise estimation NN storage unit 133, correlation estimation NN storage unit 134, speech emphasis NN storage unit 135, acoustic component calculation unit 138, acoustic feature quantity estimation unit 139, noise component calculation unit 140, feature quantity integration unit 143, masking estimation unit 144 and speech restoration unit 145 of the speech emphasis device 130 in the first embodiment.

[0183] However, the acoustic component calculation unit 138 calculates the acoustic component based on the inference mixed speech data divided for each block by the block division unit 347 .

[0184] In addition, the masking estimation unit 144 also provides the estimated masking to the noise restoration unit 348. Here, it is sufficient to provide the noise masking, but when the masking estimation unit 144 does not estimate the noise masking, the masking estimation unit 144 generates a masking that emphasizes the noise based on the target speech masking, and provides the generated noise masking to the noise restoration unit 348. In addition, for example, when the training masking is expressed by the ratio of the power spectrum of the target speech and the power spectrum of the mixed speech, the noise masking can be obtained by subtracting each element of the masking that emphasizes the target speech from the mixed speech from 1.

[0185] The block division unit 347 and the noise restoration unit 348 of the speech sound emphasis device 430 in the fourth embodiment are the same as the block division unit 347 and the noise restoration unit 348 of the speech sound emphasis device 330 in the third embodiment.

[0186] However, the noise restoration unit 348 in the fourth embodiment supplies the restored noise component to the noise likelihood determination unit 449 .

[0187] The noise likelihood determination unit 449 calculates the noise likelihood as the likelihood of restoring the noise component, and determines whether the noise likelihood is equal to or greater than a predetermined threshold value.

[0188] For example, the noise likelihood determination unit 449 receives the restored noise component from the noise restoration unit 348 and calculates the noise likelihood. Then, the noise likelihood determination unit 449 supplies the restored noise component corresponding to the block having a noise likelihood greater than or equal to the threshold value to the noise feature amount estimation unit 441 .

[0189] Here, when restoring the time frequency representation of the noise component as N×time, the noise likelihood is calculated for each block corresponding to the time frame. The noise likelihood can be calculated using NN as described in Non-Patent Document 1. The threshold value can be obtained experimentally, for example.

[0190] The noise feature quantity estimation unit 441 inputs the restored noise component output by the noise likelihood judgment unit 449 into the noise estimation NN in the next block of the block corresponding to the restored noise component received from the noise likelihood judgment unit 449, estimates the restored noise feature quantity, combines the restored noise feature quantity with the noise feature quantity estimated based on the noise component received from the noise component calculation unit 140 in the time direction, and thereby calculates the combined noise feature quantity.

[0191] On the other hand, the noise feature quantity estimation section 441 estimates the noise feature quantity based on the noise component received from the noise component calculation section 140 in the next block of the block from which the restored noise component is not received from the noise likelihood determination section 449 .

[0192] Then, the noise feature estimation unit 441 provides the combined noise feature to the correlation estimation unit 342 in the next block of the block corresponding to the restored noise component received from the noise likelihood judgment unit 449, and provides the noise feature to the correlation estimation unit 342 in other blocks.

[0193] In other words, the noise feature estimation unit 441 generates a combined noise feature when the noise likelihood is equal to or greater than the threshold.

[0194] The correlation estimation unit 442 inputs the sound feature quantity estimated by the sound feature quantity estimation unit 139 and the noise feature quantity estimated by the noise feature quantity estimation unit 341 into the correlation estimation NN stored in the correlation estimation NN storage unit 134 for the block of noise feature quantities collected from the noise feature quantity estimation unit 441, and estimates the correlation between the two feature quantities.

[0195] On the other hand, the correlation estimation unit 442 inputs the acoustic feature quantity estimated by the acoustic feature quantity estimation unit 139 and the combined noise feature quantity estimated by the noise feature quantity estimation unit 341 to the correlation estimation NN stored in the correlation estimation NN storage unit 134, and estimates the correlation between the two feature quantities. The correlation is provided to the feature quantity integration unit 143.

[0196] The noise likelihood determination unit 449 described above can also be used as follows, for example Figure 3 As shown in (A), the system includes a memory 10 and a processor 11 such as a CPU that executes a program stored in the memory 10 .

[0197] In addition, the noise likelihood determination unit 449 can also be as follows, for example Figure 3 As shown in (B), the processing circuit 12 is composed of a single circuit, a composite circuit, a processor operated by a program, a parallel processor operated by a program, an ASIC, or an FPGA.

[0198] As described above, the noise likelihood determination unit 449 can be realized by a processing circuit network.

[0199] Fig.14 1 is a flowchart showing the operation of the speech emphasis device 430 in the fourth embodiment.

[0200] In addition, Fig.14 In the Fig.12 The steps of the same processing as the steps included in the flowchart shown are marked with Fig.12 Same label.

[0201] Fig.14 The processing in steps S50 to S53 is the same as Fig.12 The processing in steps S50 to S53 is the same as that in Fig.14 In the process, after the processing of step S53, the process proceeds to step S70.

[0202] In step S70, the noise feature quantity estimation section 441 estimates the noise feature quantity based on the noise component received from the noise component calculation section 140. Then, the process proceeds to step S55.

[0203] Fig.14 The processing in steps S55 to S59 is the same as Fig.12 The processing in steps S55 to S59 is the same as that in Fig.14 In the process, after the processing of step S59, the processing enters step S71.

[0204] In step S71, the noise likelihood determination unit 449 receives the restored noise component from the noise restoration unit 348 and calculates the noise likelihood. Then, when the calculated noise likelihood is greater than or equal to the threshold, the noise likelihood determination unit 449 provides the restored noise component to the noise feature estimation unit 441. Then, the process proceeds to step S60.

[0205] In step S60, the block division unit 347 determines whether there are any remaining blocks that have not been supplied to the acoustic component calculation unit 138. If such blocks remain (Yes in S60), the process returns to step S51 and step S70, and if such blocks do not remain (No in S60), the process ends.

[0206] In addition, when returning from step S60 to step S70, when the restored noise component is received from the noise likelihood judgment unit 449, the noise feature quantity estimation unit 441 inputs the restored noise component restored by the noise restoration unit 348 into the noise estimation NN stored in the noise estimation NN storage unit 133, estimates the restored noise feature quantity, and combines the restored noise feature quantity with the noise feature quantity estimated based on the noise component received from the noise component calculation unit 140 in the time direction, thereby calculating the combined noise feature quantity.

[0207] In this case, in step S55, the correlation estimation unit 342 inputs the acoustic feature quantity estimated by the acoustic feature quantity estimation unit 139 and the restored noise feature quantity estimated by the noise feature quantity estimation unit 141 into the correlation estimation NN stored in the correlation estimation NN storage unit 134 to estimate the correlation between the two feature quantities.

[0208] As described above, according to the fourth embodiment, only the portion with high noise likelihood among the noise estimated from the previous block is used, thereby preventing the learning of noise including estimation errors.

[0209] Description of Reference Numerals

[0210] 100, 200, 300, 400 speech emphasis system, 110, 310 learning device, 111 speech data storage unit, 112 noise data storage unit, 113 speech mixing unit, 114 component calculation unit, 115, 315 training masking estimation unit, 116, 316 model learning unit, 117 model storage unit, 118 communication unit, 319 block division unit, 130, 230, 330, 430 speech emphasis device, 131 communication unit, 132 feature estimation NN storage unit, 133 noise estimation NN storage unit, 134 phase Correlation estimation NN storage unit, 135 speech emphasis NN storage unit, 136 noise mixed speech acquisition unit, 137 noise acquisition unit, 138 sound component calculation unit, 139 sound feature quantity estimation unit, 140 noise component calculation unit, 141, 341, 441 noise feature quantity estimation unit, 142, 342, 442 correlation estimation unit, 143 feature quantity integration unit, 144 masking estimation unit, 145 speech restoration unit, 246 speech interval detection unit, 347 block segmentation unit, 348 noise restoration unit, 449 noise likelihood judgment unit.

Claims

1. An information processing device, characterized in that: The information processing device comprises: an acoustic component calculation unit that calculates an acoustic component from mixed speech data using a predetermined function, the mixed speech data including a target speech to be emphasized and noise mixed with the target speech, that is, mixed noise, the acoustic component being a component of the target speech and the mixed noise; an acoustic feature estimation unit that estimates an acoustic feature by inputting the acoustic component into a feature estimation model learned for estimating acoustic feature of speech and noise; a noise component calculation unit that calculates a noise component, that is, a noise component, from noise data that does not include the target speech but includes noise, using the predetermined function; a noise feature quantity estimation unit that estimates a noise feature quantity by inputting the noise component into a noise estimation model learned for estimating an acoustic feature quantity of noise; a correlation estimation unit that estimates the correlation between the acoustic feature quantity and the noise feature quantity by inputting the acoustic feature quantity and the noise feature quantity into a correlation estimation model learned for estimating the correlation between the acoustic feature quantity of speech and noise and the acoustic feature quantity of noise; a feature quantity synthesis unit that weights the acoustic feature quantity using the estimated correlation to calculate a comprehensive feature quantity; a masking estimation unit that estimates a target speech masking by inputting the comprehensive feature amount into a speech emphasis model learned in order to estimate a masking for emphasizing speech; as well as A speech restoration unit restores the speech in which the target speech is emphasized based on the acoustic component and the target speech mask.

2. The information processing device according to claim 1, characterized in that The information processing device further comprises: The interface part accepts the input of data; a noise-mixed speech acquisition unit that acquires the mixed speech data via the interface unit; and A noise acquisition unit acquires the noise data via the interface unit.

3. The information processing device according to claim 1, characterized in that The information processing device further comprises: An interface unit that receives input of data; and A speech section detection unit, which obtains, via the interface unit, sound data having a section including the target speech and a section not including the target speech, generates the mixed speech data from the sound data based on the data of the section including the target speech, and generates the noise data from the sound data based on the data of the section not including the target speech.

4. The information processing device according to any one of claims 1 to 3, characterized in that: The information processing device further includes a block dividing unit that divides the mixed speech data into a plurality of blocks. The acoustic component calculation unit calculates the acoustic component for each of the plurality of blocks.

5. The information processing device according to claim 4, characterized in that The masking estimation unit further estimates a noise mask for emphasizing noise, The information processing device further includes a noise restoration unit configured to calculate a restored noise component by emphasizing the noise based on the acoustic component and the noise masking. The noise feature estimation unit estimates a restored noise feature by inputting the restored noise component into the noise estimation model, and generates a combined noise feature by combining the restored noise feature with the noise feature in a time direction. When the combined noise feature amount is generated, the correlation estimation unit estimates the correlation based on the acoustic feature amount and the combined noise feature amount.

6. The information processing device according to claim 5, characterized in that: The noise feature amount estimation section generates a combined noise feature amount by combining the restored noise feature amount in a time direction with the noise feature amount estimated in a block next to the block in which the restored noise component is calculated.

7. The information processing device according to claim 5, characterized in that The information processing device further includes a noise likelihood determination unit that calculates the likelihood of the restored noise component, that is, the noise likelihood, and determines whether the noise likelihood is equal to or greater than a predetermined threshold value. The noise feature estimation unit generates the combined noise feature when the noise likelihood is equal to or greater than the threshold.

8. A program, characterized in that The program causes the computer to function as the following: an acoustic component calculation unit that calculates an acoustic component from mixed speech data using a predetermined function, the mixed speech data including a target speech to be emphasized and noise mixed with the target speech, that is, mixed noise, the acoustic component being a component of the target speech and the mixed noise; an acoustic feature estimation unit that estimates an acoustic feature by inputting the acoustic component into a feature estimation model learned for estimating acoustic feature of speech and noise; a noise component calculation unit that calculates a noise component, that is, a noise component, from noise data that does not include the target speech but includes noise, using the predetermined function; a noise feature quantity estimation unit that estimates a noise feature quantity by inputting the noise component into a noise estimation model learned for estimating an acoustic feature quantity of noise; a correlation estimation unit that estimates the correlation between the acoustic feature quantity and the noise feature quantity by inputting the acoustic feature quantity and the noise feature quantity into a correlation estimation model learned for estimating the correlation between the acoustic feature quantity of speech and noise and the acoustic feature quantity of noise; a feature quantity synthesis unit that weights the acoustic feature quantity using the estimated correlation to calculate a comprehensive feature quantity; a masking estimation unit that estimates a target speech masking by inputting the comprehensive feature amount into a speech emphasis model learned in order to estimate a masking for emphasizing speech; as well as A speech restoration unit restores the speech in which the target speech is emphasized based on the acoustic component and the target speech mask.

9. An information processing method, characterized in that: Using a predetermined function, an acoustic component is calculated from mixed speech data, the mixed speech data including a target speech to be emphasized and noise mixed with the target speech, that is, mixed noise, the acoustic component being a component of the target speech and the mixed noise, The acoustic feature quantity is estimated by inputting the acoustic component into a feature estimation model learned for estimating the acoustic feature quantity of speech and noise, Using the predetermined function, a noise component is calculated based on noise data that does not include the target speech but includes noise, The noise component is input into a noise estimation model learned for estimating an acoustic feature quantity of noise, thereby estimating a noise feature quantity. The correlation between the acoustic feature quantity and the noise feature quantity is estimated by inputting the acoustic feature quantity and the noise feature quantity into a correlation estimation model learned for estimating the correlation between the acoustic feature quantity of speech and noise and the acoustic feature quantity of noise, The sound feature quantity is weighted by using the estimated correlation to calculate a comprehensive feature quantity, By inputting the comprehensive feature quantity into a speech emphasis model learned for estimating a mask for emphasizing speech, a target speech mask is estimated. The speech in which the target speech is emphasized is restored based on the acoustic component and the target speech mask.

Images