Sound enhancement device, method, and program

Abstract

A sound enhancement device comprises a sound enhancement unit 1 that inputs an observation signal, which is an observed sound signal, and auxiliary information, which is information indicating a target of sound enhancement, into a deep learning model in which a state space model is incorporated, thereby generating an enhancement signal that is a signal obtained by enhancing a signal to be subjected to sound enhancement as determined by the auxiliary information in the observation signal.

Description

Sound enhancement device, method and program

[0001] The disclosed technology relates to a technology for enhancing sound.

[0002] Thanks to the development of deep learning technology, the performance of speech enhancement has improved dramatically in recent years.

[0003] The technology for selectively extracting only signals that satisfy specific conditions from observed signals that include signals emitted by various speakers and sound sources is called target voice enhancement technology. In particular, the technology for extracting the voice of a specific speaker is called target speaker voice enhancement technology.

[0004] Prior art technologies for enhancing target sounds use deep learning networks such as LSTM, CNN, and self-attention, which are currently the de facto standard.

[0005] For example, in Non-Patent Document 1, a target speaker speech enhancement technology is realized using a network structure that combines CNN and LSTM.

[0006] In Non-Patent Document 2, a target speaker speech enhancement technique is realized using a network structure that is an extension of a network structure called DCCRN, which has a proven track record in speech enhancement, to target speaker speech enhancement.

[0007] Wang, Q., Moreno, IL, Saglam, M., Wilson, K., Chiao, A., Liu, R., He, Y., Li, W., Pelecanos, J., Nika, M., Gruenstein, A. (2020) "VoiceFilter-Lite: Streaming Targeted Voice Separation for On-Device Speech Recognition", Proc. Interspeech 2020, 2677-2681, doi: 10.21437 / Interspeech.2020-1193Eskimez, Sefik Emre, et al. "Personalized speech enhancement: New models and comprehensive evaluation", ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2022.

[0008] To enhance the target sound with high performance, it is necessary to effectively model both the local structure and the relatively long-term structure of the features.

[0009] However, the DNN architectures proposed so far for speech enhancement of target sounds have not necessarily been optimal in this respect.

[0010] The disclosed technology aims to provide a sound enhancement device, method, and program that can perform sound enhancement based on a model that takes into account relatively long-term dependencies.

[0011] A sound enhancement device, which is one aspect of the disclosed technology, includes a sound enhancement unit that generates an enhancement signal, which is a signal that enhances the signal that is the target of sound enhancement, determined by the auxiliary information in the observation signal, by inputting an observation signal, which is an observed sound signal, and auxiliary information, which is information indicating the target of sound enhancement, into a deep learning model that incorporates a state space model.

[0012] According to the disclosed technology, by incorporating a state space model into a deep learning model, sound enhancement can be performed based on a model that takes into account relatively long-term dependencies.

[0013] Fig. 1 is a diagram showing an example of the functional configuration of a sound enhancement device. Fig. 2 is a diagram showing an example of the processing procedure of a sound enhancement method. Fig. 3 is a diagram for explaining an example of processing by a sound enhancement unit. Fig. 4 is a diagram showing an example of the functional configuration of a sound enhancement model training device. Fig. 5 is a diagram showing an example of the processing procedure of a sound enhancement model training method. Fig. 6 is a diagram showing an example of the functional configuration of a computer. Fig. 7 is a diagram for explaining an experimental example.

[0014] Hereinafter, embodiments of the disclosed technology will be described with reference to the drawings. Note that components having the same functions in the drawings are given the same reference numerals, and redundant description will be omitted.

[0015] [Sound Enhancement Device and Method] As shown in Fig. 1, the sound enhancement device includes a sound enhancement unit 1. The sound enhancement device may further include an output unit 2 indicated by a dashed line in Fig. 1.

[0016] The sound enhancement method is realized, for example, by each component of the sound enhancement device performing the process of step S1 shown in FIG.

[0017] Each component of the sound enhancement device will be described below.

[0018] <Sound Emphasis Unit 1> The sound emphasis unit 1 receives an observation signal and auxiliary information.

[0019] The observed signal is an observed sound signal. The sound signal is a signal related to sound, such as an acoustic signal or a speech signal. The observed signal may be a single-channel signal or a multi-channel signal. The observed signal may also be a time-domain signal or a frequency-domain signal. The observed signal may also be a feature obtained from the time-domain signal or the frequency-domain signal.

[0020] If the observed signal is a one-channel signal, the enhancement signal output from the sound enhancement unit 1 is also a one-channel signal. If the observed signal is a multi-channel signal, the enhancement signal output from the sound enhancement unit 1 is also a multi-channel signal.

[0021] The auxiliary information is information indicating the target of sound enhancement. For example, the auxiliary information is a vector representing the class of the target signal. When the target of sound enhancement is a target speaker, the auxiliary information may be prior information such as a pre-registered utterance representing the target speaker. The auxiliary information may be information regarding the position of the target sound source, or a video of the face of the target speaker speaking. The information regarding the position of the target sound source is information regarding at least one of the distance and direction of the target sound source. The distance of the target sound source is the distance from a predetermined position of the target sound source. The direction of the target sound source is the direction of the target sound source relative to the predetermined position.

[0022] The sound enhancement unit 1 inputs an observed signal, which is an observed sound signal, and auxiliary information, which is information indicating a target of sound enhancement, into a deep learning model incorporating a state space model, to generate an enhancement signal, which is a signal obtained by enhancing the signal that is the target of sound enhancement determined by the auxiliary information in the observed signal (step S1). The generated enhancement signal is sent to the output unit 2.

[0023] The deep learning model incorporates a state space model. For example, the deep learning model has a structure in which part of the Conv-TasNet structure is replaced with a state space model. The deep learning model is trained using a sound enhancement model generation device and method described below.

[0024] For the Conv-TasNet structure, see e.g., References 1 and 2.

[0025] [Reference 1] Luo, Yi, and Nima Mesgarani. "Conv-tasnet: Surpassing ideal time-frequency magnitude masking for speech separation", IEEE / ACM transactions on audio, speech, and language processing 27.8 (2019): 1256-1266. [Reference 2] Delcroix, Marc, et al. "Improving speaker discrimination of target speech extraction with time-domain speakerbeam", ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2020. A state-space model is a model that can represent the time changes of a system. Existing architectures such as S4 (Structured State Spaces for Sequence Modeling), S4D (Structured State Spaces for Sequence Modeling with Simplicity), S4ND (Structured State Spaces for Sequence Modeling with No Dropout), and DSS (Descriptor State Space) can be used for the state space model.

[0026] For state space models, see, for example, references 3 and 4.

[0027] [Reference 3] Gu, Albert, et al. "On the parameterization and initialization of diagonal state space models", Advances in Neural Information Processing Systems 35 (2022): 35971-35983. [Reference 4] Gu, Albert, Karan Goel, and Christopher Re. "Efficiently modeling long sequences with structured state spaces", arXiv preprint arXiv:2111.00396 (2021). In this way, by incorporating a state space model into a deep learning model, it is possible to perform sound enhancement based on a model that takes into account not only local structure but also relatively long-term dependencies. It also enables lightweight enhancement of the target sound. This makes it possible to perform sound enhancement in real time, for example.

[0028] As shown in FIG. 3 , the deep learning model includes, for example, an expression extraction unit 11, an encoder 12, a state space module 13, a convolutional neural network 14, an integration unit 15, a state space module 16, a convolutional neural network 17, and a decoder 18.

[0029] <<Expression Extraction Unit 11>> The expression extraction unit 11 receives auxiliary information as input.

[0030] The expression extraction unit 11 receives auxiliary information indicating the target signal as input and acquires a latent expression related to the target signal. The expression extraction unit 11 can be constructed using, for example, a neural network. This neural network can use a known technique such as d-vector.

[0031] The acquired latent representation is sent to the synthesis unit 15.

[0032] <<Encoder 12>> The encoder receives an observation signal.

[0033] The encoder 12 extracts features from the input observation signal. The encoder 12 can be constructed using, for example, a neural network. If the input observation signal is a time-domain signal, the encoder 12 can use signal processing such as a short-time Fourier transform.

[0034] The extracted features are sent to the state space module 13 .

[0035] <<State Space Module 13 >> The features extracted by the encoder 12 are input to the state space module 13 .

[0036] The state space module 13 is, for example, a neural network configured from a multi-layer state space model, such as S4, S4D, S4ND, DSS, or H3.

[0037] The output of the state space module 13 is sent to a convolutional neural network 14 .

[0038] <<Convolutional Neural Network 14>> The output of the state space module 13 is input to the convolutional neural network 14.

[0039] The convolutional neural network 14 is, for example, a multi-layer convolutional neural network, and may be configured as an existing one-dimensional convolutional neural network.

[0040] The output of the convolutional neural network 14 is sent to the integrator 15 .

[0041] <<Integration Unit 15 >> The integration unit 15 receives the output of the state space module 13 and the latent expressions acquired by the expression extraction unit 11 .

[0042] The integrating unit 15 integrates the information derived from the observed signal and the information derived from the auxiliary information. In other words, the integrating unit 15 integrates the output of the state space module 13 with the latent representations acquired by the representation extraction unit 11. The integration result is sent to the state space module 16.

[0043] The integration unit 15 can perform integration using an existing method such as vector element product or addition, or FiLM (Feature-wise Linear Modulation).

[0044] <<State Space Module 16 >> The state space module 16 receives the integration result from the integration unit 15 .

[0045] The state space module 16 is, for example, a neural network configured with a multi-layer state space model, such as S4, S4D, S4ND, DSS, or H3.

[0046] The output of the state space module 16 is sent to a convolutional neural network 17 .

[0047] <<Convolutional Neural Network 17>> The output of the state space module 16 is input to the convolutional neural network 17.

[0048] The convolutional neural network 17 is, for example, a multi-layer convolutional neural network, and may be configured as an existing one-dimensional convolutional neural network.

[0049] The output of the convolutional neural network 17 is sent to a decoder 18 .

[0050] <<Decoder 18 >> The decoder 18 receives the output of the state space module 16 as input.

[0051] The decoder 18 converts the representation generated by the upstream processing layer, in other words, the output of the state space module 16, into a desired representation, such as a waveform or feature quantity on a sample-by-sample basis, to obtain an enhancement signal. The decoder 18 can be constructed using, for example, a neural network. Furthermore, if the input observation signal is a time-domain signal, the decoder 18 can use signal processing such as a short-time inverse Fourier transform.

[0052] It should be noted that there may be a plurality of state space modules 13 and a plurality of convolutional neural networks 14. Furthermore, the convolutional neural network 14 may be provided in front of the state space module 13.

[0053] Similarly, there may be multiple state space modules 16 and multiple convolutional neural networks 17. Furthermore, the convolutional neural network 17 may be provided in front of the state space module 16.

[0054] Also, one of the state space module 13 and the state space module 16 may be omitted.

[0055] <Output Unit 2> The output unit 2 receives the emphasis signal generated by the sound emphasis unit 1 as input.

[0056] The output unit 2 outputs the emphasis signal from the sound emphasis device (step S2).

[0057] [Sound Emphasis Model Learning Device and Method] As shown in Fig. 3, the sound emphasis model learning device includes a sound emphasis unit 3 and an optimization unit 4. The sound emphasis model learning method is realized, for example, by each component of the sound emphasis model learning device performing the processes from step S3 to step S4 shown in Fig. 4.

[0058] Each component of the sound enhancement model learning device will be described below.

[0059] <Sound Emphasis Unit 3> The sound emphasis unit 3 receives the observation signal and auxiliary information.

[0060] The sound enhancement unit 3 inputs the observed signal, which is an observed sound signal, and the auxiliary information, which is information indicating the target of sound enhancement, into a deep learning model incorporating a state space model, to generate an enhancement signal, which is a signal obtained by enhancing the signal that is the target of sound enhancement determined by the auxiliary information in the observed signal (step S3). The generated enhancement signal is sent to the optimization unit 4.

[0061] The sound enhancement unit 3 is the same as the sound enhancement unit 1 described in [Sound enhancement device and method], and therefore a description thereof will be omitted here. The observed signal, auxiliary information, and deep learning model are also the same as those described above, and therefore a description thereof will be omitted here.

[0062] <Optimization Unit 4> The optimization unit 4 receives as input the emphasis signal generated by the sound emphasis unit 3 and the target signal, which is the correct data corresponding to the observed signal and auxiliary information input to the sound emphasis unit 3.

[0063] The optimization unit 4 uses the input emphasis signal and the input target signal to optimize the parameters of the deep learning model of the sound emphasis unit 3 (step S4). For example, the optimization unit 4 optimizes the parameters of the deep learning model of the sound emphasis unit 3 based on the value of a loss function calculated using the input emphasis signal and the input target signal.

[0064] The parameters of a deep learning model are specifically the learnable parameters among the parameters of the deep learning model. An example of a loss used in the loss function is the SNR loss.

[0065] For example, the optimization unit 4 can optimize the parameters using existing methods such as the steepest descent method, momentum, RMSProp, Adam, etc.

[0066] [Modifications] The specific configurations of the embodiments of the disclosed technology are not limited to the configurations described above. The specific configurations of the embodiments of the disclosed technology can be appropriately modified in design, etc., within the scope of the spirit of the embodiments of the disclosed technology.

[0067] The various processes described in the embodiments of the disclosed technology may not only be performed chronologically in the order described, but may also be performed in parallel or individually depending on the processing capacity of the device performing the processes or as needed.

[0068] For example, data may be exchanged directly between the components of the sound enhancement device, or may be exchanged via a storage unit (not shown).

[0069] Furthermore, a device (terminal) for using the device, system, or method of the present invention via a network (telecommunications line) may also be provided. The "device (terminal) for use" may be provided with functions (e.g., control function, decoding function, restoration function, input / output function, etc.) necessary to obtain the effects of implementing the device, system, or method of the present invention.

[0070] It goes without saying that other modifications are possible without departing from the spirit of the present invention.

[0071] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or technical standard was specifically and individually indicated to be incorporated by reference.

[0072] [Program, Recording Medium] The functions realized by the components described in this specification may be implemented in circuitry or processing circuitry, including general-purpose processors, application-specific processors, integrated circuits, ASICs (Application Specific Integrated Circuits), CPUs (Central Processing Units), conventional circuits, and / or combinations thereof, programmed to realize the described functions. A processor includes transistors and other circuits and is considered to be circuitry or processing circuitry. A processor may also be a programmed processor that executes a program stored in a memory.

[0073] In this specification, a circuitry, unit, or means is hardware that is programmed to realize or performs the described functions, which may be any hardware disclosed herein or any hardware known to be programmed to realize or perform the described functions.

[0074] If the hardware is a processor considered to be a type of circuitry, the circuitry, means, or unit is a combination of the hardware and software used to configure the hardware and / or processor.

[0075] The various processes described above can be implemented by loading a program that executes each step of the above method into the recording unit 2020 of the computer 2000 shown in Figure 6, and operating the control unit 2010, input unit 2030, output unit 2040, display unit 2050, etc.

[0076] The program describing the processing contents can be recorded on a computer-readable recording medium, which may be, for example, a magnetic recording device, an optical disk, a magneto-optical recording medium, a semiconductor memory, or any other suitable recording medium.

[0077] The program may be distributed by, for example, selling, transferring, lending, etc. portable recording media such as DVDs and CD-ROMs on which the program is recorded. Furthermore, the program may be stored in a storage device of a server computer, and then transferred from the server computer to other computers via a network, thereby distributing the program.

[0078] A computer that executes such a program may first temporarily store the program recorded on a portable recording medium or transferred from a server computer in its own storage device. Then, when executing a process, the computer reads the program stored on its own recording medium and executes the process in accordance with the read program. Alternatively, the computer may read the program directly from a portable recording medium and execute the process in accordance with the program. Furthermore, the computer may execute the process in accordance with the program each time a program is transferred from a server computer to the computer. Alternatively, the server computer may not transfer the program to the computer, but may instead execute the process through a so-called ASP (Application Service Provider) service, which realizes the processing function by issuing an execution instruction and obtaining the results. Furthermore, the server computer may execute the process at the terminal using a so-called SaaS (Software as a Service) service, which allows users to use part of a server computer along with the program. In this embodiment, the program includes information used for processing by an electronic computer that is equivalent to a program (such as data that is not a direct instruction to a computer but has properties that dictate computer processing).

[0079] Furthermore, in this embodiment, the device is configured by executing a predetermined program on a computer, but at least a part of the processing contents may be realized by hardware.

[0080] [Comparative Evaluation] The results of the comparative evaluation are shown in Figure 7. The performance evaluation criteria are SDR (Signal to Distortion Ratio), which is an index showing speech enhancement performance, and RTF (Real Time Factor), which indicates the calculation time required to process one second of speech. A higher SDR indicates better enhancement performance, and a lower RTF indicates lower calculation costs required for processing.

[0081] The comparative evaluation shows that the proposed model structure has succeeded in significantly reducing computational costs while improving speech enhancement performance compared to conventional model structures.

Claims

1. A sound enhancement device including a sound enhancement unit that generates an enhancement signal, which is a signal that enhances the signal in the observation signal that is the target of sound enhancement determined by the auxiliary information, by inputting an observation signal, which is an observed sound signal, and auxiliary information, which is information indicating the target of sound enhancement, into a deep learning model that incorporates a state space model.

2. The sound enhancement device according to claim 1, wherein the deep learning model has a structure in which a part of a Conv-TasNet structure is replaced with a state space model.

3. A sound enhancement method including a sound enhancement step of inputting an observed signal, which is an observed sound signal, and auxiliary information, which is information indicating a target of sound enhancement, into a deep learning model incorporating a state space model, to generate an enhanced signal, which is a signal obtained by enhancing the signal in the observed signal that is the target of sound enhancement determined by the auxiliary information.

4. A program for causing a computer to execute each step of the sound enhancement model training method of claim 3.

Images