Signal extraction device and signal extraction method

Abstract

In a signal extraction device (10), an acquisition unit (15a) acquires a mixed signal of a plurality of sound sources and clue information for a target sound source. An estimation unit (15c) inputs, into a trained model (14a), data from a sampled step and the acquired mixed signal and clue information of the target sound source, and estimates a score corresponding to said data. An extraction unit (15d) uses the estimated score to extract a signal of the target sound source from the mixed signal of the plurality of sound sources.

Description

Signal extraction device and signal extraction method

[0001] The present invention relates to a signal extraction device and a signal extraction method.

[0002] In recent years, technologies for extracting the voice of a specific speaker from a mixed voice signal have been studied in order to improve the performance of voice recognition and speaker recognition, make voices easier to hear in conversation applications, etc. For example, a discriminative target sound source extraction system has been proposed in which an observed signal and some kind of clue that can identify a target sound, such as a pre-recorded voice of a target speaker, are input into a DNN (Deep Neural Network) model, and a target sound source signal is output (see Non-Patent Document 1).

[0003] Also, a method for removing noise from observed signals has been disclosed, which uses a diffusion model to generate data by inversely changing a Markov process that changes data to Gaussian noise (see Non-Patent Document 2).

[0004] Katerina Zmolikova, “SpeakerBeam: Speaker Aware Neural Network for Target Speaker Extraction in Speech Mixtures”, IEEE JOURNAL OF SELECTED TOPICS IN SIGNAL PROCESSING, VOL. 13, NO. 4, August 2019 Julius Richter, “Speech Enhancement and Dereverberation with Diffusion-based Generative Models”, IEEE / ACM Transactions on Audio, Speech and Language Processing, 2023

[0005] However, with conventional techniques, it has been difficult to extract a target sound source signal from a mixed signal with high accuracy. For example, the auditory quality of the extracted target sound source signal is insufficient. Furthermore, it has been difficult to improve the quality even when the target sound source signal includes a failure to extract a signal different from the target signal.

[0006] The present invention has been made in view of the above, and has an object to extract a target sound source signal from a mixed signal with high accuracy in the prior art.

[0007] In order to solve the above-mentioned problems and achieve the object, the signal extraction device according to the present invention is characterized by having an acquisition unit that acquires a mixed signal of multiple sound sources and clue information of a target sound source, an estimation unit that inputs the acquired mixed signal, the clues of the target sound source, and data at a sampled step into a trained model and estimates a score corresponding to the data, and an extraction unit that extracts a signal of the target sound source from the mixed signal of the multiple sound sources using the estimated score.

[0008] According to the present invention, it is possible to extract a target sound source signal from a mixed signal with high accuracy.

[0009] FIG. 1 is a diagram for explaining an overview of the signal extraction device of this embodiment. FIG. 2 is a diagram for explaining an overview of the signal extraction device of this embodiment. FIG. 3 is a diagram for explaining an overview of the signal extraction device of this embodiment. FIG. 4 is a diagram for explaining an overview of the signal extraction device of this embodiment. FIG. 5 is a diagram for explaining an overview of the signal extraction device of this embodiment. FIG. 6 is a schematic diagram illustrating an example of the general configuration of the signal extraction device of this embodiment. FIG. 7 is a diagram for explaining signal extraction processing. FIG. 8 is a diagram for explaining signal extraction processing. FIG. 9 is a flowchart showing the signal extraction processing procedure. FIG. 10 is a diagram for explaining an example. FIG. 11 is a diagram for explaining an example. FIG. 12 is a diagram showing an example of a computer that executes a signal extraction program.

[0010] Hereinafter, an embodiment of the present invention will be described in detail with reference to the drawings. Note that the present invention is not limited to this embodiment. In addition, in the description of the drawings, the same parts are designated by the same reference numerals.

[0011] [Outline of Signal Extraction Device] FIGS. 1 to 5 are diagrams illustrating an outline of the signal extraction device of this embodiment. First, FIG. 1 illustrates a conventional deterministic target sound source extraction process. As shown in FIG. 1, in deterministic target sound source extraction, an observed signal y and some kind of clue c capable of identifying target sound, such as pre-recorded speech from a target speaker, are input to a speech extraction model, and a target sound source signal is output. This speech extraction model is trained using a DNN to minimize the difference between the correct signal and the output signal. Because this speech extraction model is not a data generation model that follows a probability distribution, it can only generate deterministic speech that outputs the same result regardless of timing. Therefore, there is no mechanism for improving when a failure, such as a signal that is different from the target, is included.

[0012] On the other hand, Fig. 2 illustrates the processing of a conventional diffusion model. As shown in Fig. 2, in the diffusion model, the data x ０ to Gaussian noise x Ｔ For a Markov process (forward process) that changes x ０ From x Ｔ The data x is then transformed in the reverse direction using a Markov process. ０ In this case, the DNN generates the first step p(x Ｔ |x Ｔ－１ ) is learned to achieve this.

[0013] FIG. 3 also illustrates processing of a Score Based Diffusion model that uses a value called Score in the Diffusion model shown in FIG. 2. Specifically, as shown in FIG. 3(a), the Score Based Diffusion model uses Stochastic Differential Equations (SDE) that describe continuous state transitions. In this case, the DNN (Score model) is trained using the correct Score value corresponding to the data to be generated as training data, as shown in FIG. 3(b). Then, as shown in FIG. 3(c), the trained Score model is used to calculate one-step transitions (data x at step t) ｔ ) is repeated to generate data.

[0014] 4 illustrates a process for removing noise from a speech signal (see Non-Patent Document 2) using the Score Based Diffusion model shown in FIG. 3. As shown in FIGS. 4(a) and 4(b), this technology uses a DNN (Score model) similar to that shown in FIG. 3 to generate clean speech from which noise has been removed under the conditions of the observed signal y. This DNN is a noise removal model and cannot extract the target sound source. Furthermore, there is no mechanism for improving the model even if a failure is included.

[0015] Therefore, the signal extraction device of this embodiment uses the Score Based Diffusion model (Score model) shown in FIG. 3 to generate a signal of the target sound source under the conditions of the observed signal y and the clue c of the target sound source.

[0016] 5 illustrates the signal extraction process of this embodiment. Specifically, the score model of the signal extraction device is generated by generating a score x as shown in FIG. ０ to Gaussian noise x Ｔ For the forward process shown in the following equation (1), ０ From x ＴThen, the data x at step t is changed in the reverse direction as shown in the following equation (2). ｔ By repeating the inference, data x ０ Generate.

[0017]

[0018] In this way, the signal extraction device generates a speech signal of a target sound source using a diffusion model, which is a generative model, thereby enabling improvements in auditory quality, such as improvements in the performance of PESQ (Perceptual Evaluation of Speech Quality).

[0019] Furthermore, in the signal extraction device, the inference results change probabilistically depending on the random number seed used when sampling the inference step number t, etc., as shown in Figures 5(b) and 5(c). Therefore, the signal extraction device performs ensemble of the inference results, which makes it possible to improve the accuracy of the inference results.

[0020] [Configuration of the Signal Extraction Device] Fig. 6 is a schematic diagram illustrating the general configuration of a signal extraction device of this embodiment. As illustrated in Fig. 6, a signal extraction device 10 of this embodiment is realized by a general-purpose computer such as a personal computer, and includes an input unit 11, an output unit 12, a communication control unit 13, a storage unit 14, and a control unit 15.

[0021] The input unit 11 is realized using input devices such as a microphone, keyboard, and mouse, and inputs various instruction information such as a command to start processing to the control unit 15 in response to input operations by an operator. The output unit 12 is realized by a display device such as a liquid crystal display, a printing device such as a printer, etc. For example, the output unit 12 displays the results of a signal extraction process described below.

[0022] The communication control unit 13 is realized by a NIC (Network Interface Card) or the like, and controls communication between the control unit 15 and external devices via telecommunication lines such as a LAN (Local Area Network) or the Internet. For example, the communication control unit 13 controls communication between the control unit 15 and a management device or the like that manages various types of information.

[0023] The storage unit 14 is realized by a semiconductor memory element such as a RAM (Random Access Memory) or a flash memory, or a storage device such as a hard disk or an optical disk. The storage unit 14 stores in advance processing programs for operating the signal extraction device 10 and data used during execution of the processing programs, or temporarily stores the data each time processing is performed. The storage unit 14 may be configured to communicate with the control unit 15 via the communication control unit 13.

[0024] In this embodiment, the storage unit 14 stores a model 14a or a cascade model 140a, etc., which are used in the signal extraction process described later.

[0025] The control unit 15 is realized using a CPU (Central Processing Unit), NP (Network Processor), FPGA (Field Programmable Gate Array), etc., and executes processing programs stored in memory. As a result, the control unit 15 functions as an acquisition unit 15a, a learning unit 15b, an estimation unit 15c, and an extraction unit 15d, as exemplified in FIG. 6, and executes signal extraction processing. Note that these functional units may each be implemented in different hardware, or some of them may be implemented in different hardware. The control unit 15 may also include other functional units.

[0026] The acquisition unit 15a acquires a mixed signal of multiple sound sources and clue information of a target sound source. For example, the acquisition unit 15a acquires an observed signal y, which is a mixed signal of multiple sound sources, and some kind of clue c that can identify the target sound source, such as a pre-recorded voice of a target speaker, from a microphone, a management device that manages various information, or the like, via the input unit 11 or the communication control unit 13. The acquisition unit 15a may store the acquired observed signal y and clue information c of the target sound source in the storage unit 14.

[0027] The learning unit 15b uses the acquired mixed signal y, the clue c of the target sound source, and the sampled data x at the t-th step. ｔ When you input the data x ｔSpecifically, as shown in FIG. 5B, the model 14a is generated by learning to estimate a score corresponding to data x ｔ When the observed signal y and the clue c are input, x ｔ A model (Score model) 14a that outputs a score corresponding to the score is generated by learning.

[0028] This model 14a is a Gaussian noise x Ｔ Data to be generated from x ０ The data x is then converted into ０ Specifically, the model 14a is a diffusion model that generates the data x at step t using the score, as illustrated in FIG. ｔ By repeating the inference, data x ０ This is a score-based diffusion model that generates

[0029] The estimation unit 15c estimates the mixed signal y, the clue c of the target sound source, and the sampled data x at the t-th step. ｔ Specifically, as shown in FIG. 5C, the mixed signal y and clues c and x of the target sound source are input to the trained model 14a, and the score corresponding to the data is estimated. ｔ and get the score output.

[0030] The extraction unit 15d extracts the signal of the target sound source from the mixed signal y of the multiple sound sources by using the estimated score. Specifically, as illustrated in FIG. 5C, the extraction unit 15d extracts x Ｔ－Δｔ The process of estimating the inference result x ０ By repeating this process until we obtain the signal x of the target sound source, ０ Extract.

[0031] Here, if the random number seed used during sampling is changed, the inference result will change probabilistically. Therefore, the extraction unit 15d may use the average value of the target sound source signals extracted in each of a predetermined number of processes as the extraction result. In other words, the extraction unit 15d may perform ensemble to change the random number seed, repeat inference a predetermined number of times, and calculate the average value of the inference results. This makes it possible to improve the accuracy of the inference.

[0032] 7 and 8 are diagrams for explaining the signal extraction process. First, Fig. 7 illustrates the process of the model 14a (DNN, Score model) shown in Fig. 5.

[0033] Specifically, during training (see FIG. 5B), the training unit 15b first receives the mixed speech y and the target speaker speech x as observed signals. ０ , using the target speaker's voice c randomly selected as a cue for the target sound source, data (y, x ０ , c) is created.

[0034] The learning unit 15b also samples step number t from a uniform distribution of 0≦t≦T, and calculates x ０ Using x ｔ Sampling is performed.

[0035]

[0036] Then, the learning unit 15b applies the data (y, x ｔ , c) and outputs a score. The learning unit 15b also calculates a loss function to minimize the difference between the correct score and the output score, and performs a backward calculation.

[0037] During inference (see FIG. 5(c)), the estimation unit 15c calculates x Ｔ Also, [0, T] is divided into N parts, and Δt = T / N. (x Ｔ , x Ｔ－Δｔ , ..., x ０Specifically, the estimation unit 15c applies discontinuous state transitions to the model 14a using the data (y, x ｔ , c) and obtains the score output. The extraction unit 15d also uses the output score to extract x Ｔ－Δｔ The process of estimating the inference result x ０ Repeat until you get

[0038] Note that changing the random number seed during sampling will probabilistically change the inference results. Therefore, the extraction unit 15d may perform ensemble, changing the random number seed and repeating the inference a predetermined number of times to calculate the average value of the inference results. This makes it possible to improve the accuracy of the inference.

[0039] 8A illustrates the processing of a model (Discriminative TSE (Target Source Extraction)) 141a of the deterministic target sound source extraction processing illustrated in Fig. 1. 8B illustrates the processing of a model (cascade model) 140a that combines the model 141a of the deterministic target sound source extraction processing of Fig. 8A and the model 14a of the signal extraction processing illustrated in Fig. 7.

[0040] In the cascade model 140a, the learning unit 15b performs multitask learning using two loss functions: a loss function of the score and a loss function of the target speaker's voice x. The cascade model 140a can be used in two ways: as a diffusion model that performs inference using the score, and as a deterministic model that uses a direct estimate of x.

[0041] Specifically, when training the cascade model 140a, the training unit 15b uses the mixed speech y and the target speaker speech x as observed signals, similarly to when training the model 14a in FIG. ０ , using the speech of a randomly selected target speaker c as a cue for the target sound source, data (y, x ０ , c) is created.

[0042] The learning unit 15b also samples step number t from a uniform distribution of 0≦t≦T, and calculates x ０ Using x ｔ Sampling is performed.

[0043] The learning unit 15b inputs data (y, c) to the model 141a of the model 140a in FIG. 8B to obtain an output of x^. The learning unit 15b also inputs data (x^) to the model 142a. ｔ , x^) is input to obtain an output score. Then, the learning unit 15b calculates a loss function of the difference between the correct score and the output score, and calculates the relationship between x^ and x ０ and the loss function of the difference between them are calculated to minimize each other, and learning is performed by performing backward calculations.

[0044] When the cascade model 140a is estimated, the estimation unit 15c calculates x according to the above formulas (3) to (5), in the same way as when the model 14a of FIG. Ｔ The estimation unit 15c divides [0, T] into N parts to obtain Δt=T / N, and then performs sampling of (x Ｔ , x Ｔ－Δｔ , ..., x ０ Specifically, the estimation unit 15c inputs data (y, x) to the cascade model 140a, and performs discontinuous state transitions by a reverse process. ｔ , c) and obtains the score output. The extraction unit 15d uses the output score to extract x Ｔ－Δｔ The process of estimating the inference result x ０ Repeat until you get

[0045] 7, if the random number seed used during sampling is changed, the inference result will change probabilistically. Therefore, the extraction unit 15d may perform ensemble to change the random number seed, repeat the inference a predetermined number of times, and calculate the average value of the inference results. This makes it possible to improve the accuracy of the inference.

[0046] [Signal Extraction Processing] Next, the signal extraction processing performed by the signal extraction device 10 according to this embodiment will be described with reference to Fig. 9. Fig. 9 is a flowchart showing the procedure of the signal extraction processing. The flowchart in Fig. 9 starts, for example, when the user performs an operation input to instruct the start of the processing.

[0047] First, the acquisition unit 15a acquires an observed signal y, which is a mixed signal of multiple sound sources, and a clue c of a target sound source (step S1).

[0048] Next, the estimation unit 15c estimates the obtained observed signal y, the clue c of the target sound source, and the sampled data x at the t-th step. ｔ The mixed signal y and the clues c and x of the target sound source are input to the trained model 14a, and the score corresponding to the data is estimated (step S2). ｔ and get the score output.

[0049] Then, the extraction unit 15d extracts the signal of the target sound source from the mixed signal y of the multiple sound sources using the estimated score (step S3). Specifically, the extraction unit 15d extracts x Ｔ－Δｔ The process of estimating the inference result x ０ By repeating this process until we obtain the signal x of the target sound source, ０ This completes the signal extraction process.

[0050] [Effects] As described above, in the signal extraction device 10 of this embodiment, the acquisition unit 15a acquires a mixed signal of multiple sound sources and clue information of a target sound source. The estimation unit 15c inputs the acquired mixed signal, the clue information of the target sound source, and sampled data at the t-th step into the trained model 14a and estimates a score corresponding to the data. The extraction unit 15d extracts a signal of the target sound source from the mixed signal of multiple sound sources using the estimated score.

[0051] Specifically, the trained model 14a is a diffusion model that generates the data to be generated by an inverse process that changes the data from Gaussian noise to the data to be generated step by step.

[0052] In this way, by using a generative model based on a probability distribution, it is possible to improve the performance of PESQ and other auditory quality. Therefore, it becomes possible to extract a target sound source signal from a mixed signal with high accuracy.

[0053] Furthermore, the extraction unit 15d sets the average value of the target sound source signals extracted in each of the predetermined number of processes as the extraction result. In this way, by utilizing the fact that the inference result changes probabilistically when the random number seed is changed, it is possible to improve the accuracy of the inference.

[0054] [Example] FIGS. 10 and 11 are diagrams for explaining an example. First, in the example shown in FIG. 10, the model 14a of the above embodiment was used, and the inference process was repeated 10 times with different random number seeds. FIG. 10 illustrates each inference result and an ensemble result obtained by simply averaging the inference results in the time-frequency domain. As illustrated in FIG. 10(a), it was confirmed that the SDR (Source to Distortion Ratio) of the ensemble result (Ensemble) indicated by "x" was better than the SDR of each inference result (Extracted Samples) indicated by "·".

[0055] An example of the ensemble is shown in Fig. 10(b). As can be seen from the bold-framed area in the figure, it was confirmed that the ensemble result (Ensemble) better represents the inference result of the target sound source (Target) for the mixed signal (Mixture).

[0056] 11 illustrates an example of a perceptual evaluation of speech quality (PESQ) as an evaluation result of each of the above methods. As shown in FIG. 11, it was confirmed that the model 14a of the present invention illustrated in FIG. 7, using ensembles, performed better than the conventional discriminative TSE methods illustrated in FIGS. 1 and 8(a). It was also confirmed that the cascade model 140a of the present invention illustrated in FIG. 8(b) performed better than the conventional discriminative TSE methods, regardless of whether or not an ensemble was used.

[0057] [Program] A program describing the processing performed by the signal extraction device 10 according to the above embodiment in a computer-executable language can also be created. In one embodiment, the signal extraction device 10 can be implemented by installing a signal extraction program that executes the above signal extraction processing as package software or online software on a desired computer. For example, by executing the signal extraction program on an information processing device, the information processing device can function as the signal extraction device 10. The information processing device referred to here includes desktop and notebook personal computers. Other examples of information processing devices include mobile communication terminals such as smartphones, mobile phones, and PHS (Personal Handyphone System) phones, as well as slate terminals such as PDAs (Personal Digital Assistants). The functions of the signal extraction device 10 may also be implemented on a cloud server.

[0058] 12 is a diagram showing an example of a computer that executes a signal extraction program. The computer 1000 includes, for example, a memory 1010, a CPU 1020, a hard disk drive interface 1030, a disk drive interface 1040, a serial port interface 1050, a video adapter 1060, and a network interface 1070. These components are connected by a bus 1080.

[0059] The memory 1010 includes a ROM (Read Only Memory) 1011 and a RAM 1012. The ROM 1011 stores a boot program such as a BIOS (Basic Input Output System). The hard disk drive interface 1030 is connected to a hard disk drive 1031. The disk drive interface 1040 is connected to a disk drive 1041. A removable storage medium such as a magnetic disk or optical disk is inserted into the disk drive 1041. The serial port interface 1050 is connected to a mouse 1051 and a keyboard 1052, for example. The video adapter 1060 is connected to a display 1061, for example.

[0060] Here, the hard disk drive 1031 stores, for example, an OS (Operating System) 1091, an application program 1092, a program module 1093, and program data 1094. The various pieces of information described in the above embodiments are stored in the hard disk drive 1031 or the memory 1010, for example.

[0061] The signal extraction program is stored in the hard disk drive 1031 as, for example, a program module 1093 in which instructions to be executed by the computer 1000 are written. Specifically, the program module 1093 in which each process executed by the signal extraction device 10 described in the above embodiment is written is stored in the hard disk drive 1031.

[0062] Furthermore, data used for information processing by the signal extraction program is stored as program data 1094, for example, in the hard disk drive 1031. Then, the CPU 1020 reads the program module 1093 and the program data 1094 stored in the hard disk drive 1031 into the RAM 1012 as necessary, and executes each of the above-described procedures.

[0063] Note that the program module 1093 and program data 1094 related to the signal extraction program are not limited to being stored in the hard disk drive 1031, and may be stored in, for example, a removable storage medium and read by the CPU 1020 via the disk drive 1041. Alternatively, the program module 1093 and program data 1094 related to the signal extraction program may be stored in another computer connected via a network such as a LAN or a WAN (Wide Area Network), and read by the CPU 1020 via the network interface 1070.

[0064] Although the present invention has been described above as an embodiment, the present invention is not limited to the description and drawings that form part of the disclosure of the present invention. In other words, other embodiments, examples, and operational techniques that can be made by those skilled in the art based on the present invention are all included in the scope of the present invention.

[0065] REFERENCE SIGNS LIST 10 Signal extraction device 11 Input unit 12 Output unit 13 Communication control unit 14 Storage unit 14a, 140a, 141a, 142a Model 15 Control unit 15a Acquisition unit 15b Learning unit 15c Estimation unit 15d Extraction unit

Claims

1. A signal extraction device comprising: an acquisition unit that acquires a mixed signal of multiple sound sources and clue information of a target sound source; an estimation unit that inputs the acquired mixed signal, the clue information of the target sound source, and data at a sampled step into a trained model and estimates a score corresponding to the data; and an extraction unit that extracts a signal of the target sound source from the mixed signal of multiple sound sources using the estimated score.

2. The signal extraction device according to claim 1, characterized in that the trained model is a diffusion model that generates the data by an inverse process of changing the data to be generated stepwise from Gaussian noise.

3. The signal extraction device according to claim 1, wherein the extraction section takes as an extraction result an average value of the signal of the target sound source extracted in each processing step a predetermined number of times.

4. A signal extraction method executed by a signal extraction device, comprising: an acquisition step of acquiring a mixed signal of multiple sound sources and clue information of a target sound source; an estimation step of inputting the acquired mixed signal, the clue information of the target sound source, and data at a sampled step into a trained model and estimating a score corresponding to the data; and an extraction step of extracting a signal of the target sound source from the mixed signal of multiple sound sources using the estimated score.

Images