Extraction device, extraction method, learning device, learning method, and program

Abstract

To extract a target voice from a mixed voice accurately and easily.SOLUTION: A generation unit 131a of an extraction device 10 generates data of a predetermined format from activity information representing a time at which a target sound source emits sound, and a mixed sound. The generation unit 131a generates a weighted sum of an output obtained by inputting the mixed sound into auxiliary NN and the activity information, a coupling vector obtained by coupling the activity information and the mixed sound, and the like. An extraction unit 131b extracts the sound of the target sound source from the mixed sound using the data generated by the generation unit 131a and an extraction network.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to an extraction device, an extraction method, a learning device, a learning method, and a program. [Background technology]

[0002] 2. Description of the Related Art SpeakerBeam is known as a technique for extracting the voice of a target speaker from a mixed voice signal obtained from the voices of a plurality of speakers (see, for example, Non-Patent Document 1).

[0003] For example, the method described in Non-Patent Document 1 has a main NN (neural network) that converts a mixed speech signal into the time domain and extracts the speech of a target speaker from the time-domain mixed speech signal, and an auxiliary NN that extracts features from the speech signal of the target speaker.By inputting the output of the auxiliary NN to an adaptation layer provided in the middle part of the main NN, the target speaker's speech signal contained in the time-domain mixed speech signal is estimated and output.

[0004] FIG. 13 is a diagram illustrating a conventional speaker beam. As shown in FIG. 13, in the conventional speaker beam, the mixed sound y t is input to the extraction network. Also, the target speaker's voice a t is input to the auxiliary network.

[0005] Furthermore, the time average e of the output of the auxiliary network (equation (1)) is input between the first and second extraction blocks of the extraction network. Then, the mask ^m t (^ just above m) to create the mixed sound y t From target voice ^x t is extracted.

[0006]

number

[0007] [Non-Patent Document 1] Marc Delcroix, et al. “Improving speaker discrimination of target speech extraction with time-domain SpeakerBeam”, https: / / arxiv.org / pdf / 2001.08378.pdf Summary of the Invention [Problem to be solved by the invention]

[0008] However, the conventional techniques have a problem in that it may not be possible to extract the target voice from the mixed voice with high accuracy and ease.

[0009] For example, the method described in Non-Patent Document 1 requires the voice of the target speaker to be registered in advance. In addition, for example, due to fatigue or changes in physical condition caused by a long meeting, the characteristics of the target speaker's voice may deviate from the characteristics of the voice registered in advance. [Means for solving the problem]

[0010] In order to solve the above-mentioned problems and achieve the objective, the extraction device is characterized by having a generation unit that generates data of a predetermined format from activity information indicating the time when a target sound source emitted sound and a mixed sound, and an extraction unit that extracts the sound of the target sound source from the mixed sound using the data generated by the generation unit and an extraction NN (neural network).

[0011] The learning device also includes a generation unit that generates data of a predetermined format from activity information indicating the time when the target sound source emitted sound and a mixed sound, an extraction unit that extracts the sound of the target sound source from the mixed sound using the data generated by the generation unit and an extraction NN (neural network), and an update unit that updates parameters of the extraction NN so that a loss function calculated based on the sound extracted by the extraction unit is optimized. Effect of the Invention

[0012] According to the present invention, a target voice can be extracted from a mixed voice with high accuracy and with ease. [Brief description of the drawings]

[0013] [Figure 1] FIG. 1 is a diagram illustrating an example of the configuration of an extraction device according to the first embodiment. [Diagram 2] FIG. 2 is a diagram illustrating diarization. [Diagram 3] FIG. 3 is a diagram illustrating the activity information. [Figure 4] FIG. 4 is a diagram illustrating an example of a model configuration. [Diagram 5] FIG. 5 is a diagram illustrating an example of a model configuration. [Figure 6] FIG. 6 is a diagram illustrating an example of a model configuration. [Figure 7] FIG. 7 is a diagram illustrating an example of the configuration of the learning device according to the first embodiment. [Figure 8] FIG. 8 is a flowchart showing a flow of processing of the extraction device according to the first embodiment. [Figure 9] FIG. 9 is a flowchart showing a flow of processing of the learning device according to the first embodiment. [Figure 10] FIG. 10 shows the experimental results. [Figure 11] FIG. 11 shows the experimental results. [Figure 12] FIG. 12 is a diagram illustrating an example of a computer that executes a program. [Figure 13] FIG. 13 is a diagram illustrating a conventional speaker beam. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0014] Hereinafter, an embodiment of an extraction device, an extraction method, a learning device, a learning method, and a program according to the present application will be described in detail with reference to the drawings. Note that the present invention is not limited to the embodiment described below.

[0015] [First embodiment] 1 is a diagram showing an example of the configuration of an extraction device according to the first embodiment. As shown in FIG. 1, the extraction device 10 includes an interface unit 11, a storage unit 12, and a control unit 13.

[0016] The extraction device 10 receives an input of a mixed sound including sounds from a plurality of sound sources. The extraction device 10 also extracts a sound of a target sound source from the mixed sound and outputs the extracted sound.

[0017] In this embodiment, the sound source is a speaker. In this case, the mixed sound is a mixture of sounds uttered by multiple speakers. For example, the mixed sound is obtained by recording the sound of a conference in which multiple speakers participate with a microphone. In the following description, the "sound source" may be replaced with the "speaker" as appropriate.

[0018] In this embodiment, not only the voice uttered by a speaker but also sounds from any sound source can be handled. For example, the extraction device 10 can receive input of a mixed sound whose sound source is an acoustic event such as the sound of a musical instrument or a car siren, extract the sound of the target sound source, and output it. In the following description, "voice" may be replaced with "sound" as appropriate.

[0019] The interface unit 11 is an interface for inputting and outputting data. For example, the interface unit 11 is a network interface card (NIC). The interface unit 11 may be connected to an output device such as a display and an input device such as a keyboard.

[0020] The storage unit 12 is a storage device such as a hard disk drive (HDD), a solid state drive (SSD), an optical disk, etc. The storage unit 12 may be a semiconductor memory in which data can be rewritten, such as a random access memory (RAM), a flash memory, or a non-volatile static random access memory (NVSRAM). The storage unit 12 stores an operating system (OS) and various programs executed by the extraction device 10.

[0021] 1, the storage unit 12 stores model information 121. The model information 121 is parameters for constructing a model, etc. For example, the model information 121 is weights and biases for constructing each neural network, which will be described later.

[0022] The control unit 13 controls the entire extraction device 10. The control unit 13 is, for example, an electronic circuit such as a CPU (Central Processing Unit), an MPU (Micro Processing Unit), or a GPU (Graphics Processing Unit), or an integrated circuit such as an ASIC (Application Specific Integrated Circuit) or an FPGA (Field Programmable Gate Array). The control unit 13 also has an internal memory for storing programs that define various processing procedures and control data, and executes each process using the internal memory.

[0023] The control unit 13 functions as various processing units by executing various programs. For example, the control unit 13 includes a signal processing unit 131.

[0024] The signal processing unit 131 extracts the target speech from the mixed speech by using a model constructed from the model information 121. The model constructed from the model information 121 is assumed to be a model trained by a learning device. The signal processing unit 131 includes a generation unit 131a and an extraction unit 131b.

[0025] The generating unit 131a generates data in a predetermined format from activity information indicating the time when the target sound source emitted a sound and the mixed sound.

[0026] Here, the activity information will be described. The activity information is information that indicates the time when the target sound source emits a sound.

[0027] For example, activity information can be obtained by known techniques such as visual-based voice activity detection (VAD) (Reference 1), personal VAD (Reference 2), and diarization (References 3-5). Reference 1: P. Liu and Z. Wang, “Voice activity detection using visual information,” in Proc. of ICASSP'04, 2004, vol. 1, pp. I-609. Reference 2: S. Ding, Q. Wang, S.-Y. Chang, L. Wan, and I. Lopez Moreno, “Personal VAD: Speaker-Conditioned Voice Activity Detection,” in Proc. of Odyssey'20, 2020, pp. 433-439. Reference 3: D. Garcia-Romero, D. Snyder, G. Sell, D. Povey, and A. McCree, “Speaker diarization using deep neural network embeddings,” Proc. of ICASSP'17, pp. 4930-4934, 2017. Reference 4: Z. Huang, S. Watanabe, Y. Fujita, P. Garcia, Y. Shao, D. Povey, and S. Khudanpur, “Speaker diarization with region proposal network,” Proc. of ICASSP'20, pp. 6514-6518, 2020. Reference 5: I. Medennikov, M. Korenevsky, T. Prisyach, YY Khokhlov, M. Korenevskaya, I. Sorokin, TV Timofeeva, A. Mitrofanov, A. Andrusenko, I. Podluzhny, A. Laptev, and A. Romanenko, “Target-speaker voice activity detection: a novel approach for multi-speaker diarization in a dinner party scenario,” ArXiv, vol. abs / 2005.07272, 2020.

[0028] For example, visual-based VAD is effective when video of the target speaker is available, personal VAD is effective when audio of the target speaker is available, and diarization is used when neither video nor audio of the target speaker is available.

[0029] In particular, by combining the current highly accurate diarization with this embodiment, it is possible to improve the speech recognition accuracy in a practical and simple manner in situations such as meetings where overlapping occurs.

[0030] Here, overlap refers to a state in which the voice of a target speaker overlaps with the voice of a speaker other than the target speaker in a mixed voice. In the activity information, the time period in which the target speaker is active may include overlap or may not include overlap.

[0031] Diarization is explained in FIG 2. As shown in FIG 2, diarization makes it possible to identify the time periods in which speaker A, speaker B, and speaker C each speak.

[0032] FIG. 3 is a diagram for explaining the activity information. As shown in FIG. 3, the activity information may take a value of 1 during a time period when the sound source is emitting sound, and a value of 0 during a time period when the sound source is not emitting sound. In this case, p t ∈{0,1}.

[0033] The extraction unit 131b extracts the sound of the target sound source from the mixed sound by using the data generated by the generation unit 131a and the extraction network.

[0034] Here, the model constructed based on the model information 121 may be called a speaker activity driven speech extraction neural network (ADEnet). The generation unit 131a and the extraction unit 131b perform processing using ADEnet. Hereinafter, multiple variations of ADEnet will be described.

[0035] In addition, although ADEnet in this embodiment uses activity information as input, the method of acquiring the activity information is not limited to a specific method and any method may be used.

[0036] [ADEnet-auxiliary] In the ADEnet-auxiliary, the generation unit 131a generates a weighted sum of the output obtained by inputting the mixed sound to the auxiliary network and the activity information. The extraction unit 131b inputs the mixed sound and the weighted sum to an extraction network, thereby extracting the sound of the target sound source from the mixed sound.

[0037] Fig. 4 is a diagram showing an example of the configuration of a model. In the model (ADEnet-auxiliary) in Fig. 4, an extraction network and an auxiliary network are used. Both the extraction network and the auxiliary network are neural networks.

[0038] As shown in FIG. 4, the generation unit 131a outputs a mixed voice y t Then, the generation unit 131a generates the activity information p t Calculate the weighted sum e of the outputs of the auxiliary network with weights

[0039]

number

[0040] Furthermore, the extraction unit 131b extracts the mixed voice y t and the weighted sum e are input to the extraction network, and the mask ^m t Then, the extraction unit 131b obtains the mixed voice y t and mask^m t The element-wise product of the target voice ^x t Calculate as follows.

[0041] [ADEnet-input] In the ADEnet-input, the generation unit 131a generates a combined vector by combining activity information and a mixed sound, both of which are represented by vectors. The extraction unit 131b inputs the combined vector to an extraction network to extract the sound of the target sound source from the mixed sound.

[0042] Fig. 5 is a diagram showing an example of the configuration of a model. In the model (ADEnet-input) in Fig. 5, an extraction network is used.

[0043] As shown in FIG. 5, the generation unit 131a generates a mixed voice y t and activity information t Concatenate [y t T ,p t ] T get.

[0044] Then, the extraction unit 131b extracts [y t T ,p t ] T is input to the extraction network, and the mask ^m t Then, the extraction unit 131b obtains the mixed voice y t and mask^m t The element-wise product of the target voice ^x t Calculate as follows.

[0045] [ADEnet-mix] In ADEnet-mix, the generator 131a generates a weighted sum of the output obtained by inputting the mixed sound to the auxiliary network and the activity information in addition to the coupling vector. The extractor 131b inputs the coupling vector and the weighted sum to the extraction network to extract the sound of the target sound source from the mixed sound.

[0046] Fig. 6 is a diagram showing an example of the configuration of a model. In the model (ADEnet-mix) in Fig. 6, an extraction network and an auxiliary network are used.

[0047] As shown in FIG. 6, the generation unit 131a generates a mixed voice y t and activity information t Concatenate [y t T ,p t ] T get.

[0048] Then, the extraction unit 131b extracts the weighted sum e and [y t T ,p t ] T and are input to the extraction network, and the mask ^m t Then, the extraction unit 131b obtains the mixed voice y t and mask^m t The element-wise product of the target voice ^x t Calculate as follows.

[0049] As described herein, the extraction unit 131b can calculate the target speech using a mask, or the extraction unit 131b may directly calculate the target speech using an extraction network or the like without using a mask.

[0050] Here, the configuration of the learning device will be described with reference to Fig. 7. Fig. 7 is a diagram showing an example of the configuration of the learning device according to the first embodiment. As shown in Fig. 7, the learning device 20 has an interface unit 21, a storage unit 22, and a control unit 23.

[0051] The interface unit 21 is an interface for inputting and outputting data. For example, the interface unit 21 is a NIC. The interface unit 21 may be connected to an output device such as a display and an input device such as a keyboard.

[0052] The storage unit 22 is a storage device such as an HDD, an SSD, an optical disk, etc. The storage unit 22 may be a semiconductor memory in which data can be rewritten, such as a RAM, a flash memory, an NVSRAM, etc. The storage unit 22 stores an OS and various programs executed by the learning device 20.

[0053] 7, the storage unit 22 stores model information 221. The model information 221 is parameters for constructing a model, etc. For example, the model information 221 is weights and biases for constructing each neural network, etc.

[0054] The model information 221 may be updated by the learning device 20 and passed to the extraction device 10. The passed model information 221 is stored as the model information 121 by the extraction device 10.

[0055] The control unit 23 controls the entire learning device 20. The control unit 23 is, for example, an electronic circuit such as a CPU, an MPU, or a GPU, or an integrated circuit such as an ASIC or an FPGA. The control unit 23 also has an internal memory for storing programs that define various processing procedures and control data, and executes each process using the internal memory.

[0056] The control unit 23 functions as various processing units by the operation of various programs. For example, the control unit 23 includes a signal processing unit 231, a loss calculation unit 232, and an update unit 233.

[0057] The signal processing unit 231 extracts the target sound from the mixed sound by using a model constructed from the model information 221. The signal processing unit 231 includes a generation unit 231a and an extraction unit 231b.

[0058] The generating unit 231a generates data in a predetermined format from the mixed sound and activity information indicating the time when the target sound source emitted the sound. The extracting unit 231b extracts the sound of the target sound source from the mixed sound by using the data generated by the generating unit 231a and an extraction network.

[0059] The generating unit 231a and the extracting unit 231b perform the same processes as the generating unit 131a and the extracting unit 131b, respectively.

[0060] The loss calculation unit 232 calculates a loss function based on the sound extracted by the extraction unit 231b. For example, the loss function is calculated based on the sound ^x t and the correct answer x tThe loss function is not limited to the signal-to-noise ratio, but may be a signal-to-distortion ratio (SDR) and a mean square error (MSE), etc.

[0061] The update unit 233 updates the parameters of the NN for extraction so that the loss function calculated based on the sound extracted by the extraction unit 231b is optimized. The update unit 233 can update the parameters by a known method such as the backpropagation method.

[0062] [Processing flow of the first embodiment] Fig. 8 is a flowchart showing a process flow of the extraction device according to the first embodiment. As shown in Fig. 8, first, the extraction device 10 generates data in a predetermined format from the combined voice and activity information (step S101).

[0063] Next, the extraction device 10 extracts the target voice from the mixed voice using the generated data and the extraction network (step S102).

[0064] Fig. 9 is a flowchart showing the flow of processing performed by the study device according to the first embodiment. As shown in Fig. 9, first, the study device 20 generates data in a predetermined format from the combined voice and activity information (step S201).

[0065] Next, the learning device 20 extracts the target speech from the mixed speech using the generated data and the extraction network (step S202).

[0066] Here, the learning device 20 calculates a loss function that optimizes the network (step S203), and then updates the parameters of the network so that the loss function is optimized (step S204).

[0067] If the learning device 20 determines that the parameters have converged (step S205, Yes), it ends the process. On the other hand, if the learning device 20 determines that the parameters have not converged (step S205, No), it returns to step S201 and repeats the process.

[0068] [Advantages of the first embodiment] As described above, the generation unit 131a generates data in a predetermined format from the mixed sound and activity information indicating the time when the target sound source emitted the sound. The extraction unit 131b extracts the sound of the target sound source from the mixed sound by using the data generated by the generation unit 131a and the extraction network.

[0069] In this way, the extraction device 10 extracts the target sound by using the activity information. Therefore, for example, in the extraction process by the extraction device 10, it is not necessary to register the target speaker's voice in advance. Furthermore, the extraction process by the extraction device 10 can reduce the influence of changes in the characteristics of the target speaker's voice. As a result, according to this embodiment, the target sound can be extracted from the mixed sound with high accuracy and ease.

[0070] Conventionally, there have been various types of data that serve as clues for speaker separation. For example, about 10 seconds of the speaker's voice, a video of the speaker's face, a combination of voice and video, etc. are used as clues. It is necessary to prepare a model for each type.

[0071] In contrast, in this embodiment, it is sufficient to prepare a model corresponding to the activity information, regardless of the format of the clue data.

[0072] The generation unit 131a generates a weighted sum of the output obtained by inputting the mixed sound to the auxiliary network and the activity information. The extraction unit 131b inputs the mixed sound and the weighted sum to an extraction network, thereby extracting the sound of the target sound source from the mixed sound. This method corresponds to the ADEnet-auxiliary in FIG. 4.

[0073] ADEnet-auxiliary can achieve the same performance as removing the discrepancy between the mixed voice in the speaker beam and the voice of the target speaker registered in advance, especially if the overlap can be eliminated.

[0074] The generating unit 131a generates a combined vector by combining the activity information and the mixed sound, both of which are represented by vectors. The extracting unit 131b inputs the combined vector to an extraction network to extract the sound of the target sound source from the mixed sound. This method corresponds to the ADEnet-input in FIG. 5.

[0075] Since ADEnet-input does not require an auxiliary network, it can achieve a simpler configuration than ADEnet-auxiliary.

[0076] The generator 131a further generates a weighted sum of the output obtained by inputting the mixed sound to the auxiliary network and the activity information. The extractor 131b inputs the combined vector and the weighted sum to the extraction network to extract the sound of the target sound source from the mixed sound. This method corresponds to ADEnet-mix in FIG. 6.

[0077] ADEnet-mix can reflect the features of both the joint vector and the weighted sum in the extraction results.

[0078] [Experimental Results] An experiment using this embodiment will be described. In the experiment, activity information for learning was prepared by adding noise to the oracle speaker activity estimated based on the teacher data. The oracle speaker activity is the activity of the target voice estimated (extracted) from the correct voice data (teacher data) using a voice section extraction method (for example, Reference 6). Reference 6: “https: / / github.com / wiseman / py-webrtcvad”

[0079] Specifically, we removed overlapping time intervals from the oracle speaker activity, and modified the beginning and end of each time interval in which the target speaker was active with values ​​sampled from the range -1 s to 1 s.

[0080] First, the comparison results of SDR (signal to distortion ratio) when using oracle speaker activity as is and when noise is added are shown in Figure 10. Figure 10 shows the experimental results.

[0081] Noisy activity training indicates whether noise was added to the training data. Activity signal at test time is the SDR depending on whether overlap was present during extraction and whether noise was added (Oracle or +Noise).

[0082] As shown in Fig. 10, some models of this embodiment may exceed the SDR of the speaker beam. The SDR of the speaker beam is 9.4. In addition, the LibriSpeech corpus (V. Panayotov, G. Chen, D. Povey, and S. Khudanpur, "Librispeech: an asr corpus based on public domain audio books," in Proc. of ICASSP'15, 2015, pp. 5206-5210.) is used in the experiment in Fig. 10.

[0083] In addition, Figure 10 shows that adding noise to the training data (checking Noisy activity training) improves the robustness of the model.

[0084] Next, the results of an experiment using data simulating a meeting situation (Z. Chen, T. Yoshioka, L. Lu, T. Zhou, Z. Meng, Y. Luo, J. Wu, and J. Li, “Continuous speech separation: Dataset and analysis,” Proc. of ICASSP'20, pp. 7284-7288, 2020.) are shown in Figure 11. Figure 11 shows the experimental results.

[0085] In the experiment in Fig. 11, activity information from TS-VAD described in Reference 5 was used. Fig. 11 also shows the concatenated minimum-permutation word error rate (cpWER), i.e., an evaluation value including errors due to diarization.

[0086] From Fig. 11, we can see that as the overlap ratio increases, adopting ADEnet is more advantageous than not adopting ADEnet.

[0087] [System configuration, etc.] In addition, each component of each device shown in the figure is a functional concept, and does not necessarily have to be physically configured as shown in the figure. In other words, the specific form of distribution and integration of each device is not limited to that shown in the figure, and all or a part of it can be functionally or physically distributed and integrated in any unit according to various loads, usage conditions, etc. Furthermore, each processing function performed by each device can be realized in whole or in any part by a CPU (Central Processing Unit) and a program analyzed and executed by the CPU, or can be realized as hardware using wired logic.

[0088] Furthermore, among the processes described in this embodiment, all or part of the processes described as being performed automatically can be performed manually, or all or part of the processes described as being performed manually can be performed automatically by a known method. In addition, the information including the processing procedures, control procedures, specific names, various data and parameters shown in the above documents and drawings can be changed arbitrarily unless otherwise specified.

[0089] [program] As an embodiment, the extraction device 10 and the learning device 20 can be implemented by installing a program for executing the above-mentioned extraction process or learning process of the voice signal as package software or online software on a desired computer. For example, by having an information processing device execute a program for the above-mentioned extraction process, the information processing device can function as the extraction device 10. The information processing device here includes desktop and notebook personal computers. In addition, the information processing device also includes mobile communication terminals such as smartphones, mobile phones, and PHS (Personal Handyphone System), and even slate terminals such as PDA (Personal Digital Assistant), etc.

[0090] The extraction device 10 and the learning device 20 can also be implemented as a server device that provides a service related to the extraction process or learning process of the above-mentioned voice signal to a client, the client being a terminal device used by a user. For example, the server device is implemented as a server device that receives a mixed voice signal as an input and provides a service of extracting a voice signal of a target speaker. In this case, the server device may be implemented as a Web server or as a cloud that provides a service by outsourcing.

[0091] 12 is a diagram showing an example of a computer that executes a program. The computer 1000 includes, for example, a memory 1010 and a CPU 1020. The computer 1000 also includes a hard disk drive interface 1030, a disk drive interface 1040, a serial port interface 1050, a video adapter 1060, and a network interface 1070. These components are connected by a bus 1080.

[0092] The memory 1010 includes a ROM (Read Only Memory) 1011 and a RAM 1012. The ROM 1011 stores a boot program such as a BIOS (Basic Input Output System). The hard disk drive interface 1030 is connected to a hard disk drive 1090. The disk drive interface 1040 is connected to a disk drive 1100. A removable storage medium such as a magnetic disk or an optical disk is inserted into the disk drive 1100. The serial port interface 1050 is connected to a mouse 1110 and a keyboard 1120, for example. The video adapter 1060 is connected to a display 1130, for example.

[0093] The hard disk drive 1090 stores, for example, an OS 1091, an application program 1092, a program module 1093, and program data 1094. That is, a program that defines each process of the extraction device 10 is implemented as a program module 1093 in which computer-executable code is written. The program module 1093 is stored, for example, in the hard disk drive 1090. For example, a program module 1093 for executing the same process as the functional configuration of the extraction device 10 is stored in the hard disk drive 1090. The hard disk drive 1090 may be replaced with an SSD.

[0094] Furthermore, setting data used in the processing of the above-described embodiment is stored as program data 1094, for example, in the memory 1010 or the hard disk drive 1090. Then, the CPU 1020 reads out the program module 1093 or the program data 1094 stored in the memory 1010 or the hard disk drive 1090 into the RAM 1012 as necessary, and executes them.

[0095] Note that the program module 1093 and the program data 1094 are not limited to being stored in the hard disk drive 1090, but may be stored in, for example, a removable storage medium and read by the CPU 1020 via the disk drive 1100 or the like. Alternatively, the program module 1093 and the program data 1094 may be stored in another computer connected via a network (such as a local area network (LAN) or wide area network (WAN)). The program module 1093 and the program data 1094 may then be read by the CPU 1020 from the other computer via the network interface 1070. [Explanation of symbols]

[0096] 10 Extraction device 20 Learning Device 11, 21 Interface section 12, 22 Storage section 13, 23 Control section 121, 221 Model Information 131, 231 Signal processing unit 131a, 231a generation section 131b, 231b extraction part

Claims

1. A generation unit that generates a weighted sum of an output obtained by inputting a mixed sound into an auxiliary NN (neural network) and activity information indicating a time when a target sound source emitted a sound; an extraction unit that extracts the sound of the target sound source from the mixed sound by inputting the mixed sound and the weighted sum to an extraction NN; An extraction apparatus comprising:

2. A generation unit that generates a combined vector by combining activity information, which is information indicating a time when a target sound source emitted a sound, and a mixed sound, which is also represented by a vector; an extraction unit that extracts the sound of the target sound source from the mixed sound by inputting the combined vector to an extraction neural network (NN); An extraction apparatus comprising:

3. The generation unit further generates a weighted sum of an output obtained by inputting the mixed sound to an auxiliary neural network and the activity information; 3. The extraction device according to claim 2, wherein the extraction unit extracts the sound of the target sound source from the mixed sound by inputting the combined vector and the weighted sum to the NN for extraction.

4. A method of extraction performed by an extraction device, comprising: a generation step of generating a weighted sum of an output obtained by inputting the mixed sound into an auxiliary NN (neural network) and activity information indicating a time when the target sound source emitted a sound; an extraction step of extracting the sound of the target sound source from the mixed sound by inputting the mixed sound and the weighted sum to an extraction NN; An extraction method comprising the steps of:

5. An extraction method performed by an extraction device, comprising: a generating step of generating a combined vector by combining activity information, which is information indicating a time when a target sound source emits a sound and is represented by a vector, and a mixed sound, which is represented by a vector; an extraction step of extracting the sound of the target sound source from the mixed sound by inputting the combined vector to an extraction neural network (NN); An extraction method comprising the steps of:

6. A generation unit that generates a weighted sum of an output obtained by inputting the mixed sound into an auxiliary NN (neural network) and activity information indicating a time when the target sound source emitted a sound; an extraction unit that extracts the sound of the target sound source from the mixed sound by inputting the mixed sound and the weighted sum to an extraction NN; an update unit that updates parameters of the NN for extraction so that a loss function calculated based on the sound extracted by the extraction unit is optimized; A learning device comprising:

7. A generation unit that generates a combined vector by combining activity information, which is information representing a time when a target sound source emitted a sound, and a mixed sound, which is represented by a vector; an extraction unit that extracts the sound of the target sound source from the mixed sound by inputting the combined vector to an extraction neural network (NN); an update unit that updates parameters of the NN for extraction so that a loss function calculated based on the sound extracted by the extraction unit is optimized; A learning device comprising:

8. A learning method performed by a learning device, comprising: a generation step of generating a weighted sum of an output obtained by inputting the mixed sound into an auxiliary NN (neural network) and activity information indicating a time when the target sound source emitted a sound; an extraction step of extracting the sound of the target sound source from the mixed sound by inputting the mixed sound and the weighted sum to an extraction NN; an updating step of updating parameters of the NN for extraction so that a loss function calculated based on the sounds extracted by the extraction step is optimized; A learning method comprising:

9. A learning method executed by a learning device, comprising: a generating step of generating a combined vector by combining activity information, which is information indicating a time when a target sound source emits a sound and is represented by a vector, and a mixed sound, which is represented by a vector; an extraction step of extracting the sound of the target sound source from the mixed sound by inputting the combined vector to an extraction neural network (NN); an updating step of updating parameters of the NN for extraction so that a loss function calculated based on the sounds extracted by the extraction step is optimized; A learning method comprising:

10. A program for causing a computer to function as the extraction device according to any one of claims 1 to 3 or the learning device according to claim 6 or 7.

Images