Knowledge distillation device and target sound extraction device, method for same, and program

Abstract

This knowledge distillation device distills knowledge to obtain a student model through a learning process using training data based on a teacher model. The teacher model performs inference processing on the inputs of a time-series first mixed signal in which a plurality of acoustic signals are mixed and first correspondence information corresponding to a specific first segment, to obtain and output time-series first extraction information corresponding to an acoustic signal belonging to the first segment from the first mixed signal. The student model performs inference processing on the inputs of a time-series second mixed signal in which a plurality of acoustic signals are mixed and second correspondence information corresponding to a specific second segment, to obtain and output time-series second extraction information corresponding to an acoustic signal belonging to the second segment from the second mixed signal. The inference processing of the teacher model includes non-causal processing in which first input information based on the first mixed signal up to the present and in the future is used to obtain first output information corresponding to the first extraction information at present. The inference processing of the student model includes causal processing in which second input information based on the second mixed signal up to the present is used, and information based on the second mixed signal in the future is not used, to obtain second output information corresponding to the second extraction information at present.

Description

Knowledge distillation device, target sound extraction device, their methods, and programs

[0001] The present invention relates to the field of machine learning, and in particular to knowledge distillation techniques.

[0002] A technique is known that uses a deep neural network (DNN) to extract an acoustic signal belonging to a specific sound event class from a mixed signal that is a mixture of acoustic signals emitted from multiple sound sources (see, for example, Non-Patent Document 1). This technique is called target sound extraction (TSE).

[0003] M. Delcroix et al., "SoundBeam: Target Sound Extraction Conditioned on Sound-Class Labels and Enrollment Clues for Increased Performance and Continuous Learning", IEEE / ACM TASLP, vol.31, pp. 121-136, 2023.

[0004] In the inference process for target sound extraction, higher extraction performance can be expected by performing acausal processing in which current information is calculated based on not only previous information but also future information, rather than causal processing in which current information is calculated using only previous information (current information, or current and past information). However, using future information for the inference process causes delays in the process. Therefore, in applications where this delay is a problem, it is desirable to perform the inference process without performing acausal processing. On the other hand, high extraction performance cannot be obtained even if a target sound extraction model that performs inference processing without acausal processing is trained using normal training data.

[0005] In the present invention, knowledge distillation is performed to obtain a student model through a learning process using training data based on a teacher model. Here, the teacher model performs inference processing to receive input of a first mixed signal in a time series obtained by mixing a plurality of acoustic signals and first correspondence information corresponding to a specific first segment, and obtain from the first mixed signal time-series first extracted information corresponding to the acoustic signal belonging to the first segment, and output the information. The student model performs inference processing to receive input of a second mixed signal in a time series obtained by mixing a plurality of acoustic signals and second correspondence information corresponding to a specific second segment, and obtain from the second mixed signal time-series second extracted information corresponding to the acoustic signal belonging to the second segment, and output the information. The inference processing of the teacher model includes a non-causal process to obtain first output information corresponding to the current first extracted information using first input information based on the first mixed signal before and after the present and in the future. The inference processing of the student model includes a causal process to obtain second output information corresponding to the current second extracted information using second input information based on the second mixed signal before the present, without using information based on the second mixed signal in the future.

[0006] By using a student model that has been knowledge-distilled based on a teacher model, it is possible to achieve target sound extraction with low delay and high extraction performance.

[0007] FIG. 1 is a block diagram illustrating the configuration of an object sound extraction system according to each embodiment. FIG. 2 is a block diagram illustrating the functional configurations of a teacher model device and a knowledge distillation device according to each embodiment. FIG. 3 is a block diagram illustrating the functional configuration of an object sound extraction device according to each embodiment. FIG. 4 is a diagram illustrating the relationship between a teacher model and a student model in knowledge distillation. FIG. 5A is a block diagram illustrating the configuration of the main layer of the teacher model. FIG. 5B is a block diagram illustrating the configuration of the main layer of the student model. FIGS. 6A and 6B are conceptual diagrams illustrating the configuration of the main layer of the teacher model. FIG. 6C is a conceptual diagram illustrating the configuration of the main layer of the student model. FIG. 7 is a block diagram illustrating the configuration of a mixing block. FIG. 8 is a block diagram illustrating the configuration of a target block. FIG. 9 is a block diagram illustrating the configuration of a 1D convolution block. FIG. 10 is a block diagram illustrating the hardware configuration of each device in the embodiments.

[0008] Hereinafter, embodiments of the present invention will be described with reference to the drawings. [First Embodiment] First, a first embodiment of the present invention will be described. <Configuration> As illustrated in Fig. 1, an object sound extraction system 1 of this embodiment has a teacher model device 11, a knowledge distillation device 12, and an object sound extraction device 13. The teacher model device 11, the knowledge distillation device 12, and the object sound extraction device 13 are configured to be able to communicate with each other via a network, for example.

[0009] As illustrated in Fig. 2, the teacher model device 11 of this embodiment has a storage unit 111, an input unit 112, an inference processing unit (model application unit) 113, and an output unit 114. The knowledge distillation device 12 of this embodiment has a learning data setting unit 120, a storage unit 121, an input unit 122, a knowledge distillation unit 123, and an output unit 124. The knowledge distillation unit 123 has a parameter transfer unit 1231 and a teacher-student learning unit 1232. As illustrated in Fig. 3, the object sound extraction device 13 of this embodiment has a storage unit 131, an input unit 132, an inference processing unit 133, and an output unit 134.

[0010] <Processing> In this embodiment, the knowledge distillation unit 123 of the knowledge distillation device 12 obtains a student model through a learning process using learning data based on a teacher model. This process will be described in detail below.

[0011] <Teacher model> Teacher model TSE ｔｅａｃｈｅｒ is a model that extracts a target sound by inference processing. That is, the teacher model TSE ｔｅａｃｈｅｒ is a set of multiple acoustic signals x ｎ ∈R Ｋ A time series mixed signal (first mixed signal) y∈R Ｋ and specific division (first division) s ｎ ∈s={s １ , ..., s Ｎ} corresponding to the correspondence information (first correspondence information) o ｓ For the input of ｓ Acoustic signal x belonging to ｎ Extracted information (first extracted information) ^x of the time series corresponding to ｓ ｔｅａｃｈｅｒ The inference process is performed to obtain and output ^x from the mixed signal y. ｓｔｅａｃｈｅｒ = TSE ｔｅａｃｈｅｒ (y, o ｓ ) This teacher model TSE ｔｅａｃｈｅｒ The inference process is performed using a mixed signal (first mixed signal) y∈R Ｋ The current extracted information (first extracted information) ^x is calculated using the input information (first input information) based on ｓ ｔｅａｃｈｅｒ The non-causal process includes obtaining output information (first output information) corresponding to the above.

[0012] Teacher Model TSE ｔｅａｃｈｅｒ may be, for example, a model based on a Deep Neural Network (DNN) such as a Convolutional Neural Network (CNN), a model based on a Hidden Markov Model (HMM), or other machine learning models.

[0013] mixed signal y∈R Ｋ may be a signal obtained by mixing a plurality of acoustic signals in a space, or may be a signal obtained by mixing a plurality of acoustic signals by acoustic processing. ｎ represents the section to which the acoustic information belongs. ｎ For example, the specific category s may correspond to a sound source that emitted the acoustic signal, a type of acoustic information, a volume of acoustic information, or other attributes of acoustic information. ｎ may be a single division or multiple divisions. The mixed signal y is expressed, for example, as follows: Here, x ｎ ∈R Ｋ is division s ｎ represents an acoustic signal belonging to Ｋ represents other background noise. ｎ Acoustic signal x belonging to ｎ If x does not exist, ｎ is zero. R represents a real number, K is a positive integer constant representing the number of samples of the acoustic signal x (the number of predetermined acoustic signals x), and R Ｋrepresents K real numbers, n∈{1,...,N}, where N is the partition s ｎ is a positive integer representing the number of audio signals x ｎ Examples of the signal y are time-series signals (discrete signals) such as speech, music, and environmental sounds. ｎ is the time domain signal.

[0014] Specific categories ｎ Corresponding information o ｓ is a specific division s ｎ or specific categories ｎ Acoustic signal x belonging to ｎ This is information for identifying a specific category s. ｎ may be singular or plural. ｓ is, for example, a specific segment s ｎ Acoustic signal x belonging to ｎ The waveform may be information representing a specific section s ｎ Acoustic signal x belonging to ｎ It may be information representing the feature quantity of a specific section s ｎ It may be a label (class label) representing a specific division s ｎ It may be a natural language query expressing a particular segment s ｎ The correspondence information o may be information that serves as a clue for identifying the correspondence information o. ｓ Examples of the specific division s are described in Non-Patent Document 1. ｎ The corresponding information o is a label representing ｓ is a specific division s ｎ The nth element o corresponding to ｓ ｎ is 1, and other elements o ｓ ｍ N-dimensional vector o with 0 ｓ = {o ｓ １ , ..., o ｓ Ｎ}. Here, a specific section s ｎ If is singular, the corresponding information o ｓ is a 1-hot vector, and a particular partition s ｎIn the case where there are multiple (ω pieces, ω is an integer of 2 or more), the correspondence information o ｓ is a ω-hot vector.

[0015] Category s ｎ Acoustic signal x belonging to ｎ Extracted information ^x of the time series corresponding to ｓ ｔｅａｃｈｅｒ is, for example, a specific segment s ｎ or a signal estimated to belong to a specific section s ｎ , or information representing the probability of an acoustic signal belonging to a particular category s ｎ The extracted information ^x may be statistical information of the acoustic signals belonging to the ｓ ｔｅａｃｈｅｒ is the time domain signal.

[0016] Figure 4 shows the teacher model TSE. ｔｅａｃｈｅｒ The teacher model TSE shown in FIG. ｔｅａｃｈｅｒ has an encoding (encoder) layer 1110 , a corresponding information encoding layer 1120 , a main layer 1130 , a Hadamard stack 1140 , and a decoding (decoder) layer 1150 .

[0017] <Encoding Layer 1110> The encoding layer 1110 encodes the input mixed signal y∈R Ｋ Encode the mixed signal Y ｔｅａｃｈｅｒ ∈R Ｄ×Ｋ’ This layer converts the data into Y and outputs it. The relationship is shown below. teacher =Encoder(y) where D is a positive integer representing the number of embedding dimensions, K' is a positive integer representing the length of the time frame of the signal in the coding domain (a predetermined time interval in the coding domain), and R Ｄ×Ｋ’represents D×K' real numbers. Encoder(α1) is a layer that converts the time-domain signal α1 into a coding-domain signal. A specific example of Encoder(α1) is the 1-D convolution layer described in Reference 1. Alternatively, the coding-domain signal may be a time-frequency domain signal. Reference 1: Y. Luo and N. Mesgarani, "Conv-TasNet: Surpassing ideal time-frequency magnitude masking for speech separation," IEEE / ACM Trans.ASLP, vol. 27, no. 8, pp. 1256-1266, 2019.

[0018] The correspondence information encoding layer 1120 encodes the input correspondence information o ｓ The encoding correspondence information e ｓ ｔｅａｃｈｅｒ ∈R Ｄ This layer converts the input signal into a vector and outputs it. The relationship is shown below. e s teacher =Encoder c (o s ) Encoder ｃ (α1) is, for example, a layer that converts the time domain signal α1 into a coding domain signal. ｃ Specific examples of (α1) include the 1-hot encoder and Enrl encoder described in Reference 1, and combinations thereof. ｓ is an N-dimensional vector such as a 1-hot encoder, the correspondence information encoding layer 1120 encodes the correspondence information o ｓ The encoding correspondence information e ｓ ｔｅａｃｈｅｒ ∈R Ｄ The relationship is shown below. e s teacher =Wo s where W∈R Ｄ×Ｎ represents an embedding matrix. A specific example of the embedding matrix W is described in, for example, Non-Patent Document 1.

[0019] <Main Layer 1130> The main layer 1130 receives the encoded mixed signal Yｔｅａｃｈｅｒ and encoding correspondence information e ｓ ｔｅａｃｈｅｒ On the other hand, the encoding correspondence information e ｓ ｔｅａｃｈｅｒ Extracted information ^x of the time series corresponding to ｓ ｔｅａｃｈｅｒ To obtain the target mask M ｓ ｔｅａｃｈｅｒ Outputs M s teacher =MainBlock(Y teacher ,e s teacher )

[0020] As illustrated in FIG. 5A, the main layer 1130 includes, for example, a mixed block 1131, a fused block 1132, and a target block 1133 (see, for example, Reference 1).

[0021] The mixing block 1131 receives the coded mixed signal Y ｔｅａｃｈｅｒ the internal mixed signal Z ｔｅａｃｈｅｒ ∈R Ｄ×Ｋ’ This layer converts the input data into a vector and outputs it. The relationship is shown below. Z teacher =MixBlock(Y teacher )

[0022] The fusion block 1132 receives the internally mixed signal Z ｔｅａｃｈｅｒ and encoding correspondence information e ｓ ｔｅａｃｈｅｒ The internal target signal Z ｓ ｔｅａｃｈｅｒ ∈R Ｄ×Ｋ’ This layer converts the input data into a vector and outputs it. The relationship is shown below. Z s teacher =Adapt(Z teacher ,e s teacher )

[0023] The target block 1133 receives the internal target signal Z ｓ ｔｅａｃｈｅｒ The target mask M ｓ ｔｅａｃｈｅｒ ∈R Ｄ×Ｋ’ This layer converts the input signal into a vector and outputs it. The relationship is shown below. M s teacher =TgtBlock(Z s teacher )

[0024] <Hadamard Stack 1140> Returning to FIG. 4, the Hadamard stack 1140 is a multi-layer stack of the input target mask M ｓ ｔｅａｃｈｅｒ and the encoded mixed signal Y ｔｅａｃｈｅｒ The Hadamard product M ｓ ｔｅａｃｈｅｒ ○Y ｔｅａｃｈｅｒ Here, α1○α2 represents the Hadamard product of α1 and α2 (the product of α1 and α2 for each element).

[0025] <Decoding Layer 1150> The decoding layer 1150 receives the Hadamard product M ｓ ｔｅａｃｈｅｒ ○Y ｔｅａｃｈｅｒ Extract information ^x ｓ ｔｅａｃｈｅｒ This layer converts the input data into the output data. The relationship is shown below. ^x s teacher =Decoder(M s teacher ○Y teacher The decoder (α2) is a layer that converts the coding domain signal α2 into a time domain signal. A specific example of the decoder (α2) is the 1-D convolution layer described in Reference 1.

[0026] <Teacher model TSE ｔｅａｃｈｅｒ Inference process of the teacher model TSE ｔｅａｃｈｅｒ The inference process is to derive current extracted information (first extracted information) ^x using input information (first input information) based on the mixed signal (first mixed signal) y before and after the present and in the future. ｓ ｔｅａｃｈｅｒ That is, the teacher model TSE includes a non-causal process for obtaining output information (first output information) corresponding to the teacher model TSE. ｔｅａｃｈｅｒ is the input mixed signal y and the corresponding information o ｓ Extract information from ^x ｓ ｔｅａｃｈｅｒA non-causal process is performed somewhere in the process of obtaining the mixture signal y. The mixture signal before the present means the present mixture signal and / or the past mixture signal. The present mixture signal means, for example, the mixture signal in the coding domain in the current time frame. The past mixture signal means, for example, the mixture signal in the coding domain in a time frame earlier than the current time frame. The future mixture signal means, for example, the mixture signal in the coding domain in a time frame later than the current time frame. The input information and the output information are the input mixture signal y and the corresponding information o ｓ Extract information from ^x ｓ ｔｅａｃｈｅｒ The input information and the output information are, for example, signals in the coding domain. ｔｅａｃｈｅｒ The inference process may include a single acausal process or multiple acausal processes.

[0027] Teacher Model TSE ｔｅａｃｈｅｒ In addition to the non-causal processing, the inference process of (a) further uses input information (third input information) based on the mixed signal (first mixed signal) y before the present to derive the current extracted information (first extracted information) ^x without using information based on the future mixed signal (first mixed signal) y. ｓ ｔｅａｃｈｅｒ That is, the teacher model TSE may include a causal process for obtaining output information (third output information) corresponding to the teacher model TSE. ｔｅａｃｈｅｒ is the input mixed signal y and the corresponding information o ｓ Extract information from ^x ｓ ｔｅａｃｈｅｒ In other words, the teacher model TSE may be obtained by performing non-causal processing and causal processing at any point in the process. ｔｅａｃｈｅｒ is the input mixed signal y and the corresponding information o ｓ Extract information from ^x ｓ ｔｅａｃｈｅｒ It is also possible to perform non-causal processing in part of the process to obtain the teacher model TSE, and perform causal processing in another part. ｔｅａｃｈｅｒ The inference process may include a single causal process or multiple causal processes.

[0028] For example, the main layer 1130 illustrated in Figures 4 and 5 may include an acausal block that performs acausal processing (Figure 6A), or may include an acausal block that performs acausal processing and a causal block that performs causal processing (Figure 6B). For example, the mixed block 1131 (Figure 5A) may include one or more acausal blocks, or may include one or more acausal blocks and one or more causal blocks. The target block 1133 (Figure 5A) may include one or more acausal blocks, or may include one or more acausal blocks and one or more causal blocks.

[0029] Teacher Model TSE ｔｅａｃｈｅｒ The non-causal process included in the inference process is, for example, convolution of input information (first input information) based on a mixed signal (first mixed signal) y before the present and in the future to obtain current extracted information (first extracted information) ^x ｓ ｔｅａｃｈｅｒ For example, the non-causal processing may include a convolution process in a convolutional neural network that convolves input information of a coding domain in pre-present and future time frames based on a pre-present and future mixed signal y to obtain output information of the coding domain in the current time frame. For example, a convolution process in which the receptive field of a convolution filter that performs the convolution includes future time frames is a non-causal process.

[0030] Teacher Model TSE ｔｅａｃｈｅｒ The non-causal process included in the inference process is, for example, a process of statistically processing input information (first input information) based on a mixed signal (first mixed signal) y before the present and in the future to obtain current extracted information (first extracted information) ^x ｓ ｔｅａｃｈｅｒ For example, the non-causal processing may include a normalization process for performing statistical processing on input information of the coding domain in pre-present and future time frames based on the pre-present and future mixed signal y to obtain output information of the coding domain in the current time frame. A specific example of the statistical processing is processing for calculating a variance or an average.

[0031] Teacher Model TSE ｔｅａｃｈｅｒ The causal process included in the inference process is, for example, to perform convolution of input information (third input information) based on the mixed signal (first mixed signal) y before the present without using information based on the future mixed signal (first mixed signal) y, and obtain the current extracted information (first extracted information) ^x ｓ ｔｅａｃｈｅｒ For example, this causal processing may include a convolution processing in a convolutional neural network that performs convolution of input information of the coding domain in a time frame before the present based on the mixed signal y before the present to obtain output information of the coding domain in the current time frame. For example, a convolution processing in which the receptive field of the convolution filter that performs the convolution does not include future time frames is a causal processing.

[0032] Teacher Model TSE ｔｅａｃｈｅｒ The causal process included in the inference process is, for example, to perform statistical processing of input information (first input information) based on the mixed signal (first mixed signal) y before the present, without using information based on the future mixed signal (first mixed signal) y, to obtain the current extracted information (first extracted information) ^x ｓ ｔｅａｃｈｅｒ For example, the non-causal processing may include a normalization process that performs statistical processing on input information in the coding domain in pre-present and future time frames based on the pre-present mixed signal y to obtain output information in the coding domain in the current time frame.

[0033] <Student Model> Student Model TSE ｓｔｕｄｅｎｔ is a model that extracts the target sound by inference processing. That is, the student model TSE ｓｔｕｄｅｎｔ is a set of multiple acoustic signals x ｎ ∈R Ｋ A time series mixed signal (second mixed signal) y∈R Ｋ and specific divisions (second divisions) ｎ ∈s={s １ , ..., s Ｎ} corresponding to the corresponding information (second corresponding information) o ｓ For the input ofｓ Acoustic signal x belonging to ｎ Extracted information (second extracted information) ^x of the time series corresponding to ｓ ｓｔｕｄｅｎｔ The inference process is performed to obtain and output ^x from the mixed signal y. ｓ ｓｔｕｄｅｎｔ = TSE ｓｔｕｄｅｎｔ (y, o ｓ ) This student model TSE ｓｔｕｄｅｎｔ The inference process is performed by using a future mixed signal (second mixed signal) y∈R Ｋ Without using information based on Ｋ The current extracted information (second extracted information) ^x is calculated using the input information (second input information) based on ｓ ｓｔｕｄｅｎｔ The process includes a causal process for obtaining output information (second output information) corresponding to the above.

[0034] Student Model TSE ｓｔｕｄｅｎｔ For example, the model may be based on a Deep Neural Network (DNN) such as a Convolutional Neural Network (CNN), a model based on a Hidden Markov Model (HMM), or other machine learning models.

[0035] Figure 4 shows the student model TSE. ｓｔｕｄｅｎｔ The student model TSE illustrated in FIG. ｓｔｕｄｅｎｔ has an encoding (encoder) layer 1210 , a corresponding information encoding layer 1220 , a main layer 1230 , a Hadamard stack 1240 , and a decoding (decoder) layer 1250 .

[0036] <Encoding Layer 1210> The encoding layer 1210 encodes the input mixed signal y∈R Ｋ Encode the mixed signal Y ｓｔｕｄｅｎｔ ∈R Ｄ×Ｋ’ This layer converts the data into Y and outputs it. The relationship is shown below. student =Encoder(y)

[0037] The correspondence information encoding layer 1220 encodes the input correspondence information o ｓ The encoding correspondence information e ｓ ｓｔｕｄｅｎｔ ∈R Ｄ This layer converts the input signal into a vector and outputs it. The relationship is shown below. es student =Encoder c (o s ) Encoder ｃ Specific examples of the correspondence information o include the 1-hot encoder and the Enrl encoder described in Reference 1, and a combination thereof. ｓ is an N-dimensional vector such as a 1-hot encoder, the correspondence information encoding layer 1220 encodes the correspondence information o ｓ The encoding correspondence information e ｓ ｓｔｕｄｅｎｔ ∈R Ｄ The relationship is shown below. e s student =Wo s

[0038] <Main Layer 1230> The main layer 1230 receives the encoded mixed signal Y ｓｔｕｄｅｎｔ and encoding correspondence information e ｓ ｓｔｕｄｅｎｔ On the other hand, the encoding correspondence information e ｓ ｓｔｕｄｅｎｔ Extracted information ^x of the time series corresponding to ｓ ｓｔｕｄｅｎｔ To obtain the target mask M ｓ ｓｔｕｄｅｎｔ Outputs M s student =MainBlock(Y student ,e s student )

[0039] As illustrated in FIG. 5B, the main layer 1230 includes, for example, a mixed block 1231, a fused block 1232, and a target block 1233 (see, for example, Reference 1).

[0040] The mixing block 1231 receives the coded mixed signal Y ｓｔｕｄｅｎｔ the internal mixed signal Z ｓｔｕｄｅｎｔ ∈R Ｄ×Ｋ’ This layer converts the input data into a vector and outputs it. The relationship is shown below. Z student =MixBlock(Y student )

[0041] The fusion block 1232 receives the internally mixed signal Zｓｔｕｄｅｎｔ and encoding correspondence information e ｓ ｓｔｕｄｅｎｔ The internal target signal Z ｓ ｓｔｕｄｅｎｔ ∈R Ｄ×Ｋ’ This layer converts the input data into a vector and outputs it. The relationship is shown below. Z s student =Adapt(Z student ,e s student )

[0042] The target block 1233 receives the internal target signal Z ｓ ｓｔｕｄｅｎｔ The target mask M ｓ ｓｔｕｄｅｎｔ ∈R Ｄ×Ｋ’ This layer converts the input signal into a vector and outputs it. The relationship is shown below. M s student =TgtBlock(Z s student )

[0043] <Hadamard Stack 1240> Returning to FIG. 4, the Hadamard stack 1240 is a multi-layer stack of the input target mask M ｓ ｓｔｕｄｅｎｔ and the encoded mixed signal Y ｓｔｕｄｅｎｔ The Hadamard product M ｓ ｓｔｕｄｅｎｔ ○Y ｓｔｕｄｅｎｔ and output it.

[0044] <Decoding Layer 1250> The decoding layer 1250 receives the Hadamard product M ｓ ｓｔｕｄｅｎｔ ○Y ｓｔｕｄｅｎｔ Extract information ^x ｓ ｓｔｕｄｅｎｔ This layer converts the input data into the output data. The relationship is shown below. ^x s student =Decoder(M s student ○Y student )

[0045] <Student Model TSE ｓｔｕｄｅｎｔ Inference process of the student model TSE ｓｔｕｄｅｎｔThe inference process of (a) uses input information (second input information) based on the mixed signal (second mixed signal) y before the present to derive the current extracted information (second extracted information) ^x without using information based on the future mixed signal (second mixed signal) y. ｓ ｓｔｕｄｅｎｔ That is, the student model TSE includes a causal process for obtaining output information (second output information) corresponding to the ｓｔｕｄｅｎｔ is the input mixed signal y and the corresponding information o ｓ Extract information from ^x ｓ ｓｔｕｄｅｎｔ Causal processing is performed somewhere in the process to obtain the student model TSE. ｓｔｕｄｅｎｔ The inference process may include a single causal process or multiple causal processes. For example, the main layer 1230 illustrated in FIGS. 4 and 5 includes a causal block that performs causal processes (FIG. 6C). For example, the mixed block 1231 (FIG. 5B) may include a single or multiple causal blocks. The target block 1233 (FIG. 5B) may include a single or multiple causal blocks. However, the student model TSE of this embodiment ｓｔｕｄｅｎｔ The inferential process does not involve non-causal processes.

[0046] Student Model TSE ｓｔｕｄｅｎｔ The causal process included in the inference process is, for example, to perform convolution of input information (second input information) based on the mixed signal (second mixed signal) y before the present to obtain current extracted information (second extracted information) ^x without using information based on the future mixed signal (second mixed signal) y. ｓ ｓｔｕｄｅｎｔ For example, this causal processing may include a convolution process in a convolutional neural network that convolves input information of the coding domain in a time frame previous to the current based on the mixed signal y previous to the current to obtain output information of the coding domain in the current time frame.

[0047] Student Model TSE ｓｔｕｄｅｎｔ The causal process included in the inference process is, for example, to perform statistical processing of input information (second input information) based on the mixed signal (second mixed signal) y before the present, without using information based on the future mixed signal (second mixed signal) y, to obtain the current extracted information (second extracted information) ^xｓ ｓｔｕｄｅｎｔ For example, the non-causal processing may include a normalization process that performs statistical processing on input information in the coding domain in pre-present and future time frames based on the pre-present mixed signal y to obtain output information in the coding domain in the current time frame.

[0048] <Pre-processing> As a pre-processing, the learning data (y, s ｎ , x ｓ ) is stored. ｓ , x ｓ ) is a mixed signal y∈R Ｋ , a specific section s ｎ Corresponding information o corresponding to ∈s ｓ , and a particular partition s ｎ Extracted information x of time series corresponding to the acoustic signals belonging to ｓ Furthermore, the storage unit 111 of the teacher model device 11 and the storage unit 121 of the knowledge distillation device 12 store the trained teacher model TSE ｔｅａｃｈｅｒ Information (for example, model parameters) specifying the teacher model TSE is stored. ｔｅａｃｈｅｒ To learn the ｓ , x ｓ ) is used. ｓ , x ｓ The contents of the knowledge distillation device 12 may be entirely the same as those stored in the storage unit 121 of the knowledge distillation device 12, or may be partially or entirely different.

[0049] <Teacher model TSE ｔｅａｃｈｅｒ Pre-processing using the teacher model TSE> The teacher model device 11 (FIG. 2) ｔｅａｃｈｅｒ First, we generate training data for knowledge distillation (KD) using the mixed signal y and the corresponding information o ｓ are input to the input section 112 of the teacher model device 11 and the input section 122 of the knowledge distillation device 12. These mixed signals y and corresponding information o ｓThe contents of the mixed signal y and the corresponding information o may be the same as those stored in the storage unit 121 of the knowledge distillation device 12 in the pre-processing, or may be partially or entirely different. ｓ The contents of the knowledge distillation device 12 are preferably partially or entirely different from those stored in the storage unit 121 of the knowledge distillation device 12 in the pre-processing.

[0050] The mixed signal y and the corresponding information o are input to the input unit 112 of the teacher model device 11. ｓ is input to the inference processing unit 113. The inference processing unit 113 extracts the teacher model TSE ｔｅａｃｈｅｒ The mixed signal y and the corresponding information o ｓ and the teacher model TSE is input. ｔｅａｃｈｅｒ Extracted information ^x output from ｓ ｔｅａｃｈｅｒ The extracted information is output. ｓ ｔｅａｃｈｅｒ is output from the output unit 114 and input to the input unit 122 of the knowledge distillation device 12.

[0051] The input unit 122 of the knowledge distillation device 12 receives the input mixed signal y and the corresponding information o ｓ , and extracted information ^x ｓ ｔｅａｃｈｅｒ to the learning data setting unit 120. The learning data setting unit 120 sets these pairs as learning data (y, o ｓ , ^x ｓ ｔｅａｃｈｅｒ ) and store it in the storage unit 121. By repeating these processes, the learning data (y, o ｓ , ^x ｓ ｔｅａｃｈｅｒ ) is accumulated.

[0052] <Learning process of student model by knowledge distillation> Teacher model TSE ｔｅａｃｈｅｒ In the non-causal process included in the inference process of ｓ ｔｅａｃｈｅｒ In order to obtain output information corresponding to the extracted information ^x, not only input information based on the mixed signal y before the current time but also input information based on the mixed signal y in the future is used. ｓ ｔｅａｃｈｅｒOn the other hand, the teacher model TSE has the advantage of being able to model the extraction accuracy. ｔｅａｃｈｅｒ In the non-causal process included in the inference process of ｓ ｔｅａｃｈｅｒ Since input information based on the future mixed signal y is used to obtain output information corresponding to the future mixed signal y, there is a disadvantage that the processing delay is larger than in causal processing that does not use input information based on the future mixed signal y.

[0053] In contrast, the student model TSE ｓｔｕｄｅｎｔ In the causal process included in the inference process of ｓ ｓｔｕｄｅｎｔ To obtain the output information corresponding to , we do not use input information based on the future mixed signal y. Therefore, we simply use the student model TSE ｓｔｕｄｅｎｔ In the learning, the extracted information ^x ｓ ｓｔｕｄｅｎｔ However, it is difficult to improve the accuracy of the student model TSE. ｓｔｕｄｅｎｔ The causal processing included in the inference processing of has the advantage of having a smaller processing delay than non-causal processing that also uses input information based on the future mixed signal y.

[0054] In this embodiment, in order to combine the advantages of both, a teacher model TSE including acausal processing is used. ｔｅａｃｈｅｒ By using the learning process by knowledge distillation using the learning data based on ｓｔｕｄｅｎｔ That is, the knowledge distillation unit 123 of the knowledge distillation device 12 (FIG. 2) obtains the learning data (y, o ｓ , x ｓ ), (y, o ｓ , ^x ｓ ｔｅａｃｈｅｒ ) and teacher model TSE ｔｅａｃｈｅｒ Using this, learning processing by knowledge distillation is performed to obtain the student model TSE. ｓｔｕｄｅｎｔ (i.e., student model TSE ｓｔｕｄｅｎｔ The model parameters of the student model TSE obtained in this way are then output. ｓｔｕｄｅｎｔ By using this, it is possible to achieve target sound extraction with low delay and high extraction performance.

[0055] The learning process of the student model by knowledge distillation in this embodiment includes, for example, PT (Parameter Transfer) learning performed by the parameter transfer unit 1231 ( FIG. 2 ) and TS (Teacher-student) learning performed by the teacher-student learning unit 1232 ( FIG. 4 ).

[0056] <PT Learning> The PT learning of this embodiment is performed using a teacher model TSE ｔｅａｃｈｅｒ Some of the model parameters are used as the student model TSE ｓｔｕｄｅｎｔ That is, the parameter transfer unit 1231 (FIG. 2) transfers the teacher model TSE stored in the storage unit 121 to the ｔｅａｃｈｅｒ Some of the model parameters are used as the student model TSE ｓｔｕｄｅｎｔ The parameter transfer unit 1231 sets the model parameters of the teacher model TSE, for example. ｔｅａｃｈｅｒ A teacher model TSE that does not affect the non-causal processes included in the inference process ｔｅａｃｈｅｒ The model parameters of the student model TSE ｓｔｕｄｅｎｔ The parameter transfer unit 1231 sets a part of the model parameters of the teacher model TSE that does not affect the non-causal processing, for example. ｔｅａｃｈｅｒ All of the model parameters of the student model TSE ｓｔｕｄｅｎｔ It may be set as part of the model parameters of the teacher model TSE that does not affect non-causal processing. ｔｅａｃｈｅｒ Some of the model parameters are used as the student model TSE ｓｔｕｄｅｎｔ It may be set as part of the model parameters of the teacher model TSE. ｔｅａｃｈｅｒ From Student Model TSE ｓｔｕｄｅｎｔ The model parameters used in the training model TSE are, for example, ｔｅａｃｈｅｒ and student model TSE ｓｔｕｄｅｎｔ For example, as shown in FIG. 4, the parameter transfer unit 1231 transfers the model parameters of the blocks common to the teacher model TSE ｔｅａｃｈｅｒ The model parameters of the encoding layer 1110, the model parameters of the corresponding information encoding layer 1120, and the model parameters of the decoding layer 1150 are respectively defined as the student model TSE ｓｔｕｄｅｎｔThe parameter transfer unit 1231 sets (reuses) the model parameters of the coding layer 1210, the model parameters of the corresponding information coding layer 1220, and the model parameters of the decoding layer 1250. ｔｅａｃｈｅｒ Only a part of the model parameters of the encoding layer 1110, the model parameters of the corresponding information encoding layer 1120, and the model parameters of the decoding layer 1150 are used as the student model TSE. ｓｔｕｄｅｎｔ The parameter transfer unit 1231 may set the model parameters of the coding layer 1210, the model parameters of the corresponding information coding layer 1220, and the model parameters of the decoding layer 1250. ｔｅａｃｈｅｒ In the main layer 1130, all or part of the model parameters of the blocks that do not affect the non-causal processing are used as the student model TSE. ｓｔｕｄｅｎｔ may be set to the model parameters of the corresponding blocks in the main layer 1230.

[0057] <TS Learning> In this embodiment, TS learning is performed using learning data (y, o ｓ , x ｓ ), (y, o ｓ , ^x ｓ ｔｅａｃｈｅｒ ) (training data including the first mixed signal, the first correspondence information, and the first extracted information) to generate a student model TSE ｓｔｕｄｅｎｔ That is, the teacher-student learning unit 1232 (FIG. 2) trains the learning data (y, o ｓ , x ｓ ), (y, o ｓ , ^x ｓ ｔｅａｃｈｅｒ ) as a student model TSE ｓｔｕｄｅｎｔ The loss function L when applied to ＴＳ To make the student model TSE smaller, ｓｔｕｄｅｎｔ The process of updating the model parameters is repeated. The loss function L is ＴＳ Indicates L TS =αL ext (^x s ,x s ,τ)+(1-α)L ext (^x s student ,^x steacher ,τ') where α, τ, and τ' are hyperparameters (e.g., constants). α is a real number that satisfies 0≦α<1, and τ and τ' are 0 or positive real numbers. ^x ｓ is the training data (y, o ｓ , x ｓ ) y, o ｓ Student model TSE ｓｔｕｄｅｎｔ When input into the student model TSE ｓｔｕｄｅｎｔ The extracted information is output from ｓ ｓｔｕｄｅｎｔ is the training data (y, o ｓ , ^x ｓ ｔｅａｃｈｅｒ ) y, o ｓ Student model TSE ｓｔｕｄｅｎｔ When input into the student model TSE ｓｔｕｄｅｎｔ The extracted information is output from the parentheses. Note that the superscript "^" of "^x" should be placed directly above the "x" as shown below. However, in this specification, due to limitations on notation, it may be written as "^x". ext (^x s ,x s ,τ) satisfies the following: L ext (^x s student ,^x s teacher , τ') is also the same. ＴＳ Among them, the following L ＫＤ are the components based on knowledge distillation (Figure 4). KD =(1-α)L ext (^x s student ,^x s teacher ,τ') The smaller the value of the hyperparameter α (the closer α is to 0), the more emphasis is placed on knowledge distillation during learning.

[0058] The knowledge distillation unit 123 (FIG. 2) performs, for example, PT learning and then TS learning. That is, the knowledge distillation unit 123 performs, for example, a teacher model TSE that does not affect non-causal processing. ｔｅａｃｈｅｒ The model parameters of the student model are set as part of the model parameters of the learning data (y, oｓ , x ｓ ), (y, o ｓ , ^x ｓ ｔｅａｃｈｅｒ ) (learning data including the first mixed signal, the first correspondence information, and the first extracted information) to learn the remaining part of the model parameters of the student model. However, this does not limit the present invention. For example, the knowledge distillation unit 123 may perform PT learning after performing TS learning, or may perform only TS learning without performing PT learning. The knowledge distillation unit 123 ( FIG. 2 ) learns the remaining part of the model parameters of the student model using the student model TSE obtained by performing the learning process by knowledge distillation as described above. ｓｔｕｄｅｎｔ The output unit 124 outputs information (for example, model parameters) specifying the student model TSE ｓｔｕｄｅｎｔ The information specifying the student model TSE is sent to the target sound extraction device 13. ｓｔｕｄｅｎｔ The information specifying the target sound is input to the input unit 132 of the target sound extraction device 13 (FIG. 3) and stored in the storage unit 131. ｓｔｕｄｅｎｔ is set in the object sound extraction device 13.

[0059] <Target sound extraction process> In the target sound extraction process, the trained student model TSE ｓｔｕｄｅｎｔ The target sound extraction device 13 (FIG. 3) is set to receive a time-series mixed signal y in which a plurality of acoustic signals are mixed and a specific segment s ｎ Corresponding information o corresponding to ∈s ｓ The target sound extraction device 13 receives the input mixed signal y and the corresponding information o ｓ Student model TSE ｓｔｕｄｅｎｔ Apply to category s ｎ Extracted information ^x of time series corresponding to the acoustic signals belonging to ｓ is obtained from the mixed signal y and output.

[0060] That is, the mixed signal y and the corresponding information o ｓ is input to the input unit 132 and sent to the inference processing unit 133. The inference processing unit 133 receives the trained student model TSE from the storage unit 131. ｓｔｕｄｅｎｔ and extracts the input mixed signal y and the corresponding information o ｓ Student model TSE ｓｔｕｄｅｎｔ Applying to extract information ^xｓ This process is performed, for example, by using the student model TSE ｓｔｕｄｅｎｔ Extracted information ^x of the inference process ｓ ｓｔｕｄｅｎｔ Extract information ^x ｓ is replaced by

[0061] <Features of this embodiment> In this embodiment, the teacher model TSE ｔｅａｃｈｅｒ Learning data (y, o ｓ , ^x ｓ ｔｅａｃｈｅｒ ) by the learning process using the student model TSE ｓｔｕｄｅｎｔ Here, knowledge distillation was performed to obtain the teacher model TSE. ｔｅａｃｈｅｒ is a set of multiple acoustic signals x ｎ ∈R Ｋ A time series mixed signal (first mixed signal) y∈R Ｋ and specific division (first division) s ｎ ∈s={s １ , ..., s Ｎ} corresponding to the correspondence information (first correspondence information) o ｓ For the input of ｓ Acoustic signal x belonging to ｎ Extracted information (first extracted information) ^x of the time series corresponding to ｓ ｔｅａｃｈｅｒ The teacher model TSE is obtained from the mixed signal y and output. ｔｅａｃｈｅｒ The inference process is performed using a mixed signal (first mixed signal) y∈R Ｋ The current extracted information (first extracted information) ^x is calculated using the input information (first input information) based on ｓ ｔｅａｃｈｅｒ The student model TSE includes a non-causal process for obtaining output information (first output information) corresponding to ｓｔｕｄｅｎｔ is a set of multiple acoustic signals x ｎ ∈R Ｋ A time series mixed signal (second mixed signal) y∈R Ｋ and specific divisions (second divisions) ｎ ∈s={s １ , ..., s Ｎ} corresponding to the corresponding information (second corresponding information) o ｓ For the input of ｓ Acoustic signal x belonging toｎ Extracted information (second extracted information) ^x of the time series corresponding to ｓ ｓｔｕｄｅｎｔ The student model TSE is then subjected to inference processing to obtain and output the mixed signal y. ｓｔｕｄｅｎｔ The inference process is performed by using a future mixed signal (second mixed signal) y∈R Ｋ Without using information based on Ｋ The current extracted information (second extracted information) ^x is calculated using the input information (second input information) based on ｓ ｓｔｕｄｅｎｔ In this embodiment, the teacher model TSE including the non-causal process is used to obtain output information (second output information) corresponding to the ｔｅａｃｈｅｒ By using the learning process by knowledge distillation using the learning data based on ｓｔｕｄｅｎｔ We obtain a teacher model TSE that includes non-causal processing with high extraction performance. ｔｅａｃｈｅｒ By performing knowledge distillation using learning data based on ｓｔｕｄｅｎｔ The student model TSE thus obtained can be obtained. ｓｔｕｄｅｎｔ The inference process of is based on causal processing that does not use future information, so the delay is small. ｓｔｕｄｅｎｔ By using this, it is possible to achieve target sound extraction with low delay and high extraction performance.

[0062] Teacher Model TSE ｔｅａｃｈｅｒ In addition to the non-causal processing, the inference process of (a) further uses input information (third input information) based on the mixed signal (first mixed signal) y before the present to derive the current extracted information (first extracted information) ^x without using information based on the future mixed signal (first mixed signal) y. ｓ ｔｅａｃｈｅｒ In this case, the teacher model TSE may include a causal process for obtaining output information (third output information) corresponding to the teacher model TSE. ｔｅａｃｈｅｒ Part of the inference process is the student model TSE ｓｔｕｄｅｎｔ Since this is a causal process similar to part of the inference process of the student model TSE by knowledge distillation, ｓｔｕｄｅｎｔ In other words, the learning accuracy of the student model TSE in knowledge distillation is improved. ｓｔｕｄｅｎｔThe convergence speed of the loss function L ＴＳ Even in this case, the function value of the teacher model TSE ｔｅａｃｈｅｒ Since part of the inference process is acausal, the student model TSE obtained by this knowledge distillation ｓｔｕｄｅｎｔ The extraction performance of the student model TSE obtained in this way is high. ｓｔｕｄｅｎｔ By using this, it is possible to achieve target sound extraction with small delay and higher extraction performance.

[0063] For example, (1) teacher model TSE ｔｅａｃｈｅｒ The non-causal process included in the inference process is a mixed signal (first mixed signal) y∈R Ｋ The current extracted information (first extracted information) ^x is obtained by convolution of the input information (first input information) based on ｓ ｔｅａｃｈｅｒ and a process of obtaining output information (first output information) corresponding to the teacher model TSE. ｔｅａｃｈｅｒ The causal process included in the inference process is to perform statistical processing of input information (third input information) based on the mixed signal (first mixed signal) y before the current time to obtain the current extracted information (first extracted information) ^x ｓ ｔｅａｃｈｅｒ and a process of obtaining output information (third output information) corresponding to the student model TSE. ｓｔｕｄｅｎｔ The causal process included in the inference process is the mixed signal (second mixed signal) y∈R Ｋ Statistical processing of input information (second input information) based on the current extracted information (second extracted information) ^x ｓ ｓｔｕｄｅｎｔ The process may include a process of obtaining output information (second output information) corresponding to the above.

[0064] For example, (2) teacher model TSE ｔｅａｃｈｅｒ The non-causal process included in the inference process is a mixed signal (first mixed signal) y∈R Ｋ Statistical processing of input information (first input information) based on the current extracted information (first extracted information) ^x ｓ ｔｅａｃｈｅｒ and a process of obtaining output information (first output information) corresponding to the teacher model TSE. ｔｅａｃｈｅｒThe causal process included in the inference process is to perform convolution of input information (third input information) based on the mixed signal (first mixed signal) y before the current time to obtain the current extracted information (first extracted information) ^x ｓ ｔｅａｃｈｅｒ and a process of obtaining output information (third output information) corresponding to the student model TSE. ｓｔｕｄｅｎｔ The causal process included in the inference process is the mixed signal (second mixed signal) y∈R Ｋ The current extracted information (second extracted information) ^x is obtained by convolution of the input information (second input information) based on ｓ ｓｔｕｄｅｎｔ The process may include a process of obtaining output information (second output information) corresponding to the above.

[0065] [Second Embodiment] In this embodiment, the present invention is implemented based on "The fully-convolutional time-domain audio separation network (Conv-TasNet)," which is a target sound extraction technology described in Reference 1. The following description will focus on differences from the first embodiment, and the same reference numbers will be used to simplify the description of matters common to the first embodiment.

[0066] 1 , an object sound extraction system 2 of this embodiment has a teacher model device 21, a knowledge distillation device 22, and an object sound extraction device 23. The teacher model device 21, the knowledge distillation device 22, and the object sound extraction device 23 are configured to be able to communicate with each other via a network, for example.

[0067] As illustrated in Fig. 2, the teacher model device 21 of this embodiment has a storage unit 111, an input unit 112, an inference processing unit 213, and an output unit 114. The knowledge distillation device 22 of this embodiment has a learning data setting unit 120, a storage unit 121, an input unit 122, a knowledge distillation unit 223, and an output unit 124. The knowledge distillation unit 223 has a parameter transfer unit 2231 and a teacher-student learning unit 2232. As illustrated in Fig. 3, the object sound extraction device 23 of this embodiment has a storage unit 131, an input unit 132, an inference processing unit 233, and an output unit 134.

[0068] <Processing> In this embodiment, the knowledge distillation unit 223 of the knowledge distillation device 22 obtains a student model through a learning process using learning data based on a teacher model. This process will be described in detail below.

[0069] <Teacher Model> Teacher model TSE of this embodiment ｔｅａｃｈｅｒ is a model based on a convolutional neural network. As shown in FIG. 4, the teacher model TSE ｔｅａｃｈｅｒ has an encoding layer 1110 , a corresponding information encoding layer 1120 , a main layer 2130 , a Hadamard stack 1140 , and a decoding layer 1150 .

[0070] <Main Layer 2130> As illustrated in FIG. 5A, the main layer 2130 receives the encoded mixed signal Y ｔｅａｃｈｅｒ and encoding correspondence information e ｓ ｔｅａｃｈｅｒ On the other hand, the target mask M ｓ ｔｅａｃｈｅｒ The main layer 2130 in this embodiment includes a mixing block 2131, a fusion block 1132, and a target block 2133.

[0071] The mixing block 2131 receives the coded mixed signal Y ｔｅａｃｈｅｒ the internal mixed signal Z ｔｅａｃｈｅｒ ∈R Ｄ×Ｋ’ This layer converts the input data into a vector and outputs it. The relationship is shown below. Z teacher =MixBlock(Y teacher )

[0072] The fusion block 1132 receives the internally mixed signal Z ｔｅａｃｈｅｒ and encoding correspondence information e ｓ ｔｅａｃｈｅｒ The internal target signal Z ｓ ｔｅａｃｈｅｒ ∈R Ｄ×Ｋ’ This layer converts the input data into a vector and outputs it. The relationship is shown below. Z s teacher =Adapt(Z teacher ,e s teacher )

[0073] The target block 2133 receives the internal target signal Z ｓ ｔｅａｃｈｅｒThe target mask M ｓ ｔｅａｃｈｅｒ ∈R Ｄ×Ｋ’ This layer converts the input signal into a vector and outputs it. The relationship is shown below. M s teacher =TgtBlock(Z s teacher )

[0074] 7, the mixing block 2131 of this embodiment includes a layer normalization block (LN block) 2131a, a 1×1-convolution operation block 2131b, and a 1-D convolution block 2000. The layer normalization block 2131a receives the encoded mixed signal Y ｔｅａｃｈｅｒ is input, and the signal output from the layer normalization block 2131a is input to the 1×1 convolution block 2131b. The signal output from the 1×1 convolution block 2131b in response to this input is input to the 1D convolution block 2000. In response to this, the 1D convolution block 2000 outputs an internal mixing signal Z ｔｅａｃｈｅｒ ∈R Ｄ×Ｋ’ will be output.

[0075] 8, the target block 2133 of this embodiment includes a plurality of 1D convolution blocks 2000, a linear rectification block (PReLU block) 2133a, a 1×1 convolution operation block 2133b, and an activation function block 2133c. The first 1D convolution block 2000 receives an internal target signal Z ｓ ｔｅａｃｈｅｒis input, and this 1D convolution block 2000 outputs output information out. The output information out becomes input information in of the next 1D convolution block 2000, and the next 1D convolution block 2000 outputs the next output information out. This process is repeated, and the output information out from the last 1D convolution block 2000 is input to the linear rectification block 2133a. In response to this, the output information output from the linear rectification block 2133a is input to the 1×1 convolution block 2133b. In response to this, the output information output from the 1×1 convolution block 2133b is input to the activation function block 2133c. In response to this, the activation function block 2133c calculates the target mask M ｓ ｔｅａｃｈｅｒ Output.

[0076] As illustrated in FIG. 9 , the 1D convolution block 2000 of this embodiment includes a 1×1 convolution block 2001, an activation function block 2002, a layer normalization block 2003, a dilated convolution block (dilated convolution operation block, depthwise convolution operation block) 2004, an activation function block 2005, a layer normalization block 2006, a 1×1 convolution block 2007, and an adder 2008. As illustrated in FIG. 9 , input information in to the 1D convolution block 2000 is first input to the 1×1 convolution block 2001. In response to this, output information output from the 1×1 convolution block 2001 is input to the activation function block 2002. In response to this, output information output from the activation function block 2002 is input to the layer normalization block 2003. In response to this, output information output from the layer normalization block 2003 is input to the dilated convolution block 2004. On the other hand, the output information output from the dilated convolution block 2004 is input to an activation function block 2005. On the other hand, the output information output from the activation function block 2005 is input to a layer normalization block 2006. On the other hand, the output information output from the layer normalization block 2006 is input to a 1×1 convolution block 2007. On the other hand, the output information output from the 1×1 convolution block 2007 is input to an adder 2008. The adder 2008 adds the output information output from the 1×1 convolution block 2007 and the input information in to the 1D convolution block 2000 to obtain and output an output signal out.

[0077] The 1x1 convolution blocks 2131b, 2133b, 2001, and 2007 are blocks that perform the 1x1-conv operation described in Reference 1. The linear rectification block 2133a is a block that performs parametric rectified linear unit (PReLU) processing described in Reference 1. The activation function blocks 2133c, 2002, and 2005 are layers that apply activation functions to input information and output the function values. There are no limitations on the activation functions, and known activation functions such as ReLU functions and sigmoid functions can be used.

[0078] The layer normalization blocks 2131a, 2003, and 2006 perform statistical processing of input information to obtain output information. The input information to the layer normalization blocks 2131a, 2003, and 2006 in this embodiment is Z=[z1,...,z Γ ]∈R H×Γ Here, the element z of the input information Z is t =[z 1,t ,...,z H,t ] T is an H-dimensional vector, where H is a positive integer, h = 1, ..., H. t represents the time frame, t = 1, ..., Γ. α T represents the transpose of α. The layer normalization blocks 2131a, 2003, and 2006 perform the following statistical processing on the input information Z, and calculate the current time frame t ｃ The output information for [LN[z1],...,LN[z Γ ]]∈R H×Γ Output. Here, μ[Z] represents the average (statistical value, statistical quantity) of the input information Z. μ[Z] is expressed as follows: β∈R Ｈ and γ∈R Ｈ are the learnable scale and bias model parameters, respectively. [1,...,1] T ∈R H σ[Z] represents the variance (statistical value, statistical quantity) of the input information Z. σ[Z] is expressed as follows: Whether the processing of the layer normalization blocks 2131a, 2003, and 2006 is causal or non-causal is determined by the current time frame tｃ The time range t = 1, ..., Γ of the statistical processing of the above equations (2) and (3) is determined by the current time frame t ｃ If only the previous time frame is used, the processing of the layer normalization blocks 2131a, 2003, and 2006 is a causal processing. In other words, 1≦Γ≦t ｃ If the above equation is satisfied, the processing of the layer normalization blocks 2131a, 2003, and 2006 is a causal processing. The layer normalization blocks 2131a, 2003, and 2006 that perform causal processing are collectively referred to as cumulative global LN (cgLN). On the other hand, if the time range t=1,...,Γ of the statistical processing of equations (2) and (3) is the current time frame t ｃ The processing of the layer normalization blocks 2131a, 2003, and 2006 is acausal if it includes not only previous but also future time frames. ｃ <If Γ is satisfied, the processing of the layer normalization blocks 2131a, 2003, and 2006 is acausal processing. The layer normalization blocks 2131a, 2003, and 2006 that perform acausal processing are collectively referred to as global LN (gLN).

[0079] The dilated convolution block 2004 is a block that performs the depthwise convolution (D-conv) operation described in Reference 1. The dilated convolution block 2004 performs convolution of input information to obtain output information. This convolution includes convolution of input information of multiple time frames. When the dilated convolution block 2004 performs the convolution of the current time frame t ｃ Only the input information of the previous time frame is convolved to obtain the current time frame t ｃ If the dilated convolution block 2004 obtains output information of the current time frame t, the processing of the dilated convolution block 2004 is a causal processing. The dilated convolution block 2004 that performs the causal processing is called a causal processing. On the other hand, if the dilated convolution block 2004 obtains output information of the current time frame t ｃ The current time frame t is calculated by convolving not only the input information of the previous time frame but also the input information of the future time frame. ｃIf the output information obtained is:

[0080] As described above, the teacher model TSE of this embodiment ｔｅａｃｈｅｒ The inference process by the layer normalization block 2131a includes non-causal processing. For example, at least one of the layer normalization blocks 2131a, 2003, and 2006 may be a gLN that performs non-causal processing. For example, the dilated convolution block 2004 may be a non-causal that performs non-causal processing. ｔｅａｃｈｅｒ The inference processing by the MLN may include causal processing and non-causal processing. For example, the layer normalization blocks 2131a, 2003, and 2006 may be gLNs that perform non-causal processing, and the dilated convolution block 2004 may be causal. For example, the layer normalization blocks 2131a, 2003, and 2006 may be cgLNs that perform causal processing, and the dilated convolution block 2004 may be non-causal. However, these are merely examples and do not limit the present invention.

[0081] <Student Model> Student model TSE of this embodiment ｓｔｕｄｅｎｔ is a model based on a convolutional neural network. As shown in FIG. 4, the student model TSE ｓｔｕｄｅｎｔ has an encoding layer 1210 , a corresponding information encoding layer 1220 , a main layer 2230 , a Hadamard stack 1240 , and a decoding layer 1250 .

[0082] <Main Layer 2230> As illustrated in FIG. 5B, the main layer 2230 receives the encoded mixed signal Y ｓｔｕｄｅｎｔ and encoding correspondence information e ｓ ｓｔｕｄｅｎｔ On the other hand, the target mask M ｓ ｓｔｕｄｅｎｔ The main layer 2230 in this embodiment includes a mixing block 2231, a fusion block 2232, and a target block 2233.

[0083] The mixing block 2231 receives the coded mixed signal Y ｓｔｕｄｅｎｔ the internal mixed signal Zｓｔｕｄｅｎｔ ∈R Ｄ×Ｋ’ This layer converts the input data into a vector and outputs it. The relationship is shown below. Z student =MixBlock(Y student )

[0084] The fusion block 1232 receives the internally mixed signal Z ｓｔｕｄｅｎｔ and encoding correspondence information e ｓ ｓｔｕｄｅｎｔ The internal target signal Z ｓ ｓｔｕｄｅｎｔ ∈R Ｄ×Ｋ’ This layer converts the input data into a vector and outputs it. The relationship is shown below. Z s student =Adapt(Z student ,e s student )

[0085] The target block 2233 receives the internal target signal Z ｓ ｓｔｕｄｅｎｔ The target mask M ｓ ｓｔｕｄｅｎｔ ∈R Ｄ×Ｋ’ This layer converts the input signal into a vector and outputs it. The relationship is shown below. M s student =TgtBlock(Z s student )

[0086] 7, the mixing block 2231 of this embodiment includes a layer normalization block (LN block) 2231a, a 1×1-convolution operation block 2231b, and a 1-D convolution block 2000. The layer normalization block 2231a stores the encoded mixed signal Y ｓｔｕｄｅｎｔ is input, and the signal output from the layer normalization block 2231a is input to the 1×1 convolution block 2231b. The signal output from the 1×1 convolution block 2231b in response to this input is input to the 1D convolution block 2000. In response to this, the 1D convolution block 2000 outputs an internal mixing signal Z ｓｔｕｄｅｎｔ ∈R Ｄ×Ｋ’ will be output.

[0087] 8, the target block 2233 of this embodiment includes a plurality of 1D convolution blocks 2000, a linear rectification block (PReLU block) 2233a, a 1×1 convolution block (1×1-convolution operation block) 2233b, and an activation function block 2233c. The first 1D convolution block 2000 receives an internal target signal Z ｓ ｓｔｕｄｅｎｔ is input, and this 1D convolution block 2000 outputs output information out. The output information out becomes input information in of the next 1D convolution block 2000, and the next 1D convolution block 2000 outputs the next output information out. This process is repeated, and the output information out from the last 1D convolution block 2000 is input to the linear rectification block 2233a. In response to this, the output information output from the linear rectification block 2233a is input to the 1×1 convolution block 2233b. In response to this, the output information output from the 1×1 convolution block 2233b is input to the activation function block 2233c. In response to this, the activation function block 2233c calculates the target mask M ｓ ｓｔｕｄｅｎｔ Output.

[0088] The 1x1 convolution blocks 2231b and 2233b are blocks that perform the 1x1-conv operation described in Reference 1. The linear rectification block 2233a is a block that performs parametric rectified linear unit (PReLU) processing described in Reference 1. The activation function block 2233c is a layer that applies an activation function to input information and outputs the function value. There are no limitations on the activation function, and known activation functions such as the ReLU function and the Sigmond function can be used, for example.

[0089] The layer normalization block 2231a performs statistical processing of input information to obtain output information. The input information to the layer normalization block 2231a in this embodiment is Z=[z1,...,z Γ ]∈R H×ΓThe layer normalization block 2231 performs statistical processing of the input information Z according to the equations (1), (2), and (3) to obtain the current time frame t ｃ The output information for [LN[z1],...,LN[z Γ ]]∈R H×Γ Whether the processing of the layer normalization block 2231a is a causal processing or a non-causal processing is determined by the current time frame t ｃ The time range t = 1, ..., Γ of the statistical processing of equations (2) and (3) is determined by the time range t = 1, ..., Γ of the statistical processing of equations (2) and (3). ｃ If only the previous time frame exists, the processing of the layer normalization block 2231a is a causal processing. In other words, 1≦Γ≦t ｃ If the above formula (2) and (3) are satisfied, the processing of the layer normalization block 2231a is a causal processing. As mentioned above, the layer normalization block 2231a that performs the causal processing is collectively called cumulative global LN (cgLN). On the other hand, if the time range t=1,...,Γ of the statistical processing of the formula (2) and (3) is the current time frame t ｃ The processing of the layer normalization block 2231a is acausal if it includes not only previous but also future time frames. ｃ <Γ is satisfied, the processing of the layer normalization block 2231a is acausal processing. As described above, the layer normalization blocks 2231a that perform acausal processing are collectively referred to as global LN (gLN).

[0090] As described above, the student model TSE of this embodiment ｓｔｕｄｅｎｔ The inference processing by includes non-causal processing. For example, the layer normalization block 2231a may be a gLN that performs non-causal processing. For example, the layer normalization block 2231a may be a gLN that performs non-causal processing, and the dilated convolution block 2004 may be a causal that performs causal processing. For example, the layer normalization block 2231a may be a cgLN that performs causal processing, and the dilated convolution block 2004 may be a non-causal that performs non-causal processing. However, these are merely examples and do not limit the present invention.

[0091] <Features of this embodiment> As in the first embodiment, in this embodiment, the teacher model TSE ｔｅａｃｈｅｒ Learning data (y, o ｓ , ^x ｓ ｔｅａｃｈｅｒ ) by the learning process using the student model TSE ｓｔｕｄｅｎｔ The knowledge distillation was performed to obtain the student model TSE. ｓｔｕｄｅｎｔ By using this, it is possible to achieve target sound extraction with low delay and high extraction performance.

[0092] Also, the teacher model TSE ｔｅａｃｈｅｒ In addition to the non-causal processing, the inference process of (a) further uses input information (third input information) based on the mixed signal (first mixed signal) y before the present to derive the current extracted information (first extracted information) ^x without using information based on the future mixed signal (first mixed signal) y. ｓ ｔｅａｃｈｅｒ The obtained student model TSE may include a causal process for obtaining output information (third output information) corresponding to the ｓｔｕｄｅｎｔ By using this, it is possible to achieve target sound extraction with small delay and higher extraction performance.

[0093] [Modification of the Second Embodiment] The target blocks 2133 and 2233 of the second embodiment may be replaced with blocks including a skip structure. As illustrated in FIG. 8, the target blocks 2133 and 2233 of this modification further include adders 2133d and 2233d in addition to the target blocks 2133 and 2233 of the second embodiment. The internal target signal Z ｓ ｔｅａｃｈｅｒ or Z ｓ ｓｔｕｄｅｎｔis input, and this 1D convolution block 2000 outputs output information out and skip information skip. The output information out becomes input information in for the next 1D convolution block 2000, and the next 1D convolution block 2000 outputs the next output information out. The skip information skip is input to adders 2133d and 2233d. This process is repeated, and the output information out from the last 1D convolution block 2000 is input to adders 2133d and 2233d. The adders 2133d and 2233d output output information obtained by adding the values ​​of the input information. The output information of adders 2133d and 2233d is input to linear rectification block 2133a. In contrast, the output information output from linear rectification block 2133a is input to 1×1 convolution block 2133b. On the other hand, the output information output from the 1×1 convolution block 2133b is input to the activation function block 2133c. On the other hand, the activation function block 2133c calculates the target mask M ｓ ｔｅａｃｈｅｒ Output.

[0094] As illustrated in Fig. 9, the 1D convolution block 2000 of this modification further includes a 1x1 convolution block 2010 in addition to the 1D convolution block 2000 of the second embodiment. In this modification, output information output from the activation function block 2005 is input to the layer normalization block 2006. In contrast, output information output from the layer normalization block 2006 is input to the 1x1 convolution block 2007 and the 1x1 convolution block 2010. The 1x1 convolution block 2010 to which this output information is input outputs skip information "skip." The rest is the same as in the second embodiment.

[0095] [Experimental Results] Next, experimental results showing the effect of knowledge distillation according to this embodiment will be shown. In the experiment, the following models a) to d) were used to extract the target sound by inference processing. All of the models were the teacher model TSE exemplified in the second embodiment. ｔｅａｃｈｅｒ and student model TSE ｓｔｕｄｅｎｔ The structure is the same as above. The ReLU function was used as the activation function. In the model (a), the dilated convolution block 2004 of the 1D convolution block 2000 (FIG. 9) is non-causal, performing non-causal processing, and the layer normalization blocks 2131a, 2231a, 2003, and 2006 of the main layers 2130 and 2230 (FIGS. 7 and 9) are gLNs performing non-causal processing. The inference processing of the model (a) is non-causal as a whole, without including causal processing.

[0096] In the model (b), the dilated convolution block 2004 of the 1D convolution block 2000 (FIG. 9) is causal, performing causal processing, and the layer normalization blocks 2131a, 2231a, 2003, and 2006 of the main layers 2130 and 2230 (FIGS. 7 and 9) are cgLN, performing causal processing. The inference process of the model (b) is causal as a whole, without including non-causal processing.

[0097] In the model (c), the dilated convolution block 2004 of the 1D convolution block 2000 (FIG. 9) is non-causal, performing non-causal processing, and the layer normalization blocks 2131a, 2231a, 2003, and 2006 of the main layers 2130 and 2230 (FIGS. 7 and 9) are cgLNs performing causal processing. The inference processing of the model (c) includes partly causal processing and partly non-causal processing, and is partially causal / non-causal.

[0098] In the model (d), the dilated convolution block 2004 of the 1D convolution block 2000 (FIG. 9) is causal, performing causal processing, and the layer normalization blocks 2131a, 2231a, 2003, and 2006 of the main layers 2130 and 2230 (FIGS. 7 and 9) are gLNs performing acausal processing. The inference processing of the model (d) includes partly causal processing and partly acausal processing, and is partially causal / acausal.

[0099] When target sounds were extracted using models (a), (b), (c), and (d) trained without knowledge distillation, the signal-to-distortion ratio improvement (SDRi) (see Non-Patent Document 1, etc.) was 8.26, 6.55, 7.74, and 7.95 dB, respectively. Note that a higher SDRi indicates higher target sound extraction accuracy. Thus, when training was performed without knowledge distillation, the causal model (b) had the lowest extraction performance, the non-causal model (a) had the highest extraction performance, and the partially causal / non-causal models (c) and (d) had extraction performances between them. More specifically, when using the causal model (b), the SDRi (performance) decreased by 1.71 dB compared to the non-causal model (a). The partially causal / non-causal models (c) and (d) performed worse than the non-causal model (a), but better than the causal model (b). The partially causal / non-causal models (c) and (d) performed worse than the causal model (a), but better than the causal model (b). Of the partially causal / non-causal models (c) and (d), model (d) using a non-causal layer normalization block (gLN) performed better than model (c) using a non-causal dilated convolution block (non-causal), and had an SDRi 0.31 dB lower than the causal model (a).

[0100] Next, as shown in (1) to (5) below, models (a), (c), and (d) are used as teacher models TSE. ｔｅａｃｈｅｒ The student model TSE (model (b)) is used as ｓｔｕｄｅｎｔ We compare the extraction performance when knowledge distillation is performed. In all of (1) to (5), model (b) is the student model TSE. ｓｔｕｄｅｎｔ In (1), model (a) is the teacher model TSE. ｔｅａｃｈｅｒ In (2), model (c) is the teacher model TSE ｔｅａｃｈｅｒ In (3), (4), and (5), model (d) is the teacher model TSE. ｔｅａｃｈｅｒTS learning is performed in (1), (2), (3), and (4), but not in (5). PT learning is performed in (1), (2), (3), and (5), but not in (4). As a result of the experiment, the student model TSE obtained in (1), (2), (3), (4), and (5) ｓｔｕｄｅｎｔ The SDRi of (a) and (b) were 6.93, 7.02, 7.35, 7.07, and 6.82 [dB], respectively, which were 0.38, 0.47, 0.80, 0.52, and 0.27 [dB] better than the model (b) trained without knowledge distillation. Thus, in all of (1), (2), (3), (4), and (5), the teacher model TSE, which performs non-causal processing, ｔｅａｃｈｅｒ A student model TSE that performs causal processing using ｓｔｕｄｅｎｔ By using knowledge distillation, we were able to improve the extraction performance compared to learning without knowledge distillation. In particular, model (d) is used as the teacher model TSE. ｔｅａｃｈｅｒ The student model TSE was used as the knowledge distillation model, and both TS learning and PT learning were performed. ｓｔｕｄｅｎｔ In this way, the performance of the teacher model TSE, which performs partially causal and non-causal processing, was maximized. ｔｅａｃｈｅｒ Using the student model TSE, which performs causal processing, ｓｔｕｄｅｎｔ The student model TSE obtained by knowledge distillation ｓｔｕｄｅｎｔ It was found that the performance of the student model TSE can be maximized. ｓｔｕｄｅｎｔ The combination of these two improves the performance of the student model TSE. ｓｔｕｄｅｎｔ It was found that the performance can be further improved.

[0101] [Hardware Configuration] The teacher model devices 11 and 21, the knowledge distillation devices 12 and 22, and the object sound extraction devices 13 and 23 in each embodiment are devices configured by a general-purpose or dedicated computer including, for example, a processor (hardware processor) such as a CPU (central processing unit) and memories such as RAM (random-access memory) and ROM (read-only memory) executing a predetermined program. That is, the teacher model devices 11 and 21, the knowledge distillation devices 12 and 22, and the object sound extraction devices 13 and 23 in each embodiment have, for example, a processing circuit configured to implement each unit possessed by each of them. This computer may have one processor and memory, or multiple processors and memories. This program may be installed on the computer or may be recorded in advance in a ROM or the like. Furthermore, some or all of the processing units may be configured using electronic circuits that independently realize processing functions, rather than electronic circuits that realize functional configurations by loading programs like a CPU. Furthermore, the electronic circuits constituting one device may include multiple CPUs.

[0102] FIG. 10 is a block diagram illustrating the hardware configuration of the teacher model devices 11 and 21, the knowledge distillation devices 12 and 22, and the object sound extraction devices 13 and 23 in each embodiment. As illustrated in FIG. 10, the teacher model devices 11 and 21, the knowledge distillation devices 12 and 22, and the object sound extraction devices 13 and 23 in this example include a CPU (Central Processing Unit) 10a, an input unit 10b, an output unit 10c, a RAM (Random Access Memory) 10d, a ROM (Read Only Memory) 10e, an auxiliary storage device 10f, a communication unit 10h, and a bus 10g. The CPU 10a in this example includes a control unit 10aa, a calculation unit 10ab, and a register 10ac, and executes various calculation processes according to various programs loaded into the register 10ac. The input unit 10b is an input terminal, keyboard, mouse, touch panel, etc. through which data is input. The output unit 10c is an output terminal, display, etc. through which data is output. The communication unit 10h is a LAN card or the like controlled by the CPU 10a that has loaded a predetermined program. The RAM 10d is a static random access memory (SRAM) or dynamic random access memory (DRAM), and has a program area 10da where the predetermined program is stored and a data area 10db where various data is stored. The auxiliary storage device 10f is a hard disk, magneto-optical disk (MO), semiconductor memory, or the like, and has a program area 10fa where the predetermined program is stored and a data area 10fb where various data is stored. The bus 10g connects the CPU 10a, input unit 10b, output unit 10c, RAM 10d, ROM 10e, communication unit 10h, and auxiliary storage device 10f so that information can be exchanged. The CPU 10a writes the program stored in the program area 10fa of the auxiliary storage device 10f to the program area 10da of RAM 10d in accordance with the loaded OS (Operating System) program. Similarly, the CPU 10a writes various data stored in the data area 10fb of the auxiliary storage device 10f to the data area 10db of the RAM 10d.The addresses on RAM 10d where these programs and data are written are stored in register 10ac of CPU 10a. Control unit 10aa of CPU 10a sequentially reads these addresses stored in register 10ac, reads programs and data from areas on RAM 10d indicated by the read addresses, causes calculation unit 10ab to sequentially execute calculations indicated by the programs, and stores the calculation results in register 10ac. With this configuration, the functional configurations of teacher model devices 11 and 21, knowledge distillation devices 12 and 22, and target sound extraction devices 13 and 23 are realized.

[0103] The above-mentioned program can be recorded on a computer-readable recording medium. Examples of computer-readable recording media include non-transitory recording media. Examples of such recording media include magnetic recording devices, optical disks, magneto-optical recording media, and semiconductor memories.

[0104] This program may be distributed, for example, by selling, transferring, or lending a portable recording medium, such as a DVD or CD-ROM, on which the program is recorded. Furthermore, the program may be distributed by storing the program in a storage device of a server computer and transferring the program from the server computer to other computers via a network. As described above, a computer that executes such a program may first temporarily store the program recorded on a portable recording medium or transferred from the server computer in its own storage device. Then, when executing a process, the computer reads the program stored in its own storage device and executes processing in accordance with the read program. Alternatively, the program may be executed by a computer that reads the program directly from a portable recording medium and executes processing in accordance with the program. Furthermore, the computer may execute processing in accordance with the received program each time a program is transferred from the server computer to the computer. Alternatively, the server computer may not transfer the program to the computer, but may instead execute the processing function simply by issuing an execution instruction and obtaining the results, thereby executing the processing described above through a so-called ASP (Application Service Provider) service. In this embodiment, the program includes information used for processing by an electronic computer that is equivalent to a program (such as data that is not a direct instruction to a computer but has properties that dictate computer processing).

[0105] In each embodiment, the device is configured by executing a predetermined program on a computer, but at least a part of the processing contents may be realized by hardware.

[0106] It should be noted that the present invention is not limited to the above-described embodiments. For example, in the above-described embodiments, the student model TSE ｓｔｕｄｅｎｔ The inference process of the teacher model TSE did not include non-causal processes. ｔｅａｃｈｅｒIf the inference process of is less than the non-causal process involved, then the student model TSE ｓｔｕｄｅｎｔ The inference process may include acausal processes.

[0107] Furthermore, the various processes described above may not only be executed in chronological order as described, but may also be executed in parallel or individually depending on the processing capabilities of the devices that execute the processes or as necessary. Needless to say, other modifications are possible within the scope of the invention as defined in the claims.

[0108] 11, 21 Teacher model device 12, 22 Knowledge distillation device 13, 23 Target sound extraction device

Claims

1. a knowledge distillation unit that obtains a student model by a learning process using learning data based on a teacher model; the teacher model performs an inference process in which, in response to input of a first mixed signal in a time series in which a plurality of acoustic signals are mixed and first correspondence information corresponding to a specific first segment, the teacher model obtains and outputs first extracted information in a time series from the first mixed signal corresponding to the acoustic signals belonging to the first segment; the student model performs an inference process in which, in response to input of a second mixed signal in a time series in which a plurality of acoustic signals are mixed and second correspondence information corresponding to a specific second division, the student model obtains second extracted information in a time series from the second mixed signal and outputs the second extracted information corresponding to the acoustic signals belonging to the second division; the inference process of the teacher model includes a non-causal process of obtaining first output information corresponding to the current first extracted information using first input information based on the first mixed signal before and in the future, the inference process of the student model includes a causal process for obtaining second output information corresponding to the current second extracted information using second input information based on the second mixed signal in the past without using information based on the second mixed signal in the future; Knowledge distillation apparatus.

2. A knowledge distillation apparatus according to claim 1, The inference process of the teacher model further includes a causal process of obtaining third output information corresponding to the current first extracted information using third input information based on the first mixed signal before the present, without using information based on the first mixed signal in the future.

3. A knowledge distillation apparatus according to claim 2, (1) The non-causal processing included in the teacher model's inference processing includes processing to obtain the first output information by performing convolution of the first input information, the causal processing included in the teacher model's inference processing includes processing to obtain the third output information by performing statistical processing of the third input information, and the causal processing included in the student model's inference processing includes processing to obtain the second output information by performing statistical processing of the second input information, or (2) The non-causal processing included in the inference processing of the teacher model includes processing to obtain the first output information by performing statistical processing of the first input information, the causal processing included in the inference processing of the teacher model includes processing to obtain the third output information by performing convolution of the third input information, and the causal processing included in the inference processing of the student model includes processing to obtain the second output information by performing convolution of the second input information. Knowledge distillation apparatus.

4. A knowledge distillation apparatus according to claim 1, The aforementioned knowledge distillation unit, a parameter transfer unit that sets model parameters of the teacher model that do not affect the non-causal processing as part of model parameters of the student model; a teacher-student learning unit that learns a remaining part of model parameters of the student model using the learning data including the first mixed signal, the first correspondence information, and the first extracted information; Knowledge distillation apparatus having.

5. an inference processing unit that applies a time-series mixed signal in which a plurality of acoustic signals are mixed and correspondence information corresponding to a specific category to a student model, and obtains and outputs time-series extracted information corresponding to the acoustic signals belonging to the category from the mixed signal; The aforementioned student model was obtained through a learning process using training data based on the teacher model. the teacher model performs an inference process in which, in response to input of a first mixed signal in a time series in which a plurality of acoustic signals are mixed and first correspondence information corresponding to a specific first segment, the teacher model obtains and outputs first extracted information in a time series from the first mixed signal corresponding to the acoustic signals belonging to the first segment; the student model performs an inference process in which, in response to input of a second mixed signal in a time series in which a plurality of acoustic signals are mixed and second correspondence information corresponding to a specific second division, the student model obtains second extracted information in a time series from the second mixed signal and outputs the second extracted information corresponding to the acoustic signals belonging to the second division; the inference process of the teacher model includes a non-causal process of obtaining first output information corresponding to the current first extracted information using first input information based on the first mixed signal before and in the future, the inference process of the student model includes a causal process for obtaining second output information corresponding to the current second extracted information using second input information based on the second mixed signal in the past without using information based on the second mixed signal in the future; Target sound extraction device.

6. A method of knowledge distillation using a knowledge distillation apparatus, It has a knowledge distillation step in which a student model is obtained by a learning process using training data based on a teacher model, the teacher model performs an inference process in which, in response to input of a first mixed signal in a time series in which a plurality of acoustic signals are mixed and first correspondence information corresponding to a specific first segment, the teacher model obtains and outputs first extracted information in a time series from the first mixed signal corresponding to the acoustic signals belonging to the first segment; the student model performs an inference process in which, in response to input of a second mixed signal in a time series in which a plurality of acoustic signals are mixed and second correspondence information corresponding to a specific second division, the student model obtains second extracted information in a time series from the second mixed signal and outputs the second extracted information corresponding to the acoustic signals belonging to the second division; the inference process of the teacher model includes a non-causal process of obtaining first output information corresponding to the current first extracted information using first input information based on the first mixed signal before and in the future, the inference process of the student model includes a causal process for obtaining second output information corresponding to the current second extracted information using second input information based on the second mixed signal in the past without using information based on the second mixed signal in the future; Knowledge distillation method.

7. A method for extracting a target sound using a target sound extraction device, an inference processing step of applying a time-series mixed signal in which a plurality of acoustic signals are mixed and correspondence information corresponding to a specific category of the mixed signal to a student model, and obtaining and outputting time-series extracted information corresponding to the acoustic signals belonging to the category from the mixed signal; The aforementioned student model was obtained through a learning process using training data based on the teacher model. the teacher model performs an inference process in which, in response to input of a first mixed signal in a time series in which a plurality of acoustic signals are mixed and first correspondence information corresponding to a specific first segment, the teacher model obtains and outputs first extracted information in a time series from the first mixed signal corresponding to the acoustic signals belonging to the first segment; the student model performs an inference process in which, in response to input of a second mixed signal in a time series in which a plurality of acoustic signals are mixed and second correspondence information corresponding to a specific second division, the student model obtains second extracted information in a time series from the second mixed signal and outputs the second extracted information corresponding to the acoustic signals belonging to the second division; the inference process of the teacher model includes a non-causal process of obtaining first output information corresponding to the current first extracted information using first input information based on the first mixed signal before and in the future, the inference process of the student model includes a causal process for obtaining second output information corresponding to the current second extracted information using second input information based on the second mixed signal in the past without using information based on the second mixed signal in the future; Target sound extraction method.

8. A program for causing a computer to function as a knowledge distillation apparatus according to any of claims 1 to 4 or as a target sound extraction apparatus according to claim 5.

Images