Learning device, training method, and training program
The learning device addresses catastrophic forgetting by updating only a portion of the neural network and using vector representations to adapt to new domains without labels, effectively preventing accuracy drops and reducing human labor in the learning process.
Patent Information
- Application Number
- PCT/JP2023/019481
- Authority / Receiving Office
- WO · WO
- Patent Type
- Applications
- Current Assignee / Owner
- Filing Date
- 2023-05-25
- Publication Date
- 2024-11-28
AI Technical Summary
Trained models experience significant estimation accuracy drops when transitioning from the original domain to a different applied domain, known as catastrophic forgetting, and existing methods that prevent this require heavy human labor for label association.
A learning device that updates only a portion of the neural network, using a clean signal and a mixed signal with noise, and employs a student learning model to estimate a vector representation that approaches the original domain teacher model without using labels, by extracting clean and mixed feature amounts and calculating a value based on teacher and student vector representations.
Prevents catastrophic forgetting without the need for labels, allowing the learning device to adapt to new domains while maintaining original domain performance, reducing human burden and improving learning efficiency.
Smart Images

Figure JP2023019481_28112024_PF_FP_ABST
Abstract
Description
Learning device, learning method, and learning program
[0001] The present disclosure relates to a learning device, a learning method, and a learning program.
[0002] A trained model may be used in signal processing. The trained model may be trained in multiple environments. Hereinafter, the initial training environment is referred to as the original domain. The training environment after the original domain is referred to as the application domain. A trained model obtained by training in the original domain may be retrained in the application domain. When the original domain and the application domain are different, the estimation accuracy of the trained model obtained by training in the original domain may be significantly reduced. This phenomenon is called catastrophic forgetting. Therefore, according to Non-Patent Document 1, the neural network is partially updated rather than the entire neural network being updated. This prevents catastrophic forgetting.
[0003] International Publication No. 2016 / 143125
[0004] Yuki Takashima et al., "Preventing Catastrophic Forgetting by Partial Parameter Update in Continuous Training for End-to-End Speech Recognition," Sound Lectures Autumn, 2022; Ashish Vaswani et al., "Attention Is All You Need," in Proc. IPS, 2017; Ryo Aihara et al., "Deep Clustering-Based Single-Channel Speech Separation and Recent Advances," Acoust. Sci. & Tech. 41, 2, 2020 Ethan Perez et al. “FiLM: Visual Reasoning with a General Conditioning Layer”, in proc. AAAI, 20018
[0005] In the partial update proposed in Non-Patent Document 1, signals, which are training data, are associated with labels, and supervised learning is performed. However, the task of associating the signals with labels is performed manually. This places a heavy burden on the human beings. Therefore, the method proposed in Non-Patent Document 1 cannot be said to be optimal.
[0006] The goal of this disclosure is to prevent catastrophic forgetting without using labels.
[0007] According to one aspect of the present disclosure, there is provided a learning device, wherein the learning device performs learning in an application domain, which is a subsequent learning environment of an original domain, which is a learning environment. The learning device includes an acquisition unit that acquires a first clean signal, a mixed signal that is a signal obtained by mixing the first clean signal with a noise signal, and an original-domain supervised-trained model that is a trained model obtained by training in the original domain; a first extraction unit that uses the first clean signal to extract clean features that are features of the first clean signal; a first estimation unit that uses the original-domain supervised-trained model and the clean features to estimate a supervised vector representation that is a vector representation of information in which the clean features are aggregated; a second extraction unit that uses the mixed signal to extract mixed features that are features of the mixed signal; a student learning model whose initial state is the same as the original-domain supervised-trained model and a second estimation unit that uses the mixed features to estimate a student vector representation that is a vector representation of information in which the mixed features are aggregated; a calculation unit that calculates a value based on the supervised vector representation and the student vector representation; and a learning unit that uses the value to train the student learning model so that the estimation of the student learning model approaches the estimation of the original-domain supervised-trained model.
[0008] According to the present disclosure, catastrophic forgetting can be prevented without using labels.
[0009] 1 is a diagram illustrating a signal processing system according to a first embodiment; 2 is a diagram illustrating hardware included in a learning device according to the first embodiment; 3 is a block diagram illustrating functions of the learning device according to the first embodiment; 4 is a flowchart illustrating an example of processing performed by the learning device according to the first embodiment; 5 is a block diagram illustrating functions of an estimation device according to the first embodiment; 6 is a flowchart illustrating an example of processing performed by the estimation device according to the first embodiment; 7 is a block diagram illustrating functions of a learning device according to a second embodiment; 8 is a flowchart illustrating an example of processing performed by the learning device according to the second embodiment; 9 is a block diagram illustrating functions of a learning device according to a third embodiment; 10 is a flowchart illustrating an example of processing performed by the learning device according to the third embodiment; 11 is a block diagram illustrating functions of an estimation device according to the third embodiment; 12 is a flowchart illustrating an example of processing performed by the estimation device according to the third embodiment; 13 is a block diagram illustrating functions of a learning device according to a fourth embodiment; 14 is a flowchart illustrating an example of processing performed by the learning device according to the fourth embodiment; 15 is a block diagram illustrating functions of an estimation device according to the fourth embodiment; 16 is a flowchart illustrating an example of processing performed by the learning device according to the fourth embodiment; 17 is a block diagram illustrating functions of an estimation device according to the fourth embodiment; 18 is a flowchart illustrating an example of processing performed by the estimation device according to the fourth embodiment; 19 is a block diagram illustrating functions of an estimation device according to the fourth embodiment; 20 is a flowchart illustrating an example of processing performed by the estimation device according to the fourth embodiment;
[0010] Hereinafter, embodiments will be described with reference to the drawings. The following embodiments are merely examples, and various modifications are possible within the scope of the present disclosure.
[0011] 1 is a diagram showing a signal processing system according to embodiment 1. The signal processing system includes a learning device 100 and an estimation device 200. The learning device 100 is a device that executes a learning method.
[0012] Next, the hardware of the learning device 100 will be described. Fig. 2 is a diagram showing the hardware of the learning device of embodiment 1. The learning device 100 is a computer. The learning device 100 has a processor 101, a volatile storage device 102, and a non-volatile storage device 103.
[0013] The processor 101 controls the entire learning device 100. For example, the processor 101 is a central processing unit (CPU) or a field programmable gate array (FPGA). The processor 101 may be a multiprocessor. The learning device 100 may also include a processing circuit.
[0014] The volatile memory device 102 is the main memory device of the learning device 100. For example, the volatile memory device 102 is a RAM (Random Access Memory). The non-volatile memory device 103 is an auxiliary memory device of the learning device 100. For example, the non-volatile memory device 103 is a HDD (Hard Disk Drive) or an SSD (Solid State Drive). The memory area secured in the volatile memory device 102 or the non-volatile memory device 103 is called a memory unit. Like the learning device 100, the estimation device 200 has a processor, a volatile memory device, and a non-volatile memory device.
[0015] The learning phase and the utilization phase will be described below. In the learning phase, the learning device 100 will be described. In the utilization phase, the estimation device 200 will be described. <Learning Phase>
[0016] Next, the functions of the learning device 100 will be described. FIG. 3 is a block diagram showing the functions of the learning device of embodiment 1. The learning device 100 has an acquisition unit 110, a mixing unit 120, an extraction unit 130, an estimation unit 140, an extraction unit 150, an estimation unit 160, a calculation unit 170, a learning unit 180, and an output unit 190. The extraction unit 130 is also referred to as a first extraction unit. The estimation unit 140 is also referred to as a first estimation unit. The extraction unit 150 is also referred to as a second extraction unit. The estimation unit 160 is also referred to as a second estimation unit.
[0017] Some or all of the acquisition unit 110, the mixing unit 120, the extraction unit 130, the estimation unit 140, the extraction unit 150, the estimation unit 160, the calculation unit 170, the learning unit 180, and the output unit 190 may be realized by a processing circuit. Also, some or all of the acquisition unit 110, the mixing unit 120, the extraction unit 130, the estimation unit 140, the extraction unit 150, the estimation unit 160, the calculation unit 170, the learning unit 180, and the output unit 190 may be realized as program modules executed by the processor 101. For example, the program executed by the processor 101 is also referred to as a learning program. For example, the learning program is recorded on a recording medium.
[0018] The acquisition unit 110 acquires the clean signal and the mixed signal. For example, the acquisition unit 110 acquires the clean signal and the mixed signal from a storage unit. Also, for example, the acquisition unit 110 acquires the clean signal and the mixed signal from an external device. Note that the external device is a device that exists outside the learning device 100. For example, the external device is a cloud server. Also, the external device is not shown in the drawing.
[0019] The clean signal is a signal that does not include a noise signal. The clean signal is also called a first clean signal. The mixed signal is a signal that is a mixture of the clean signal and a noise signal.
[0020] Here, as will be described later, the acquiring unit 110 may acquire the mixed signal generated by the mixing unit 120. Furthermore, when the acquiring unit 110 acquires the mixed signal from a storage unit or an external device, the learning device 100 does not need to include the mixing unit 120.
[0021] Furthermore, the acquisition unit 110 acquires the original domain supervised trained model 11 from a storage unit or an external device. The original domain supervised trained model 11 is a trained model obtained by training in the original domain. The original domain supervised trained model 11 is a neural network composed of multiple layers. The original domain supervised trained model 11 may use a method similar to LSTM (Long Short-Term Memory), a method combining one-dimensional convolution operations, or the Transformer described in Non-Patent Document 2. The number of layers is not limited.
[0022] The initial state of the student learning model 12 depicted in FIG. 3 is the same as the original domain supervised trained model 11. As will be described later, the student learning model 12 learns. As a result of learning, the student learning model 12 becomes different from the original domain supervised trained model 11. Furthermore, learning is performed in a learning environment after the original domain. Therefore, the learning environment of the learning performed by the learning device 100 is the application domain. Furthermore, the application domain is an environment different from the original domain.
[0023] The mixer 120 generates a mixed signal using the clean signal and the noise signal. Note that the noise signal is stored in a storage unit or an external device, for example.
[0024] The extraction unit 130 uses the clean signal to extract clean features, which are features of the clean signal. For example, the extraction unit 130 extracts, as the clean features, a time series of power spectra obtained by performing a short-term Fourier transform (STFT) on the clean signal. The clean features are also referred to as first clean features.
[0025] The estimation unit 140 estimates a teacher vector representation using the original domain supervised-trained model 11 and the clean features. In detail, when the estimation unit 140 inputs the clean features to the original domain supervised-trained model 11, the original domain supervised-trained model 11 outputs a teacher vector representation.
[0026] Here, the teacher vector representation is a vector representation of information in which clean features are aggregated. Note that aggregation can also be rephrased as degeneration. Therefore, the teacher vector representation can be expressed as a vector representation of information in which clean features are degenerated.
[0027] The extraction unit 150 extracts a mixed feature, which is a feature of the mixed signal, by using the mixed signal. For example, the extraction unit 150 extracts, as the mixed feature, a time series of a power spectrum obtained by performing a short-time Fourier transform on the mixed signal.
[0028] The estimation unit 160 estimates a student vector representation using the student learning model 12 and the mixed feature. In particular, when the estimation unit 160 inputs the mixed feature to the student learning model 12, the student learning model 12 outputs a student vector representation. Note that the student vector representation is a vector representation of information in which the mixed feature is aggregated.
[0029] The calculation unit 170 calculates a value based on the teacher vector representation and the student vector representation. Specifically, the calculation unit 170 calculates the value L using a loss function expressed by Equation (1). p Calculate.
[0030]
[0031] h N ^t is the teacher vector representation. N ^s is a student vector representation. "^" (hat) is a symbol indicating exponentiation. When "P=1", the value L 1 represents the L1 norm. When "P=2", the value L 2 represents the L2 norm.
[0032] The loss function is described in Non-Patent Document 3. p may be called an error or loss.
[0033] The learning unit 180 adjusts the calculated value (i.e., the value L p ) to train the student learning model 12. In other words, the learning unit 180 uses the value to train the student learning model 12 so that the vector representation output by the student learning model 12 approaches the vector representation output by the original domain teacher-trained model 11. For example, the learning unit 180 executes processing using the student learning model 12, the calculated value, and an optimization method such as Adam (Adaptive moment), and then adjusts the weight coefficient of the student learning model 12 based on backpropagation. Also, for example, the learning unit 180 trains the student learning model 12 so that the calculated value becomes equal to or less than a predetermined threshold.
[0034] The output unit 190 outputs the clean features, the teacher vector representation, the mixed features, the student vector representation, and the values calculated by the calculation unit 170. For example, the output unit 190 outputs the clean features and the like to a display connectable to the learning device 100. For example, by outputting the clean features and the like to a display, the user can recognize the learning status.
[0035] Next, the processing executed by the learning device 100 will be described using a flowchart. FIG. 4 is a flowchart showing an example of processing executed by the learning device of embodiment 1. (Step S11) The acquisition unit 110 acquires a clean signal, a mixed signal, and the original domain supervised trained model 11. (Step S12) The extraction unit 130 extracts clean features using the clean signal. (Step S13) The estimation unit 140 estimates a teacher vector representation using the original domain supervised trained model 11 and the clean features. (Step S14) The extraction unit 150 extracts mixed features using the mixed signal.
[0036] (Step S15) The estimation unit 160 estimates a student vector representation using the student learning model 12 and the mixed features. (Step S16) The calculation unit 170 calculates a value based on the teacher vector representation and the student vector representation. (Step S17) The learning unit 180 uses the calculated values to learn the student learning model 12. (Step S18) The output unit 190 outputs the clean features, the teacher vector representation, the mixed features, the student vector representation, and the values calculated by the calculation unit 170. Note that the execution order of steps S12 to S15 may be different from that of FIG. 4.
[0037] 4 using different clean signals and different mixed signals. For example, the learning device 100 repeatedly learns the student learning model 12 until the value output by the calculation unit 170 becomes equal to or less than a predetermined threshold. In this way, the student learning model 12 is learned multiple times. For example, after learning is completed, the learning device 100 transmits the student learning model 12 to the estimation device 200.
[0038] According to the first embodiment, the learning device 100 learns the student learning model 12 in the application domain. Therefore, the influence of the application domain is reflected in the estimation of the student learning model 12. Furthermore, the learning device 100 learns the student learning model 12 so that the estimation of the student learning model 12 approaches the estimation of the original domain teacher-trained model 11. Therefore, the estimation of the student learning model 12 is influenced by the original domain. Furthermore, the learning device 100 performs learning without using labels. Therefore, the learning device 100 can prevent catastrophic forgetting without using labels.
[0039] Next, an example of the utilization phase in Embodiment 1 will be described. <Utilization Phase> Fig. 5 is a block diagram showing the functions of the estimation device in Embodiment 1. The estimation device 200 includes an acquisition unit 210, an extraction unit 220, an estimation unit 230, and an estimation unit 240.
[0040] Some or all of the acquiring unit 210, the extracting unit 220, the estimating unit 230, and the estimating unit 240 may be realized by a processing circuit included in the estimating device 200. Alternatively, some or all of the acquiring unit 210, the extracting unit 220, the estimating unit 230, and the estimating unit 240 may be realized as program modules executed by a processor included in the estimating device 200. The acquiring unit 210 acquires the mixed signal. For example, the acquiring unit 210 acquires the mixed signal from a volatile storage device or a non-volatile storage device included in the estimating device 200. Alternatively, for example, the acquiring unit 210 acquires the mixed signal from an external device.
[0041] The acquisition unit 210 acquires the student learning model 12 from the learning device 100. Here, the student learning model 12 may be referred to as an encoding neural network. The acquisition unit 210 acquires the trained model 21. For example, the acquisition unit 210 acquires the trained model 21 from a volatile storage device or a non-volatile storage device included in the estimation device 200. Furthermore, for example, the acquisition unit 210 acquires the trained model 21 from an external device. Here, the trained model 21 may be referred to as a decoding neural network.
[0042] The extraction unit 220 extracts a mixed feature, which is a feature of the mixed signal, by using the mixed signal. For example, the extraction unit 220 extracts, as the mixed feature, a time series of a power spectrum obtained by performing a short-time Fourier transform on the mixed signal.
[0043] The estimation unit 230 estimates a vector representation using the student learning model 12 and the mixed feature. In particular, when the estimation unit 230 inputs the mixed feature to the student learning model 12, the student learning model 12 outputs the vector representation. Note that the vector representation is a vector representation of information in which the mixed feature is aggregated.
[0044] The estimation unit 240 estimates a label using the trained model 21 and the vector representation. The label is information estimated based on information in which mixed features are aggregated. For example, if the mixed signal is speech and the estimation device 200 performs speech recognition, the label is a character string indicating the content of the speech. Furthermore, for example, if the mixed signal is speech and the estimation device 200 performs emotion estimation, the label is information indicating the emotion estimated from the speech.
[0045] Next, the processing executed by the estimation device 200 will be described using a flowchart. FIG. 6 is a flowchart showing an example of the processing executed by the estimation device of embodiment 1. (Step S21) The acquisition unit 210 acquires a mixed signal, a student learning model 12, and a trained model 21. (Step S22) The extraction unit 220 extracts mixed features using the mixed signal. (Step S23) The estimation unit 230 estimates a vector representation using the student learning model 12 and the mixed features. (Step S24) The estimation unit 240 estimates a label using the trained model 21 and the vector representation.
[0046] Second Embodiment Next, a second embodiment will be described. In the second embodiment, differences from the first embodiment will be mainly described. In the second embodiment, descriptions of the commonalities between the first embodiment and the second embodiment will be omitted.
[0047] <Learning Phase> Fig. 7 is a block diagram showing the functions of the learning device according to the second embodiment. The acquisition unit 110 acquires weights 13 corresponding to noise signals included in the mixed signal. Specifically, the acquisition unit 110 acquires weights 13 from a storage unit or an external device. For example, if the noise signal is the sound of running water, the weights 13 are weights corresponding to the sound of running water. For example, if the noise signal is the sound of a car running, the weights 13 are weights corresponding to the sound of a car running.
[0048] The calculation unit 170 calculates a value based on the weight 13, the teacher vector representation, and the student vector representation. Specifically, the calculation unit 170 calculates the value L using a loss function expressed by Equation (2). p Calculate.
[0049]
[0050] The weight 13 is λ x λ x is the weight corresponding to the x-th noise signal among the Y types of noise signals. xN ^s is the student vector representation estimated using the mixture features of the mixture signal including the x-th noise signal. xN ^t is the teacher vector expression corresponding to the student vector expression. "^" (hat) is a symbol indicating exponentiation.
[0051] Next, the processing executed by the learning device 100 will be described using a flowchart. Fig. 8 is a flowchart showing an example of processing executed by the learning device of embodiment 2. The processing in Fig. 8 differs from the processing in Fig. 4 in that steps S11a and 16a are executed. Therefore, steps S11a and 16a will be described in Fig. 8. Further, a description of the processing other than steps S11a and 16a will be omitted.
[0052] (Step S11a) The acquisition unit 110 acquires the clean signal, the mixed signal, the original domain supervised trained model 11, and the weights 13. (Step S16a) The calculation unit 170 calculates a value based on the weights 13, the teacher vector representation, and the student vector representation.
[0053] According to the second embodiment, the learning device 100 can appropriately perform learning according to the noise signal by using the weight 13 .
[0054] Embodiment 3 Next, embodiment 3 will be described. In embodiment 3, differences from embodiment 1 will be mainly described. Furthermore, in embodiment 3, descriptions of matters common to embodiment 1 will be omitted.
[0055] <Learning phase>
[0056] 9 is a block diagram showing the functions of the learning device according to the third embodiment. The learning device 100 further includes an extraction unit 191 and an estimation unit 192. The extraction unit 191 is also referred to as a third extraction unit. The estimation unit 192 is also referred to as a third estimation unit. Part or all of the extraction unit 191 and the estimation unit 192 may be implemented by a processing circuit. Alternatively, part or all of the extraction unit 191 and the estimation unit 192 may be implemented as program modules executed by the processor 101.
[0057] The acquisition unit 110 acquires a noise signal from a storage unit or an external device. The noise signal is the same as the noise signal contained in the mixed signal. The acquisition unit 110 acquires a noise-trained model 14 from the storage unit or an external device.
[0058] The extraction unit 191 uses a noise signal to extract noise features, which are features of the noise signal. For example, the extraction unit 191 extracts, as the noise feature, a time series of a power spectrum obtained by performing a short-time Fourier transform on the noise signal. The estimation unit 192 uses the noise-learned model 14 and the noise feature to estimate a noise vector representation. In detail, when the estimation unit 192 inputs the noise feature to the noise-learned model 14, the noise-learned model 14 outputs the noise vector representation. Note that the noise vector representation is a vector representation of information in which the noise feature is aggregated.
[0059] The estimation unit 160 estimates a student vector representation using the student learning model 12, the mixed feature, and the noise vector representation. In particular, when the estimation unit 160 inputs the mixed feature and the noise vector representation to the student learning model 12, the student learning model 12 outputs a student vector representation. Here, by inputting the noise vector representation, the student learning model 12 can identify which information in the mixed feature is related to the noise signal. Therefore, the student learning model 12 estimates the student vector representation while identifying information related to the noise signal.
[0060] Furthermore, the estimation unit 160 may use the method described in Non-Patent Document 4 when inputting the mixed feature and the noise vector representation to the student learning model 12.
[0061] Next, the processing executed by the learning device 100 will be described using a flowchart. Fig. 10 is a flowchart showing an example of processing executed by the learning device of embodiment 3. The processing in Fig. 10 differs from the processing in Fig. 4 in that steps S11b, 14a, 14b, and 15a are executed. Therefore, steps S11b, 14a, 14b, and 15a will be described in Fig. 10. Further, a description of processing other than steps S11b, 14a, 14b, and 15a will be omitted.
[0062] (Step S11b) The acquisition unit 110 acquires the clean signal, the mixed signal, the original domain teacher-trained model 11, the noise signal, and the noise-trained model 14. (Step S14a) The extraction unit 191 extracts noise features using the noise signal. (Step S14b) The estimation unit 192 estimates a noise vector representation using the noise-trained model 14 and the noise features. (Step S15a) The estimation unit 160 estimates a student vector representation using the student learning model 12, the mixed features, and the noise vector representation. Note that the execution order of steps S12 to S15a may be different from that of FIG. 10.
[0063] According to the third embodiment, the robustness of the student learning model 12 is improved. Furthermore, the student learning model 12 can estimate noise more accurately through learning.
[0064] Next, an example of the utilization phase in embodiment 3 will be described. <Utilization Phase> Fig. 11 is a block diagram showing the functions of the estimation device according to embodiment 3. The estimation device 200 further includes a detection unit 250, an extraction unit 260, and an estimation unit 270.
[0065] Some or all of the detection unit 250, extraction unit 260, and estimation unit 270 may be realized by a processing circuit included in the estimation device 200. Furthermore, some or all of the detection unit 250, extraction unit 260, and estimation unit 270 may be realized as program modules executed by a processor included in the estimation device 200.
[0066] The acquisition unit 210 further acquires the noise-trained model 14 from a volatile storage device or a non-volatile storage device included in the estimation device 200 .
[0067] The detection unit 250 detects a noise signal included in the mixed signal. For example, the detection unit 250 detects the noise signal using the method described in Patent Document 1. Furthermore, for example, the detection unit 250 detects the noise signal using the power of the mixed signal and a threshold value.
[0068] The extraction unit 260 uses the noise signal to extract noise features, which are feature quantities of the noise signal. For example, the extraction unit 260 extracts, as the noise features, a time series of power spectra obtained by performing a short-time Fourier transform on the noise signal.
[0069] The estimation unit 270 estimates a noise vector representation using the noise-learned model 14 and the noise feature. In particular, when the estimation unit 270 inputs the noise feature to the noise-learned model 14, the noise-learned model 14 outputs a noise vector representation. Note that the noise vector representation is a vector representation of information in which the noise feature is aggregated.
[0070] The estimation unit 230 estimates a vector representation using the student learning model 12, the mixed feature, and the noise vector representation. In particular, when the estimation unit 230 inputs the mixed feature and the noise vector representation to the student learning model 12, the student learning model 12 outputs the vector representation.
[0071] Next, the processing executed by the estimation device 200 will be described using a flowchart. Fig. 12 is a flowchart showing an example of the processing executed by the estimation device of embodiment 3. The processing in Fig. 12 differs from the processing in Fig. 6 in that steps S21a, 22a, 22b, 22c, and 23a are executed. Therefore, in Fig. 12, steps S21a, 22a, 22b, 22c, and 23a will be described. Further, descriptions of the processing other than steps S21a, 22a, 22b, 22c, and 23a will be omitted.
[0072] (Step S21a) The acquisition unit 210 acquires the mixed signal, the student learning model 12, the trained model 21, and the noise trained model 14. (Step S22a) The detection unit 250 detects a noise signal contained in the mixed signal. (Step S22b) The extraction unit 260 extracts noise features using the noise signal. (Step S22c) The estimation unit 270 estimates a noise vector representation using the noise trained model 14 and the noise features. (Step S23a) The estimation unit 230 estimates a vector representation using the student learning model 12, the mixed features, and the noise vector representation.
[0073] Embodiment 4 Next, embodiment 4 will be described. In embodiment 4, differences from embodiment 1 will be mainly described. Furthermore, in embodiment 4, description of matters common to embodiment 1 will be omitted.
[0074] <Learning Phase> Fig. 13 is a block diagram showing the functions of the learning device according to the fourth embodiment. The learning device 100 further includes an extraction unit 193 and an estimation unit 194. The extraction unit 193 is also referred to as a fourth extraction unit. The estimation unit 194 is also referred to as a fourth estimation unit. Part or all of the extraction unit 193 and the estimation unit 194 may be implemented by a processing circuit. Alternatively, part or all of the extraction unit 193 and the estimation unit 194 may be implemented as program modules executed by the processor 101.
[0075] Here, the clean signal input to the extraction unit 130 is referred to as a first clean signal. The acquisition unit 110 acquires a second clean signal from a storage unit or an external device. The second clean signal is a signal different from the first clean signal. For example, the first clean signal and the second clean signal are sound signals uttered by the same speaker. The acquisition unit 110 acquires the clean trained model 15 from a storage unit or an external device.
[0076] The extracting unit 193 uses the second clean signal to extract a second clean feature that is a feature of the second clean signal. For example, the extracting unit 193 extracts, as the second clean feature, a time series of a power spectrum obtained by performing a short-time Fourier transform on the second clean signal.
[0077] The estimation unit 194 estimates a clean vector representation using the clean trained model 15 and the second clean feature. In particular, when the estimation unit 194 inputs the second clean feature to the clean trained model 15, the clean trained model 15 outputs a clean vector representation. Note that the clean vector representation is a vector representation of information in which the second clean feature is aggregated.
[0078] The estimation unit 160 estimates a student vector representation using the student learning model 12, the mixed features, and the clean vector representation. In particular, when the estimation unit 160 inputs the mixed features and the clean vector representation to the student learning model 12, the student learning model 12 outputs a student vector representation. Here, by receiving the clean vector representation, the student learning model 12 can identify which information among the mixed features is related to the clean signal. Therefore, the student learning model 12 estimates the student vector representation while identifying information related to the clean signal.
[0079] Furthermore, the estimation unit 160 may use the method described in Non-Patent Document 4 when inputting the mixed feature and the clean vector representation to the student learning model 12.
[0080] Next, the processing executed by the learning device 100 will be described using a flowchart. Fig. 14 is a flowchart showing an example of processing executed by the learning device of embodiment 4. The processing in Fig. 14 differs from the processing in Fig. 4 in that steps S11c, 14c, 14d, and 15b are executed. Therefore, steps S11c, 14c, 14d, and 15b will be described in Fig. 14. Further, a description of processing other than steps S11c, 14c, 14d, and 15b will be omitted.
[0081] (Step S11c) The acquisition unit 110 acquires the first clean signal, the mixed signal, the original-domain teacher-trained model 11, the second clean signal, and the clean trained model 15. (Step S14c) The extraction unit 193 extracts second clean features using the second clean signal. (Step S14d) The estimation unit 194 estimates a clean vector representation using the clean trained model 15 and the second clean features. (Step S15b) The estimation unit 160 estimates a student vector representation using the student learning model 12, the mixed features, and the clean vector representation. Note that the execution order of steps S12 to S15b may be different from that of FIG. 14.
[0082] According to the fourth embodiment, the robustness of the student learning model 12 is improved. Furthermore, the student learning model 12 can estimate a clean signal more accurately through learning. Furthermore, the learning device 100 uses different clean signals to train the student learning model 12. Therefore, even if different speech contents are input, the student learning model 12 can more easily estimate which is the clean signal.
[0083] Next, an example of the utilization phase will be shown. <Utilization Phase> Fig. 15 is a block diagram showing the functions of the estimation device according to embodiment 4. The estimation device 200 further includes an extraction unit 280 and an estimation unit 290.
[0084] A part or all of the extraction unit 280 and the estimation unit 290 may be realized by a processing circuit included in the estimation device 200. Alternatively, a part or all of the extraction unit 280 and the estimation unit 290 may be realized as a program module executed by a processor included in the estimation device 200.
[0085] The acquisition unit 210 further acquires the clean trained model 15 from a volatile storage device or a non-volatile storage device included in the estimation device 200. The acquisition unit 210 further acquires a clean signal from the volatile storage device or the non-volatile storage device included in the estimation device 200. Note that this clean signal is different from the clean signal included in the mixed signal acquired by the acquisition unit 210.
[0086] The extraction unit 280 extracts clean features, which are features of the clean signal, by using a clean signal different from the clean signal contained in the mixed signal. For example, the extraction unit 280 extracts, as the clean features, a time series of power spectra obtained by performing a short-time Fourier transform on the clean signal.
[0087] The estimation unit 290 estimates a clean vector representation using the clean trained model 15 and the clean features. In particular, when the estimation unit 290 inputs the clean features to the clean trained model 15, the clean trained model 15 outputs a clean vector representation. Note that the clean vector representation is a vector representation of information in which the clean features are aggregated.
[0088] The estimation unit 230 estimates a vector representation using the student learning model 12, the mixed feature, and the clean vector representation. In particular, when the estimation unit 230 inputs the mixed feature and the clean vector representation to the student learning model 12, the student learning model 12 outputs the vector representation.
[0089] Next, the processing executed by the estimation device 200 will be described using a flowchart. Fig. 16 is a flowchart showing an example of the processing executed by the estimation device of embodiment 4. The processing in Fig. 16 differs from the processing in Fig. 6 in that steps S21b, 22d, 22e, and 23b are executed. Therefore, steps S21b, 22d, 22e, and 23b will be described in Fig. 16. Further, a description of the processing other than steps S21b, 22d, 22e, and 23b will be omitted.
[0090] (Step S21b) The acquisition unit 210 acquires the mixed signal, the student learning model 12, the trained model 21, the clean signal, and the clean trained model 15. (Step S22d) The extraction unit 280 extracts clean features using the clean signal. (Step S22e) The estimation unit 290 estimates a clean vector representation using the clean trained model 15 and the clean features. (Step S23b) The estimation unit 230 estimates a vector representation using the student learning model 12, the mixed features, and the clean vector representation.
[0091] The features of the above-described embodiments can be combined with each other as appropriate.
[0092] 11 Original domain teacher trained model, 12 Student trained model, 13 Weights, 14 Noise trained model, 15 Clean trained model, 21 Trained model, 100 Learning device, 101 Processor, 102 Volatile storage device, 103 Non-volatile storage device, 110 Acquisition unit, 120 Mixing unit, 130 Extraction unit, 140 Estimation unit, 150 Extraction unit, 160 Estimation unit, 170 Calculation unit, 180 Learning unit, 190 Output unit, 191 Extraction unit, 192 Estimation unit, 193 Extraction unit, 194 Estimation unit, 200 Estimation device, 210 Acquisition unit, 220 Extraction unit, 230 Estimation unit, 240 Estimation unit, 250 Detection unit, 260 Extraction unit, 270 estimation section, 280 extraction section, 290 estimation section.
Claims
1. A learning device that performs learning in an application domain, which is a learning environment subsequent to an original domain, which is a learning environment, comprising: an acquisition unit that acquires a first clean signal, a mixed signal that is a signal obtained by mixing the first clean signal and a noise signal, and an original domain teacher-trained model that is a trained model obtained by learning in the original domain; a first extraction unit that uses the first clean signal to extract clean features that are features of the first clean signal; a first estimation unit that uses the original domain teacher-trained model and the clean features to estimate a teacher vector representation that is a vector representation of information in which the clean features are aggregated; a second extraction unit that uses the mixed signal to extract mixed features that are features of the mixed signal; a second estimation unit that uses a student learning model whose initial state is the same as that of the original domain teacher-trained model and the mixed features to estimate a student vector representation that is a vector representation of information in which the mixed features are aggregated; and a calculation unit that calculates a value based on the teacher vector representation and the student vector representation. a learning unit that uses the value to train the student learning model so that the estimate of the student learning model approaches an estimate of the original domain teacher-trained model.
2. The learning device according to claim 1, wherein the acquisition unit acquires a weight corresponding to the noise signal contained in the mixed signal, and the calculation unit calculates a value based on the weight, the teacher vector representation, and the student vector representation.
3. The learning device described in claim 1, further comprising: a third extraction unit; and a third estimation unit, wherein the acquisition unit acquires the noise signal and a noise learned model, the third extraction unit uses the noise signal to extract noise features that are features of the noise signal, the third estimation unit uses the noise learned model and the noise features to estimate a noise vector representation that is a vector representation of information in which the noise features are aggregated, and the second estimation unit estimates the student vector representation using the student learning model, the mixed features, and the noise vector representation.
4. The learning device of claim 1, further comprising: a fourth extraction unit; and a fourth estimation unit, wherein the acquisition unit acquires a second clean signal and a clean learned model; the fourth extraction unit uses the second clean signal to extract second clean features which are features of the second clean signal; the fourth estimation unit uses the clean learned model and the second clean features to estimate a clean vector representation which is a vector representation of information in which the second clean features are aggregated; and the second estimation unit estimates the student vector representation using the student learning model, the mixed features, and the clean vector representation.
5. The learning device according to claim 1 , further comprising an output unit that outputs the clean features, the teacher vector representation, the mixed features, the student vector representation, and the value.
6. A learning method, comprising: a learning device that performs learning in an application domain, which is a learning environment subsequent to an original domain that is a learning environment, acquiring a first clean signal, a mixed signal that is a signal obtained by mixing the first clean signal and a noise signal, and an original domain teacher-trained model that is a trained model obtained by learning in the original domain, extracting clean features that are features of the first clean signal using the first clean signal, estimating a teacher vector representation that is a vector representation of information in which the clean features are aggregated using the original domain teacher-trained model and the clean features, extracting mixed features that are features of the mixed signal using the mixed signal, estimating a student vector representation that is a vector representation of information in which the mixed features are aggregated using a student model whose initial state is the same as that of the original domain teacher-trained model and the mixed features, calculating a value based on the teacher vector representation and the student vector representation, and training the student learning model using the value so that the estimate of the student learning model approaches the estimate of the original domain teacher-trained model.
7. A learning program that causes a computer that performs learning in an application domain, which is a learning environment subsequent to an original domain, to execute processes of acquiring a first clean signal, a mixed signal that is a signal obtained by mixing the first clean signal and a noise signal, and an original domain teacher-trained model that is a trained model obtained by learning in the original domain, extracting clean features that are features of the first clean signal using the first clean signal, estimating a teacher vector representation that is a vector representation of information in which the clean features are aggregated using the original domain teacher-trained model and the clean features, extracting mixed features that are features of the mixed signal using the mixed signal, estimating a student vector representation that is a vector representation of information in which the mixed features are aggregated using a student learning model whose initial state is the same as that of the original domain teacher-trained model and the mixed features, calculating a value based on the teacher vector representation and the student vector representation, and training the student learning model using the value so that the estimate of the student learning model approaches the estimate of the original domain teacher-trained model.
Citation Information
Patent Citations
Speech segment detection device and method for detecting speech segment
WO2016143125A1
Patient-specific deep learning image denoising methods and systems
JP2020064609A
Sound inspection system and control method
JP2021135585A
Audio data processing method, device, electronic device, and computer program
JP2022527527A
