Learning device, conversion device, learning method, conversion method, and program
The learning device combines distillation and noise removal by using a teacher model and noise addition to update a student model, addressing the limitations of existing methods and enhancing speech representation learning for improved performance in speech processing tasks.
Patent Information
- Application Number
- PCT/JP2023/019030
- Authority / Receiving Office
- WO · WO
- Patent Type
- Applications
- Current Assignee / Owner
- Filing Date
- 2023-05-22
- Publication Date
- 2024-11-28
AI Technical Summary
Current methods do not effectively combine distillation and noise removal in speech representation learning, limiting the performance of speech processing applications.
A learning device and method that incorporates a teacher model, noise addition, and noise removal distillation process to update a student model, allowing it to learn noise-robust representations by minimizing the difference between teacher and student data, thereby combining distillation and noise removal.
The approach enhances the student model's ability to produce noise-robust speech representations, improving performance in tasks like keyword classification and speech emotion recognition compared to existing methods.
Smart Images

Figure JP2023019030_28112024_PF_FP_ABST
Abstract
Description
Learning device, conversion device, learning method, conversion method, and program
[0001] The present invention relates to a learning device, a conversion device, a learning method, a conversion method, and a program.
[0002] Studies have been conducted on learning methods for models that learn from datasets and output data representations that can be applied to various applications. For example, there is a multi-stage learning method called HuBERT, which combines a mask prediction method and clustering, in which the learning process is divided into two stages, and the speech representations output by the learning model in the first stage are clustered to obtain a target representation in advance, and the mask prediction result is made to approach the target representation in the second stage of learning (Non-Patent Document 1). There is also a method called WavLM, which further improves performance by combining noise removal with HuBERT (Non-Patent Document 2).
[0003] On the other hand, there is a method for realizing small-sized speech expression learning by distilling a high-performance trained speech expression learning model (Non-Patent Document 3).
[0004] However, a combined technique of distillation and noise removal has not been realized.
[0005] W.-N. Hsu, et al., “HuBERT: Self-supervised speech representation learning by masked prediction of hidden units,” IEEE / ACM Trans. Audio, Speech, Language Process., p.3451-3460, 2021. S. Chen, et al., “WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processing,” IEEE J. Sel. Top. Signal Process., vol. 16, no. 6, p.1505-1518, 2022.H.-J. Chang, et al., “DistilHuBERT: Speech representation learning by layer-wise distillation of hidden-unit bert,” in ICASSP, 2022, pp. 7087-7091.Jinsen, et al., “Masked Modeling Duo: Learning Representations by Encouraging Both Networks to Model the Input,” to appear at ICASSP, 2023. Kaiming He, et al., "Masked autoencoders are scalable vision learners," arXiv preprint arXiv:2111.06377, 2021.
[0006] The problem that the present invention aims to solve is to combine distillation and noise removal.
[0007] One aspect of the present invention is a learning device that includes a control unit that acquires, through learning, a target representation conversion process, which is a process of converting the representation of data to be converted into a target representation that is a predetermined representation, and the control unit executes a teacher model process that generates teacher data by inputting the data to be converted into a teacher model, a noise addition process that adds noise to the data to be converted, a student model process that generates student data by inputting the data with the noise added into a student model, and a noise removal and distillation process that updates the student model based on the teacher data and the student data, and the student model process at the time when a predetermined condition for ending the update is met is the target representation conversion process.
[0008] One aspect of the present invention is a conversion device comprising: a conversion target acquisition unit that acquires data to be converted into a target expression, which is a predetermined expression; and an expression conversion unit that performs a target expression conversion process that converts the expression of the data to be converted into a target expression, which is a predetermined expression, wherein the target expression conversion process executes a teacher model process that generates teacher data by inputting the data to be converted into a teacher model; a noise addition process that adds noise to the data to be converted; a student model process that generates student data by inputting the data with the noise added into a student model; and a noise removal and distillation process that updates the student model based on the teacher data and the student data, and the student model process is the student model process at the point when a predetermined condition regarding the end of the update is met.
[0009] One aspect of the present invention is a learning method that includes a control step of obtaining, through learning, a target representation conversion process, which is a process of converting the representation of data to be converted into a target representation that is a predetermined representation, wherein the control step executes a teacher model process that generates teacher data by inputting the data to be converted into a teacher model, a noise addition process that adds noise to the data to be converted, a student model process that generates student data by inputting the data with the noise added into a student model, and a noise removal and distillation process that updates the student model based on the teacher data and the student data, and wherein the student model process at the time when a predetermined condition regarding the end of the update is met is the target representation conversion process.
[0010] One aspect of the present invention is a conversion method comprising: a conversion target acquisition step for acquiring data to be converted into a target representation, which is a predetermined representation; and an expression conversion step for performing a target representation conversion process for converting the representation of the data to be converted into a target representation, which is a predetermined representation. The target representation conversion process executes a teacher model process for generating teacher data by inputting the data to be converted into a teacher model; a noise addition process for adding noise to the data to be converted; a student model process for generating student data by inputting the data with the noise added into a student model; and a noise removal and distillation process for updating the student model based on a loss calculated by the student model process and a loss calculated based on the teacher data and the student data, and the student model process is the student model process at the point when a predetermined condition regarding the end of the update is satisfied.
[0011] The present invention allows for the combination of distillation and noise removal.
[0012] 1 is a diagram illustrating an example of the configuration of an expression conversion system according to an embodiment. FIG. 2 is a diagram illustrating an overview of the flow of processing executed by a learning unit according to an embodiment. FIG. 3 is a flowchart illustrating an example of the flow of learning processing executed by a learning unit according to an embodiment. FIG. 4 is a diagram illustrating an overview of the flow of processing by M2D, which is an example of a student model. FIG. 5 is a diagram illustrating an example of the hardware configuration of a learning device according to an embodiment. FIG. 6 is a diagram illustrating an example of the configuration of a control unit according to an embodiment. FIG. 7 is a diagram illustrating an example of the hardware configuration of a conversion device according to an embodiment. FIG. 8 is a diagram illustrating an example of the configuration of a control unit according to an embodiment. FIG. 9 is a flowchart illustrating an example of the flow of processing executed by a conversion device according to an embodiment. FIG. 10 is a diagram illustrating an example of the results of an experiment according to an embodiment.
[0013] 1 is a diagram illustrating an example of the configuration of an expression conversion system 100 according to an embodiment. The expression conversion system 100 includes a learning device 1 and a conversion device 2. The learning device 1 acquires, through learning, a process (hereinafter referred to as a "target expression conversion process") for converting an expression of data to be converted into a predetermined expression (hereinafter referred to as a "target expression").
[0014] The target representation is, for example, a representation embedding. The target representation is, for example, a representation of 768 floating-point values. The target representation may be, for example, a representation of 1024 floating-point values. The target representation may be, for example, a representation of 2048 floating-point values. The representation conversion process is a type of learning model.
[0015] The data to be converted may be any data obtained based on image data, acoustic signal data, natural language data, or general time-series data, as long as it is expressed as a tensor. Therefore, the data to be converted may be, for example, image data, a spectrogram of acoustic signal data, or a series of word samples in which each word is expressed as an M-dimensional vector (M is a natural number greater than or equal to 1) (hereinafter referred to as a "natural language series"). The data to be converted may be, for example, general time-series data in which symbols and numbers other than letters are expressed as vectors.
[0016] A spectrogram of the acoustic signal data is obtained based on the acoustic signal data. A natural language sequence is obtained based on the natural language data.
[0017] The learning device 1 includes a learning unit 10. The learning unit 10 executes a teacher model process, a noise addition process, a student model process, and a noise removal and distillation process.
[0018] The teacher model processing is processing using a teacher model that has learned how to convert data to be converted into a target representation. The teacher model is, for example, HuBERT, as disclosed in Non-Patent Document 1. The teacher model is not limited to HuBERT, and may be any model that learns a clustered target. Teacher model processing generates teacher data from the data to be converted.
[0019] The noise addition process is a process of adding noise to the data to be converted. The ratio of the data to be converted to the noise is set in advance. The noise addition process is a process of mixing, for example, the spectrogram data to be converted with noise at a preset ratio.
[0020] The student model processing is a process of generating student data from data to which noise has been added based on a student model. The student model is, for example, Masked Modeling Duo (M2D) disclosed in Non-Patent Document 4.
[0021] The denoising distillation process is a process that combines distillation and noise removal. The denoising distillation process updates the student model so that the difference between the teacher data and the student data is reduced, thereby causing the student model to learn the target representation conversion process of the teacher model. In the denoising distillation process, the loss between the teacher data and the student data (hereinafter referred to as "teacher-student loss") is calculated as the difference between the teacher data and the student data. The teacher-student loss may be the mean square error (MSE) between the teacher data and the student data, or L1, which is the mean absolute value of the difference. In the denoising distillation process, parameters in the student model process are updated based on the teacher-student loss.
[0022] In the noise removal distillation process, the target representation conversion process of the teacher model is distilled onto the student model, and noise removal is performed using the target representation conversion process of the student model simultaneously. Noise removal is a target representation conversion process that outputs the same target representation when a noisy input is used as when a noise-free input is used. By distilling the target representation conversion process of the teacher model regardless of the noise added to the input, the student model achieves a process that is robust to noise and imitates the teacher model.
[0023] Here, an example of the flow of processing executed by the learning unit 10 will be described using diagrams and formulas. FIG. 2 is a diagram illustrating an outline of the flow of processing executed by the learning unit 10 in the embodiment. Data x to be converted is input to a teacher model, and teacher data z 1 Furthermore, noise is added to the data x and input to the student model, and the student data z 2 is generated.
[0024] Teacher data z 1 and student data z 2 Between teachers and students, loss L int is calculated.
[0025] 3 is a flowchart showing an example of the flow of the learning process executed by the learning unit 10 in the embodiment. The learning unit 10 acquires data to be converted (step S101). Next, the learning unit 10 executes processing using a teacher model (step S102). Next, the learning unit 10 adds noise to the data to be converted (step S103). Next, the learning unit 10 executes processing using a student model (step S104).
[0026] Next, the learning unit 10 executes a noise removal and distillation process (step S105). In the noise removal and distillation process, the teacher-student loss is calculated and the student model is updated. Next, the learning unit 10 determines whether an update termination condition is satisfied (step S106). If the update termination condition is satisfied (step S106: YES), the process ends. On the other hand, if the update termination condition is not satisfied (step S106: NO), the process returns to step S101.
[0027] The processing by the student model at the time when the update termination condition is satisfied is the target representation conversion processing. The processing of steps S102 to S103 and the processing of step S104 may be executed in parallel.
[0028] In the denoising and distillation process, the learning unit 10 may update the student model based on the teacher-student loss as well as the loss calculated in the student model process (hereinafter referred to as "student data loss"). For example, the learning unit 10 calculates an overall loss by combining the teacher-student loss and the student data loss, and updates the student model based on the overall loss. For example, when M2D is used for the student model, the student data loss is the loss due to mask prediction of M2D.
[0029] The learning unit 10 calculates the teacher-student loss L int and student data loss L stu By combining these, the total loss L com Calculate.
[0030] λ int and λ stu is the weighting.
[0031] 4 is a diagram showing an outline of a processing flow by M2D, which is an example of a student model. In the example of FIG. 4, the learning unit 10 generates first data and second data based on data in which noise is added to data x (hereinafter referred to as "noise-added data"). In the example of FIG. 4, data D101 is an example of the first data, and data D102 is an example of the second data.
[0032] Data D101 is data in which a portion of noise-added data has been masked. Masking refers to a process that prevents data to be processed from being processed by a predetermined other process. In other words, masking refers to a process that restricts information access. The predetermined other process is, for example, a mask data representation prediction process. An example of the result of masking processing will be explained by showing an example of how masked data is handled in the next stage. For example, if input data is a series of "010101010101" and is converted by masking into data of the series "****0101****", the data "*" will not be subject to processing in the next stage. This is the result of masking. In this example, "*" will not be subject to processing in the next stage. In other words, in this example, "*" is an example of data with restricted information access.
[0033] In Fig. 4, solid patches represent masked patches. Non-solid patches in Fig. 4 represent non-masked patches. Patch P1 is an example of a solid patch. Patch P2 is an example of a non-solid patch. A collection of masked patches is an example of mask data.
[0034] 4, the mask data in the first data is the non-mask data in the second data, and therefore data including the non-mask data in the first data and the non-mask data in the second data includes noise-added data.
[0035] However, it is not necessary that all of the mask data in the first data be non-mask data in the second data. The non-mask data in the second data may be part of the mask data in the first data.
[0036] The determination of which data of the noise-added data will be used as the mask data in the first data may be performed in any manner, for example, randomly. Hereinafter, the process of determining which data of the noise-added data will be used as the mask data in the first data will be referred to as the first mask data determination process.
[0037] As described above, the second data is, for example, data determined as mask data in the first data. However, as described above, all of the data determined as mask data in the first data does not necessarily have to be second data. An example of how to determine the second data when only some, but not all, of the data determined as mask data in the first data is second data will be described. In such a case, the determination of which of the data determined as mask data in the first data will be used as the mask data may be performed in any manner, for example, randomly.
[0038] Hereinafter, the process of determining which of the data determined as mask data in the first data should be used as mask data will be referred to as the "second mask data determination process." As can be seen from the above explanation, if the second data is all of the data determined as mask data in the first data, the second mask data determination process does not necessarily need to be performed. This is because the data determined as mask data in the first data is the second data.
[0039] In the example of Fig. 4, the first mask data determination process and the second mask data determination process are performed by the learning unit 10. However, the first mask data determination process and the second mask data determination process do not necessarily have to be performed by the learning unit 10. For example, a device other than the learning device 1 may perform the process up to generating the first data and the second data, and the generated first data and second data may be executed by the learning unit 10. In such a case, the learning unit 10 does not perform the first mask data determination process and the second mask data determination process. Note that if the second data is all of the mask data in the first data, the second mask data determination process does not need to be performed.
[0040] In the example of Fig. 4, the learning unit 10 performs a first conversion process on the first data. θ (·) represents the first conversion process. θrepresents the result of the addition process. The addition process is a process of adding mask tokens to the result of the first conversion process on the first data, the number of which is equal to the number of elements belonging to the mask data. The mask token is information indicating whether or not an element in the noise-added data belongs to the mask data.
[0041] The elements belonging to the mask data are elements of the tensor representing the noise-added data that belong to the mask data in the first data. Therefore, the elements belonging to the mask data are, for example, patches that are determined to be mask data in the first mask data determination process.
[0042] In other words, the addition process is a process of adding a vector obtained by adding information indicating the position of patch P1 in the noise-added data and information indicating that it is a mask token to the result of the first conversion process. In this way, the result of the addition process is an example of a result based on the result of the first conversion process. When the addition process is executed as in the example of Figure 4, the addition process is executed by, for example, the learning unit 10.
[0043] In the example of Fig. 4, the learning unit 10 performs a mask data representation prediction process on the result of the addition process. In this way, the addition process is performed after the first representation conversion process and before the mask data representation prediction process. The function q θ (•) represents the mask data representation prediction process. θ represents the result of the mask data representation prediction process.
[0044] In the example of Fig. 4, the learning unit 10 performs the second conversion process on the second data. ξ (·) represents the second transformation process. The function f θ and a function f representing the second conversion process ξ Both of these are parameterized functions, and the parameter values are updated by the noise removal and distillation process described above.
[0045] Function f θ means a function f with a parameter value θ, and the function f ξThe symbol means a function f whose parameter value is ξ. Therefore, the function f θ The symbol and the function f ξ The symbol z' in FIG. 4 indicates that the functions of the first and second transformation processes are the same except for the difference in parameters. ξ represents the result of the second conversion process.
[0046] In the example of FIG. 4, the learning unit 10 uses y θ and z', the result of the second transformation process ξ and the student data loss L in the student model based on stu The student data loss L may be calculated. stu is the teacher-student loss as well as y θ and z' ξ It may be the mean square error (MSE) between the two, or L1, which is the mean absolute value of the differences.
[0047] 5 is a diagram illustrating an example of the hardware configuration of the learning device 1 according to the embodiment. The learning device 1 includes a control unit 11 including a processor 91 such as a CPU (Central Processing Unit) and a memory 92 connected via a bus, and executes a program. By executing the program, the learning device 1 functions as a device including the control unit 11, an input unit 12, a communication unit 13, a storage unit 14, and an output unit 15.
[0048] More specifically, the processor 91 reads the program stored in the storage unit 14 and stores the read program in the memory 92. When the processor 91 executes the program stored in the memory 92, the learning device 1 functions as a device including a control unit 11, an input unit 12, a communication unit 13, a storage unit 14, and an output unit 15.
[0049] The control unit 11 controls the operation of various functional units included in the learning device 1. The control unit 11 includes a learning unit 10. Therefore, the control unit 11 executes, for example, teacher model processing, noise addition processing, student model processing, and noise removal and distillation processing. The control unit 11 may also execute additional processing.
[0050] The control unit 11 controls, for example, the operation of the output unit 15. The control unit 11 records, in the storage unit 14, various pieces of information generated by the execution of various processes, such as teacher model processing, noise addition processing, student model processing, and noise removal and distillation processing.
[0051] The input unit 12 includes input devices such as a mouse, keyboard, and touch panel. The input unit 12 may be configured as an interface that connects these input devices to the study device 1. The input unit 12 accepts input of various information for the study device 1.
[0052] The communication unit 13 includes a communication interface for connecting the learning device 1 to an external device. The communication unit 13 communicates with the external device via wired or wireless communication. The external device is, for example, a device that is the source of the data to be converted. When the external device is the source of the data to be converted, the communication unit 13 acquires the data to be converted by communicating with the device that is the source of the data to be converted.
[0053] The external device is, for example, a device that transmits original data. When the external device is the device that transmits original data, the communication unit 13 acquires the original data by communicating with the device that transmits the original data. The external device is, for example, a device that transmits acoustic signal data.
[0054] The external device is, for example, a device that has transmitted acoustic signal data. When the external device is the device that has transmitted acoustic signal data, the communication unit 13 acquires the acoustic signal data by communicating with the device that has transmitted the acoustic signal data.
[0055] The external device is, for example, a device that has transmitted natural language data. When the external device is a device that has transmitted natural language data, the communication unit 13 acquires the natural language data by communicating with the device that has transmitted the natural language data.
[0056] The external device is a device that transmits general time series data in which symbols and numbers other than characters are expressed as vectors, for example. When the external device is a device that transmits general time series data, the communication unit 13 acquires the general time series data by communicating with the device that transmits the general time series data.
[0057] The various information input to the communication unit 13 may be input to the input unit 12 instead of the communication unit 13 .
[0058] The external device is, for example, the conversion device 2. The communication unit 13 communicates with the conversion device 2 to transmit to the conversion device 2 information indicating the content of the processing by the student model at the time when the update termination condition is satisfied.
[0059] The storage unit 14 is configured using a computer-readable storage medium device (non-transitory computer-readable recording medium) such as a magnetic hard disk drive or a semiconductor storage device. The storage unit 14 stores various information related to the learning device 1. The storage unit 14 stores information input via, for example, the input unit 12 or the communication unit 13. The storage unit 14 stores various information generated by the operation of the control unit 11, for example.
[0060] The output unit 15 outputs various types of information. The output unit 15 includes a display device such as a CRT (Cathode Ray Tube) display, a liquid crystal display, or an organic EL (Electro-Luminescence) display. The output unit 15 may be configured as an interface that connects these display devices to the learning device 1. The output unit 15 outputs information input to the input unit 12, for example. The output unit 15 may also display the results of processing by the control unit 11, for example.
[0061] 6 is a diagram showing an example of the configuration of the control unit 11 in the embodiment. The control unit 11 includes a learning unit 10, a data acquisition unit 110, a storage control unit 120, a communication control unit 130, and an output control unit 140.
[0062] The data acquisition unit 110 acquires data to be transmitted to the learning unit 10. The data to be transmitted to the learning unit 10 may be data to be converted, or may be data to which noise has been added.
[0063] When the input unit 12 or the communication unit 13 acquires acoustic signal data, the data acquisition unit 110 acquires data obtained based on the acoustic signal data acquired by the input unit 12 or the communication unit 13, which is expressed as a tensor, as original data.
[0064] When the input unit 12 or the communication unit 13 acquires natural language data, the data acquisition unit 110 acquires data obtained based on the natural language data acquired by the input unit 12 or the communication unit 13, which is expressed as a tensor, as original data.
[0065] When the input unit 12 or the communication unit 13 acquires general time series data in which symbols and numbers other than letters are expressed as vectors, the data acquisition unit 110 acquires, as original data, data obtained based on the general time series data acquired by the input unit 12 or the communication unit 13 and expressed as tensors.
[0066] The storage control unit 120 records various information in the storage unit 14. The communication control unit 130 controls the operation of the communication unit 13. The output control unit 140 controls the operation of the output unit 15.
[0067] 7 is a diagram showing an example of the hardware configuration of the conversion device 2 according to the embodiment. The conversion device 2 includes a control unit 21 including a processor 93 such as a CPU and a memory 94 connected via a bus, and executes a program. By executing the program, the conversion device 2 functions as a device including the control unit 21, an input unit 22, a communication unit 23, a storage unit 24, and an output unit 25.
[0068] More specifically, the processor 93 reads the program stored in the storage unit 24 and stores the read program in the memory 94. When the processor 93 executes the program stored in the memory 94, the conversion device 2 functions as a device including the control unit 21, the input unit 22, the communication unit 23, the storage unit 24, and the output unit 25.
[0069] The control unit 21 controls the operation of various functional units included in the conversion device 2. The control unit 21 acquires, for example, information obtained by the learning device 1, indicating the content of processing by the student model at the time when the update termination condition is satisfied, and records the information in the storage unit 24.
[0070] The control unit 21 executes processing using the student model. The execution of processing using the student model by the control unit 21 is executed, for example, by the control unit 21 reading and executing information indicating the content of processing using the student model recorded in the storage unit 24. The control unit 21 controls, for example, the operation of the output unit 25. The control unit 21 records, for example, various information generated by the execution of processing using the student model in the storage unit 24.
[0071] The input unit 22 includes input devices such as a mouse, a keyboard, and a touch panel. The input unit 22 may be configured as an interface that connects these input devices to the conversion device 2. The input unit 22 accepts input of various information to the conversion device 2.
[0072] The communication unit 23 includes a communication interface for connecting the conversion device 2 to an external device. The communication unit 23 communicates with the external device via wired or wireless communication. The external device is, for example, a device that transmits data to be converted into a target expression. The communication unit 23 acquires data to be converted into a target expression through communication with such an external device. The external device is, for example, the learning device 1. The communication unit 23 acquires information indicating the content of the target expression conversion process through communication with the learning device 1.
[0073] The various information input to the communication unit 23 may be input to the input unit 22 instead of the communication unit 23 .
[0074] The storage unit 24 is configured using a computer-readable storage medium device (non-transitory computer-readable recording medium) such as a magnetic hard disk device or a semiconductor storage device. The storage unit 24 stores various information related to the conversion device 2. The storage unit 24 stores information input via, for example, the input unit 22 or the communication unit 23. The storage unit 24 stores various information generated by, for example, the operation of the control unit 21. The storage unit 24 stores, for example, the contents of the target expression conversion process.
[0075] The output unit 25 outputs various types of information. The output unit 25 is, for example, a communication interface connected to be able to communicate with a device that executes a downstream task. The output unit 25 may be configured to include a display device such as a CRT display, a liquid crystal display, or an organic EL display. The output unit 25 may be configured as an interface that connects these display devices to the conversion device 2. The output unit 25 outputs, for example, information input to the input unit 22. The output unit 25 may also output, for example, the results of processing performed by a student model.
[0076] 8 is a diagram showing an example of the configuration of the control unit 21 in the embodiment. The control unit 21 includes a conversion target acquisition unit 210, an expression conversion unit 220, a storage control unit 230, a communication control unit 240, and an output control unit 250. The conversion target acquisition unit 210 acquires data that is input to the communication unit 23 and is the target of conversion of an expression into a target expression.
[0077] The expression conversion unit 220 converts the expression of the data acquired by the conversion object acquisition unit 210 using a target expression conversion process.
[0078] The storage control unit 230 records various information in the storage unit 24. The communication control unit 240 controls the operation of the communication unit 23. The output control unit 250 controls the operation of the output unit 25.
[0079] 9 is a flowchart showing an example of the flow of processing executed by the conversion device 2 in the embodiment. The conversion target acquisition unit 210 acquires data to be converted into a target expression, which has been input to the communication unit 23 (step S201). Next, the expression conversion unit 220 converts the expression of the data acquired in step S201 using a target expression conversion process (step S202). Next, the output control unit 250 controls the operation of the output unit 25 to cause the output unit 25 to output the result of step S202 (step S203). Note that the output destination of the output unit 25 may be, for example, a device that executes a downstream task.
[0080] In this embodiment, noise-added data is input to a student model to generate student data, and distillation is performed based on the teacher data and the student data to update the parameters of the student model, thereby combining noise removal and distillation.
[0081] (Experimental Results) Figure 10 is a diagram showing an example of the experimental results in this embodiment. This embodiment (M2D-S) was performed three times, changing the length of the conversion target data, which is audio data, to 4.0 seconds, 5.12 seconds, and 6.08 seconds. In this experiment, the ratio of added noise was 20%, and λ atu and λ int were set to 1 and 0.5, respectively. In addition, tasks were performed using three comparative methods: wav2vec 2.0, HuBERT (Non-Patent Document 1), and WavLM (Non-Patent Document 3). Five tasks were performed: phoneme recognition (PR), keyword classification (KS), utterance intention classification (IC), speaker identification (SID), and speech emotion recognition (ER).
[0082] The method of this embodiment showed results showing higher performance than the comparison targets in keyword classification and voice emotion recognition.
[0083] 100... Expression conversion system, 1... Learning device, 2... Conversion device, 10... Learning unit, 11... Control unit, 12... Input unit, 13... Communication unit, 14... Memory unit, 15... Output unit, 110... Data acquisition unit, 120... Memory control unit, 130... Communication control unit, 140... Output control unit, 21... Control unit, 22... Input unit, 23... Communication unit, 24... Memory unit, 25... Output unit, 210... Conversion target acquisition unit, 220... Expression conversion unit, 230... Memory control unit, 240... Communication control unit, 250... Output control unit, 91... Processor, 92... Memory, 93... Processor, 94... Memory
Claims
1. A learning device comprising: a control unit that acquires, through learning, a target representation conversion process, which is a process of converting a representation of data to be converted into a target representation that is a predetermined representation; the control unit executes: a teacher model process that generates teacher data by inputting the data to be converted into a teacher model; a noise addition process that adds noise to the data to be converted; a student model process that generates student data by inputting the data with the noise added into a student model; and a noise removal and distillation process that updates the student model based on the teacher data and the student data; and the student model process at the point in time when a predetermined condition for ending the update is satisfied is the target representation conversion process.
2. The learning device according to claim 1, wherein the control unit updates the student model in the noise removal and distillation process based on a loss calculated by the student model process and a loss calculated based on the teacher data and the student data.
3. The learning device according to claim 1 or 2, wherein the data to be converted is voice data.
4. A conversion device comprising: a conversion target acquisition unit that acquires data to be converted into a target representation, which is a predetermined specified representation; and an expression conversion unit that performs a target representation conversion process to convert the expression of the data to be converted into a target representation, which is a predetermined specified representation, wherein the target representation conversion process executes: a teacher model process that generates teacher data by inputting the data to be converted into a teacher model; a noise addition process that adds noise to the data to be converted; a student model process that generates student data by inputting the data with the noise added into a student model; and a noise removal and distillation process that updates the student model based on the teacher data and the student data, wherein the student model process is the student model process at the point in time when a predetermined condition for ending the update is satisfied.
5. A learning method comprising: a control step of obtaining, through learning, a target representation conversion process, which is a process of converting a representation of data to be converted into a target representation that is a predetermined representation, wherein the control step executes: a teacher model process of generating teacher data by inputting the data to be converted into a teacher model; a noise addition process of adding noise to the data to be converted; a student model process of generating student data by inputting the data with the noise added into a student model; and a noise removal and distillation process of updating the student model based on the teacher data and the student data; and the student model process at the point in time when a predetermined condition for ending the update is satisfied is the target representation conversion process.
6. A conversion method comprising: a conversion target acquisition step of acquiring data to be converted into a target representation, which is a predetermined expression; and an expression conversion step of performing a target representation conversion process of converting the expression of the data to be converted into a target representation, which is a predetermined expression, wherein the target representation conversion process executes: a teacher model process of generating teacher data by inputting the data to be converted into a teacher model; a noise addition process of adding noise to the data to be converted; a student model process of generating student data by inputting the data with the noise added into a student model; and a noise removal and distillation process of updating the student model based on a loss calculated by the student model process and a loss calculated based on the teacher data and the student data, wherein the student model process is the student model process at the point in time when a predetermined condition for ending the update is satisfied.
7. A program for causing a computer to function as either the learning device according to any one of claims 1 to 3 or the conversion device according to claim 4.
Citation Information
Patent Citations
Method and device for compressing neural network models, method and device for corpus translation, electronic device, program, and recording medium
JP2022509892A
Artificial intelligence (AI) method for cleaning data for training ai models
WO2021195688A1
