Model learning method, model learning device, and program

The model learning method addresses the high calculation costs of SSL models by using knowledge distillation to transfer the teacher model's rich expression to a smaller student model, achieving accurate speech recognition with reduced computational costs and processing times.

WO2024228264A1PCT designated stage expired Publication Date: 2024-11-07NT T INC
View PDF 6 Cites 0 Cited by

Patent Information

Application Number
PCT/JP2023/017192
Authority / Receiving Office
WO · WO
Patent Type
Applications
Current Assignee / Owner
Filing Date
2023-05-02
Publication Date
2024-11-07

AI Technical Summary

Technical Problem

Self-supervised learning (SSL) models for speech recognition and related tasks face high calculation costs and processing times due to the long input sequence lengths of audio waveforms, which are exacerbated by the use of deep convolutional layers, making it challenging to achieve high accuracy with smaller model sizes.

Method used

A model learning method that generates an acoustic feature sequence from audio digital signals and uses knowledge distillation to transfer the rich expression of a teacher model to a student model with a smaller size, reducing the model size and calculation costs by using acoustic feature sequences as input and adjusting the convolutional layer configurations to match temporal resolutions.

Benefits of technology

This approach allows for complex speech expressions to be realized with a student model that has a smaller model size than the teacher model, maintaining accuracy equivalent to the teacher model while significantly reducing calculation costs and processing times.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure JP2023017192_07112024_PF_FP_ABST
    Figure JP2023017192_07112024_PF_FP_ABST
Patent Text Reader

Abstract

To realize a complicated voice expression by a student model having a model size smaller than that of a teacher model. In a model training method according to an embodiment of the present disclosure: an acoustic feature quantity sequence X is generated from a voice digital signal A; a teacher model expression sequence Y is generated from the voice digital signal A using a teacher model TM, which is a trained model trained by self-supervised learning; and a student model SM having a model size smaller than that of the teacher model TM model is trained using the acoustic feature quantity sequence X and the teacher model expression sequence Y.
Need to check novelty before this filing date? Find Prior Art

Description

Model learning method, model learning device, and program

[0001] The present disclosure relates to the art of knowledge distillation.

[0002] Self-supervised learning (SSL) for speech representation is a technology that allows speech representation to be learned in advance using a large amount of unlabeled speech data. By using a speech representation model trained using SSL, it is possible to solve subsequent tasks such as speech recognition, speaker recognition, and emotion recognition (hereinafter also referred to as "post-tasks") with high accuracy even when only a small amount of labeled speech data is available (see, for example, Non-Patent Document 1).

[0003] Generally, speech SSL models use speech waveforms as input. However, in some cases, the performance of post-task models can be improved by using only spectral features such as log-log-melt filter banks (FBANKs) to achieve accuracy comparable to that of speech waveforms. However, in order to achieve high accuracy in post-task tasks, it is important to obtain complex speech representations that utilize the temporal information contained in the speech waveforms. Therefore, speech waveforms are often used as inputs. In these cases, deep convolutional layers such as CNN7 (Convolutional Neural Network 7) are often used as preprocessing layers for encoders such as Transformers to obtain complex speech representations, thereby reducing the speech waveform sequence.

[0004] Using speech waveforms as input allows for the creation of highly versatile SSL models. However, compared to feature inputs such as FBANK, the time series length of speech waveforms is significantly longer, resulting in significantly increased computational costs. Furthermore, using speech waveforms to implement deep convolutional layers not only increases the computational cost to a non-negligible extent, but also increases the computational cost and processing time of downstream tasks such as inference.

[0005] One method proposed to solve this problem is to compress model size using knowledge distillation (KD). Knowledge distillation achieves this by training a large model (hereinafter also referred to as a "teacher model") in advance and transferring its rich representations to a small model (hereinafter also referred to as a "student model"). In this case, since the student model also uses speech waveforms as input, only the encoder is changed, and the deep convolutional layers are not changed (see, for example, Non-Patent Document 2).

[0006] A. Mohamed et al., "Self-Supervised Speech Representation Learning: A Review," in IEEE Journal of Selected Topics in Signal Processing, vol. 16, no. 6, pp. 1179-1210, Oct. 2022, doi: 10.1109 / JSTSP.2022.3207050.H. -J. Chang, S. -w. Yang and H. -y. Lee, "Distilhubert: Speech Representation Learning by Layer-Wise Distillation of Hidden-Unit Bert," ICASSP 2022 - 2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Singapore, Singapore, 2022, pp. 7087-7091, doi: 10.1109 / ICASSP43922.2022.9747490.

[0007] However, since the input used in Non-Patent Document 2 is merely a speech waveform, the input sequence length is longer than pre-processed feature sequences such as FBANK, and therefore the convolutional layer, which is the pre-processing stage of the encoder, must also be configured deeply. This poses the problem of high computational costs and long processing times for subsequent tasks.

[0008] In order to solve the above problem, the present disclosure aims to realize complex speech expression using a student model that has a smaller model size than a teacher model.

[0009] In order to solve the above problem, a model learning method according to one aspect of the present disclosure generates an acoustic feature sequence from an audio digital signal, generates a teacher model representation sequence from the audio digital signal using a teacher model, which is a trained model trained by self-supervised learning, and learns a student model having a smaller model size than the teacher model using the acoustic feature sequence and the teacher model representation sequence.

[0010] According to the present disclosure, the above configuration makes it possible to realize complex speech expressions using a student model that is smaller in model size than a teacher model.

[0011] Fig. 1 is a diagram showing an example of the functional configuration of a model learning device according to the present disclosure. Fig. 2 is a flowchart showing an example of the processing of the model learning device 1. Fig. 3 is a diagram showing an example of the functional configuration of a modified example of the model learning device according to the present disclosure. Fig. 4 is a flowchart showing an example of the processing of the model learning device 1a. Fig. 5 is a diagram showing an example of the functional configuration of a computer.

[0012] Hereinafter, embodiments of the present disclosure will be described in detail with reference to the accompanying drawings. Components having the same functions will be denoted by the same reference numerals, and redundant description will be omitted.

[0013] 1, a model learning device 1 according to an embodiment of the present disclosure includes, for example, an audio signal acquisition unit 10, an audio digital signal storage unit 20, a feature extraction unit 30, a feature storage unit 40, a teacher expression sequence generation unit 50, and a learning unit 60. The model learning device 1 performs the processing shown in FIG. 2 to realize a learning method according to this embodiment.

[0014] (Audio Signal Acquisition Unit 10) The audio signal acquisition unit 10 performs a predetermined AD conversion process on the input analog audio signal (audio signal A') to generate an audio digital signal (audio digital signal A), and outputs it to the communication device 20 (step S10). The audio signal A' is, for example, audio recorded with a PC microphone or an IC recorder. The audio signal A' does not necessarily need to be recorded in advance, and may be configured to be output directly to the audio signal acquisition unit 10 using a predetermined microphone or the like.

[0015] (Audio digital signal storage unit 20) The input audio digital signal A is stored and output to the feature extraction unit 30 and the teacher expression sequence generation unit 50 (step S20). The audio digital signal storage unit 20 may be configured as a part of the function of the audio signal acquisition unit 10.

[0016] (Feature Extraction Unit 30) The feature extraction unit 30 generates an acoustic feature sequence (acoustic feature sequence X) for each utterance from the input digital audio signal A, and outputs it to the feature accumulation unit 40 (step S30).

[0017] The acoustic feature sequence X to be generated may, for example, be 1 to 12 dimensions of MFCCs (Mel-Frequency Cepstrum Coefficients) based on short-time frame analysis of the digital speech signal A, and their dynamic feature parameters such as ΔMFCC and ΔΔMFCC, as well as power, Δpower, and ΔΔpower. Cepstral Mean Normalization (CMN) processing may be performed on the MFCCs. Alternatively, a log-log-mel filter bank (FBANK) may be used. The acoustic feature sequence X is not limited to MFCCs and power, and parameters used to identify special utterances (such as autocorrelation peak values ​​and group delays) may also be used.

[0018] (Feature Storage Unit 40) The feature storage unit 40 stores the acoustic feature sequence X input from the feature extraction unit 30 and outputs it to the learning unit 60 (step S40). The feature storage unit 40 may be configured as a part of the function of the feature extraction unit 30.

[0019] (Teacher expression sequence generation unit 50) The teacher expression sequence generation unit 50 generates a teacher model expression sequence (teacher model expression sequence Y) from the input audio digital signal A using a teacher model (teacher model TM), which is a trained model trained by self-supervised learning (SSL), and outputs it to the learning unit 60 (step S50).

[0020] The teacher expression sequence generation unit 50 has a teacher model TM, which is a trained model by SSL. The teacher model TM includes, for example, a deep convolutional layer (convolutional layer N T ) and a multi-layer encoder (encoder E T The teacher model TM may have any other configuration as long as it is a trained model trained by SSL using a speech waveform as input. The generated teacher model representation sequence Y may be the representation sequence of only the final layer of the encoder, or may be the representation sequence of all of the intermediate layers.

[0021] (Learning unit 60) Using the acoustic feature sequence X input from the feature storage unit 40 and the teacher model expression sequence Y input from the teacher expression sequence generation unit 50, a student model (student model SM) having a smaller model size than the teacher model TM is trained (steps S60-1, S60-2). The student model SM as a trained model is used for, for example, the above-mentioned subsequent tasks such as speech recognition, speaker recognition, and emotion recognition. The student model SM is trained by convolutional layers N T A convolutional layer with a model size smaller than S ) and encoder E T An encoder with a smaller model size than S ) and.

[0022] The learning unit 60 uses the acoustic feature sequence X as input data and the teacher model expression sequence Y as correct answer data (teacher data) to learn the student model SM. The student model SM is learned through the following processes (i) and (ii).

[0023] (i) A loss function (loss function LS1) is calculated using the acoustic feature sequence X as input data and the teacher model expression sequence Y as correct answer data (step S60-1).

[0024] (ii) Based on the calculation result of the loss function LS1, the model parameters P of the student model SM are updated (step S60-2). Examples of the loss function LS1 include a frame-by-frame L1 loss function or cosine similarity, but other loss functions may also be used.

[0025] The learning unit 60 repeats the above-mentioned learning using other data until the calculation result of the loss function LS1 satisfies a predetermined threshold, thereby completing the learning and generating a student model SM as a learned model.

[0026] Through the learning in steps S60-1 and S60-2, a student model SM having a smaller model size than the teacher model TM is trained using the acoustic feature sequence X and the teacher model representation sequence Y. The student model SM has significantly reduced computational cost compared to the teacher model TM due to three points: (i) the input sequence length is degenerated, (ii) the number of parameters (number of layers or number of dimensions) in the convolutional layer is reduced, and (iii) the number of parameters is reduced by an arbitrarily determined encoder size. Meanwhile, because the SSL representation of the teacher model TM is transferred as is to the student model SM, accuracy degradation in various subsequent tasks due to pre-processed feature input can also be minimized.

[0027] In other words, the rich representation of the teacher model TM trained on speech waveforms is transferred to a smaller student model SM through knowledge distillation using the acoustic feature sequence X as input data. Therefore, the trained student model SM can be guaranteed to have the same level of accuracy as the teacher model TM. This makes it possible to minimize the degradation of accuracy in various downstream tasks that accompany input of feature sequences such as conventional FBANK.

[0028] As a result, complex speech expressions can be realized using the student model SM, which has a smaller model size than the teacher model.

[0029] [Modification] The above-described model learning device 1 may be configured as a model learning device 1a shown in Fig. 3. In the model learning device 1a, the learning unit 60 of the model learning device 1 is replaced with a learning unit 60a. The model learning device 1a performs the processing shown in Fig. 4 to realize the learning method according to this modification.

[0030] 60a includes a student model SMa. In addition to the functions of the student model SM described above, the student model SMa also has a self-supervised learning function for generating a label (label V) from an acoustic feature sequence X. In other words, the student model SMa is configured as a model for SSL.

[0031] During learning by the learning unit 60a, in addition to knowledge distillation using the teacher model expression sequence Y as correct answer data, the label V generated from the acoustic feature sequence X as a result of the student model SMa learning as an SSL model may be used as correct answer data.

[0032] The learning unit 60a learns the student model SMa by performing the following processes (i) to (iv): (i) As a preprocessing of the loss function LS1, the convolution layer N T Setting the layer configuration, setting the generation conditions for the acoustic feature sequence X, and setting the convolution layer N S At least one of the layer structure settings is adjusted to reduce the difference in time resolution between the acoustic feature sequence X and the teacher model expression sequence Y (step S60a-1).

[0033] When knowledge distillation is performed using the teacher model representation sequence Y as the correct answer data, the sequence length of the teacher model representation sequence Y may differ from the sequence length generated by the student model SM. This is because the convolutional layer N T The settings of the analysis conditions of the acoustic feature sequence X and the convolutional layer of the student model SM (convolutional layer N S This occurs because the time resolution is different between the settings of the convolutional layer N T , analysis conditions for the acoustic feature sequence X, and the convolution layer N SThe discrepancy can be resolved by adjusting the settings. The temporal resolution may also be adjusted by adding a transposed convolution layer to the SSL representation output by the student model SMa. However, these adjustments of the temporal resolution are merely examples, and the temporal resolution may also be adjusted by other methods.

[0034] (ii) Using the acoustic feature sequence X as input data and the teacher model expression sequence Y as correct data, a loss function (loss function LS1) is calculated (step S60a-2).

[0035] (iii) Using the acoustic feature sequence X as input data and the label V generated by the student model SMa as ground truth data, a loss function (loss function LS2) is calculated (step S60a-3). The loss function LS2 may be, for example, wav2vec2.0, but may also be any other SSL method.

[0036] (iv) Based on the calculation results of the loss function LS1 and the loss function LS2, the model parameters Pa of the student model SMa are updated (step S60a-4). For example, a method of performing the update using a weighted sum of the loss function LS1 and the loss function LS2 can be considered, but any other method that can take both loss functions into consideration may be used.

[0037] The learning unit 60a repeats the above-mentioned learning using other data until the calculation results of the loss function LS1 and the loss function LS2 satisfy a predetermined threshold, thereby completing the learning and generating a student model SMa as a learned model.

[0038] Although the embodiments and modifications of this disclosure have been described above, the specific configuration is not limited to these embodiments and modifications, and it goes without saying that appropriate design changes and the like are included in this disclosure as long as they do not deviate from the spirit of this disclosure. The various processes described in this embodiment and modifications may not only be executed in chronological order according to the order described, but may also be executed in parallel or individually depending on the processing capacity of the device that executes the processes or as needed.

[0039] [Program, Recording Medium] The various processes described above can be implemented by loading a program that executes each step of the above method into the recording unit 2020 of the computer 2000 shown in FIG. 5, and operating the control unit 2010, input unit 2030, output unit 2040, display unit 2050, etc.

[0040] The program describing the processing contents can be recorded on a computer-readable recording medium, which may be, for example, a magnetic recording device, an optical disk, a magneto-optical recording medium, a semiconductor memory, or any other suitable recording medium.

[0041] The program may be distributed by, for example, selling, transferring, lending, etc. portable recording media such as DVDs and CD-ROMs on which the program is recorded. Furthermore, the program may be stored in a storage device of a server computer, and then transferred from the server computer to other computers via a network, thereby distributing the program.

[0042] A computer that executes such a program may first temporarily store the program recorded on a portable recording medium or transferred from a server computer in its own storage device. Then, when executing a process, the computer reads the program stored on its own recording medium and executes the process in accordance with the read program. Alternatively, the computer may read the program directly from a portable recording medium and execute the process in accordance with the program. Furthermore, the computer may execute the process in accordance with each program transferred from the server computer. Alternatively, the server computer may not transfer the program to the computer, but may execute the process through a so-called ASP (Application Service Provider) service, which realizes the processing function by issuing an execution instruction and obtaining the results. In this embodiment, the program includes information used for processing by a computer that is equivalent to a program (e.g., data that is not a direct instruction to the computer but has properties that define computer processing).

[0043] Furthermore, in this embodiment, the device is configured by executing a predetermined program on a computer, but at least a part of the processing contents may be realized by hardware.

[0044] 1, 1a Model learning device 10 Audio signal acquisition unit 20 Audio digital signal storage unit 30 Feature extraction unit 40 Feature storage unit 50 Teacher expression sequence generation unit 60, 60a Learning unit SM Student model TM Teacher model A' Audio signal A Audio digital signal E T , E S Encoder LS1, LS2 Loss function N T , N S Convolutional layer P, Pa Model parameters V Label X Acoustic feature sequence Y Teacher model representation sequence

Claims

1. A model learning method comprising: generating an acoustic feature sequence from a digital audio signal; generating a teacher model representation sequence from the digital audio signal using a teacher model, which is a trained model trained by self-supervised learning; and learning a student model having a smaller model size than the teacher model, using the acoustic feature sequence and the teacher model representation sequence.

2. The learning method according to claim 1, wherein the learning comprises: calculating a first loss function using the acoustic feature sequence as input data and the teacher model representation sequence as correct answer data; and updating model parameters of the student model based on a calculation result of the first loss function.

3. The learning method according to claim 1, wherein the student model further has a self-supervised learning function for generating labels from the acoustic feature sequence, and the learning comprises: calculating a first loss function using the acoustic feature sequence as input data and the teacher model representation sequence as correct answer data; calculating a second loss function using the acoustic feature sequence as input data and the labels generated by the student model as correct answer data; and updating model parameters of the student model based on the calculation results of the first loss function and the calculation results of the second loss function.

4. The learning method according to claim 3, wherein the learning is performed using a weighted sum of the calculation result of the first loss function and the calculation result of the second loss function.

5. The learning method according to claim 1, wherein the teacher model has a first convolutional layer and a first encoder, and the student model has a second convolutional layer having a model size smaller than that of the first convolutional layer and a second encoder having a model size smaller than that of the first encoder.

6. The learning method according to claim 5, wherein the learning comprises adjusting at least one of a setting of a layer configuration of the first convolutional layer, a setting of a generation condition of the acoustic feature sequence, and a setting of a layer configuration of the second convolutional layer as a preprocessing for calculating a first loss function, thereby making an adjustment to reduce a difference in time resolution between the acoustic feature sequence and the teacher model representation sequence.

7. A model learning device comprising: a feature extraction unit that generates an acoustic feature sequence from a digital audio signal; a teacher representation sequence generation unit that generates a teacher model representation sequence from the digital audio signal using a teacher model that is a learned model learned by self-supervised learning; and a learning unit that learns a student model having a smaller model size than the teacher model using the acoustic feature sequence and the teacher model representation sequence.

8. A program for causing a computer to function as the model learning device according to claim 7.

Citation Information

Patent Citations

  • End-to-end long-time speech recognition method

    CN113516968A

  • Lightweight end-to-end speech synthesis system construction method based on multi-frame prediction

    CN115376484A

  • Conditional teacher-student learning for model training

    US20200334538A1

  • Model training method and apparatus

    US20200357384A1

  • Self-Adaptive Distillation

    US20220309340A1