Continuous self-supervising multi-type voice acoustic feature representation method based on distillation

Abstract

The invention relates to the technical field of deep learning, in particular to a distillation-based continuous self-supervising multi-type voice acoustic feature representation method, which comprises the following steps of: acquiring voice data; preprocessing the collected voice data, and distributing the preprocessed different types of voice data to different training stages; initializing a masking self-supervision feature extraction model; constructing a replay buffer area according to reconstruction features output by the model, and performing data enhancement; updating the masking self-supervision feature extraction model in the next stage, and performing feature distillation by using a replay buffer area; obtaining acoustic features of multiple types of voices to be extracted; according to the method, multiple types of voice data are trained in sequence, and data conflicts of different data can be prevented; and secondly, feature distillation is carried out on replay buffer data by using cosine similarity as a regular term to retain knowledge, so that old knowledge can be prevented from being forgotten, and the plasticity of the model can be enhanced.

Description

Technical Field

[0001] The present invention relates to the field of deep learning technology, and in particular to a method for representing multi-type speech acoustic features through continuous self-supervision based on distillation. Background Art

[0002] Due to the decline in the flexibility and coordination of the vocal organs, Parkinson's disease patients often experience symptoms such as difficulty in pronunciation and unstable pronunciation in the early stages of the disease. Acoustic experts have designed a variety of corpora, including sustained vowels, repeated syllables, and situational dialogues, for the analysis and evaluation of the subjects' speech ability. Since it is difficult to collect speech data from Parkinson's disease patients and the annotation cost is high, there are relatively few public Parkinson's disease speech datasets. Moreover, most of these public datasets are derived from the speech of people in a single region, and have the characteristics of different collection methods, insufficient capacity, and small differences in the pronunciation characteristics of the subjects' native language. These characteristics of Parkinson's disease speech datasets affect the application of artificial intelligence technology in this field, mainly manifested in the following aspects: (1) Obtaining sufficient pre-training speech data is a challenging, expensive, and time-consuming process. Under existing conditions, it is difficult to increase the amount of labeled data. Therefore, it is difficult to enhance the performance of the network model by increasing the available training data. (2) In different Parkinson's disease speech datasets, the probability distribution of speech data is inconsistent due to language differences and corpus differences. Therefore, the Parkinson's disease speech feature extraction model trained on a single type of speech dataset often cannot avoid the impact of type differences in multi-type scenarios (different languages, different corpora), resulting in the extracted features being unable to fully reflect the pathological information of Parkinson's disease. Therefore, it is urgent to seek a new Parkinson's disease speech acoustic feature extraction method that aims to fully utilize Parkinson's disease speech data resources while avoiding time-consuming and labor-intensive manual labeling.

[0003] Standard strong supervised learning is dependent on data, which often requires time-consuming and labor-intensive manual labeling, which limits strong supervised learning in practical applications. Continuous self-supervised learning is an effective pre-training method in the field of deep learning. It can directly learn the underlying structure representation of data from unlabeled data streams. More importantly, it has a certain memory function for the learned data stream information. The high-level features learned in the end help improve the performance and convergence speed of downstream tasks.

[0004] Currently, there are two main strategies for resisting catastrophic forgetting in continuous self-supervised learning: (1) The first strategy selects appropriate samples in each data stream as replay samples, and adds the replay samples to the training in subsequent data stream training to prevent catastrophic forgetting. However, the repeated appearance of replay samples in subsequent training may lead to the leakage of private information in certain fields. (2) The second strategy adds some regularization techniques to the network optimization objectives, such as knowledge distillation, so that the network can maintain consistency with the previous state to a certain extent in the training of subsequent data streams. This type of strategy tends to limit the plasticity of the network and make it unable to adapt to the learning of new features. Summary of the invention

[0005] The purpose of the present invention is to provide a distillation-based continuous self-supervised multi-type speech acoustic feature representation method to solve the problems raised in the above background technology.

[0006] In order to solve the above technical problems, the present invention provides the following technical solutions:

[0007] A method for continuously self-supervising multi-type speech acoustic feature representation based on distillation, the method comprising:

[0008] Step 1: Collection of speech data: Collect different types of subject speech data from existing public data sets, or collect different types of subject speech data;

[0009] Step 2: Preprocess the collected speech data and assign different types of preprocessed speech data to different training stages. Assuming that the total number of tasks is T (corresponding to T training stages), the speech data subset of the tth training stage is denoted as V t , extract the Mel spectrogram features corresponding to the speech data subsets of each training stage, construct the training subsets of each training stage, and the training subset of the tth training stage is recorded as D t ;

[0010] Step 3: Initialize a masked self-supervised feature extraction model in the first training stage, and use the training subset D1 of the first training stage generated in step 2 to model the model to obtain a masked self-supervised feature extraction model M1; construct a replay buffer Β1 according to the reconstructed features output by the model M1;

[0011] Step 4: When t ≥ 2, in the training subset D of the tth training stage t The masked self-supervised model obtained in the t-1th stage is trained and the model parameters are updated to obtain the masked self-supervised feature extraction model M t ; According to model M t The reconstructed features of the output construct the replay buffer Β t; and use cosine similarity as a regularizer for feature distillation on the replay buffer;

[0012] Step 5: Let t = t + 1, and repeat step 4 until the training is completed at t = T, and finally obtain the masked self-supervised feature extraction model M T , and its output is the acoustic features of the multi-type speech to be extracted.

[0013] Preferably, the voice data in step 1 includes:

[0014] The voice data adopts multiple types of voice data, including continuous vowels, repeated syllables and situational dialogues;

[0015] Among them, sustained vowels include sustained vowel / a / , sustained vowel / i / , sustained vowel / o / ; repeated syllables include / pa-ka-la / , / pa-ta-la / ; situational dialogues include a variety of different dialogue contents.

[0016] Preferably, step 2 comprises:

[0017] S201, dividing the original speech signal into frames, and using the Librosa audio processing library to remove the silent frames at the beginning and end of the speech; converting the pre-processed speech data into a spectrogram, and passing it through a Mel filter bank to obtain the corresponding Mel spectrogram features;

[0018] S202, according to the different types of speech data mentioned in the improvement of step 1, the speech data subsets are divided (such as into a continuous vowel / a / subset, a repeated syllable / pa-ka-la / subset, a situational dialogue subset, etc.); different speech data subsets are assigned to different training stages;

[0019] Among them, the speech data subset of the tth training stage is denoted as V t , 1≤t≤T;

[0020] V t The corresponding Mel spectrogram set constitutes the training subset D of the tth training stage t , 1≤t≤T;

[0021] To include n t The Mel spectrogram feature set of samples, where the Mel spectrogram feature x i,t is the i-th sample in the t-th training subset, and each sample has no label.

[0022] Preferably, step 3 comprises:

[0023] S301, train a basic masked self-supervised speech feature extraction model M1, and use the training subset of the first training stage As input to the masked self-supervised speech feature extraction model;

[0024] Using the mask image as an excuse task, the Mel spectrogram features in D1 are randomly masked, and then the masked Mel spectrogram features are reconstructed through the Transformer encoder to obtain the reconstructed Mel spectrogram features.

[0025] The L1 loss is calculated between the input Mel-level spectrogram features and the reconstructed Mel-level spectrogram features, and the calculation formula is:

[0026] Among them, x i,t (i=1,2,…,n t ) is the input sequence, i.e. the original Mel spectrogram feature, y i,t (i=1,2,…,n t ) is the output sequence of the masked self-supervised model, i.e., the reconstructed Mel-spectrogram features;

[0027] S302. Use K-means clustering method to select representative samples for replay:

[0028] The reconstructed spectrogram features output by the masked self-supervised feature extraction model M1 in S301 Cluster into C clusters, select k samples closest to the center and q samples farthest from the center from each cluster to form the replay buffer Β1 = {m 1,1 ,m 2,1 ,…,m i,1 ,…,m k+q,1};

[0029] S303, in order to increase the diversity of the replay buffer and prevent the model from overfitting, according to the formula: x′=λx1+(1-λ)x2, a binary hybrid strategy is used to perform data enhancement on the buffer B1;

[0030] Among them, λ represents a uniform random variable between [0,1]; x′ represents the generated new sample, that is, the enhanced sample; x1 and x2 represent samples randomly selected from B1 respectively;

[0031] Get the updated replay buffer Β1={m 1,1 ,m 2,1 ,…,m i,1 ,…,m k+q,1 ,m k+q+1,1 …,m n,1};

[0032] Among them, m k+q+1,1 …,m n,1 Represents an enhanced sample.

[0033] Preferably, step 4 comprises:

[0034] S401, in the training subset D t Calculate the L1 loss on the model M t-1 Update the parameters to obtain the updated masked self-supervised feature extraction model M t ;

[0035] S402, copy the masked self-supervised model M obtained by training in the t-1th training stage t-1 , whose model parameters represent the set of knowledge learned in the t-1th training stage;

[0036] S403, replay buffer B t-1 Feature input model M t , output the reconstructed Mel spectrogram feature E t ={e 1,t ,e 2,t ,…,e i,t ,…,e n,t}; Replay buffer B t-1 Feature input M t-1 , output the reconstructed Mel spectrogram feature E t-1 ={e 1,(t-1) ,e 2,(t-1) ,…,e i,(t-1) ,…,e n,(t-1)};

[0037] Calculate E t and E t-1 The cosine similarity between the two models is used as a regular term to prevent the features learned in the previous training stage from being greatly different when training on the data of the current training stage. When the cosine similarity output by the two models is greater than the threshold, the features are similar, indicating that the old knowledge has not been forgotten. Using cosine similarity as a regular term will not directly affect the model's learning of new features. As long as the cosine similarity is non-negative, the total loss will not increase, which not only prevents the old knowledge from being forgotten, but also enhances the plasticity of the new features.

[0038] Based on this, the loss function is defined as:

[0039] Among them, S(e i,t ,e i,t-1 ) computes two self-supervised models M t and M t-1 The cosine similarity of the output features of

[0040]

[0041] Among them, ||·||2 means finding the second norm, (·)T Represents the transpose of a vector. The value range of cosine similarity is between [-1,1], 1 means completely similar, -1 means completely opposite, and 0 means they are orthogonal;

[0042] S404, in the training subset D t Referring to S302, representative samples are selected to form a replay buffer B. t ={m 1,t ,m 2,t ,…,m i,t ,…,m n,t}; Refer to S303 for buffer B t Use binary hybrid strategy for data enhancement and construct a new replay buffer B t And clear B t-1 data.

[0043] Preferably, a speech feature representation system for implementing a continuous self-supervised multi-type speech acoustic feature representation method based on distillation comprises: a speech feature extraction module, a training subset generation module, a sequential self-supervised model training module, a replay buffer module and a feature distillation module;

[0044] The speech feature extraction module is used to pre-process the speech data and extract the Mel spectrum features of the speech;

[0045] The training subset generation module is used to divide the speech data into speech data subsets and allocate them to each training stage. The Mel spectrograms corresponding to the speech data subsets constitute the training subsets of each training stage.

[0046] The sequential self-supervised model training module is used to input the Mel-spectrogram features of speech data into the self-supervised feature extraction model in the order of the training phase, perform random masking and reconstruction, and extract the output of the last layer of the model as the acoustic features of multiple types of speech;

[0047] The replay buffer module is used to retain part of the data in the current stage as buffer data for the next stage and perform data enhancement;

[0048] The feature distillation module is used to input the replay buffer data into M t and the copied M t-1 In , the cosine similarity of the input features is calculated for feature distillation.

[0049] A computer-readable storage medium having a computer program stored thereon, characterized in that: when the program is executed by a processor, the steps in the above-mentioned method for continuous self-supervised multi-type speech acoustic feature representation based on distillation are implemented.

[0050] A computer device includes a memory, a processor, and a computer program stored in the memory and running on the processor. When the processor executes the program, the steps in the above-mentioned method for continuous self-supervised multi-type speech acoustic feature representation based on distillation are implemented.

[0051] Compared with the prior art, the beneficial effects achieved by the present invention are:

[0052] The present invention adopts sequential self-supervised training for multiple types of speech data. Compared with joint self-supervised training, this method can prevent data conflicts between different data. Secondly, cosine similarity is used as a regular term to perform feature distillation on the replay buffer data to retain knowledge, which can prevent the old knowledge from being forgotten while enhancing the plasticity of new features.

[0053] The present invention utilizes a sequential self-supervised model to learn generalized representations of different types of speech data sets without labels for training data, thereby preventing conflicts between different types of data. At the same time, a continuous learning method is introduced to prevent catastrophic forgetting of the model and improve the plasticity of the model. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] The accompanying drawings are used to provide a further understanding of the present invention and constitute a part of the specification. Together with the embodiments of the present invention, they are used to explain the present invention and do not constitute a limitation of the present invention. In the accompanying drawings:

[0055] Figure 1 It is a flow chart of a method for continuously self-supervising multi-type speech acoustic feature representation based on distillation of the present invention;

[0056] Figure 2 It is a schematic diagram of characteristic distillation of the present invention. DETAILED DESCRIPTION

[0057] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0058] See also Figure 1-Figure 2 , the present invention provides a technical solution:

[0059] Embodiment 1:

[0060] like Figure 1 As shown, the present invention is a method for representing multi-type speech features through continuous self-supervision based on distillation, which specifically includes:

[0061] Step 1, voice signal collection / collection: collect different types of subject voice data from existing public data sets, or collect different types of subject voice data, including sustained vowels, repeated syllables and situational dialogues. Among them, sustained vowels can include sustained vowels / a / , sustained vowels / i / , sustained vowels / o / , etc.; repeated syllables can include / pa-ka-la / , / pa-ta-la / ; situational dialogues can also have a variety of different dialogue contents.

[0062] Step 2: Preprocess the speech data collected / collected in step 1, and assign different types of preprocessed speech data to different training stages. Assuming that the total number of tasks is T (corresponding to T training stages, T is an integer greater than 1), the speech data subset of the tth training stage is denoted as V t (1≤t≤T); extract the Mel spectrogram features corresponding to the speech data subset of each training stage, construct the training subset of each training stage, and the training subset of the tth training stage is recorded as D t (1≤t≤T). Specifically include:

[0063] S201, divide the original speech signal into frames, and use the Librosa audio processing library to remove the silent frames at the beginning and end of the speech. Convert the pre-processed speech data into a spectrogram, and obtain the corresponding Mel spectrogram features through a Mel filter bank;

[0064] S202: Divide the speech data into subsets according to the different speech data types mentioned in the improvement of step 1, and assign different speech data subsets to different training stages, wherein the speech data subset of the t-th training stage is denoted as V t (1≤t≤T), V t The corresponding Mel spectrogram set constitutes the training subset D of the tth training stage t .

[0065] To include n t The Mel spectrogram feature set of samples, where the Mel spectrogram feature x i,t is the i-th sample in the t-th training subset, and each sample has no label.

[0066] Step 3: Initialize a masked self-supervised feature extraction model in the first training stage, use the training subset D1 of the first training stage generated in step 2 to model the model, and obtain the masked self-supervised feature extraction model M1; construct a replay buffer B1 according to the reconstructed features output by the model M1. Specifically including:

[0067] S301, train a basic masked self-supervised speech feature extraction model M1, and use the training subset of the first training stage As the input of the masked self-supervised speech feature extraction model. Using the masked image as an excuse task, the Mel spectrogram features in D1 are randomly masked, that is, 15% of the frames in each input sequence are selected for masking, and then the masked Mel spectrogram features are reconstructed through the Transformer encoder to obtain the reconstructed Mel spectrogram features. The L1 loss is calculated between the input Mel-level spectrogram features and the reconstructed Mel-level spectrogram features, and the calculation formula is:

[0068] Among them, x i,t (i=1,2,…,n t ) is the input sequence (i.e. the original Mel spectrogram features), y i,t (i=1,2,…,n t ) is the output sequence of the masked self-supervised model (i.e., the reconstructed Mel-spectrogram features).

[0069] S302, using the K-means clustering method to select representative samples (i.e., the points closest to the center of each cluster and the points at the distribution boundary) for replay. Specifically, the reconstructed spectrogram features output by the masked self-supervised feature extraction model M1 in S301 are Clustering is done into C clusters, and from each cluster, k samples closest to the center and q samples farthest from the center are selected to form a replay buffer Β1 = {m 1,1 ,m 2,1 ,…,m i,1 ,…,m k+q,1}.

[0070] S303, in order to increase the diversity of the replay buffer and prevent the model from overfitting, the buffer B1 is enhanced by using a binary mixing (InstanceMixup) strategy, as shown below: x′=λx1+(1-λ)x2;

[0071] Where λ is a uniform random variable between [0,1]; x′ is the generated new sample, i.e. the enhanced sample; x1 and x2 are samples randomly selected from B1 respectively. The updated replay buffer

[0072] Β1={m 1,1 ,m 2,1 ,…,m i,1 ,…,m k+q,1 ,m k+q+1,1 …,m n,1}, where m k+q+1,1 …,m n,1 To enhance the sample;

[0073] Step 4: Training subset D in the tth (t≥2) training stage tThe masked self-supervised model obtained in the (t-1)th stage is trained and the model parameters are updated to obtain the masked self-supervised feature extraction model M t According to model M t The reconstructed features of the output construct the replay buffer Β t ; and use cosine similarity as a regular term in the replay buffer for feature distillation. Specifically include:

[0074] S401, in the training subset D t Calculate the L1 loss (refer to the L1 loss calculation formula in S301) for model M t-1 Update the parameters to obtain the updated masked self-supervised feature extraction model M t ;

[0075] S402, copy the masked self-supervised model M obtained by training in the t-1th training stage t-1 , whose model parameters represent the set of knowledge learned in the t-1th training stage;

[0076] S403, replay buffer B t-1 Feature input model M t , output the reconstructed Mel spectrogram feature E t ={e 1,t ,e 2,t ,…,e i,t ,…,e n,t}; Replay buffer B t-1 Feature input M t-1 , output the reconstructed Mel spectrogram feature E t-1 ={e 1,(t-1) ,e 2,(t-1) ,…,e i,(t-1) ,…,e n,(t-1)}. Calculate E t and E t-1 The cosine similarity between the two models is used as a regularization term to prevent large differences between the features learned in the previous training phase and the features learned in the current training phase. Here, when the features output by the two models are similar, it means that the old knowledge has not been forgotten. Using cosine similarity as a regularization term will not directly affect the model's learning of new features. As long as the cosine similarity is non-negative, the total loss will not increase. This prevents the old knowledge from being forgotten while enhancing the plasticity of the new features. The loss function here is defined as follows:

[0077]

[0078] Among them, S(e i,t ,e i,t-1 ) computes two self-supervised models M t and M t-1The cosine similarity of the output features is defined as follows:

[0079] Among them, ||·||2 means finding the second norm, (·) T Represents the transpose of a vector. The value range of cosine similarity is between [-1,1], 1 means completely similar, -1 means completely opposite, and 0 means they are orthogonal;

[0080] S404, in the training subset D t Referring to S302, representative samples are selected to form a replay buffer B. t ={m 1,t ,m 2,t ,…,m i,t ,…,m n,t}; Refer to S303 for buffer B t Use the binary mixing (InstanceMixup) strategy for data enhancement and construct a new replay buffer B t And clear B t-1 data.

[0081] Step 5: Let t = t + 1, and repeat step 4 until the training is completed at t = T, and finally obtain the masked self-supervised feature extraction model M T , and its output is the acoustic features of the multi-type speech to be extracted.

[0082] A continuous self-supervised multi-type speech feature representation system based on distillation is also provided, the system comprising:

[0083] Speech feature extraction module: pre-processes speech data and extracts the Mel spectrum features of speech;

[0084] Training subset generation module: divides the speech data into speech data subsets and assigns them to each training stage. The Mel spectrograms corresponding to the speech data subsets constitute the training subsets of each training stage.

[0085] Sequential self-supervised model training module: The Mel-spectrogram features of speech data are input into the self-supervised feature extraction model in the order of the training phase, and randomly masked and reconstructed. The output of the last layer of the model is extracted as the acoustic features of multiple types of speech.

[0086] Replay buffer module: retains part of the data of the current stage as the buffer data of the next stage and performs data enhancement;

[0087] Feature distillation module: input the replay buffer data into M t and the copied M t-1 In , the cosine similarity of the input features is calculated for feature distillation;

[0088] The invention uses three Parkinson's disease speech data sets in different languages, including the Maxlittle data set (native English), the Italian Parkinson's disease data set (native Italian) and the self-collected data set (native Chinese), of which the Maxlittle data set contains vowel data, and the other two data sets contain sustained vowels / a / , repeated syllables and designated text reading scenario dialogue content. The vowel / a / can reflect the ability of the vocal cords to vibrate continuously and the health of the laryngeal muscles; repeated syllables can measure the regularity of the movement of organs such as the lips, teeth and tongue, and the occlusion ability of the mandibular joint; the designated text reading scenario dialogue can help comprehensively evaluate all vocal components and the comprehensibility of pronunciation. When analyzing the various motor symptoms of Parkinson's disease through speech signals, it is necessary to use a signal processing algorithm to analyze the test speech and extract speech feature information that can characterize the pathological characteristics of Parkinson's disease motor symptoms.

[0089] In summary, the use of Mel spectrograms that retain more original speech information is used as the input of the self-supervised model; the self-supervised model based on the Transformer encoder can capture longer time series dependencies when facing time series data. Integrating multilingual datasets can provide the model with richer and more diverse speech features, which will help to explore potential commonalities and differences and improve the learning effect and performance of the model.

[0090] Embodiment 2:

[0091] The computer-readable storage medium of this embodiment stores a computer program thereon, which, when executed by a processor, implements the steps in a method for continuous self-supervised multi-type speech acoustic feature representation based on distillation in Example 1.

[0092] The computer-readable storage medium of this embodiment may be an internal storage unit of the terminal, such as a hard disk or memory of the terminal; the computer-readable storage medium of this embodiment may also be an external storage device of the terminal, such as a plug-in hard disk, a smart memory card, a secure digital card, a flash memory card, etc. equipped on the terminal; further, the computer-readable storage medium may also include both an internal storage unit of the terminal and an external storage device.

[0093] The computer-readable storage medium of this embodiment is used to store computer programs and other programs and data required by the terminal. The computer-readable storage medium can also be used to temporarily store data that has been output or is to be output.

[0094] Embodiment 3:

[0095] The computer device of this embodiment includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, the steps of a method for continuous self-supervised multi-type speech acoustic feature representation based on distillation in Embodiment 1 are implemented.

[0096] In this embodiment, the processor may be a central processing unit, or other general-purpose processors, digital signal processors, application-specific integrated circuits, readily available programmable gate arrays or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc. The memory may include a read-only memory and a random access memory, and provide instructions and data to the processor. A portion of the memory may also include a non-volatile random access memory. For example, the memory may also store information about the device type.

[0097] Those skilled in the art will appreciate that the disclosed content of the embodiments may be provided as methods, systems, or computer program products. Therefore, the present solution may adopt the form of hardware embodiments, software embodiments, or embodiments combining software and hardware. Moreover, the present solution may adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage and optical storage, etc.) containing computer-usable program codes.

[0098] The present solution is described with reference to the method according to the embodiment of the present solution and the flowchart and / or block diagram of the computer program product. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of the processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions; these computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 one or more processes and / or methods Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0099] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 one or more processes and / or methods Figure 1 A function specified in one or more boxes.

[0100] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 one or more processes and / or methods Figure 1 The steps for the functions specified in one or more boxes.

[0101] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing related hardware through a computer program, and the program can be stored in a computer-readable storage medium, and when the program is executed, it can include the processes of the embodiments of the above-mentioned methods. The storage medium can be a disk, an optical disk, a read-only memory (ROM) or a random access memory (RAM).

[0102] Finally, it should be noted that the above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art can still modify the technical solutions described in the aforementioned embodiments or replace some of the technical features therein by equivalents. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A method for continuously self-supervising multi-type speech acoustic feature representation based on distillation, characterized by: The method comprises: Step 1: Voice data collection: collect different types of subject voice data from existing public datasets; Step 2: Preprocess the collected speech data and assign different types of preprocessed speech data to different training stages. Assuming that the total number of tasks is T, corresponding to T training stages, the speech data subset of the tth training stage is recorded as V t , extract the Mel spectrogram features corresponding to the speech data subsets of each training stage, construct the training subsets of each training stage, and the training subset of the tth training stage is recorded as D t ; Step 3: Initialize a masked self-supervised feature extraction model in the first training stage, and use the training subset D1 of the first training stage generated in step 2 to model the model to obtain a masked self-supervised feature extraction model M1; construct a replay buffer Β1 according to the reconstructed features output by the model M1; Step 4: When t ≥ 2, in the training subset D of the tth training stage t The masked self-supervised model obtained in the t-1th stage is trained and the model parameters are updated to obtain the masked self-supervised feature extraction model M t ; According to model M t The reconstructed features of the output construct the replay buffer Β t ; and use cosine similarity as a regularizer for feature distillation on the replay buffer; Step 5: Let t = t + 1, and repeat step 4 until the training is completed at t = T, and finally obtain the masked self-supervised feature extraction model M T , and its output is the acoustic features of the multi-type speech to be extracted.

2. The method for continuously self-supervising multi-type speech acoustic feature representation based on distillation as claimed in claim 1, characterized in that: The voice data in step 1 includes: The voice data adopts multiple types of voice data, including continuous vowels, repeated syllables and situational dialogues; Among them, sustained vowels include sustained vowel / a / , sustained vowel / i / , sustained vowel / o / ; repeated syllables include / pa-ka-la / , / pa-ta-la / ; situational dialogues include a variety of different dialogue contents.

3. The method for representing multi-type speech acoustic features based on continuous self-supervision based on distillation as claimed in claim 1, characterized in that: The step 2 comprises: S201, dividing the original speech signal into frames, and using the Librosa audio processing library to remove the silent frames at the beginning and end of the speech; converting the pre-processed speech data into a spectrogram, and passing it through a Mel filter bank to obtain the corresponding Mel spectrogram features; S202, dividing the voice data subsets according to the different voice data types mentioned in the improvement of step 1; and assigning different voice data subsets to different training stages; Among them, the speech data subset of the tth training stage is denoted as V t , 1≤t≤T; V t The corresponding Mel spectrogram set constitutes the training subset D of the tth training stage t , 1≤t≤T; To include n t The Mel spectrogram feature set of samples, where the Mel spectrogram feature x i,t is the i-th sample in the t-th training subset, and each sample has no label.

4. The method for representing acoustic features of multiple types of speech based on continuous self-supervision based on distillation as claimed in claim 1, characterized in that: The step 3 comprises: S301, train a basic masked self-supervised speech feature extraction model M1, and use the training subset of the first training stage As input to the masked self-supervised speech feature extraction model; Using the mask image as an excuse task, the Mel spectrogram features in D1 are randomly masked, and then the masked Mel spectrogram features are reconstructed through the Transformer encoder to obtain the reconstructed Mel spectrogram features. The L1 loss is calculated between the input Mel-level spectrogram features and the reconstructed Mel-level spectrogram features, and the calculation formula is: Among them, x i,t (i=1,2,…,n t ) is the input sequence, i.e. the original Mel spectrogram feature, y i,t (i=1,2,…,n t ) is the output sequence of the masked self-supervised model, i.e., the reconstructed Mel-spectrogram features; S302. Use K-means clustering method to select representative samples for replay: The reconstructed spectrogram features output by the masked self-supervised feature extraction model M1 in S301 Cluster into C clusters, select k samples closest to the center and q samples farthest from the center from each cluster to form the replay buffer Β1 = {m 1,1 ,m 2,1 ,…,m i,1 ,…,m k+q,1 }; S303, in order to increase the diversity of the replay buffer and prevent the model from overfitting, according to the formula: x′=λx1+(1-λ)x2, a binary hybrid strategy is used to perform data enhancement on the buffer B1; Among them, λ represents a uniform random variable between [0,1]; x′ represents the generated new sample, that is, the enhanced sample; x1 and x2 represent samples randomly selected from B1 respectively; Get the updated replay buffer Β1 = {m 1,1 ,m 2,1 ,…,m i,1 ,…,m k+q,1 ,m k+q+1,1 …,m n,1 }; Among them, m k+q+1,1 …,m n,1 Represents an enhanced sample.

5. The method for continuously self-supervising multi-type speech acoustic feature representation based on distillation as claimed in claim 1, characterized in that: The step 4 comprises: S401, in the training subset D t Calculate the L1 loss on the model M t-1 Update the parameters to obtain the updated masked self-supervised feature extraction model M t ; S402, copy the masked self-supervised model M obtained by training in the t-1th training stage t-1 , whose model parameters represent the set of knowledge learned in the t-1th training stage; S403, replay buffer B t-1 Feature input model M t , output the reconstructed Mel spectrogram feature E t ={e 1,t ,e 2,t ,…,e i,t ,…,e n,t }; Replay buffer B t-1 Feature input M t-1 , output the reconstructed Mel spectrogram feature E t-1 ={e 1,(t-1) ,e 2,(t-1) ,…,e i,(t-1) ,…,e n,(t-1) }; Calculate E t and E t-1 The cosine similarity between them is used as a regular term. When the cosine similarity of the outputs of the two models is greater than the threshold, the features are similar, indicating that the old knowledge has not been forgotten. Based on this, the loss function is defined as: Among them, S(e i,t ,e i,t-1 ) computes two self-supervised models M t and M t-1 The cosine similarity of the output features of Among them, ||·||2 means finding the second norm, (·) T Represents the transpose of a vector. The value range of cosine similarity is between [-1,1], 1 means completely similar, -1 means completely opposite, and 0 means they are orthogonal; S404, in the training subset D t Referring to S302, representative samples are selected to form a replay buffer B. t ={m 1,t ,m 2,t ,…,m i,t ,…,m n,t }; Refer to S303 for buffer B t Use binary hybrid strategy for data enhancement and construct a new replay buffer B t And clear B t-1 data.

6. A speech feature representation system for implementing a method for continuously self-supervising multi-type speech acoustic feature representation based on distillation according to any one of claims 1 to 5, characterized in that: The system comprises: a speech feature extraction module, a training subset generation module, a sequential self-supervisory model training module, a replay buffer module and a feature distillation module; The speech feature extraction module is used to pre-process the speech data and extract the Mel spectrum features of the speech; The training subset generation module is used to divide the speech data into speech data subsets and allocate them to each training stage, and the Mel spectrograms corresponding to the speech data subsets constitute the training subsets of each training stage; The sequential self-supervised model training module is used to input the Mel-spectrogram features of the speech data into the self-supervised feature extraction model in the order of the training phase, perform random masking and reconstruction, and extract the output of the last layer of the model as the acoustic features of multiple types of speech; The replay buffer module is used to retain part of the data of the current stage as buffer data of the next stage and perform data enhancement; The feature distillation module is used to input the replay buffer data into M t and the copied M t-1 In , the cosine similarity of the input features is calculated for feature distillation.

7. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the steps in a method for continuous self-supervised multi-type speech acoustic feature representation based on distillation as described in any one of claims 1-6 are implemented.

8. A computer device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that: When the processor executes the program, it implements the steps in a method for continuous self-supervised multi-type speech acoustic feature representation based on distillation as described in any one of claims 1-6.

Images