A distillation-based continuous self-supervised multi-type speech acoustic feature representation method

Abstract

The present invention relates to the field of deep learning technology, and specifically to a method for continuously self-supervising the acoustic feature representation of multiple types of speech based on distillation, comprising: collecting speech data; preprocessing the collected speech data, and allocating the preprocessed different types of speech data to different training stages; initializing a masked self-supervised feature extraction model; constructing a replay buffer according to the reconstructed features output by the model and performing data enhancement; updating the masked self-supervised feature extraction model in the next stage, and performing feature distillation using the replay buffer; obtaining the acoustic features of the multiple types of speech to be extracted; the method trains the multiple types of speech data in sequence, and can prevent data conflicts between different data; secondly, using cosine similarity as a regular term to perform feature distillation on the replay buffer data to retain knowledge, which can prevent the forgetting of old knowledge while enhancing the plasticity of the model.

Description

Technical Field

[0001] The present invention relates to the field of deep learning technology, and in particular to a method for representing multi-type speech acoustic features through continuous self-supervision based on distillation. Background Art

[0002] Due to the decline in the flexible coordination ability of the vocal organs, Parkinson's disease patients often experience symptoms such as difficulty in pronunciation and unstable pronunciation in the early stages of the disease. Acoustic experts have designed a variety of corpora, including sustained vowels, repeated syllables, and situational dialogues, for the analysis and evaluation of the subjects' speech ability. Since the collection of speech data from Parkinson's disease patients is difficult and the annotation cost is high, there are relatively few public Parkinson's disease speech datasets. Moreover, most of these public datasets are derived from the speech of people in a single region, and have the characteristics of different collection methods, insufficient capacity, and small differences in the pronunciation characteristics of the subjects' native language. These characteristics of Parkinson's disease speech datasets affect the application of artificial intelligence technology in this field, mainly manifested in the following aspects: (1) Obtaining sufficient pre-training speech data is a challenging, expensive, and time-consuming process. Under existing conditions, there are certain difficulties in increasing the amount of labeled data. Therefore, it is difficult to enhance the performance of the network model by increasing the available training data. (2) In different Parkinson's disease speech datasets, language differences, corpus differences, etc. lead to inconsistent probability distributions of speech data. Therefore, Parkinson's disease speech feature extraction models trained on a single speech dataset often cannot avoid the impact of type differences in multi-type scenarios (different languages ​​and different corpora), resulting in the extracted features not fully reflecting the pathological information of Parkinson's disease. Therefore, it is urgent to seek a new method for extracting acoustic features from Parkinson's disease speech that can fully utilize Parkinson's disease speech data resources while avoiding time-consuming and labor-intensive manual labeling.

[0003] Standard strongly supervised learning relies on data, which often requires time-consuming and labor-intensive manual labeling, limiting its practical application. Continuous self-supervised learning is an effective pre-training method in deep learning. It can directly learn the underlying structural representation of data from unlabeled data streams. More importantly, it retains the learned data stream information to a certain extent. The resulting high-level features help improve the performance and convergence speed of downstream tasks.

[0004] Currently, there are two main types of strategies for resisting catastrophic forgetting in continuous self-supervised learning: (1) The first type of strategy selects appropriate samples in each data stream as replay samples, and in subsequent data stream training, adds the replay samples to the training to prevent catastrophic forgetting. However, the repeated appearance of replay samples in subsequent training may lead to the leakage of private information in certain fields. (2) The second type of strategy adds some regularization techniques to the network optimization objectives, such as knowledge distillation, so that the network can maintain consistency with the previous state to a certain extent in the training of subsequent data streams. This type of strategy easily limits the plasticity of the network and cannot adapt to the learning of new features. Summary of the Invention

[0005] The purpose of the present invention is to provide a distillation-based continuous self-supervised multi-type speech acoustic feature representation method to solve the problems raised in the above background technology.

[0006] In order to solve the above technical problems, the present invention provides the following technical solutions:

[0007] A distillation-based continuous self-supervised multi-type speech acoustic feature representation method, the method includes:

[0008] Step 1: Voice data collection: Collect different types of subject voice data from existing public datasets, or collect different types of subject voice data;

[0009] Step 2: Preprocess the collected speech data and assign different types of preprocessed speech data to different training stages. Assuming that the total number of tasks is T (corresponding to T training stages), the speech data subset of the tth training stage is recorded as V t , extract the Mel spectrogram features corresponding to the speech data subset of each training stage, construct the training subset of each training stage, and the training subset of the tth training stage is recorded as D t ;

[0010] Step 3: Initialize a masked self-supervised feature extraction model in the first training phase, and use the training subset D1 of the first training phase generated in step 2 to train the model to obtain a masked self-supervised feature extraction model M1; construct a replay buffer B1 based on the reconstructed features output by the model M1;

[0011] Step 4: When t≥2, the training subset D in the tth training stage t The masked self-supervised model obtained in the t-1th stage is trained and the model parameters are updated to obtain the masked self-supervised feature extraction model M t ; According to model M t The reconstructed features of the output construct the replay buffer Β t; and use cosine similarity as a regularizer for feature distillation on the replay buffer;

[0012] Step 5: Let t = t + 1, and repeat step 4 until the training is completed at t = T, and finally obtain the masked self-supervised feature extraction model M T , and its output is the acoustic features of the multi-type speech to be extracted.

[0013] Preferably, the voice data in step 1 includes:

[0014] The voice data adopts multiple types of voice data, including sustained vowels, repeated syllables and situational dialogues;

[0015] Among them, sustained vowels include sustained vowel / a / , sustained vowel / i / , and sustained vowel / o / ; repeated syllables include / pa-ka-la / and / pa-ta-la / ; and situational dialogues include a variety of different dialogue contents.

[0016] Preferably, step 2 includes:

[0017] S201, framing the original speech signal and removing the silence frames at the beginning and end of the speech using the Librosa audio processing library; converting the preprocessed speech data into a spectrogram, and passing it through a Mel filter bank to obtain the corresponding Mel spectrogram features;

[0018] S202, dividing the speech data into subsets (e.g., into a sustained vowel / a / subset, a repeated syllable / pa-ka-la / subset, a situational dialogue subset, etc.) according to the different speech data types mentioned in the improvement of step 1; and assigning different speech data subsets to different training stages;

[0019] Among them, the speech data subset of the tth training stage is recorded as V t , 1≤t≤T;

[0020] V t The corresponding Mel spectrogram set constitutes the training subset D of the t-th training stage t , 1≤t≤T;

[0021] To include n t Mel spectrogram feature set of samples, where the Mel spectrogram feature x i,t is the i-th sample in the t-th training subset, and each sample has no label.

[0022] Preferably, step 3 includes:

[0023] S301, train a basic masked self-supervised speech feature extraction model M1, and use the training subset of the first training stage As input to the masked self-supervised speech feature extraction model;

[0024] Using the mask image as an excuse task, the Mel spectrogram features in D1 are randomly masked, and then the masked Mel spectrogram features are reconstructed through the Transformer encoder to obtain the reconstructed Mel spectrogram features.

[0025] The L1 loss is calculated between the input Mel-level spectrogram features and the reconstructed Mel-level spectrogram features, and its calculation formula is:

[0026] Among them, x i,t (i=1,2,…,n t ) is the input sequence, i.e. the original Mel spectrogram feature, y i,t (i=1,2,…,n t ) is the output sequence of the masked self-supervised model, i.e., the reconstructed Mel-spectrogram features;

[0027] S302. Use the K-means clustering method to select representative samples for replay:

[0028] The reconstructed spectrogram features output by the masked self-supervised feature extraction model M1 in S301 Cluster into C clusters, select k samples closest to the center and q samples farthest from the center from each cluster to form the replay buffer Β1={m 1,1 ,m 2,1 ,…,m i,1 ,…,m k+q,1};

[0029] S303. To increase the diversity of the replay buffer and prevent model overfitting, a binary hybrid strategy is used to perform data enhancement on buffer B1 according to the formula: x′=λx1+(1-λ)x2;

[0030] Where λ represents a uniform random variable between [0,1]; x′ represents the generated new sample, that is, the enhanced sample; x1 and x2 represent samples randomly selected from B1 respectively;

[0031] Get the updated replay buffer Β1={m 1,1 ,m 2,1 ,…,m i,1 ,…,m k+q,1 ,m k+q+1,1 …,m n,1};

[0032] Among them, m k+q+1,1 …,m n,1 represents the enhanced sample.

[0033] Preferably, step 4 includes:

[0034] S401, in the training subset D t Calculate the L1 loss on the model M t-1 Update the parameters to obtain the updated masked self-supervised feature extraction model M t ;

[0035] S402, copy the masked self-supervised model M obtained by training in the t-1th training stage t-1 , whose model parameters represent the set of knowledge learned in the t-1th training stage;

[0036] S403, replay buffer B t-1 Feature input model M t , output the reconstructed Mel spectrogram feature E t ={e 1,t ,e 2,t ,…,e i,t ,…,e n,t}; will replay buffer B t-1 Feature input M t-1 , output the reconstructed Mel spectrogram feature E t-1 ={e 1,(t-1) ,e 2,(t-1) ,…,e i,(t-1) ,…,e n,(t-1)};

[0037] Calculate E t and E t-1 The cosine similarity between the two models is used as a regularization term to prevent large differences between the features learned in the previous training stage and the current training stage data. When the cosine similarity output by the two models is greater than the threshold, the features are similar, indicating that the old knowledge has not been forgotten. Using cosine similarity as a regularization term will not directly affect the model's learning of new features. As long as the cosine similarity is non-negative, the total loss will not increase. This prevents the forgetting of old knowledge while enhancing the plasticity of new features.

[0038] Based on this, the loss function is defined as:

[0039] Among them, S(e i,t ,e i,t-1 ) calculated two self-supervised models M t and M t-1 The cosine similarity of the output features of

[0040]

[0041] Among them, ||·||2 means finding the two-norm, (·)T Represents the transpose of a vector. The cosine similarity value range is between [-1, 1], where 1 represents complete similarity, -1 represents complete oppositeness, and 0 represents orthogonality.

[0042] S404, in the training subset D t Referring to S302, select representative samples to form a replay buffer B. t ={m 1,t ,m 2,t ,…,m i,t ,…,m n,t}; Refer to S303 for buffer B t Adopt binary hybrid strategy for data enhancement and construct a new replay buffer B t and clear B t-1 data.

[0043] Preferably, a speech feature representation system for implementing a distillation-based continuous self-supervised multi-type speech acoustic feature representation method comprises: a speech feature extraction module, a training subset generation module, a sequential self-supervised model training module, a replay buffer module and a feature distillation module;

[0044] The speech feature extraction module is used to preprocess the speech data and extract the Mel spectrum features of the speech;

[0045] The training subset generation module is used to divide the speech data into speech data subsets and assign them to each training stage. The Mel spectrograms corresponding to the speech data subsets constitute the training subsets of each training stage.

[0046] The sequential self-supervised model training module is used to input the Mel-spectrogram features of speech data into the self-supervised feature extraction model in the order of the training phase, perform random masking and reconstruction, and extract the output of the last layer of the model as the acoustic features of multiple types of speech;

[0047] The replay buffer module is used to retain part of the data in the current stage as the buffer data for the next stage and perform data enhancement;

[0048] The feature distillation module is used to input the replay buffer data into M t and the copied M t-1 In

[15] , the cosine similarity of the input features is calculated for feature distillation.

[0049] A computer-readable storage medium having a computer program stored thereon, characterized in that when the program is executed by a processor, the steps in the above-mentioned method for continuous self-supervised multi-type speech acoustic feature representation based on distillation are implemented.

[0050] A computer device includes a memory, a processor, and a computer program stored in the memory and running on the processor. When the processor executes the program, the steps in the above-mentioned distillation-based continuous self-supervised multi-type speech acoustic feature representation method are implemented.

[0051] Compared with the prior art, the present invention has the following beneficial effects:

[0052] The present invention adopts sequential self-supervised training for multi-type speech data. Compared with joint self-supervised training, this method can prevent data conflicts between different data. Secondly, cosine similarity is used as a regularizer to perform feature distillation on the replay buffer data to retain knowledge, which can prevent the forgetting of old knowledge while enhancing the plasticity of new features.

[0053] The present invention utilizes a sequential self-supervised model to learn generalized representations of different types of speech data sets without labeling the training data, thereby preventing conflicts between different types of data. At the same time, it introduces a continuous learning method to prevent catastrophic forgetting of the model and improve the plasticity of the model. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] The accompanying drawings are used to provide a further understanding of the present invention and constitute a part of the specification. Together with the embodiments of the present invention, they are used to explain the present invention and do not constitute a limitation of the present invention. In the accompanying drawings:

[0055] Figure 1 This is a flow chart of a method for continuously self-supervising multi-type speech acoustic feature representation based on distillation of the present invention;

[0056] Figure 2 It is a schematic diagram of characteristic distillation of the present invention. DETAILED DESCRIPTION

[0057] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0058] See also Figure 1-Figure 2 , the present invention provides a technical solution:

[0059] Example 1:

[0060] like Figure 1 As shown, the present invention is a continuous self-supervised multi-type speech feature representation method based on distillation, which specifically includes:

[0061] Step 1: Speech signal collection / collection: Collect different types of subject speech data from existing public datasets, or collect speech data from different types of subjects, including sustained vowels, repeated syllables, and contextual dialogues. Sustained vowels can include sustained vowels / a / , sustained vowels / i / , sustained vowels / o / , etc.; repeated syllables can include / pa-ka-la / and / pa-ta-la / ; and contextual dialogues can also contain a variety of different conversation contents.

[0062] Step 2: Preprocess the speech data collected / collected in step 1 and assign different types of preprocessed speech data to different training stages. Assuming that the total number of tasks is T (corresponding to T training stages, T is an integer greater than 1), the speech data subset of the tth training stage is denoted as V t (1≤t≤T); extract the Mel spectrogram features corresponding to the speech data subset of each training stage, construct the training subset of each training stage, and the training subset of the tth training stage is recorded as D t (1≤t≤T). Specifically including:

[0063] S201. Frame the original speech signal and remove the silence frames at the beginning and end of the speech using the Librosa audio processing library. Convert the pre-processed speech data into a spectrogram and pass it through a Mel filter bank to obtain the corresponding Mel spectrogram features.

[0064] S202: Divide the speech data into subsets according to the different speech data types mentioned in the improvement of step 1, and assign different speech data subsets to different training stages, wherein the speech data subset of the t-th training stage is denoted as V t (1≤t≤T), V t The corresponding Mel spectrogram set constitutes the training subset D of the t-th training stage t .

[0065] To include n t Mel spectrogram feature set of samples, where the Mel spectrogram feature x i,t is the i-th sample in the t-th training subset, and each sample has no label.

[0066] Step 3: Initialize a masked self-supervised feature extraction model in the first training phase, and use the training subset D1 of the first training phase generated in step 2 to train the model to obtain a masked self-supervised feature extraction model M1; and construct a replay buffer B1 based on the reconstructed features output by the model M1. Specifically, the following steps are performed:

[0067] S301, train a basic masked self-supervised speech feature extraction model M1, and use the training subset of the first training stage As the input of the masked self-supervised speech feature extraction model. Using the masked image as an excuse task, the Mel spectrogram features in D1 are randomly masked, that is, 15% of the frames in each input sequence are selected for masking, and then the masked Mel spectrogram features are reconstructed through the Transformer encoder to obtain the reconstructed Mel spectrogram features. The L1 loss is calculated between the input Mel-level spectrogram features and the reconstructed Mel-level spectrogram features, and its calculation formula is:

[0068] Among them, x i,t (i=1,2,…,n t ) is the input sequence (i.e. the original Mel spectrogram features), y i,t (i=1,2,…,n t ) is the output sequence of the masked self-supervised model (i.e., the reconstructed Mel-spectrogram features).

[0069] S302, using the K-means clustering method to select representative samples (i.e., the points closest to the center of each cluster and the points at the distribution boundary) for replay. Specifically, the reconstructed spectrogram features output by the masked self-supervised feature extraction model M1 in S301 are Cluster into C clusters, select k samples closest to the center and q samples farthest from the center from each cluster to form a replay buffer Β1 = {m 1,1 ,m 2,1 ,…,m i,1 ,…,m k+q,1}.

[0070] S303, in order to increase the diversity of the replay buffer and prevent the model from overfitting, the buffer B1 is enhanced by using a binary mixing (InstanceMixup) strategy, as shown below: x′=λx1+(1-λ)x2;

[0071] Where λ is a uniform random variable between [0,1]; x′ is the generated new sample, i.e. the enhanced sample; x1 and x2 are samples randomly selected from B1. The updated replay buffer

[0072] Β1={m 1,1 ,m 2,1 ,…,m i,1 ,…,m k+q,1 ,m k+q+1,1 …,m n,1}, where m k+q+1,1 …,m n,1 To enhance the sample;

[0073] Step 4: Training subset D in the tth (t≥2) training stage tThe masked self-supervised model obtained in the (t-1)th stage is trained and the model parameters are updated to obtain the masked self-supervised feature extraction model M t ; According to model M t The reconstructed features of the output construct the replay buffer Β t ; and use cosine similarity as a regularizer to perform feature distillation on the replay buffer. Specifically:

[0074] S401, in the training subset D t Calculate the L1 loss (refer to the L1 loss calculation formula in S301) for the model M t-1 Update the parameters to obtain the updated masked self-supervised feature extraction model M t ;

[0075] S402, copy the masked self-supervised model M obtained by training in the t-1th training stage t-1 , whose model parameters represent the set of knowledge learned in the t-1th training stage;

[0076] S403, replay buffer B t-1 Feature input model M t , output the reconstructed Mel spectrogram feature E t ={e 1,t ,e 2,t ,…,e i,t ,…,e n,t}; will replay buffer B t-1 Feature input M t-1 , output the reconstructed Mel spectrogram feature E t-1 ={e 1,(t-1) ,e 2,(t-1) ,…,e i,(t-1) ,…,e n,(t-1)}. Calculate E t and E t-1 The cosine similarity between the two models is used as a regularization term to prevent large differences between the features learned in the previous training phase and the features learned in the current training phase. Here, when the features output by the two models are similar, it means that the old knowledge has not been forgotten. Using cosine similarity as a regularization term does not directly affect the model's learning of new features. As long as the cosine similarity is non-negative, the overall loss will not increase. This prevents the forgetting of old knowledge while enhancing the plasticity of new features. The loss function here is defined as follows:

[0077]

[0078] Among them, S(e i,t ,e i,t-1 ) calculated two self-supervised models M t and M t-1The cosine similarity of the output features is defined as follows:

[0079] Among them, ||·||2 means finding the two-norm, (·) T Represents the transpose of a vector. The cosine similarity value range is between [-1, 1], where 1 represents complete similarity, -1 represents complete oppositeness, and 0 represents orthogonality.

[0080] S404, in the training subset D t Referring to S302, select representative samples to form a replay buffer B. t ={m 1,t ,m 2,t ,…,m i,t ,…,m n,t}; Refer to S303 for buffer B t Adopt the binary mixing (InstanceMixup) strategy for data enhancement and construct a new replay buffer B t and clear B t-1 data.

[0081] Step 5: Let t = t + 1, and repeat step 4 until the training is completed at t = T, and finally obtain the masked self-supervised feature extraction model M T , and its output is the acoustic features of the multi-type speech to be extracted.

[0082] A continuous self-supervised multi-type speech feature representation system based on distillation is also provided, which includes:

[0083] Speech feature extraction module: pre-processes speech data and extracts the Mel spectrum features of speech;

[0084] Training subset generation module: divides the speech data into speech data subsets and assigns them to each training stage. The Mel spectrograms corresponding to the speech data subsets constitute the training subsets of each training stage.

[0085] Sequential self-supervised model training module: The Mel-spectrogram features of the speech data are input into the self-supervised feature extraction model in the order of the training phase, randomly masked and reconstructed, and the output of the last layer of the model is extracted as the acoustic features of multiple types of speech;

[0086] Replay buffer module: retains part of the data of the current stage as the buffer data of the next stage and performs data enhancement;

[0087] Feature distillation module: input the replay buffer data into M t and the copied M t-1 In

[15] , the cosine similarity of the input features is calculated for feature distillation;

[0088] The invention uses three Parkinson's disease speech datasets in different languages, including the Maxlittle dataset (native English), the Italian Parkinson's disease dataset (native Italian), and the self-collected dataset (native Chinese). The Maxlittle dataset contains vowel data, and the other two datasets contain sustained vowels / a / , repeated syllables, and designated text reading scenario dialogue content. The vowel / a / can reflect the ability of the vocal cords to vibrate continuously and the health of the laryngeal muscles; repeated syllables can measure the regularity of the movements of organs such as the lips, teeth, and tongue, as well as the occlusion ability of the mandibular joint; and designated text reading scenario dialogues can help comprehensively evaluate all vocal components and the comprehensibility of pronunciation. When analyzing the various motor symptoms of Parkinson's disease through speech signals, it is necessary to use signal processing algorithms to analyze the test speech and extract speech feature information that can characterize the pathological characteristics of Parkinson's disease motor symptoms.

[0089] In summary, using mel-spectrograms, which retain more original speech information, as input for self-supervised models, and Transformer encoder-based self-supervised models, can capture longer-term temporal dependencies when dealing with time series data. Integrating multilingual datasets can provide the model with richer and more diverse speech features, helping to uncover potential commonalities and differences, and improving model learning and performance.

[0090] Example 2:

[0091] The computer-readable storage medium of this embodiment stores a computer program thereon, which, when executed by a processor, implements the steps of a method for continuous self-supervised multi-type speech acoustic feature representation based on distillation in Example 1.

[0092] The computer-readable storage medium of this embodiment may be an internal storage unit of the terminal, such as a hard disk or memory of the terminal; the computer-readable storage medium of this embodiment may also be an external storage device of the terminal, such as a plug-in hard disk, a smart memory card, a secure digital card, a flash memory card, etc. equipped on the terminal; further, the computer-readable storage medium may also include both an internal storage unit of the terminal and an external storage device.

[0093] The computer-readable storage medium of this embodiment is used to store computer programs and other programs and data required by the terminal. The computer-readable storage medium can also be used to temporarily store data that has been output or is to be output.

[0094] Example 3:

[0095] The computer device of this embodiment includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, the steps of the method for continuous self-supervised multi-type speech acoustic feature representation based on distillation in Example 1 are implemented.

[0096] In this embodiment, the processor can be a central processing unit, or other general-purpose processors, digital signal processors, application-specific integrated circuits, off-the-shelf programmable gate arrays or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor, etc. The memory can include read-only memory and random access memory, and provide instructions and data to the processor. A part of the memory can also include non-volatile random access memory. For example, the memory can also store information about the device type.

[0097] Those skilled in the art will appreciate that the disclosed contents of the embodiments may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of a hardware embodiment, a software embodiment, or an embodiment combining software and hardware. Furthermore, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage and optical storage, etc.) containing computer-usable program code.

[0098] The present solution is described with reference to the flowcharts and / or block diagrams of the methods and computer program products according to the embodiments of the present solution. It should be understood that each process and / or block in the flowcharts and / or block diagrams, as well as the combination of the processes and / or blocks in the flowcharts and / or block diagrams, can be implemented by computer program instructions; these computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the steps in the flowcharts and / or block diagrams. Figure 1 a process or multiple processes and / or methods Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0099] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 a process or multiple processes and / or methods Figure 1 The function specified in one or more boxes.

[0100] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing the instructions executed on the computer or other programmable device for implementing the process. Figure 1 a process or multiple processes and / or methods Figure 1 The steps for the function specified in one or more boxes.

[0101] Those skilled in the art will appreciate that all or part of the processes in the above-described method embodiments can be implemented by instructing related hardware through a computer program. The program can be stored in a computer-readable storage medium, and when executed, the program can include the processes in the above-described method embodiments. The storage medium can be a magnetic disk, an optical disk, a read-only memory (ROM), or a random access memory (RAM).

[0102] Finally, it should be noted that the above descriptions are merely preferred embodiments of the present invention and are not intended to limit the present invention. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art will be able to modify the technical solutions described in the aforementioned embodiments or substitute equivalents for some of the technical features. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention shall be included within the scope of protection of the present invention.

Claims

1. A distillation-based continuous self-supervised multi-type speech acoustic feature representation method, characterized by: The method comprises: Step 1: Voice data collection: Collect different types of subject voice data from existing public datasets; Step 2: Preprocess the collected speech data and assign different types of preprocessed speech data to different training stages. Assuming the total number of tasks is , corresponding to T training stages, the The speech data subset of the training stage is denoted as , extract the Mel spectrogram features corresponding to the speech data subset of each training stage, construct the training subset of each training stage, and the training subset of the tth training stage is recorded as ; Step 3: Initialize a masked self-supervised feature extraction model in the first training phase, using the training subset of the first training phase generated in step 2. The model is modeled to obtain the masked self-supervised feature extraction model According to the model The reconstructed features of the output are used to construct the replay buffer ; Step 4 When , in the training subset of the tth training stage The masked self-supervised model obtained in the t-1th stage is trained and the model parameters are updated to obtain the masked self-supervised feature extraction model According to the model The reconstructed features of the output are used to construct the replay buffer ; and use cosine similarity as a regularizer for feature distillation on the replay buffer; Step 5: , repeat step 4 until The training is completed when , and the masked self-supervised feature extraction model is finally obtained , its output is the acoustic features of the multi-type speech to be extracted; Wherein, the step 4 includes: S401, in the training subset Calculate the L1 loss on the model Update the parameters to obtain the updated masked self-supervised feature extraction model ; Among them, the L1 loss is calculated as follows: ; in, is the input sequence, i.e. the original Mel spectrogram features, The output sequence of the masked self-supervised model is the reconstructed Mel-spectrogram features; S402, copy The masked self-supervised model trained in the training phase , whose model parameters represent the The collection of knowledge learned in the training phase; S403, replay the buffer Feature Input Model , output the reconstructed Mel spectrogram features ; Replay buffer Feature Input , output the reconstructed Mel spectrogram features ; calculate and The cosine similarity between them is used as a regular term. When the cosine similarity of the outputs of the two models is greater than the threshold, the features are similar, indicating that the old knowledge has not been forgotten. Based on this, the loss function is defined as: ; in, Two self-supervised models were calculated and The cosine similarity of the output features of ; in, Indicates the second norm, Represents the transpose of the vector. The cosine similarity value range is between [-1, 1], where 1 represents complete similarity, -1 represents complete oppositeness, and 0 represents orthogonality. S404, in the training subset Refer to S302 to select representative samples and form a replay buffer ; Refer to S303 for buffer zone Adopting binary hybrid strategy for data enhancement and constructing new replay buffer and clear data.

2. The method for continuously self-supervising multi-type speech acoustic feature representation based on distillation according to claim 1, characterized in that: The voice data in step 1 includes: The voice data adopts multiple types of voice data, including sustained vowels, repeated syllables and situational dialogues; Among them, sustained vowels include sustained vowel / a / , sustained vowel / i / , and sustained vowel / o / ; repeated syllables include / pa-ka-la / and / pa-ta-la / ; and situational dialogues include a variety of different dialogue contents.

3. The method for continuously self-supervising multi-type speech acoustic feature representation based on distillation according to claim 1, characterized in that: The step 2 comprises: S201, framing the original speech signal and removing the silence frames at the beginning and end of the speech using the Librosa audio processing library; converting the preprocessed speech data into a spectrogram, and passing it through a Mel filter bank to obtain the corresponding Mel spectrogram features; S202, dividing the voice data into subsets according to the different voice data types mentioned in the improvement of step 1; and assigning different voice data subsets to different training stages; Among them, The speech data subset of the training stage is denoted as , ; The corresponding Mel spectrogram set constitutes the training subset for each training phase , ; To include Mel spectrogram feature set of samples, where Mel spectrogram feature is the i-th sample in the t-th training subset, and each sample has no label.

4. The method for continuously self-supervising multi-type speech acoustic feature representation based on distillation according to claim 1, characterized in that: The step 3 comprises: S301. Train a basic masked self-supervised speech feature extraction model , the training subset of the first training stage As input to the masked self-supervised speech feature extraction model; Using masked images as a pretext task, The Mel spectrogram features in the mel language are randomly masked, and then the masked mel spectrogram features are reconstructed through the Transformer encoder to obtain the reconstructed mel spectrogram features ; The L1 loss is calculated between the input Mel-level spectrogram features and the reconstructed Mel-level spectrogram features, and its calculation formula is: ; in, is the input sequence, i.e. the original Mel spectrogram features, The output sequence of the masked self-supervised model is the reconstructed Mel-spectrogram features; S302. Use the K-means clustering method to select representative samples for replay: The masked self-supervised feature extraction model in S301 Output reconstructed spectrogram features Cluster into C clusters, select k samples closest to the center and q samples farthest from the center from each cluster to form the replay buffer ; S303. To increase the diversity of the replay buffer and prevent the model from overfitting, according to the formula: , for the buffer Adopting binary hybrid strategy for data augmentation; in, represents a uniform random variable between [0,1]; Represents the generated new sample, that is, the enhanced sample; 、 Respectively represent A randomly selected sample from Get the updated replay buffer ; in, represents the enhanced sample.

5. A speech feature representation system for implementing the distillation-based continuous self-supervised multi-type speech acoustic feature representation method according to any one of claims 1 to 4, characterized in that: The system includes: a speech feature extraction module, a training subset generation module, a sequential self-supervised model training module, a replay buffer module and a feature distillation module; The speech feature extraction module is used to pre-process the speech data and extract the Mel spectrum features of the speech; The training subset generation module is used to divide the speech data into speech data subsets and allocate them to each training stage, and the Mel spectrograms corresponding to the speech data subsets constitute the training subsets of each training stage; The sequential self-supervised model training module is used to input the Mel spectrogram features of the speech data into the self-supervised feature extraction model in the order of the training phase, perform random masking and reconstruction, and extract the output of the last layer of the model as the acoustic features of multiple types of speech; The replay buffer module is used to retain part of the data of the current stage as buffer data of the next stage and perform data enhancement; The feature distillation module is used to input the replay buffer data into and copied In [15], the cosine similarity of the input features is calculated for feature distillation.

6. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the steps of the method for continuous self-supervised multi-type speech acoustic feature representation based on distillation as described in any one of claims 1 to 4 are implemented.

7. A computer device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that: When the processor executes the program, it implements the steps in a distillation-based continuous self-supervised multi-type speech acoustic feature representation method as described in any one of claims 1-4.

Images