Speech representation model pre-training method and device, electronic equipment and storage medium

Abstract

The present disclosure relates to a speech representation model pre-training method and device, electronic equipment and storage medium. The method comprises: randomly initializing a mapping matrix and a codebook set, performing vector mapping processing on each audio frame in the audio frame sequence of the sample speech based on the mapping matrix to obtain a mapping vector of each audio frame; searching for a target codebook vector matched with each mapping vector from the codebook set and taking the target index thereof as a reference discretization label of the corresponding audio frame; inputting the masked audio frame sequence into a speech representation model to be trained to perform speech representation processing to obtain a speech representation result; predicting a discretization label for the representation result corresponding to each masked position in the speech representation result; and adjusting the model parameters of the speech representation model to be trained based on the difference between the predicted discretization label corresponding to each masked position and the corresponding reference discretization label to obtain a pre-trained speech representation model. The present disclosure improves the pre-training speed and stability of the speech representation model.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the technical field of computer, and particularly relates to a speech representation model pre-training method and device, electronic equipment and storage medium. BACKGROUND

[0002] With the development of machine learning technology, in order to get rid of the dependence of the speech recognition model on a large amount of speech annotation data, the speech representation model is currently pre-trained using unannotated speech data, so that the speech recognition model meeting the downstream task can be obtained based on the pre-trained speech representation model combined with a small amount of speech annotation data, and because the pre-training process of the speech representation model is irrelevant to the downstream task, the speech representation model obtained through pre-training can quickly adapt to multiple different downstream tasks, achieving the effect of repeated use.

[0003] However, the pre-training method of the speech representation model in the related art has the problems of slow training speed, convergence difficulty and poor stability. SUMMARY

[0004] The present disclosure provides a speech representation model pre-training method, device, electronic equipment and storage medium to at least solve the problem of slow training speed, convergence difficulty and poor stability of the pre-training method of the speech representation model in the related art. The technical solutions of the present disclosure are as follows:

[0005] According to a first aspect of the embodiments of the present disclosure, a speech representation model pre-training method is provided, comprising:

[0006] generating a mapping matrix and a codebook set through a random initialization method; the codebook set represents the corresponding relationship between the codebook vectors and the indexes;

[0007] performing vector mapping processing on each audio frame in the sequence of audio frames corresponding to the sample speech based on the mapping matrix, to obtain a mapping vector of each audio frame;

[0008] searching for a target codebook vector matching each mapping vector from the codebook set, and taking the target index corresponding to the target codebook vector matching each mapping vector as the reference discrete label of the audio frame corresponding to each mapping vector;

[0009] inputting the masked sequence of audio frames into a speech representation model to be trained for speech representation processing, to obtain a speech representation result; the masked sequence of audio frames is obtained based on masking processing on multiple positions in the sequence of audio frames;

[0010] performing discrete label prediction on the representation result corresponding to each masked position in the speech representation result, to obtain a predicted discrete label corresponding to each masked position;

[0011] Based on a difference between a predicted discrete label corresponding to each of the occluded positions and a reference discrete label of an audio frame of the occluded position, adjusting model parameters of the to-be-trained speech representation model until a preset training end condition is met, to obtain a pre-trained speech representation model.

[0012] In some example embodiments, the vector mapping processing based on the mapping matrix is performed on each audio frame in the sequence of audio frames corresponding to the sample speech to obtain a mapping vector of each of the audio frames, including:

[0013] Extracting a frequency domain feature of each audio frame in the sequence of audio frames to obtain a sequence of frame features corresponding to the sequence of audio frames;

[0014] Performing global standardization processing on the sequence of frame features to obtain a sequence of standardized frame features;

[0015] Determining a product of the mapping matrix and each standardized frame feature in the sequence of standardized frame features to obtain a mapping vector of each audio frame in the sequence of audio frames.

[0016] In some example embodiments, the global standardization processing on the sequence of frame features to obtain a sequence of standardized frame features includes:

[0017] Concatenating every continuous preset number of frame features in the sequence of frame features to obtain a sequence of concatenated frame features;

[0018] Performing global standardization processing on the sequence of concatenated frame features to obtain a sequence of standardized frame features.

[0019] In some example embodiments, the inputting of the occluded sequence of audio frames into the to-be-trained speech representation model for speech representation processing to obtain a speech representation result includes:

[0020] Performing a preset number of downsampling processes on the sequence of frame features to obtain a sequence of downsampled frame features; each downsampled frame feature in the sequence of downsampled frame features has a same time length as a concatenated frame feature at a corresponding position in the sequence of concatenated frame features;

[0021] Replacing a plurality of downsampled frame features in the sequence of downsampled frame features with a preset occlusion identifier to obtain an occluded sequence of downsampled frame features;

[0022] Inputting the occluded sequence of downsampled frame features into the to-be-trained speech representation model for speech representation processing to obtain a speech representation result.

[0023] In some example embodiments, the step of obtaining the prediction discrete label corresponding to each of the masked positions in the speech representation result comprises:

[0024] obtaining the representation result corresponding to each of the masked positions in the speech representation result;

[0025] for each of the representation results corresponding to the masked positions, predicting a probability that the representation result belongs to each of the reference discrete labels, and determining the reference discrete label corresponding to the maximum probability as the prediction discrete label corresponding to the masked position.

[0026] In some example embodiments, the step of taking the target index corresponding to the target codebook vector matched with each of the mapping vectors as the reference discrete label of the audio frame corresponding to each of the mapping vectors comprises:

[0027] performing information entropy calculation on the target index sequence to obtain a current sequence information entropy, wherein the target index sequence is composed of the target indexes corresponding to the target codebook vectors matched with each of the mapping vectors;

[0028] when the current sequence information entropy exceeds a preset sequence information entropy range, adjusting the codebook set to obtain an adjusted codebook set;

[0029] taking the adjusted codebook set as the codebook set, and performing the steps of searching the target codebook vector matched with each of the mapping vectors from the codebook set, and performing information entropy calculation on the target index sequence to obtain a current sequence information entropy, until the current sequence information entropy does not exceed the preset sequence information entropy range;

[0030] taking each of the target indexes in the target index sequence corresponding to the current sequence information entropy not exceeding the preset sequence information entropy range as the reference discrete label of the corresponding audio frame.

[0031] In some example embodiments, the preset sequence information entropy range comprises a preset sequence information entropy upper limit and a preset sequence information entropy lower limit; and the step of adjusting the codebook set to obtain an adjusted codebook set comprises:

[0032] when the current sequence information entropy is less than the preset sequence information entropy lower limit, increasing the size of the codebook set, and taking the codebook set with the increased size as the adjusted codebook set;

[0033] when the current sequence information entropy is greater than the preset sequence information entropy upper limit, decreasing the size of the codebook set, and taking the codebook set with the decreased size as the adjusted codebook set.

[0034] In some example embodiments, the generating the mapping matrix and the codebook set by random initialization includes:

[0035] The mapping matrix is obtained by initializing the weights according to a uniform distribution.

[0036] The codebook set is obtained by initializing the codebook vectors according to a standard normal distribution.

[0037] In some example embodiments, the codebook set includes a plurality of sub-codebook sets; and the searching, from the codebook set, for a target codebook vector matching each mapping vector includes:

[0038] For each audio frame, a target sub-codebook vector most similar to the mapping vector of the audio frame is searched from each sub-codebook set; and the target sub-codebook vector in each sub-codebook set constitutes a target codebook vector matching the mapping vector of the audio frame.

[0039] According to a second aspect of the embodiments of the present disclosure, a voice representation model pre-training apparatus is provided, including:

[0040] A random initialization unit configured to perform generating a mapping matrix and a codebook set by random initialization; the codebook set representing a corresponding relationship between a codebook vector and an index;

[0041] An audio frame mapping unit configured to perform vector mapping processing on each audio frame in a sequence of audio frames corresponding to a sample voice based on the mapping matrix, to obtain a mapping vector of each audio frame;

[0042] A discretized label determination unit configured to perform searching, from the codebook set, for a target codebook vector matching each mapping vector, and taking a target index corresponding to the target codebook vector matching each mapping vector as a reference discretized label of an audio frame corresponding to each mapping vector;

[0043] A voice representation unit configured to perform inputting a masked sequence of audio frames into a voice representation model to be trained for voice representation processing, to obtain a voice representation result; the masked sequence of audio frames being obtained based on masking processing on a plurality of positions in the sequence of audio frames;

[0044] A label prediction unit configured to perform discretized label prediction on a representation result corresponding to each masked position in the voice representation result, to obtain a predicted discretized label corresponding to each masked position.

[0045] The model parameter adjusting unit is configured to adjust model parameters of the speech representation model to be trained based on a difference between a predicted discretized label corresponding to each of the occluded positions and a reference discretized label of an audio frame of the occluded position until a preset training end condition is met, to obtain a pre-trained speech representation model.

[0046] In some example embodiments, the audio frame mapping unit comprises:

[0047] The frequency domain feature extraction unit is configured to perform extraction of frequency domain features of each audio frame in the audio frame sequence to obtain a frame feature sequence corresponding to the audio frame sequence.

[0048] The global normalization unit is configured to perform global normalization processing on the frame feature sequence to obtain a normalized frame feature sequence.

[0049] The mapping unit is configured to perform determination of a product of the mapping matrix and each normalized frame feature in the normalized frame feature sequence to obtain a mapping vector of each audio frame in the audio frame sequence.

[0050] In some example embodiments, the global normalization unit comprises:

[0051] The frame feature concatenation unit is configured to perform concatenation of every continuous preset number of frame features in the frame feature sequence to obtain a concatenated frame feature sequence.

[0052] The global normalization sub-unit is configured to perform global normalization processing on the concatenated frame feature sequence to obtain a normalized frame feature sequence.

[0053] In some example embodiments, the speech representation unit comprises:

[0054] The down-sampling unit is configured to perform down-sampling processing of the frame feature sequence for a preset number of times to obtain a down-sampled frame feature sequence; a time length of each down-sampled frame feature in the down-sampled frame feature sequence is the same as a time length of a concatenated frame feature at a corresponding position in the concatenated frame feature sequence.

[0055] The replacement unit is configured to perform replacement of a plurality of down-sampled frame features in the down-sampled frame feature sequence with a preset occlusion identifier to obtain an occluded down-sampled frame feature sequence.

[0056] The speech representation sub-unit is configured to perform input of the occluded down-sampled frame feature sequence to a speech representation model to be trained for speech representation processing to obtain a speech representation result.

[0057] In some example embodiments, the label prediction unit comprises:

[0058] a characterizing result acquisition unit configured to perform acquisition of a characterizing result corresponding to each of the masked positions in the speech characterizing result;

[0059] a prediction sub-unit configured to perform, for each of the characterizing results corresponding to the masked positions, prediction of a probability that the characterizing result belongs to each of the reference discretized labels, and determine a reference discretized label corresponding to a maximum probability as a predicted discretized label corresponding to the masked position.

[0060] In some example embodiments, the discretized label determination unit comprises:

[0061] an information entropy calculation unit configured to perform information entropy calculation on a target index sequence to obtain a current sequence information entropy, the target index sequence being composed of target indices corresponding to target codebook vectors matching the mapping vectors;

[0062] a codebook set adjustment unit configured to perform adjustment on the codebook set to obtain an adjusted codebook set, in a case where the current sequence information entropy exceeds a preset sequence information entropy range;

[0063] a repeated execution unit configured to perform the steps of searching for target codebook vectors matching the mapping vectors from the codebook set, performing information entropy calculation on a target index sequence to obtain a current sequence information entropy, with the adjusted codebook set as the codebook set, until the current sequence information entropy does not exceed the preset sequence information entropy range;

[0064] a determination sub-unit configured to perform determination of each target index in a target index sequence corresponding to a current sequence information entropy not exceeding the preset sequence information entropy range as a reference discretized label of a corresponding audio frame.

[0065] In some example embodiments, the preset sequence information entropy range comprises a preset sequence information entropy upper limit and a preset sequence information entropy lower limit; and the codebook set adjustment unit comprises:

[0066] a first codebook adjustment unit configured to perform, when the current sequence information entropy is less than the preset sequence information entropy lower limit, increase of a size of the codebook set, and take the codebook set after the size increase as the adjusted codebook set;

[0067] a second codebook adjustment unit configured to perform, when the current sequence information entropy is greater than the preset sequence information entropy upper limit, decrease of the size of the codebook set, and take the codebook set after the size decrease as the adjusted codebook set.

[0068] In some example embodiments, the random initialization unit comprises:

[0069] a first random initialization unit configured to perform random initialization of weights by uniform distribution to obtain the mapping matrix;

[0070] a second random initialization unit configured to perform random initialization of codebook vectors by standard normal distribution to obtain the codebook set.

[0071] In some example embodiments, the codebook set includes a plurality of sub-codebook sets; the discrete label determination unit, when searching for target codebook vectors matching the mapping vectors from the codebook set, is specifically configured to perform, for each audio frame, searching for a target sub-codebook vector most similar to the mapping vector of the audio frame from each sub-codebook set; and the target sub-codebook vector in each sub-codebook set constitutes the target codebook vector matching the mapping vector of the audio frame.

[0072] According to a third aspect of the embodiments of the present disclosure, an electronic device is provided, comprising:

[0073] a processor;

[0074] a memory for storing instructions executable by the processor;

[0075] wherein the processor is configured to execute the instructions to implement the voice representation model pre-training method of the first aspect.

[0076] According to a fourth aspect of the embodiments of the present disclosure, a computer-readable storage medium is provided, when instructions in the computer-readable storage medium are executed by a processor of an electronic device, the electronic device can execute the voice representation model pre-training method of the first aspect.

[0077] According to a fifth aspect of the embodiments of the present disclosure, a computer program product is provided, comprising a computer program, when the computer program is executed by a processor, the voice representation model pre-training method of the first aspect is implemented.

[0078] The embodiment of the disclosure generates a mapping matrix and a codebook set by a random initialization method, and then performs vector mapping processing on each audio frame in the sequence of audio frames corresponding to the sample speech based on the mapping matrix to obtain a mapping vector of each audio frame, searches for a target codebook vector matched with each audio frame from the codebook set based on the mapping vector of each audio frame, takes a target index corresponding to the target codebook vector matched with each audio frame as a reference discrete label of each audio frame, inputs the sequence of masked audio frames into a speech representation model to be trained for speech representation processing to obtain a speech representation result, predicts a discrete label for the representation result corresponding to each masking position in the speech representation result to obtain a predicted discrete label corresponding to each masking position, and then adjusts the model parameters of the speech representation model to be trained based on the difference between the predicted discrete label corresponding to each masking position and the reference discrete label of the original audio frame of the masking position until a preset training end condition is met to obtain a pre-trained speech representation model. In the above technical solution, the mapping matrix and the codebook set initialized randomly are fixed in the entire pre-training process, so that the same input audio frame in the entire pre-training process always corresponds to a uniquely determined discrete label, and the sample speech data does not need to be discretized in advance. The sample speech data can be prepared for efficient and stable speech pre-training, and the pre-training speed, convergence and stability of the speech representation model are improved.

[0079] It should be understood that the above general description and the following detailed description are only exemplary and explanatory, and cannot limit the disclosure. BRIEF DESCRIPTION OF DRAWINGS

[0080] The accompanying drawings incorporated in the specification and forming a part of it illustrate embodiments consistent with the present disclosure and together with the description serve to explain the principles of the present disclosure, and do not limit the present disclosure.

[0081] Figure 1 is an application environment schematic diagram of a speech representation model pre-training method according to an exemplary embodiment;

[0082] Figure 2 is a flowchart schematic diagram of a speech representation model pre-training method according to an exemplary embodiment;

[0083] Figure 3 is a flowchart schematic diagram of another speech representation model pre-training method according to an exemplary embodiment;

[0084] Figure 4 is a flowchart schematic diagram of another speech representation model pre-training method according to an exemplary embodiment;

[0085] Figure 5is a structural block diagram of a voice representation model pre-training device according to an exemplary embodiment;

[0086] Figure 6 is a structural block diagram of an electronic device according to an exemplary embodiment. DETAILED DESCRIPTION

[0087] In order for those skilled in the art to better understand the technical solutions of the present disclosure, the technical solutions in the embodiments of the present disclosure will be described clearly and completely below with reference to the drawings.

[0088] It should be noted that the terms "first", "second", and the like in the specification and claims of the present disclosure and the above-described drawings are used to distinguish similar objects, and do not necessarily have to describe a specific order or a chronological sequence. It should be understood that the data thus used can be interchanged under appropriate circumstances, so that the embodiments of the present disclosure described herein can be implemented in an order other than that illustrated or described herein. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present disclosure. Rather, they are merely examples of devices and methods consistent with some aspects of the present disclosure as detailed in the appended claims.

[0089] It should also be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data for display, analyzed data, etc.) involved in the present disclosure are all information and data authorized by the user or authorized by all parties.

[0090] Since voice is a continuous modal information, there is usually a lot of noise fluctuation, so when pre-training the voice representation model, it is usually necessary to discretize the voice. A reasonable discretization method can eliminate this data fluctuation to some extent, thereby helping the voice representation model to model the essential information of the voice data. For voice, the essential information is the limited phonetic unit sequence. In addition, after voice discretization, it can be equivalent to text information, so mature deep learning models and algorithms in the text field can be directly used.

[0091] One of the voice pre-training methods in the related art is to use a machine learning algorithm to train the voice discretization network and the voice representation model at the same time. In the training process, the parameters of the voice discretization network and the voice representation model are updated. Since the voice discretization network is not accurate at the beginning of training, the model needs a long period to learn good discretization mapping, resulting in slow training speed, difficulty in convergence, and poor stability.

[0092] Another voice pre-training manner in the related art is to perform clustering processing on a large amount of voice data, and then train a voice representation model by taking a cluster center serial number as a discretization label. The clustering processing usually requires hundreds of G of running memory and takes a long time. In addition, the stability of discretization by clustering of voice information is not enough, and more iterations of pre-training are usually required to obtain better voice representation results, resulting in slow training speed and poor stability.

[0093] In view of this, the embodiments of the present disclosure provide a voice representation model pre-training method, which pre-trains a voice representation model by simply, quickly and stably extracting voice discretization labels, thereby improving the pre-training speed and stability of the voice representation model. The voice representation model pre-training method of the embodiments of the present disclosure will be described in detail below with reference to the accompanying drawings.

[0094] Please refer to Figure 1 which shows an application environment schematic diagram according to an exemplary embodiment. The application environment can include a terminal 110 and a server 120, which can be connected through a wired network or a wireless network.

[0095] The terminal 110 can be a smartphone, a tablet computer, a notebook computer, a desktop computer, etc., but is not limited thereto. The terminal 110 can be installed with a client software such as an application program (Application, APP for short) providing a voice recognition function. The application program can be a standalone application program or a subprogram in an application program. A user of the terminal 110 can log in to the application program through pre-registered user information, which can include an account and a password. For example, the application program can include a social application program, a live broadcast application program, a video application program, etc.

[0096] The server 120 can be a server providing a background service such as voice recognition for the application program in the terminal 110. The server 120 can store a voice recognition model, which can be obtained by the server 120 based on a preset voice recognition task by model fine-tuning a pre-trained voice representation model. The pre-trained voice representation model can be obtained by the server 120 based on an unannotated sample voice by pre-training a to-be-trained voice representation model. The to-be-trained voice representation model can be an encoding network based on a Transformer structure.

[0097] In the embodiments of the present disclosure, the server 120 can be a stand-alone physical server, a server cluster or a distributed system composed of multiple physical servers, an edge computing node, a cloud server providing cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN (Content Delivery Network), and basic cloud computing services such as big data and artificial intelligence platforms.

[0098] Figure 2 is a flowchart of a voice representation model pre-training method according to an exemplary embodiment, as shown in Figure 2 The method comprises the following steps:

[0099] In step S201, a mapping matrix and a codebook set are generated by random initialization.

[0100] The codebook set represents the correspondence between the codebook vectors and the indexes.

[0101] Specifically, the mapping matrix is used to map the input into a vector matching the dimension of the codebook vector, and the size of the mapping matrix can be set based on actual needs. The codebook set includes multiple codebook vectors, each of which corresponds to a unique index. For example, the index can be represented by a serial number. The size of the codebook set can be represented by the number of codebook vectors in the codebook set, and the specific size can be set based on actual needs, for example, it can be set to include 100 codebook vectors, and the index can use serial numbers from 1 to 100.

[0102] It should be noted that the mapping matrix and the codebook set generated by random initialization in the embodiments of the present disclosure are fixed in the model parameter adjustment process of the voice representation model, that is, they are not learnable and updatable in this process.

[0103] For example, generating the mapping matrix and the codebook set by random initialization can include: initializing the weights by uniform distribution to obtain the mapping matrix; initializing the codebook vectors by standard normal distribution to obtain the codebook set.

[0104] In a specific implementation, the uniform distribution random initialization of the weights can be realized by the nn.init.xavier_uniform() function, which can calculate a reasonable initialization range according to the input and output dimensions, and then use the uniform distribution to randomly initialize the weights in this range, thereby obtaining the mapping matrix generated by random initialization.

[0105] The codebook vector randomly initialized by the standard normal distribution can be implemented by the F.normalize(torch.randn()) function, where the torch.randn() function can generate a random tensor subject to the standard normal distribution, and then the F.normalize() function is used to perform L2 norm normalization on the tensor. By using the random number of the standard normal distribution for initialization and L2 normalization, the codebook vector can be ensured to have a unit norm.

[0106] In the above embodiments, the weights are randomly initialized by the uniform distribution to obtain the randomly initialized mapping matrix, and the codebook vector is randomly initialized by the standard normal distribution to obtain the randomly initialized codebook set, so that the reference discrete label can be more quickly and stably determined based on the mapping matrix and the codebook set subsequently, which is beneficial to improve the training speed and stability of the language representation model.

[0107] In step S203, each audio frame in the sequence of audio frames corresponding to the sample speech is subjected to vector mapping processing based on the mapping matrix to obtain a mapping vector of each audio frame.

[0108] Specifically, after obtaining the randomly initialized mapping matrix and codebook set, the mapping matrix and the codebook set can be used to perform speech discretization processing on the sample speech to obtain the reference discrete label of each audio frame in the sequence of audio frames corresponding to the sample speech. The reference discrete label of each audio frame is a target index corresponding to a target codebook vector in the codebook set that matches the audio frame.

[0109] Specifically, for each audio frame in the sequence of audio frames corresponding to the sample speech, the mapping matrix generated by the random initialization method can be used for vector mapping processing to map the audio frame into a mapping vector.

[0110] The sequence of audio frames corresponding to the sample speech can be a plurality of audio frames obtained by performing frame processing and windowing processing on the sample speech, each audio frame having a specified time length. The specified time length can be set based on actual needs, for example, the specified time length can be selected in the short time range of 10-30 ms, and for example, 10 ms can be selected.

[0111] In specific implementations, the frame processing can be in a continuous segmentation manner or in an overlapping segmentation manner. The overlapping segmentation manner can make the frames smoothly transition between each other and maintain continuity. The overlapping part of the previous frame and the next frame is referred to as a frame shift. The ratio of the frame shift to the frame length can be 0-1 / 2. The windowing processing refers to function mapping of the audio signal after the frame processing by using a window function, so that the two adjacent audio data frames can smoothly transition, reduce the problem of signal discontinuity at the start and end of the data frame, make the global have higher continuity, avoid the Gibbs effect, and through the windowing processing, the audio signal originally without periodicity can also present part of the characteristics of the periodic function, which is beneficial to signal analysis and processing. For example, the window function can be any type of window function in a Hamming window, a rectangular window, a Hanning window, etc.

[0112] In some example implementations, the step S203 can include, in implementation, extracting a frequency domain feature of each audio frame in the audio frame sequence to obtain a frame feature sequence corresponding to the audio frame sequence; performing global normalization processing on the frame feature sequence to obtain a normalized frame feature sequence; and determining a product between the mapping matrix and each normalized frame feature in the normalized frame feature sequence to obtain a mapping vector of each audio frame in the audio frame sequence.

[0113] Specifically, the frequency domain feature of each audio frame can be a Mel-Frequency Cepstral Coefficient (MFCC) or a Mel filter bank (FBank) coefficient. For example, the frequency domain feature is an 80-dimensional FBank feature.

[0114] The global normalization processing is used to compress the space where the frame features are located, so as to reduce the difference between different frame features. In specific implementations, when performing the global normalization processing on the frame feature sequence, the mean and the standard deviation of the frame features in the frame feature sequence can be calculated first; then for each frame feature, the normalized frame feature corresponding to the frame feature can be obtained by subtracting the mean and dividing by the standard deviation, so that the normalized frame features corresponding to the frame features in the frame feature sequence constitute the normalized frame feature sequence.

[0115] Specifically, for each normalized frame feature, the product between the mapping matrix and the normalized frame feature is calculated, specifically, the dot product, and the product is taken as the mapping vector of the audio frame corresponding to the normalized frame feature.

[0116] The implementation manner extracts the frequency domain features of each audio frame to obtain a corresponding frame feature sequence, performs global standardization processing on the frame feature sequence to obtain a standardized frame feature sequence, and then maps the standardized frame feature sequence to a mapping vector through a randomly initialized mapping matrix, so that the speech vectors can be effectively concentrated, the utilization rate of the codebook after discretization mapping is high enough, the accuracy of the mapping vector of each audio frame is improved, and the accuracy of the subsequent discretization label is improved, thereby improving the training effect of the speech representation model.

[0117] In step S205, a target codebook vector matching each mapping vector is searched from the codebook set, and a target index corresponding to the target codebook vector matching each mapping vector is taken as a reference discretization label of an audio frame corresponding to each mapping vector.

[0118] Specifically, for the mapping vector of each audio frame, the similarity between the mapping vector and each codebook vector in the codebook set is calculated, the codebook vector corresponding to the maximum similarity is determined as the target codebook vector matching the mapping vector, and then based on the correspondence between the codebook vector and the index, the target index corresponding to the target codebook vector is obtained, and the target index is determined as the reference discretization label of the audio frame corresponding to the mapping vector. Then, the reference discretization labels of each audio frame in the audio frame sequence constitute a reference discretization label sequence.

[0119] In some exemplary implementation manners, in order to improve the accuracy of the discretization label and make the discretization have diversity, i.e., different speech signals with different meanings should be discretized into different labels, and then improve the effect of speech pre-training, the aforementioned randomly initialized codebook set can include a plurality of sub-codebook sets, and then in step S205, when searching for a target codebook vector matching each mapping vector from the codebook set, for each audio frame, a target sub-codebook vector most similar to the mapping vector of the audio frame can be searched from each sub-codebook set, and then the target sub-codebook vectors in each sub-codebook set constitute the target codebook vector matching the mapping vector of the audio frame. Then, the target index corresponding to the target codebook vector includes the target sub-index corresponding to each target sub-codebook vector. By using the combination of a plurality of sub-codebook sets, the number of required codebook vectors can be significantly reduced under the premise of ensuring the diversity of discretization, and the pre-training speed is improved.

[0120] In step S207, the masked audio frame sequence is input into the speech representation model to be trained for speech representation processing, and a speech representation result is obtained.

[0121] The sequence of masked audio frames is obtained by performing a masking process on a plurality of positions in the sequence of audio frames. The masking process can specifically be replacing the audio frames of the masked positions in the sequence of audio frames with preset masking identifiers, for example, the preset masking identifier can be [MASK]. In a specific implementation, a plurality of positions can be randomly selected from the sequence of audio frames for the masking process. The plurality of masked positions can account for 15% of the total number of positions in the sequence of audio frames.

[0122] In actual applications, the frequency domain features of each audio frame in the sequence of audio frames can be extracted, for example, FBank features, to obtain a sequence of frame features corresponding to the sequence of audio frames. Then, a plurality of positions in the sequence of frame features are masked to obtain a sequence of masked frame features. The sequence of masked frame features is input into the to-be-trained speech representation model for speech representation processing to obtain a speech representation result.

[0123] The speech representation result includes representation results corresponding to each position of the sequence of audio frames, specifically including representation results corresponding to each masked position and representation results of each unmasked position.

[0124] In a specific implementation, the to-be-trained speech representation model can be an encoder based on a Transformer structure.

[0125] In step S209, the representation results corresponding to each masked position in the speech representation result are respectively predicted as discrete labels to obtain predicted discrete labels corresponding to each masked position.

[0126] For example, the representation results corresponding to each masked position in the speech representation result can be obtained. For each representation result corresponding to the masked position, the probability of the representation result belonging to each reference discrete label in a sequence of reference discrete labels is predicted. The reference discrete label corresponding to the maximum probability is determined as the predicted discrete label corresponding to the masked position. Thus, each masked position is accurately classified and predicted in combination with the sequence of reference discrete labels. Since the reference discrete labels are obtained based on a randomly initialized fixed mapping matrix and a codebook set, the stability and speed of the classification and prediction are improved, which is conducive to improving the training effect of the language representation model.

[0127] In step S211, based on the difference between the predicted discrete labels corresponding to each masked position and the reference discrete labels of the audio frames of the masked positions, the model parameters of the to-be-trained speech representation model are adjusted until a preset training end condition is met, to obtain a pre-trained speech representation model.

[0128] Specifically, based on a difference between the predicted discrete label corresponding to each occluded position and a reference discrete label of an audio frame of the occluded position, a loss value is calculated using a preset loss function, and then the model parameters of the to-be-trained speech representation model are adjusted in reverse based on the loss value, and the to-be-trained speech representation model is iteratively trained based on the adjusted to-be-trained speech representation model until a preset training end condition is met to end the training, and the model corresponding to the model parameters at the end of the training is determined as the pre-trained speech representation model. The preset training end condition can be that the number of iterations reaches a preset number of iterations, or the loss value reaches a preset loss threshold, or the difference between the loss values of adjacent two iterations reaches a preset difference threshold. Exemplarily, the preset loss function can use a classification loss function, such as a cross-entropy loss function.

[0129] In a specific implementation, a classifier based on classification of the reference discrete label sequence can be connected after the to-be-trained speech representation model, and the classifier is used to calculate the loss value.

[0130] It should be noted that in the above step S211, only the model parameters of the to-be-trained speech representation model are adjusted, and the foregoing mapping matrix and codebook set are not updated.

[0131] The technical scheme of the embodiments of the present disclosure adopts a random initialization discretization method, and the discretization is determined throughout the pre-training process, that is, the same input always corresponds to a uniquely determined discrete label, and at the same time, there is no need to discretize the data in advance, and the original speech data can be prepared for efficient and stable speech pre-training, thereby improving the training speed and stability of the speech representation model.

[0132] In some exemplary embodiments, in order to reduce the computational amount of the pre-training process and further improve the training efficiency, when the frame feature sequence is globally standardized to obtain a standardized frame feature sequence, the following can be included: concatenating every continuous preset number of frame features in the frame feature sequence to obtain a concatenated frame feature sequence; and globally standardizing the concatenated frame feature sequence to obtain the standardized frame feature sequence.

[0133] The preset number can be set based on actual experience, for example, the preset number can be 4, and then every continuous 4 frame features in the frame feature sequence can be concatenated into a concatenated frame feature, thereby obtaining the concatenated frame feature sequence, and then the mean and standard deviation of the concatenated frame features in the concatenated frame feature sequence are calculated, and each concatenated frame feature is subtracted by the mean and then divided by the standard deviation to obtain the standardized frame feature sequence.

[0134] The above embodiment reduces the calculation amount of discretization, improves the discretization speed, and further facilitates improving the training efficiency of the speech representation model by splicing every continuous preset number of frame features in the frame feature sequence to obtain a spliced frame feature sequence, and performing global standardization on the spliced frame feature sequence and then performing vector mapping processing based on a mapping matrix.

[0135] Therefore, in order to utilize the reference discretization label sequence obtained based on the frame feature splicing, the step S207 can include the following steps when implemented:

[0136] performing preset number of downsampling processes on the frame feature sequence to obtain a downsampled frame feature sequence;

[0137] replacing a plurality of downsampled frame features in the downsampled frame feature sequence with preset masking marks to obtain a masked downsampled frame feature sequence;

[0138] inputting the masked downsampled frame feature sequence into the speech representation model to be trained to perform speech representation processing, and obtaining a speech representation result.

[0139] In specific implementations, the downsampling process can be implemented by convolution processing of a convolution layer, and the preset number of downsampling processes can be implemented by convolution processing of a preset number of cascaded convolution layers in sequence, and each convolution layer is implemented to perform one downsampling process. For example, if the time length of a frame feature is 10 ms, and every continuous 4 frame features are spliced into one spliced frame feature with a time length of 40 ms, two cascaded convolution layers can be used to perform convolution processing on the frame feature sequence in sequence, so as to obtain a downsampled frame feature sequence composed of downsampled frame features with a time length of 40 ms.

[0140] replacing a plurality of downsampled frame features in the downsampled frame feature sequence with preset masking marks, such as [MASK], to obtain a masked downsampled frame feature sequence, and then inputting the masked downsampled frame feature sequence into the speech representation model to be trained to perform speech representation processing and obtain a speech representation result.

[0141] The above embodiment reduces the calculation amount of discretization, improves the discretization speed, and further facilitates improving the training efficiency of the speech representation model by splicing every continuous preset number of frame features in the frame feature sequence to obtain a spliced frame feature sequence, and performing global standardization on the spliced frame feature sequence and then performing vector mapping processing based on a mapping matrix.

[0142] In some exemplary embodiments, in order to ensure that the reference discretization labels obtained by voice discretization have sufficient diversity, i.e. different voice signals with different meanings should be discretized into different labels, so as to improve the accuracy of the pre-trained voice representation model, as shown in Figure 3 The foregoing step S205, when implementing the target index corresponding to the target codebook vector matched with each mapping vector as the reference discretization label of the audio frame corresponding to each mapping vector, can include:

[0143] In step S301, information entropy calculation is performed on the target index sequence to obtain the current sequence information entropy.

[0144] The target index sequence is composed of target indexes corresponding to target codebook vectors matched with each mapping vector. The current sequence information entropy can represent the diversity degree of the current target index sequence. Generally, the greater the sequence information entropy, the higher the diversity degree; on the contrary, the smaller the sequence information entropy, the lower the diversity degree.

[0145] In a specific implementation, the information entropy calculation on the target index sequence can be implemented by using the following formula (1):

[0146]

[0147] Wherein, H(X) represents the information entropy of the target index sequence X={x 1, x 2, ……,x n}; p(x i ) represents the occurrence probability of the target index x i in the target index sequence; n represents the sequence length of the target index sequence X. Specifically, p(x i ) can be calculated by using the following formula (2):

[0148]

[0149] Wherein, f(x i ) represents the frequency of the target index x i in the target index sequence X, which can be calculated by using the following formula (3):

[0150] f(x i )=N xi / n (3)

[0151] Wherein, N xi represents the number of times of the target index x i in the target index sequence X.

[0152] In step S303, when the current sequence information entropy exceeds the preset sequence information entropy range, the codebook set is adjusted to obtain an adjusted codebook set.

[0153] The preset sequence information entropy range can be set based on actual experience. The adjustment of the codebook set can be re-generating the codebook set by random initialization.

[0154] Specifically, a preset sequence information entropy reference value H base can be set based on actual experience, and then an acceptable variation range Δ relative to the preset sequence information entropy reference value H base is set, and then the preset sequence information entropy range can be obtained based on H base ± Δ. Specifically, the preset sequence information entropy range can be represented as (H base - Δ, H base + Δ), where H base - Δ is the lower limit of the preset sequence information entropy, and H base + Δ is the upper limit of the preset sequence information entropy. For example, the preset sequence information entropy reference value H base can be set to 8.43.

[0155] Based on this, continuing to refer to Figure 3 , the above-mentioned adjustment of the codebook set to obtain an adjusted codebook set can include:

[0156] In step S3031, when the current sequence information entropy is less than the lower limit of the preset sequence information entropy, the size of the codebook set is increased, and the codebook set after the size is increased is taken as the adjusted codebook set.

[0157] In step S3033, when the current sequence information entropy is greater than the upper limit of the preset sequence information entropy, the size of the codebook set is reduced, and the codebook set after the size is reduced is taken as the adjusted codebook set.

[0158] Specifically, when the current sequence information entropy is less than the lower limit of the preset sequence information entropy H base - Δ, it indicates that the current discretization diversity is not enough, the diversity of the target index sequence is not enough, and the speech pre-training can cause mode collapse, at which time the size of the codebook set can be increased to expand the codebook vector space, so that the codebook set after the size is increased includes more codebook vectors.

[0159] When the current sequence information entropy is greater than the upper limit of the preset sequence information entropy H base + Δ, it indicates that the current discretization is too dispersed, the diversity of the target index sequence is too high, and it will cause the speech pre-training to be difficult to converge, at which time the size of the codebook set can be reduced to reduce the codebook vector space, so that the codebook set after the size is reduced includes fewer codebook vectors.

[0160] In the embodiments of the present disclosure, the adjustment of the codebook set can be reinitializing the codebook vectors randomly.

[0161] The above-mentioned embodiments increase the size of the codebook set when the current sequence information entropy is less than the preset lower limit of the sequence information entropy, and decrease the size of the codebook set when the current sequence information entropy is greater than the preset upper limit of the sequence information entropy, so as to control the size of the codebook set in a proper range, thereby facilitating the effective regulation of the discretization diversity.

[0162] In step S305, the adjusted codebook set is taken as the codebook set, and the steps of searching the target codebook vector matching each mapping vector from the codebook set in the aforementioned step S205, and steps S301 to S303 are performed until the current sequence information entropy does not exceed the preset sequence information entropy range, and the following step S307 is entered.

[0163] In step S307, each target index in the target index sequence corresponding to the current sequence information entropy not exceeding the preset sequence information entropy range is taken as the reference discretization label of the corresponding audio frame.

[0164] The above-mentioned embodiments calculate the information entropy of the target index sequence obtained after discretization to obtain the current sequence information entropy, and then regulate the discretization process based on the current sequence information entropy, so as to ensure that the final obtained reference discretization label sequence has sufficient diversity, thereby ensuring the accuracy of the pre-trained voice representation model, and improving the convergence speed, pre-training efficiency and stability of the pre-training.

[0165] In order to facilitate the understanding of the technical solutions of the embodiments of the present disclosure, the voice representation model pre-training method of the embodiments of the present disclosure will be exemplarily described below in combination with Figure 4

[0166] As Figure 4 ​As shown, first, a randomly initialized, fixed full connection mapping matrix and codebook set throughout the pre-training process are created. Second, Fbank features with a time length of 10 ms are extracted from the input sample speech, obtaining a frame feature sequence of 80 dimensions per frame. Then, the frame feature sequence is globally standardized to compress the space distribution of the frame features. After that, each frame feature after global standardization is mapped to a vector through a randomly initialized mapping matrix, and the similarity is calculated in the codebook set to obtain the serial number index of the most similar target codebook vector of the vector as the corresponding reference discrete label. Finally, the frame feature sequence is randomly masked and sent to the speech representation model (Transformer) to be trained for speech representation, obtaining the output speech representation result, using the output speech representation result to classify and predict (or called discrete label prediction) the input masked part, and the classified label is the corresponding reference discrete label, and then the difference between the predicted discrete label corresponding to each input masked part and the reference discrete label corresponding to the corresponding input masked part is calculated to calculate the classification loss, and the model parameters of the speech representation model (Transformer) to be trained are adjusted based on the classification loss and iteratively trained to obtain the pre-trained speech representation model. The above technical solution does not need to train the discrete module, the pre-training process is efficient, easy to converge and stable, without the cumbersome and time-consuming discrete label extraction process, the discrete quality is good enough, and experiments have verified that it can meet the related downstream tasks of speech, songs, speakers, etc.

[0167] Figure 5 is a structural block diagram of a speech representation model pre-training device according to an exemplary embodiment. Referring to Figure 5 The speech representation model pre-training device 500 includes:

[0168] The random initialization unit 510 is configured to generate a mapping matrix and a codebook set by a random initialization method; the codebook set represents the corresponding relationship between the codebook vector and the index;

[0169] The audio frame mapping unit 520 is configured to perform vector mapping processing on each audio frame in the audio frame sequence corresponding to the sample speech based on the mapping matrix, to obtain the mapping vector of each audio frame;

[0170] The discrete label determination unit 530 is configured to search for a target codebook vector matching each mapping vector from the codebook set, and take the target index corresponding to the target codebook vector matching each mapping vector as the reference discrete label of the audio frame corresponding to each mapping vector;

[0171] The speech representation unit 540 is configured to perform speech representation processing on the masked audio frame sequence by inputting the masked audio frame sequence into a speech representation model to be trained, to obtain a speech representation result; the masked audio frame sequence is obtained based on masking processing on a plurality of positions in the audio frame sequence;

[0172] The label prediction unit 550 is configured to perform discrete label prediction on the representation result corresponding to each masked position in the speech representation result, to obtain a predicted discrete label corresponding to each masked position.

[0173] The model parameter adjustment unit 560 is configured to perform model parameter adjustment of the speech representation model to be trained based on a difference between the predicted discrete label corresponding to each masked position and a reference discrete label of the audio frame of the masked position, until a preset training end condition is met, to obtain a pre-trained speech representation model.

[0174] In some example embodiments, the audio frame mapping unit 520 includes:

[0175] The frequency domain feature extraction unit is configured to perform frequency domain feature extraction on each audio frame in the audio frame sequence, to obtain a frame feature sequence corresponding to the audio frame sequence.

[0176] The global normalization unit is configured to perform global normalization processing on the frame feature sequence, to obtain a normalized frame feature sequence.

[0177] The mapping unit is configured to perform determination of a product of the mapping matrix and each normalized frame feature in the normalized frame feature sequence, to obtain a mapping vector of each audio frame in the audio frame sequence.

[0178] In some example embodiments, the global normalization unit includes:

[0179] The frame feature concatenation unit is configured to perform concatenation of every continuous preset number of frame features in the frame feature sequence, to obtain a concatenated frame feature sequence.

[0180] The global normalization sub-unit is configured to perform global normalization processing on the concatenated frame feature sequence, to obtain a normalized frame feature sequence.

[0181] In some example embodiments, the speech representation unit 540 includes:

[0182] The down-sampling unit is configured to perform down-sampling processing on the frame feature sequence for a preset number of times, to obtain a down-sampled frame feature sequence; a time length of each down-sampled frame feature in the down-sampled frame feature sequence is the same as a time length of a concatenated frame feature at a corresponding position in the concatenated frame feature sequence.

[0183] a replacing unit configured to perform replacing a plurality of down-sampling frame features in the down-sampling frame feature sequence with preset masking marks to obtain a masked down-sampling frame feature sequence;

[0184] a speech representation sub-unit configured to perform inputting the masked down-sampling frame feature sequence into a speech representation model to be trained for speech representation processing to obtain a speech representation result.

[0185] In some example embodiments, the label prediction unit 550 comprises:

[0186] a representation result obtaining unit configured to perform obtaining a representation result corresponding to each masked position in the speech representation result;

[0187] a prediction sub-unit configured to perform, for each representation result corresponding to the masked position, predicting a probability that the representation result belongs to each reference discretized label, and determining a reference discretized label corresponding to the maximum probability as a predicted discretized label corresponding to the masked position.

[0188] In some example embodiments, the discretized label determination unit 530 comprises:

[0189] an information entropy calculation unit configured to perform information entropy calculation on a target index sequence to obtain a current sequence information entropy; the target index sequence is composed of target indexes corresponding to target codebook vectors matched with the mapping vectors;

[0190] a codebook set adjustment unit configured to perform, in a case where the current sequence information entropy exceeds a preset sequence information entropy range, adjusting the codebook set to obtain an adjusted codebook set;

[0191] a repeated execution unit configured to perform, taking the adjusted codebook set as the codebook set, the steps of searching, from the codebook set, a target codebook vector matched with each mapping vector, performing information entropy calculation on a target index sequence to obtain a current sequence information entropy, until the current sequence information entropy does not exceed the preset sequence information entropy range;

[0192] a determination sub-unit configured to perform taking each target index in a target index sequence corresponding to a current sequence information entropy that does not exceed the preset sequence information entropy range as a reference discretized label of a corresponding audio frame.

[0193] In some example embodiments, the preset sequence information entropy range comprises a preset sequence information entropy upper limit and a preset sequence information entropy lower limit; and the codebook set adjustment unit comprises:

[0194] The first codebook adjusting unit is configured to increase the size of the codebook set when the current sequence information entropy is less than the preset sequence information entropy lower limit, and take the codebook set after the size is increased as the adjusted codebook set.

[0195] The second codebook adjusting unit is configured to decrease the size of the codebook set when the current sequence information entropy is greater than the preset sequence information entropy upper limit, and take the codebook set after the size is decreased as the adjusted codebook set.

[0196] In some example embodiments, the random initialization unit 510 includes:

[0197] The first random initialization unit is configured to initialize the weight by uniform distribution to obtain the mapping matrix.

[0198] The second random initialization unit is configured to initialize the codebook vector by standard normal distribution to obtain the codebook set.

[0199] In some example embodiments, the codebook set includes a plurality of sub-codebook sets; and the discretization label determining unit 530, when searching for the target codebook vector matching each mapping vector from the codebook set, is specifically configured to search for, for each audio frame, a target sub-codebook vector most similar to the mapping vector of the audio frame from each sub-codebook set; and the target sub-codebook vector in each sub-codebook set constitutes the target codebook vector matching the mapping vector of the audio frame.

[0200] As to the apparatus in the above-described embodiments, the specific manners in which each module performs operations have been described in detail in the embodiments of the method, and thus will not be described in detail here.

[0201] In an example embodiment, an electronic device is also provided, including a processor; a memory for storing processor-executable instructions; wherein the processor is configured to implement the voice representation model pre-training method provided in any of the above-described embodiments when executing the instructions stored on the memory.

[0202] The electronic device can be a terminal, a server or a similar computing device. Take the electronic device as a server for example, Figure 6 is a block diagram of an electronic device running a voice representation model pre-training method according to an example embodiment, as shown in Figure 6As shown, the server 600 can vary significantly due to different configurations or performance. It may include one or more Central Processing Units (CPUs) 1010 (CPUs 610 may include, but are not limited to, microprocessors such as MCUs or programmable logic devices such as FPGAs), a memory 630 for storing data, and one or more storage media 620 (e.g., one or more mass storage devices) for storing application programs 623 or data 622. The memory 630 and storage media 620 may be temporary or persistent storage. The program stored in the storage media 620 may include one or more modules, each module may include a series of instruction operations on the server. Furthermore, the CPU 610 may be configured to communicate with the storage media 620 and execute the series of instruction operations stored in the storage media 620 on the server 600. Server 600 may also include one or more power supplies 660, one or more wired or wireless network interfaces 650, one or more input / output interfaces 640, and / or one or more operating systems 621, such as Windows Server™, Mac OS X™, Unix™, Linux™, FreeBSD™, etc.

[0203] The input / output interface 640 can be used to receive or send data via a network. Specific examples of the network described above may include a wireless network provided by the communication provider of server 600. In one example, the input / output interface 640 includes a network interface controller (NIC), which can connect to other network devices via a base station to communicate with the Internet. In another example, the input / output interface 640 may be a radio frequency (RF) module for wireless communication with the Internet.

[0204] Those skilled in the art will understand that Figure 6 The structure shown is for illustrative purposes only and does not limit the structure of the aforementioned electronic device. For example, server 600 may also include... Figure 6 The more or fewer components shown, or having the same Figure 6 The different configurations shown.

[0205] In one exemplary embodiment, a computer-readable storage medium is also provided, which, when the instructions in the computer-readable storage medium are executed by a processor of an electronic device, enables the electronic device to perform the speech representation model pre-training method provided in any of the above embodiments.

[0206] In an exemplary embodiment, a computer program product is also provided, comprising a computer program which, when executed by a processor, implements the voice representation model pre-training method provided in any of the above embodiments.

[0207] Other embodiments of the disclosure will be apparent to those skilled in the art from consideration of the specification and practice of the features disclosed herein. It is intended that the specification and examples be considered as exemplary only, with the true scope and spirit of the disclosure being indicated by the following claims.

[0208] It will be understood that the present disclosure is not limited to the precise structures hereinbefore described and illustrated in the drawings, and that various modifications and changes can be made without departing from the scope thereof. The scope of the present disclosure is indicated by the appended claims.

Claims

1. A method for pre-training a speech representation model, characterized in that, include: The mapping matrix and codebook set are generated through random initialization. The codebook set represents the correspondence between codebook vectors and indices; Based on the mapping matrix, vector mapping processing is performed on each audio frame in the audio frame sequence corresponding to the sample speech to obtain the mapping vector of each audio frame. Search for a target codebook vector that matches each of the mapping vectors from the codebook set, and use the target index corresponding to the target codebook vector that matches each of the mapping vectors as the reference discretization label of the audio frame corresponding to each of the mapping vectors; The masked audio frame sequence is input into the speech representation model to be trained for speech representation processing to obtain the speech representation result; the masked audio frame sequence is obtained by masking multiple positions in the audio frame sequence; For the speech representation results, the representation results corresponding to each occluded position are discretized and labeled to obtain the predicted discretized label corresponding to each occluded position. Based on the difference between the predicted discretized label corresponding to each of the occluded positions and the reference discretized label of the audio frame at the occluded position, the model parameters of the speech representation model to be trained are adjusted until the preset training termination condition is met, thereby obtaining the pre-trained speech representation model.

2. The method according to claim 1, characterized in that, The step of performing vector mapping processing on each audio frame in the audio frame sequence corresponding to the sample speech based on the mapping matrix to obtain the mapping vector of each audio frame includes: Extract the frequency domain features of each audio frame in the audio frame sequence to obtain the frame feature sequence corresponding to the audio frame sequence; The frame feature sequence is subjected to global standardization to obtain a standardized frame feature sequence; The mapping vector of each audio frame in the audio frame sequence is obtained by determining the product between the mapping matrix and each standardized frame feature in the standardized frame feature sequence.

3. The method according to claim 2, characterized in that, The global standardization process for the frame feature sequence to obtain the standardized frame feature sequence includes: The frame feature sequence is spliced ​​together by concatenating a predetermined number of consecutive frame features to obtain a spliced ​​frame feature sequence. The spliced ​​frame feature sequence is subjected to global standardization to obtain a standardized frame feature sequence.

4. The method according to claim 3, characterized in that, The step of inputting the masked audio frame sequence into the speech representation model to be trained for speech representation processing, and obtaining the speech representation result includes: The frame feature sequence is downsampled a preset number of times to obtain a downsampled frame feature sequence; the time length of each downsampled frame feature in the downsampled frame feature sequence is the same as the time length of the corresponding spliced ​​frame feature in the spliced ​​frame feature sequence. Replace multiple downsampled frame features in the downsampled frame feature sequence with preset masking identifiers to obtain a masked downsampled frame feature sequence; The masked downsampled frame feature sequence is input into the speech representation model to be trained for speech representation processing to obtain the speech representation result.

5. The method according to claim 1, characterized in that, The speech representation results are then subjected to discretized label prediction for each occluded position, resulting in the following predicted discretized labels for each occluded position: Obtain the representation results corresponding to each occluded position in the speech representation results; For each representation result corresponding to the occluded position, predict the probability that the representation result belongs to each of the reference discretization labels, and determine the reference discretization label corresponding to the highest probability as the predicted discretization label corresponding to the occluded position.

6. The method according to claim 1, characterized in that, The step of using the target index corresponding to the target codebook vector that matches each of the mapping vectors as the reference discretization label of the audio frame corresponding to each of the mapping vectors includes: The information entropy of the target index sequence is calculated to obtain the information entropy of the current sequence; the target index sequence is composed of the target indices corresponding to the target codebook vectors that match each of the mapping vectors. If the current sequence information entropy exceeds a preset sequence information entropy range, the codebook set is adjusted to obtain an adjusted codebook set. The adjusted codebook set is used as the codebook set. The steps of searching for target codebook vectors that match each of the mapping vectors from the codebook set, calculating the information entropy of the target index sequence, and obtaining the information entropy of the current sequence are performed until the information entropy of the current sequence does not exceed the preset information entropy range. Each target index in the target index sequence corresponding to the current sequence information entropy, which does not exceed the preset sequence information entropy range, is used as a reference discretization tag for the corresponding audio frame.

7. The method according to claim 6, characterized in that, The preset sequence information entropy range includes a preset sequence information entropy upper limit and a preset sequence information entropy lower limit; the adjustment of the codebook set to obtain the adjusted codebook set includes: When the current sequence information entropy is less than the preset sequence information entropy lower limit, the size of the codebook set is increased, and the codebook set with the increased size is used as the adjusted codebook set; When the current sequence information entropy is greater than the preset sequence information entropy upper limit, the size of the codebook set is reduced, and the codebook set with the reduced size is used as the adjusted codebook set.

8. The method according to claim 1, characterized in that, The generation of the mapping matrix and codebook set through random initialization includes: The mapping matrix is ​​obtained by randomly initializing the weights using a uniform distribution; The codebook set is obtained by randomly initializing the codebook vector using a standard normal distribution.

9. The method according to claim 1, characterized in that, The codebook set includes multiple sub-codebook sets; the step of searching for a target codebook vector that matches each of the mapping vectors from the codebook set includes: For each audio frame, a target subcodebook vector that is most similar to the mapping vector of the audio frame is searched from each of the subcodebook sets; wherein, the target subcodebook vectors in each of the subcodebook sets constitute the target codebook vector that matches the mapping vector of the audio frame.

10. A pre-training device for a speech representation model, characterized in that, include: The random initialization unit is configured to generate the mapping matrix and codebook set through random initialization. The codebook set represents the correspondence between codebook vectors and indices; The audio frame mapping unit is configured to perform vector mapping processing on each audio frame in the audio frame sequence corresponding to the sample speech based on the mapping matrix, so as to obtain the mapping vector of each audio frame. The discretization tag determination unit is configured to perform a search from the codebook set for a target codebook vector that matches each of the mapping vectors, and to use the target index corresponding to the target codebook vector that matches each of the mapping vectors as a reference discretization tag for the audio frame corresponding to each of the mapping vectors. The speech representation unit is configured to perform speech representation processing by inputting the masked audio frame sequence into the speech representation model to be trained, and to obtain a speech representation result; the masked audio frame sequence is obtained by masking multiple positions in the audio frame sequence. The label prediction unit is configured to perform discretized label prediction on the representation results corresponding to each occluded position in the speech representation results, so as to obtain the predicted discretized label corresponding to each occluded position. The model parameter adjustment unit is configured to perform adjustments based on the difference between the predicted discretized label corresponding to each occluded position and the reference discretized label of the audio frame at the occluded position, until the preset training termination condition is met, thereby obtaining a pre-trained speech representation model.

11. An electronic device, characterized in that, include: processor; Memory used to store the processor's executable instructions; The processor is configured to execute the instructions to implement the speech representation model pre-training method as described in any one of claims 1 to 9.

12. A computer-readable storage medium, characterized in that, When the instructions in the computer-readable storage medium are executed by the processor of the electronic device, the electronic device is enabled to perform the speech representation model pre-training method as described in any one of claims 1 to 9.

13. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by the processor, it implements the speech representation model pre-training method as described in any one of claims 1 to 9.

Images