Speaker embedding extraction method and device, equipment, storage medium and product

Abstract

The invention discloses a speaker embedding extraction method and device, equipment, a storage medium and a product. The method comprises the following steps: acquiring a speaker embedding sequence of a to-be-recognized first voice segment; the speaker embedding sequence is generated based on speaker embedding of each voice segment intercepted by the first voice segment according to a duration increasing mode; and inputting the speaker embedding sequence into a pre-trained prediction model, predicting to obtain speaker embedding of the second voice segment, and taking the predicted speaker embedding as speaker embedding of the first voice segment. Since the speaker embedding sequence comprises the incidence relation of speaker embedding in continuous time change, the prediction model can predict the input speaker embedding sequence based on learned priori knowledge to obtain speaker embedding of a voice segment with longer duration, the accuracy of speaker embedding extraction can be improved, and the accuracy of speaker embedding extraction can be improved. And the application effects of tasks such as subsequent speaker logs and voiceprint verification can be improved.

Description

Technical Field

[0001] The present application relates to the field of speech processing technology, and in particular to a speaker embedding extraction method, device, equipment, storage medium and product. Background Art

[0002] In speech processing technology, in multi-speaker scenarios, such as meeting records, court trials, and telephone conferences, in order to automatically identify and separate the voices of different speakers for subsequent speech analysis and processing, it is necessary to introduce speaker diarization (also known as voiceprint segmentation and clustering) technology. Speaker diarization solves the problem of "who spoke and when". In simple terms, it analyzes the speech of multiple speakers and distinguishes who spoke and when.

[0003] Speaker logging technology usually involves multiple steps in analyzing and processing speech. First, voice activity detection is performed to remove non-speech parts, such as background sounds, noise, and silence. Then, the entire speech is divided into multiple speech segments. Next, a speaker embedding extractor is used to extract the speaker embedding (Speaker Embedding) of each speech segment. Finally, these speaker embeddings are used for scoring and clustering to classify these speech segments into corresponding speakers.

[0004] The so-called speaker embedding extractor is a deep learning model, whose direct input is usually an acoustic feature converted from speech, and its output is a vector, called speaker embedding. An excellent speaker embedding extractor needs to meet the following requirements: the speaker embeddings extracted from multiple segments of speech of the same person are clustered together in high-dimensional space, and there is a certain distance between the speaker embeddings of different people in high-dimensional space. Only in this way can different speakers be distinguished well. Therefore, the quality of speaker embedding determines the results of subsequent scoring and clustering. However, in order to ensure that the segmented speech segment contains only one person's speech, the segmented speech segment is often relatively short, usually 1 to 2 seconds. The speaker embedding extraction model in the related art can obtain enough information in the case of long speech so that the extracted speaker embedding is of high quality. However, the quality of speaker embedding extracted from such short speech is low, which affects the subsequent scoring and clustering, resulting in the final result being not good enough. Summary of the invention

[0005] In view of this, embodiments of the present application provide a speaker embedding extraction method, apparatus, device, storage medium and product, aiming to improve the extraction quality of speaker embedding.

[0006] The technical solution of the embodiment of the present application is implemented as follows:

[0007] In a first aspect, an embodiment of the present application provides a speaker embedding extraction method, comprising:

[0008] Obtaining a speaker embedding sequence of a first speech segment to be recognized; wherein the speaker embedding sequence is generated based on speaker embeddings of each speech segment intercepted by the first speech segment in increasing time length;

[0009] The speaker embedding sequence is input into a pre-trained prediction model to predict the speaker embedding of a second speech segment, and the predicted speaker embedding is used as the speaker embedding of the first speech segment; wherein the second speech segment is a speech segment corresponding to a set duration added continuously in time to the first speech segment.

[0010] In the above solution, the step of obtaining a speaker embedding sequence of a first speech segment to be recognized includes:

[0011] Incrementally segmenting the acoustic features of the first speech segment to be recognized based on the initial truncation duration and the incremental duration to obtain the acoustic features of each speech segment after truncation; wherein the next speech segment to be truncate is a speech segment with the incremental duration added to the previous speech segment;

[0012] Inputting the acoustic features of each speech segment after being cut into a pre-trained speaker embedding extractor to obtain the speaker embedding of each speech segment after being cut; and inputting the acoustic features of the speech segment of each incremental duration into the speaker embedding extractor to obtain the speaker embedding of the speech segment of each incremental duration;

[0013] The speaker embeddings of the intercepted speech segments are sequentially concatenated in pairs with the speaker embeddings of the speech segments of the corresponding incremental durations to obtain a speaker embedding sequence of the first speech segment.

[0014] In the above scheme, the prediction model adopts a sequence-to-sequence architecture, including an encoder and a decoder, wherein the encoder is used to convert the speaker embedding sequence into a first output feature and a second output feature, and the decoder is used to obtain a predicted speaker embedding based on the first output feature, the second output feature and the speaker embedding of the first speech segment at an initial time step, and to obtain a new predicted speaker embedding based on the first output feature, the hidden layer output of the previous time step and the speaker embedding predicted at the previous time step at the next time step, until the speaker embedding of the second speech segment is predicted.

[0015] In the above scheme, the encoder includes a bidirectional gated recurrent unit, the speaker embedding sequence serves as the input of the bidirectional gated recurrent unit, the encoder obtains the first output feature based on the hidden layer output of the bidirectional gated recurrent unit, and obtains the second output feature based on the general output of the bidirectional gated recurrent unit.

[0016] In the above scheme, the method further includes:

[0017] A training sample set is obtained, and a speech segment is intercepted in an increasing manner in duration for each speech sample in the training sample set, and an acoustic feature set of the intercepted speech segment is constructed; wherein the acoustic feature set includes: a first set of acoustic features representing an existing part and a second set of acoustic features representing a part to be predicted;

[0018] The prediction model is trained based on the acoustic feature set until a trained prediction model is obtained.

[0019] In the above solution, the training of the prediction model based on the acoustic feature set includes:

[0020] Inputting the acoustic feature set into a pre-trained speaker embedding extractor to obtain speaker embeddings of acoustic features of each existing part in the first set and speaker embeddings of acoustic features of each to-be-predicted part in the second set;

[0021] Extracting acoustic features of incremental parts obtained based on two adjacent interceptions in the first set, and inputting the acoustic features of the incremental parts into the speaker embedding extractor to obtain speaker embeddings of the acoustic features of each incremental part;

[0022] For the speech sample, the speaker embedding of the acoustic features of the existing part and the speaker embedding of the acoustic features of the corresponding incremental part are sequentially concatenated in pairs to obtain a speaker embedding sequence of each speech sample;

[0023] Based on the prediction model and the speaker embedding sequence of each speech sample, predict the predicted speaker embedding of the to-be-predicted part of each speech sample, and obtain the loss value based on the predicted speaker embedding and the speaker embedding of the acoustic features of each to-be-predicted part obtained by the speaker embedding extractor;

[0024] Model parameters of the prediction model are updated based on the loss value.

[0025] In a second aspect, an embodiment of the present application provides a speaker embedding extraction device, comprising:

[0026] An acquisition module, configured to acquire a speaker embedding sequence of a first speech segment to be recognized; wherein the speaker embedding sequence is generated based on speaker embeddings of each speech segment intercepted by the first speech segment in increasing time length;

[0027] A prediction module is used to input the speaker embedding sequence into a pre-trained prediction model, predict the speaker embedding of a second speech segment, and use the predicted speaker embedding as the speaker embedding of the first speech segment; wherein the second speech segment is a speech segment corresponding to a set duration added continuously in time on the basis of the first speech segment.

[0028] In a third aspect, an embodiment of the present application provides an electronic device, comprising: a processor and a memory for storing a computer program that can be run on the processor, wherein the processor, when used to run the computer program, executes the steps of the method described in the first aspect of the embodiment of the present application.

[0029] In a fourth aspect, an embodiment of the present application provides a computer storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the method described in the first aspect of the embodiment of the present application are implemented.

[0030] In a fifth aspect, an embodiment of the present application provides a computer program product, including a computer program, which, when executed by a processor, implements the steps of the method described in the first aspect of the embodiment of the present application.

[0031] The technical solution provided by the embodiment of the present application obtains a speaker embedding sequence of a first speech segment to be recognized; wherein the speaker embedding sequence is generated based on the speaker embedding of each speech segment cut out of the first speech segment in an increasing manner in time length; the speaker embedding sequence is input into a pre-trained prediction model to predict the speaker embedding of the second speech segment, and the predicted speaker embedding is used as the speaker embedding of the first speech segment; wherein the second speech segment is a speech segment corresponding to a set time length added continuously in time on the basis of the first speech segment. Since the speaker embedding sequence contains the correlation relationship of the speaker embedding in continuous time changes, the prediction model can predict the input speaker embedding sequence based on the learned prior knowledge to obtain the speaker embedding of the speech segment with a longer time length, that is, it realizes the prediction of the speaker embedding of the speech segment with a longer time length based on the speech segment with a shorter time length, which can improve the accuracy of speaker embedding extraction and facilitate the improvement of the application effect of subsequent tasks such as speaker log and voiceprint verification. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] Figure 1 This is a flowchart of the speaker embedding extraction method according to an embodiment of the present application;

[0033] Figure 2 This is a schematic diagram of the principle of the speaker embedding extraction method of the application embodiment of this application;

[0034] Figure 3 This is a schematic diagram of the structure of the prediction model of the application embodiment of the present application;

[0035] Figure 4 This is a schematic diagram of the structure of a speaker embedding and extraction device according to an embodiment of the present application;

[0036] Figure 5 It is a schematic diagram of the structure of an electronic device according to an embodiment of the present application. DETAILED DESCRIPTION

[0037] The present application is further described in detail below in conjunction with the accompanying drawings and embodiments.

[0038] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein in the specification of this application are only for the purpose of describing specific embodiments and are not intended to limit this application.

[0039] In the related art, there are mainly three ways to extract speaker embedding with better discrimination from short speech:

[0040] The first method is to improve the method of extracting acoustic features. Acoustic features are the direct input of the speaker embedding extractor, usually using Mel-frequency cepstral coefficients. This type of method proposes a new extraction method or combines multiple extraction methods to improve the effect.

[0041] The second method is to improve the speaker embedding extractor. The main reason why the speaker embedding has low discrimination is that the speaker embedding extractor has difficulty extracting enough speaker features from shorter speech, that is, the model's feature extraction ability is not strong enough. Therefore, this type of method mainly adds some network layers on the basis of the existing model to mine more information from short speech, thereby extracting better speaker embedding. Usually, some structures are added to the original model, such as multi-feature fusion mechanism, attention mechanism, residual block, multi-scale, etc.

[0042] The third method is to directly use a deep learning model to map the speaker embedding extracted from short speech to the speaker embedding extracted from long speech.

[0043] The improvement effect of the above method 1 is general, it increases the computational difficulty, and it can only be applied to specific models, and cannot guarantee universality; the above method 2 often increases the difficulty of training, and the model becomes larger and more complex, but its performance is not significantly improved; the above method 3 is to directly map the poor speaker embedding to the good speaker embedding. There is a certain correlation between the poor embedding and the good embedding, but the input of the mapping model is only one speaker embedding, so it is difficult for the model to learn complex correlations based on limited input.

[0044] Based on this, in various embodiments of the present application, the method of directly mapping the poor speaker embedding corresponding to the short speech segment to the good speaker embedding corresponding to the long speech segment is changed. The speaker embedding sequence is first extracted from the short speech segment in an increasing manner in time. The speaker embedding sequence contains the correlation between the speaker embedding in time changes, and the speaker embedding sequence is predicted in combination with the pre-trained prediction model. The speaker embedding of the longer speech segment can be predicted, thereby improving the accuracy of speaker embedding extraction. In other words, the quality of speaker embedding improves with the increase of speech length. Suppose there is a relatively long speech of someone, and the speaker embedding extracted from it is good. A small segment of speech is intercepted, and the speaker embedding extracted from it is poor. The small segment comes from the long speech, so the speaker embeddings corresponding to the two have a certain correlation. If this relationship can be found, the poor speaker embedding can be converted into a relatively good speaker embedding. Based on this, the method of the embodiment of the present application can use poor speaker embedding to predict good speaker embedding by introducing speaker embedding sequence and prediction model.

[0045] Here, speaker embedding refers to mapping the speaker's speech features to a vector representation in a high-dimensional space. This vector can capture the speaker's unique characteristics and can be used for tasks such as speaker recognition and voiceprint verification.

[0046] The embodiment of the present application provides a speaker embedding extraction method, which can be independently applied to electronic devices with data processing capabilities, such as terminal devices, servers, etc., or can be implemented by the cooperation between terminal devices and servers; wherein the terminal device can specifically be a computer, a smart phone, a personal digital assistant (PDA), etc.; the server can specifically be an application server or a Web server. In actual deployment, the server can be an independent server or a cluster server. Figure 1 As shown, the method includes:

[0047] Step 101, obtaining a speaker embedding sequence of a first speech segment to be recognized; wherein the speaker embedding sequence is generated based on speaker embeddings of each speech segment intercepted by the first speech segment in increasing time length;

[0048] Step 102: input the speaker embedding sequence into a pre-trained prediction model, predict the speaker embedding of a second speech segment, and use the predicted speaker embedding as the speaker embedding of the first speech segment; wherein the second speech segment is a speech segment corresponding to a set duration added continuously in time to the first speech segment.

[0049] It should be noted that since the speaker embedding sequence contains the correlation between speaker embeddings in continuous time changes, the prediction model can predict the input speaker embedding sequence based on the learned prior knowledge to obtain the speaker embedding of the longer speech segment, that is, it realizes the prediction of the speaker embedding of the longer speech segment (that is, the second speech segment) based on the shorter speech segment (that is, the first speech segment), which can improve the accuracy of speaker embedding extraction and facilitate the improvement of the application effects of subsequent speaker logs, voiceprint verification and other tasks.

[0050] Exemplarily, the acquiring of a speaker embedding sequence of a first speech segment to be recognized includes:

[0051] Incrementally segmenting the acoustic features of the first speech segment to be recognized based on the initial truncation duration and the incremental duration to obtain the acoustic features of each speech segment after truncation; wherein the next speech segment to be truncate is a speech segment with the incremental duration added to the previous speech segment;

[0052] Inputting the acoustic features of each speech segment after being cut into a pre-trained speaker embedding extractor to obtain the speaker embedding of each speech segment after being cut; and inputting the acoustic features of the speech segment of each incremental duration into the speaker embedding extractor to obtain the speaker embedding of the speech segment of each incremental duration;

[0053] The speaker embeddings of the intercepted speech segments are sequentially concatenated in pairs with the speaker embeddings of the speech segments of the corresponding incremental durations to obtain a speaker embedding sequence of the first speech segment.

[0054] For example, assuming that the duration of the first speech segment is 2s (seconds), the initial truncation duration and the incremental duration are both 200ms (milliseconds), then the durations of the incrementally intercepted speech segments are 200ms, 400ms, 600ms, ..., 2s respectively. Starting from the second intercepted speech segment, each speech segment corresponds to an incremental duration of 200ms. The acoustic features of each intercepted speech segment and the corresponding incremental duration speech segment are converted into speaker embeddings represented by high-dimensional vectors through a pre-trained speaker embedding extractor, and then the speaker embeddings of each intercepted speech segment and the speaker embeddings of the corresponding incremental duration speech segment are concatenated in pairs to obtain a speaker embedding sequence. For example, assuming that the speaker embeddings of the first to tenth speech segments after incremental interception are E1, E2, ..., E10, starting from the second speech segment, the speaker embeddings of the corresponding speech segments of incremental duration are E11, E12, ..., E19. Since the first speech segment has no corresponding speech segment of incremental duration, it is concatenated with itself by default. In this way, the speaker embedding sequence obtained after pairwise concatenation is [E1, E1], [E2, E11], [E3, E12], [E4, E13], [E5, E14], [E6, E15], [E7, E16], [E8, E17], [E9, E18], [E10, E19].

[0055] It can be understood that the speaker embedding sequence introduced in the embodiment of the present application includes the correlation relationship between the speaker embeddings of the incremental speech fragments in time changes, and establishes a corresponding relationship between the speaker embeddings of the incremental speech fragments and the speaker embeddings of the incremental part, which can realize the increase of output constraints in the training process of the prediction model, so that the prediction model can more accurately establish the correlation relationship between the long speech fragment to be predicted and the known short speech fragment, thereby improving the accuracy of the predicted speaker embedding.

[0056] Exemplarily, the prediction model adopts a sequence-to-sequence (Seq2Seq) architecture, including an encoder and a decoder, wherein the encoder is used to convert the speaker embedding sequence into a first output feature and a second output feature, and the decoder is used to obtain a predicted speaker embedding based on the first output feature, the second output feature and the speaker embedding of the first speech segment at an initial time step, and to obtain a new predicted speaker embedding based on the first output feature, the hidden layer output of the previous time step and the speaker embedding predicted at the previous time step at the next time step, until the speaker embedding of the second speech segment is predicted.

[0057] Here, the sequence-to-sequence architecture is a deep learning architecture that can receive a sequence as input and generate another sequence as output through a specific generation method. A key feature of this architecture is that the lengths of the input sequence and the output sequence can be different, thereby solving the limitation of the recurrent neural network (RNN) when processing sequences of unequal lengths. Among them, the encoder is responsible for encoding the input sequence into a vector of fixed length, which captures the speech information of the input sequence; the decoder can use this vector to generate an output sequence. Since the quality of speaker embedding improves with the increase of the length of the speech segment, the last speaker embedding in the speaker embedding sequence predicted by the decoder is used as the final predicted speaker embedding in the embodiment of the present application.

[0058] It should be noted that the embodiment of the present application predicts speaker embedding based on the combination of the prediction model and the speaker embedding sequence, which has the following advantages:

[0059] 1) Considering temporal correlation: By extracting speaker embedding sequences in increasing order of duration, the embodiment of the present application fully considers the temporal correlation of speech signals. In this way, the prediction model can better capture the changes and evolution of speaker embeddings at different time points, thereby more accurately modeling the relationship between speakers.

[0060] 2) Introducing prior relationships: A speaker embedding sequence is provided as input, which allows the prediction model to refer to certain prior relationships in the speech signal. This prior information can help the prediction model better understand and capture the relationship between speakers, so that the prediction model can understand how the speaker behaves in the speech signal through the speaker embedding sequence.

[0061] 3) Increase of output constraints: Different from directly outputting the final speaker embedding, the embodiment of the present application requires the prediction model to predict the speaker embedding of continuous longer speech that increases in time. In this way, the output constraints are increased during the prediction model training process, forcing the prediction model to more accurately predict the speaker embedding at each time point, thereby improving the accuracy and robustness of the prediction model.

[0062] 4) More accurate associations: The prediction model can more accurately learn and capture the complex associations between speakers. Since the prediction model can observe the gradual changes of speakers from the speaker embedding sequence, it can better understand the relationship between sound features at different time points and specific speakers, thereby improving the accuracy of the associations.

[0063] Exemplarily, the encoder includes a bidirectional gated recurrent unit (GRU), the speaker embedding sequence serves as the input of the bidirectional gated recurrent unit, and the encoder obtains the first output feature based on the hidden layer output of the bidirectional gated recurrent unit, and obtains the second output feature based on the general output of the bidirectional gated recurrent unit.

[0064] It should be noted that the bidirectional GRU can combine forward and backward information to improve the performance of the prediction model. When the forward GRU processes the input sequence, it can only see the input before the current position; while when the backward GRU processes the input sequence, it can see the input after the current position. By splicing the output and hidden states of these two directions together, the model can better understand the contextual information of the input sequence, thereby improving the accuracy of the prediction.

[0065] Here, the general output of the bidirectional GRU refers to the result of splicing the outputs in the two directions together. Specifically, the bidirectional GRU consists of two independent GRU networks, which model the input sequence from the forward and backward directions respectively. The GRU network in each direction will generate an output sequence, and the two output sequences are the same in time steps but in opposite directions. Finally, the two output sequences will be spliced ​​together to form a new output sequence, which contains information in both the forward and backward directions. Exemplarily, the general output input of the bidirectional GRU is processed by the convolution layer, ReLU function (Rectified Linear Unit, linear rectification function), pooling layer, and linear layer in sequence to obtain the aforementioned second output feature.

[0066] Here, the hidden layer output of the bidirectional GRU refers to the result of splicing the hidden states in two directions together. In the forward GRU network, the hidden state represents all the information from the beginning of the sequence to the current time step; in the backward GRU network, the hidden state represents all the information from the end of the sequence to the current time step. The two hidden state sequences are the same in time step, but in opposite directions. Eventually, the two hidden state sequences will be spliced ​​together to form a new hidden state sequence, which contains information in both the forward and backward directions. Exemplarily, the vectors in the two directions of the hidden layer output of the bidirectional GRU are spliced ​​and input into the linear layer, and the aforementioned first output feature is obtained after conversion by the linear layer.

[0067] It should be noted that the introduction of the bidirectional GRU structure in the encoder can better learn the contextual information of the speaker embedding sequence and improve the model performance of the prediction model.

[0068] Exemplarily, the decoder includes a GRU and a linear layer, and the entire process of the decoder is: the input of the GRU includes two parts, namely the hidden input and the general input. For the initial time step, the second output feature mentioned above is used as the hidden input, and the general input is composed of two parts, namely the first output feature mentioned above and the speaker embedding extracted from some acoustic features. Then the output of the GRU is input to the linear layer, and the linear layer outputs the predicted speaker embedding. Next, the predicted speaker embedding and the first output feature mentioned above are concatenated as the general input of the GRU in the calculation of the next time step, and the hidden layer output of the GRU in the previous time step is used as the hidden layer input of the next GRU. After inputting into the GRU, the linear layer outputs the next predicted speaker embedding. Finally, this process is repeated until the final speaker embedding to be predicted is obtained.

[0069] Here, the number of time steps for the decoder to run in the prediction model can be pre-set. Considering that the longer the model prediction length is, the more difficult it is to train, while the improvement is not obvious for shorter predictions, technical personnel in this field can reasonably set the number of prediction time steps according to needs.

[0070] Exemplarily, the method further includes:

[0071] A training sample set is obtained, and a speech segment is intercepted in an increasing manner in duration for each speech sample in the training sample set, and an acoustic feature set of the intercepted speech segment is constructed; wherein the acoustic feature set includes: a first set of acoustic features representing an existing part and a second set of acoustic features representing a part to be predicted;

[0072] The prediction model is trained based on the acoustic feature set until a trained prediction model is obtained.

[0073] Here, speech samples in the training sample set are intercepted into speech segments in increasing duration, and an acoustic feature set is constructed based on the intercepted speech segments, wherein the speech sample can be understood as the speech data of a single speaker, and the speakers corresponding to different speech samples can be the same or different. The prediction model is trained based on the constructed acoustic feature set, so that the prediction model can learn the correlation between the speaker embeddings of the same speaker in time continuity, so that the speaker embedding of long speech can be better predicted based on the speaker embedding of short speech.

[0074] Exemplarily, the training of the prediction model based on the acoustic feature set includes:

[0075] Inputting the acoustic feature set into a pre-trained speaker embedding extractor to obtain speaker embeddings of acoustic features of each existing part in the first set and speaker embeddings of acoustic features of each to-be-predicted part in the second set;

[0076] Extracting acoustic features of incremental parts obtained based on two adjacent interceptions in the first set, and inputting the acoustic features of the incremental parts into the speaker embedding extractor to obtain speaker embeddings of the acoustic features of each incremental part;

[0077] For the speech sample, the speaker embedding of the acoustic features of the existing part and the speaker embedding of the acoustic features of the corresponding incremental part are sequentially concatenated in pairs to obtain a speaker embedding sequence of each speech sample;

[0078] Based on the prediction model and the speaker embedding sequence of each speech sample, predict the predicted speaker embedding of the to-be-predicted part of each speech sample, and obtain the loss value based on the predicted speaker embedding and the speaker embedding of the acoustic features of each to-be-predicted part obtained by the speaker embedding extractor;

[0079] Model parameters of the prediction model are updated based on the loss value.

[0080] It can be understood that if the prediction model is based on the speaker embedding sequence of the speech sample, the speaker embedding corresponding to multiple time steps can be predicted. For example, assuming that the duration of the speech sample is 3s, the constructed acoustic feature set of the intercepted speech segment includes the acoustic features corresponding to the incremental interception duration of 200ms, 400ms, 600ms, ..., 3s, among which the acoustic features corresponding to 200ms, 400ms, 600ms, ..., 2s constitute the first set, and the acoustic features corresponding to 220ms, 240ms, ..., 3s constitute the second set. The second set is composed of acoustic features. The prediction model can predict the speaker embeddings of 220ms, 240ms, ..., 3s based on the speaker embedding sequence of the speech sample. In addition, the speaker embedding extractor can be used to extract the acoustic features corresponding to 220ms, 240ms, ..., 3s to obtain the actual speaker embedding, and the loss value is calculated based on the predicted and actual speaker embeddings. The model parameters of the prediction model are then updated based on the loss value, thereby realizing the training of the prediction model and obtaining the trained prediction model.

[0081] The present application is further described in detail below in conjunction with an application example.

[0082] Figure 2The schematic diagram of the principle of the speaker embedding extraction method of this application embodiment is shown. Assuming that the speech segment to be recognized is a short speech segment, the predicted speaker embedding corresponds to a long speech segment, and it is used as the speaker embedding of the short speech segment. Since the speaker embedding of the long speech segment has better discrimination, the extraction quality of the speaker embedding can be improved. Figure 2 The left side of shows the long acoustic features corresponding to the long speech segment, where the part above the dotted line is the short acoustic features corresponding to the short speech segment. The short acoustic features can be intercepted in increasing time to obtain the acoustic feature set of the existing part, and then the speaker embedding sequence is constructed; the role of the prediction model is to predict the speaker embedding of the part to be predicted based on the speaker embedding sequence of the existing part, and then the speaker embedding of the long speech segment can be obtained. The specific process is as follows: first, the acoustic features of the short speech segment are incrementally intercepted, and then each intercepted acoustic feature is input into the trained speaker embedding extractor, the corresponding speaker embedding is output, and the speaker embedding of the incremental part is extracted, and the two are spliced ​​together, so as to obtain the spliced ​​speaker embedding sequence; then this speaker embedding sequence is input into the prediction model; finally, the prediction model outputs the predicted speaker embedding. The last predicted speaker embedding is the final "good" speaker embedding.

[0083] The prediction model of this application embodiment adopts Figure 3 The structure shown is a Seq2Seq architecture. The prediction model includes an encoder and a decoder.

[0084] The encoder consists of a bidirectional GRU layer, a one-dimensional convolutional layer, a pooling layer, and three linear layers. The hidden layer dimension of the bidirectional GRU layer is 512, the output dimension of the one-dimensional convolutional layer is 1500, the statistical pooling layer used in the pooling layer calculates the mean and variance of the channel, and the dimension of the linear layer is 512.

[0085] The whole process of the encoder is: first Figure 2 The concatenated speaker embedding sequence is used as input to the bidirectional GRU layer. The output of the bidirectional GRU layer includes the general output and the hidden layer output. Since it is a bidirectional GRU, the hidden layer outputs two vectors, which are concatenated and input into the linear layer to obtain the potential feature 1 (corresponding to the first output feature mentioned above). The general output of the bidirectional GRU is then input into the one-dimensional convolutional layer, activated using ReLU, and then the output of the convolutional layer is pooled. Specifically, the mean and variance of each channel are calculated to obtain the mean vector and variance vector, and the two are concatenated; finally, the pooled output is input into two linear layers to obtain the potential feature 2 (corresponding to the second output feature mentioned above).

[0086] The decoder consists of a GRU unit and a linear layer. The hidden layer dimension of the GRU unit is 512, and the output dimension of the linear layer is the number of elements in the speaker embedding vector. The entire process of the decoder is: first input the data into the GRU layer. The input of the GRU consists of two parts, namely hidden input and general input. The latent feature 2 is used as the hidden input, and the general input is composed of two parts, namely latent feature 1 and Figure 2 The speaker embedding extracted from some acoustic features has been extracted; then the output of the GRU layer is input to the linear layer, and the linear layer outputs the predicted speaker embedding. Then the predicted speaker embedding and the latent feature 1 are concatenated as the general input of the GRU layer in the next calculation, and the hidden layer output of the previous GRU is used as the hidden layer input of the next GRU layer. After inputting into the GRU, the linear layer outputs the next predicted speaker embedding; finally, this process is repeated until the final predicted speaker embedding is obtained.

[0087] The training process of the prediction model of this application embodiment is as follows:

[0088] Before training the prediction model, a trained speaker embedding extractor G(f,θ) is required.

[0089] Assuming there are N speech samples for training, the training process for each batch is as follows:

[0090] (1) Randomly select n speech samples from the training set and calculate the acoustic features of each speech sample. The frame length and frame shift used in the calculation process depend on the trained speaker embedding extraction model. Finally, n complete acoustic features are obtained.

[0091] (2) Randomly extract consecutive z-frame data from each complete acoustic feature.

[0092] (3) The data of the z frame is Figure 2 The acoustic feature set F = {f ij ,i=1,…,n,j=1,…,P}, where {f ij , i=1,…,n,j=1,…,p} represents the existing part of the acoustic feature set (corresponding to the first set mentioned above), and {f ij ,i=1,…,n,j=p+1,…,P} represents the acoustic feature set of the part to be predicted (corresponding to the second set mentioned above). Here, considering that the longer the model prediction length is, the more difficult it is to train, and the improvement is not obvious when the prediction is shorter, the prediction length is fixed to 10, that is, Pp=10.

[0093] (4) Input the acoustic feature set F into the speaker embedding extractor G(f,θ) to obtain the corresponding speaker embedding set. Represents the speaker embedding of the existing part, using Represents the speaker embedding of the part to be predicted. ij ,i=1,…,n,j=1,…,p} is input into the speaker embedding extractor G(f,θ) to obtain the corresponding speaker embedding set

[0094] (5) E I With E Z Spliced ​​together and input into the prediction model Y(e,θ') to obtain the predicted acoustic features

[0095]

[0096] (6) Using the expected output E A and the predicted output E O Calculate the cosine distance loss, specifically:

[0097]

[0098] (7) Back propagation, using the RMSprop optimizer to update the parameter θ' of the prediction model Y(e,θ'). The learning rate is 0.0001.

[0099] (8) Return to step 1 until the model loss no longer decreases.

[0100] The use process of the prediction model of this application embodiment includes: in the speaker embedding extraction stage of the speaker log, the voice clips are no longer directly input into the speaker embedding extractor, but the voice clips are incrementally intercepted, the speaker embedding of each intercepted data is extracted, and the speaker embedding of the incremental data is extracted, the two are spliced ​​and input into the trained prediction model, and the last step output of the decoder of the prediction model is used as the final speaker embedding of the voice clip.

[0101] In an application example, a speaker embedding extractor based on ResNet34 is used as the extraction model. The model is trained using two datasets, VoxCeleb1 and VoxCeleb2, using Fbank as the acoustic feature, with 60 channels, 25ms frame length, and 10ms frame shift. The batch size is 500, the learning rate is 0.001, and the training is 100 rounds. The learning rate is reduced by 2% each round. The Adam optimizer is used, and the number of speaker embedding elements is 512.

[0102] For the training of the prediction model: the value of z mentioned above is 200, the batch size is 128, the RMSprop optimizer is used, the learning rate is initially 0.0001, and it is reduced by 2% each round, for a total of 80 rounds of training.

[0103] For the training of the technical model in the aforementioned method 3 (hereinafter referred to as the enhanced model): the batch size is 128, the learning rate is 0.001, a total of 80 rounds of training, and the rate is reduced by 2% each round. The Adam optimizer is used, and the speaker embedding of the 2-second long speech is used as the expected output, and the speaker embedding of the 1.5-second long speech is used as the model input.

[0104] The test data uses the Full-corpus part of the AMI dataset. Each speech in the test set is segmented using a window length of 1.5s and a window shift of 0.75s. Speaker embedding is extracted according to the method of the embodiment of the present application, scored using cosine similarity, and clustered using spectral clustering. The speaker diarization error rate (DER) is used to evaluate the effect of the speaker log task, and the results are shown in Table 1:

[0105] Table 1

[0106]

[0107] It can be seen from the table that under the two test conditions, the DER of the model proposed in the embodiment of the present application is the lowest, indicating that its performance is the best.

[0108] In order to implement the method of the embodiment of the present application, the embodiment of the present application also provides a speaker embedding and extraction device, which corresponds to the above-mentioned speaker embedding and extraction method, and each step in the above-mentioned speaker embedding and extraction method embodiment is also fully applicable to the present speaker embedding and extraction device embodiment.

[0109] like Figure 4 As shown, the speaker embedding extraction device includes: an acquisition module 401 and a prediction module 402. The acquisition module 401 is used to acquire a speaker embedding sequence of a first speech segment to be recognized; wherein the speaker embedding sequence is generated based on the speaker embeddings of each speech segment cut out of the first speech segment in an increasing manner in time length; the prediction module 402 is used to input the speaker embedding sequence into a pre-trained prediction model, predict the speaker embedding of a second speech segment, and use the predicted speaker embedding as the speaker embedding of the first speech segment; wherein the second speech segment is a speech segment corresponding to a set time length added continuously in time on the basis of the first speech segment.

[0110] In some embodiments, the acquisition module 401 is specifically used to:

[0111] Incrementally segmenting the acoustic features of the first speech segment to be recognized based on the initial truncation duration and the incremental duration to obtain the acoustic features of each speech segment after truncation; wherein the next speech segment to be truncate is a speech segment with the incremental duration added to the previous speech segment;

[0112] Inputting the acoustic features of each speech segment after being cut into a pre-trained speaker embedding extractor to obtain the speaker embedding of each speech segment after being cut; and inputting the acoustic features of the speech segment of each incremental duration into the speaker embedding extractor to obtain the speaker embedding of the speech segment of each incremental duration;

[0113] The speaker embeddings of the intercepted speech segments are sequentially concatenated in pairs with the speaker embeddings of the speech segments of the corresponding incremental durations to obtain a speaker embedding sequence of the first speech segment.

[0114] Exemplarily, the prediction model adopts a sequence-to-sequence architecture, including an encoder and a decoder, wherein the encoder is used to convert the speaker embedding sequence into a first output feature and a second output feature, and the decoder is used to obtain a predicted speaker embedding based on the first output feature, the second output feature and the speaker embedding of the first speech segment at an initial time step, and to obtain a new predicted speaker embedding based on the first output feature, the hidden layer output of the previous time step and the speaker embedding predicted at the previous time step at a next time step, until the speaker embedding of the second speech segment is predicted.

[0115] Exemplarily, the encoder includes a bidirectional gated recurrent unit, the speaker embedding sequence serves as an input of the bidirectional gated recurrent unit, the encoder obtains the first output feature based on the hidden layer output of the bidirectional gated recurrent unit, and obtains the second output feature based on the general output of the bidirectional gated recurrent unit.

[0116] Exemplarily, the speaker embedding extraction device further includes: a training module 403, which is used to:

[0117] A training sample set is obtained, and a speech segment is intercepted in an increasing manner in duration for each speech sample in the training sample set, and an acoustic feature set of the intercepted speech segment is constructed; wherein the acoustic feature set includes: a first set of acoustic features representing an existing part and a second set of acoustic features representing a part to be predicted;

[0118] The prediction model is trained based on the acoustic feature set until a trained prediction model is obtained.

[0119] Exemplarily, the training module 403 is specifically used for:

[0120] Inputting the acoustic feature set into a pre-trained speaker embedding extractor to obtain speaker embeddings of acoustic features of each existing part in the first set and speaker embeddings of acoustic features of each to-be-predicted part in the second set;

[0121] Extracting acoustic features of incremental parts obtained based on two adjacent interceptions in the first set, and inputting the acoustic features of the incremental parts into the speaker embedding extractor to obtain speaker embeddings of the acoustic features of each incremental part;

[0122] For the speech sample, the speaker embedding of the acoustic features of the existing part and the speaker embedding of the acoustic features of the corresponding incremental part are sequentially concatenated in pairs to obtain a speaker embedding sequence of each speech sample;

[0123] Based on the prediction model and the speaker embedding sequence of each speech sample, predict the predicted speaker embedding of the to-be-predicted part of each speech sample, and obtain the loss value based on the predicted speaker embedding and the speaker embedding of the acoustic features of each to-be-predicted part obtained by the speaker embedding extractor;

[0124] Model parameters of the prediction model are updated based on the loss value.

[0125] In actual application, the acquisition module 401, the prediction module 402 and the training module 403 can be implemented by a processor in the electronic device. Of course, the processor needs to run the computer program in the memory to implement its functions.

[0126] It should be noted that: the speaker embedding extraction device provided in the above embodiment only uses the division of the above program modules as an example to illustrate when performing speaker embedding extraction. In actual applications, the above processing can be assigned to different program modules as needed, that is, the internal structure of the device is divided into different program modules to complete all or part of the above-described processing. In addition, the speaker embedding extraction device provided in the above embodiment and the speaker embedding extraction method embodiment belong to the same concept, and the specific implementation process is detailed in the method embodiment, which will not be repeated here.

[0127] Based on the hardware implementation of the above program modules and in order to implement the method of the embodiment of the present application, the embodiment of the present application also provides an electronic device. Figure 5 Only an exemplary structure of the electronic device is shown, not all structures, and it can be implemented as needed. Figure 5 Partial or complete structure shown.

[0128] like Figure 5 As shown, the electronic device 500 provided in the embodiment of the present application includes: at least one processor 501, a memory 502, a user interface 503 and at least one network interface 504. The various components in the electronic device 500 are coupled together through a bus system 505. It can be understood that the bus system 505 is used to realize the connection and communication between these components. In addition to the data bus, the bus system 505 also includes a power bus, a control bus and a status signal bus. However, for the sake of clarity, in Figure 5Various buses are labeled as bus system 505 .

[0129] The user interface 503 may include a display, a keyboard, a mouse, a trackball, a click wheel, keys, buttons, a touch pad or a touch screen.

[0130] The memory 502 in the embodiment of the present application is used to store various types of data to support the operation of the electronic device. Examples of such data include: any computer program used to operate on the electronic device.

[0131] The speaker embedding extraction method disclosed in the embodiment of the present application can be applied to the processor 501, or implemented by the processor 501. The processor 501 may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the speaker embedding extraction method can be completed by the hardware integrated logic circuit or software instructions in the processor 501. The above-mentioned processor 501 can be a general-purpose processor, a digital signal processor (DSP, Digital Signal Processor), or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, etc. The processor 501 can implement or execute the various methods, steps and logic block diagrams disclosed in the embodiment of the present application. The general-purpose processor can be a microprocessor or any conventional processor, etc. The steps of the method disclosed in the embodiment of the present application can be directly embodied as a hardware decoding processor to execute, or a combination of hardware and software modules in the decoding processor to execute. The software module can be located in a storage medium, and the storage medium is located in the memory 502. The processor 501 reads the information in the memory 502 and completes the steps of the speaker embedding extraction method provided in the embodiment of the present application in combination with its hardware.

[0132] In an exemplary embodiment, the electronic device may be implemented by one or more application specific integrated circuits (ASIC), DSP, programmable logic device (PLD), complex programmable logic device (CPLD), field programmable gate array (FPGA), general processor, controller, microcontroller (MCU), microprocessor, or other electronic components to execute the aforementioned method.

[0133] It can be understood that the memory 502 can be a volatile memory or a non-volatile memory, and can also include both volatile and non-volatile memories. Among them, the non-volatile memory can be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), a magnetic random access memory (FRAM), a flash memory, a magnetic surface memory, an optical disk, or a compact disc read-only memory (CD-ROM); the magnetic surface memory can be a disk memory or a tape memory. The volatile memory can be a random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of RAM are available, such as static random access memory (SRAM), synchronous static random access memory (SSRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDRSDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link dynamic random access memory (SLDRAM), and direct rambus random access memory (DRRAM).The memories described in the embodiments of the present application are intended to include, but are not limited to, these and any other suitable types of memories.

[0134] In an exemplary embodiment, the present application also provides a computer storage medium, which can be a computer-readable storage medium, for example, a memory 502 storing a computer program, and the computer program can be executed by a processor 501 of an electronic device to complete the steps described in the method of the present application embodiment. The computer-readable storage medium can be a memory such as a ROM, a PROM, an EPROM, an EEPROM, a Flash Memory, a magnetic surface memory, an optical disk, or a CD-ROM.

[0135] In an exemplary embodiment, the embodiment of the present application further provides a computer program product, including a computer program, which can be executed by the processor 501 of the electronic device 500 to complete the steps described in the method of the embodiment of the present application.

[0136] It should be noted that: "first", "second", etc. are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence.

[0137] In addition, the technical solutions described in the embodiments of the present application can be combined arbitrarily without conflict.

[0138] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A speaker embedding extraction method, characterized in that: include: Obtaining a speaker embedding sequence of a first speech segment to be recognized; wherein the speaker embedding sequence is generated based on speaker embeddings of each speech segment intercepted by the first speech segment in increasing time length; The speaker embedding sequence is input into a pre-trained prediction model to predict the speaker embedding of a second speech segment, and the predicted speaker embedding is used as the speaker embedding of the first speech segment; wherein the second speech segment is a speech segment corresponding to a set duration added continuously in time to the first speech segment.

2. The method according to claim 1, characterized in that The step of obtaining a speaker embedding sequence of a first speech segment to be recognized includes: Incrementally segmenting the acoustic features of the first speech segment to be recognized based on the initial truncation duration and the incremental duration to obtain the acoustic features of each speech segment after truncation; wherein the next speech segment to be truncate is a speech segment with the incremental duration added to the previous speech segment; Inputting the acoustic features of each speech segment after being cut into a pre-trained speaker embedding extractor to obtain the speaker embedding of each speech segment after being cut; and inputting the acoustic features of the speech segment of each incremental duration into the speaker embedding extractor to obtain the speaker embedding of the speech segment of each incremental duration; The speaker embeddings of the intercepted speech segments are sequentially concatenated in pairs with the speaker embeddings of the speech segments of the corresponding incremental durations to obtain a speaker embedding sequence of the first speech segment.

3. The method according to claim 1, characterized in that The prediction model adopts a sequence-to-sequence architecture, including an encoder and a decoder, wherein the encoder is used to convert the speaker embedding sequence into a first output feature and a second output feature, and the decoder is used to obtain a predicted speaker embedding based on the first output feature, the second output feature and the speaker embedding of the first speech segment at an initial time step, and obtain a new predicted speaker embedding based on the first output feature, the hidden layer output of the previous time step and the speaker embedding predicted at the previous time step at a next time step, until the speaker embedding of the second speech segment is predicted.

4. The method according to claim 3, characterized in that The encoder includes a bidirectional gated recurrent unit, the speaker embedding sequence is used as an input of the bidirectional gated recurrent unit, the encoder obtains the first output feature based on the hidden layer output of the bidirectional gated recurrent unit, and obtains the second output feature based on the general output of the bidirectional gated recurrent unit.

5. The method according to claim 1, characterized in that The method further comprises: A training sample set is obtained, and a speech segment is intercepted in an increasing manner in duration for each speech sample in the training sample set, and an acoustic feature set of the intercepted speech segment is constructed; wherein the acoustic feature set includes: a first set of acoustic features representing an existing part and a second set of acoustic features representing a part to be predicted; The prediction model is trained based on the acoustic feature set until a trained prediction model is obtained.

6. The method according to claim 5, characterized in that The training of the prediction model based on the acoustic feature set includes: Inputting the acoustic feature set into a pre-trained speaker embedding extractor to obtain speaker embeddings of acoustic features of each existing part in the first set and speaker embeddings of acoustic features of each to-be-predicted part in the second set; Extracting acoustic features of incremental parts obtained based on two adjacent interceptions in the first set, and inputting the acoustic features of the incremental parts into the speaker embedding extractor to obtain speaker embeddings of the acoustic features of each incremental part; For the speech sample, the speaker embedding of the acoustic features of the existing part and the speaker embedding of the acoustic features of the corresponding incremental part are sequentially concatenated in pairs to obtain a speaker embedding sequence of each speech sample; Based on the prediction model and the speaker embedding sequence of each speech sample, predict the predicted speaker embedding of the to-be-predicted part of each speech sample, and obtain the loss value based on the predicted speaker embedding and the speaker embedding of the acoustic features of each to-be-predicted part obtained by the speaker embedding extractor; Model parameters of the prediction model are updated based on the loss value.

7. A speaker embedding extraction device, characterized in that: include: An acquisition module, configured to acquire a speaker embedding sequence of a first speech segment to be recognized; wherein the speaker embedding sequence is generated based on speaker embeddings of each speech segment intercepted by the first speech segment in increasing time length; A prediction module is used to input the speaker embedding sequence into a pre-trained prediction model, predict the speaker embedding of a second speech segment, and use the predicted speaker embedding as the speaker embedding of the first speech segment; wherein the second speech segment is a speech segment corresponding to a set duration added continuously in time on the basis of the first speech segment.

8. An electronic device, characterized in that: include: A processor and a memory for storing a computer program that can be executed on the processor, wherein: The processor is used to execute the steps of the method according to any one of claims 1 to 6 when running a computer program.

9. A computer storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

Images