A target speaker activity detection system based on generative algorithm

Abstract

The application discloses a target speaker activity detection system based on a generative algorithm, relates to the field of speech, and uses a generative method to detect a target speaker, realizes a generative algorithm based on flow matching Flow-Matching in a sequence-to-sequence target speaker voice activity detection Seq2Seq-TSVAD system, and maps a binary label sequence to a dense hidden space before the generative algorithm is applied. The system performance based on the generative algorithm exceeds a baseline based on a discriminative algorithm.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of speech, in particular to a target speaker activity detection system based on generative algorithm. BACKGROUND

[0002] Target speaker activity detection is usually considered as a discriminative task, and discriminative methods are used to produce fixed results. In this paper, we first explore the use of generative methods for target speaker detection. We implement a flow-matching (FM) based generative algorithm in a sequence-to-sequence target speaker voice activity detection (Seq2Seq-TSVAD) system. Our experiments show that directly applying generative methods to the raw binary label sequence space output by target speaker activity detection is ineffective. To address this problem, we propose to map the binary label sequence to a dense latent space before applying the generative algorithm. Our Flow-TSVAD method outperforms the Seq2Seq-TSVAD system. In addition, we observe that the FM algorithm converges quickly in the inference phase, and only two steps of inference are needed to obtain satisfactory results. As a generative model, Flow-TSVAD allows different results to be sampled by running the model multiple times. Furthermore, fusing the results of different sampling instances can further improve performance. SUMMARY

[0003] In view of the above defects of the prior art, the technical problem to be solved by the present application is how to perform target activity detection. To achieve the above purpose, the present application provides a target speaker activity detection system based on generative algorithm, characterized in that the present application uses a generative method for target speaker detection, and implements a flow-matching (FM) based generative algorithm in a sequence-to-sequence target speaker voice activity detection (Seq2Seq-TSVAD) system. Before applying the generative algorithm, the binary label sequence is mapped to a dense latent space.

[0004] Further, the flow-matching (FM) based generative algorithm maps a simple distribution to an unknown distribution by constructing a flow path, and this process uses an ordinary differential equation to represent:

[0005]

[0006] Further, to construct the flow path, a neural network is used to fit the vector field u in the ordinary differential equation t .

[0007] Further, in order to map the binary label sequence to a dense latent space, a label auto-encoder model Label-AE is proposed.

[0008] Further, the target speaker activity detection system is composed of three parts, including a speaker encoder R-vector, a Conformer and a decoder.

[0009] Further, the speech feature X is input into a pre-trained speaker encoder R-vector to obtain a frame-level speaker representation H, and the speaker representation H is further sent to the Conformer for further processing to obtain a feature O more suitable for speech activity detection, and finally output through the decoder.

[0010] Further, the decoder has five inputs, a positional encoding Positional Embedding, a speaker registration vector e, a sample zt on the flow path, an output O of the Conformer model, and time information t involved in the flow model modeling.

[0011] Further, the decoder block takes the zt as a direct input and sends it to the self-attention (elf-attention module.

[0012] Further, the speaker registration vector e is used as conditional information and spliced into the query input of the cross-attention module cross-attention.

[0013] Further, the output O of the Conformer is used as the value (value, V) input and the key (key, K) input of the cross-attention module; the positional encoding positional embedding is also spliced into the key input; and the final output vt of the decoder is the estimation of the vector field ut in the flow model.

[0014] The flow matching algorithm of the present application can help the system to obtain better results when solving ordinary differential equations through multiple iterations. In addition, the generative algorithm can obtain different results in the multiple sampling process, and the fusion of different results can further improve the system performance.

[0015] The concept, specific structure and technical effects of the present application will be further described below in combination with the drawings, so as to fully understand the purpose, features and effects of the present application. BRIEF DESCRIPTION OF DRAWINGS

[0016] Figure 1 is a system overall diagram of a preferred embodiment of the present application;

[0017] Figure 2 Figure 1 is a decoder module diagram of a preferred embodiment of the present application;

[0018] Figure 3 Figure 2 is a system inference diagram of a preferred embodiment of the present application. DETAILED DESCRIPTION

[0019] The technical content of the present application will be more clearly understood and facilitated to be understood by referring to the following description of the preferred embodiments of the present application with reference to the accompanying drawings. The present application can be embodied in many different forms and the scope of the present application should not be limited to the embodiments described herein.

[0020] In the drawings, components of the same structure are designated by the same reference numerals, and components having similar structures or functions are designated by similar reference numerals. The size and thickness of each component shown in the drawings are arbitrarily shown, and the present application is not limited to the size and thickness of each component. In order to make the drawing clearer, the thickness of some components is appropriately exaggerated in some places in the drawing.

[0021] Figure 1 The overall structure of the system is shown. First, the speech feature X is input into a pre-trained speaker encoder R-vector to obtain a frame-level speaker representation H. The speaker representation H is further sent to the Conformer for further processing to obtain a feature O more suitable for speech activity detection. For the decoder part, there are five inputs in total:

[0022] (1) Positional Embedding: encodes the position information of the input

[0023] (2) Speaker registration vector e

[0024] (3) A sample on the flow path, zt

[0025] (4) The output O of the Confomer model

[0026] (5) The time information t involved in the flow model modeling, which is modeled into the model in the form of a time vector.

[0027] Figure 2The structure of the decoder block is shown in more detail. The decoder block takes zt as direct input and sends it to the self-attention module, in addition, the speaker embedding vector e is used as conditional information and concatenated to the query (Q) input of the cross-attention module. The output O of the Conformer is used as the value (V) input and key (K) input of the cross-attention module. In addition, the positional embedding is also concatenated to the key input. In addition, the final output vt of the decoder is the estimation of the vector field ut in the flow model.

[0028] Figure 3 The inference process of the system is shown, in which we can integrate the vector field vt output by the decoder to obtain the label in the latent space, and then use the decoder of the Label-AE to map the label in the latent space back to the original binary sequence.

[0029] The above detailed description of the preferred embodiments of the present application. It should be understood that those of ordinary skill in the art can make many modifications and changes without creative labor based on the concept of the present application. Therefore, any technical solutions obtained by logical analysis, reasoning or limited experiments based on the existing technology within the scope of the concept of the present application should be within the protection scope determined by the claims.

Claims

1. A target speaker activity detection system based on a generative algorithm, characterized in that: The present invention uses a generative approach to perform target speaker detection and implements a flow-matching generative algorithm in a sequence-to-sequence target speaker voice activity detection (Seq2Seq-TSVAD) system. Before applying the generative algorithm, the binary label sequence is mapped into a dense latent space. The flow-matching generation algorithm maps a simple distribution to an unknown distribution by constructing a flow path. This process is represented by an ordinary differential equation: ( ), at time t=0, represents Gaussian distribution; z is the variable in the distribution, Represents the probability density of the distribution when t=0; At time t The derivative with respect to time t, the derivative ( ) Integrating from time t=0 to time t=1, we can get , we can get the distribution of voice activity detection labels in this article.

2. The target speaker activity detection system based on a generative algorithm according to claim 1, wherein: In order to construct the flow path, it is necessary to use a neural network to fit the vector field in the ordinary differential equation .

3. The target speaker activity detection system based on a generative algorithm according to claim 2, wherein: In order to map the binary label sequence into a dense latent space, a label auto-encoder model Label-AE is proposed.

4. The target speaker activity detection system based on a generative algorithm according to claim 3, wherein: The target speaker activity detection system consists of three parts, including a speaker encoder R-vector, a Conformer, and a decoder.

5. The target speaker activity detection system based on a generative algorithm according to claim 4, wherein: The speech feature X will be input into a pre-trained speaker encoder R-vector to obtain a frame-level speaker representation H. The speaker representation H will be further sent to the Conformer for further processing to obtain a feature O that is more suitable for voice activity detection, and finally output by the decoder.

6. The target speaker activity detection system based on a generative algorithm according to claim 5, wherein: The decoder has five inputs: positional embedding, speaker registration vector e, a sample zt on the flow path, the output O of the Confomer model, and the time information t involved in the flow model modeling.

7. The target speaker activity detection system based on a generative algorithm according to claim 6, wherein: The decoder block takes zt as direct input and feeds it into the self-attention module.

8. The target speaker activity detection system based on a generative algorithm according to claim 7, wherein: The speaker registration vector e is used as conditional information and is concatenated into the query input of the cross-attention module.

9. The target speaker activity detection system based on a generative algorithm according to claim 8, wherein: The output O of the Conformer is used as the value (V) input and key (K) input of the cross-attention module; the positional embedding is also concatenated to the key input; the final output vt of the decoder is an estimate of the vector field ut in the flow model.

Images