Target speaker activity detection system based on generative algorithm

Abstract

The invention discloses a target speaker activity detection system based on a generative algorithm, and relates to the voice field, the target speaker is detected by using a generation method, and the generation algorithm based on Flow-Matching is realized in a sequence-to-sequence target speaker voice activity detection Seq2Seq-TSVAD system; before a generation algorithm is applied, a binary tag sequence is mapped to a dense hidden space. The system performance based on the generative algorithm exceeds the baseline based on the discrimination algorithm.

Description

Technical Field

[0001] The present invention relates to the field of speech, and in particular to a target speaker activity detection system based on a generative algorithm. Background Art

[0002] Target speaker activity detection is usually considered as a discriminative task, using discriminative methods to produce fixed results. In this paper, we explore the use of generative methods for target speaker detection for the first time. We implement a generative algorithm based on Flow-Matching (FM) in a sequence-to-sequence target speaker voice activity detection (Seq2Seq-TSVAD) system. Our experiments show that directly applying generative methods to the raw binary label sequence space output by target speaker activity detection is ineffective. To address this issue, we propose to map the binary label sequence into a dense latent space before applying the generative algorithm. Our Flow-TSVAD method outperforms the Seq2Seq-TSVAD system. In addition, we observe that the FM algorithm converges quickly in the inference stage and only needs two steps of inference to obtain satisfactory results. As a generative model, Flow-TSVAD allows sampling different results by running the model multiple times. In addition, fusing the results of different sampled instances can further improve the performance. Summary of the invention

[0003] In view of the above-mentioned defects in the prior art, the technical problem to be solved by the present invention is how to perform target active person detection. To achieve the above-mentioned purpose, the present invention provides a target speaker activity detection system based on a generative algorithm, which is characterized in that the present invention uses a generative method to perform target speaker detection, and implements a flow-matching based generation algorithm in a sequence-to-sequence target speaker voice activity detection Seq2Seq-TSVAD system; before applying the generation algorithm, the binary label sequence is mapped to a dense latent space.

[0004] Furthermore, the flow-matching generation algorithm maps a simple distribution to an unknown distribution by constructing a flow path. This process is represented by an ordinary differential equation:

[0005]

[0006] Furthermore, in order to construct the flow path, it is necessary to use a neural network to fit the vector field u in the ordinary differential equation t .

[0007] Furthermore, in order to map the binary label sequence to a dense latent space, a label auto-encoder model Label-AE is proposed.

[0008] Furthermore, the target speaker activity detection system consists of three parts, including a speaker encoder R-vector, a Conformer and a decoder.

[0009] Furthermore, the speech feature X will be input into a pre-trained speaker encoder R-vector to obtain a frame-level speaker representation H, and the speaker representation H will be further sent to the Conformer for further processing to obtain a feature O that is more suitable for voice activity detection, and finally output through the decoder.

[0010] Furthermore, the decoder has five inputs: positional embedding, speaker registration vector e, a sample zt on the flow path, output O of the Confomer model, and time information t involved in modeling the flow model.

[0011] Furthermore, the decoder block takes the zt as direct input and sends it to the self-attention module.

[0012] Furthermore, the speaker registration vector e is used as conditional information and concatenated into the query input of the cross-attention module.

[0013] Furthermore, the output O of the Conformer is used as the value (V) input and the key (K) input of the cross-attention module; the positional embedding is also concatenated to the key input; the final output vt of the decoder is an estimate of the vector field ut in the flow model.

[0014] The flow matching algorithm of the present invention will go through multiple iterations when solving ordinary differential equations, which can help the system achieve better results. In addition, the generative algorithm will obtain different results during multiple sampling processes, and the fusion of different results can further improve the system performance.

[0015] The concept, specific structure and technical effects of the present invention will be further described below in conjunction with the accompanying drawings to fully understand the purpose, characteristics and effects of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 It is a system overall diagram of a preferred embodiment of the present invention;

[0017] Figure 2 It is a decoder module diagram of a preferred embodiment of the present invention;

[0018] Figure 3 It is a system reasoning diagram of a preferred embodiment of the present invention. DETAILED DESCRIPTION

[0019] The following describes several preferred embodiments of the present invention with reference to the drawings in the specification, so that the technical content is clearer and easier to understand. The present invention can be embodied in many different forms of embodiments, and the protection scope of the present invention is not limited to the embodiments mentioned in the text.

[0020] In the drawings, components with the same structure are indicated by the same numerical reference numerals, and components with similar structures or functions are indicated by similar numerical reference numerals. The size and thickness of each component shown in the drawings are arbitrarily shown, and the present invention does not limit the size and thickness of each component. In order to make the illustration clearer, the thickness of the components is appropriately exaggerated in some places in the drawings.

[0021] Figure 1 The overall structure of the system is shown. First, the speech feature X is input into a pre-trained speaker encoder R-vector to obtain a frame-level speaker representation H. The speaker representation H is further sent to the Conformer for further processing to obtain a feature O that is more suitable for voice activity detection. For the decoder part, there are five inputs:

[0022] (1) Positional Embedding: Encodes the input position information

[0023] (2) Speaker registration vector e

[0024] (3) A sample on the flow path, zt

[0025] (4) Output of the Confomer model O

[0026] (5) The time information t involved in flow modeling is modeled into the model in the form of a time vector.

[0027] Figure 2The structure of the decoder block is shown in more detail. The decoder block takes zt as direct input and sends it to the self-attention module. In addition, the speaker registration vector e is used as conditional information and is concatenated to the query (Q) input of the cross-attention module. The output O of the Conformer is used as the value (V) input and key (K) input of the cross-attention module. In addition, the positional embedding is also concatenated to the key input. In addition, the final output vt of the decoder is an estimate of the vector field ut in the flow model.

[0028] Figure 3 It shows the reasoning process of the system. During the reasoning process, we can integrate the ordinary differential equation according to the vector field vt output by the decoder to obtain the label in the latent space, and then use the decoder of Label-AE to map the label in the latent space back to the original binary sequence.

[0029] The preferred specific embodiments of the present invention are described in detail above. It should be understood that ordinary technicians in the field can make many modifications and changes based on the concept of the present invention without creative work. Therefore, all technical solutions that can be obtained by technicians in the technical field based on the concept of the present invention through logical analysis, reasoning or limited experiments on the basis of the prior art should be within the scope of protection determined by the claims.

Claims

1. A target speaker activity detection system based on a generative algorithm, characterized in that: The present invention uses a generation method to perform target speaker detection, and implements a flow-matching generation algorithm in a sequence-to-sequence target speaker voice activity detection Seq2Seq-TSVAD system; before applying the generation algorithm, the binary label sequence is mapped to a dense latent space.

2. The target speaker activity detection system based on a generative algorithm as claimed in claim 1, characterized in that: The flow matching flow-matching generation algorithm maps a simple distribution to an unknown distribution by constructing a flow path flowpath. This process is represented by an ordinary differential equation: At time t = 0, φ0(z) represents a Gaussian distribution; z is the variable in the distribution, φ0(z) represents the probability density of the distribution when t=0; u t At time t, φ t (z) The derivative with respect to time t, the derivative u t (φ t By integrating (z) from time t=0 to time t=1, we can get φ1(z), and we can get the distribution of the voice activity detection labels in this paper.

3. The target speaker activity detection system based on a generative algorithm as claimed in claim 2, characterized in that: In order to construct the flow path, a neural network is used to fit the vector field u in the ordinary differential equation t .

4. The target speaker activity detection system based on a generative algorithm as claimed in claim 3, characterized in that: In order to map the binary label sequence to a dense latent space, a label auto-encoder model Label-AE is proposed.

5. The target speaker activity detection system based on a generative algorithm as claimed in claim 4, characterized in that: The target speaker activity detection system consists of three parts, including a speaker encoder R-vector, a Conformer and a decoder.

6. The target speaker activity detection system based on a generative algorithm as claimed in claim 5, characterized in that: The speech feature X will be input into a pre-trained speaker encoder R-vector to obtain a frame-level speaker representation H. The speaker representation H will be further sent to the Conformer for further processing to obtain a feature O that is more suitable for voice activity detection, and finally output through the decoder.

7. The target speaker activity detection system based on a generative algorithm as claimed in claim 6, characterized in that: The decoder has five inputs: positional embedding, speaker registration vector e, a sample zt on the flow path, the output O of the Confomer model, and the time information t involved in modeling the flow model.

8. The target speaker activity detection system based on a generative algorithm as claimed in claim 7, characterized in that: The decoder block takes zt as direct input and feeds it into the self-attention module.

9. The target speaker activity detection system based on a generative algorithm as claimed in claim 8, characterized in that: The speaker registration vector e is used as conditional information and concatenated into the query input of the cross-attention module.

10. The target speaker activity detection system based on a generative algorithm as claimed in claim 9, characterized in that: The output O of the Conformer is used as the value (value, V) input and key (key, K) input of the cross-attention module; the positional embedding is also concatenated to the key input; the final output vt of the decoder is an estimate of the vector field ut in the flow model.

Images