Multi-voice separation method based on lightweight dual-path Transform network

Abstract

The invention discloses a multi-voice separation method based on a lightweight dual-path Transform network, and the method comprises the steps: collecting audio multi-voice data, and carrying out the preprocessing of the data, and forming a data set; the method comprises the following steps: constructing a dual-path Transform network model DPTNet, and introducing a recurrent neural network to optimize the dual-path Transform network model DPTNet; and training the dual-path network model DPTNet, and performing engineering deployment based on the trained model. The method is beneficial to obtaining higher-quality audio fingerprint recognition capability, sound source separation capability and voice enhancement function, can be used for tracking and positioning the position of a sound source, helps positioning and tracking related applications, can be expanded to the medical field, can be used for heart sound segmentation, namely, recognition of specific signals of the heart, and can be applied to the field of medical science. The method helps to diagnose cardiovascular and other medical problems, and has technical innovation and practical application value.

Description

Technical Field

[0001] This invention belongs to the field of multi-person voice separation technology, and particularly relates to a multi-person voice separation method based on a lightweight dual-path Transformer network. Background Technology

[0002] With the development of deep learning, audio algorithms have seen explosive growth in recent years, finding applications in fields such as smart healthcare, speech recognition, and sound source localization, with continuous improvements in efficiency and speed. From cloud services to edge computing on PCs, and now to mobile computing, the field has evolved significantly. Despite increasingly powerful hardware, lightweight models remain a core pursuit in algorithm design. Only in this way can AI algorithms be accessed anytime, anywhere, to unleash their full potential.

[0003] However, existing audio multi-voice separation technologies have the following drawbacks: traditional methods (such as NMF and CASA) rely on the Close-set assumption and cannot handle unseen combinations of sound sources. Deep learning methods generally rely on GPU acceleration, making it difficult to achieve real-time processing on mobile devices. Existing models suffer from technical bottlenecks such as high computational complexity (>100MB) and long inference latency (>30 seconds / minute of audio). The lack of long-sequence speech modeling capabilities and context-aware mechanisms leads to limited separation accuracy and deployment challenges in scenarios with limited hardware resources. Summary of the Invention

[0004] Purpose of the Invention: The purpose of this invention is to provide a multi-voice separation method based on a lightweight dual-path Transformer network. This achieves a more lightweight audio multi-voice separation method with stronger speech recognition capabilities.

[0005] Technical solution: The present invention provides a multi-person voice separation method based on a lightweight dual-path Transformer network, comprising the following steps:

[0006] Step 1: Collect multi-person audio data, preprocess it to form a dataset;

[0007] Step 2: Using the dataset as input, the decoder reconstructs the source waveform as output by transforming the mask features, and constructs a dual-path Transformer network model DPTNet, which includes three modules: scaled dot product attention, multi-head attention, and position feedforward network.

[0008] Step 3: Introduce a recurrent neural network to optimize the dual-path Transformer network model DPTNet;

[0009] Step 4: Train the dual-path Transformer network model DPTNet to obtain the trained model.

[0010] Step 5: Deploy the trained model in an engineering manner.

[0011] Furthermore, step 1 specifically involves dividing the data preprocessing into four parts: generating JSON files, sorting by filename, loading data, and dividing the dataset. The entire JSON file is sorted by audio ID, and the dataset is divided into training, validation, and test sets according to the proportions.

[0012] Furthermore, step 2 specifically includes the following steps:

[0013] Step 2.1: Use the encoder to segment the mixed waveform into corresponding features in the intermediate feature space;

[0014] x∈R 1×T Representing speech mixture, it is divided into overlapping vectors x∈R of length L samples. L×I Where I is the number of vectors, W represents the transformation matrix in the encoder, and the encoder receives X and outputs the speech signal x∈R. N×I as follows:

[0015] X = ReLU(x*W)

[0016] Step 2.2: Construct a separation layer using a dual-path transformer network. In the partitioning stage, X is divided into overlapping blocks of length K and step size H. Then, all blocks are connected into a 3-D tensor D∈R. N×K×P ;

[0017] Step 2.3: Processed through a dual-path transformer, including three core modules: scaled dot product attention, multi-head attention, and position feedforward network;

[0018] Step 2.4: After overlapping and adding, the decoder reconstructs the source waveform by transforming the masking features.

[0019] Furthermore, in step 2.3, the scaled dot product attention specifically involves: scaled dot product attention is a self-attention mechanism that associates different positions in the input sequence to calculate the representation of the input. The final output of scaled dot product attention is calculated as a weighted sum of values, where the weight of each value is calculated by the queried attention function and the corresponding key, specifically including the following steps:

[0020] Step 2.3.1: Multiply the one-hot encoding of each word in the input sequence by the word embedding matrix to obtain the word vector, denoted as X. The number of rows in X represents the sentence length, and each row is the mathematical expression of each word vector. To learn expressions with multiple meanings, three weights W are assigned to X. Q W K W VAfter linear mapping, three matrices are formed: Query, Key, and Value, denoted as Q, K, and V respectively, where X = [x1, x2, ... x...]. n ];

[0021]

[0022] Q i ,K i V i ∈R l×d / h It is a mapping of queries, keys, and values; Parameter matrix;

[0023] Step 2.3.2: Calculate the dot product of Q and the transpose of K to obtain the attention matrix; the dot product is used to measure the similarity between two vectors, the more similar the vectors, the larger the dot product. Each row of the resulting attention matrix represents the degree of association between the current character and other characters; divide the attention matrix by... It is then transformed back into a standard normal distribution; then softmax normalization is performed along the column dimension so that the sum of each row is 1;

[0024] Step 2.3.3: Use the attention matrix to weight V. Multiply the columns of matrix V by the rows of the attention matrix in turn. Each row of matrix V represents the mathematical expression of each character vector. Use the attention weights to perform a weighted linear combination of these mathematical expressions so that each character vector contains the information of all character vectors in the current sentence.

[0025]

[0026] Furthermore, in step 2.3, the multi-head attention consists of multiple scaled dot product attention modules. The multi-head attention uses different, learnable linear projections to perform h linear mappings on the input to obtain parallel queries, keys, and values ​​respectively. Then, scaled dot product attention is performed simultaneously on the mapped queries, keys, and values.

[0027] The vectors after self-attention are concatenated to obtain the vector representation of the character. Each line represents the vector representation of one character. The multi-head self-attention mechanism is as follows:

[0028] MultiHead=Concat(head1,...,head h W O

[0029] W O ∈R d×d It is a parameter matrix.

[0030] Furthermore, in step 2.3, the location feedforward network consists of two linear transformations with a ReLU activation in between, and the calculation process is as follows:

[0031] Mid = LayerNorm(Z + MultiHead)

[0032] FFN = ReLU(MidW2 + b2)W2 + b2

[0033] Output = LayerNorm(Mid + FFN)

[0034] Where Z∈R I×d It is an input of length l and dimension d, and FFN is a feedforward network that outputs based on position. b2∈R d ,d ff = 4 × d.

[0035] Furthermore, step 2.4 specifically involves: inter-transformer The output is used to learn a mask for each source through 2-D convolutional layers; the mask is then transformed back into a sequence M by overlapping and adding. s ∈R N×I The mask encoder features of the s-th source are obtained by element-wise multiplication between X and Ms:

[0036] Y S =X·M S

[0037] The decoder reconstructs the source waveform by transforming masking features. In the decoder, a transposed convolution module is used to reconstruct the separated speech signal Y for the s-th source. S ∈R L×I ;

[0038] y S =Y S *V

[0039] Where V∈R N×L The value in is the parameter of the transposed convolution module, and then the overlapping addition method is applied to obtain the final waveform y. S ∈R 1×T The structure and function of the decoder are symmetrical to those of the encoder.

[0040] Furthermore, step 4 specifically involves replacing the first fully connected layer in the feedforward network with a recurrent neural network. During the two-path transformer processing stage, the output D from the segmentation stage is passed to B two-path transformer DPT stacks. Each DPT consists of intra-transformer and inter-transformer components, which respectively model local and global information. The Intra-transformer processing block first independently models the local blocks, acting on the second dimension of D.

[0041]

[0042] Use the inter-transformer to aggregate information from all blocks to learn global dependencies and execute on the last dimension of D:

[0043]

[0044] Where b = 1, ..., B and Layer normalization in each sub-transformer is applied across all dimensions.

[0045] Furthermore, step 5 specifically involves: the designed multi-voice audio separation algorithm implements three functions: single voice + noise separation, dual voice separation, and dual voice + noise separation; using JIT technology, the PyTorch model is exported as a C++ callable model, and tracing is used to convert the PyTorch model into a Torch script, then Libtorch is used to load the serialized PyTorch model in C++; finally, a minimal application is built using CMake and LibTorch to implement its functions; the data to be separated is passed to the executable program to provide the user with the separated audio.

[0046] Beneficial Effects: Compared with existing technologies, this invention has the following significant advantages: This invention employs a dual-path Transformer network (DPTNet) and introduces a recurrent neural network to optimize sequence modeling. It can learn sequential information without positional encoding, improving the processing capability for long sequences. This invention is beneficial for obtaining higher-quality audio fingerprint recognition capabilities, sound source separation capabilities, and speech enhancement functions. It can be used for sound source location tracking and localization, aiding in related applications and can be extended to the medical field, such as heart sound segmentation, i.e., identifying specific cardiac signals to help diagnose cardiovascular and other medical problems. It possesses both technological innovation and practical application value. This invention uses DPTNet as the model for this multi-voice audio separation algorithm. It also uses two commonly used multi-voice audio separation models, DPRNN and Conv-TasNet-gLN, for comparative experiments. The model accuracy format is FP 32. DPTNet's SDR reaches 16.05, significantly outperforming the two comparative models, and its inference speed and model size are also ideal. Attached Figure Description

[0047] Figure 1 This is an overall flowchart of the present invention;

[0048] Figure 2 A diagram of a two-path Transformer network for end-to-end speech separation;

[0049] Figure 3 Here is a diagram of the transformer architecture;

[0050] Figure 4 Diagram of the Transformer attention mechanism;

[0051] Figure 5 Diagram of dual-path converter processing;

[0052] Figure 6 For data preprocessing graphs;

[0053] Figure 7 This is a segmentation image;

[0054] Figure 8 Diagram of dual-path converter processing;

[0055] Figure 9 Here is a diagram of the Transformer layer structure;

[0056] Figure 10 Diagram of the Transformer attention mechanism;

[0057] Figure 11 Graph showing the calculation process for Q, Z, and V;

[0058] Figure 12 A diagram illustrating the head calculation process;

[0059] Figure 13 A diagram illustrating the process of multi-head self-attention mechanism;

[0060] Figure 14 This is a location-based feedforward network diagram; Detailed Implementation

[0061] The technical solution of the present invention will be further described below with reference to the accompanying drawings.

[0062] This invention provides a method and system for multi-person voice separation based on a lightweight dual-path Transformer network. It includes:

[0063] Firstly, a core technical approach using a dual-path Transformer network. The multi-voice audio algorithm, while reading the data, also needs to perform a series of processing steps on the data; the processed results will then serve as the input data for the model. For example... Figure 1 As shown.

[0064] Secondly, a language separation model is provided. A dual-path Transformer network (DPTNet) for end-to-end speech separation is presented, which introduces direct context awareness in speech sequence modeling. By introducing an improved transformer, DPTNet can directly interact with speech sequences, thereby achieving direct context awareness of the speech sequences. Figure 2 As shown.

[0065] Thirdly, a dual-path Transformer algorithm is provided. The Transformer is a model based on an encoder-decoder structure. In this invention, the Transformer specifically refers to the encoder part, which consists of three core modules: scaled dot product attention, multi-head attention, and a position feedforward network. The speech separation model of this invention consists of three stages: an encoder, a separation layer, and a decoder. First, the encoder converts segments of the mixed waveform into corresponding features in an intermediate feature space. Then, the features are fed into the separation layer to construct a mask for each source. Finally, the decoder reconstructs the source waveform by transforming the mask features. This invention outlines the encoder and decoder and describes in detail the separation layer, namely the dual-path Transformer network of this invention. Figure 3-5 As shown.

[0066] Fourthly, the project provides an engineering deployment of the network model. To make it easier and faster for users to use the model, an offline application for this project was released using VS2022. DPTNet was used as the model for this audio multi-voice separation algorithm. Simultaneously, this invention used two commonly used audio multi-voice separation models, DPRNN and Conv-TasNet-gLN, for comparative experiments. The model accuracy format was FP32. DPTNet achieved an SDR of 16.05, significantly outperforming the two comparative models, and its inference speed and model size were also ideal.

[0067] Comparison of the effects of each model

[0068] Model SDR SI_SNR ModelSize DPTNet 16.05 15.81 3.76M DPRNN 15.6 15.0 2.6M Conv-TasNet-gLN 13.5 12.9 5.1M

[0069] Comparison of the effects of each function

[0070] Entity Category SDR SI_SNR STOI Solo 14.19 13.95 0.9013 Dual voices plus noise 14.08 13.64 0.8734 Dual voices 16.05 15.81 0.9577

[0071] This invention provides three main functions: single-voice + noise separation, multi-voice separation, and multi-voice + noise separation. It boasts high detection accuracy, with an SDR of up to 16.05, an STOI of up to 0.9377, and an SI_SNR of approximately 14, making it applicable to various complex scenarios. It employs a state-of-the-art (SOTA) model in the field of multi-voice audio separation algorithms, achieving a balance between speed and accuracy.

[0072] Example

[0073] Data preprocessing: This invention divides data preprocessing into four parts: generating JSON files, sorting by filename, loading data, and dataset partitioning, as detailed below. Figure 6As shown. Due to the large size of the dataset, this invention saves the dataset address and data length as a JSON file for easy computer reading. A JSON file excerpt is shown; the entire JSON file content is sorted by audio ID to more easily identify abnormal audio files. Data from WSJ0 is used, with 20,000 files used for training, 5,000 for validation, and 3,000 for testing. The data volume of the training, validation, and test sets is shown.

[0074] Model Training: First, an encoder is used to convert segments of the mixed waveform into corresponding features in an intermediate feature space. Then, these features are fed into a separation layer to construct a mask for each source. Finally, the decoder reconstructs the source waveform by transforming the mask features. The speech separation model of this invention consists of three stages: an encoder, a separation layer, and a decoder. First, an encoder is used to convert segments of the mixed waveform into corresponding features in an intermediate feature space. Then, these features are fed into a separation layer to construct a mask for each source. Finally, the decoder reconstructs the source waveform by transforming the mask features. This invention outlines the encoder and decoder and describes in detail the separation layer, namely the dual-path transformer network of this invention.

[0075] Encoder: First, this invention utilizes an encoder to segment the mixed waveform into corresponding features in the intermediate feature space; if this invention uses x∈R 1×T If we represent speech mixing, then this invention can divide it into an overlapping vector x∈R of length L samples. L×I , where I is the number of vectors. The encoder receives x and outputs a speech signal x∈R. N×I as follows:

[0076] X = ReLU(x*W)

[0077] The separation layer, also known as the dual-path transformer network, consists of three stages: splitting, dual-path transformer processing, and overlapping addition. It is inspired by the common dual-path network.

[0078] Partitioning: First, the partitioning stage divides X into overlapping blocks of length K and step size H. Then, all blocks are concatenated into a 3-D tensor D∈R. N×K×P .like Figure 7 As shown.

[0079] Dual-path transformer processing: Broadly speaking, a Transformer consists of an encoder and a decoder. The encoder and decoder share the same model structure, except that the decoder is used to generate a left-context-only version. To avoid confusion, the Transformer in this invention specifically refers to the encoder part, which consists of three core modules: scaled dot product attention, multi-head attention, and a position feedforward network. The dual-path transformer processing structure is as follows: Figure 8 As shown.

[0080] Each Transformer layer consists of a multi-head attention module and a feedforward network module. These two modules are each wrapped by a residual connection and perform layer normalization on the output. The structure of the Transformer layer is as follows: Figure 9 As shown.

[0081] Scaling dot product attention: This is an effective self-attention mechanism that associates different positions in the input sequence to compute the representation of the input, such as... Figure 10 As shown. The final output of this module is calculated as a weighted sum of values, where the weight of each value is calculated by the query's attention function with the corresponding key.

[0082] Multi-head attention: Consists of multiple scaled dot-product attention modules that perform h linear mappings on the input using different, learnable linear projections to obtain parallel queries, keys, and values. Scaled dot-product attention is then simultaneously applied to the queries, keys, and values ​​of these mappings.

[0083] The one-hot encoding of each word in the input sequence is multiplied by the word embedding matrix to obtain a word vector, denoted as X. The number of rows in X represents the sentence length, and each row is the mathematical expression of each word vector. To learn expressions with multiple meanings, for X (X=[x1,x2,...x...)... n Assign three weights W Q W K W V After linear mapping, three matrices are formed: Query, Key, and Value, denoted as Q, K, and V, respectively. For example... Figure 11 As shown.

[0084]

[0085] First, the attention matrix is ​​obtained by calculating the dot product of Q and the transpose of K. The dot product is usually used to measure the similarity between two vectors; the more similar the vectors, the larger the dot product. Here, each row of the attention matrix obtained in this invention represents the degree of association between the current character and other characters. The attention matrix is ​​then divided by... It then transforms back into a standard normal distribution. Next, softmax normalization is performed along the column dimensions to make the sum of each row equal to 1. Standardizing before performing softmax makes the results more stable, allowing for a more balanced gradient to be obtained during backpropagation.

[0086] Then, this invention uses an attention matrix to weight V, sequentially multiplying the columns of matrix V by the rows of the attention matrix. Each row in matrix V represents the mathematical expression of each character vector. These mathematical expressions are weighted linearly and combined using attention weights, so that each character vector contains information from all character vectors in the current sentence. The process is as follows: Figure 12 As shown.

[0087]

[0088] The head calculation process involves concatenating the vectors after self-attention to obtain the vector representation of each character, with each line representing the vector representation of one character. The multi-head self-attention mechanism works as follows: Figure 13 As shown.

[0089] MultiHead=Concat(head1,...,head h W O

[0090] Location feedforward networks: such as Figure 14 As shown, this is a fully connected feedforward network. It consists of two linear transformations with a ReLU activation in between. In addition to the three core modules, the transformer also includes several residual and normalization layers.

[0091] Residual connections and LayerNorm: The vector after self-attention is transposed to have the same dimension as the input vector Z, and then they are summed to form a residual connection, such as Z + Attention(Q,K,V). After each module's operation, the values ​​before and after the operation are summed. LayerNorm normalizes the hidden layers in the neural network to a standard normal distribution, accelerating model convergence.

[0092] Feedforward network: A feedforward network performs an activation function operation after two fully connected layers. The calculation process is as follows:

[0093] Mid = LayerNorm(Z + MultiHead)

[0094] FFN = ReLU(MidW2 + b2)W2 + b2

[0095] Output = LayerNorm(Mid + FFN)

[0096] Where Z∈R I×d It is an input of length l and dimension d, Q i ,K i V i ∈R l×d / h It is a mapping of queries, keys, and values. and WO ∈R d×d It is a parameter matrix. FFN is a feedforward network that outputs based on position. Wherein, b2∈R d ,d ff = 4 × d.

[0097] Overlapping addition: the last inter-transformer The output is used to learn a mask for each source through 2D convolutional layers. The mask is then transformed back into a sequence M by overlapping and adding. s ∈R N×I The mask encoder features of the s-th source are obtained by element-wise multiplication between X and Ms:

[0098] Y S =X·M S

[0099] The decoder reconstructs the source waveform by transforming masking features. Within the decoder, a transposed convolutional module is used to reconstruct the separated speech signal Y from the s-th source. S ∈R L×I

[0100] y S =Y S *V

[0101] Where V∈R N×L The value in is the parameter of the transposed convolution module. Then, the overlap-addition method is applied to obtain the final waveform y. S ∈R 1×T The structure and function of the decoder are symmetrical to those of the encoder.

[0102] Model Optimization: One thing missing from the converter in the model is how to utilize the sequential information in the speech sequence. The origin converter adds positional encoding to the input embedding to represent sequential information, i.e., sine and cosine functions or learned parameters. However, this invention finds that positional encoding is not suitable for two-path networks, often leading to model divergence during training. To learn the order information, this invention replaces the first fully connected layer in the feedforward network with a recurrent neural network. This invention illustrates this improved converter in the figure and applies it to the next two-path converter processing stage. In the two-path converter processing stage, the output D of the segmentation stage is passed to B two-path converter (DPT) stacks, each DPT consisting of intra-converter and inter-converter layers, which are dedicated to modeling local and global information, respectively. The Intra-transformer processing block first independently models the local blocks, which acts on the second dimension of D:

[0103]

[0104] Then, the inter-transformer is used to aggregate information from all blocks to learn global dependencies and execute on the last dimension of D:

[0105]

[0106] Where b = 1, ..., B and Please note that layer normalization in each sub-transformer applies to all dimensions.

[0107] In fact, this structure means that each element in the speech sequence interacts directly with only a few elements and with the rest through intermediate elements. This fact has a slight negative impact on direct context-aware modeling. However, the dual-path structure allows the method of this invention to effectively model extremely long speech sequences. Overall, the minor drawbacks of the dual-path structure far outweigh its benefits.

[0108] Network Model Engineering Deployment: The designed multi-voice audio separation algorithm implements three functions: single voice + noise separation, dual voice separation, and dual voice + noise separation. Using JIT technology, the PyTorch model is exported as a C++ callable model. Tracing is used to convert the PyTorch model into a Torch script, and then Libtorch is used to load the serialized PyTorch model in C++. Finally, a minimal application is built using CMake and LibTorch to implement its functionality. Passing the data to be separated into the executable program provides the user with the separated audio.

Claims

1. A method for multi-person voice separation based on a lightweight dual-path Transformer network, characterized in that, Includes the following steps: Step 1: Collect multi-person audio data, preprocess it to form a dataset; Step 2: Using the dataset as input, the decoder reconstructs the source waveform as output by transforming the mask features, and constructs a dual-path Transformer network model DPTNet, which includes three modules: scaled dot product attention, multi-head attention, and position feedforward network. Step 3: Introduce a recurrent neural network to optimize the dual-path Transformer network model DPTNet; Step 4: Train the dual-path Transformer network model DPTNet to obtain the trained model. Step 5: Deploy the trained model in an engineering manner.

2. The multi-person voice separation method based on a lightweight dual-path Transformer network according to claim 1, characterized in that, Step 1 specifically involves: dividing data preprocessing into four parts: generating JSON files, sorting by filename, loading data, and dividing the dataset; the entire JSON file is sorted by audio ID; and the dataset is divided into training set, validation set, and test set according to the proportions.

3. The multi-person voice separation method based on a lightweight dual-path Transformer network according to claim 1, characterized in that, Step 2 specifically includes the following steps: Step 2.1: Use the encoder to segment the mixed waveform into corresponding features in the intermediate feature space; x∈R 1×T Representing speech mixture, it is divided into overlapping vectors x∈R of length L samples. L×I Where I is the number of vectors, W represents the transformation matrix in the encoder, and the encoder receives X and outputs the speech signal x∈R. N×I as follows: X = ReLU(x*W) Step 2.2: Construct a separation layer using a dual-path transformer network. In the partitioning stage, X is divided into overlapping blocks of length K and step size H. Then, all blocks are connected into a 3-D tensor D∈R. N×K×P ; Step 2.3: Processed through a dual-path transformer, including three core modules: scaled dot product attention, multi-head attention, and position feedforward network; Step 2.4: After overlapping and adding, the decoder reconstructs the source waveform by transforming the masking features.

4. The multi-person voice separation method based on a lightweight dual-path Transformer network according to claim 3, characterized in that, In step 2.3, the scaled dot product attention specifically involves: scaled dot product attention is a self-attention mechanism that associates different positions in the input sequence to calculate the representation of the input. The final output of scaled dot product attention is calculated as a weighted sum of values, where the weight of each value is calculated by the queried attention function and the corresponding key. Specifically, it includes the following steps: Step 2.3.1: Multiply the one-hot encoding of each word in the input sequence by the word embedding matrix to obtain the word vector, denoted as X. The number of rows in X represents the sentence length, and each row is the mathematical expression of each word vector. To learn expressions with multiple meanings, three weights W are assigned to X. Q W K W V After linear mapping, three matrices are formed: Query, Key, and Value, denoted as Q, K, and V respectively, where X = [x1, x2, ... x...]. n ]; Q i =ZW i Q ,K i =ZW i K ,V i =ZW i V ,i∈[1,h] Q i ,K i V i ∈R l×d / h It is a mapping of queries, keys, and values; W i Q W i K W i V ∈R d×d / h Parameter matrix; Step 2.3.2: Calculate the dot product of Q and the transpose of K to obtain the attention matrix; the dot product is used to measure the similarity between two vectors, the more similar the vectors, the larger the dot product. Each row of the resulting attention matrix represents the degree of association between the current character and other characters; divide the attention matrix by... It is then transformed back into a standard normal distribution; then softmax normalization is performed along the column dimension so that the sum of each row is 1; Step 2.3.3: Use the attention matrix to weight V. Multiply the columns of matrix V by the rows of the attention matrix in turn. Each row of matrix V represents the mathematical expression of each character vector. Use the attention weights to perform a weighted linear combination of these mathematical expressions so that each character vector contains the information of all character vectors in the current sentence.

5. A multi-person voice separation method based on a lightweight dual-path Transformer network according to claim 4, characterized in that, In step 2.3, the multi-head attention consists of multiple scaled dot product attention modules. The multi-head attention uses different, learnable linear projections to perform h linear mappings on the input to obtain parallel queries, keys, and values ​​respectively. Then, scaled dot product attention is performed simultaneously on the mapped queries, keys, and values. The vectors after self-attention are concatenated to obtain the vector representation of the character. Each line represents the vector representation of one character. The multi-head self-attention mechanism is as follows: MultiHead=Concat(head1,...,head h )W O W O ∈R d×d It is a parameter matrix.

6. A multi-person voice separation method based on a lightweight dual-path Transformer network according to claim 5, characterized in that, In step 2.3, the location feedforward network consists of two linear transformations with a ReLU activation in between. The calculation process is as follows: Mid = LayerNorm(Z + MultiHead) FFN = ReLU(MidW2 + b2)W2 + b2 Output = LayerNorm(Mid + FFN) Where Z∈R I×d It is an input of length l and dimension d, and FFN is a feedforward network that outputs based on position. b2∈R d ,d ff = 4 × d.

7. A multi-person voice separation method based on a lightweight dual-path Transformer network according to claim 3, characterized in that, Step 2.4 specifically refers to: inter-transformer The output is used to learn a mask for each source through 2-D convolutional layers; The mask is converted back to sequence M by overlapping and adding. s ∈R N×I The mask encoder features of the s-th source are obtained by element-wise multiplication between X and Ms: Y S =X·M S The decoder reconstructs the source waveform by transforming masking features. In the decoder, a transposed convolution module is used to reconstruct the separated speech signal Y for the s-th source. S ∈R L×I ; and S And S *V Where V∈R N×L The value in is the parameter of the transposed convolution module, and then the overlapping addition method is applied to obtain the final waveform y. S ∈R 1 ×T The structure and function of the decoder are symmetrical to those of the encoder.

8. A multi-person voice separation method based on a lightweight dual-path Transformer network according to claim 1, characterized in that, Step 4 specifically involves replacing the first fully connected layer in the feedforward network with a recurrent neural network. During the two-path transformer processing stage, the output D from the segmentation stage is passed to B two-path transformer (DPT) stacks. Each DPT consists of intra-transformer and inter-transformer components, respectively dedicated to modeling local and global information. The Intra-transformer processing block first independently models the local blocks, acting on the second dimension of D. Use the inter-transformer to aggregate information from all blocks to learn global dependencies and execute on the last dimension of D: Where b = 1, ..., B and Layer normalization in each sub-transformer is applied across all dimensions.

9. A multi-person voice separation method based on a lightweight dual-path Transformer network according to claim 1, characterized in that, Step 5 specifically involves the following steps: The designed multi-voice audio separation algorithm implements three functions: single voice + noise separation, dual voice separation, and dual voice + noise separation. Using JIT technology, the PyTorch model is exported as a C++ callable model. Tracing is used to convert the PyTorch model into a Torch script, and then Libtorch is used to load the serialized PyTorch model in C++. Finally, a minimal application is built using CMake and LibTorch to implement its functionality. The data to be separated is passed to the executable program to provide the user with the separated audio.

Images