Speech conversion algorithm based on state space model and fusion strategy

Abstract

The invention discloses a voice conversion algorithm based on a state space model and a fusion strategy, belongs to the field of voice signal processing, and aims to solve the problems of simplification of a feature fusion strategy (such as simple feature addition), high model calculation complexity, insufficient long-distance modeling dependency capability and the like in an existing voice conversion method. The invention provides two core innovations: 1, a novel feature fusion strategy based on cross attention and a gating mechanism, and 2, an efficient encoder architecture combined with a state space model. The voice conversion method mainly comprises two stages: the first stage is a data preprocessing stage, and cutting, voice conversion and labeling are mainly performed on an original voice data set so as to provide subsequent training; the second stage is a training stage of a voice conversion algorithm, namely a core stage, and mainly comprises a data feature extraction stage, a voice synthesis encoder and a decoder. The method has good performance in the traditional voice conversion task, and the current system is expected to have better adaptive capability.

Description

TECHNICAL FIELD

[0001] The application belongs to the field of speech signal processing, and particularly relates to a speech conversion method based on a state space model and a fusion strategy. BACKGROUND

[0002] In recent years, voice conversion (VC) technology has made significant progress, evolving from early speaker-specific adaptation to the current zero-shot voice conversion paradigm. The core goal of zero-shot voice conversion is to transfer the timbre characteristics of a source speaker to any unseen target speaker while strictly preserving the semantic content and prosodic features of the original speech. Existing methods still have obvious limitations in feature decoupling ability, computational efficiency, and generated speech quality.

[0003] Representative research works such as StableVC and StreamVC reveal the bottlenecks of current technology. StableVC can achieve independent control of timbre and style, but its reliance on diffusion sampling mechanism leads to a significant decrease in inference speed, and the feature fusion only uses simple concatenation operation, which is difficult to fully model the complex correlation between cross-modal. StreamVC supports low-latency streaming processing and is suitable for real-time communication scenarios, but its causal modeling architecture severely limits the ability to capture long-distance temporal dependencies, and lacks an explicit control mechanism for style features. The common defects of these methods are that the coupling degree of timbre and style transfer is high, and they perform poorly in balancing real-time performance and speech quality.

[0004] The core challenges facing current zero-shot voice conversion can be summarized as follows: Insufficient feature fusion strategy: existing methods (such as StableVC) mostly use simple concatenation or addition operations without attention guidance, leading to interference between timbre and style information; Low efficiency of long sequence modeling: traditional RNN / CNN architectures are difficult to efficiently capture cross-sentence prosodic continuity (such as StreamVC limited by causal modeling bottlenecks);

[0005] To address the above challenges, the present application proposes two major technical breakthroughs:

[0006] Innovative feature fusion strategy: cross-attention mechanism is introduced to realize dynamic fusion between features, and cascaded gating mechanism is used for multi-scale feature selection and enhancement; Efficient encoding architecture based on state space model: a bidirectional Mamba-SSM backbone network is used to model long sequence dependencies with linear complexity. SUMMARY

[0007] To solve the problems of feature fusion strategy simplification (such as simple feature addition) and high model calculation complexity and insufficient long-distance dependence modeling capability in existing voice conversion methods, the present application proposes two core innovations: 1, a new feature fusion strategy based on cross-attention and gating mechanism 2, an efficient encoder architecture combined with a state space model

[0008] The voice conversion method described in the present application mainly consists of two stages: the first stage is the data preprocessing stage, mainly cutting, converting voice, and labeling the original voice data set to provide subsequent training. The second stage is the training stage of the voice conversion algorithm, which is the core stage, mainly consisting of a data feature extraction stage, a voice synthesis encoder, and a decoder.

[0009] The specific scheme of the present application is shown in Figure 1

[0010] Step 1: Data preprocessing

[0011] The purpose of data preprocessing is that the original voice is a long voice, which is the voice of two different dialogues, and the original voice needs to be cut and merged into the voice of the same dialogue, so the voice needs to be preprocessed.

[0012] Step 1-1: Voice cutting

[0013] Pyannote / speaker-diarization-3.1 is used to cut the long voice into two different dialogues.

[0014] Step 1-2: Voice to text model

[0015] SenseVoice is an advanced system focused on high-precision multi-modal voice analysis, demonstrating outstanding performance in voice recognition, emotion recognition, and audio event detection. Its multi-language recognition capability is based on over 400,000 hours of high-quality data training, supporting accurate transcription of more than 50 languages, with actual performance exceeding mainstream models such as Whisper; In the rich text analysis aspect, the system not only realizes high-accuracy emotion state recognition (reaching the industry SOTA level), but also detects more than 20 kinds of human-computer interaction events such as applause, laughter, and cough. Using an innovative non-autoregressive end-to-end architecture, the SenseVoice-Small model achieves extremely fast inference - only 70 milliseconds are required to process 10 seconds of audio, with efficiency improved by 15 times compared to Whisper-Large.

[0016] Step 1-3: LLM dialogue person labeling

[0017] The model of deepseek-7b is used to label the voice-to-text to identify different segments of the two dialogues in the same voice segment. ​

[0018] DeepSeek is a series of advanced large language models developed by DeepSeek, with powerful natural language understanding and generation capabilities, supporting smooth conversations, text creation, translation, and summary tasks. Its outstanding features include 128K tokens super-long context processing, efficient analysis of long documents and complex code libraries; excellent code generation and mathematical reasoning capabilities, which can assist programming and logic problem solving; and efficient reasoning optimization, which reduces computing costs while maintaining high performance. DeepSeek model supports multi-modal expansion and is optimized for search enhancement, knowledge question and answer scenarios, providing open source and API services to help developers and enterprises quickly build intelligent applications.

[0019] Step 1-4 same dialogue voice merging

[0020] Step 2 voice conversion algorithm model

[0021] The specific model diagram of the algorithm is as follows: Figure 2

[0022] Step 2-1 encoder

[0023] The original speaker's voice is first analyzed by MFCC to extract Mel-frequency spectrum features containing phonetic content and prosodic information, and at the same time, the VoiceEncoder is used to obtain deep acoustic embedding features that encode the speaker's timbre characteristics. After the fusion of these two parts of features, the MFCC features extracted separately for the target speaker are input into the double-path processing stream. Each path passes through three layers of Res2Net modules with multi-branch residual structure to enhance local feature expression, SE channel attention mechanism with dynamic recalibration feature channel weight function, and multi-head self-attention module for establishing global temporal dependency, and finally inputs the Mamba module based on the state space model SSM for sequence modeling, outputting high-order feature representations of the original speaker and the target speaker respectively. This design realizes multiple complementary advantages: MFCC ensures phonetic content fidelity, VoiceEncoder provides timbre decoupling capability; the combination of Res2NetBC and SE enhances local features and optimizes channel distribution; multi-head attention and Mamba effectively balance global context modeling and long sequence processing efficiency. Finally, the cascade processing of Res2NetBC-SE-attention-Mamba forms a feature abstraction chain from local to global, producing double-path features that retain the essence of the source voice content and incorporate target timbre characteristics.

[0024] Step 2-2 feature fusion

[0025] The encoder dual-path features in step 2-1 are cross-modal fused through a Fusion module. The core advantage of this fusion module is to efficiently realize the dynamic fusion of two kinds of features and solve the potential feature conflict problem. Its core method is to use a cross-attention mechanism composed of a Fusion module to replace simple feature splicing and establish deep interaction between content-related features and speaker-related features. This mechanism accurately captures key associations and intelligently adjusts the fusion ratio by dynamically calculating the attention weight between features, generating intermediate representations with rich semantic information and high fusion, effectively avoiding source interference. Further, the module introduces a series of multi-layer learnable gating mechanisms, Switch modules, to refine the preliminary fused features hierarchically. These modules serve as multi-scale information filters, using convolution to extract context and combining gating operations to perform key tasks: adaptively filtering out conflicts, information flow control routing key data, and feature enhancement optimizing expression. This processing effectively preserves the semantic integrity of the original speech while giving the ability to flexibly and smoothly convert features such as the target speaker's timbre.

[0026] Step 2-3 decoder

[0027] First, the fused feature results in step 2-2 are input into a Mamba layer to model long-term temporal dependencies. Then, through three layers of Res2Net convolution layers containing residual connections and multi-scale designs, deep nonlinear transformations are performed using ReLU activation to refine local spectral patterns. The refined features are further processed by a second Mamba layer to integrate temporal relationships, and finally projected to 1024 dimensions through a high-dimensional projection, and output the initial 80-dimensional Mel spectrum through a LinearNorm layer. BRIEF DESCRIPTION OF DRAWINGS

[0028] Figure 1 Overall flowchart

[0029] Figure 2 Voice conversion algorithm model diagram DETAILED DESCRIPTION

[0030] Step 1: Data preprocessing

[0031] The purpose of data preprocessing is that the original speech is a long speech of two different dialogues, which needs to be cut and merged into the same dialogue, so the speech needs to be preprocessed.

[0032] Step 1-1 speech cutting

[0033] pyannote / speaker-diarization-3.1 is used to cut the long speech into two different dialogues, and the default parameters are used.

[0034] Step 1-2 Speech-to-Text Model

[0035] After cutting, use sensevoice for speech-to-text, convert the cut segments one by one, set vad_model="fsmn-vad" voice activity detection (VAD) model, batch_size_s=60 batch size of 60 seconds of audio, merge_length_s=30 maximum merged segment length of 30 seconds.

[0036] Step 1-3 LLM performs dialog person annotation

[0037] Use the model of deepseek-7b to annotate the speech-to-text to identify different segments of the two dialog persons in the same segment of speech.

[0038] The prompt word for the large model is:

[0039] """

[0040] Now the input is a segment of dialogue text, formatted as an alternating dialogue of one question and one answer. Each round of dialogue has been labeled as 0 and 1, but it has not been specified which number corresponds to the questioner and which number corresponds to the answerer. Please analyze the content of the dialogue and output the following information:

[0041] Clearly indicate which number (0 or 1) corresponds to the questioner

[0042] Please output the results in the following format:

[0043] [Questioner Number]: 0 or 1

[0044] [Answerer Number]: 0 or 1

[0045] Note: Your judgment should be based entirely on the language features of the dialogue content (such as interrogative words, question marks, answer sentence patterns, etc.), without introducing external assumptions.

[0046] """

[0047] Step 2 Speech Conversion Algorithm Model

[0048] The specific model diagram of the algorithm is as follows: Figure 2

[0049] Step 2-1 Encoder

[0050] Step 2-1-1 Feature Extraction

[0051] The original speaker's voice is first analyzed by MFCC to extract 80-dimensional Mel-spectrum features containing phonetic content and prosodic information, and at the same time, VoiceEncoder is used to obtain 256-dimensional deep acoustic embedding features that encode the speaker's voice characteristics.

[0052] F MFCC = MFCC(v) # (1)

[0053] where: v is voice data, MFCC is mel graph extraction method, F MFCC is the result MFCC The specific formula is as follows

[0054] y(n) = x(n) - a x(n-1) # (2)

[0055] where x(n): time domain sampling value of input signal, a: pre-emphasis coefficient, y(n): pre-emphasis output signal, n: sample index of window function

[0056]

[0057] N: window length, w(n): weight value of Hamming window at index n, y(n) w(n): windowed time domain signal, cos: cosine function

[0058]

[0059] k: frequency index, X(k): complex spectrum value of the kth frequency index, j: imaginary unit,

[0060]

[0061] P(k): power spectral density of the kth frequency index, X(k): complex spectrum value of the kth frequency index, N: window length

[0062]

[0063] f: linear frequency, m: corresponding Mel frequency, log: logarithm

[0064]

[0065] H m (k): frequency response value of the mth Mel filter, P(k): power spectral density of the kth frequency index, log S(m): logarithm of filter bank energy

[0066]

[0067] M: total number of Mel filter banks, m: corresponding Mel frequency, c(n): n-order MFCC coefficient

[0068] e spk = VoiceEncoder(v) # (9)

[0069] where v is the source or target speech, VoiceEncoder is a pre-trained model in Resemblyzer library (https: / / github.com / resemble-ai / Resemblyzer)

[0070] After the concatenation of these two parts, the backbone of the encoder is composed of three Res2Net blocks, which adopts a progressive feature refinement strategy. The first block uses a larger 5x1 convolution kernel to establish macro temporal correlation and map the input to a 512-dimensional high-dimensional feature space.

[0071] F1 = Conv1D 1×1 (X) # (10)

[0072] Split F1 into s subsets along the channel dimension

[0073]

[0074] where each subset and

[0075]

[0076] For i = 1, 2,..., s-1:

[0077] Y = Concat(Y (0) ,Y (1) ,…,Y (s-1) ) # (13)

[0078] Resnet = Conv1D 1×1 (Y) # (14)

[0079] where x is the input feature, Conv1D 1×1 maps the input channel number by 1x1 convolution, k': convolution kernel size, d i : dilated rate, +: element-wise addition, realizes residual connection, Concat: concatenates all subset outputs along the channel dimension, s: the number of split subsets, which is set to 4, w: the channel number of each subset, the i-th subset after splitting, Y (i) : the output of the i-th subset, C out : the output channel number, Y is the output feature, and Resnet is the final data result of this module

[0080] Subsequently, the SE (Squeeze-and-Excitation) channel attention mechanism is introduced after the first level Res2Net feature extraction, and the features extracted by Res2Net are taken as the input of SE to accurately evaluate the channel importance using the features at this time. First, GAP(x) performs global average pooling on the input features, compresses the spatial information of each channel into a scalar, and extracts global statistical features; then, W1·GAP(x) performs dimension reduction transformation on the compressed features through the first fully connected layer, σ() applies the ReLU activation function to introduce nonlinearity, W2 restores to the original channel dimension through the second fully connected layer, and finally σ() uses the Sigmoid activation function to generate channel attention weights in the range of [0, 1]. This series of operations learns the interdependence between channels; finally, ⊙x applies the learned attention weights to the original input features through element-wise multiplication, which strengthens important channels and suppresses unimportant channels, so that the network can adaptively adjust the feature response and improve the representation ability and performance of the model.

[0081] SE(x) = σ(W2δ(W1·GAP(x)))⊙x#(15)

[0082] where x is the input feature map, GAP(x) is the global average pooling of the input feature, W1 is the weight matrix of the first fully connected layer, δ is the ReLU activation function, W2 is the weight matrix of the second fully connected layer, σ is the Sigmoid function, and ⊙ represents element-wise multiplication

[0083] The output of the above SE is taken as the input of the second and third layers of res2net, and the Res2net formulas (10)-(14) have been introduced, which all use 3x1 convolution kernel for feature extraction. This design is similar to the multi-scale feature pyramid in the visual field, which can capture time series patterns of different granularity. Each Res2Net module uses a multi-branch structure with scale=4 and parallel dilated convolution paths to process multi-time scale information, while the residual connection ensures effective gradient propagation.

[0084] The above features are put into the input of multi-head self-attention (MHSA), and the number of heads of multi-head attention is set to 4. The calculation process can be summarized as follows: first, the input tensor X (shape (B, C, T), representing batch size, feature dimension and sequence length respectively) is transposed to (B, T, C), so that the attention calculation is performed on the sequence dimension. Then, the self-attention is calculated by the multi-head attention mechanism (MultiHead), where the query (Q), key (K) and value (V) all come from the transposed input X TThe core of multi-head attention is split-head computation: each attention head first projects Q, K, V into different subspaces (by learnable weight matrices W i Q ,W i K ,W i V ), then computes scaled dot-product attention Finally, the outputs of all heads are weightedly aggregated V i V . The concatenated outputs are linearly transformed W O to get the final attention result. This result is added to the original input X T (Residual connection), then normalized by LayerNorm, and finally transposed back to shape (B, C, T). The whole process effectively captures the dependencies of sequences and stabilizes training through multi-head parallel computation, residual connection, and normalization.

[0085]

[0086] where X is the input matrix, X T is the transpose of the input matrix, MultiHead(Q, K, V) is the multi-head attention computation, LayerNorm is the layer normalization, W O : the weight matrix of the output linear transformation, Concat: concatenation along the feature dimension, W i Q ,W i K ,W i V : learnable weight matrix, head i : the attention output of the i-th head, softmax is the activation function.

[0087] The output of the multi-head attention is input to the following Mamba, where d_state = 16 (the hidden state dimension of the state space model (SSM)), d_conv = 4 (the kernel size of the local convolution), expand = 2 (the expansion factor), and the bidirectional Mamba module implements efficient bidirectional context modeling for long sequences. This module contains two independent Mamba layers: the forward layer processes the original sequence to capture the accumulation of historical information from the past to the current time; the backward layer processes the reverse copy of the sequence to capture the future information flow from the future to the current time.

[0088] First, the input features are projected to the internal dimension

[0089] x' = Linear1(x) # (17)

[0090] x1, x2 = split(x', dim=-1) # (18)

[0091] y1 = SSM(x1) # (19)

[0092] y = y1 O SiLU(x2) # (20)

[0093] Mamba = Linear2(y) # (21)

[0094] where x: input feature, Linear1: linear transformation layer, project input to higher dimension, x': projected feature, split(x', dim=-1): split x' along the last dimension (dim=-1), x1, x2: split features, used for SSM transformation and gating mechanism respectively, SSM: state space model, y1: hidden state sequence output by SSM, SiLU: activation function, y: gated feature, Linear2: linear layer, map feature back to original input dimension, Mamba: final output of the module.

[0095] The detailed formula of SSM is as follows

[0096] Parameter generation:

[0097] Delta = softplus(xW Δ ) # (22)

[0098] B = xW B # (23)

[0099] C = xW C # (24)

[0100] where: x is the input vector, W Δ : step projection weight, Delta: discretization step, softplus: activation function, W B : input matrix projection weight, W C : output matrix projection weight, B: dynamic input matrix, C: dynamic output matrix

[0101] Discretization

[0102]

[0103] where A: fixed state transition matrix, exp: matrix exponential operation, discrete state matrix, discrete input matrix, element-wise multiplication, t: current time step, Delta t : discretization step of the current time step, I: identity matrix

[0104] State update:

[0105]

[0106] y t =C t h t (28)

[0107] where h t : hidden state, y t : one-step output, C: dynamic output matrix, t: current time step, ⊙: element-wise multiplication

[0108] Output

[0109] y = [y1, y2,..., y L ] (29)

[0110] where: y: final output sequence of ssm

[0111] Step 2-2 decoder

[0112] The decoder adopts a hierarchical architecture to process the original speaker and target speaker features output by the encoder. First, the features input from the above encoder and the target speech embedding features obtained by formula (9) are fused through the Fusion module, and the detailed fusion strategy is in step 2-3. The fusion result is input into a Mamba layer to model long-term temporal dependencies and projected to 512-dimensional features, called dim_pre. Subsequently, through three layers of Res2Net convolution layers containing residual connections and multi-scale design scale=4, deep nonlinear transformation is performed using ReLU activation to refine local spectral patterns. The refined features are further processed by the second layer Mamba to further integrate the temporal relationship, and finally projected to 1024 dimensions through high-dimensional projection, and output the preliminary 80-dimensional Mel spectrum through the LinearNorm layer.

[0113] The decoder core is a three-stage design of Mamba-Resnet-Mamba, corresponding to the feature dimension transformation of 288 to 512 (dim_pre), and finally outputting 80 dimensions after 1024-dimensional projection. The first Mamba layer d_model is set to 288, which takes advantage of the linear complexity of the state space model to process long sequences and deeply understand the complex interaction of speech content and speaker information in the fused features. The middle Res2Net convolution core focuses on the depth refinement and quality improvement of local features. The second Mamba layer is responsible for integrating the refined features to ensure the global consistency and temporal coherence of the output sequence, and stabilizing the training process through the LinearNorm layer. This design fully combines the efficient long sequence modeling capability of Mamba and the strong feature extraction advantage of convolutional network, where Mamba parameters d_state=16 (state space model (SSM) hidden state dimension), d_conv=4 (local convolution kernel size), expand=2 (expansion factor), Resnet parameters kernel_size=3 (convolution kernel size), scale=4.

[0114]

[0115] where x is the input feature, Mamba is the state space model formula (17)-(29), W and b are dimension projection, ReLU is the activation function, Res2Net is the model introduced in step 2-1, and z T is the conversion.

[0116] Step 2-3 Feature Fusion Strategy

[0117] The core method is to use the cross-attention mechanism to form a Fusion module, replacing the simple feature concatenation, and establishing a deep interaction between content-related features and speaker-related features. This mechanism dynamically calculates the attention weight between features, accurately captures key associations and intelligently adjusts the fusion ratio, generating intermediate representations with rich semantic information and high fusion, effectively avoiding source interference. Further, the module introduces a series of multi-layer learnable gating mechanisms Switch modules to refine the preliminary fusion features hierarchically. These modules serve as multi-scale information filters, using convolution to extract context and combining gating operations to perform key tasks: adaptive filtering to remove conflicts, information flow control to route key data, and feature enhancement to optimize expression. This processing effectively preserves the semantic integrity of the original speech, while giving the ability to flexibly and smoothly convert features such as the target speaker's voice tone.

[0118] First, calculate the dot product of the query Q and the key K, and divide by to obtain more stable gradients. Then, the attention weights are calculated using the softmax function, and these weights are applied to the values V to obtain the fused features. This process enables the model to dynamically focus on the importance of different features.

[0119]

[0120] where x1, x2 are the input feature vectors, W q : query vector, W k : key vector, W v value vector, softmax: convert matching degree to probability distribution, scaling factor

[0121] The input x2 is passed through a multi-layer perceptron (MLP) which contains two linear layers and an activation function σ. In this way, x2 is nonlinearly transformed to extract richer features and enhance the expressive power of the model.

[0122] F2 = W2σ(W1x2 + b1) + b2 # (33)

[0123] where W1, W2: learnable weight matrices, b1, b2: learnable bias vectors, σ(): nonlinear activation function

[0124] The fused features F1 and the MLP output features F2 are concatenated to form a new feature matrix F concat . Then, the attention weights α are calculated through another linear layer, which are used for subsequent feature weighting. The introduction of the activation function σ helps the model to learn nonlinear features.

[0125] F concat = [F1; F2]

[0126] α = σ(W a F concat + b a ) # (34)

[0127] where F1, F2 are the previous features, [F1; F2]: concatenate F1, F2 along the feature dimension, W a : learnable weight matrix, b a : learnable bias term, σ(): nonlinear activation function

[0128] Here, F switch is the layer-by-layer processing of the concatenated features F concat through four Switch modules. Each Switch module extracts features through convolution and gating mechanisms and selectively retains important features. This cumulative process enables the model to effectively select and enhance feature representations.

[0129]

[0130] The formula of Switch is as follows,

[0131] X conv = ReLU(BatchNorm(Conv1d(X))) # (36)

[0132]

[0133] Y = X conv ⊙ G T # (38)

[0134] X is the input feature, where Conv1d: 1D convolution, BatchNorm: batch normalization, ReLU: activation function, W g and b g : weights and biases of the gating linear layer, sigma: sigmoid function, and: element-wise multiplication

[0135] The feature F switch processed by the Switch module is element-wise multiplied with the attention weight alpha to obtain the weighted feature. This weighted fusion ensures that the features output by the model are more concentrated and meaningful. Finally, the final output Y is generated through linear transformation W final and bias b final , and the dimension of the output feature remains the same as the input feature.

[0136] Y = W final (F switch ⊙ alpha) + b final # (39)

[0137] W final : linear transformation, and: element-wise multiplication, b final : bias

[0138] Experimental results

[0139] 1. Indicators

[0140] MCD: measures the difference between the converted speech and the target speech in the Mel-Frequency Cepstral Coefficients (MFCC) domain, the smaller the value, the closer the timbre and spectral features of the converted speech and the target speech.

[0141] MOS: the naturalness of the converted speech is evaluated using the neural MOS predictor (SSL-MOS), which is the baseline system of VoiceMOS Challenge 2022, and the score is 1-5, the higher the score, the more natural the converted speech.

[0142] Cosine: Cosine similarity measure speaker similarity of converted speech

[0143] 2. Many-to-many conversion

[0144] When you want to convert multiple source speakers to multiple target speakers, and both source speakers and target speakers appear in the training set, this type of task is called many-to-many conversion.

[0145] MCD MOS Cosine AutoVC 9.93 3.25 0.34 Our VC 8.52 3.57 0.45

[0146] 3. Zero-shot conversion

[0147] In addition to performing well in traditional voice conversion tasks, the current system is expected to have better adaptive ability. Therefore, the zero-shot conversion task is performed. That is, the target speaker is invisible in the training set, and the same sex and different sex mos in zero-shot are tested.

[0148] MOS same-sex mos oppo-sex mos AutoVC 3.08 3.15 3.05 Our VC 3.41 3.45 3.44

Claims

1. A speech conversion algorithm based on a state-space model and a fusion strategy, characterized in that: Step 1: Data Preprocessing The original audio is a long audio file containing the voices of two different speakers. The original audio needs to be segmented and merged into a single voice, which requires audio preprocessing. Step 1-1 Speech Cutting Long audio clips are segmented into two separate conversations. Step 1-2: Speech-to-Text Conversion Steps 1-3: LLM perform speaker annotation The text after speech-to-text conversion is annotated to identify different segments of the same speech between two speakers; Steps 1-4: Merging the voices of the same speaker Step 2: Speech Conversion Algorithm Step 2-1 Encoder The original speaker's speech is first processed by extracting Mel-spectral features containing phoneme content and prosodic information, and simultaneously obtaining deep acoustic embedding features that encode the speaker's timbre characteristics. After feature fusion, these two parts are processed together with the separately extracted MFCC features of the target speaker and then enter a dual-path processing flow. Each path sequentially passes through three layers: a Res2Net module that uses a multi-branch residual structure to enhance local feature representation, an SE channel attention mechanism with dynamic recalibration of feature channel weights, and a multi-head self-attention module for establishing global temporal dependencies. Finally, the input is used for sequence modeling based on the state-space model SSM, and the outputs are high-order feature representations of the original speaker and the target speaker, respectively. Step 2-2 Feature Fusion The encoder dual-path features from step 2-1 are fused across modalities using the Fusion module; a series of multi-layer learnable gating mechanisms, the Switch module, are introduced to refine the fused features hierarchically. Step 2-3 Decoder First, the fused feature results from step 2-2 are input into a Mamba layer to model long-term temporal dependencies. Then, a three-layer Res2Net convolutional layer containing residual connections and multi-scale design is used to perform deep nonlinear transformation with ReLU activation to refine local spectral patterns. The refined features are then processed by a second Mamba layer to further integrate temporal relationships. Finally, the features are projected to 1024 dimensions and output as an initial 80-dimensional Mel spectrum through a LinearNorm layer.

2. The method according to claim 1, characterized in that, Step 2, the speech conversion algorithm, is as follows: Step 2-1 Encoder Step 2-1-1 Feature Extraction The original speaker's speech was first analyzed by MFCC to extract 80-dimensional Mel-spectral features containing phoneme content and prosodic information. At the same time, 256-dimensional deep acoustic embedding features encoding the speaker's timbre characteristics were obtained by VoiceEncoder. F MFCC =MFCC(v)#(1) Where: v represents speech data, MFCC is the mel graph extraction method, and F... MFCC For the result The specific formula for MFCC is as follows: y(n)=x(n)-α·x(n-1)#(2) Where x(n): time-domain sampled value of the input signal, α: pre-emphasis coefficient, y(n): output signal after pre-emphasis, and n: sample index of the window function. N: Window length, w(n): Weight of the Hamming window at index n, y(n)·w(n): Time-domain signal after windowing, cos: Cosine function k: frequency index, X(k): complex spectral value of the k-th frequency index, j: imaginary unit. P(k): Power spectral density of the k-th frequency index, X(k): Complex spectral value of the k-th frequency index, N: Window length f: linear frequency, m: corresponding Mel-frequency, log: logarithm H m (k): Frequency response of the m-th Mel filter, P(k): Power spectral density at the k-th frequency index, logS(m): Logarithm of the filter bank energy M: Total number of Mel filter banks, m: Corresponding Mel frequency, c(n): nth order MFCC coefficient e spk =VoiceEncoder(v)#(9) Where v is the source or target speech, and VoiceEncoder is a pre-trained model from the Resemblyzer library; After these two feature sets are concatenated, the encoder's feature extraction backbone consists of three Res2Net modules, employing a progressive feature refinement strategy. The first layer uses 5×1 convolutional kernels to establish macroscopic temporal correlations and maps the input to a 512-dimensional high-dimensional feature space. F1=Conv1D 1×1 (X)#(10) Divide F1 into s subsets equally along the channel dimension. Each subset and For i = 1, 2, ..., s-1: Y=Concat(Y (0) ,AND (1) ,…,AND (s-1) )#(13) Resnet=Conv1D 1×1 (Y)#(14) Where x is the input feature, Conv1D 1×1 The input channel number is mapped using a 1×1 convolution, where k′ is the kernel size and d is the input channel number. i : Expansion rate, +: element-wise addition to achieve residual concatenation, Concat: concatenate all subset outputs along the channel dimension, s: number of subsets (scale set to 4), w: number of channels in each subset. The i-th subset after splitting, Y (i) : The output of the i-th subset, C out : Number of output channels, Y is the output feature, and ResNet is the final data result of this module. Subsequently, after the first-level Res2Net feature extraction, the SE channel attention mechanism uses the Res2Net-extracted features as input to SE to accurately evaluate channel importance. First, GAP(x) performs global average pooling on the input features, compressing the spatial information of each channel into a scalar and extracting global statistical features. Next, W1·GAP(x) performs dimensionality reduction transformation on the compressed features through the first fully connected layer, σ() introduces nonlinearity by applying the ReLU activation function, W2 restores the original channel dimension through the second fully connected layer, and finally σ() uses the Sigmoid activation function to generate channel attention weights in the range [0,1]. Finally, ⊙x applies the learned attention weights to the original input features by element-wise multiplication. SE(x)=σ(W2δ(W1·GAP(x)))⊙x#(15) Where x is the input feature map, GAP(x) is the global average pooling of the input features, W1 is the weight matrix of the first fully connected layer, δ is the ReLU activation function, W2 is the weight matrix of the second fully connected layer, σ is the Sigmoid function, and ⊙ represents element-wise multiplication. The output of the SE above is used as the input of the second and third res2net layers below. The Res2net formulas (10)-(14) have been introduced. They all use 3×1 convolution kernels for feature extraction. This design is similar to the multi-scale feature pyramid in the field of vision, which can capture temporal patterns of different granularities. Each Res2Net module uses a multi-branch structure with scale=4 and parallel dilated convolution paths to process multi-time scale information. At the same time, the gradient is effectively propagated through residual connections. The above features are fed into the input of a multi-head self-attention (MHSA) algorithm. Here, the number of heads in the MHSA is set to 4. The calculation process is as follows: First, the input tensor X has a shape of (B, C, T), representing the batch size, feature dimension, and sequence length, respectively. After transposing, it becomes (B, T, C), so that the attention calculation is performed along the sequence dimension. Next, the self-attention mechanism is used to calculate the self-attention, where the query (Q), key (K), and value (V) all come from the transposed input X. T The core of multi-head attention is separate computation: each attention head first projects Q, K, and V into different subspaces, and then uses a learnable weight matrix. Project, then calculate the scaled dot product attention. Finally, weighted aggregation WV i V The outputs of all heads are concatenated and then subjected to a linear transformation W. O The final attention result is obtained; this result is compared with the original input X. T The summation is the residual connection, followed by LayerNorm normalization, and finally transposed back to the shape of (B,C,T); the whole process involves multi-head parallel computation, residual connection, and normalization. Where X is the input matrix, X T The input matrix is ​​the transpose, MultiHead(Q,K,V) is the multi-head attention computation, LayerNorm is the layer normalization, and W is the multi-head attention computation. O : Outputs the weight matrix of the linear transformation, Concat: concatenates along the feature dimension, W i Q W i K W i V Learnable weight matrix, head i The attention output of the i-th head, where softmax is the activation function; The output of the multi-head attention is fed into the Mamba module below. Here, d_state = 16 (the hidden state dimension of the State Space Model (SSM)), d_conv = 4 (the kernel size of the local convolution), and expand factor = 2. The bidirectional Mamba module implements efficient bidirectional context modeling for long sequences. This module contains two independent Mamba layers: the forward layer processes the original sequence, capturing the historical information accumulated from the past to the present; the backward layer processes the reversed copy of the sequence, capturing the future information flow from the future to the present. First, project the input features onto the internal dimensions. x′=Linear1(x)#(17) x1,x2=split(x′,dim=-1)#(18) y1=SSM(x1)#(19) y=y1⊙SiLU(x2)#(20) Mamba=Linear2(y)#(21) Where x: input feature, Linear1: linear transformation layer, which projects the input to a higher dimension, x′: projected feature, split(x′,dim=-1): splits x′ along the last dimension (dim=-1), x1, x2: split features, used for SSM transformation and gating mechanism respectively, SSM: state space model, y1: hidden state sequence output by SSM, SiLU: activation function, y: gated feature, Linear2: linear layer, which maps features back to the original input dimension, Mamba: final output of the module; The detailed formula for SSM is as follows: Parameter generation: Δ=softplus(xW Δ )#(22) B=xW B #(23) C=xW C #(24) Where: x is the input vector, W Δ Step size projection weights, Δ: discretization step size, softplus: activation function, W B : Input matrix projection weights, W C A: Projection weights of the output matrix, B: Dynamic input matrix, C: Dynamic output matrix Discretization Where A: fixed state transition matrix, exp: matrix exponentiation. Discrete state matrix, Discrete input matrix, element-wise multiplication, t: current time step, Δ t : Discretization step size of the current time step, I: Identity matrix Status Update: y t =C t h t #(28) Where h t : Hidden state, y t : Single-step output, C: Dynamic output matrix, t: Current time step, ⊙: Element-wise multiplication Output y=[y1,y2,...,y L ]#(29) Where: y: the final output sequence of ssm.

3. The method according to claim 2, characterized in that, Step 2, the speech conversion algorithm, also includes: Step 2-2 Decoder The decoder uses a hierarchical architecture to process the original speaker and target speaker features output by the encoder. First, the feature input obtained from the encoder above and the target speech embedding features obtained by formula (9) are fused through the Fusion module. The detailed fusion strategy is in steps 2-3. The fusion result is input into a Mamba layer to model long temporal dependencies and projected onto 512-dimensional features, called dim_pre. Then, through three layers of Res2Net convolutional layers containing residual connections and multi-scale design scale=4, deep nonlinear transformation is performed using ReLU activation to refine the local spectral pattern. The refined features are then processed by the second Mamba layer to further integrate temporal relationships. Finally, they are projected to 1024 dimensions through high-dimensional projection and output as a preliminary 80-dimensional Mel spectrum through the LinearNorm layer. The core of this decoder is a three-stage design of Mamba-ResNet-Mamba, corresponding to a feature dimension transformation from 288 to 512, followed by a 1024-dimensional projection to finally output 80 dimensions. The first Mamba layer, d_model, is set to 288, leveraging the linear complexity advantage of the state-space model to process long sequences and deeply understand the complex interaction between speech content and speaker information in the fused features. The middle Res2Net convolutional core focuses on the deep refinement and quality improvement of local features. The second Mamba layer is responsible for integrating the refined features, ensuring the global consistency and temporal coherence of the output sequence, and stabilizing the training process through the LinearNorm layer. The parameters of Mamba are the hidden state dimension d_state = 16, the kernel size of the local convolution d_conv = 4, and the expansion factor expand = 2. The parameters of ResNet are the convolution kernel size kernel_size = 3 and scale = 4. Where x is the input feature, Mamba is the state-space model formula (17)-(29), W and b are dimensional projections, ReLU is the activation function, Res2Net is the model introduced in step 2-1, and z T For the transformation; Step 2-3 Feature fusion strategy First, calculate the dot product of query Q and key K, and then divide by... Scaling is performed to obtain a more stable gradient; then, the softmax function is used to calculate attention weights, and these weights are applied to the value V to obtain fused features; this F1 process allows the model to dynamically focus on the importance of different features; Where x1 and x2 are the input feature vectors, W q : Query vector, W k : Key vector, W v Value vector, softmax: transforms the matching degree into a probability distribution. scaling factor The input x2 is passed through a multilayer perceptron (MLP), which contains two linear layers and an activation function σ; in this way, x2 undergoes a nonlinear transformation. F2=W2σ(W1x2+b1)+b2#(33) Where W1 and W2 are learnable weight matrices, b1 and b2 are learnable bias vectors, and σ() is a non-linear activation function. The fused feature F1 and the MLP output feature F2 are concatenated to form a new feature matrix F. concat Then, the attention weight α is calculated through another linear layer, and this weight is used for subsequent feature weighting; the introduction of the activation function σ helps the model learn non-linear features; F concat =[F1;F2] α=σ(W a F concat +b a )#(34) Where F1 and F2 are the preceding features, [F1; F2]: concatenates F1 and F2 along the feature dimension, W a : Learnable weight matrix, b a : Learnable bias term, σ(): Non-linear activation function Here, F switch The splicing feature F is achieved through four Switch modules. concat Layer-by-layer processing; each Switch module extracts features through convolution and gating mechanisms, and selectively retains important features; The formula for Switch is as follows: X conv =ReLU(BatchNorm(Conv1d(X)))#(36) Y=X conv ⊙G T #(38) X represents the input features, where Conv1d is a 1D convolution, BatchNorm is batch normalization, ReLU is the activation function, and W is the input feature. g and b g : Weights and biases of the gated linear layer, σ: sigmoid function, ⊙: element-wise multiplication The feature F processed by the Switch module switch The weighted features are obtained by element-wise multiplication with the attention weight α; then, through linear transformation W... final and bias b final Generate the final output Y, whose dimensions are consistent with the input features; Y=W final (F switch ⊙α)+b final #(39) W final Linear transformation; ⊙: element-wise multiplication; b final : Bias.

Images