Speaker verification method and system based on dual-stream low-rank adaptive and adversarial decoupling

Abstract

The application discloses a speaker verification method and system based on double-flow low-rank adaptive and anti-decoupling, comprising: based on a pre-trained speech network, a double-flow low-rank adaptive anti-decoupling network is constructed, and the original weight parameters of the pre-trained speech network are frozen; based on a language feature extraction branch and a speaker feature extraction branch, original speech data is subjected to feature extraction respectively to obtain language features and speaker features; the language features are input into the shared discriminator to perform language classification prediction to complete language boundary anchoring; after gradient inversion processing of the speaker features, the speaker features are input into the shared discriminator which has completed language boundary anchoring to perform anti-decoupling; identity recognition is performed based on the speaker features subjected to anti-decoupling constraint; corresponding training losses are calculated respectively, and iterative updating is performed based on the training losses. The application can improve the acceptance rate of cross-language speech of the same person and the rejection rate of the same language speech of different persons, and is suitable for high-precision speaker verification in a multi-language environment.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to a speaker identification method and system based on dual-stream low-rank adaptive and adversarial decoupling. Background Technology

[0002] Currently, deep entanglement of language and speaker features is a common problem in cross-linguistic speaker identification tasks. This problem directly leads to a systematic performance degradation of the system in the aforementioned complex and difficult scenarios, severely restricting the practical application of cross-linguistic speaker identification technology. To solve the above-mentioned language-speaker feature entanglement problem, existing technologies mainly adopt adversarial training, unsupervised domain adaptation, and feature decomposition techniques. The core idea of ​​these techniques is to treat different languages ​​as different data domains, and to force the cross-linguistic feature distribution to be aligned through algorithms to eliminate differences, or to train a discriminator to identify and remove language information in speech. Some studies have also attempted to introduce fine-grained speech features such as phonemes to help separate the pronunciation content from the speaker's identity. Essentially, all of these methods forcibly mask or filter language interference items when extracting speaker features.

[0003] However, these existing solutions have significant shortcomings in practical applications and are difficult to meet the application needs of real-world scenarios: First, they severely impair the distinguishability of speaker features, which is the core defect of existing technologies. In the standard adversarial decoupling process, the "blind" discriminator used by existing methods may unintentionally over-penalize cues that are related to language but contain important speaker distinguishing features when eliminating language information, causing the model to lose key identity information while stripping away language features. Second, they pose a risk of catastrophic forgetting and lack parameter efficiency. Current cross-linguistic speaker identification systems often introduce large-scale pre-trained speech models such as WavLM and w2v-BERT, which can easily destroy the rich features originally learned by the pre-trained models, leading to catastrophic forgetting. Moreover, there is currently a lack of an adaptive mechanism in the field that can accurately decouple real language cues while preserving the generalization ability of pre-trained models in a parameter-efficient manner.

[0004] In summary, existing cross-language speaker identification technologies are still hampered by the deep entanglement of language and speaker features, and existing solutions have many shortcomings, failing to achieve accurate decoupling of identity features and language features, and making it difficult to balance system recognition performance and parameter efficiency. Summary of the Invention

[0005] Therefore, it is necessary to provide a speaker identification method and system based on dual-stream low-rank adaptive and adversarial decoupling to address the above-mentioned technical problems, so as to solve at least one of the problems existing in the prior art.

[0006] Firstly, a speaker verification method based on dual-stream low-rank adaptive and adversarial decoupling is provided, including: Based on a pre-trained speech network, a dual-stream low-rank adaptive adversarial decoupling network is constructed, and the original weight parameters of the pre-trained speech network are frozen. The dual-stream low-rank adaptive adversarial decoupling network includes a language feature extraction branch, a speaker feature extraction branch, and a shared discriminator. Based on the language feature extraction branch and the speaker feature extraction branch, feature extraction is performed on the original speech data to obtain language features and speaker features; The language features are input into the shared discriminator to perform language classification prediction, and the shared discriminator is optimized based on the language classification results to complete language boundary anchoring; After the speaker features are processed by gradient inversion, they are input into a shared discriminator that has completed language boundary anchoring, and adversarially decoupled from the language features. Identity recognition is performed based on speaker features after adversarial decoupling constraints to obtain identity recognition results; The training losses corresponding to the language features, speaker features, and speaker features after adversarial decoupling constraints are calculated respectively, and the dual-stream low-rank adaptive adversarial decoupling network is iteratively updated based on the training losses.

[0007] In one possible implementation, the step of calculating the training losses corresponding to the language features, speaker features, and speaker features after adversarial decoupling constraints, and iteratively updating the two-stream low-rank adaptive adversarial decoupling network based on the training losses includes: Based on the language category prediction results and the actual language labels, calculate the language classification loss; Based on the shared discriminator with completed language boundary anchoring, the speaker features are used to predict the language category, and the adversarial prediction result is obtained. Based on the adversarial prediction result and the real language label, the adversarial loss is calculated. Based on the identity recognition results and the real identity labels, calculate the identity recognition loss; The weight parameters of the dual-stream low-rank adaptive adversarial decoupling network are iteratively updated by integrating the language classification loss, identity recognition loss, and adversarial loss.

[0008] In one possible implementation, the iterative update of the weight parameters of the two-stream low-rank adaptive adversarial decoupling network by fusing the language classification loss, identity recognition loss, and adversarial loss includes: The weight coefficients of the language classification loss, identity recognition loss, and adversarial loss are dynamically adjusted in stages. Based on the weight coefficients of each stage, the language classification loss, identity recognition loss, and adversarial loss are weighted and summed to obtain the total loss function; Based on the total loss function, the parameter update gradient is calculated using the backpropagation algorithm; The gradient is updated according to the parameters, and the weight parameters of the two-stream low-rank adaptive adversarial decoupling network are updated until the total loss function converges or the preset number of iterations is reached, thus obtaining the trained two-stream low-rank adaptive adversarial decoupling network.

[0009] In one possible implementation, the step of performing identity recognition based on speaker features after adversarial decoupling constraints to obtain the identity recognition result includes: Based on the adversarial loss generated during the adversarial decoupling process, the corresponding adversarial loss gradient is calculated. The negative penalty gradient is obtained by inverting the sign of the adversarial loss gradient. The speaker features are processed by the negative penalty gradient to obtain the speaker features after the adversarial decoupling constraint; The speaker features after the adversarial decoupling constraint are input into the speaker classifier for identity recognition, and the identity recognition result is obtained.

[0010] In one possible implementation, the processing of the original speech data based on the dual-stream low-rank adaptive adversarial decoupling network to extract language features and speaker features includes: Feature extraction is performed on the raw speech data to obtain basic acoustic features; The basic acoustic features are respectively input into the language feature extraction branch and the speaker feature extraction branch; The language features are obtained by extracting language-related features from the basic acoustic features through the language feature extraction branch. Speaker features are obtained by extracting speaker-related features from the basic acoustic features through the speaker feature extraction branch.

[0011] In one possible implementation, the two-stream low-rank adaptive adversarial decoupling network further includes a gradient inversion layer and a speaker classifier; The output of the pre-trained speech network is connected to the input of the language feature extraction branch and the input of the speaker feature extraction branch in a one-to-one correspondence. The output of the language feature extraction branch is connected to the first input of the shared discriminator; The first output of the speaker feature extraction branch is connected to the input of the gradient inversion layer, and the output of the gradient inversion layer is connected to the second input of the shared discriminator. The second output of the speaker feature extraction branch is connected to the input of the speaker classifier.

[0012] In one possible implementation, the language feature extraction branch and the speaker feature extraction branch are independent low-rank adaptive modules, and the output terminal of the language feature extraction branch is connected to the rank of the language feature extraction branch which is lower than the rank of the speaker feature extraction branch.

[0013] Secondly, a speaker verification system based on dual-stream low-rank adaptive and adversarial decoupling is provided, including: The network construction unit is used to construct a two-stream low-rank adaptive adversarial decoupling network based on a pre-trained speech network and freeze the original weight parameters of the pre-trained speech network. The two-stream low-rank adaptive adversarial decoupling network includes a language feature extraction branch, a speaker feature extraction branch, and a shared discriminator. The feature extraction unit is used to extract features from the original speech data based on the language feature extraction branch and the speaker feature extraction branch, respectively, to obtain language features and speaker features. The language boundary anchoring unit is used to input the language features into the shared discriminator to perform language classification prediction, and optimize the shared discriminator based on the language classification results to complete the language boundary anchoring. The adversarial decoupling unit is used to input the speaker features after gradient inversion into the shared discriminator that has completed language boundary anchoring, and to perform adversarial decoupling from the language features; The identity recognition unit is used to perform identity recognition based on the speaker features after adversarial decoupling constraints, and obtain the identity recognition result. The iterative update unit is used to calculate the training loss corresponding to the language features, speaker features and speaker features after adversarial decoupling constraints, respectively, and to iteratively update the two-stream low-rank adaptive adversarial decoupling network based on the training loss.

[0014] Thirdly, a computer device is provided, including a memory, a processor, and computer-readable instructions stored in the memory and executable on the processor, wherein the processor, when executing the computer-readable instructions, implements the steps of the speaker verification method based on dual-stream low-rank adaptive and adversarial decoupling as described above.

[0015] Fourthly, a readable storage medium is provided, the readable storage medium storing computer-readable instructions, which, when executed by a processor, implement the steps of the speaker verification method based on dual-stream low-rank adaptive and adversarial decoupling as described in any of the preceding claims.

[0016] The aforementioned speaker identification method and system based on dual-stream low-rank adaptive and adversarial decoupling includes the following steps: constructing a dual-stream low-rank adaptive adversarial decoupling network based on a pre-trained speech network, and freezing the original weight parameters of the pre-trained speech network. The dual-stream low-rank adaptive adversarial decoupling network includes a language feature extraction branch and a speaker feature extraction branch. Based on the language feature extraction branch and the speaker feature extraction branch, features are extracted from the original speech data to obtain language features and speaker features, respectively. After gradient inversion, the speaker features are adversarially decoupled from the language features to obtain adversarially decoupled speaker features. Based on the language features, speaker features, and adversarially decoupled speaker features, the corresponding training loss is calculated, and the dual-stream low-rank adaptive adversarial decoupling network is iteratively updated based on the training loss. In this embodiment, by constructing language feature extraction branches and speaker feature extraction branches on the basis of a pre-trained speech network, and simultaneously freezing the original weight parameters of the pre-trained speech network, catastrophic forgetting of the model can be effectively avoided, preserving the general feature extraction capability and strong generalization performance of the pre-trained model. By extracting language features and speaker features separately through the language feature extraction branches and speaker feature extraction branches, and combining gradient inversion and adversarial decoupling mechanisms, language-related interference information mixed in the speaker features can be effectively removed, resulting in pure and stable speaker features after adversarial decoupling constraints. Furthermore, through a joint optimization method of multiple features corresponding to multiple losses, the network can accurately focus on the speaker's own biometric features during training, significantly reducing the impact of language differences on the recognition results and improving the robustness and recognition accuracy of speaker identification in cross-language scenarios. At the same time, a low-rank adaptive structure is adopted, which effectively reduces the number of parameters and computational overhead while ensuring the decoupling effect, achieving efficient and stable cross-language speaker identification. Attached Figure Description

[0017] To more clearly illustrate the technical solutions of the embodiments of this application, the drawings used in the description of the embodiments of this application will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0018] Figure 1 This is a schematic diagram of a network architecture for a speaker verification method based on dual-stream low-rank adaptive and adversarial decoupling in one embodiment of this application; Figure 2 This is a flowchart illustrating a speaker identification method based on dual-stream low-rank adaptive and adversarial decoupling in one embodiment of this application; Figure 3This is a schematic diagram of the fraction density distribution under the worst-case scenario (SS-DL vs DS-SL) in one embodiment of this application; Figure 4 This is a schematic diagram of the speaker verification system based on dual-stream low-rank adaptive and adversarial decoupling in one embodiment of this application; Figure 5 This is a schematic diagram of a computer device according to one embodiment of this application. Detailed Implementation

[0019] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0020] The speaker identification method based on dual-stream low-rank adaptive and adversarial decoupling provided in this embodiment can be applied to, for example... Figure 1 The network architecture of the dual-stream low-rank adaptive adversarial decoupling network shown employs a parameter-efficient fine-tuning framework based on dual-stream low-rank adaptive (Dual-LoRA) and anchored adversarial decoupling. While maximizing the retention of the acoustic feature extraction capabilities of the pre-trained model, it achieves precise decoupling of language features and speaker features with a minimal number of parameters, thereby improving the accuracy and robustness of speaker identification in cross-linguistic scenarios. This network is primarily deployed on top of a parameter-frozen pre-trained speech network (backbone network) and includes core components such as a pre-trained speech network, task switcher, language feature extraction branch, speaker feature extraction branch, shared discriminator, gradient inversion layer, and speaker classifier.

[0021] Specifically, the pre-trained speech network serves as the foundation of the architecture. Its original weight parameters are frozen throughout the process to prevent catastrophic forgetting and to fully utilize its pre-trained acoustic feature extraction capabilities, providing a high-quality basic acoustic representation for subsequent feature separation. A task switcher connects the pre-trained backbone network to two independent low-rank adaptive feature extraction branches (language feature extraction branch and speaker feature extraction branch), controlling their independent operating states. The language feature extraction branch inputs language features into the shared discriminator to anchor its language classification capabilities, while the speaker feature extraction branch outputs speaker features without adversarial decoupling constraints. The gradient is directly mapped to the shared discriminator through the gradient inversion layer. It is then subjected to adversarial training against the language anchoring decision boundary of the shared discriminator, and the adversarial loss is calculated. This generates a parameter update gradient with clear linguistic basis. The gradient inversion layer then negates the sign of the parameter update gradient to obtain a negative penalty gradient, which is fed back to the speaker feature extraction branch to purify the speaker features and remove linguistic interference information. Finally, the speaker classifier receives the speaker features after adversarial decoupling constraints to complete the identity recognition. The components are jointly optimized through the loss function to form a closed loop, ultimately achieving efficient and accurate speaker identification in cross-language scenarios, balancing parameter efficiency and recognition accuracy.

[0022] The pre-trained speech network consists of a multi-layer network structure (e.g., Layer 1 to Layer N) and statistical pooling layers. During training and inference, its original weight parameters remain completely frozen. Two sets of parallel low-rank adaptive branches are globally injected into each layer of the pre-trained speech network, serving as language feature extraction and speaker feature extraction branches, respectively. Low-rank matrix technology is used to provide isolated parameter spaces for language recognition and speaker identification tasks, logically avoiding mutual interference during feature extraction. The shared discriminator can employ a multi-layer perceptron projection head structure, with its input connected to both the language feature extraction branch and the gradient inversion layer. Its core function is to perform language classification, thereby anchoring and defining accurate linguistic decision boundaries for the entire network using real language variation data, providing a reliable linguistic reference for subsequent adversarial decoupling. The gradient inversion layer is connected in series between the speaker feature extraction branch and the shared discriminator, forming a complete adversarial link. During backpropagation, this gradient inversion layer multiplies the gradient returned by the shared discriminator by a negative coefficient. Since the shared discriminator has anchored accurate language classification capabilities through the language feature extraction branch, the adversarial loss gradient processed by the gradient inversion layer will be propagated back to the speaker feature extraction branch, forcing this branch to accurately remove the language-related clues it carries during the parameter update process. Ultimately, at both the physical structure and parameter space levels, the speaker identity features and language content features are completely decoupled, ensuring the purity and discriminability of the speaker features.

[0023] In one embodiment, such as Figure 2 As shown, a speaker verification method based on dual-stream low-rank adaptive and adversarial decoupling is provided, including the following steps: In step S110, a dual-stream low-rank adaptive adversarial decoupling network is constructed based on the pre-trained speech network, and the original weight parameters of the pre-trained speech network are frozen. The dual-stream low-rank adaptive adversarial decoupling network includes a language feature extraction branch, a speaker feature extraction branch, and a shared discriminator. Optionally, a basic backbone network framework is built based on a pre-trained speech network, embedding a low-rank adaptive (LoRA) module to construct a network structure containing a language feature extraction branch and a speaker feature extraction branch. The language feature extraction branch uses low-rank parameter configuration, while the speaker feature extraction branch uses relatively higher-dimensional parameter settings to ensure that the two branches operate independently and do not interfere with each other. During network construction, the weights of the pre-trained speech network are first fixed to avoid damaging its pre-trained acoustic feature extraction capabilities during subsequent fine-tuning. Then, a task switching module is deployed on this network to achieve independent control and collaborative operation of the two branches.

[0024] Specifically, the language feature extraction branch and the speaker feature extraction branch are globally injected into all network layers of the pre-trained speech network, covering the entire feature space from low-level acoustic features to high-level semantic features. This achieves deep decoupling of language information and speaker information across all layers. At the same time, the original weight parameters of the pre-trained speech network are kept frozen throughout the training process and do not participate in updates, thereby effectively avoiding catastrophic forgetting of the model and preserving the general feature extraction capability and strong generalization performance of the pre-trained model. This improves the model training efficiency while ensuring the stability and accuracy of cross-language speaker verification.

[0025] In addition, the dual-stream low-rank adaptive adversarial decoupling network may also include a shared discriminator, which may be a multilayer perceptron projection head connected to the language feature extraction branch, receiving the language features extracted by the language feature extraction branch and anchoring the language boundaries.

[0026] In step S120, feature extraction is performed on the original speech data based on the language feature extraction branch and the speaker feature extraction branch respectively to obtain language features and speaker features; Optionally, after the dual-stream low-rank adaptive adversarial decoupling network is constructed, the original speech data is input into the network for feature extraction. The language feature extraction branch focuses on language-related information in the speech, prioritizing features related to language categories for subsequent language classification and recognition. The speaker feature extraction branch focuses on capturing signals of voiceprint biometrics related to the speaker's identity, without adding unnecessary language interference information, thus achieving simultaneous and independent extraction of the two types of features.

[0027] It should be noted that both feature extraction branches in the dual-stream low-rank adaptive adversarial decoupling network adopt a low-rank adaptive (LoRA) structure, updating only a small number of parameters within each branch without requiring full fine-tuning of the pre-trained speech network. This ensures efficient parameter optimization while avoiding the destruction of the feature representation of the pre-trained model, effectively preserving its original performance. Through parallel extraction via two branches, both linguistic and speaker features can be obtained simultaneously, achieving separation of the two types of features.

[0028] In step S130, the language features are input into the shared discriminator to perform language classification prediction, and the shared discriminator is optimized based on the language classification results to complete language boundary anchoring; Optionally, the extracted language features are first input into the shared discriminator for language classification prediction, and the language classification loss is calculated based on the language classification prediction results and the real language labels using a preset loss function (such as the cross-entropy loss function). The shared discriminator is then optimized using the language classification loss, thereby anchoring and defining accurate language decision boundaries for the shared discriminator using real language variation data.

[0029] Specifically, the language boundary anchoring refers to the process of using real language features for classification supervision and parameter optimization to enable the shared discriminator to define clear and stable decision boundaries for different language categories in a high-dimensional feature space. This allows for accurate differentiation of feature vectors corresponding to different languages, avoiding overlap of different language features in the feature space.

[0030] In step S140, the speaker features are processed by gradient inversion and then input into the shared discriminator that has completed language boundary anchoring, so as to decouple them adversarially from the language features. Optionally, the extracted speaker features are input into a gradient inversion layer in the network architecture. This layer maintains the identity mapping of the input features during forward propagation, without altering the original representation of the speaker features, and only plays a role during backpropagation. Subsequently, the speaker features processed by the gradient inversion layer are input into a shared discriminator. Leveraging the language classification capabilities anchored by the shared discriminator through language features, adversarial decoupling operations are performed. Based on its own anchored language recognition capabilities, the shared discriminator accurately identifies language-related interference information in the input speaker features and outputs updated gradient parameters.

[0031] In step S150, identity recognition is performed based on the speaker features after adversarial decoupling constraints to obtain the identity recognition result; Optionally, during backpropagation, the gradient inversion layer negates the sign of the parameter update gradient output by the shared discriminator, generating a negative penalty gradient. This negative penalty gradient then acts inversely on the speaker feature extraction branch, specifically suppressing and removing linguistic interference information mixed in the speaker features. This results in speaker features that have been freed from linguistic interference and retain only the speaker's biological characteristics (such as pronunciation habits, timbre, and intonation). These are the speaker features based on adversarial decoupling constraints. They are then input into the speaker classifier for identity recognition, yielding the identification result. Compared to the original speaker features without adversarial decoupling, these features exhibit stronger discriminative power and better robustness, effectively avoiding identity recognition bias caused by feature interference from different languages.

[0032] In step S160, the training losses corresponding to the language features, speaker features, and speaker features after adversarial decoupling constraints are calculated respectively, and the dual-stream low-rank adaptive adversarial decoupling network is iteratively updated based on the training losses.

[0033] Optionally, language features are used for language category discrimination, and a language classification loss is calculated to constrain the language feature extraction branch to focus on learning language-related information. Speaker features are used for language category adversarial discrimination, and an adversarial loss is calculated. A gradient reversal mechanism is then used to further weaken the language interference components in the speaker features, resulting in speaker features after adversarial decoupling constraints. Then, speaker identity verification is performed using these adversarial decoupling constraints, and an identity recognition loss is calculated, allowing the network to focus on the speaker's stable biometric expressions. The above three types of losses are weighted and fused to obtain the total loss function. The parameters of each branch in the dual-stream low-rank adaptive adversarial decoupling network are updated using the backpropagation algorithm. While keeping the weights of the pre-trained speech backbone network frozen, only the parameters of the low-rank adaptive module are iterated until the total loss converges or reaches the preset number of iterations, thus completing network training and enabling the model to have higher recognition accuracy and robustness in cross-language scenarios.

[0034] It should be noted that when training is completed and the actual deployment and inference phase begins, auxiliary modules such as the language feature extraction branch and the shared discriminator, which are only used in the training phase, can be directly discarded. At the same time, the weight parameters of the trained speaker feature extraction branch can be directly merged into the original weights of the frozen pre-trained backbone network through matrix addition to complete weight fusion. Finally, the merged network is used to extract speaker representations and to complete the verification based on representation similarity.

[0035] This application provides a speaker identification method based on dual-stream low-rank adaptive and adversarial decoupling, comprising: constructing a dual-stream low-rank adaptive adversarial decoupling network based on a pre-trained speech network, and freezing the original weight parameters of the pre-trained speech network; the dual-stream low-rank adaptive adversarial decoupling network includes a language feature extraction branch, a speaker feature extraction branch, and a shared discriminator; extracting features from the original speech data based on the language feature extraction branch and the speaker feature extraction branch respectively to obtain language features and speaker features; inputting the language features into the shared discriminator to perform language classification prediction, and optimizing the shared discriminator based on the language classification result to complete language boundary anchoring; inputting the speaker features after gradient inversion processing into the shared discriminator that has completed language boundary anchoring, and adversarially decoupling them from the language features; performing identity recognition based on the speaker features constrained by adversarial decoupling to obtain identity recognition results; calculating the training losses corresponding to the language features, speaker features, and speaker features constrained by adversarial decoupling respectively, and iteratively updating the dual-stream low-rank adaptive adversarial decoupling network based on the training losses. In this embodiment, by constructing language feature extraction branches and speaker feature extraction branches on the basis of a pre-trained speech network, and simultaneously freezing the original weight parameters of the pre-trained speech network, catastrophic forgetting of the model can be effectively avoided, preserving the general feature extraction capability and strong generalization performance of the pre-trained model. By extracting language features and speaker features separately through the language feature extraction branches and speaker feature extraction branches, and combining gradient inversion and adversarial decoupling mechanisms, language-related interference information mixed in the speaker features can be effectively removed, resulting in pure and stable speaker features after adversarial decoupling constraints. Furthermore, through a joint optimization method of multiple features corresponding to multiple losses, the network can accurately focus on the speaker's own biometric features during training, significantly reducing the impact of language differences on the recognition results and improving the robustness and recognition accuracy of speaker identification in cross-language scenarios. At the same time, a low-rank adaptive structure is adopted, which effectively reduces the number of parameters and computational overhead while ensuring the decoupling effect, achieving efficient and stable cross-language speaker identification.

[0036] In one embodiment of this application, the step of calculating the training losses corresponding to the language features, speaker features, and speaker features after adversarial decoupling constraints, and iteratively updating the two-stream low-rank adaptive adversarial decoupling network based on the training losses, includes: Based on the language category prediction results and the actual language labels, calculate the language classification loss; Based on the shared discriminator with completed language boundary anchoring, the speaker features are used to predict the language category, and the adversarial prediction result is obtained. Based on the adversarial prediction result and the real language label, the adversarial loss is calculated. Based on the identity recognition results and the real identity labels, calculate the identity recognition loss; The weight parameters of the dual-stream low-rank adaptive adversarial decoupling network are iteratively updated by integrating the language classification loss, identity recognition loss, and adversarial loss.

[0037] Optionally, a first pass of forward propagation and language boundary anchoring is performed using a shared discriminator. After receiving cross-lingual speech data input, the system activates a language feature extraction branch, which specifically extracts language features related to the speech content. These language features are then directly input into the shared discriminator, which predicts the language category and outputs the prediction result. Subsequently, a preset loss function (such as cross-entropy loss) is used to compare the predicted language category with the real language labels corresponding to the input speech data, calculating the language classification loss. This language classification loss is used to train the shared discriminator to accurately identify the language of the current speech data, establishing a true linguistic decision boundary for the shared discriminator, thereby achieving language anchoring.

[0038] Among them, the language classification loss It can be calculated using the following formula: ; in, Indicates the first c One-hot encoding of real language tags in language-like systems; Represents language feature embedding; C Indicates the total number of language categories; c Represents a language category index; This represents the probability that the discriminator D predicts for the c-th language class.

[0039] Then, a second pass of forward propagation and feature adversarial decoupling is performed, activating the speaker feature extraction branch to extract identity-related speaker features. The speaker's characteristics On one hand, it is used for identity supervision loss; on the other hand, it is obtained through gradient inversion layer. The adversarial loss is then calculated by the shared discriminator; where the GRL maintains the identity mapping in forward propagation and propagates the adversarial loss back to the speaker branch gradient multiplied by -η in back propagation, thereby constraining the speaker branch output to gradually remove linguistic information.

[0040] Among them, combating losses It can be calculated using the following formula. ; in, GRL stands for Gradient Reversal Layer. This represents the speaker features after the gradient inversion layer, with its forward values ​​and... The difference lies in the fact that the gradient is multiplied by a negative coefficient during backpropagation; Speaker feature embedding ， Discriminator D For the c Predicted probabilities for language-like systems.

[0041] During backpropagation, the gradient inversion layer negates the gradient returned by the discriminator. Since the discriminator now has accurate language recognition capabilities, its negative gradient can force the speaker feature adaptation branch to accurately target and eliminate real language cues without harming valid identity features unrelated to language, thereby achieving adversarial decoupling of features.

[0042] Then, the speaker features processed by the inverse penalty gradient can be input into the speaker classifier, which predicts the speaker's identity and outputs the identity prediction result. Combined with the real identity label corresponding to the input speech data, the identity recognition loss is calculated using a preset loss function (such as the sub-center arcmargin loss function). This loss function, by introducing a multi-center modeling mechanism, can effectively address the extremely high intra-class variance problem caused by different pronunciation habits in cross-linguistic speech scenarios, thereby significantly reducing the system's error rate. This identity recognition loss is used to constrain the speaker feature extraction branch to focus on learning the speaker's own biometric features, thereby improving the accuracy of speaker identity recognition.

[0043] Finally, the language classification loss, identity recognition loss, and adversarial loss are fused, and the parameter update gradients of each feature extraction branch (low-rank adaptive module) are calculated using the backpropagation algorithm. While keeping the original weight parameters of the pre-trained speech backbone network frozen, only trainable parameters such as the LoRA branch, the shared discriminator, and the speaker supervision head from the training phase are updated. This loss calculation and parameter update process is repeated until the preset convergence condition is met or the preset number of iterations is reached, completing network training. This ensures that the trained network can accurately identify speakers in cross-linguistic scenarios, balancing parameter efficiency and recognition robustness. During the inference phase, the language branch and the shared discriminator are discarded, and the speaker LoRA weights are combined.

[0044] Of which, the total loss This can be expressed by the following formula: ; in, The weight hyperparameters represent the language classification loss. The weight hyperparameters represent the adversarial loss. and All can be dynamically adjusted according to actual needs.

[0045] In one embodiment of this application, the iterative update of the weight parameters of the two-stream low-rank adaptive adversarial decoupling network by fusing the language classification loss, identity recognition loss, and adversarial loss includes: The weight coefficients of the language classification loss, identity recognition loss, and adversarial loss are dynamically adjusted in stages. Based on the weight coefficients of each stage, the language classification loss, identity recognition loss, and adversarial loss are weighted and summed to obtain the total loss function; Based on the total loss function, the parameter update gradient is calculated using the backpropagation algorithm; The gradient is updated according to the parameters, and the weight parameters of the two-stream low-rank adaptive adversarial decoupling network are updated until the total loss function converges or the preset number of iterations is reached, thus obtaining the trained two-stream low-rank adaptive adversarial decoupling network.

[0046] Optionally, to avoid model instability caused by gradient conflicts in the early stages of adversarial training, and to achieve deep decoupling of language features and speaker features while balancing language classification accuracy and speaker identification precision, a phased dynamic adjustment strategy can be adopted to adjust the weight coefficients of each loss function. For example, during the initialization phase of model training, the weight coefficients can be adjusted. Set to 1.0, Setting it to 0 activates only the backpropagation of language classification loss and identity recognition loss, disabling gradient updates for adversarial loss. The core objective of this stage is to allow the language feature extraction branch and the shared discriminator to learn reliable language category decision boundaries first, establishing accurate language anchors for subsequent adversarial training, while ensuring the speaker feature extraction branch possesses basic identity discrimination capabilities, preventing the model from getting stuck in local optima during the early stages of training. During the transition phase, the weight coefficients can be adjusted... Set to 0.2, Setting the weight to 0.2, adversarial training is cautiously introduced while retaining language classification supervision. This stage achieves a smooth transition from language feature learning to feature decoupling by reducing the weight of the language classification loss and slightly increasing the weight of the adversarial loss, avoiding the damage to learned identity features caused by abrupt adversarial gradient changes. In the decoupling stage, further... Set to 0.2, Setting it to 0.5 increases the weight of the adversarial loss, enhancing the decoupling effect between language and speaker features. In this stage, while maintaining language classification supervision, the speaker feature extraction branch is driven by a high-weight adversarial loss to thoroughly remove language-related information, while retaining core identity-discriminating features, ultimately achieving robust speaker verification in cross-language scenarios.

[0047] Within each training iteration, the identity recognition loss, language classification loss, and adversarial loss are weighted and summed according to the weight coefficients of the current stage to obtain the total loss function for that stage. Then, based on the total loss function, the parameter update gradients of the speaker feature extraction branch, language feature extraction branch, and shared discriminator in the two-stream low-rank adaptive adversarial decoupling network are calculated using the backpropagation algorithm. The trainable weight parameters of the network are then iteratively updated based on the calculated parameter update gradients. This process of phased weight adjustment, total loss calculation, gradient backpropagation, and parameter update is repeated until the total loss function converges or the preset maximum number of iterations is reached, ultimately resulting in the trained two-stream low-rank adaptive adversarial decoupling network.

[0048] During the model inference stage, the language feature extraction branch and the shared discriminator module can be discarded, and only the LoRA adaptation weights of the speaker feature extraction branch can be retained and merged into the frozen pre-trained backbone network. This achieves efficient and robust speaker verification in cross-language scenarios without increasing additional computational overhead.

[0049] In one embodiment of this application, the step of performing identity recognition based on speaker features after adversarial decoupling constraints to obtain an identity recognition result includes: The gradient of the adversarial loss is calculated based on the adversarial loss generated during the adversarial decoupling process. The negative penalty gradient is obtained by inverting the sign of the adversarial loss gradient. The speaker features are purified by the negative penalty gradient to obtain the speaker features after adversarial decoupling constraints.

[0050] Optionally, the adversarial loss is calculated based on the adversarial prediction results output by the shared discriminator and the real language labels. Backpropagation is then performed based on this adversarial loss to calculate the corresponding adversarial loss gradient. Subsequently, the adversarial loss gradient is inverted using a gradient inversion layer to obtain a negative penalty gradient. Since the shared discriminator has been trained with language classification loss, it possesses accurate language recognition capabilities. The negative penalty gradient it returns forces the speaker feature extraction branch to accurately target and remove language-related interference components from the speaker features, rather than the speaker's own biometric features. Finally, the negative penalty gradient is used to constrain and clean the speaker feature extraction branch, suppressing and stripping away mixed language information from the speaker features while preserving the speaker's inherent voiceprint biometric features and their distinguishability, thus obtaining the adversarial decoupling constrained speaker features after removing language interference.

[0051] In one embodiment of this application, the processing of the original speech data based on the dual-stream low-rank adaptive adversarial decoupling network to extract language features and speaker features respectively includes: Feature extraction is performed on the raw speech data to obtain basic acoustic features; The basic acoustic features are respectively input into the language feature extraction branch and the speaker feature extraction branch; The language features are obtained by extracting language-related features from the basic acoustic features through the language feature extraction branch. Speaker features are obtained by extracting speaker-related features from the basic acoustic features through the speaker feature extraction branch.

[0052] Optionally, the acquired raw speech data undergoes preprocessing to remove ambient noise, echo interference, and invalid silence segments. Simultaneously, the speech signal is normalized, the sampling rate is unified, and frame length and frame shift are set to eliminate speech differences caused by different acquisition devices and environments, resulting in standardized speech data that meets feature extraction standards, ensuring the accuracy of subsequent feature extraction. The standardized speech data is input into a pre-trained speech network for high-dimensional basic acoustic feature extraction, generating basic acoustic features containing language information and speaker characteristics. Since the pre-trained backbone network has all model parameters frozen throughout the entire runtime, it does not perform any parameter updates, ensuring the stability and accuracy of the basic features. After basic acoustic feature extraction, these features can be input into the language feature extraction branch and the speaker feature extraction branch, respectively. The language feature extraction branch focuses on capturing language-related information from the basic acoustic features, providing support for subsequent language classification and language recognition using a shared discriminator; the speaker feature extraction branch focuses on capturing the speaker's biometric features from the basic acoustic features.

[0053] It should be noted that this dual-stream low-rank adaptive (LoRA) network includes a pre-trained speech network (backbone network) with frozen weight parameters, and two independent low-rank adaptive (LoRA) branches corresponding to the speaker task and the language task, respectively. For any frozen layer of the backbone network, its output features are calculated jointly by the original frozen weights and the low-rank incremental weights of the corresponding task. The low-rank incremental weights of the two branches are independent of each other and adopt an asymmetric rank design, with the rank of the speaker feature extraction branch being higher than that of the language feature extraction branch. For any network layer of the pre-trained speech network, the features (language features or speaker features) extracted through the low-rank adaptive (LoRA) branches can be expressed by the following formula: ; in, and Let represent the trainable low-rank matrix for task t; d Feature Dimension ； This represents the frozen weight parameters of the corresponding network layer in the pre-trained speech network; △ This represents the trainable low-rank incremental weights corresponding to task t (the speaker feature extraction branch and the language feature extraction branch each have independent Δ). ), α represents the globally shared scaling hyperparameter, used to balance the ratio of low-rank incremental weights to the original frozen weights, and to unify the training scale of the two branches; The low-rank number is specific to task t; t is the task identifier, used to distinguish between the speaker feature extraction branch and the language feature extraction branch; x is the input feature vector (basic acoustic features), which serves as the shared input for the speaker feature extraction branch and the language feature extraction branch.

[0054] like Figure 1 As shown, in one embodiment of this application, the dual-stream low-rank adaptive adversarial decoupling network further includes a gradient inversion layer and a speaker classifier; The output of the pre-trained speech network is connected one-to-one with the input of the language feature extraction branch and the input of the speaker feature extraction branch; The output of the language feature extraction branch is connected to the first input of the shared discriminator. The first output of the speaker feature extraction branch is connected to the input of the gradient inversion layer, and the output of the gradient inversion layer is connected to the second input of the shared discriminator. The second output of the speaker feature extraction branch is connected to the input of the speaker classifier.

[0055] Optionally, the dual-stream low-rank adaptive adversarial decoupling network may specifically include: a pre-trained speech network, a language feature extraction branch, a speaker feature extraction branch, a shared discriminator, a gradient inversion layer, and a speaker classifier. The pre-trained speech network employs a large-scale pre-trained speech model with frozen parameters to extract basic speech features from the input speech signal, outputting basic speech features with strong generalization capabilities. The language feature extraction branch and the speaker feature extraction branch each contain a set of independent, learnable low-rank matrices. The language feature extraction branch can be used to specifically extract language-related representations from the basic speech features, and its output is directly connected to the first input of the shared discriminator, providing accurate language supervision signals to the shared discriminator. This enables the discriminator to learn stable and reliable language category decision boundaries, providing a language anchoring basis for subsequent adversarial decoupling. The speaker feature extraction branch is used to focus on extracting speaker identity-related representations from basic speech features. Its first output is connected to the input of the gradient inversion layer, and the output of the gradient inversion layer is connected to the second input of the shared discriminator. This allows the speaker features, after gradient inversion processing, to be fed into the discriminator for adversarial learning, forcibly removing mixed linguistic information from the speaker features. Ultimately, this achieves complete decoupling of speaker identity and linguistic content at the physical and parametric levels, resulting in pure speaker features unaffected by language. The second output of the speaker feature extraction branch is connected to the input of the speaker classifier, allowing the pure speaker features to be fed into the classifier for identity discrimination and loss supervision, ensuring high discriminative power of the speaker features. Through the above structural connections and signal flow, the dual-stream low-rank adaptive adversarial decoupling network constructed in this embodiment can efficiently and accurately decouple linguistic information from speaker identity information while maintaining the capabilities of the pre-trained model. This significantly improves the stability, accuracy, and generalization ability of cross-language speaker verification, making it suitable for practical applications such as multilingual voice authentication and cross-language voiceprint recognition.

[0056] Furthermore, the dual-stream low-rank adaptive adversarial decoupling network may also include a task switching module, which can be used to independently control the feature calculation paths of the speech feature extraction branch and the speaker feature extraction branch, so that speech feature extraction and speaker feature extraction can be executed in parallel in an independent parameter space without interfering with each other, thus structurally achieving feature decoupling.

[0057] It should be noted that the language feature extraction branch and the speaker feature extraction branch are independent low-rank adaptive modules, and the output terminal of the language feature extraction branch is connected to a branch with a rank lower than that of the speaker feature extraction branch. For example, the rank of the speaker feature extraction branch is lower than that of the speaker feature extraction branch. Set to 16, the rank of the language feature extraction branch Setting it to 4, this asymmetric rank design allows the speaker feature extraction branch to have a larger feature representation capacity, ensuring the ability to distinguish speaker identity features. At the same time, it keeps the language feature extraction branch lightweight, serving only as a language anchor to provide language supervision information, without competing for computing resources with the core speaker recognition task. This improves model training efficiency while ensuring the stability and accuracy of cross-language speaker verification.

[0058] In this embodiment, a dual-stream low-rank adaptive adversarial decoupling network is constructed. This network employs independent language feature extraction branches and speaker feature extraction branches that are globally injected into all network layers and designed with asymmetric rank. Combined with a language anchoring adversarial decoupling mechanism, gradient inversion processing, and a phased dynamic loss weighted training strategy, the deep, full-level decoupling of speaker identity features and language-related features is achieved with only a minimal number of parameters, while maintaining the pre-trained speech network weights frozen throughout the process and without compromising the original general feature extraction capabilities. This effectively avoids catastrophic forgetting and feature degradation during cross-language adaptation, while precisely suppressing language interference and preserving the core speaker discrimination attributes to the greatest extent. This significantly improves the accuracy, stability, and generalization performance of speaker verification under cross-language, multilingual, unknown language, and complex acoustic conditions, greatly reducing the computational overhead of training and inference. It possesses advantages such as high efficiency, lightweight design, and strong robustness.

[0059] Furthermore, to verify the performance of the trained two-stream low-rank adaptive adversarial decoupling network (a parameter-efficient fine-tuning framework based on two-stream low-rank adaptation and anchored adversarial decoupling), this paper designs a systematic experiment to comprehensively evaluate the model's performance. The specific experimental setup is as follows: Dataset configuration: The experiment adopts a tiered dataset scheme to balance fair comparison and performance verification in the challenge. Fairness comparison datasets: All single-system analyses and ablation experiments were conducted using only publicly available datasets, including VoxBlink (VB) and VoxCeleb (VC), to ensure fairness in comparison with baseline methods.

[0060] Challenge Evaluation Dataset: Core performance validation is based on the TidyVoice Challenge dataset (TidyVoiceX). This dataset contains a training set (3,666 speakers, 262k voice recordings) and a development set (808 speakers, 60k voice recordings).

[0061] The final submitted data was pre-selected: following the open-condition rules of the challenge, the speech network was initialized using a self-developed large-scale internal multilingual corpus (approximately 18,000 hours, covering 396 languages).

[0062] Comparative experimental configurations: To verify the effectiveness of the anchored adversarial decoupling mechanism, three control configurations were set up, defined as follows: No Adversity: When training the proposed parameter-efficient fine-tuning framework based on dual-stream low-rank adaptation and anchored adversarial decoupling, let... =0. At this point, the two branches update independently with no gradient interaction. This setting can serve as the primary control group.

[0063] Standard Adversarial (Std Adv): The baseline is constructed using the standard DANN method, where the discriminator is not guided by a language feature extraction branch and achieves blind decoupling directly based on the predicted language category.

[0064] The method provided in this application adopts a parameter-efficient fine-tuning framework based on dual-stream low-rank adaptation and adversarial decoupling, wherein the shared discriminator is anchored and guided by the language feature extraction branch.

[0065] All systems are implemented based on PyTorch, with unified training parameters and augmentation strategies, as detailed below: Basic training: Fine-tuning was performed on the TidyVoice dataset using Sub-center ArcMargin loss for 3 epochs, with the loss function hyperparameter K=3; Data augmentation: The training process incorporates MUSAN noise and RIR data augmentation.

[0066] Model and optimizer adaptation: ResNet series models (SamResNet100, ResNet293): employ the SGD optimizer, with the learning rate gradually decreasing from 0.01 to 0.0001, and setting the rank of LoRA to 0.0001. =16, =4.

[0067] w2v-BERT2-Adapter-MFA model: Employs the AdamW optimizer and sets... =32, =16, loss weight ( , ).

[0068] Training strategy: The three-stage training strategy described in the method section of this application is adopted.

[0069] To further verify the performance advantages of the proposed dual-stream low-rank adaptive and anchored adversarial decoupling parameter fine-tuning framework, clarify the impact of each core component (dual-stream low-rank adaptive structure and anchored adversarial decoupling mechanism) and pre-training data on model performance, and determine the optimal adaptation scheme of the framework, a single-system analysis experiment was conducted. The specific results are shown in Table 1 below. It focuses on the performance evaluation results of the TidyVoiceX development set, using speaker recognition accuracy and equal error rate (EER) as the core evaluation indicators, and gradually shows the performance change trend under different pre-training data initialization and different model configurations. It clearly presents the gradual optimization process of the framework from the basic configuration to the optimal configuration. Here, "data" refers to the pre-training data used for initialization, which facilitates a direct comparison of the impact of different pre-training data on model performance and provides data support for the subsequent determination of the optimal scheme.

[0070] Table 1: Performance Comparison on the TidyVoice Development Kit

[0071] Table 1 above allows for further analysis of the impact of key factors on model performance, clarifying the core advantages of the proposed method. The specific analysis is as follows: Impact of the loss function (comparison between experimental groups B0 and B1): After replacing the standard ArcFace loss with Sub-centerArcMargin loss, the EER decreased from 3.07% to 2.05%, showing a significant performance improvement. This result indicates that in cross-lingual speech scenarios, the same speaker exhibits significant intra-class differences in different languages. Using multi-center modeling (Sub-centerArcMargin loss) can more accurately characterize the speaker feature distribution, effectively improving the accuracy of speaker recognition and verifying the necessity of multi-center modeling in cross-lingual scenarios.

[0072] Comparison of efficient parameter fine-tuning and full fine-tuning (Experimental groups B1 and B2): Compared to the full fine-tuning approach (B1), experimental group B2, which introduced a LoRA adaptation layer for efficient fine-tuning on top of the frozen speech network, further reduced the EER to 1.66%. This result demonstrates that, under conditions of limited target data, efficient parameter fine-tuning can effectively alleviate the catastrophic forgetting problem, quickly adapting to the target task while preserving the general features of the pre-trained model, thus balancing training efficiency and performance.

[0073] Impact of pre-training data (comparison between experimental groups B2 and B3): When only VoxBlink (VB) was used as the initial data, the system achieved a lower EER (1.57%), indicating that compared with the mixed corpus of VB and VC, VB has better domain matching with the challenge scenario and is more suitable as the initial data for pre-training of this task, providing experimental basis for the selection of pre-training data for subsequent models.

[0074] Model Scalability and Decoupling Mechanism (Comparison between Experimental Group B4 and the Method Provided in this Application): When the speech network is expanded to SamResNet100 (B4), the baseline system already achieves a strong performance of 1.25%. Building upon this, the proposed parameter-efficient fine-tuning framework based on dual-stream low-rank adaptation and anchored adversarial decoupling further reduces the EER to 0.98%. Furthermore, on the w2v-BERT2 backbone network, an optimal EER of 0.91% is achieved. These experimental results fully validate the effectiveness of the proposed anchored adversarial decoupling strategy and the good scalability of the dual-stream low-rank adaptive framework on different backbone networks, highlighting the versatility and superiority of the proposed method.

[0075] To further verify the adaptability and robustness of the proposed method in complex cross-lingual speaker recognition scenarios, four experimental scenarios were defined for combinations of same / different speakers (SS / DS) and same / different languages ​​(SL / DL), and performance analysis was conducted. The official baseline system (overall EER of 3.07%) was compared with the proposed best method (overall EER of 0.91%), as shown in Table 2 below. Here, SS represents the same speaker, DS represents different speakers, SL represents the same language, and DL represents different languages. The table uses equal error rate (EER, in %) as the evaluation metric; a lower value indicates better speaker recognition performance in the corresponding scenario.

[0076] Table 2: Comparison of EER (%) under different test scenarios

[0077] As shown in Table 2 above, the parameter-efficient fine-tuning framework based on dual-stream low-rank adaptive and anchored adversarial decoupling proposed in this paper has achieved comprehensive performance improvement in four experimental scenarios with the same / different speakers and the same / different language combinations. The scenario with the most significant performance gain is the comparison scenario between SS-DL (same speaker, different language) and DS-SL (different speakers, same language), which is recognized as the most difficult.

[0078] This scenario, representing the worst-case scenario in cross-lingual speaker recognition, places extremely high demands on the model's feature discrimination capabilities: the model must accurately accept target trials (SS-DL) where the same speaker uses different languages, while effectively rejecting non-target trials (DS-SL) where different speakers use the same language. Experimental results show that the official baseline system is easily affected by shared language features and cannot effectively distinguish between speaker features and language features, resulting in an efficiency error (EER) as high as 5.19%. The method provided in this application significantly reduces the EER in this scenario to 1.62% by structurally decoupling these two types of features. The significant performance improvement fully validates the effectiveness of the decoupling strategy in complex scenarios.

[0079] also, Figure 3 The fractional density distributions under the worst-case scenario (SS-DL vs DS-SL) are shown, by Figure 3 It can be clearly observed that there is a serious overlap between the target score distribution and the non-target score distribution in the official baseline system, making it difficult for the model to accurately distinguish between the two types of trials. The method provided in this application can significantly widen the gap between the two score distributions, achieve clearer score separation, and further demonstrate the effectiveness of the proposed anchored adversarial decoupling strategy from a visualization perspective, providing intuitive support for the excellent performance of the model in complex cross-language scenarios.

[0080] Furthermore, to verify that the method provided in this application can effectively remove linguistic information, thereby achieving efficient decoupling between speaker features and linguistic features, a detection experiment was designed for verification. The specific experimental design is as follows: On top of the frozen speaker embedding output by the w2v-BERT2 model, an additional detection classifier is trained. The core task of this classifier is to predict the linguistic label of the input speech.

[0081] The experiment employed a joint evaluation metric, simultaneously reporting the Language Identification (LID) accuracy and the Equal Error Rate (EER) for the speaker verification task. LID accuracy measures the amount of residual linguistic information in the speaker embedding; a lower accuracy indicates more thorough removal of linguistic information and better decoupling. EER measures the model's speaker verification performance; a lower value indicates better speaker recognition ability. Through joint analysis of these two metrics, the effectiveness of the proposed method in removing linguistic information and preserving speaker features is comprehensively verified. Specific experimental results are shown in Table 3 below.

[0082] Table 3: Diagnostic Probe Analysis Based on w2v-BERT2

[0083] As shown in Table 3 above, the baseline method without adversarial training (non-adversarial) still retains a large amount of linguistic information in the speaker embedding, resulting in a high LID accuracy of 72.71% and a corresponding speaker verification EER of 1.25%. This result indicates that without adversarial decoupling, linguistic information is mixed into the speaker features, interfering with speaker verification performance. After adopting the standard adversarial training method (standard adversarial), the LID accuracy drops to 55.03%, while the speaker verification EER decreases to 0.96%. This change clearly confirms that suppressing linguistic information in the speaker embedding can effectively improve speaker verification performance, validating the rationality of the core idea of ​​"removing linguistic interference and preserving speaker features". In contrast, the parameter-efficient fine-tuning framework based on dual-stream low-rank adaptation and anchored adversarial decoupling proposed in this paper achieves dual best results: the lowest LID accuracy (49.02%) and the lowest speaker verification EER (0.91%). This result fully demonstrates that, compared with standard adversarial methods, the shared discriminator designed in this paper, guided by the anchored adversarial decoupling mechanism, can more fully extract language-related information from the speaker embedding, while more effectively preserving the speaker's identity features. It achieves simultaneous optimization of language information removal and speaker verification performance, further proving the effectiveness and superiority of the decoupling strategy proposed in this paper.

[0084] To verify the cross-architecture generalization ability and wide applicability of the proposed method, and to further verify the adaptability of the anchored adversarial decoupling strategy on different speech backbone networks, ablation experiments were designed to systematically evaluate the proposed decoupling strategy on a variety of mainstream speech backbone networks.

[0085] The experiments further expanded the evaluation scope to different types of speech network architectures, covering SamResNet100, ResNet293, and the previously validated w2v-BERT2, to comprehensively verify the generality of the method. The pre-training data settings for each backbone network are as follows: ResNet293 was pre-trained using the VoxCeleb (VC) dataset, SamResNet100 was pre-trained using the VoxBlink (VB) dataset, and w2v-BERT2 was pre-trained using a mixed dataset of VoxBlink (VB) and VoxCeleb (VC). Specific experimental results are shown in Table 4 below.

[0086] Table 4: Ablation experimental results on different backbone networks

[0087] As shown in Table 4, the standard adversarial training method (standard adversarial) consistently delivers performance improvements across all speech backbone networks compared to the non-adversarial baseline method, validating the fundamental effectiveness of adversarial training in extracting linguistic information and enhancing speaker verification performance. Taking the SamResNet100 architecture as an example, the standard adversarial method reduces the model's EER from 1.25% to 1.07%, achieving significant performance optimization.

[0088] More importantly, the parameter-efficient fine-tuning framework proposed in this paper, based on dual-stream low-rank adaptation and anchored adversarial decoupling, achieved the lowest EER on all tested model architectures (SamResNet100, ResNet293, w2v-BERT2). This experimental result fully demonstrates that the anchored adversarial decoupling mechanism designed in this paper has good cross-architecture generalization ability and can effectively adapt to different types of speech backbone networks. Its performance improvement does not depend on a specific network structure design, further verifying the broad applicability and stability of the proposed method.

[0089] To verify the performance of the proposed method in a real-world competition scenario, the official test set was evaluated during the final submission stage of the TidyVoice Challenge, following the open-conditions of the competition. The specific process and settings are as follows: First, three different types of speech backbone networks were pre-trained and initialized on a large-scale internal multilingual corpus: SamResNet100, ResNet293, and w2v-BERT2. Subsequently, the proposed parameter-efficient fine-tuning framework based on dual-stream low-rank adaptation and anchored adversarial decoupling was used to specifically adapt the three pre-trained backbone networks to the challenge task scenario.

[0090] After model adaptation, the output scores of each backbone network are calibrated to improve the reliability of the results. Finally, the output scores of the three models are fused in a 1:1:1 ratio to obtain the final evaluation results on the official test set, as shown in Table 5 below. Note: In the table, "Internal" indicates that the model was pre-trained and initialized using a large-scale multilingual internal corpus.

[0091] Table 5: Final Results on the Official TidyVoice Test Set

[0092] As shown in Table 5 above, the official test set evaluation results indicate that the system after score-level fusion achieved equal error rates (EERs) of 2.43% and 2.84% on the tv26_eval-A and tv26_eval-U test sets of the TidyVoice Challenge, respectively. Compared with the official baseline system of the competition, this result represents a relative error rate reduction of over 70%, demonstrating a significant performance improvement. Based on this excellent performance, our system ultimately achieved 3rd place in the TidyVoice Challenge.

[0093] It is noteworthy that the method presented in this application maintains consistent and stable performance in both known and unknown language testing scenarios. This phenomenon fully demonstrates that the parameter-efficient fine-tuning framework based on dual-stream low-rank adaptation and anchored adversarial decoupling proposed in this paper can effectively learn general speaker feature representations with good generalization ability, avoid overfitting the model to specific language patterns, and further verify the reliability and practicality of the method in cross-lingual and unknown language scenarios.

[0094] It should be understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0095] In one embodiment, a speaker verification system based on dual-stream low-rank adaptive and adversarial decoupling is provided. This speaker verification system corresponds one-to-one with the speaker verification methods based on dual-stream low-rank adaptive and adversarial decoupling described in the above embodiments. Figure 4 As shown, the speaker verification system based on dual-stream low-rank adaptive and adversarial decoupling includes a network construction unit 10, a feature extraction unit 20, a language boundary anchoring unit 30, an adversarial decoupling unit 40, an identity recognition unit 50, and an iterative update unit 60. Detailed descriptions of each functional module are as follows: Network construction unit 10 is used to construct a two-stream low-rank adaptive adversarial decoupling network based on a pre-trained speech network and freeze the original weight parameters of the pre-trained speech network. The two-stream low-rank adaptive adversarial decoupling network includes a language feature extraction branch, a speaker feature extraction branch, and a shared discriminator. Feature extraction unit 20 is used to extract features from the original speech data based on the language feature extraction branch and the speaker feature extraction branch respectively, to obtain language features and speaker features; The language boundary anchoring unit 30 is used to input the language features into the shared discriminator to perform language classification prediction, and optimize the shared discriminator based on the language classification results to complete language boundary anchoring. The adversarial decoupling unit 40 is used to input the speaker features after gradient inversion into the shared discriminator that has completed language boundary anchoring, and to perform adversarial decoupling with the language features; The identity recognition unit 50 is used to perform identity recognition based on the speaker features after adversarial decoupling constraints, and obtain the identity recognition result. The iterative update unit 60 is used to calculate the training loss corresponding to the language features, speaker features and speaker features after adversarial decoupling constraints, respectively, and to iteratively update the two-stream low-rank adaptive adversarial decoupling network based on the training loss.

[0096] In one embodiment of this application, the iterative update unit 40 is further configured to: Based on the language category prediction results and the actual language labels, calculate the language classification loss; Based on the shared discriminator with completed language boundary anchoring, the speaker features are used to predict the language category, and the adversarial prediction result is obtained. Based on the adversarial prediction result and the real language label, the adversarial loss is calculated. Based on the identity recognition results and the real identity labels, calculate the identity recognition loss; The weight parameters of the dual-stream low-rank adaptive adversarial decoupling network are iteratively updated by integrating the language classification loss, identity recognition loss, and adversarial loss.

[0097] In one embodiment of this application, the iterative update unit 40 is further configured to: The weight coefficients of the language classification loss, identity recognition loss, and adversarial loss are dynamically adjusted in stages. Based on the weight coefficients of each stage, the language classification loss, identity recognition loss, and adversarial loss are weighted and summed to obtain the total loss function; Based on the total loss function, the parameter update gradient is calculated using the backpropagation algorithm; The gradient is updated according to the parameters, and the weight parameters of the two-stream low-rank adaptive adversarial decoupling network are updated until the total loss function converges or the preset number of iterations is reached, thus obtaining the trained two-stream low-rank adaptive adversarial decoupling network.

[0098] In one embodiment of this application, the identity recognition unit 40 is further configured to: Based on the adversarial loss generated during the adversarial decoupling process, the corresponding adversarial loss gradient is calculated. The negative penalty gradient is obtained by inverting the sign of the adversarial loss gradient. The speaker features are processed by the negative penalty gradient to obtain the speaker features after the adversarial decoupling constraint; The speaker features after the adversarial decoupling constraint are input into the speaker classifier for identity recognition, and the identity recognition result is obtained.

[0099] In one embodiment of this application, the feature extraction unit 20 is further configured to: Feature extraction is performed on the raw speech data to obtain basic acoustic features; The basic acoustic features are respectively input into the language feature extraction branch and the speaker feature extraction branch; The language features are obtained by extracting language-related features from the basic acoustic features through the language feature extraction branch. Speaker features are obtained by extracting speaker-related features from the basic acoustic features through the speaker feature extraction branch.

[0100] In one embodiment of this application, the two-stream low-rank adaptive adversarial decoupling network further includes a gradient inversion layer and a speaker classifier; The output of the pre-trained speech network is connected to the input of the language feature extraction branch and the input of the speaker feature extraction branch in a one-to-one correspondence. The output of the language feature extraction branch is connected to the first input of the shared discriminator. The first output of the speaker feature extraction branch is connected to the input of the gradient inversion layer, and the output of the gradient inversion layer is connected to the second input of the shared discriminator. The second output of the speaker feature extraction branch is connected to the input of the speaker classifier.

[0101] In one embodiment of this application, the language feature extraction branch and the speaker feature extraction branch are independent low-rank adaptive modules, and the output terminal of the language feature extraction branch is connected to the rank of the language feature extraction branch which is lower than the rank of the speaker feature extraction branch.

[0102] In this embodiment, a dual-stream low-rank adaptive adversarial decoupling network is constructed. This network employs independent language feature extraction branches and speaker feature extraction branches that are globally injected into all network layers and designed with asymmetric rank. Combined with a language anchoring adversarial decoupling mechanism, gradient inversion processing, and a phased dynamic loss weighted training strategy, the deep, full-level decoupling of speaker identity features and language-related features is achieved with only a minimal number of parameters, while maintaining the pre-trained speech network weights frozen throughout the process and without compromising the original general feature extraction capabilities. This effectively avoids catastrophic forgetting and feature degradation during cross-language adaptation, while precisely suppressing language interference and preserving the core speaker discrimination attributes to the greatest extent. This significantly improves the accuracy, stability, and generalization performance of speaker verification under cross-language, multilingual, unknown language, and complex acoustic conditions, greatly reducing the computational overhead of training and inference. It possesses advantages such as high efficiency, lightweight design, and strong robustness.

[0103] Specific limitations regarding the speaker verification system based on dual-stream low-rank adaptive and adversarial decoupling can be found in the limitations of the speaker verification method based on dual-stream low-rank adaptive and adversarial decoupling mentioned above, and will not be repeated here. Each module in the aforementioned speaker verification system based on dual-stream low-rank adaptive and adversarial decoupling can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device, or stored in the memory of a computer device as software, so that the processor can call and execute the corresponding operations of each module.

[0104] In one embodiment, a computer device is provided, which may be a terminal device, and its internal structure diagram may be as follows: Figure 5 As shown, the computer device includes a processor, memory, and network interface connected via a system bus. The processor provides computational and control capabilities. The memory includes a readable storage medium storing computer-readable instructions. The network interface communicates with external terminals via a network connection. When executed by the processor, the computer-readable instructions implement a speaker verification method based on dual-stream low-rank adaptive and adversarial decoupling. The readable storage medium provided in this embodiment includes both non-volatile and volatile readable storage media.

[0105] In this application embodiment, a computer device is provided, including a memory, a processor, and computer-readable instructions stored in the memory and executable on the processor. When the processor executes the computer-readable instructions, it implements the steps of the speaker verification method based on dual-stream low-rank adaptation and adversarial decoupling as described above.

[0106] In this embodiment of the application, a readable storage medium is provided, which stores computer-readable instructions. When the computer-readable instructions are executed by a processor, they implement the steps of the speaker verification method based on dual-stream low-rank adaptive and adversarial decoupling described above.

[0107] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by instructing related hardware with computer-readable instructions. These computer-readable instructions can be stored in a non-volatile readable storage medium or a volatile readable storage medium. When executed, these computer-readable instructions can include the processes of the embodiments of the methods described above. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in a variety of forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), RAMbus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0108] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is used as an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above.

[0109] The above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application, and should all be included within the protection scope of this application.

Claims

1. A speaker identification method based on dual-stream low-rank adaptive and adversarial decoupling, characterized in that, The method includes: Based on a pre-trained speech network, a dual-stream low-rank adaptive adversarial decoupling network is constructed, and the original weight parameters of the pre-trained speech network are frozen. The dual-stream low-rank adaptive adversarial decoupling network includes a language feature extraction branch, a speaker feature extraction branch, and a shared discriminator. Based on the language feature extraction branch and the speaker feature extraction branch, feature extraction is performed on the original speech data to obtain language features and speaker features; The language features are input into the shared discriminator to perform language classification prediction, and the shared discriminator is optimized based on the language classification results to complete language boundary anchoring; After the speaker features are processed by gradient inversion, they are input into a shared discriminator that has completed language boundary anchoring, and adversarially decoupled from the language features. Identity recognition is performed based on speaker features after adversarial decoupling constraints to obtain identity recognition results; The training losses corresponding to the language features, speaker features, and speaker features after adversarial decoupling constraints are calculated respectively, and the dual-stream low-rank adaptive adversarial decoupling network is iteratively updated based on the training losses.

2. The speaker identification method based on dual-stream low-rank adaptive and adversarial decoupling as described in claim 1, characterized in that, The step of calculating the training losses corresponding to the language features, speaker features, and speaker features after adversarial decoupling constraints, and iteratively updating the two-stream low-rank adaptive adversarial decoupling network based on the training losses, includes: Based on the language category prediction results and the actual language labels, calculate the language classification loss; Based on the shared discriminator with completed language boundary anchoring, the speaker features are used to predict the language category, and the adversarial prediction result is obtained. Based on the adversarial prediction result and the real language label, the adversarial loss is calculated. Based on the identity recognition results and the real identity labels, calculate the identity recognition loss; The weight parameters of the dual-stream low-rank adaptive adversarial decoupling network are iteratively updated by integrating the language classification loss, identity recognition loss, and adversarial loss.

3. The speaker identification method based on dual-stream low-rank adaptive and adversarial decoupling as described in claim 2, characterized in that, The process of fusing the language classification loss, identity recognition loss, and adversarial loss to iteratively update the weight parameters of the two-stream low-rank adaptive adversarial decoupling network includes: The weight coefficients of the language classification loss, identity recognition loss, and adversarial loss are dynamically adjusted in stages. Based on the weight coefficients of each stage, the language classification loss, identity recognition loss, and adversarial loss are weighted and summed to obtain the total loss function; Based on the total loss function, the parameter update gradient is calculated using the backpropagation algorithm; The gradient is updated according to the parameters, and the weight parameters of the two-stream low-rank adaptive adversarial decoupling network are updated until the total loss function converges or the preset number of iterations is reached, thus obtaining the trained two-stream low-rank adaptive adversarial decoupling network.

4. The speaker identification method based on dual-stream low-rank adaptive and adversarial decoupling as described in claim 1, characterized in that, The identity recognition based on speaker features after adversarial decoupling constraints yields the following results: Based on the adversarial loss generated during the adversarial decoupling process, the corresponding adversarial loss gradient is calculated. The negative penalty gradient is obtained by inverting the sign of the adversarial loss gradient. The speaker features are processed by the negative penalty gradient to obtain the speaker features after the adversarial decoupling constraint; The speaker features after the adversarial decoupling constraint are input into the speaker classifier for identity recognition, and the identity recognition result is obtained.

5. The speaker identification method based on dual-stream low-rank adaptive and adversarial decoupling as described in claim 1, characterized in that, The raw speech data is processed using a dual-stream low-rank adaptive adversarial decoupling network to extract language features and speaker features, including: Feature extraction is performed on the raw speech data to obtain basic acoustic features; The basic acoustic features are respectively input into the language feature extraction branch and the speaker feature extraction branch; The language features are obtained by extracting language-related features from the basic acoustic features through the language feature extraction branch. Speaker features are obtained by extracting speaker-related features from the basic acoustic features through the speaker feature extraction branch.

6. The speaker identification method based on dual-stream low-rank adaptive and adversarial decoupling as described in any one of claims 1-5, characterized in that, The dual-stream low-rank adaptive adversarial decoupling network also includes a gradient inversion layer and a speaker classifier; The output of the pre-trained speech network is connected to the input of the language feature extraction branch and the input of the speaker feature extraction branch in a one-to-one correspondence. The output of the language feature extraction branch is connected to the first input of the shared discriminator; The first output of the speaker feature extraction branch is connected to the input of the gradient inversion layer, and the output of the gradient inversion layer is connected to the second input of the shared discriminator. The second output of the speaker feature extraction branch is connected to the input of the speaker classifier.

7. The speaker identification method based on dual-stream low-rank adaptive and adversarial decoupling as described in claim 6, characterized in that, The language feature extraction branch and the speaker feature extraction branch are independent low-rank adaptive modules, and the output terminal of the language feature extraction branch is connected to the rank of the language feature extraction branch which is lower than the rank of the speaker feature extraction branch.

8. A speaker verification system based on dual-stream low-rank adaptive and adversarial decoupling, characterized in that, The system includes: The network construction unit is used to construct a two-stream low-rank adaptive adversarial decoupling network based on a pre-trained speech network and freeze the original weight parameters of the pre-trained speech network. The two-stream low-rank adaptive adversarial decoupling network includes a language feature extraction branch, a speaker feature extraction branch, and a shared discriminator. The feature extraction unit is used to extract features from the original speech data based on the language feature extraction branch and the speaker feature extraction branch, respectively, to obtain language features and speaker features. The language boundary anchoring unit is used to input the language features into the shared discriminator to perform language classification prediction, and optimize the shared discriminator based on the language classification results to complete the language boundary anchoring. The adversarial decoupling unit is used to input the speaker features after gradient inversion into the shared discriminator that has completed language boundary anchoring, and to perform adversarial decoupling from the language features; The identity recognition unit is used to perform identity recognition based on the speaker features after adversarial decoupling constraints, and obtain the identity recognition result. The iterative update unit is used to calculate the training loss corresponding to the language features, speaker features and speaker features after adversarial decoupling constraints, respectively, and to iteratively update the two-stream low-rank adaptive adversarial decoupling network based on the training loss.

9. A computer device comprising a memory, a processor, and computer-readable instructions stored in the memory and executable on the processor, characterized in that, When the processor executes the computer-readable instructions, it implements the steps of the speaker verification method based on dual-stream low-rank adaptive and adversarial decoupling as described in any one of claims 1 to 7.

10. A readable storage medium storing computer-readable instructions, characterized in that, When the computer-readable instructions are executed by the processor, they implement the steps of the speaker verification method based on dual-stream low-rank adaptive and adversarial decoupling as described in any one of claims 1 to 7.

Images