A voiceprint recognition model training method based on multi-task learning and adversarial training

Abstract

The present application discloses a method for training a voiceprint recognition model based on multi-task learning and adversarial training, and relates to the field of voiceprint technology. The method comprises: obtaining an audio sample set to be trained, wherein each audio sample in the audio sample set is annotated with a user label and a domain label; obtaining the human voiceprint features and domain features of each audio sample; filtering out the domain features from the human voiceprint features, and obtaining the filtered human voiceprint features as the human voice universal features; training the adversarial network based on the human voice universal features, user labels, and domain labels of each audio sample until the adversarial network meets the preset conditions, thereby obtaining a trained voiceprint recognition model. In this way, the domain adversarial training method is adopted to enhance the robustness and generalization ability of the trained voiceprint recognition model in different environments, thereby improving the recognition accuracy of the voiceprint recognition model in the face of environmental noise and improving the user experience.

Description

Technical Field

[0001] The present application relates to the field of voiceprint technology, and more specifically, to a voiceprint recognition model training method based on multi-task learning and adversarial training. Background Art

[0002] Voiceprint recognition is a biometric technology that uses speech to identify the speaker. Voiceprint recognition technology is used in many scenarios, such as telephones, mobile phones, applications, access control systems, smart speakers, robots, etc.

[0003] In related technologies, the speaker's speech is generally input into a pre-trained voiceprint recognition model, which then identifies the speaker. However, in actual application scenarios, external factors such as ambient noise may affect the voiceprint recognition model, resulting in a lower recognition accuracy and a reduced user experience. Summary of the Invention

[0004] In view of this, this application proposes a voiceprint recognition model training method based on multi-task learning and adversarial training.

[0005] In a first aspect, an embodiment of the present application provides a voiceprint recognition model training method based on multi-task learning and adversarial training, the method comprising: obtaining an audio sample set to be trained, each audio sample in the audio sample set being annotated with a user label and a domain label, the user label being used to characterize the identity of the speaker in the audio sample, and the domain label being used to characterize the scene type of the scene in which the speaker is located in the audio sample; obtaining the human voiceprint features and domain features of each audio sample, the domain features being used to characterize the scene features of the scene in which the speaker is located in each audio sample; filtering out the domain features from the human voiceprint features to obtain the filtered human voiceprint features as universal human voice features; training an adversarial network based on the universal human voice features, the user labels, and the domain labels of each audio sample until the adversarial network meets preset conditions, thereby obtaining a trained voiceprint recognition model.

[0006] In the solution provided by this application, a set of audio samples to be trained is obtained, wherein each audio sample in the audio sample set is annotated with a user label and a domain label; the human voiceprint features and domain features of each audio sample are obtained; the domain features are filtered out from the human voiceprint features to obtain the filtered human voiceprint features as human voice universal features; based on the human voice universal features, user labels, and domain labels of each audio sample, an adversarial network is trained until the adversarial network meets preset conditions, thereby obtaining a trained voiceprint recognition model. In this way, in addition to the original user labels, domain labels are added during the model training phase. While learning the human voiceprint features of the speaker, domain features are also learned, i.e., domain adversarial training is adopted, which enhances the robustness and generalization ability of the voiceprint recognition model; furthermore, domain features are filtered out from the human voiceprint features to obtain human voice universal features. Since human voice universal features are not related to the domain, the robustness of the trained voiceprint recognition model in different environments can be further improved, thereby improving the recognition accuracy of the voiceprint recognition model in the face of environmental noise and improving the user experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0007] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For those skilled in the art, other drawings can be obtained based on these drawings without creative work.

[0008] Figure 1 A flow chart of a voiceprint recognition model training method based on multi-task learning and adversarial training provided in an embodiment of the present application is shown.

[0009] Figure 2 A flow chart of a voiceprint recognition model training method based on multi-task learning and adversarial training provided in another embodiment of the present application is shown.

[0010] Figure 3 Shown Figure 2 Schematic diagram of the sub-step flow chart of step S260 in one embodiment.

[0011] Figure 4 Shown Figure 3 Schematic diagram of the sub-step flow chart of step S264 in one embodiment.

[0012] Figure 5 A flow chart of a voiceprint recognition model training method based on multi-task learning and adversarial training provided in another embodiment of the present application is shown.

[0013] Figure 6A flowchart of a voiceprint recognition model training method based on multi-task learning and adversarial training provided in yet another embodiment of the present application is shown.

[0014] Figure 7 This is a block diagram of a voiceprint recognition model training device based on multi-task learning and adversarial training provided according to an embodiment of the present application.

[0015] Figure 8 It is a block diagram of an electronic device for executing a voiceprint recognition model training method based on multi-task learning and adversarial training according to an embodiment of the present application.

[0016] Figure 9 It is a storage unit for storing or carrying program codes for implementing the voiceprint recognition model training method based on multi-task learning and adversarial training according to an embodiment of the present application. DETAILED DESCRIPTION

[0017] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application.

[0018] Voiceprint recognition is a biometric technology that uses speech to identify the speaker. Voiceprint recognition technology is used in many scenarios, such as telephones, mobile phones, applications, access control systems, smart speakers, robots, etc.

[0019] In related technologies, the speaker's speech is generally input into a pre-trained voiceprint recognition model, which then identifies the speaker. However, in actual application scenarios, external factors such as ambient noise may affect the voiceprint recognition model, resulting in a lower recognition accuracy and a reduced user experience.

[0020] To address these issues, the inventors proposed a method for training a voiceprint recognition model based on multi-task learning and adversarial training. This method iteratively trains an adversarial network based on universal voice features, user labels, and domain labels obtained after filtering out domain features, until the adversarial network meets preset conditions, resulting in a trained voiceprint recognition model. This method is described in detail below.

[0021] Please refer to Figure 1 , Figure 1 A flow chart of a voiceprint recognition model training method based on multi-task learning and adversarial training is provided in an embodiment of the present application. Figure 1 The voiceprint recognition model training method based on multi-task learning and adversarial training provided in the embodiment of the present application is described in detail. The voiceprint recognition model training method based on multi-task learning and adversarial training may include the following steps:

[0022] Step S110: Obtain an audio sample set to be trained, wherein each audio sample in the audio sample set is annotated with a user label and a domain label, wherein the user label is used to characterize the identity of the speaker in the audio sample, and the domain label is used to characterize the scene type of the scene in which the speaker in the audio sample is located.

[0023] In this embodiment, during the original model training process, the model input is speech and the output is the speaker (i.e., user label). In other words, the labels guiding model learning only include known user labels. However, in actual applications, the audio to be recognized may contain many factors that are different from the audio samples used in the training phase. In other words, there are differences between the audio to be recognized and the training samples. For example, the audio to be recognized may contain environmental noise, or the device used to collect the audio to be recognized may be different from the device used to collect the audio samples, resulting in channel mismatch, which in turn causes a decrease in the recognition accuracy of the trained voiceprint recognition model.

[0024] Based on this, in order to make the voiceprint recognition model maintain good adaptability in the face of different application environments, audio from multiple fields can be used for model training. Specifically, audio containing multiple fields is obtained as the above-mentioned audio sample to be trained, wherein each audio sample in the audio sample set is marked with a user label and a domain label, the user label is used to characterize the identity of the speaker in the audio sample, and the domain label is used for the scene type of the scene where the speaker is in the audio sample. The scene type can include multiple types, for example, a quiet room, a street cafe, a noisy street, etc., which is not limited in this embodiment. In this way, the trained voiceprint recognition model can maintain a stable recognition effect in different recording environments, and improve the recognition accuracy of the voiceprint recognition model when facing noise in different environments. Among them, the user label and the domain label can be marked when the audio sample is collected.

[0025] In some implementations, the domain label can characterize the scene type of the speaker in the audio sample, as well as the audio acquisition channel. For example, an audio sample acquired using a handheld microphone in a quiet room belongs to one domain, while an audio sample acquired using a computer in a street cafe belongs to another domain. In this way, the domain label can encompass both the recording environment and the audio acquisition channel, allowing the trained voiceprint recognition model to maintain stable recognition results in different application environments and improve the recognition accuracy of the voiceprint recognition model in the presence of noise and different acquisition devices.

[0026] In other implementations, each audio sample may be labeled with not only a user tag and a domain tag, but also a user region tag and a user age tag, etc. This is not a limitation in this embodiment. This can further improve the recognition accuracy of the trained voiceprint recognition model for people of different ages and from different regions, thereby increasing the robustness of the voiceprint recognition model.

[0027] Step S120: Acquire the human voiceprint features and domain features of each audio sample, where the domain features are used to characterize the scene features of the scene where the speaker in each audio sample is located.

[0028] In this embodiment, feature extraction can be performed on each audio sample to obtain target audio features; based on the first nonlinear layer, the above-mentioned human voiceprint features are obtained from the target audio features; based on the second nonlinear layer, the above-mentioned domain features are obtained from the target audio features. The domain features are used to characterize the scene features of the speaker in each audio sample, and can also be understood as other audio features in the target audio features other than the human voiceprint features. The first nonlinear layer may include a first activation function, based on which the human voiceprint features are obtained from the target audio features; the second nonlinear layer may include a second activation function, based on which the domain features are obtained from the target audio features. In this way, extracting human voiceprint features and domain features based on nonlinear activation functions can more accurately extract human voiceprint features and domain features, thereby improving the convergence speed of subsequent models and reducing model training time.

[0029] Step S130: filtering out the domain features from the human voiceprint features, and obtaining filtered human voiceprint features as human voice universal features.

[0030] In this embodiment, even though the human voiceprint features have been extracted from the target audio features through the first nonlinear layer, there will still be a certain error, that is, the extracted human voiceprint features may still include domain features. Therefore, in order to extract purer human voiceprint features, even if the human voiceprint features contain as few or no domain features as possible, the domain features in the human voiceprint features can be filtered out to obtain the filtered human voiceprint features as the above-mentioned human voice universal features. Specifically, the human voiceprint features are essentially human voice feature vectors, and the domain features are essentially domain feature vectors. Therefore, to filter out the domain features from the human voiceprint features, the difference between the human voice feature vector and the domain feature vector can be obtained to obtain a new feature vector, and the new feature vector can be used as the human voice universal feature.

[0031] Step S140: Based on the universal human voice features, the user label, and the domain label of each audio sample, the adversarial network is trained until the adversarial network meets preset conditions, thereby obtaining a trained voiceprint recognition model.

[0032] In this embodiment, universal features of human voice can be input into an adversarial network to obtain user labels and domain labels output by the adversarial network. Based on the difference between the output user labels and the user labels marked in the audio samples, as well as the difference between the output domain labels and the domain labels marked in the audio samples, the adversarial network is trained until the adversarial network meets the preset conditions, that is, until the adversarial network has converged. The part of the network that outputs the user labels in the adversarial network is used as the voiceprint recognition model obtained after training. When the adversarial network converges, when the universal features of human voice of the input sample audio are recognized by the adversarial network, the user label of the sample audio can be correctly determined, while the domain label of the sample audio cannot be correctly determined. It can be understood that at this time, the universal features of human voice learned by the adversarial network are not related to the domain, which means that the trained voiceprint recognition model can be applied to different domains and can accurately identify the user label (identity) of the speaker in the audio to be recognized in different domains.

[0033] In this embodiment, a set of audio samples marked with user labels and domain labels is used to perform domain adversarial training to obtain the final voiceprint recognition model. In this way, the robustness and generalization ability of the voiceprint recognition model when facing different domains are enhanced; and the domain features are filtered from the human voiceprint features to obtain the universal features of the human voice, and then the adversarial network is trained based on the universal features of the human voice. In this way, it can be further ensured that the model can converge in time, that is, the convergence speed of the model is improved, the training time of the model is reduced, and the above-mentioned voiceprint recognition model can be trained in a shorter time.

[0034] Please refer to Figure 2 , Figure 2 A flow chart of a voiceprint recognition model training method based on multi-task learning and adversarial training is provided in another embodiment of the present application. Figure 2 The voiceprint recognition model training method based on multi-task learning and adversarial training provided in the embodiment of the present application is described in detail. The voiceprint recognition model training method based on multi-task learning and adversarial training may include the following steps:

[0035] Step S210: Obtain an audio sample set to be trained, wherein each audio sample in the audio sample set is annotated with a user label and a domain label, wherein the user label is used to characterize the identity of the speaker in the audio sample, and the domain label is used to characterize the scene type of the scene in which the speaker in the audio sample is located. The user label is a first user label, and the domain label is a first domain label.

[0036] Step S220: Acquire the human voiceprint features and domain features of each audio sample, where the domain features are used to characterize the scene features of the scene where the speaker in each audio sample is located.

[0037] Step S230: filtering out the domain features from the human voiceprint features, and obtaining filtered human voiceprint features as human voice universal features.

[0038] In this embodiment, the specific implementation of steps S210 to S230 can refer to the content of the above embodiments and will not be repeated here.

[0039] Step S240: inputting the universal human voice feature of each audio sample into a domain discrimination network in the adversarial network to obtain a second domain label;

[0040] In this embodiment, the domain discrimination network may include a domain classifier. After obtaining the universal vocal features of each audio sample, the universal vocal features are further input into the domain discrimination network. The domain classifier in the domain discrimination network can classify and identify the universal vocal features and output a second domain label. The second domain label can be understood as the domain label corresponding to the audio sample identified by the domain classifier based on the universal vocal features.

[0041] Step S250: Input the universal human voice feature of each audio sample into the user discrimination network in the adversarial network to obtain a second user label.

[0042] In this embodiment, the user discrimination network may include a speaker classifier. After obtaining universal voice features for each audio sample, the universal voice features are further input into the user discrimination network. Accordingly, the speaker classifier in the user discrimination network can classify and identify the universal voice features and output a second user label. The second user label can be understood as the user label (identity) corresponding to the speaker in the audio sample, identified by the speaker classifier based on the universal voice features.

[0043] Step S260: Based on the first user tag, the second user tag, the first field tag, and the second field tag, obtain the total loss value of the adversarial network, and iteratively train the adversarial network according to the total loss value until the adversarial network meets the preset conditions, and use the trained user discrimination network as the voiceprint recognition model.

[0044] In some embodiments, see Figure 3 , step S260 may include:

[0045] Step S261: Determine a first loss value based on the difference between the first user label and the second user label.

[0046] In this embodiment, the first loss value can be obtained according to the following formula:

[0047]

[0048] Among them, L CE represents the cross-entropy loss function, f i represents the universal features of human voice obtained from the i-th input audio sample, Indicates the user label to which the i-th input audio sample belongs. M represents the audio sample set containing M speakers’ audio samples. M is a positive integer greater than 1, and the value of i is [1, M]. spk The difference between the first user label and the second user label after the universal feature of human voice passes through the adversarial network can also be informally referred to as L spk It is used to measure the accuracy of user label classification based on universal features of human voice. spk The smaller the value, the higher the accuracy of user label classification based on universal voice features.

[0049] Step S262: Determine a second loss value according to the difference between the first domain label and the second domain label.

[0050] In this embodiment, the second loss value can be obtained according to the following formula:

[0051]

[0052] Among them, L CE represents the cross-entropy loss function, f i It represents the universal voice feature obtained from the i-th input audio sample. M represents the audio samples of M speakers in the audio sample set. M is a positive integer greater than 1. The value of i is [1, M]. Represents the domain label of the i-th input audio sample. If the aforementioned audio sample set comes from N domains, then The value of L is [1, N], where N is a positive integer greater than 1. domain Represents the difference between the first domain label and the second domain label, which can also be understood as L domain It is used to measure the accuracy of domain classification using universal features of human voice. domain The larger the value, the worse the accuracy of domain classification based on universal features of human voice.

[0053] Step S263: Obtain the sum of the first loss value and the second loss value as a first sum.

[0054] Step S264: Determine the total loss value according to the first sum value.

[0055] Based on this, after obtaining the first loss value and the second loss value, the sum of the first loss value and the second loss value can be obtained as the first sum value, and the total loss value can be determined according to the first sum value.

[0056] In some implementations, the first sum value can be directly used as the total loss value. In this case, the loss of the adversarial network mainly comes from: user label discrimination loss and domain discrimination loss.

[0057] In other embodiments, see Figure 4 , step S264 may include:

[0058] Step S2641: Obtain the total amount of information contained in the domain features of each audio sample.

[0059] Specifically, based on the information acquisition function, the total amount of information contained in the domain feature is acquired;

[0060] The information acquisition function can be expressed as follows:

[0061]

[0062] Where M is a positive integer greater than 1, and ||di||F is used to characterize the Frobenius norm of the domain characteristics of the i-th input audio sample. The Frobenius norm can also be called the Hilbert-Schmidt norm, which is a simple and useful upper bound of the matrix norm. It can be understood popularly that L info It is used to limit the total amount of information contained in the domain features of the audio sample, that is, L info Used to measure the total amount of information related to the field. info The smaller the value, the less total information contained in the domain feature. The above formula for obtaining the total information of the domain feature can be understood as a domain information restriction function, which limits the total information in the domain feature to below a preset value. This ensures that when filtering out the domain feature from the human voiceprint feature, it avoids the excessive total information in the domain feature, which may lead to the mis-filtering of some human voiceprint features from the ultimately filtered human voice universal features. This, in turn, leads to the poor performance of the voiceprint recognition model trained based on the human voice universal features, and reduced recognition accuracy.

[0063] Step S2642: Obtain the sum of the total amount of information and the first sum value as the total loss value.

[0064] Based on this, the total amount of information obtained can be considered as the third loss value, and the information acquisition function can be considered as a loss function. The sum of the total information and the first sum value is obtained as the total loss value. At this time, the loss of the above-mentioned adversarial network mainly comes from: user label discrimination loss, domain discrimination loss, and total information loss. In this way, by limiting the amount of information contained in the domain features, the filtered universal features of human voice can be made independent of the domain, which improves the convergence speed of the model, thereby reducing the training time and improving the efficiency of model training.

[0065] Optionally, the formula for obtaining the first loss value can be understood as a user label classification loss function, and the formula for obtaining the second loss value can be understood as a domain label classification loss function. The gradients generated by the user label classification loss function and the domain information restriction function are forward propagated to subsequent network layers, while the gradients generated by the domain label classification loss function are reverse propagated to subsequent network layers. In this way, during the entire training process, the voiceprint recognition model can learn to distinguish the user labels of the speakers in the audio samples while making the learned universal features of the human voice not significantly correlated with the domain, thereby ensuring the robustness and generalization ability of the voiceprint recognition model when facing different domains.

[0066] In this embodiment, the preset conditions may be: the total loss value is less than the preset value, the total loss value no longer changes, or the number of training times reaches the preset number, etc. It is understandable that after the adversarial network is iteratively trained for multiple training cycles based on the audio sample set, wherein each training cycle includes multiple iterative trainings, the parameters in the adversarial network are continuously optimized, so that the above-mentioned total loss value becomes smaller and smaller, and finally becomes smaller to a fixed value, or is less than the above-mentioned preset value. At this time, it is indicated that the adversarial network has converged; of course, it can also be determined that the adversarial network has converged after the number of training times reaches the preset number. At this time, the user discrimination network in the adversarial network can be used as a voiceprint recognition model. Among them, the preset value and the preset number of times are both pre-set, and their values ​​can also be adjusted according to different application scenarios. This embodiment does not impose any restrictions on this.

[0067] In this embodiment, the universal voice features of each audio sample are input into the domain discrimination network in the adversarial network to obtain a second domain label. The universal voice features of each audio sample are also input into the user discrimination network in the adversarial network to obtain a second user label. Based on the first user label, the second user label, the first domain label, the second domain label, and the total information contained in the domain features, a total loss value of the adversarial network is obtained. The adversarial network is then iteratively trained based on the total loss value until the adversarial network meets preset conditions. The trained user discrimination network is then used as the voiceprint recognition model. In this manner, domain adversarial training is performed using a set of audio samples labeled with user and domain labels to obtain a final voiceprint recognition model. This improves the recognition accuracy of the voiceprint recognition model in different domains. Furthermore, by introducing a domain information restriction function, the learned universal voice features are not significantly correlated with the domain. This improves the robustness and generalization ability of the voiceprint recognition model while also increasing the convergence speed of the trained model, thereby reducing the training time and improving the efficiency of model training.

[0068] Please refer to Figure 5 , Figure 5 A flow chart of a voiceprint recognition model training method based on multi-task learning and adversarial training is provided as another embodiment of the present application. Figure 5 The voiceprint recognition model training method based on multi-task learning and adversarial training provided in the embodiment of the present application is described in detail. The voiceprint recognition model training method based on multi-task learning and adversarial training may include the following steps:

[0069] Step S310: Obtain an audio sample set to be trained, wherein each audio sample in the audio sample set is annotated with a user label and a domain label. The user label is used to characterize the identity of the speaker in the audio sample, and the domain label is used to characterize the scene type of the scene in which the speaker in the audio sample is located.

[0070] Step S320: Acquire the human voiceprint features and domain features of each audio sample, where the domain features are used to characterize the scene features of the scene where the speaker in each audio sample is located.

[0071] Step S330: filtering out the domain features from the human voiceprint features, and obtaining filtered human voiceprint features as human voice universal features.

[0072] Step S340: Based on the universal human voice features, the user label, and the domain label of each audio sample, the adversarial network is trained until the adversarial network meets preset conditions, thereby obtaining a trained voiceprint recognition model.

[0073] In this embodiment, the specific implementation of steps S310 to S340 can refer to the content of the above embodiments and will not be repeated here.

[0074] Step S350: Acquire the audio to be recognized input by the user.

[0075] In this embodiment, after the voiceprint recognition model is trained, it can be used to perform voiceprint recognition on the audio to be recognized input by the user. The audio to be recognized can be acquired through the audio acquisition device configured in the electronic device itself, or it can be acquired through an external audio acquisition device, and this embodiment does not limit this. The audio to be recognized can be acquired in a quiet room, a street cafe, a noisy street, or other scenes, and this embodiment does not limit this.

[0076] Step S360: extracting the human voiceprint features of the audio to be recognized as target voiceprint features.

[0077] Step S370: Input the target voiceprint feature into the voiceprint recognition model to obtain a user label corresponding to the user.

[0078] Based on this, after obtaining the audio to be identified, the human voiceprint features of the audio to be identified can be extracted through the feature extraction network, and the extracted human voiceprint features can be used as the target voiceprint features; the target voiceprint features are then input into the voiceprint recognition model. Correspondingly, the speaker classifier in the voiceprint recognition model can perform recognition based on the target voiceprint features and output the recognition result, i.e., the user label. For example, if the above-mentioned voiceprint recognition model is applied to an access control system, when the user inputs the audio to be identified, the access control system can extract the human voiceprint features of the audio to be identified through the voiceprint recognition model, and determine the user label of the user based on the human voiceprint features, i.e., determine the user's identity information; if it is determined that the user label of the user matches any one of the preset user labels in the preset user label set, then it is determined that the user has the authority to open the access control; if it is determined that the user label of the user does not match any one of the preset user labels in the preset user label set, then it is determined that the user does not have the authority to open the access control. The access control system based on the above voiceprint recognition model can accurately identify the user's identity regardless of whether the environment in which the user speaks the audio to be recognized is relatively noisy or relatively quiet. Of course, the above voiceprint recognition model can also be applied to scenarios such as application login, access control systems, smart speakers, robots, etc. that require the user's voiceprint characteristics, and this embodiment does not limit this.

[0079] In this embodiment, the trained voiceprint recognition model can be used to identify user labels for audio input by the user. This improves the robustness of the voiceprint recognition model and allows accurate identification of user identities even when faced with audio input from users in various fields. This improves the accuracy of voiceprint recognition and enhances the user experience.

[0080] See Figure 6 , Figure 6 A flow chart of a voiceprint recognition model training method based on multi-task learning and adversarial training is provided as another embodiment of the present application. Figure 6 The voiceprint recognition model training method based on multi-task learning and adversarial training provided in the embodiment of the present application is elaborated in detail.

[0081] In this embodiment, an audio sample set to be trained is obtained, and feature extraction is performed on each audio sample in the audio sample set based on a feature extraction network to obtain target audio features; based on a first nonlinear transformation layer, a human voiceprint feature is obtained from the target audio feature, wherein the human voiceprint feature is represented by a vector s; based on a second nonlinear transformation layer, a domain feature is obtained from the target audio feature, wherein the domain feature is represented by a vector d; a difference vector F between vector s and vector d is obtained, and the difference vector F is used as a universal feature of human voice; the universal feature of human voice is input into a speaker classifier C in the adversarial network spk , get the second user label, based on the user label classification loss function L spk Calculate the difference between the second user label and the first user label marked in the audio sample as the first loss value; input the universal feature of human voice into the domain classifier C in the adversarial network domain , get the second domain label, based on the domain label classification loss function L domain Calculate the difference between the second domain label and the first domain label marked in the audio sample as the second loss value; and limit the function L based on the domain information info , limit the total amount of information in the domain features, and use the total amount of information as the third loss value; obtain the sum of the first loss value, the second loss value and the third loss value, Loss = L spk +L domain +L info , serving as the total loss value, Loss. The gradients generated by the user label classification loss function and the domain information restriction function are forward propagated to subsequent network layers, while the gradients generated by the domain label classification loss function are back propagated to subsequent network layers. Finally, the adversarial network is iteratively trained based on the total loss value until it meets the preset conditions. The trained user discrimination network is then used as the voiceprint recognition model.

[0082] Domain adversarial training involves iteratively training the adversarial network based on domain labels, user labels, and universal voice features. This is known as domain adversarial training, a method in transfer learning. Its goal is to map data from the source domain and other domains into the same feature space, ensuring they share the same distribution. This allows models trained on data from the source domain to be directly applied to data from other domains. Domain adversarial training typically consists of three modules: a feature learning module, a task classification module, and a domain discrimination module. The feature learning module learns feature representations from the raw data, enabling the task classification module to distinguish the labels of the source domain data while preventing the domain discrimination module from distinguishing the domain to which the data belongs. The task classification module performs classification and discrimination tasks on the source domain data, aiming to classify the source domain data as accurately as possible. The domain discrimination module performs domain classification on the features, aiming to identify the domain to which the data belongs as accurately as possible. After domain adversarial learning, the features generated by the feature learning module can be used to build effective voiceprint recognition models in real-world applications, even when encountering speech scenarios and acquisition devices that differ from those used in the training phase.

[0083] In this embodiment, domain adversarial training is performed on an audio sample set marked with user labels and domain labels to obtain the final voiceprint recognition model. In addition to improving the recognition accuracy of the voiceprint recognition model when facing different domains, a domain information restriction function is introduced to make the learned universal features of human voice not significantly correlated with the domain. While improving the robustness and generalization ability of the voiceprint recognition model, the convergence speed of the training model is also improved, thereby reducing the training time of the model training and improving the efficiency of the model training.

[0084] Please refer to Figure 7 , which shows a structural block diagram of a voiceprint recognition model training device 400 based on multi-task learning and adversarial training provided by an embodiment of the present application. The device 400 may include: a sample acquisition module 410, a feature acquisition module 420, a feature filtering module 430 and a model training module 440.

[0085] The sample acquisition module 410 is used to obtain an audio sample set to be trained, where each audio sample in the audio sample set is annotated with a user label and a domain label. The user label is used to characterize the identity of the speaker in the audio sample, and the domain label is used to characterize the scene type of the scene in which the speaker in the audio sample is located.

[0086] The feature acquisition module 420 is used to acquire the human voiceprint feature and the domain feature of each audio sample. The domain feature is used to characterize the scene feature of the scene where the speaker in each audio sample is located.

[0087] The feature filtering module 430 is configured to filter out the domain features from the human voiceprint features, and obtain the filtered human voiceprint features as human voice universal features.

[0088] The model training module 440 is used to train the adversarial network based on the universal human voice features, the user labels and the domain labels of each audio sample until the adversarial network meets the preset conditions, obtain the trained adversarial network, and use the user discrimination network in the trained adversarial network as the voiceprint recognition model.

[0089] In some embodiments, the human voiceprint feature is a human voice feature vector, and the domain feature is a domain feature vector. The feature filtering module 430 can be specifically used to obtain the difference between the human voice feature vector and the domain feature vector as the human voice universal feature.

[0090] In some embodiments, the adversarial network includes a user discrimination network and a domain discrimination network, the user label is a first user label, the domain label is a first domain label, and the model training module 440 may include: a domain label acquisition unit, a user label acquisition unit, and an iterative training unit. Among them, the domain label acquisition unit can be used to input the universal human voice features of each audio sample into the domain discrimination network to obtain a second domain label. The user label acquisition unit can be used to input the universal human voice features of each audio sample into the user discrimination network to obtain a second user label. The iterative training unit can be used to obtain the total loss value of the adversarial network based on the first user label, the second user label, the first domain label, and the second domain label, and iteratively train the adversarial network according to the total loss value until the adversarial network meets the preset conditions, and use the trained user discrimination network as a voiceprint recognition model.

[0091] In this manner, the iterative training unit may include: a first loss value acquisition subunit, a second loss value acquisition subunit, a sum value acquisition subunit, and a total loss value acquisition subunit. The first loss value acquisition subunit may be configured to determine a first loss value based on the difference between the first user tag and the second user tag. The second loss value acquisition subunit may be configured to determine a second loss value based on the difference between the first domain tag and the second domain tag. The sum value acquisition subunit may be configured to obtain the sum of the first loss value and the second loss value as a first sum value. The total loss value acquisition subunit may be configured to determine the total loss value based on the first sum value.

[0092] In some embodiments, the first loss value obtaining subunit may be specifically configured to obtain the first loss value:

[0093]

[0094] Among them, L spk is the first loss value, L CE is the cross entropy loss function, f i The universal features of human voice obtained for the audio sample of the i-th input, is the first user tag to which the audio sample inputted at the ith time belongs, M represents that the audio sample set contains audio samples of M speakers, M is a positive integer greater than 1, and i is a positive integer not less than 1 and not greater than M.

[0095] In some embodiments, the second loss value obtaining subunit may be specifically configured to obtain the second loss value:

[0096]

[0097] Among them, L domain is the second loss value, L CE represents the cross entropy loss function, f i represents the universal human voice feature obtained from the audio sample of the i-th input, M represents the audio samples of M speakers contained in the audio sample set, M is a positive integer greater than 1, and i is a positive integer not less than 1 and not greater than M. represents the first domain label of the i-th input audio sample.

[0098] In some embodiments, the total loss value acquisition subunit can be specifically used to: obtain the total amount of information contained in the domain features of each audio sample; and obtain the sum of the total amount of information and the first sum value as the total loss value.

[0099] In some embodiments, the total loss value acquisition subunit may be specifically configured to: acquire the total amount of information contained in the domain feature based on an information amount acquisition function; and acquire the sum of the total amount of information and the first sum value as the total loss value. The information amount acquisition function includes:

[0100]

[0101] Wherein, M is a positive integer greater than 1, and ||di||F is used to characterize the Frobenius norm of the domain feature of the i-th input audio sample.

[0102] In some embodiments, the feature acquisition module 420 may include an audio feature acquisition unit, a vocal feature acquisition unit, and a domain feature acquisition unit. The audio feature acquisition unit may be configured to extract features from each audio sample to obtain target audio features. The vocal feature acquisition unit may be configured to acquire the vocal voiceprint features from the target audio features based on a first linear layer. The domain feature acquisition unit may be configured to acquire the domain features from the target audio features based on a second linear layer.

[0103] In some embodiments, the voiceprint recognition model training device 400 based on multi-task learning and adversarial training may further include an audio acquisition module, a feature extraction module, and a user identification module. The audio acquisition module may be used to acquire audio to be recognized from a user input. The feature extraction module may be used to extract voiceprint features of the human voice in the audio to be recognized as target voiceprint features. The user identification module may be used to input the target voiceprint features into the voiceprint recognition model to obtain a user label corresponding to the user.

[0104] Those skilled in the art will clearly understand that, for the convenience and brevity of description, the specific working processes of the above-described devices and modules can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0105] In several embodiments provided in this application, the coupling between modules may be electrical, mechanical or other forms of coupling.

[0106] In addition, the functional modules in the various embodiments of the present application may be integrated into a processing module, or each module may exist physically separately, or two or more modules may be integrated into a single module. The above-mentioned integrated modules may be implemented in the form of hardware or software functional modules.

[0107] In summary, in the solution provided by the embodiment of the present application, an audio sample set to be trained is obtained, and each audio sample in the audio sample set is marked with a user label and a domain label; the human voiceprint features and domain features of each audio sample are obtained; the domain features are filtered out from the human voiceprint features to obtain the filtered human voiceprint features as the human voice universal features; based on the human voice universal features, user labels and domain labels of each audio sample, the adversarial network is trained until the adversarial network meets the preset conditions, and a trained voiceprint recognition model is obtained. Among them, the audio sample set marked with user labels and domain labels is used for domain adversarial training to obtain the final voiceprint recognition model, which enhances the robustness and generalization ability of the voiceprint recognition model when facing different domains; and the domain features are filtered out from the human voiceprint features to obtain the human voice universal features, and then the adversarial network is trained based on the human voice universal features. In this way, it can be further ensured that the model can converge in time, that is, the convergence speed of the model is improved, the training time of the model is reduced, and the above-mentioned voiceprint recognition model can be trained in a shorter time.

[0108] The following will be combined Figure 8 An electronic device provided by this application is described.

[0109] Reference Figure 8 , Figure 8 The following is a block diagram of an electronic device 500 provided in an embodiment of the present application. The voiceprint recognition model training method based on multi-task learning and adversarial training provided in an embodiment of the present application can be executed by the electronic device 500. The electronic device 500 can be a device capable of running applications, such as a smartphone, tablet computer, smartwatch, laptop computer, desktop computer, server, voice recorder, etc.

[0110] The electronic device 500 in the embodiment of the present application may include one or more of the following components: a processor 501, a memory 502, and one or more applications, wherein the one or more applications may be stored in the memory 502 and configured to be executed by one or more processors 501, and the one or more programs are configured to execute the method as described in the aforementioned method embodiment.

[0111] The processor 501 may include one or more processing cores. The processor 501 utilizes various interfaces and circuits to connect various components within the electronic device 500. It executes various functions and processes data for the electronic device 500 by running or executing instructions, programs, code sets, or instruction sets stored in the memory 502, as well as accessing data stored in the memory 502. Optionally, the processor 501 may be implemented using at least one of the following hardware forms: a digital signal processing (DSP), a field-programmable gate array (FPGA), or a programmable logic array (PLA). The processor 501 may integrate one or a combination of a central processing unit (CPU), a graphics processing unit (GPU), and a modem. The CPU primarily processes the operating system, user interface, and application programs; the GPU is responsible for rendering and drawing display content; and the modem handles wireless communications. It is understood that the modem may also be integrated into the processor 501 and implemented separately via a communication chip.

[0112] The memory 502 may include a random access memory (RAM) or a read-only memory (ROM). The memory 502 may be used to store instructions, programs, codes, code sets, or instruction sets. The memory 502 may include a program storage area and a data storage area, wherein the program storage area may store instructions for implementing an operating system, instructions for implementing at least one function (such as a touch function, a sound playback function, an image playback function, etc.), instructions for implementing the following various method embodiments, etc. The data storage area may also store data (such as the various corresponding relationships described above) created by the electronic device 500 during use.

[0113] Those skilled in the art will clearly understand that, for the convenience and brevity of description, the specific working processes of the above-described devices and modules can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0114] In several embodiments provided in this application, the coupling or direct coupling or communication connection between the modules shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or modules, which can be electrical, mechanical or other forms.

[0115] In addition, the functional modules in the various embodiments of the present application may be integrated into a processing module, or each module may exist physically separately, or two or more modules may be integrated into a single module. The above-mentioned integrated modules may be implemented in the form of hardware or software functional modules.

[0116] Please refer to Figure 9 , which shows a block diagram of a computer-readable storage medium provided in an embodiment of the present application. The computer-readable medium 600 stores program code, which can be called by a processor to execute the method described in the above method embodiment.

[0117] The computer-readable storage medium 600 can be an electronic memory such as a flash memory, an EEPROM (Electrically Erasable Programmable Read-Only Memory), an EPROM, a hard disk, or a ROM. Alternatively, the computer-readable storage medium 600 includes a non-transitory computer-readable storage medium. The computer-readable storage medium 600 has storage space for program code 610 for executing any of the method steps described above. These program codes can be read from or written to one or more computer program products. The program code 610 can be compressed, for example, in a suitable form.

[0118] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A voiceprint recognition model training method based on multi-task learning and adversarial training, characterized in that: The method comprises: Obtaining an audio sample set to be trained, wherein each audio sample in the audio sample set is annotated with a first user tag and a first domain tag, wherein the first user tag is used to characterize the identity of a speaker in the audio sample, and the first domain tag is used to characterize the scene type of the scene in which the speaker in the audio sample is located; Acquire a human voiceprint feature and a domain feature of each audio sample, wherein the domain feature is used to characterize a scene feature of a scene in which a speaker in each audio sample is located; Filtering the domain feature from the human voiceprint feature to obtain the filtered human voiceprint feature as the human voice universal feature; Based on the universal human voice features, the user label and the domain label of each audio sample, an adversarial network including a user discrimination network and a domain discrimination network is trained until the adversarial network meets the preset conditions, thereby obtaining a trained voiceprint recognition model: the universal human voice features of each audio sample are input into the domain discrimination network to obtain a second domain label; the universal human voice features of each audio sample are input into the user discrimination network to obtain a second user label; a first loss value is determined based on the difference between the first user label and the second user label; a second loss value is determined based on the difference between the first domain label and the second domain label; a sum of the first loss value and the second loss value is obtained as a first sum value; a total loss value is determined based on the first sum value, and the adversarial network is iteratively trained based on the total loss value until the adversarial network meets the preset conditions, and the trained user discrimination network is used as the voiceprint recognition model.

2. The method according to claim 1, characterized in that The human voiceprint feature is a human voice feature vector, the domain feature is a domain feature vector, and filtering the domain feature from the human voiceprint feature to obtain the filtered human voiceprint feature as the human voice universal feature includes: A difference between the human voice feature vector and the domain feature vector is obtained as the human voice universal feature.

3. The method according to claim 1, characterized in that Determining the total loss value according to the first sum value includes: Obtain the total amount of information contained in the domain features of each audio sample; The sum of the total amount of information and the first sum is obtained as the total loss value.

4. The method according to claim 1, wherein Obtain the first loss value, include: in, is the first loss value, is the cross entropy loss function, For the i The universal features of human voice obtained by inputting the audio samples, For the i The first user label to which the input audio sample belongs, M Characterize that the audio sample set contains M Audio samples of speakers, M is a positive integer greater than 1, i is a positive integer not less than 1 and not greater than M.

5. The method according to claim 1, wherein Obtaining the second loss value includes: in, is the second loss value, represents the cross entropy loss function, Indicates the i The universal features of human voice obtained by inputting the audio samples, M Indicates that the audio sample set contains M Audio samples of speakers, M is a positive integer greater than 1, i is a positive integer not less than 1 and not greater than M, Indicates the i The first field labels of the input audio samples.

6. The method according to claim 3, characterized in that The total amount of information contained in the acquired domain features includes: Based on the information acquisition function, the total amount of information contained in the domain features is acquired; The information acquisition function includes: Wherein, M is a positive integer greater than 1, and ||di||F is used to characterize the Frobenius norm of the domain feature of the i-th input audio sample.

7. The method according to claim 1, characterized in that The method of obtaining the human voiceprint feature and domain feature of each audio sample further includes: Performing feature extraction on each of the audio samples to obtain target audio features; Based on the first nonlinear layer, obtaining the human voiceprint feature from the target audio feature; Based on a second nonlinear layer, the domain feature is obtained from the target audio feature.

8. The method according to any one of claims 1 to 7, characterized in that After training the adversarial network based on the universal human voice features, the user label, and the domain label of each audio sample until the adversarial network meets preset conditions and a trained voiceprint recognition model is obtained, the method further includes: Get the audio input to be recognized by the user; Extracting the voiceprint features of the human voice of the audio to be recognized as target voiceprint features; The target voiceprint feature is input into the voiceprint recognition model to obtain a user label corresponding to the user.

Images