Voiceprint recognition model training method and device, electronic equipment and storage medium

Abstract

The voiceprint recognition model training method and device, the electronic equipment and the storage medium provided by the application comprise the following steps: obtaining a training sample; inputting the training sample into a voiceprint recognition model to be trained; calculating a covariance matrix of a label category; determining a first loss function according to a speaker feature and the covariance matrix of the label category corresponding to the speaker feature; training the voiceprint recognition model by using the first loss function to obtain a trained voiceprint recognition model; and through the above manner, the covariance matrix of the corresponding label category is used to realize semantic data enhancement of the speaker feature, the speaker feature and the covariance matrix of the corresponding label category are integrated into the calculation of the first loss function, data enhancement in a feature space is realized in the training process of the model, the problem of difficult data enhancement of the voiceprint recognition model can be solved, the effect of realizing semantic data enhancement is achieved, and the effect of improving the recognition accuracy of the voiceprint recognition model is achieved.

Description

Technical Field

[0001] This application relates to the field of voiceprint recognition technology and also to the field of artificial intelligence technology, specifically to a voiceprint recognition model training method, device, electronic device and storage medium. Background Technology

[0002] Voiceprint recognition (VPR), also known as speaker recognition (SR), is a type of biometric identification technology that includes speaker identification and speaker verification. The production of human voice is a complex physiological process between the language center and the vocal organs. Although each person's voice has a certain degree of variability due to factors such as age, physical condition, and emotional fluctuations, the size and shape of each person's vocal organs, such as the tongue, teeth, lungs, and nasal cavity, are fundamentally different, thus making each person's voice distinguishable. Voiceprint recognition is a technology that uses speech feature parameters contained in the speech waveform, reflecting the physiological and behavioral characteristics of a specific speaker, to automatically identify the speaker's identity.

[0003] Furthermore, speaker verification, utilizing voiceprint recognition technology, compares two given audio clips and verifies whether they come from the same speaker. In recent years, with the significant advancements in deep neural networks, speaker verification technology has been greatly influenced, gradually shifting from traditional probabilistic statistical methods to deep learning-based approaches. Speaker verification has now become an important technology in daily life, used in applications such as biometric authentication and intelligent control. However, the performance of current speaker verification systems in real-world industrial applications remains unsatisfactory. The lack of sufficient data is a key challenge to performance robustness in complex scenarios, thus the demand for audio data is increasing daily.

[0004] In the training of neural networks used for voiceprint recognition and speaker verification, data augmentation is an important technique for addressing missing data, increasing the number of training samples, and enhancing their distribution diversity. Commonly used data augmentation techniques include adding noise, reverberation, and velocity perturbations, as well as applying SpecAugment (directly masking the spectrum during training), a technique commonly used in speech recognition, to these neural networks. These methods directly manipulate the original speech signal and are signal-level augmentation methods, thus limiting the diversity of the generated augmented samples. Furthermore, these augmentation methods also incur significant computational and I / O overhead.

[0005] Existing technologies also propose using deep generative models such as Generative Adversarial Networks (GANs) and Variational Autoencoders (VAEs) to learn the distribution of speaker features with noise and generate new features from this distribution. However, these methods require explicit enhancement of samples using complex deep generative models, which significantly slows down the training process of speaker recognition models. Another approach is to enhance the data by generating a pure noise distribution from specified noisy data and then generating new noise from it to add to the original data features. The drawback of this approach is that this distribution depends on additional specific noise datasets, and the diversity of generated samples is far from sufficient. Summary of the Invention

[0006] In view of the above problems, embodiments of this application provide a voiceprint recognition model training method, apparatus, electronic device and storage medium to solve the above technical problems.

[0007] In a first aspect, embodiments of this application provide a method for training a voiceprint recognition model, including:

[0008] Obtain training samples, wherein the training samples include speech data and the label categories corresponding to the speech data;

[0009] The training samples are input into the voiceprint recognition model to be trained. The speaker features corresponding to each training sample are obtained through the speaker encoder of the voiceprint recognition model. The speaker features are then identified through the classifier of the voiceprint recognition model.

[0010] The covariance matrix of the corresponding label category is calculated based on multiple speaker features corresponding to each label category, wherein the covariance matrix is ​​used to characterize the semantic transformation direction of all speaker features of the corresponding label category;

[0011] A first loss function is determined based on the speaker features and the covariance matrix of the label categories corresponding to the speaker features;

[0012] The voiceprint recognition model is trained using a first loss function to obtain the trained voiceprint recognition model.

[0013] Optionally, after calculating the covariance matrix of the corresponding label category based on multiple speaker features corresponding to each label category, the method further includes:

[0014] The difficulty coefficient of the speaker feature is calculated based on the weight matrix of the label category corresponding to the speaker feature and the speaker feature itself, wherein the difficulty coefficient is used to characterize the recognition difficulty of the corresponding speaker feature;

[0015] Accordingly, determining the first loss function based on the speaker features and the covariance matrix of the label categories corresponding to the speaker features includes:

[0016] The first loss function is determined based on the speaker features, the difficulty coefficients corresponding to the speaker features, and the covariance matrix of the label categories corresponding to the speaker features.

[0017] Optionally, the first loss function is:

[0018]

[0019] Where N is the total number of training samples, C is the number of label categories, and f i Let yi be the speaker feature of the i-th training sample, and w be the label category of the i-th training sample. yi Let w be the weight matrix corresponding to the label category yi of the i-th training sample. j Let ∑yi be the weight matrix corresponding to the j-th label category, and ∑yi be the covariance matrix of the label category yi corresponding to the i-th training sample. i Let be the difficulty coefficient of the i-th training sample. m is the first hyperparameter, s is the second hyperparameter, and λ is the third hyperparameter.

[0020] Optionally, the step of calculating the covariance matrix of the corresponding label category based on multiple speaker features corresponding to each label category includes:

[0021] The category mean center μ of the label category is obtained based on the speaker features of multiple training samples corresponding to the label category. j , Among them, f k Let M be the speaker features of the k-th training sample in the j-th label category, and M be the number of training samples in the j-th label category.

[0022] Based on the category mean center and all speaker features of the label category, obtain each matrix element ∑j(m, n) in the covariance matrix of the label category. Where ∑j(m,n) are the matrix elements in the m-th row and n-th column of the covariance matrix ∑j. and Let be the speaker features of the k-th training sample in the j-th label category, representing the values ​​in the m-th and n-th dimensions, respectively. and The class mean center μ of the j-th label category is respectively j The values ​​in the m-th and n-th dimensions, where D is the dimension of the speaker features.

[0023] Optionally, after training the voiceprint recognition model using the first loss function to obtain the trained voiceprint recognition model, the method further includes:

[0024] The speaker encoder of the trained voiceprint recognition model is used to extract voiceprint features from each speech sample in the training set to obtain the speaker features corresponding to the speech sample.

[0025] The covariance matrix of the corresponding label category is calculated based on multiple speaker features corresponding to each label category, wherein the covariance matrix is ​​used to characterize the semantic transformation direction of all speaker features of the corresponding label category;

[0026] Based on the speaker features and the covariance matrix of the corresponding label categories, a Gaussian distribution is established for the speaker features, and speaker enhancement features with infinite semantic transformation are sampled from the Gaussian distribution.

[0027] Optionally, after establishing a Gaussian distribution for the label category based on multiple speaker features and the covariance matrix corresponding to the label category, and sampling speaker enhancement features with infinite semantic transformation from the Gaussian distribution, the method further includes:

[0028] A training dataset for the label category is constructed using multiple speaker features and multiple speaker enhancement features of the label category;

[0029] The classifier of the voiceprint recognition model is trained using the training dataset and the first loss function to obtain an optimized voiceprint recognition model.

[0030] Secondly, embodiments of this application provide a voiceprint recognition method, including:

[0031] The speech data to be recognized is input into the voiceprint recognition model, and the recognition result of the speech is output. The voiceprint recognition model is trained and obtained according to the above-mentioned voiceprint recognition model training method.

[0032] Thirdly, embodiments of this application provide a voiceprint recognition model training device, comprising:

[0033] A training sample construction module is used to obtain training samples, wherein the training samples include speech data and the label categories corresponding to the speech data;

[0034] The voiceprint recognition module is used to input the training samples into the voiceprint recognition model to be trained, obtain the speaker features corresponding to each training sample through the speaker encoder of the voiceprint recognition model, and recognize the speaker features through the classifier of the voiceprint recognition model.

[0035] The sample augmentation module is used to calculate the covariance matrix of the corresponding label category based on multiple speaker features corresponding to each label category, wherein the covariance matrix is ​​used to characterize the semantic transformation direction of all speaker features of the corresponding label category;

[0036] The loss calculation module is used to determine a first loss function based on the speaker features and the covariance matrix of the label categories corresponding to the speaker features;

[0037] The parameter adjustment module is used to train the voiceprint recognition model using a first loss function to obtain the trained voiceprint recognition model.

[0038] Fourthly, embodiments of this application provide an electronic device, including a processor and a memory coupled to the processor, the memory storing program instructions executable by the processor; when the processor executes the program instructions stored in the memory, it implements the above-described voiceprint recognition model training method or the above-described voiceprint recognition method.

[0039] Fifthly, embodiments of this application provide a storage medium storing program instructions, which, when executed by a processor, implement the aforementioned voiceprint recognition model training method or the aforementioned voiceprint recognition method.

[0040] The voiceprint recognition model training method, apparatus, electronic device, and storage medium provided in this application include the following steps: acquiring training samples; inputting the training samples into the voiceprint recognition model to be trained; acquiring speaker features corresponding to each training sample through the speaker encoder of the voiceprint recognition model; recognizing the speaker features through the classifier of the voiceprint recognition model; calculating the covariance matrix of the corresponding label category based on multiple speaker features corresponding to each label category; determining a first loss function based on the speaker features and the covariance matrix of the label category corresponding to the speaker features; training the voiceprint recognition model using the first loss function to obtain the trained voiceprint recognition model; through the above method, semantic data enhancement of speaker features is achieved by utilizing the covariance matrix of the corresponding label category, integrating speaker features and the covariance matrix of the corresponding label category into the calculation of the first loss function, realizing data enhancement in the feature space during the model training process, solving the problem of difficult data enhancement in voiceprint recognition models, achieving the effect of semantic data enhancement, and also improving the recognition accuracy of the voiceprint recognition model. Furthermore, the training method of this application does not require direct data augmentation of the speech training samples, which helps to simplify the model training process. At the same time, data augmentation can be reflected in the construction of the first loss function without changing the structure of the voiceprint recognition model, and can be compatible with voiceprint recognition models with various network structures.

[0041] These or other aspects of this application will become more apparent in the following description of the embodiments. Attached Figure Description

[0042] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0043] Figure 1 A schematic flowchart of a voiceprint recognition model training method provided in an embodiment of this application is shown.

[0044] Figure 2 A schematic diagram of the structure of a voiceprint recognition model training device provided in an embodiment of this application is shown.

[0045] Figure 3 A schematic diagram of the structure of an electronic device provided in an embodiment of this application is shown.

[0046] Figure 4 A schematic diagram of the structure of a storage medium provided in an embodiment of this application is shown. Detailed Implementation

[0047] The embodiments of this application are described in detail below. Examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain this application, and should not be construed as limiting this application.

[0048] To enable those skilled in the art to better understand the solutions of this application, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of them. All other embodiments obtained by those skilled in the art based on the embodiments of this application without creative effort are within the scope of protection of this application.

[0049] In the embodiments of this application, it should be noted that, in this document, relational terms such as first and second are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations.

[0050] Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0051] In the description of the embodiments of this application, the words "example" or "for example" are used to indicate exemplification, illustration, or description. Any embodiment or design described as "example" or "for example" in the embodiments of this application is not to be construed as being more preferred or having more advantages than another embodiment or design. The use of the words "example" or "for example" is intended to present relative concepts in a clear manner.

[0052] Furthermore, in the embodiments of this application, "multiple" refers to two or more. Therefore, in the embodiments of this application, "multiple" can also be understood as "at least two". "At least one" can be understood as one or more, such as one, two, or more. For example, including at least one means including one, two, or more, and is not limited to which ones are included. For example, including at least one of A, B, and C, then it could include A, B, C, A and B, A and C, B and C, or A and B and C.

[0053] It should be noted that in the embodiments of this application, "and / or" describes the relationship between associated objects, indicating that there can be three relationships. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone. In addition, the character " / ", unless otherwise specified, generally indicates that the associated objects before and after it are in an "or" relationship.

[0054] It should be noted that in the embodiments of this application, "connection" can be understood as electrical connection. The connection between two electrical components can be a direct or indirect connection between the two electrical components. For example, the connection between A and B can be a direct connection between A and B, or an indirect connection between A and B through one or more other electrical components.

[0055] Figure 1 This is a schematic flowchart illustrating a voiceprint recognition model training method according to an embodiment of this application. It should be noted that if substantially the same result is obtained, the method of this application is not based on... Figure 1 The illustrated process sequence is limited. For example... Figure 1 As shown, the training method for this voiceprint recognition model includes the following steps:

[0056] S11, Obtain training samples, wherein the training samples include speech data and the label categories corresponding to the speech data;

[0057] Each label category corresponds to a speaker identity; for example, the speaker identity can be speaker A, speaker B, or speaker C. The training samples include speech data from multiple speakers, with each speaker corresponding to multiple speech data points. The label categories corresponding to the speech data are the actual label categories. For example, in this embodiment, the total number of training samples is N, which is the total number of speech data points, and the number of speakers is C, which is the number of label categories. Each speaker (label category) corresponds to multiple speech data points.

[0058] S12, the training samples are input into the voiceprint recognition model to be trained, the speaker features corresponding to each training sample are obtained through the speaker encoder of the voiceprint recognition model, and the speaker features are identified through the classifier of the voiceprint recognition model.

[0059] The voiceprint recognition model includes a speaker encoder and a classifier. The speaker encoder extracts the speaker's voiceprint features from the speech data; these voiceprint features are also called speaker characteristics. The classifier identifies the speaker's identity (the label category to which the speaker belongs) based on these speaker characteristics. The recognition result for each speaker characteristic includes the probability that the speaker characteristic belongs to each label category. For example, when the label categories include speaker A, speaker B, and speaker C, the recognition result includes the probability that the speaker characteristic belongs to speaker A, the probability that the speaker characteristic belongs to speaker B, and the probability that the speaker characteristic belongs to speaker C.

[0060] S13, calculate the covariance matrix of the corresponding label category based on the multiple speaker features corresponding to each label category, wherein the covariance matrix is ​​used to characterize the semantic transformation direction of all speaker features of the corresponding label category;

[0061] The speech data is input into the speaker encoder, and the speaker feature data corresponding to the speech data is output. The speaker feature data can include feature vectors of different feature dimensions. For example, different feature dimensions can include the speaker's oral physiological feature dimension, the speaker's vocal cord physiological feature dimension, the speaker's age-related feature dimension, the speaker's environment-related feature dimension, and the cross feature dimension of the speaker's oral physiological feature and vocal cord physiological feature, etc.

[0062] Specifically, based on the actual label categories of the speech data in the training samples, the speaker features corresponding to each label category are determined, and then the covariance matrix of the label category is calculated based on the multiple speaker features corresponding to each label category.

[0063] As one implementation method, the covariance matrix of the label category can be calculated as follows:

[0064] S131, Obtain the category mean center μ of the label category based on the speaker features of multiple training samples corresponding to the label category. j , Among them, f k Let M be the speaker features of the k-th training sample in the j-th label category, and M be the number of training samples in the j-th label category.

[0065] S132, based on the category mean center and all speaker features of the label category, obtain each matrix element ∑j(m, n) in the covariance matrix of the label category. Where ∑j(m,n) are the matrix elements in the m-th row and n-th column of the covariance matrix ∑j. and Let be the speaker features of the k-th training sample in the j-th label category, representing the values ​​in the m-th and n-th dimensions, respectively. and The class mean center μ of the j-th label category is respectively j The values ​​in the m-th and n-th dimensions, where D is the dimension of the speaker features.

[0066] In this embodiment, the covariance matrix of the tag category represents the semantic transformation direction of each speaker feature in that tag category.

[0067] S14, determine the first loss function based on the speaker features and the covariance matrix of the label categories corresponding to the speaker features;

[0068] In this context, each matrix element ∑j(m, n) in the covariance matrix represents a semantic transformation direction of the corresponding two-dimensional feature vectors (the m-th and n-th dimensions) in the speaker features. The product of each speaker feature in the label category and a matrix element can represent a semantic data enhancement feature of that speaker feature. In this way, the first loss function includes a part used to characterize the semantic data enhancement of the speaker features. During the calculation of the first loss function, multiple semantic data enhancement features of each speaker feature are incorporated, realizing semantic enhancement in the feature space. The voiceprint recognition model trained according to the first loss function can better learn the enhancement features of the speaker features and improve the recognition accuracy of the voiceprint recognition model.

[0069] S15, The voiceprint recognition model is trained using the first loss function to obtain the trained voiceprint recognition model;

[0070] In the process of training the voiceprint recognition model using the first loss function, the training of the voiceprint recognition model can be considered complete when the result of the first loss function converges; or, the training can also be considered complete when the number of training iterations reaches a preset threshold. The user then outputs the speaker's identity based on the voice data using the obtained trained voiceprint recognition model.

[0071] In this embodiment, semantic data augmentation of speaker features is achieved using the covariance matrix of the corresponding label category. The speaker features and the covariance matrix of the corresponding label category are integrated into the calculation of the first loss function. This achieves data augmentation in the feature space during model training, solving the problem of difficult data augmentation in voiceprint recognition models. It achieves semantic data augmentation and improves the recognition accuracy of the voiceprint recognition model. Furthermore, the training method of this application does not require direct data augmentation of the speech training samples, simplifying the model training process. Simultaneously, data augmentation can be incorporated into the construction of the first loss function without altering the structure of the voiceprint recognition model, making it compatible with voiceprint recognition models of various network structures.

[0072] As one implementation method, a difficulty coefficient is defined for each speaker feature based on the probability of speaker feature recognition. After step S13 and before step S14, the following steps are also included:

[0073] S21, calculate the difficulty coefficient of the speaker feature based on the weight matrix of the label category corresponding to the speaker feature and the speaker feature, wherein the difficulty coefficient is used to characterize the recognition difficulty of the corresponding speaker feature;

[0074] In the last fully connected layer of the classifier, the weight matrix is ​​W, where the weight matrix...

[0075] in, Corresponding to the first tag category, Corresponding to the second tag category, Corresponding to the j-th tag category, This corresponds to the Cth tag category.

[0076] The weight matrix is ​​determined based on the true label category corresponding to the speaker features, and the difficulty coefficient is calculated based on the weight matrix. The i-th speaker feature f... i The difficulty level is dill i , yi is the speaker's characteristic f i The actual tag category, w yiw is the weight matrix of the classifier corresponding to the label category yi. yi The weight matrix W of the last fully connected layer of the classifier is the part corresponding to the label category yi. For w yi The transpose of the matrix, Indicate speaker characteristics f i Let be the probability of label category yi, when When the value is large, it indicates the speaker's characteristics f. i The recognition difficulty is relatively low; when When the value is small, it indicates the speaker's characteristics f i The identification of these is quite difficult; therefore, The higher the level, the greater the difficulty. i The lower; The lower the value, the lower the difficulty level. i The higher.

[0077] Accordingly, step S14 specifically includes the following steps:

[0078] S141, determine the first loss function based on the speaker features, the difficulty coefficient corresponding to the speaker features, and the covariance matrix of the label category corresponding to the speaker features;

[0079] By incorporating the difficulty coefficient into the first loss function, it is possible to further set the boundaries between different speaker label categories based on the sample difficulty. Using this first loss function, better speaker features can be learned, thereby further improving the recognition accuracy of the voiceprint recognition model.

[0080] In some implementations, a first loss function can be constructed based on the AM-Softmax loss function. Specifically, the first loss function is:

[0081]

[0082] Where N is the total number of training samples, C is the number of label categories, and f i Let yi be the speaker feature of the i-th training sample, and w be the label category of the i-th training sample. yi Let w be the weight matrix corresponding to the label category yi of the i-th training sample. j Let ∑yi be the weight matrix corresponding to the j-th label category, and ∑yi be the covariance matrix of the label category yi corresponding to the i-th training sample. i Let be the difficulty coefficient of the i-th training sample. m is the first hyperparameter, s is the second hyperparameter, and λ is the third hyperparameter.

[0083] In the first loss function, For the part that enhances the semantic data, sm*dill i The part that represents the difficulty level of integration.

[0084] As one implementation method, after the voiceprint recognition model is trained, the trained voiceprint recognition model can be further optimized. After step S14, the following steps are also included:

[0085] S31, using the speaker encoder of the trained voiceprint recognition model to extract voiceprint features from each speech sample in the training set, and obtain the speaker features corresponding to the speech sample.

[0086] The voice samples include voice data from multiple speaker identities, with each speaker corresponding to multiple voice data and each speaker identity corresponding to a label category.

[0087] S32, calculate the covariance matrix of the corresponding label category based on the multiple speaker features corresponding to each label category, wherein the covariance matrix is ​​used to characterize the semantic transformation direction of all speaker features of the corresponding label category;

[0088] For specific calculation steps, please refer to the description of step S13 above.

[0089] S33, Based on the speaker features and the covariance matrix of the corresponding label categories, establish a Gaussian distribution for the speaker features, and sample speaker enhancement features with infinite semantic transformation from the Gaussian distribution;

[0090] Among them, speaker characteristics f i Construct a Gaussian distribution N(f) centered at the center i ,λ∑ yi ), where f i Let f be the feature of the i-th speaker, and ∑yi be the feature f of the i-th speaker. i The covariance matrix of the corresponding label category yi, where λ is a hyperparameter, f i Along from N(f) i ,λ∑ yi Enhanced features are generated by randomly sampling the speaker's features in each semantic transformation direction, thereby enhancing the speaker's features f. i The enhancement.

[0091] In this embodiment, speaker enhancement features can be obtained by sampling each speaker feature along the direction of semantic transformation using an infinite number of Gaussian distributions.

[0092] Furthermore, following step S22, the following steps are also included:

[0093] S34, construct a training dataset for the label category using multiple speaker features and multiple speaker enhancement features of the label category;

[0094] Each speaker feature was augmented with data. The training data for each label category included the speaker feature and at least one augmented speaker feature corresponding to that speaker feature. The training data for all label categories formed the training dataset.

[0095] S35, The classifier of the voiceprint recognition model is trained using the training dataset and the first loss function to obtain an optimized voiceprint recognition model.

[0096] The training dataset is input into the classifier of the voiceprint recognition model, and the classifier is trained using the first loss function. Since the speaker features of each label category are enhanced, the training samples of the classifier are enriched, which helps to improve the recognition accuracy.

[0097] This application embodiment can achieve voiceprint recognition model training and voiceprint recognition through artificial intelligence. This application embodiment can acquire and process relevant data based on artificial intelligence technology. Artificial intelligence (AI) is the theory, method, technology, and application system that uses digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to obtain optimal results.

[0098] Foundational technologies for artificial intelligence generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing, operating / interactive systems, and mechatronics. AI software technologies mainly encompass computer vision, robotics, biometrics, speech processing, natural language processing, and machine learning / deep learning.

[0099] One embodiment of the application provides a voiceprint recognition method, including the following steps:

[0100] The speech data to be recognized is input into the voiceprint recognition model, and the recognition result of the speech is output.

[0101] The voiceprint recognition model is obtained by training according to the above-mentioned voiceprint recognition model training method.

[0102] Figure 2 This is a schematic diagram of the structure of a voiceprint recognition model training device according to an embodiment of this application. Figure 2As shown, the voiceprint recognition model training device 20 includes: a training sample construction module 21, a voiceprint recognition module 22, a sample enhancement module 23, a loss calculation module 24, and a parameter adjustment module 25. The training sample construction module 21 is used to acquire training samples, which include speech data and corresponding label categories. The voiceprint recognition module 22 is used to input the training samples into the voiceprint recognition model to be trained, obtain speaker features corresponding to each training sample through the speaker encoder of the voiceprint recognition model, and recognize the speaker features through the classifier of the voiceprint recognition model. The sample enhancement module 23 is used to calculate the covariance matrix of the corresponding label category based on multiple speaker features corresponding to each label category, where the covariance matrix is ​​used to characterize the semantic transformation direction of all speaker features of the corresponding label category. The loss calculation module 24 is used to determine a first loss function based on the speaker features and the covariance matrix of the label category corresponding to the speaker features. The parameter adjustment module 25 is used to train the voiceprint recognition model using the first loss function to obtain the trained voiceprint recognition model.

[0103] In one implementation, the loss calculation module 24 is further configured to: calculate the difficulty coefficient of the speaker feature based on the weight matrix of the label category corresponding to the speaker feature and the speaker feature, wherein the difficulty coefficient is used to characterize the recognition difficulty of the corresponding speaker feature.

[0104] Accordingly, the loss calculation module 24 is further configured to: determine a first loss function based on the speaker features, the difficulty coefficient corresponding to the speaker features, and the covariance matrix of the label category corresponding to the speaker features.

[0105] As one implementation method, the first loss function is:

[0106]

[0107] Where N is the total number of training samples, C is the number of label categories, and f i Let yi be the speaker feature of the i-th training sample, and w be the label category of the i-th training sample. yi Let w be the weight matrix corresponding to the label category yi of the i-th training sample. j Let ∑yi be the weight matrix corresponding to the j-th label category, and ∑yi be the covariance matrix of the label category yi corresponding to the i-th training sample. i Let be the difficulty coefficient of the i-th training sample. m is the first hyperparameter, s is the second hyperparameter, and λ is the third hyperparameter.

[0108] In one implementation, the sample augmentation module 23 is further configured to: obtain the category mean center μ of the label category based on the speaker features of multiple training samples corresponding to the label category. j , Among them, f k Let M be the speaker feature of the k-th training sample in the j-th label category, and M be the number of training samples in the j-th label category; based on the category mean center and all speaker features of the label category, obtain each matrix element ∑j(m, n) in the covariance matrix of the label category. Where ∑j(m,n) are the matrix elements in the m-th row and n-th column of the covariance matrix ∑j. and Let be the speaker features of the k-th training sample in the j-th label category, representing the values ​​in the m-th and n-th dimensions, respectively. and The class mean center μ of the j-th label category is respectively j The values ​​in the m-th and n-th dimensions, where D is the dimension of the speaker features.

[0109] As one implementation, the voiceprint recognition model training device 20 further includes an optimization training module, used for: extracting voiceprint features from each speech sample in the training set using the speaker encoder of the trained voiceprint recognition model to obtain the speaker features corresponding to the speech sample; calculating the covariance matrix of the corresponding label category based on multiple speaker features corresponding to each label category, wherein the covariance matrix is ​​used to characterize the semantic transformation direction of all speaker features of the corresponding label category; establishing a Gaussian distribution for the speaker features based on the speaker features and the corresponding label category covariance matrix, and sampling speaker enhancement features with infinite semantic transformation from the Gaussian distribution.

[0110] In one implementation, the optimization training module is further configured to: construct a training dataset for the label category using multiple speaker features and multiple speaker enhancement features of the label category; and train the classifier of the voiceprint recognition model using the training dataset and the first loss function to obtain an optimized voiceprint recognition model.

[0111] Figure 3 This is a schematic diagram of the structure of an electronic device according to an embodiment of this application. Figure 3 As shown, the electronic device 30 includes a processor 31 and a memory 32 coupled to the processor 31.

[0112] The memory 32 stores program instructions for implementing the voiceprint recognition model training method or voiceprint recognition method of any of the above embodiments.

[0113] The processor 31 is used to execute program instructions stored in the memory 32 for voiceprint recognition model training or voiceprint recognition.

[0114] The processor 31 can also be referred to as a CPU (Central Processing Unit). The processor 31 may be an integrated circuit chip with signal processing capabilities. The processor 31 can also be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. A general-purpose processor can be a microprocessor or any conventional processor.

[0115] See Figure 4 , Figure 4 This is a schematic diagram of the structure of a storage medium according to an embodiment of this application. The storage medium of this embodiment stores program instructions 41 capable of implementing all the above methods. The storage medium can be non-volatile or volatile. The program instructions 41 can be stored in the storage medium in the form of a software product, including several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) or processor to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks, or terminal devices such as computers, servers, mobile phones, and tablets.

[0116] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces, or indirect coupling or communication connection between apparatuses or units, and may be electrical, mechanical, or other forms.

[0117] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated units described above can be implemented in hardware or as software functional units. The above are merely embodiments of this application and do not limit the patent scope of this application. Any equivalent structural or procedural transformations made based on the description and drawings of this application, or direct or indirect applications in other related technical fields, are similarly included within the patent protection scope of this application.

[0118] The above are merely preferred embodiments of this application and are not intended to limit this application in any way. Although this application has disclosed preferred embodiments as above, it is not intended to limit this application. Any person skilled in the art can make some modifications or alterations to the above-disclosed technical content to create equivalent embodiments without departing from the scope of the technical solution of this application. Any simple modifications, equivalent changes and alterations made to the above embodiments based on the technical essence of this application without departing from the scope of the technical solution of this application shall still fall within the scope of the technical solution of this application.

Claims

1. A method for training a voiceprint recognition model, characterized in that, include: Obtain training samples, wherein the training samples include speech data and the label categories corresponding to the speech data; The training samples are input into the voiceprint recognition model to be trained. The speaker features corresponding to each training sample are obtained through the speaker encoder of the voiceprint recognition model. The speaker features are then identified through the classifier of the voiceprint recognition model. The covariance matrix of the corresponding label category is calculated based on multiple speaker features corresponding to each label category, wherein the covariance matrix is ​​used to characterize the semantic transformation direction of all speaker features of the corresponding label category; The difficulty coefficient of the speaker feature is calculated based on the weight matrix of the label category corresponding to the speaker feature and the speaker feature itself, wherein the difficulty coefficient is used to characterize the recognition difficulty of the corresponding speaker feature; Based on the speaker features, the difficulty coefficients corresponding to the speaker features, and the covariance matrix of the label categories corresponding to the speaker features, a first loss function is determined, wherein, The first loss function is: in, The total number of training samples. For the number of tag categories, For the first Speaker features of each training sample For the first The label category of each training sample. For the first The label category of each training sample The corresponding weight matrix, For the first The weight matrix corresponding to each label category For the first The label category corresponding to each training sample The covariance matrix, For the first The difficulty coefficient of each training sample. , As the first hyperparameter, This is the second hyperparameter. This is the third hyperparameter; The voiceprint recognition model is trained using a first loss function to obtain the trained voiceprint recognition model.

2. The voiceprint recognition model training method according to claim 1, characterized in that, The step of calculating the covariance matrix of the corresponding label category based on multiple speaker features corresponding to each label category includes: The category mean center of the label category is obtained based on the speaker features of multiple training samples corresponding to the label category. , ,in, For the first The first of the tag categories Speaker features of each training sample For the first The number of training samples for each label category; Based on the category mean center and all speaker features of the label category, obtain each matrix element in the covariance matrix of the label category. , ,in, Covariance matrix The Middle line, number Column matrix elements, and The first The first of the tag categories Speaker features of the training samples in the th training sample peacekeeping The numerical value of the dimension. and The first Center of category means for each label category In the peacekeeping The numerical value of the dimension. This refers to the dimension of speaker characteristics.

3. The voiceprint recognition model training method according to claim 2, characterized in that, After training the voiceprint recognition model using the first loss function to obtain the trained voiceprint recognition model, the method further includes: The speaker encoder of the trained voiceprint recognition model is used to extract voiceprint features from each speech sample in the training set to obtain the speaker features corresponding to the speech sample. The covariance matrix of the corresponding label category is calculated based on multiple speaker features corresponding to each label category, wherein the covariance matrix is ​​used to characterize the semantic transformation direction of all speaker features of the corresponding label category; Based on the speaker features and the covariance matrix of the corresponding label categories, a Gaussian distribution is established for the speaker features, and speaker enhancement features with infinite semantic transformation are sampled from the Gaussian distribution.

4. The voiceprint recognition model training method according to claim 3, characterized in that, After establishing a Gaussian distribution for the speaker features based on the covariance matrix of the speaker features and the corresponding label categories, and sampling speaker enhancement features with infinite semantic transformation from the Gaussian distribution, the method further includes: A training dataset for the label category is constructed using multiple speaker features and multiple speaker enhancement features of the label category; The classifier of the voiceprint recognition model is trained using the training dataset and the first loss function to obtain an optimized voiceprint recognition model.

5. A voiceprint recognition method, characterized in that, include: The speech data to be recognized is input into the voiceprint recognition model, and the recognition result of the speech is output. The voiceprint recognition model is trained and obtained by the voiceprint recognition model training method according to any one of claims 1 to 4.

6. A voiceprint recognition model training device, characterized in that, include: A training sample construction module is used to obtain training samples, wherein the training samples include speech data and the label categories corresponding to the speech data; The voiceprint recognition module is used to input the training samples into the voiceprint recognition model to be trained, obtain the speaker features corresponding to each training sample through the speaker encoder of the voiceprint recognition model, and recognize the speaker features through the classifier of the voiceprint recognition model. The sample augmentation module is used to calculate the covariance matrix of the corresponding label category based on multiple speaker features corresponding to each label category, wherein the covariance matrix is ​​used to characterize the semantic transformation direction of all speaker features of the corresponding label category; it is also used to calculate the difficulty coefficient of the speaker feature based on the weight matrix of the label category corresponding to the speaker feature and the speaker feature, wherein the difficulty coefficient is used to characterize the recognition difficulty of the corresponding speaker feature. The loss calculation module is used to determine a first loss function based on the speaker features, the difficulty coefficient corresponding to the speaker features, and the covariance matrix of the label category corresponding to the speaker features, wherein... The first loss function is: in, The total number of training samples. For the number of tag categories, For the first Speaker features of each training sample For the first The label category of each training sample. For the first The label category of each training sample The corresponding weight matrix, For the first The weight matrix corresponding to each label category For the first The label category corresponding to each training sample The covariance matrix, For the first The difficulty coefficient of each training sample. , As the first hyperparameter, This is the second hyperparameter. This is the third hyperparameter; The parameter adjustment module is used to train the voiceprint recognition model using a first loss function to obtain the trained voiceprint recognition model.

7. An electronic device, characterized in that, The system includes a processor and a memory coupled to the processor, the memory storing program instructions executable by the processor; when the processor executes the program instructions stored in the memory, it implements the voiceprint recognition model training method as described in any one of claims 1 to 4 or the voiceprint recognition method as described in claim 5.

8. A storage medium, characterized in that, The storage medium stores program instructions, which, when executed by a processor, enable the implementation of the voiceprint recognition model training method as described in any one of claims 1 to 4 or the voiceprint recognition method as described in claim 5.

Images