A method and device for verifying speaker identity in a far-field scene, and an electronic device

Abstract

The present disclosure provides a method and device for verifying the identity of a speaker in a far-field scene, and an electronic device, wherein the method comprises: obtaining target identity information selected by a target user and a voice of the target user; dividing the voice of the target user into a plurality of target voice segments, and obtaining a plurality of target voiceprint feature vectors corresponding to the plurality of target voice segments respectively by using a trained speaker verification model, wherein the trained speaker verification model uses a multi-layer structure of an SE Res-D Block backbone network; and performing similarity comparison between the plurality of target voiceprint feature vectors and a target voiceprint feature space corresponding to the target identity information to verify the identity of the target user. Through the present disclosure, the problem that the related art directly mixes near-field voice data and far-field voice data into a model for learning and training, resulting in low accuracy of the model when performing far-field speaker verification, is solved, and the effect of improving the accuracy of far-field speaker verification is achieved.

Description

Technical Field

[0001] This invention relates to the field of speech recognition technology, and in particular to a method, apparatus, and electronic device for verifying the identity of a speaker in a far-field scenario. Background Technology

[0002] Voiceprint recognition is a technology that extracts the voice features of a speaker to verify their identity. Depending on the task, voiceprint recognition can be divided into speaker verification and speaker identification. Speaker verification involves verifying the speech to determine whether the speaker is the person they claim to be, and then accepting or rejecting the claim.

[0003] However, in far-field speech recognition tasks, due to various noises and reverberation caused by multiple reflections and refractions of sound waves in far-field environments such as living rooms, basements, classrooms, and auditoriums, the voiceprint features in the speech data acquired by the recording device under far-field conditions are masked by reverberation effects, background noise, multipath reflections, and other interference factors. The intra-class and inter-class distances exhibit different distributions than in the near field, which reduces the model's ability to learn voiceprint features.

[0004] For far-field speech processing, front-end processing is the mainstream approach. This involves using signal processing methods or deep learning to manipulate the speech spectrum to reduce interference from factors such as reverberation, echo, and noise. In tasks like far-field speech recognition, front-end processing significantly improves performance. However, in the field of speaker recognition, front-end processing can severely damage the speaker characteristics contained in the speech data, especially with deep learning-based methods. Because deep learning methods have poor interpretability, their alterations to the speech spectrum also have poor interpretability.

[0005] In recent years, the x-vector model and its improved versions have achieved excellent results in speaker verification. Later, voiceprint recognition models, represented by the ECAPA-TDNN model, inherited the system architecture of the x-vector model, improved its topology, and further enhanced its performance in speaker verification.

[0006] The voiceprint recognition model based on the ECAPA-TDNN model is an improvement upon the x-vertor model and is one of the most advanced voiceprint recognition models currently available. It mainly consists of a Squeeze-Excitation (SE)Res2Blocks module, a multi-layer feature aggregation and summation module, and a context- and channel-dependent statistical pooling module. The SE Res2Blocks module comprises a Res2Net block and an SE Net block. The Res2Net block constructs internal multi-layered residual connections, reducing model parameters and improving performance. The SE Net block calculates the average vector of frame-level features across the temporal domain through compression operations, then calculates the weights of each channel through activation operations, and finally multiplies it with the original input to enhance important features while weakening less important ones.

[0007] When training the far-field speaker verification model, near-field and far-field speech data are mixed and fed into the model for learning and training. The loss of the model is calculated based on the true labels of the data, and then the model is iteratively optimized using optimization algorithms.

[0008] Ablation experiments show that in the SE Res2Blocks module of the ECAPA-TDNN model, the SE Net block is the one that truly affects model performance. Models with SE Net blocks show a significant performance improvement compared to those without, but models with Res2Net blocks show only a minor performance improvement. This indicates that this module does not effectively utilize the Res2Net network structure, consuming unnecessary computational and storage resources.

[0009] Meanwhile, experiments have shown that when training a far-field speaker recognition model, if the near-field and far-field speech data are directly mixed and fed into the model for training using the traditional training method, the model can only learn relatively shallow common features between near-field and far-field speech, and cannot learn deeper common features. Therefore, the model still cannot achieve ideal results in far-field speaker verification, and the accuracy is low.

[0010] Currently, no effective solution has been proposed to address the problem that directly mixing near-field and far-field speech data into the model for learning and training, resulting in low accuracy of the model when performing far-field speaker verification. Summary of the Invention

[0011] The purpose of this disclosure is to address the shortcomings of existing technologies by providing a method, apparatus, electronic device, and computer-readable storage medium for verifying speaker identity in far-field scenarios. This aims to at least solve the problem that related technologies directly mix near-field and far-field speech data and feed them into a model for learning and training, resulting in low accuracy of the model when performing far-field speaker verification.

[0012] According to one aspect of this disclosure, a method for verifying speaker identity in a far-field scenario is provided, comprising:

[0013] Obtain the target user's selected identity information and the target user's voice;

[0014] The speech of the target user is divided into multiple target speech segments, and multiple target voiceprint feature vectors corresponding to the multiple target speech segments are obtained using a pre-trained speaker verification model. The pre-trained speaker verification model uses a multi-layer structure of the SE Res-D Block backbone network.

[0015] The similarity of the plurality of target voiceprint feature vectors with the target voiceprint feature space corresponding to the target identity information is compared, and the identity of the target user is verified based on the comparison results. The target voiceprint feature space includes a plurality of voiceprint feature vectors.

[0016] According to another aspect of this disclosure, a speaker identity verification device is provided in a far-field scenario, comprising:

[0017] The first acquisition unit is used to acquire the target identity information selected by the target user and the voice of the target user;

[0018] The second acquisition unit is used to divide the speech of the target user into multiple target speech segments, and use a pre-trained speaker verification model to obtain multiple target voiceprint feature vectors corresponding to the multiple target speech segments respectively. The pre-trained speaker verification model uses a multi-layer structure of SE Res-D Block backbone network.

[0019] The verification unit is used to compare the similarity between the plurality of target voiceprint feature vectors and the target voiceprint feature space corresponding to the target identity information, and to verify the identity of the target user based on the comparison result, wherein the target voiceprint feature space includes a plurality of voiceprint feature vectors.

[0020] According to another aspect of this disclosure, an electronic device is provided, comprising:

[0021] Processor; and

[0022] Stored program memory,

[0023] The program includes instructions that, when executed by the processor, cause the processor to perform the speaker identity verification method in the far-field scenario described in this disclosure.

[0024] According to another aspect of this disclosure, a non-transitory computer-readable storage medium is provided storing computer instructions for causing the computer to perform the speaker identity verification method in the far-field scenario described in this disclosure.

[0025] One or more technical solutions provided in this disclosure, based on a multi-layer speaker verification model with an SE Res-D Block backbone network, can fully learn the voiceprint features of far-field speech data without destroying the voiceprint features. After obtaining the target user's selected identity information and the target user's speech, the pre-trained speaker verification model is used to obtain multiple target voiceprint feature vectors corresponding to the target user's speech. Then, the multiple target voiceprint feature vectors are compared with the target voiceprint feature space corresponding to the target identity information, and the identity of the target user is verified based on the comparison results. This can solve the problem that related technologies directly mix near-field and far-field speech data and put them into the model for learning and training, resulting in low accuracy of the model when performing far-field speaker verification, thus improving the accuracy of far-field voiceprint recognition. Attached Figure Description

[0026] Further details, features, and advantages of this disclosure are disclosed in the following description of exemplary embodiments in conjunction with the accompanying drawings, in which:

[0027] Figure 1 A flowchart is shown of a method for verifying speaker identity in a far-field scenario according to an exemplary embodiment of the present disclosure;

[0028] Figure 2 A schematic diagram of the speaker verification model according to a preferred embodiment of the present disclosure is shown;

[0029] Figure 3 A schematic diagram of the structure of the SE Res_D Block backbone network according to a preferred embodiment of the present disclosure is shown;

[0030] Figure 4 A schematic diagram of the structure of an SE Net according to a preferred embodiment of the present disclosure is shown;

[0031] Figure 5 A schematic diagram of a speaker registration and verification process according to a preferred embodiment of the present disclosure is shown;

[0032] Figure 6 A schematic block diagram of a speaker identity verification device in a far-field scenario according to an exemplary embodiment of the present disclosure is shown;

[0033] Figure 7 A structural block diagram of an exemplary electronic device that can be used to implement embodiments of the present disclosure is shown. Detailed Implementation

[0034] Embodiments of this disclosure will now be described in more detail with reference to the accompanying drawings. While some embodiments of this disclosure are shown in the drawings, it should be understood that this disclosure can be implemented in various forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of this disclosure. It should be understood that the accompanying drawings and embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of protection of this disclosure.

[0035] It should be understood that the steps described in the method embodiments of this disclosure may be performed in different orders and / or in parallel. Furthermore, the method embodiments may include additional steps and / or omit the steps shown. The scope of this disclosure is not limited in this respect.

[0036] The term "comprising" and its variations as used herein are open-ended, meaning "including but not limited to". The term "based on" means "at least partially based on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one additional embodiment"; the term "some embodiments" means "at least some embodiments". Definitions of other terms will be given in the description below. It should be noted that the concepts of "first", "second", etc., used in this disclosure are only used to distinguish different devices, modules, or units, and are not intended to limit the order of functions performed by these devices, modules, or units or their interdependencies.

[0037] It should be noted that the terms "a" and "a plurality of" used in this disclosure are illustrative rather than restrictive, and those skilled in the art should understand that, unless otherwise expressly indicated in the context, they should be understood as "one or more".

[0038] The names of messages or information exchanged between multiple devices in the embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of such messages or information.

[0039] The present disclosure is described below with reference to the accompanying drawings.

[0040] This exemplary embodiment provides a method for verifying speaker identity in a far-field scenario. Figure 1 A flowchart illustrating a speaker identity verification method in a far-field scenario according to an exemplary embodiment of this disclosure is shown, such as... Figure 1 As shown, the method includes the following steps:

[0041] Step S101: Obtain the target identity information selected by the target user and the voice of the target user;

[0042] Step S102: Divide the speech of the target user into multiple target speech segments, and use the trained speaker verification model to obtain multiple target voiceprint feature vectors corresponding to the multiple target speech segments respectively. The trained speaker verification model uses a multi-layer structure of SE Res-D Block backbone network.

[0043] Step S103: Compare the similarity between the plurality of target voiceprint feature vectors and the target voiceprint feature space corresponding to the target identity information, and verify the identity of the target user based on the comparison result, wherein the target voiceprint feature space includes a plurality of voiceprint feature vectors.

[0044] By using the above steps, the multi-layer speaker verification model based on the SE Res-D Block backbone network is used to verify the identity of the target user. This solves the problem that related technologies directly mix near-field and far-field speech data and feed them into the model for learning and training, resulting in low accuracy of the model when performing far-field speaker verification. This improves the accuracy of far-field speaker recognition.

[0045] In some embodiments, the speaker identity verification method in a far-field scenario may further include:

[0046] Obtain the user's identity information and at least one voice message from the user in different scenarios;

[0047] The trained speaker verification model is used to obtain at least one voiceprint feature vector corresponding to each of the at least one speech;

[0048] The at least one voiceprint feature vector is used to form the voiceprint feature space corresponding to the user's identity information;

[0049] The user's identity information is used as registered identity information, and the registered identity information is mapped and stored to its corresponding voiceprint feature space. The registered identity information includes the target identity information.

[0050] In some embodiments, the speaker identity verification method in a far-field scenario may further include:

[0051] Obtain a training dataset, wherein the training dataset includes multiple datasets, the multiple datasets include a near-field speech dataset and a far-field speech dataset, and each dataset includes at least one speech data from at least one speaker;

[0052] Construct a speaker verification model;

[0053] The speaker verification model is trained using the training dataset to obtain the trained speaker verification model, wherein the trained speaker verification model is used to obtain the voiceprint feature vector corresponding to the speech.

[0054] In some embodiments, obtaining the training dataset includes:

[0055] The speech data in the training dataset is augmented to obtain augmented speech data;

[0056] The enhanced speech data is then normalized.

[0057] In some embodiments, constructing the speaker verification model includes:

[0058] Obtain the 80-dimensional Mel spectrogram of the input parameters, and transform the 80-dimensional Mel spectrogram into an 80*T feature map;

[0059] The 80*T feature map is processed by Layer 0 to extract shallow features and then outputs a C*T feature map.

[0060] The feature map of C*T is processed by Layer 1, Layer 02, and Layer 3 to extract deep features, and then connected to the first fully connected layer with output channel C' to obtain the feature map of C'*T.

[0061] The feature map of C'*T is processed by attention statistical pooling layer and then output as a 2*1 feature map.

[0062] The 2*1 feature map is passed through a second fully connected layer with output channel R to obtain an R*1 voiceprint feature vector;

[0063] The speaker verification model includes Layer 0, Layer 1, Layer 02, Layer 3, the first fully connected layer, the attention statistical pooling layer, and the second fully connected layer.

[0064] In some embodiments, training the speaker verification model using the training dataset includes:

[0065] The speaker verification model is trained using the near-field speech dataset to obtain a pre-trained model;

[0066] Fix the other layers of the speaker verification model except for Layer 0, and train Layer 0 using the far-field speech dataset;

[0067] Fix the other layers of the speaker verification model except for the attention statistical pooling layer and the first fully connected layer, and train the attention statistical pooling layer and the first fully connected layer using the far-field speech dataset;

[0068] Fix all layers of the speaker verification model except the second fully connected layer, and train the second fully connected layer using the far-field speech dataset;

[0069] Fix the other layers of the speaker verification model except for Layer 3, and train Layer 3 using the far-field speech dataset;

[0070] Fix the other layers of the speaker verification model except for Layer 2, and train Layer 2 using the far-field speech dataset;

[0071] Fix the other layers of the speaker verification model other than Layer 1, and train Layer 1 using the far-field speech dataset.

[0072] In some embodiments, training the speaker verification model using the training dataset includes:

[0073] Read batch training data of size B and its corresponding data category labels from the training dataset, and transform the batch training data of size B into a feature map of size B*80*T;

[0074] The B*80*T feature map is used as the input to the speaker verification model to obtain a voiceprint feature vector of size B*R*1.

[0075] The loss of the speaker verification model is calculated based on the data category labels corresponding to the batch training data of size B and the voiceprint feature vector of size B*R*1.

[0076] The speaker verification model is iteratively optimized based on the loss of the speaker verification model.

[0077] The embodiments of this application will be described and illustrated below through preferred embodiments.

[0078] This preferred embodiment provides a far-field speaker verification method based on residual networks.

[0079] The residual network-based far-field speaker verification method proposed in this preferred embodiment can be applied to speaker authentication systems under far-field conditions. Application scenarios include authorization for access control of various things in daily life, such as voice-activated TV locks, voice-activated security doors, car voice-activated locks, and parking lot voice authentication. Traditional far-field speaker verification systems based on voice front-end processing often suffer from low accuracy due to interference from noise, echo, reverberation, and front-end processing factors under far-field conditions, which easily destroys voiceprint features. The residual network-based far-field speaker verification method proposed in this preferred embodiment can improve the accuracy of far-field speaker verification without destroying voiceprint features.

[0080] Inspired by classic speaker verification models such as x-vector, ECAPA_TDNN, and residual networks in deep learning, and considering the characteristics of far-field speech, this preferred embodiment proposes a far-field speaker verification algorithm based on residual networks. This preferred embodiment trains the model using the publicly available datasets AISHELL1, AISHELL2, HI-MIA, and Voxceleb1 and Voxceleb2. Considering the noise and reverberation inherent in far-field speech, this preferred embodiment first adds noise and reverberation to the training data using the publicly available noise datasets MUSAN and RIRS_NOISES. Then, this preferred embodiment uses the 80-dimensional Mel-spectrum of the corresponding speech data as the model input. The network model uses a ResNet_D Block network as the backbone network, and employs a bottleneck structure to enhance the learning ability of shallow features. The input data is classified using a loss function to train the model's speaker feature learning ability. Finally, the trained model uses a registration-verification approach to verify unknown speech. Therefore, the main process of this preferred embodiment can be divided into four main stages: data preprocessing, network model construction, model training, and model testing.

[0081] 1. Data Preprocessing Stage

[0082] The dataset used in this preferred embodiment is shown in Table 1:

[0083] Table 1 Dataset used in the invention

[0084] Dataset Name The amount of voice data used Number of speakers included Voxceleb1 148642 1121 Voxceleb2 1092009 5994 FFSVC2020 1416006 120 FFSVC2022 Supplementary Dataset 813643 120

[0085] The training dataset used in this preferred embodiment comprises a total of 3,470,248 speech data points. It includes 148,642 speech data points recorded by 1,211 speakers from the Voxceleb1 dataset, 1,092,009 speech data points recorded by 5,994 speakers from the Voxceleb2 dataset, 1,416,006 utterances recorded by 120 speakers from the FFSVC2020 dataset, and 813,643 utterances recorded by 120 speakers from the FFSVC2022 supplementary dataset. The Voxceleb1 and Voxceleb2 datasets are publicly available near-field speech datasets from Oxford University, while the FFSVC2020 and FFSVC2022 supplementary datasets are publicly available speech datasets from HillShell, and the speech data in these datasets is primarily far-field speech, including speech data recorded at distances of 1m, 3m, and 5m. Furthermore, this preferred embodiment employs various methods to enhance the speech data. The enhanced datasets in this preferred embodiment are derived from the RIRs and MUSAN datasets, both of which are publicly available datasets. The data augmentation methods and probabilities during the enhancement process are shown in Table 2. After speech enhancement, considering the low volume of the speech recorded in the far-field scene and the uneven distribution of features in each frequency segment, this preferred embodiment performs additional normalization processing on the speech data.

[0086] Table 2 Data Augmentation Methods and Probabilities

[0087]

[0088]

[0089] 2. Network Model Construction Stage

[0090] The network model design of this preferred embodiment is inspired by the classic voiceprint recognition models x-vector and ECAPA-TDNN, and improves upon the issue that the performance of the ECAPA-TDNN model is less affected by the Res2Net part in the SE Res2Block module. The specific design is as follows: Figure 2 As shown:

[0091] The model takes an 80-dimensional Mel spectrogram as input. Through pre-emphasis, framing, windowing, Fast Fourier Transform, and Mel filter bank operations, the input speech data is transformed into a feature map of size 80*T, where T is the number of frames. This preferred embodiment uses a 25ms frame window and a 10ms frame shift.

[0092] Layer 0 is the bottleneck structure. The input feature map has a shape of 80*T. To directly obtain more data from the spectrogram, 3*C convolutional channels are used during convolution. Considering the number of model parameters and training difficulty, the number of convolutional channels in the intermediate convolutional layers is 0.5*C. The output layer of Layer 0 has C convolutional channels. The 80-dimensional spectrogram data is processed by Layer 0 to extract shallow features, outputting a feature map of size C*T.

[0093] Layer 1, Layer 2, and Layer 3 use the SE Res_DB Block backbone network structure. The specific structure of SE Res_DB Block is as follows: Figure 3 As shown, the basic structure of SE Res_D consists of a one-dimensional convolutional neural network, the activation function ReLU, and BatchNormalization. The entire structure consists of two parts: the upper part is the ResNet_D network, and the lower part is the SE Net, where the SE Net structure is as follows... Figure 4 As shown, the design of the entire backbone network first uses multi-layer convolutions to extract deep features, then introduces residuals to prevent network degradation. Next, SE Net is used to model the inter-channel dependencies, and the feature response strength between channels is adaptively re-corrected through the network's global loss function. The entire network uses a backbone network to construct a multi-layer deep network, forming the core structure for learning and extracting features in the entire model.

[0094] Layer 1, Layer 2, and Layer 3 use different dilated convolution parameters to extract deep features through multiple convolutions. Inspired by residual networks, a hierarchical residual structure was incorporated into the model design. The input to Layer 3 is the sum of Layer 1 and Layer 2. After obtaining the output of Layer 3, the feature maps from Layer 1, Layer 2, and Layer 3 are concatenated together, and finally, a fully connected layer (e.g., ...) is added to the output channel. Figure 2 The FullyConnected 1) shown above yields a feature map of size C'*T.

[0095] The Attention Stat Pooling layer is an attention-based statistical pooling layer. To enable each frame of the feature map to capture the voiceprint features contained in other frames, the mean and variance data of all frames are added to each frame based on the input C'*T feature map. At this point, the feature map size is 3C'*T. After passing through a linear layer and normalization, an attention weight w of size C'*T is obtained, representing the importance of each frame in each channel.

[0096] The attention weight w and the original input x of the Attentive Stat Pooling layer are used to calculate the weighted average vector according to formulas (1) and (2) respectively. and weighted standard deviation Will and The size of the output feature map after concatenation is 2C'*1.

[0097]

[0098]

[0099] Finally, it passes through a fully connected layer 2 with an output size of R (e.g., ...). Figure 2 As shown in Fully Connected 2), a voiceprint feature embedding vector of size R*1 can be obtained.

[0100] 3. Model Training Phase

[0101] This preferred embodiment proposes a novel model training method for the speaker verification model structure described above. The training method is shown in Table 3.

[0102] Table 3 Training methods for multi-layered speaker verification models

[0103]

[0104] First, this preferred embodiment uses the near-field Voxceleb2 training data containing 5994 speakers as the training set and the Voxceleb1 dataset as the validation set to obtain a model with optimal results for near-field data. Next, the obtained model is used as a pre-trained model, and the model is further trained on the FFSVC2020 training set and the FFSVC2022 supplementary set according to the method in Table 3.

[0105] Each step in the method in Table 3 involves iteratively executing four steps: reading batch training data, extracting the embedding vectors of the data, calculating the model loss, and backpropagating to update the model parameters.

[0106] a. Read batch training speech data or data-augmented speech data of size B, along with the corresponding data category labels. Then, through pre-emphasis, frame segmentation, windowing, fast Fourier transform, and Mel filter bank operations, transform the batch training speech data into a feature map of size B*80*T, which serves as the input to the model;

[0107] b. Input the feature map of the batch training data of size B*80*T into the model. After the model encodes and extracts, a voiceprint feature vector of size B*R*1 can be obtained.

[0108] c. Combine the voiceprint feature vectors of the batch training data output by the model encoding with the corresponding labels of the batch training data to calculate the model loss. This preferred embodiment uses ArcFace Loss as the loss function for model training, and the specific expression is shown in formula (3):

[0109]

[0110] ArcFace loss function on input feature vector x i and the coefficient vector w corresponding to the weight matrix ji An angular interval m is added to the angle θ between the features, and the angle between the deep features and their corresponding weights is penalized in an additive manner, thereby simultaneously enhancing intra-class compactness and inter-class difference, while normalizing the feature vector to a fixed value s.

[0111] The Adam gradient descent optimization algorithm is used to iteratively optimize the entire model based on the model's loss.

[0112] 4. Model Testing Phase

[0113] After obtaining the trained model, speaker registration is required to test a complete speaker verification task. Considering real-world far-field speaker verification scenarios, during registration, speakers need to record their voice and name their identities according to the requirements shown in Table 4. After obtaining the speaker's recorded voice, the system will call the existing model to convert the voice data into the speaker's voiceprint feature vector. The existing voiceprint feature vectors together form the speaker's voiceprint feature space.

[0114] Table 4 Speaker Registration Scenario Requirements

[0115] Registration scenario Recording count Near field (0.1m) 5 items Far field (1m) 10 Far field (3m) 10 Far field (5m) 10

[0116] After obtaining the speaker's voiceprint feature space, the speaker verification stage begins. The speaker selects a pre-registered identity and records using a front-end recording device. The system extracts multiple speech segments from this recording and then uses a model to convert them into multiple voiceprint feature vectors. Finally, the similarity between these voiceprint feature vectors and the voiceprint feature space is compared, and the system decides whether to accept or reject the speaker's identity based on this similarity. The specific process is as follows: Figure 5 As shown.

[0117] Traditional far-field speaker verification systems based on speech front-end processing often suffer from low accuracy due to interference from noise, echo, reverberation, and front-end processing issues in the far-field environment, which can easily destroy speakerprint features. By introducing the multi-layered speaker verification model based on the SE Res-D Block backbone network in this embodiment, the speaker verification model can fully learn the speakerprint features of far-field speech data without destroying them, thereby improving the accuracy of far-field speakerprint recognition.

[0118] By introducing the SE Res-D Block backbone network in this embodiment, the original ECAPA-TDNN model based on the SE Res2Block backbone network can be compressed, which speeds up model training and reduces model storage and computational overhead.

[0119] Furthermore, the multi-layered speaker verification model training method proposed in this embodiment solves the problem that traditional speaker verification model training methods can only learn some shallower common features of near and far field voiceprints, resulting in low model recognition accuracy. By training the overall model with near-field data to obtain a pre-trained model, and then fine-tuning the model layer by layer with mixed near and far field data, the model can learn deeper common features of near and far field voiceprint data, thereby improving the recognition accuracy of the far-field speaker verification model.

[0120] It should be noted that the steps shown in the above process or in the flowchart of the accompanying figures can be executed in a computer system such as a set of computer-executable instructions, and although a logical order is shown in the flowchart, in some cases the steps shown or described may be executed in a different order than that shown here.

[0121] This exemplary embodiment also provides a speaker identity verification device in a far-field scenario, which is used to implement the above embodiments and preferred embodiments, and will not be repeated hereafter. As used below, the terms "module," "unit," "subunit," etc., can refer to a combination of software and / or hardware that performs a predetermined function. Although the device described in the following embodiments is preferably implemented in software, hardware implementation, or a combination of software and hardware, is also possible and contemplated.

[0122] Figure 6 A schematic block diagram of a speaker identity verification device in a far-field scenario according to an exemplary embodiment of the present disclosure is shown, such as... Figure 6 As shown, the device includes:

[0123] The first acquisition unit 61 is used to acquire the target identity information selected by the target user and the voice of the target user;

[0124] The second acquisition unit 62 is used to divide the speech of the target user into multiple target speech segments, and use a pre-trained speaker verification model to obtain multiple target voiceprint feature vectors corresponding to the multiple target speech segments respectively. The pre-trained speaker verification model uses a multi-layer structure of SE Res-D Block backbone network.

[0125] The verification unit 63 is used to compare the similarity between the plurality of target voiceprint feature vectors and the target voiceprint feature space corresponding to the target identity information, and to verify the identity of the target user based on the comparison result, wherein the target voiceprint feature space includes a plurality of voiceprint feature vectors.

[0126] In some embodiments, the device further includes:

[0127] The third acquisition unit is used to acquire the user's identity information and at least one voice message from the user in different scenarios;

[0128] The fourth acquisition unit is used to acquire at least one voiceprint feature vector corresponding to each of the at least one speech using the trained speaker verification model.

[0129] A component unit is used to assemble the at least one voiceprint feature vector into a voiceprint feature space corresponding to the user's identity information;

[0130] The storage unit is used to store the user's identity information as registered identity information and to map and store the registered identity information with its corresponding voiceprint feature space, wherein the registered identity information includes the target identity information.

[0131] In some embodiments, the device further includes:

[0132] The fifth acquisition unit is used to acquire a training dataset, wherein the training dataset includes multiple datasets, including a near-field speech dataset and a far-field speech dataset, and each dataset includes at least one speech data from at least one speaker;

[0133] Construction unit, used to construct speaker verification model;

[0134] The training unit is used to train the speaker verification model using the training dataset to obtain the trained speaker verification model, wherein the trained speaker verification model is used to obtain the voiceprint feature vector corresponding to the speech.

[0135] In some embodiments, the fifth acquisition unit includes:

[0136] The first processing module is used to enhance the speech data in the training dataset to obtain enhanced speech data.

[0137] The second processing module is used to normalize the enhanced speech data.

[0138] In some embodiments, the building unit includes:

[0139] The first acquisition module is used to acquire an 80-dimensional Mel spectrogram of the input parameters and convert the 80-dimensional Mel spectrogram into an 80*T feature map.

[0140] The third processing module is used to output a C*T feature map after the 80*T feature map has been processed by Layer 0 to extract shallow features.

[0141] The fourth processing module is used to extract deep features from the feature map of C*T through Layer 1, Layer 2, and Layer 3, and then connect it to the first fully connected layer with output channel C' to obtain the feature map of C'*T.

[0142] The fifth processing module is used to output a 2*1 feature map after the C'*T feature map is processed by the attention statistical pooling layer and weighted.

[0143] The sixth processing module is used to pass the 2*1 feature map through a second fully connected layer with an output channel of R to obtain an R*1 voiceprint feature vector;

[0144] The speaker verification model includes Layer 0, Layer 1, Layer 02, Layer 3, the first fully connected layer, the attention statistical pooling layer, and the second fully connected layer.

[0145] In some embodiments, the training unit includes:

[0146] The first training module is used to train the speaker verification model using the near-field speech dataset to obtain a pre-trained model.

[0147] The second training module is used to fix the other layers of the speaker verification model except for the Layer 0 layer, and train the Layer 0 layer using the far-field speech dataset.

[0148] The third training module is used to fix the attention statistical pooling layer and the first fully connected layer of the speaker verification model, and train the attention statistical pooling layer and the first fully connected layer using the far-field speech dataset.

[0149] The fourth training module is used to fix the other layers of the speaker verification model except for the second fully connected layer, and to train the second fully connected layer using the far-field speech dataset;

[0150] The fifth training module is used to fix the other layers of the speaker verification model other than the Layer 3 layer, and train the Layer 3 layer using the far-field speech dataset;

[0151] The sixth training module is used to fix the other layers of the speaker verification model other than the Layer 2 layer, and train the Layer 2 layer using the far-field speech dataset;

[0152] The seventh training module is used to fix the other layers of the speaker verification model other than Layer 1, and to train Layer 1 using the far-field speech dataset.

[0153] In some embodiments, the training unit includes:

[0154] The second acquisition module is used to read batch training data of size B and its corresponding data category labels from the training dataset, and convert the batch training data of size B into a feature map of size B*80*T.

[0155] The input module is used to take the B*80*T feature map as the input of the speaker verification model to obtain a voiceprint feature vector of size B*R*1.

[0156] The calculation module is used to calculate the loss of the speaker verification model based on the data category labels corresponding to the batch training data of size B and the voiceprint feature vector of size B*R*1.

[0157] An optimization module is used to iteratively optimize the speaker verification model based on the loss of the speaker verification model.

[0158] It should be noted that the above modules can be functional modules or program modules, and can be implemented through software or hardware. For modules implemented through hardware, the above modules can reside in the same processor; or the above modules can be located in different processors in any combination.

[0159] Exemplary embodiments of this disclosure also provide an electronic device, including: at least one processor; and a memory communicatively connected to the at least one processor. The memory stores a computer program executable by the at least one processor, the computer program being executed by the at least one processor to cause the electronic device to perform a method according to an embodiment of this disclosure.

[0160] Exemplary embodiments of this disclosure also provide a non-transitory computer-readable storage medium storing a computer program, wherein the computer program, when executed by a computer's processor, is used to cause the computer to perform a method according to embodiments of this disclosure.

[0161] Exemplary embodiments of this disclosure also provide a computer program product, including a computer program, wherein, when executed by a processor of a computer, the computer program is used to cause the computer to perform a method according to an embodiment of this disclosure.

[0162] refer to Figure 7 The present invention describes a structural block diagram of an electronic device 700 that can serve as a server or client of the present disclosure, which is an example of a hardware device that can be applied to various aspects of the present disclosure. The electronic device is intended to represent various forms of digital electronic computer devices, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0163] like Figure 7 As shown, the electronic device 700 includes a computing unit 701, which can perform various appropriate actions and processes based on a computer program stored in a read-only memory (ROM) 702 or a computer program loaded from a storage unit 708 into a random access memory (RAM) 703. The RAM 703 may also store various programs and data required for the operation of the device 700. The computing unit 701, ROM 702, and RAM 703 are interconnected via a bus 704. An input / output (I / O) interface 705 is also connected to the bus 704.

[0164] Multiple components in electronic device 700 are connected to I / O interface 705, including: input unit 706, output unit 707, storage unit 708, and communication unit 709. Input unit 706 can be any type of device capable of inputting information to electronic device 700. Input unit 706 can receive input digital or character information and generate key signal inputs related to user settings and / or function control of electronic device. Output unit 707 can be any type of device capable of presenting information and may include, but is not limited to, a display, speaker, video / audio output terminal, vibrator, and / or printer. Storage unit 708 may include, but is not limited to, disk and optical disk. Communication unit 709 allows electronic device 700 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks, and may include, but is not limited to, modems, network cards, infrared communication devices, wireless communication transceivers, and / or chipsets, such as Bluetooth devices, WiFi devices, WiMax devices, cellular communication devices, and / or the like.

[0165] The computing unit 701 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 701 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 701 performs the various methods and processes described above. For example, in some embodiments, speaker identity verification in a far-field scenario can be implemented as a computer software program tangibly contained in a machine-readable medium, such as storage unit 708. In some embodiments, part or all of the computer program can be loaded and / or installed on the electronic device 700 via ROM 702 and / or communication unit 709. In some embodiments, the computing unit 701 can be configured to perform speaker identity verification in a far-field scenario by any other suitable means (e.g., by means of firmware).

[0166] The program code used to implement the methods of this disclosure may be written in any combination of one or more programming languages. This program code may be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing apparatus, such that when executed by the processor or controller, the program code causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0167] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0168] As used in this disclosure, the terms "machine-readable medium" and "computer-readable medium" refer to any computer program product, device, and / or apparatus (e.g., disk, optical disk, memory, programmable logic device (PLD)) for providing machine instructions and / or data to a programmable processor, including machine-readable media that receive machine instructions as machine-readable signals. The term "machine-readable signal" refers to any signal for providing machine instructions and / or data to a programmable processor.

[0169] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device for displaying information to the user (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor); and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0170] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as a data server), or computing systems that include middleware components (e.g., an application server), or computing systems that include frontend components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with embodiments of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., a communication network). Examples of communication networks include local area networks (LANs), wide area networks (WANs), and the Internet.

[0171] Computer systems can include clients and servers. Clients and servers are generally located far apart and typically interact through communication networks. Client-server relationships are created by computer programs running on the respective computers and having a client-server relationship with each other.

Claims

1. A method for verifying speaker identity in a far-field scenario, characterized in that, include: Obtain the training dataset; Construct a speaker verification model; The construction of the speaker verification model includes: acquiring an 80-dimensional Mel spectrogram of input parameters, and converting the 80-dimensional Mel spectrogram into an 80*T feature map, where T is the number of frames; the 80*T feature map is processed by Layer 0 to extract shallow features, and then outputs a C*T feature map, where C is the number of convolutional channels in the output layer of Layer 0; the C*T feature map is processed by Layer 1, Layer 2, and Layer 3 to extract deep features, and then connected to a first fully connected layer with output channels C' to obtain a C'*T feature map; the C'*T feature map is processed by an attention-weighted attention-based attention pooling layer to output a 2*1 feature map; the 2*1 feature map is then ... The feature map of *1 is passed through a second fully connected layer with output channel R to obtain an R*1 voiceprint feature vector; wherein, the speaker verification model includes the Layer0 layer, the Layer1 layer, the Layer2 layer, the Layer3 layer, the first fully connected layer, the attention statistical pooling layer, and the second fully connected layer, the Layer0 layer is a bottleneck structure, and the Layer1 layer, the Layer2 layer, and the Layer3 layer use the SERes_D Block backbone network; The speaker verification model is trained using the training dataset to obtain a trained speaker verification model, wherein the trained speaker verification model is used to obtain the voiceprint feature vector corresponding to the speech. Obtain the target user's selected identity information and the target user's voice; The speech of the target user is divided into multiple target speech segments, and the trained speaker verification model is used to obtain multiple target voiceprint feature vectors corresponding to the multiple target speech segments respectively. The similarity of the plurality of target voiceprint feature vectors with the target voiceprint feature space corresponding to the target identity information is compared, and the identity of the target user is verified based on the comparison results. The target voiceprint feature space includes a plurality of voiceprint feature vectors.

2. The speaker identity verification method in a far-field scenario as described in claim 1, characterized in that, Also includes: Obtain the user's identity information and at least one voice message from the user in different scenarios; The trained speaker verification model is used to obtain at least one voiceprint feature vector corresponding to each of the at least one speech; The at least one voiceprint feature vector is used to form the voiceprint feature space corresponding to the user's identity information; The user's identity information is used as registered identity information, and the registered identity information is mapped and stored to its corresponding voiceprint feature space. The registered identity information includes the target identity information.

3. The speaker identity verification method in a far-field scenario as described in claim 1, characterized in that, The training dataset includes multiple datasets, including near-field speech datasets and far-field speech datasets, each dataset including at least one speech data point from at least one speaker.

4. The speaker identity verification method in a far-field scenario as described in claim 3, characterized in that, The acquisition of the training dataset includes: The speech data in the training dataset is augmented to obtain augmented speech data; The enhanced speech data is then normalized.

5. The speaker identity verification method in a far-field scenario as described in claim 3, characterized in that, The step of training the speaker verification model using the training dataset includes: The speaker verification model is trained using the near-field speech dataset to obtain a pre-trained model; Fix the other layers of the speaker verification model except for Layer 0, and train Layer 0 using the far-field speech dataset; Fix the other layers of the speaker verification model except for the attention statistical pooling layer and the first fully connected layer, and train the attention statistical pooling layer and the first fully connected layer using the far-field speech dataset; Fix all layers of the speaker verification model except the second fully connected layer, and train the second fully connected layer using the far-field speech dataset; Fix the other layers of the speaker verification model except for Layer 3, and train Layer 3 using the far-field speech dataset; Fix the other layers of the speaker verification model except for Layer 2, and train Layer 2 using the far-field speech dataset; Fix the other layers of the speaker verification model other than Layer 1, and train Layer 1 using the far-field speech dataset.

6. The speaker identity verification method in a far-field scenario as described in claim 3, characterized in that, The step of training the speaker verification model using the training dataset includes: Read batch training data of size B and its corresponding data category labels from the training dataset, and transform the batch training data of size B into a feature map of size B*80*T; The B*80*T feature map is used as the input to the speaker verification model to obtain a voiceprint feature vector of size B*R*1. The loss of the speaker verification model is calculated based on the data category labels corresponding to the batch training data of size B and the voiceprint feature vector of size B*R*1. The speaker verification model is iteratively optimized based on the loss of the speaker verification model.

7. A speaker identity verification device in a far-field scenario, characterized in that, include: The training unit is used to acquire the training dataset; Construct a speaker verification model; The speaker verification model is trained using the training dataset to obtain a trained speaker verification model, wherein the trained speaker verification model is used to obtain the speakerprint feature vector corresponding to the speech. The construction of the speaker verification model includes: obtaining an 80-dimensional Mel spectrogram of the input parameters, and converting the 80-dimensional Mel spectrogram into an 80*T feature map, where T is the number of frames; the 80*T feature map is processed by Layer 0 to extract shallow features, and then outputs a C*T feature map, where C is the number of convolutional channels in the output layer of Layer 0; the C*T feature map is processed by Layer 1, Layer 2, and Layer 3 to extract deep features, and then connected to a first fully connected layer with output channel C' to obtain a C'*T feature map; the C'*T feature map is processed by an attention-weighted attention pooling layer to output a 2*1 feature map; the 2*1 feature map is then processed by... The feature map of *1 is passed through a second fully connected layer with output channel R to obtain an R*1 voiceprint feature vector; wherein, the speaker verification model includes the Layer0 layer, the Layer1 layer, the Layer2 layer, the Layer3 layer, the first fully connected layer, the attention statistical pooling layer, and the second fully connected layer, the Layer0 layer is a bottleneck structure, and the Layer1 layer, the Layer2 layer, and the Layer3 layer use the SE Res_D Block backbone network; The first acquisition unit is used to acquire the target identity information selected by the target user and the voice of the target user; The second acquisition unit is used to divide the speech of the target user into multiple target speech segments, and use the trained speaker verification model to obtain multiple target voiceprint feature vectors corresponding to the multiple target speech segments respectively. The verification unit is used to compare the similarity between the plurality of target voiceprint feature vectors and the target voiceprint feature space corresponding to the target identity information, and to verify the identity of the target user based on the comparison result, wherein the target voiceprint feature space includes a plurality of voiceprint feature vectors.

8. An electronic device, characterized in that, include: processor; as well as Stored program memory, The program includes instructions that, when executed by the processor, cause the processor to perform a speaker identity verification method in a far-field scenario according to any one of claims 1-6.

9. A non-transitory computer-readable storage medium storing computer instructions, characterized in that, The computer instructions are used to cause the computer to execute the speaker identity verification method in a far-field scenario according to any one of claims 1-6.

Images