Speaking subject recognition method, device and equipment and readable storage medium
By employing a dual-path, multi-feature strategy in speaker recognition, combining a pre-trained model and a high-level acoustic feature extraction model, and adaptively allocating weights for feature fusion, the problem of low accuracy in speaker recognition in existing technologies is solved, achieving a more efficient recognition effect.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-12-11
- Publication Date
- 2026-03-13
AI Technical Summary
In existing speaker recognition methods based on pre-trained models, the mining of speaker-related information in multi-layer general representations is insufficient, resulting in low recognition accuracy.
A dual-path, multi-feature strategy is adopted. A general and robust first feature is extracted through a pre-trained model, and a fine-grained second feature is extracted by combining a high-level acoustic feature extraction model. The fine-grained fusion module adaptively allocates weights and performs feature fusion to extract the speaker vector.
It improves the accuracy of speaker identification, fully explores the multi-layered speaker information in the pre-trained model, and reduces computational overhead.
Smart Images
Figure CN121662051A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of intelligent recognition technology, and in particular to a method, apparatus, device, and readable storage medium for recognizing a speaker. Background Technology
[0002] Currently, speaker identification is mainly based on pre-trained models. Introducing pre-trained models into speaker identification can effectively improve performance. However, the mining of speaker-related information in multi-layer general representations is still insufficient, resulting in low accuracy of speaker identification.
[0003] It is evident that improving the accuracy of speaker identification is a technical problem that urgently needs to be solved by those skilled in the art. Summary of the Invention
[0004] In view of this, the purpose of the present invention is to provide a method, apparatus, device and readable storage medium for identifying a speaker, which solves the technical problem of low accuracy in speaker identification in the prior art.
[0005] To address the aforementioned technical problems, this invention provides a method for speaker recognition, comprising:
[0006] The first feature is extracted from the audio to be identified using a pre-trained model;
[0007] Spectral features are extracted from the audio to be identified, and a second feature is extracted from the spectral features using a high-level acoustic feature extraction model; wherein, the high-level acoustic feature extraction model includes two convolutional neural network models consisting of one-dimensional convolutional layers, modified linear units, and normalization layers;
[0008] The first feature and the second feature are fused to obtain a fused feature, and the speaker vector is extracted from the fused feature.
[0009] The similarity between the speaker vector and the target vector corresponding to each speaker is determined, and the speaker corresponding to the audio to be identified is determined based on the similarity.
[0010] Optionally, a first feature is extracted from the audio to be identified using a pre-trained model, including:
[0011] The initial features are obtained by using the convolutional neural network encoder in the pre-trained model to extract features.
[0012] The initial features are modeled temporally and hierarchically using a predetermined number of deep learning models based on self-attention mechanism in the pre-trained model to obtain the features corresponding to each layer;
[0013] The features corresponding to each layer are fused to obtain the first feature.
[0014] Optionally, the features corresponding to each layer are fused to obtain the first feature, including:
[0015] The features corresponding to each layer are stacked hierarchically to obtain the input features;
[0016] The input features are subjected to average pooling in the time dimension to obtain the global channel features of each layer;
[0017] The global channel features of each layer are compressed using the first fully connected layer to obtain the compressed features.
[0018] The compressed features are processed using a modified linear unit to obtain nonlinear features;
[0019] The nonlinear features are processed using a second fully connected layer to obtain the processed features;
[0020] The processed features are normalized using a normalized exponential function to obtain a weight matrix;
[0021] The first feature is obtained by fusing the input features element-wise using the weight matrix.
[0022] Optionally, determining the similarity between the speaker vector and the target vector corresponding to each speaker, and determining the speaker corresponding to the audio to be identified based on the similarity, includes:
[0023] Obtain the speaker vector determined by the pre-trained model and the high-level acoustic feature extraction model in the speaker recognition model;
[0024] Based on the speaker recognition model, the similarity between the speaker vector and the target vector corresponding to each speaker is determined, and the speaker corresponding to the audio to be recognized is determined based on the similarity; wherein, the target vector corresponding to each speaker is the correspondence between the speaker and the target vector determined during the training process.
[0025] Optionally, before extracting the first feature from the audio to be recognized using a pre-trained model, the following steps are also included:
[0026] The parameters of the pre-trained model in the speaker recognition model to be trained are frozen, the parameters of the high-level acoustic feature extraction model are trained, and the objective function in the speaker recognition model to be trained is trained based on the loss function to obtain the speaker recognition model; wherein, the high-level acoustic feature extraction model is a model that can extract acoustic features with a set maximum dimension, and the objective function is a function that maximizes the angular space interval of the target vectors corresponding to different speakers.
[0027] Optionally, the loss function is an additive angle margin marginal loss function.
[0028] Optionally, the first feature and the second feature are fused to obtain a fused feature, and the speaker vector is extracted from the fused feature, including:
[0029] The fused features are encoded using a frame-level encoder to obtain the encoded features;
[0030] The encoded features are weighted and aggregated using an attention-based statistical pooling layer to obtain aggregated features;
[0031] The aggregated features described in the fully connected layer are processed to obtain the speaker vector.
[0032] The present invention also provides a speaker recognition device, comprising:
[0033] The first feature determination module is used to extract the first feature from the audio to be identified using a pre-trained model;
[0034] The second feature determination module is used to extract spectral features from the audio to be identified, and to extract a second feature from the spectral features using a high-level acoustic feature extraction model; wherein, the high-level acoustic feature extraction model is a convolutional neural network model consisting of two convolutional layers, a modified linear unit, and a normalization layer;
[0035] The speaker vector determination module is used to fuse the first feature and the second feature to obtain a fused feature, and extract the speaker vector from the fused feature;
[0036] The speaker determination module is used to determine the similarity between the speaker vector and the target vector corresponding to each speaker, and to determine the speaker corresponding to the audio to be identified based on the similarity.
[0037] The present invention also provides a speaker recognition device, comprising:
[0038] Memory, used to store computer programs;
[0039] A processor is used to execute the computer program to implement the steps of the above-described speaker recognition method.
[0040] The present invention also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps of the above-described speaker recognition method.
[0041] The present invention also provides a computer program product, including a computer program / instructions, which, when executed by a processor, implement the steps of the above-described speaker recognition method.
[0042] As can be seen, this invention utilizes a pre-trained model to extract a first feature from the audio to be identified; extracts spectral features from the audio to be identified; and utilizes a high-level acoustic feature extraction model to extract a second feature from the spectral features. The high-level acoustic feature extraction model is a convolutional neural network model comprising two convolutional layers, rectified linear units, and normalization layers. The first and second features are fused to obtain a fused feature, and a speaker vector is extracted from the fused feature. The similarity between the speaker vector and the corresponding target vector for each speaker is determined, and the speaker corresponding to the audio to be identified is determined based on the similarity. Compared to the current insufficient mining of speaker-related information, this invention adopts a "dual-path, multi-feature" strategy: one path is based on a pre-trained model to obtain a general and robust first feature, and the other path is based on a high-level acoustic feature extraction model to retain fine-grained second features. Thus, the speaker vector is determined based on the fused feature determined by the first and second features, and the speaker corresponding to the audio to be identified is determined based on this speaker vector, improving the accuracy of speaker identification.
[0043] In addition, the present invention also provides a speaker recognition device, apparatus and readable storage medium, which also have the above-mentioned beneficial effects. Attached Figure Description
[0044] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.
[0045] Figure 1 A flowchart of a speaking subject recognition method provided in an embodiment of the present invention;
[0046] Figure 2 A flowchart illustrating a method for identifying a speaker provided in an embodiment of the present invention;
[0047] Figure 3 A schematic diagram of a speaker recognition architecture provided in an embodiment of the present invention;
[0048] Figure 4 This is a schematic diagram of the structure of a speaking subject recognition device provided in an embodiment of the present invention;
[0049] Figure 5 This is a schematic diagram of the structure of a speaker recognition device provided in an embodiment of the present invention. Detailed Implementation
[0050] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.
[0051] Some terms that appear in the description of the embodiments of this application are subject to the following interpretation:
[0052] Speaker recognition, also known as voiceprint recognition, identifies a speaker by analyzing personalized features in their speech signals. Each person's voice has unique characteristics, primarily determined by the size of their vocal cavity and the way their vocal organs are controlled. Compared to traditional biometric methods such as fingerprint, facial, and iris recognition, speaker recognition offers advantages such as convenient data collection, support for remote authentication, and low cost, making it widely used in various practical scenarios.
[0053] Please refer to Figure 1 , Figure 1 A flowchart illustrating a speaker recognition method provided in an embodiment of the present invention. The method may include:
[0054] S101 uses a pre-trained model to extract the first feature from the audio to be recognized.
[0055] Each step in this embodiment can be executed by a designated electronic device, which can be a server, a portable terminal, or other forms. The pre-trained model in this embodiment is a speech pre-trained model, such as Waveform to Vector 2.0 (wav2vec 2.0), Hidden Unit Bidirectional Encoder Representation (HuBERT), UniSpeech-SAT pre-trained general speech representation with speaker awareness, and WavLM (Wav-Large-Scale Self-Supervised Pre-trained Model for Full-Stack Speech Processing). The pre-trained model utilizes massive amounts of unlabeled audio for self-supervised training to learn richer and more expressive general representations.
[0056] It should be further noted that, based on any of the above embodiments, the extraction of the first feature from the audio to be identified using a pre-trained model may include:
[0057] S1011 uses the convolutional neural network encoder in the pre-trained model to extract features and obtain initial features.
[0058] In this embodiment, in the branch of the pre-trained model, the audio to be recognized is first input to the CNN (neural network) encoder of the pre-trained model to complete downsampling and initial feature extraction.
[0059] S1012, using a pre-set number of deep learning models based on self-attention mechanism in the pre-trained model to perform temporal and hierarchical modeling of the initial features, to obtain the features corresponding to each layer.
[0060] This embodiment uses multi-layer Transformer (a deep learning model based on self-attention mechanism) for temporal and hierarchical modeling to gradually build a high-dimensional hierarchical speech representation in order to capture richer and more speaker-related speech cues.
[0061] S1013, fuse the features corresponding to each layer to obtain the first feature.
[0062] This embodiment does not limit the specific method for fusing the features corresponding to each layer. For example, this embodiment can directly concatenate the features to obtain the first feature; or this embodiment can determine the weights corresponding to the features of each layer and fuse them based on the weights. This embodiment provides a specific method for extracting the first feature, which improves the accuracy of the first feature extraction.
[0063] It should be further explained that, based on any of the above embodiments, the fusion of the features corresponding to each layer to obtain the first feature may include:
[0064] Step 1: Stack the features corresponding to each layer hierarchically to obtain the input features;
[0065] Step 2: Perform average pooling on the input features over time to obtain the global channel features for each layer;
[0066] Step 3: Compress the global channel features of each layer using the first fully connected layer to obtain the compressed features;
[0067] Step 4: Process the compressed features using the modified linear unit to obtain nonlinear features;
[0068] Step 5: Process the nonlinear features using the second fully connected layer to obtain the processed features;
[0069] Step 6: Normalize the processed features using the normalization exponential function to obtain the weight matrix;
[0070] Step 7: After element-wise weighting of the input features using the weight matrix, fuse them to obtain the first feature.
[0071] To improve the utilization efficiency of the multi-layer representations in the pre-trained branches, this embodiment designs a fine-grained fusion module. This module can adaptively assign weights to different layers and different feature channels, thereby enhancing the expression of speaker discrimination cues without significantly increasing computational cost. The fine-grained fusion module consists of an average pooling layer, two fully connected layers, a Rectified Linear Unit (ReLU) activation layer, and a Softmax (normalized exponential function) normalization layer, and is jointly trained with the backbone network in an end-to-end manner. In this fine-grained feature fusion module, the outputs of each Transformer layer (a neural network layer based on a self-attention mechanism) of the pre-trained model are first stacked hierarchically to form the input. Its dimensions are The features of the first layer are: The model has L layers, a sequence length of T, and a feature dimension of D. Then, average pooling is performed on the time dimension to compress temporal information, obtaining the global channel description for each layer, i.e., the compressed intermediate vector. This step preserves the channel statistics of each layer, reducing the computational cost and noise sensitivity of subsequent weight generation. Next, based on the generated weight matrix: first, a fully connected layer is used to compress the dimension D to B to reduce redundancy and computation, and ReLU is introduced to introduce non-linearity; then, a second fully connected layer is used to map the dimension back to D, and Softmax is used for normalization, resulting in a feature dimension of... The weight matrix S is used to characterize the importance of different layers and channels. Finally, S is used to evaluate the original features. Element-wise weighted multiplication (dot product) is performed, and summation is performed across the layer dimensions to obtain the first feature P. This achieves both cross-layer selection (selecting the layer most useful to the speaker) and cross-channel recalibration (amplifying the discriminative channel and suppressing redundant channels). After the Transformer processing in this embodiment, the outputs of each layer are fed into the Fine-grained Fusion Module (FGFM). FGFM effectively aggregates complementary information from different depths and channels by adaptively allocating cross-layer and cross-channel weights, suppressing redundant or noise components, and finally obtaining the first feature from the pre-trained model.
[0072] S102, extract spectral features from the audio to be identified, and extract a second feature from the spectral features using a high-level acoustic feature extraction model; wherein, the high-level acoustic feature extraction model is a convolutional neural network model consisting of two convolutional layers, a modified linear unit and a normalization layer.
[0073] The high-level acoustic feature extraction model in this embodiment is a model for extracting fine-grained spectral information, capable of achieving the highest-dimensional acoustic feature recognition. High-level acoustic features, also known as deep acoustic features or semantic-level acoustic features, refer to feature representations that characterize high-level semantic information in audio signals, obtained through in-depth analysis and abstraction of original or basic acoustic parameters (such as spectrum, Mel frequency cepstral coefficients, pitch, phonemes, etc.). This embodiment can design a CNN feature extraction module (high-level acoustic feature extraction model) to transform and optimize the original acoustic feature FBank (Mel filter bank) (spectral features), making it better integrated with the general representation extracted by the pre-trained model. This module consists of two one-dimensional convolutions, each followed by a ReLU layer and a batch normalization layer. Since the FBank and the pre-trained model representation differ in distribution and dimensionality, direct fusion may lead to training instability or even affect generalization. Therefore, two levels of convolutions are introduced to enhance the complementarity of features from different sources. The first layer primarily addresses the alignment issue. First, the channel dimension of the FBank is expanded to the same size as the pre-trained representation, ensuring direct fusion later. Second, downsampling or upsampling is performed on the time axis (usually using stride or interpolation) to align the durations of the two features, reducing inconsistencies across features. Combined with ReLU and batch normalization, the numerical range is stabilized, mitigating training oscillations caused by distribution differences. The second layer further extracts higher-level speech features based on the initial transformation, making them more consistent with the general representation of the pre-trained model, thereby improving the stability and expressive power of the fused features. Finally, to achieve effective integration of the two features (denoted as the first feature P and the second feature F), learnable weights α and β are introduced, and they are weighted and summed to obtain the final fused feature E. In this embodiment, in the acoustic branch, the original audio is first converted into an FBank (Mel filter bank) to represent the spectral features of speech; then, it is further processed by a lightweight CNN feature extraction module to extract more discriminative higher-level acoustic features F. This branch emphasizes mining traditional acoustic cues with lower computational overhead, forming a feature source that complements the pre-trained branch, which helps improve robustness in multi-speaker and complex acoustic scenarios.
[0074] S103, fuse the first feature and the second feature to obtain the fused feature, and extract the speaker vector from the fused feature.
[0075] This embodiment does not limit the specific method of fusing the first and second features. For example, this embodiment can directly concatenate the first and second features, or perform weighted fusion based on an attention mechanism. This embodiment's fusion of the first and second features ensures that the information from the two branches can fully interact, and the fusion process highlights key speaker cues while maintaining temporal consistency, thereby providing high-quality input for downstream encoding.
[0076] It should be further explained that, based on any of the above embodiments, the above-mentioned fusion of the first feature and the second feature to obtain fused features, and the extraction of the speaker vector from the fused features, may include: encoding the fused features using a frame-level encoder to obtain encoded features; weighting and aggregating the encoded features using an attention statistical pooling layer to obtain aggregated features; and processing the aggregated features using a fully connected layer to obtain the speaker vector. In this embodiment, the fused features are fed into the ECAPA-TDNN frame-level encoder (a time-delay neural network emphasizing channel attention, propagation, and aggregation). The encoder further models in a multi-scale time-frequency context, highlighting stable cues related to identity, and weighting and aggregating the frame-level representation through an attention statistical pooling layer (such as weighted summarization of mean and variance), and then connecting to a fully connected layer to generate a compact speaker embedding vector (speaker vector).
[0077] S104, determine the similarity between the speaker vector and the target vector corresponding to each speaker, and determine the speaker corresponding to the audio to be identified based on the similarity.
[0078] This embodiment does not limit the method for determining similarity. For example, this embodiment may determine similarity based on cosine similarity, or it may determine similarity based on Euclidean distance.
[0079] It should be further explained that, based on any of the above embodiments, the determination of the similarity between the speaker vector and the target vector corresponding to each speaker, and the determination of the speaker corresponding to the audio to be identified based on the similarity, may include: obtaining the speaker vector determined by the pre-trained model and the high-level acoustic feature extraction model in the speaker recognition model; determining the similarity between the speaker vector and the target vector corresponding to each speaker based on the speaker recognition model; and determining the speaker corresponding to the audio to be identified based on the similarity; wherein, the target vector corresponding to each speaker is the correspondence between the speaker and the target vector determined during the training process. In this embodiment, the pre-trained model and the high-level acoustic feature extraction model are fused into the same speaker recognition model, so that the speaker recognition model can be directly trained.
[0080] It should be further explained that, based on any of the above embodiments, before extracting the first feature from the audio to be recognized using the pre-trained model, the method may further include: freezing the parameters of the pre-trained model in the speaker recognition model to be trained, training the parameters of the high-level acoustic feature extraction model, and training the objective function in the speaker recognition model to be trained based on the loss function to obtain the speaker recognition model; wherein, the high-level acoustic feature extraction model is a model capable of extracting acoustic features of a set maximum dimension, and the objective function is a function that maximizes the angular space interval of the target vectors corresponding to different speakers. The loss function in this embodiment is the additive angular margin marginal loss function. This embodiment introduces inter-class angular intervals and compresses intra-class differences, thereby improving discriminability and generalization ability. Its core idea is to introduce an angular margin m between the normalized features and the classification weight vector, and combine it with a scaling factor s to impose a stricter separation constraint on the angular similarity of the correct category, typically in the form of:
[0081] ;
[0082] Where N represents the batch size. Represents the loss function. This represents the angle between a sample and its true class weight. This represents the angle between the feature vector of the i-th sample and the weight vector of the j-th class (j being a non-true class). This objective function encourages greater spacing between the embeddings of different speakers in the angular space. In this embodiment, during overall model training, the parameters of the pre-trained model can be frozen, and only the fine-grained fusion module, the CNN feature extraction module, and the downstream speaker recognition network ECAPA-TDNN (frame-level encoder) can be updated. This significantly reduces the computational overhead during training while ensuring superior recognition performance.
[0083] This invention provides a method for speaker identification, which may include: S101, extracting a first feature from the audio to be identified using a pre-trained model; S102, extracting spectral features from the audio to be identified, and extracting a second feature from the spectral features using a high-level acoustic feature extraction model; wherein the high-level acoustic feature extraction model is a convolutional neural network model consisting of two convolutional layers, rectified linear units, and normalization layers; S103, fusing the first feature and the second feature to obtain a fused feature, and extracting a speaker vector from the fused feature; S104, determining the similarity between the speaker vector and the target vector corresponding to each speaker, and determining the speaker corresponding to the audio to be identified based on the similarity. Compared with the current insufficient mining of speaker-related information, this invention adopts a "dual-path, multi-feature" strategy: one path is based on a pre-trained model to obtain a general and robust first feature, and the other path is based on a high-level acoustic feature extraction model to retain fine-grained second features, thereby determining the speaker vector based on the fused feature determined by the first feature and the second feature, and determining the speaker corresponding to the audio to be identified based on the speaker vector, thus improving the accuracy of speaker identification.
[0084] In recent years, speech pre-trained models such as Waveform to Vector 2.0 (wav2vec 2.0), Hidden Unit Bidirectional Encoder Representation (HuBERT), Speaker-Aware Universal Speech Representation Pre-training (UniSpeech-SAT), and Large-Scale Self-Supervised Pre-trained Models for Full-Stack Speech Processing (WavLM) have been proposed and gradually applied to downstream speech tasks. Pre-trained models utilize massive amounts of unlabeled audio for self-supervised training, learning richer and more expressive universal representations; their overall performance on various tasks of the SuperB benchmark for general speech processing often outperforms traditional handcrafted features. A typical pre-trained model usually consists of a convolutional neural network (CNN) feature encoder and a multi-layer transformer. Early approaches directly replaced handcrafted features with the representations from the last transformer layer and then connected fully connected layers or time-delayed neural networks (TDNNs) downstream, achieving considerable performance improvements.
[0085] While introducing pre-trained models into speaker recognition can effectively improve performance, two challenges remain: First, the mining of speaker-related information in multi-layered general representations is still insufficient; second, existing systems based on pre-trained models often directly discard traditional acoustic features, which can reflect the spectral structure of speech, correspond to the human auditory mechanism, and contain rich individual differences; third, pre-trained models have a huge number of parameters, and speaker recognition systems based on pre-trained models usually require full-parameter fine-tuning training, which undoubtedly increases the computational cost significantly. Therefore, how to better combine the general representations of pre-trained models with acoustic features and reduce model training costs still deserves in-depth research. Based on this, this invention proposes a speaker recognition method that can fully utilize the representational capabilities of different depths of pre-trained models and combine the stability and interpretability of acoustic features to further improve the performance and robustness of speaker recognition systems.
[0086] For a clearer understanding of this invention, please refer to the following details. Figure 2 , Figure 2 A flowchart illustrating a speaker recognition method provided in this embodiment of the invention may specifically include:
[0087] S201. The original audio is first input into the CNN encoder of the pre-trained model to complete downsampling and initial feature extraction, and the initial features are obtained.
[0088] For easier understanding, please refer to Figure 3 , Figure 3 This diagram illustrates a speaker recognition architecture provided in an embodiment of the present invention. It mainly includes a pre-trained model and a high-level acoustic feature extraction component. To improve the utilization efficiency of the multi-layer representations in the pre-trained branches, a fine-grained fusion module is designed. This module can adaptively assign weights to different layers and different feature channels, thereby enhancing the expression of speaker discrimination cues without significantly increasing computational cost. The fine-grained fusion module consists of an average pooling layer, two fully connected layers, a Rectified Linear Unit (ReLU) activation layer, and a Softmax normalization layer. The entire module is jointly trained with the backbone network in an end-to-end manner. A CNN feature extraction module is designed to transform and optimize the original acoustic features (FBank), enabling better integration with the general representations extracted by the pre-trained model. This module consists of two one-dimensional convolutions, each followed by a ReLU layer and a batch normalization layer.
[0089] S202. The initial features are modeled temporally and hierarchically using multi-layer Transformers to obtain the features corresponding to each layer.
[0090] S203. Using the fine-grained fusion module, the features corresponding to each layer are fused by adaptively allocated cross-layer and cross-channel weights to obtain the first feature P of the pre-trained model.
[0091] S204. In the acoustics branch, FBank is used to extract the spectral features of the original audio.
[0092] S205. The spectral features are processed using a lightweight CNN feature extraction module to obtain the second feature F.
[0093] S206. Integrate the first feature P and the second feature F to obtain the fused feature E.
[0094] S207. The fused feature E is processed using the ECAPA-TDNN frame-level encoder to obtain the speaker embedding vector.
[0095] S208. Determine the similarity between the speaker's embedding vector and the target vector corresponding to each speaker, and determine the speaker corresponding to the current original audio.
[0096] The beneficial effects of the embodiments of the present invention may include:
[0097] (1) An efficient multi-feature integration speaker recognition architecture is proposed. The architecture adopts a multi-feature extraction strategy. The pre-trained model provides general speech features, which are integrated with high-level acoustic features in the subsequent network to achieve more discriminative speaker (speaking subject) representation learning.
[0098] (2) A fine-grained feature fusion module was designed to adaptively adjust the feature weights of different layers and channels in the pre-trained model, so as to fully explore the multi-level speaker information contained in the pre-trained model.
[0099] (3) A CNN feature extraction module was also designed to transform and optimize the original acoustic features FBank, so that they can be better integrated with the general representation extracted by the pre-trained model.
[0100] The following describes the speaker recognition device provided in the embodiments of the present invention. The speaker recognition device described below can be referred to in correspondence with the speaker recognition method described above.
[0101] Please refer to the details. Figure 4 , Figure 4 A schematic diagram of a speaker recognition device provided in an embodiment of the present invention may include:
[0102] The first feature determination module 100 is used to extract the first feature from the audio to be identified using a pre-trained model;
[0103] The second feature determination module 200 is used to extract spectral features from the audio to be identified, and to extract a second feature from the spectral features using a high-level acoustic feature extraction model; wherein, the high-level acoustic feature extraction model is a convolutional neural network model consisting of two convolutional layers, a modified linear unit, and a normalization layer.
[0104] The speaker vector determination module 300 is used to fuse the first feature and the second feature to obtain a fused feature, and extract the speaker vector from the fused feature;
[0105] The speaker determination module 400 is used to determine the similarity between the speaker vector and the target vector corresponding to each speaker, and to determine the speaker corresponding to the audio to be identified based on the similarity.
[0106] Furthermore, based on any of the above embodiments, the first feature determination module 100 may include:
[0107] An initial feature determination unit is used to extract features using the convolutional neural network encoder in the pre-trained model to obtain initial features;
[0108] The feature determination unit corresponding to each layer is used to perform temporal and hierarchical modeling of the initial features using a preset number of deep learning models based on self-attention mechanism in the pre-trained model, so as to obtain the features corresponding to each layer;
[0109] The first feature determination unit is used to fuse the features corresponding to each layer to obtain the first feature.
[0110] Furthermore, based on any of the above embodiments, the first feature determination unit may include:
[0111] The input feature determination subunit is used to stack the features corresponding to each layer hierarchically to obtain the input features;
[0112] The global channel feature subunit is used to perform average pooling on the input features in the time dimension to obtain the global channel features of each layer.
[0113] The compressed feature determination subunit is used to compress the global channel features of each layer using the first fully connected layer to obtain the compressed features;
[0114] A nonlinear feature determination unit is used to process the compressed features using a corrected linear unit to obtain nonlinear features;
[0115] The processed feature determination subunit is used to process the nonlinear features using the second fully connected layer to obtain the processed features;
[0116] The weight matrix determines the sub-unit, which is used to normalize the processed features using a normalized exponential function to obtain the weight matrix;
[0117] The first weight determination subunit is used to perform element-wise weighting and fusion of the input features using the weight matrix to obtain the first feature.
[0118] Furthermore, based on any of the above embodiments, the speaking subject determination module 400 includes:
[0119] The speaker vector determination unit is used to obtain the speaker vector determined based on the pre-trained model and the high-level acoustic feature extraction model in the speaker recognition model;
[0120] The speaker identification unit is used to determine the similarity between the speaker vector and the target vector corresponding to each speaker based on the speaker identification model, and to determine the speaker corresponding to the audio to be identified based on the similarity; wherein, the target vector corresponding to each speaker is the correspondence between the speaker and the target vector determined during the training process.
[0121] Furthermore, based on any of the above embodiments, the above-mentioned speaker recognition device may further include:
[0122] The training module is used to freeze the parameters of the pre-trained model in the speaker recognition model to be trained, train the parameters of the high-level acoustic feature extraction model, and train the objective function in the speaker recognition model to be trained based on the loss function to obtain the speaker recognition model; wherein, the high-level acoustic feature extraction model is a model that can extract acoustic features with a set maximum dimension, and the objective function is a function that maximizes the angular space interval of the target vectors corresponding to different speakers.
[0123] Furthermore, based on any of the above embodiments, the loss function is an additive angle margin marginal loss function.
[0124] Furthermore, based on any of the above embodiments, the speaker vector determination module 300 may include:
[0125] The coding feature determination unit is used to encode the fused features using a frame-level encoder to obtain the coding features;
[0126] The aggregated feature determination unit is used to perform weighted aggregation of the encoded features using an attention statistical pooling layer to obtain aggregated features;
[0127] The speaker vector determination unit is used to process the aggregated features of the fully connected layer to obtain the speaker vector.
[0128] It should be noted that the order of the modules and units in the aforementioned speaker recognition device can be changed without affecting the logic.
[0129] An embodiment of the present invention provides a speaker identification device, which may include: a first feature determination module 100, used to extract a first feature from an audio to be identified using a pre-trained model; a second feature determination module 200, used to extract spectral features from the audio to be identified, and extract a second feature from the spectral features using a high-level acoustic feature extraction model; wherein the high-level acoustic feature extraction model is a convolutional neural network model comprising two convolutional layers, a modified linear unit, and a normalization layer; a speaker vector determination module 300, used to fuse the first feature and the second feature to obtain a fused feature, and extract a speaker vector from the fused feature; and a speaker identification module 400, used to determine the similarity between the speaker vector and the target vector corresponding to each speaker, and determine the speaker corresponding to the audio to be identified based on the similarity. Compared with the current insufficient mining of information related to the speaker, this invention adopts a "dual-path, multi-feature" strategy: one path is based on a pre-trained model to obtain a general and robust first feature, and the other path is based on a high-level acoustic feature extraction model to retain a fine-grained second feature. Thus, the speaker vector is determined based on the fused feature determined by the first and second features, and the speaker corresponding to the audio to be identified is determined based on the speaker vector, thereby improving the accuracy of speaker identification.
[0130] The following is a description of a speaker recognition device provided by an embodiment of the present invention. The speaker recognition device described below can be referred to in correspondence with the speaker recognition method described above.
[0131] Please refer to Figure 5 , Figure 5 A schematic diagram of a speaker recognition device provided in an embodiment of the present invention may include:
[0132] Memory 10 is used to store computer programs;
[0133] The processor 20 is used to execute computer programs to implement the aforementioned speaker recognition method.
[0134] The memory 10, processor 20, and communication interface 30 all communicate with each other through the communication bus 40.
[0135] In this embodiment of the invention, the memory 10 is used to store one or more programs. The programs may include program code, which includes computer operation instructions. In this embodiment of the invention, the memory 10 may store programs for implementing the following functions:
[0136] The first feature is extracted from the audio to be identified using a pre-trained model;
[0137] The spectral features of the audio to be identified are extracted, and a second feature is extracted from the spectral features using a high-level acoustic feature extraction model. The high-level acoustic feature extraction model is a convolutional neural network model consisting of two convolutional layers, a modified linear unit, and a normalization layer.
[0138] The first and second features are fused to obtain fused features, and the speaker vector is extracted from the fused features.
[0139] The similarity between the speaker vector and the target vector corresponding to each speaker is determined, and the speaker corresponding to the audio to be identified is determined based on the similarity.
[0140] In one possible implementation, the memory 10 may include a program storage area and a data storage area, wherein the program storage area may store the operating system and applications required for at least one function; and the data storage area may store data created during use.
[0141] Furthermore, memory 10 may include read-only memory and random access memory, providing instructions and data to the processor. A portion of the memory may also include NVRAM. The memory stores operating systems and operating instructions, executable modules, or data structures, or subsets thereof, or extended sets thereof, wherein the operating instructions may include various operating instructions for implementing various operations. The operating system may include various system programs for implementing various basic tasks and handling hardware-based tasks.
[0142] Processor 20 can be a central processing unit (CPU), an application-specific integrated circuit, a digital signal processor, a field-programmable gate array, or other programmable logic device. Processor 20 can be a microprocessor or any conventional processor. Processor 20 can call programs stored in memory 10.
[0143] The communication interface 30 can be an interface for the communication module, used to connect with other devices or systems.
[0144] Of course, it should be noted that, Figure 5 The structure shown does not constitute a limitation on the speaker recognition device in the embodiments of the present invention. In practical applications, the speaker recognition device may include more than Figure 5 More or fewer components as shown, or combinations of certain components.
[0145] The following describes the readable storage medium (i.e., computer-readable storage medium) provided in the embodiments of the present invention. The computer-readable storage medium described below can be referred to in correspondence with the speaker recognition method described above.
[0146] The present invention also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps of the above-described speaker recognition method.
[0147] The computer-readable storage medium may include various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.
[0148] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the apparatus disclosed in the embodiments, since it corresponds to the method disclosed in the embodiments, the description is relatively simple; relevant parts can be referred to in the method section.
[0149] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this invention.
[0150] Finally, it should be noted that in this document, relationships such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus.
[0151] The present invention has provided a detailed description of a method, apparatus, device, and readable storage medium for identifying a speaker. Specific examples have been used to illustrate the principles and implementation methods of the present invention. The descriptions of the above embodiments are only for the purpose of helping to understand the method and core ideas of the present invention. At the same time, those skilled in the art will recognize that there will be changes in the specific implementation methods and application scope based on the ideas of the present invention. Therefore, the content of this specification should not be construed as a limitation of the present invention.
Claims
1. A method for identifying a speaker, characterized in that, include: The first feature is extracted from the audio to be identified using a pre-trained model; Spectral features are extracted from the audio to be identified, and a second feature is extracted from the spectral features using a high-level acoustic feature extraction model; wherein, the high-level acoustic feature extraction model includes two convolutional neural network models consisting of one-dimensional convolutional layers, modified linear units, and normalization layers; The first feature and the second feature are fused to obtain a fused feature, and the speaker vector is extracted from the fused feature. The similarity between the speaker vector and the target vector corresponding to each speaker is determined, and the speaker corresponding to the audio to be identified is determined based on the similarity.
2. The speaker recognition method according to claim 1, characterized in that, Extracting the first feature from the audio to be identified using a pre-trained model includes: The initial features are obtained by using the convolutional neural network encoder in the pre-trained model to extract features. The initial features are modeled temporally and hierarchically using a predetermined number of deep learning models based on self-attention mechanism in the pre-trained model to obtain the features corresponding to each layer; The features corresponding to each layer are fused to obtain the first feature.
3. The speaker recognition method according to claim 2, characterized in that, The features corresponding to each layer are fused to obtain the first feature, including: The features corresponding to each layer are stacked hierarchically to obtain the input features; The input features are subjected to average pooling in the time dimension to obtain the global channel features of each layer; The global channel features of each layer are compressed using the first fully connected layer to obtain the compressed features. The compressed features are processed using a modified linear unit to obtain nonlinear features; The nonlinear features are processed using a second fully connected layer to obtain the processed features; The processed features are normalized using a normalized exponential function to obtain a weight matrix; The first feature is obtained by fusing the input features element-wise using the weight matrix.
4. The speaking subject recognition method according to any one of claims 1 to 3, characterized in that, Determining the similarity between the speaker vector and the target vector corresponding to each speaker, and determining the speaker corresponding to the audio to be identified based on the similarity, includes: Obtain the speaker vector determined by the pre-trained model and the high-level acoustic feature extraction model in the speaker recognition model; Based on the speaker recognition model, the similarity between the speaker vector and the target vector corresponding to each speaker is determined, and the speaker corresponding to the audio to be recognized is determined based on the similarity; wherein, the target vector corresponding to each speaker is the correspondence between the speaker and the target vector determined during the training process.
5. The speaker recognition method according to claim 4, characterized in that, Before extracting the first feature from the audio to be identified using a pre-trained model, the following steps are also included: The parameters of the pre-trained model in the speaker recognition model to be trained are frozen, the parameters of the high-level acoustic feature extraction model are trained, and the objective function in the speaker recognition model to be trained is trained based on the loss function to obtain the speaker recognition model; wherein, the high-level acoustic feature extraction model is a model that can extract acoustic features with a set maximum dimension, and the objective function is a function that maximizes the angular space interval of the target vectors corresponding to different speakers.
6. The speaker recognition method according to claim 5, characterized in that, The loss function is the additive angle margin marginal loss function.
7. The speaker recognition method according to claim 1, characterized in that, The first feature and the second feature are fused to obtain a fused feature, and the speaker vector is extracted from the fused feature, including: The fused features are encoded using a frame-level encoder to obtain the encoded features; The encoded features are weighted and aggregated using an attention-based statistical pooling layer to obtain aggregated features; The aggregated features described in the fully connected layer are processed to obtain the speaker vector.
8. A speaker recognition device, characterized in that, include: The first feature determination module is used to extract the first feature from the audio to be identified using a pre-trained model; The second feature determination module is used to extract spectral features from the audio to be identified, and to extract a second feature from the spectral features using a high-level acoustic feature extraction model; wherein, the high-level acoustic feature extraction model is a convolutional neural network model consisting of two convolutional layers, a modified linear unit, and a normalization layer; The speaker vector determination module is used to fuse the first feature and the second feature to obtain a fused feature, and extract the speaker vector from the fused feature; The speaker determination module is used to determine the similarity between the speaker vector and the target vector corresponding to each speaker, and to determine the speaker corresponding to the audio to be identified based on the similarity.
9. A speaker recognition device, characterized in that, include: Memory, used to store computer programs; A processor for executing the computer program to implement the steps of the speaker recognition method as described in any one of claims 1 to 7.
10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the steps of the speaker recognition method as described in any one of claims 1 to 7.