Speaker voiceprint feature extraction method, device, equipment and readable storage medium

Abstract

A method, device, equipment and readable storage medium for extracting speaker voiceprint features, relating to the technical field of voiceprint recognition, including sequentially performing point-by-point convolution, depth convolution and point-by-point convolution processing on input target acoustic features based on a deep separable module to output initial features; performing feature extraction and fusion of different granularities on the initial features through multiple layers of multi-granularity feature fusion layers in a multi-granularity time-delay neural network to output speaker frame-level features; performing global feature processing on the speaker frame-level features based on a statistical pooling layer to output a feature vector with a fixed length; performing low-dimensional mapping processing on the feature vector based on a fully connected layer to extract speaker embedding features. Through this application, while effectively realizing speaker feature extraction, the amount of model parameters and computing resources can be effectively reduced.

Description

Technical Field

[0001] The present application relates to the technical field of voiceprint recognition, and in particular to a method, device, equipment and readable storage medium for extracting speaker voiceprint features. Background Art

[0002] Voiceprint is a feature extracted from speech signals to represent a speaker. Voiceprint recognition can also be called speaker verification (SV), which is a task to verify whether a given speaker is from a registered speaker. It specifically includes the following processes: extracting acoustic features from a speech of indefinite length from a single speaker, then the model extracts speaker features from the acoustic features, and finally uses cosine similarity to calculate the speaker features of different speech to determine whether they belong to the same speaker. Among them, many voiceprint recognition systems are currently based on the identification of a single speaker and are widely used in security identification, smart home, and forensic identification. Multi-speaker voiceprint recognition identifies the identities of different speakers from an audio containing multiple speakers. However, it is actually based on the voiceprint recognition of a single speaker, but it adds more complex pre-processing and post-processing operations on its basis. Therefore, the core of multi-speaker voiceprint recognition is still to extract robust and discriminative speaker features from speech.

[0003] In related technologies, speaker features are mainly extracted by using a time-delay neural network (TDNN) as the backbone network, or by using a residual network (ResNet). Among them, although the TDNN-based model can effectively extract contextual features of different time sequences, it usually requires a large number of model parameters to fully capture frequency domain features; similarly, although the ResNet-based model can model both time domain and frequency domain dimensions, it usually requires a large number of parameters and computing resources to achieve competitive performance. It can be seen that how to effectively reduce the number of parameters and computing resources while effectively realizing speaker feature extraction is a problem that needs to be solved urgently. Summary of the invention

[0004] The present application provides a method, device, equipment and readable storage medium for extracting speaker voiceprint features, which can effectively realize speaker feature extraction while effectively reducing the amount of parameters and computing resources.

[0005] In a first aspect, an embodiment of the present application provides a method for extracting a speaker's voiceprint features, comprising the following steps:

[0006] Based on the depth-wise separable module, the input target acoustic features are sequentially processed by point-wise convolution, depth-wise convolution, and point-wise convolution to output the initial features;

[0007] Extracting and fusing the initial features at different granularities respectively through multiple layers of multi-granularity feature fusion layers in a multi-granularity time-delay neural network to output speaker frame-level features;

[0008] Performing global feature processing on the speaker frame-level features based on a statistical pooling layer to output a feature vector with a fixed length;

[0009] The feature vector is subjected to low-dimensional mapping processing based on a fully connected layer to extract speaker embedding features.

[0010] In combination with the first aspect, in one implementation, the depth-separable module sequentially performs point-by-point convolution, depth-wise convolution, and point-by-point convolution processing on the input target acoustic features to output initial features, including:

[0011] Based on the depthwise separable module, performing a first point-by-point convolution, batch normalization, and linear rectification process on the target acoustic feature to obtain a first feature;

[0012] Based on the depth-separable module, perform depth convolution, batch normalization and linear rectification on the first feature to obtain a second feature;

[0013] Performing a second point-wise convolution and batch normalization process on the second feature based on the depth-wise separable module to obtain a third feature;

[0014] The third feature and the target acoustic feature are residually connected based on the depthwise separable module to output an initial feature.

[0015] In combination with the first aspect, in one implementation, the expression of the initial feature is:

[0016]

[0017] In the formula, represents the initial features, represents the target acoustic feature, represents the first point-by-point convolution, BN represents batch normalization, ReLU represents linear rectification unit, represents the depthwise convolution, represents the second point-wise convolution.

[0018] In combination with the first aspect, in one embodiment, the multi-granularity time-delay neural network includes multiple multi-granularity feature fusion modules in sequence and each multi-granularity feature fusion module includes a different number of multi-granularity feature fusion layers, so that each multi-granularity feature fusion layer extracts features of different granularities.

[0019] In combination with the first aspect, in one implementation, each layer of the multi-granularity feature fusion layer includes a first processing module, a time-delay neural network module, a phoneme-level feature pooling module, a squeeze excitation module, and a second processing module, and each layer of the multi-granularity feature fusion layer extracts features of different granularities, including:

[0020] The first processing module performs one-dimensional convolution processing on the input target features to obtain preliminary features;

[0021] The time-delay neural network module performs feature extraction on the preliminary features at different time steps to obtain a fourth feature, and the phoneme-level feature pooling module performs a phoneme-level pooling operation on the preliminary features to obtain a fifth feature;

[0022] The squeezing excitation module fuses the fourth feature and the fifth feature to obtain a fused feature;

[0023] The second processing module performs one-dimensional convolution processing on the fused features and performs residual processing with the target features to obtain features that fuse different granularities.

[0024] In combination with the first aspect, in one implementation, the squeezing excitation module fuses the fourth feature and the fifth feature to obtain a fused feature, including:

[0025] The extrusion excitation module splices the fourth feature and the fifth feature to obtain a spliced ​​feature;

[0026] The squeezing excitation module linearly transforms and activates the splicing features to obtain attention weights;

[0027] The squeeze excitation module performs a dot multiplication on the splicing feature through the attention weight to obtain a fusion feature.

[0028] In combination with the first aspect, in one implementation, the expression of the attention weight is:

[0029]

[0030] In the formula, represents the attention weight, represents the Sigmoid activation function, represents the linear rectification activation function, Represents the fourth feature, Represents the fifth characteristic, Represents the splicing feature, and represents a linear transformation, and Represents the bias term.

[0031] In a second aspect, an embodiment of the present application provides a device for extracting a speaker's voiceprint features, comprising:

[0032] A depthwise separable module, which is used to perform point-wise convolution, depth-wise convolution, and point-wise convolution on the input target acoustic features in sequence to output initial features;

[0033] A multi-granularity time-delay neural network, wherein the multi-layer multi-granularity feature fusion layers in the multi-granularity time-delay neural network are respectively used to extract and fuse features of different granularities on the initial features to output speaker frame-level features;

[0034] A statistical pooling layer, which is used to perform global feature processing on the speaker frame-level features to output a feature vector with a fixed length;

[0035] A fully connected layer is used to perform low-dimensional mapping processing on the feature vector to extract speaker embedding features.

[0036] In a third aspect, an embodiment of the present application provides a speaker voiceprint feature extraction device, the speaker voiceprint feature extraction device comprising a processor, a memory, and a speaker voiceprint feature extraction program stored in the memory and executable by the processor, wherein when the speaker voiceprint feature extraction program is executed by the processor, the steps of the speaker voiceprint feature extraction method as described above are implemented.

[0037] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, on which a speaker voiceprint feature extraction program is stored. When the speaker voiceprint feature extraction program is executed by a processor, the steps of the speaker voiceprint feature extraction method as described above are implemented.

[0038] The beneficial effects brought by the technical solution provided in the embodiments of the present application include:

[0039] By using point-by-point convolution to increase the dimension of the channel, and then using deep convolution for feature extraction, and then using point-by-point convolution for dimensionality reduction, feature extraction in high-dimensional space is achieved, which is conducive to fully capturing the time-frequency domain characteristics of the speech signal, and adjusting the dimension through secondary point-by-point convolution to reduce the number of model parameters and computational complexity; at the same time, through each layer of multi-granularity feature fusion layer in the multi-granularity time-delay neural network to extract features of different granularities and fuse the different granularity information within the layer to obtain the attention weights of features of different granularities, avoiding the single representation of features, enhancing the modeling of time-frequency domain features, and further effectively representing the speaker features; finally, the statistical pooling layer and the fully connected layer are used to extract the speaker embedding features of fixed dimensions. It can be seen that this application can effectively reduce the number of model parameters and computing resources while effectively realizing speaker feature extraction. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] Figure 1 This is a flow chart of an embodiment of a method for extracting speaker voiceprint features in this application;

[0041] Figure 2 Schematic diagram of the overall architecture of the MGFF-TDNN model involved in the embodiment of the present application;

[0042] Figure 3 This is a schematic diagram of the deeply separable module architecture involved in the embodiment of the present application;

[0043] Figure 4 A schematic diagram of t-SNE visualization display of five speaker features extracted by different models involved in the embodiment of the present application;

[0044] Figure 5 This is a schematic diagram of the hardware structure of a speaker voiceprint feature extraction device involved in the embodiment of the present application. DETAILED DESCRIPTION

[0045] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0046] In order to better understand the speaker voiceprint recognition technology, the following will briefly describe and explain the relevant content of speaker voiceprint recognition. It should be understood that speaker voiceprint recognition is often not a separate task, but is combined with other voice tasks to realize an engineering application. Therefore, the text data corresponding to the voice and the timestamp data corresponding to each word can be obtained through the voice recognition model at the front end, and the interactive voice can be sentence-separated with the help of the semantic ability of the language model, and then the corresponding voice is obtained according to the sentence timestamp to perform voiceprint recognition. It can be understood that voiceprint recognition usually includes three parts: speaker voice pre-processing, speaker feature extraction and speaker feature back-end clustering; among them, speaker voice pre-processing requires voice separation of different speakers in long conversation voice to prepare for the subsequent extraction of speaker features. The general processing steps are: voice activity detection (VAD) is performed on the audio data to obtain the human voice segment in the voice; then the human voice segment is further segmented using a fixed length time window and offset for subsequent feature extraction.

[0047] In actual application, this embodiment found that: speech quality has a great impact on the performance of the voiceprint recognition model, that is, the difference in speech quality will affect the result of speech recognition, and then affect the sentence segmentation ability of the language model, so that there is still a high possibility that multiple speakers' expressions appear in the sentence, and then the sentence segmentation result of the language model cannot be directly used for voiceprint recognition; and speaker voiceprint recognition is a multi-stage task, and the processing strategy of each stage will affect the final presentation performance, so it is necessary to take speech quality into consideration, and adopt a suitable segmentation strategy according to the result of speech quality evaluation. Among them, the mean opinion score (MOS) of speech quality evaluation takes a value of 0~5, and generally greater than or equal to 3 is considered to be normal and excellent sound quality. At this time, the text of speech recognition can be segmented, and the speech corresponding to the word segmentation result can be extracted for speaker features. If the MOS value is less than 3, the text of speech recognition is segmented for subsequent feature extraction; the above segmentation strategy can ensure that effective information can be fully extracted in the speaker feature extraction stage and the speech used for feature extraction can be avoided as much as possible.

[0048] For the speaker feature extraction part, it mainly includes acoustic feature extraction and speaker feature extraction. Among them, acoustic feature extraction is to convert audio data into time-frequency domain feature vectors to facilitate subsequent model processing and analysis. Its main processing flow includes pre-emphasis, framing, windowing, short-time Fourier transform, filter, logarithm and discrete cosine transform, etc. It should be noted that the commonly used acoustic features in the field of voiceprint recognition are Mel Frequency Cepstral Coefficients (MFCC) and Filter Bank (FBank) features. The calculation steps of the two are the same in the early stage, but in the final stage, the FBank feature is not subjected to discrete cosine transform, so that FBank contains more abstract speaker features; therefore, this embodiment will use FBank as the acoustic feature for speaker feature extraction.

[0049] For the speaker feature extraction part, speaker features are mainly extracted by using TDNN as the backbone network or ResNet is used to realize speaker feature extraction. Although the TDNN-based model can effectively extract context features of different time series, it usually requires a large number of model parameters to fully capture frequency domain features; similarly, although the ResNet-based model can model both time domain and frequency domain dimensions, it usually requires a large number of parameters and computing resources to obtain competitive performance. It can be seen that there are problems of large parameter quantity and computing resource requirements when implementing speaker feature extraction at present; therefore, this embodiment mainly improves and optimizes the speaker feature extraction part, and then forms a multi-granularity feature fusion model (Multi-granularity feature fusion TDNN, MGFF-TDNN), which is an enhanced architecture based on TDNN, which pays more attention to the context modeling of multi-granularity features, so as to effectively reduce the parameter quantity and computing resources while realizing speaker feature extraction.

[0050] For the back-end clustering of speaker features, after the processing in the first two stages, a large number of speaker features of the same dimension can be obtained from the speaker's audio data, which represent the speaker information in a certain speech segment; and the post-processing stage needs to cluster the speaker feature vectors. Commonly used clustering methods include k-means clustering and spectral clustering. Since the number of classifications will be estimated in the spectral clustering algorithm process, the spectral clustering method is generally used when the number of speakers is unknown.

[0051] It is understandable that the main step of speaker voiceprint recognition technology is the extraction of speaker features by the model. Since the backend needs to perform clustering operations on them, it is necessary to ensure that the extracted speaker features are more robust and discriminative, so that different speaker features form a good clustering effect in high-dimensional space, while reducing the spatial differences of the same speaker, thereby improving the accuracy of the backend clustering operation and the overall performance of voiceprint recognition. Therefore, the following embodiment will explain the specific improvement and optimization methods and principles of the speaker feature extraction part, so as to effectively reduce the number of parameters and computing resources through the speaker voiceprint feature extraction method provided in this embodiment.

[0052] In order to make the objectives, technical solutions and advantages of the present application clearer, the implementation methods of the present application will be further described in detail below with reference to the accompanying drawings.

[0053] In a first aspect, an embodiment of the present application provides a method for extracting speaker voiceprint features.

[0054] In one embodiment, referring to Figure 1 , Figure 1 This is a flow chart of an embodiment of the method for extracting speaker voiceprint features in this application. Figure 1 As shown, the speaker voiceprint feature extraction method includes:

[0055] Step S10: Based on the depthwise separable module, the input target acoustic features are sequentially processed by pointwise convolution, depthwise convolution and pointwise convolution to output initial features.

[0056] For example, see Figure 2 As shown in Table 1, this embodiment directly obtains FBank acoustic features as input for speaker feature extraction, and the dimension of the acoustic features is 80×T, where T represents the time series dimension. It should be noted that Table 1 only lists the main change units in MGFF-TDNN.

[0057] Table 1 MGFF-TDNN model architecture and dimension transformation table

[0058]

[0059] In order to enhance the feature modeling of the model in the frequency domain, a two-dimensional depth-wise separable convolution module (DSM) will be preferably used as a front-end extractor to capture time-frequency domain features; in the DSM, a two-dimensional depth-wise separable residual network layer will be introduced by expanding the channel dimension for the initial modeling of time-frequency domain features.

[0060] It should be understood that since this embodiment needs to perform two-dimensional convolution through DSM, as shown in Table 1, after obtaining the FBank acoustic features, the FBank acoustic features will be expanded and unsqueezed to increase the one-dimensional channel dimension, so that the shape of the features can meet the input format required by DSM; it should be noted that the specific method and principle of the unsqueeze operation are common knowledge in the field, and for the sake of simplicity of description, they are not limited here; then the features output after the unsqueeze operation are expanded in the channel dimension through the convolution module conv_1 to obtain a basic channel dimension (for example, the dimension is 16,32), and then gradually extract richer feature information, providing higher-dimensional and more refined feature representations for subsequent network layers. It can be seen that the convolution of conv_1 can increase the effective feature space dimension for the network without significantly changing the timing or frequency structure, helping the model to more efficiently identify complex timing patterns and subtle changes in subsequent layers. Therefore, this embodiment will use the features output by the conv_1 module as the target acoustic features as the input of the DSM module.

[0061] It should be understood that ordinary TDNN usually performs convolution operations on acoustic features along the time axis to capture the time domain features in the sequence. Although its convolution kernel completely covers the frequency domain features, a large number of filter groups are required to fully model the frequency domain features, which invisibly increases the number of model parameters. In this embodiment, in order to effectively model the time domain and frequency domain features, a two-dimensional deep separable module is added to the front end, see Figure 2 and Figure 3 As shown, the deep separable module uses a multi-layer separable residual structure (i.e., SeparableResNet-1 to SeparableResNet-3, i.e., multi-layer separable residual structure 1 to multi-layer separable residual structure-3) and the microstructure corresponding to each layer of the separable residual structure is an inverted residual block, which extracts features by using a combination of depth-wise convolution and point-wise convolution to reduce the computational complexity of the convolution layer and improve the network's expressiveness without substantially affecting performance. It should be noted that since the workflow and principles of each multi-layer separable residual structure SeparableResNet are the same, for the sake of simplicity of description, the subsequent embodiments will use SeparableResNet-1 as an example to explain the workflow and principles.

[0062] For details, see Figure 3As shown in the figure, in order to increase the feature expression capability and realize feature extraction in high-dimensional space, SeparableResNet-1 first uses point-by-point convolution with a convolution kernel size of 1×1 to expand the channel dimension, so that after point-by-point convolution processing of the target acoustic features, the output channel of this layer is ,in, represents the expansion factor, and its value range is 5 to 10. In this embodiment, it can be preferably set to 6. The number of channels representing the target acoustic features may preferably be 32.

[0063] Then SeparableResNet-1 uses a deep convolution with a convolution kernel size of 3×3 to extract features from the output results of the point-by-point convolution, and the output channel at this time is also ; Finally, SeparableResNet-1 uses a convolution kernel size of 1×1 to perform point-by-point convolution on the output result of the deep convolution and then outputs the initial features, that is, the output channel at this time becomes It can be seen that this embodiment realizes feature extraction in high-dimensional space, which is conducive to comprehensively capturing the time-frequency domain characteristics of the speech signal, and adjusts the dimension through secondary point-by-point convolution to ensure that the model parameters and computational complexity are maintained within an appropriate range.

[0064] Further, in one embodiment, the depth-separable module sequentially performs point-by-point convolution, depth-wise convolution and point-by-point convolution processing on the input target acoustic features to output initial features, including:

[0065] Based on the depthwise separable module, performing a first point-by-point convolution, batch normalization, and linear rectification process on the target acoustic feature to obtain a first feature;

[0066] Based on the depth-separable module, perform depth convolution, batch normalization and linear rectification on the first feature to obtain a second feature;

[0067] Performing a second point-wise convolution and batch normalization process on the second feature based on the depth-wise separable module to obtain a third feature;

[0068] Based on the depth-separable module, the third feature and the target acoustic feature are residually connected to output an initial feature; wherein the expression of the initial feature is:

[0069]

[0070] In the formula, represents the initial features, represents the target acoustic feature, represents the first point-by-point convolution, BN represents batch normalization, ReLU represents linear rectification unit, represents the depthwise convolution, represents the second point-wise convolution.

[0071] For example, it should be noted that the input dimensions of the deep separable module are: [batch_size, channel, feature_dim, time] ([B, C, F, T]), that is, batch, channel, feature and time series dimensions, which are used to perform two-dimensional convolution operations on the target acoustic features; specifically, first, in order to increase the feature expression capability and realize feature extraction in high-dimensional space, point-by-point convolution will be used to expand the channel dimension, that is, SeparableResNet-1 will perform two-dimensional convolution operations on the target acoustic features. Perform the first point-wise convolution , and perform batch normalization BN and linear rectification on the first point-by-point convolution result Processing to obtain the first feature; then use deep convolution to extract features, that is, SeparableResNet-1 performs deep convolution on the first feature Processing, and batch normalization BN and linear rectification of the deep convolution results Processing to obtain the second feature; then use point-by-point convolution to restore the dimension, that is, SeparableResNet-1 performs a second point-by-point convolution on the second feature , and the third feature is obtained by batch normalization BN processing on the second point-by-point convolution result. Here, the dimension is adjusted by secondary point-by-point convolution to ensure that the model parameters and computational complexity are maintained within an appropriate range; finally, the third feature and the target acoustic feature are After residual connection, the initial features can be output Based on this, the following expression can be used to describe the initial features The specific implementation process:

[0072]

[0073] In the formula, It can be seen that this embodiment realizes feature extraction in a high-dimensional space, which is conducive to fully capturing the time-frequency domain characteristics of the speech signal.

[0074] Step S20: extracting and fusing features of different granularities on the initial features respectively through multiple layers of multi-granularity feature fusion layers in the multi-granularity time-delay neural network to output frame-level features of the speaker; wherein the multi-granularity time-delay neural network includes a plurality of multi-granularity feature fusion modules in sequence and each multi-granularity feature fusion module includes a different number of multi-granularity feature fusion layers, so that each multi-granularity feature fusion layer extracts features of different granularities.

[0075] Exemplarily, it should be understood that from the perspective of application, a major challenge faced by speaker verification tasks is speaker modeling for short speech (usually only 3 to 5 seconds). Unlike speaker modeling for long speech, speaker modeling based on short speech should pay more attention to the detailed information of speech, that is, more fine-grained information; therefore, under the same method, the probability of extracting rich information from long speech is greater, so that people ignore the importance of fine granularity and focus on whether there is enough context. Based on this, this embodiment proposes a multi-granularity temporal delay neural network (M-TDNN) to extract feature information of different granularities and fuse the features together, thereby obtaining important features of different granularities. It should be noted that the M-TDNN provided in this embodiment is different from the multi-scale feature aggregation method mentioned in the past. The multi-scale feature aggregation method pays more attention to the feature fusion between layers, that is, the fusion of context information of different lengths; while M-TDNN emphasizes the fusion between different granularity information within the layer to avoid a single feature representation method.

[0076] For details, see Figure 2 As shown in Table 1, the MGFF-TDNN model also includes M-TDNN, which models the internal dependencies between features of different granularities through M-TDNN, so that the model can further effectively characterize speaker features. Among them, M-TDNN includes multiple multi-granularity feature fusion modules M-TDNN Blocks (i.e., M-TDNN Blocks1 to M-TDNN Blocks3, i.e., multi-granularity feature fusion module-1 to multi-granularity feature fusion module-3), and each M-TDNNBlocks includes a different number (e.g., N) of multi-granularity feature fusion layers, so as to extract features of different granularities through each multi-granularity feature fusion layer, for example, M-TDNN Blocks1 to M-TDNN Blocks3 include 3, 6, and 4 multi-granularity feature fusion layers, respectively, and the output channels of M-TDNN Blocks1 to M-TDNN Blocks3 are 128, 256, and 512, respectively; and the features of different granularities are fused to obtain the attention weights of the features of different granularities, so as to output the speaker frame-level features.

[0077] It should be noted that this embodiment uses a fusion method of one-dimensional convolution and two-dimensional convolution structure to perform feature fusion extraction, that is, a two-dimensional DSM is used in the first half, and a one-dimensional M-TDNN is used in the second half; therefore, a reshape operation will be used in the two-dimensional to one-dimensional adaptation stage, that is, the feature map generated by the DSM module (that is, the initial feature ) is flattened along the channel and frequency dimensions and then input into M-TDNN to reorganize the data shape while keeping the data content unchanged to adapt it to the requirements of the model structure.

[0078] Furthermore, in one embodiment, each layer of the multi-granularity feature fusion layer includes a first processing module, a time-delay neural network module, a phoneme-level feature pooling module, a squeeze excitation module, and a second processing module, and each layer of the multi-granularity feature fusion layer extracts features of different granularities, including:

[0079] The first processing module performs one-dimensional convolution processing on the input target features to obtain preliminary features;

[0080] The time-delay neural network module performs feature extraction on the preliminary features at different time steps to obtain a fourth feature, and the phoneme-level feature pooling module performs a phoneme-level pooling operation on the preliminary features to obtain a fifth feature;

[0081] The squeezing excitation module fuses the fourth feature and the fifth feature to obtain a fused feature;

[0082] The second processing module performs one-dimensional convolution processing on the fused features and performs residual processing with the target features to obtain features that fuse different granularities.

[0083] For example, see Figure 2 As shown in the figure, each multi-granularity feature fusion layer integrates a first processing module including a convolutional layer ConvlD, a batch normalization layer BN and a linear rectifier unit ReLU, a time-delay neural network module TDNN, a phoneme level feature pooling module (Phoneme level pooling, PLP), a squeeze-excitation module (Squeeze-excitation, SE) and a second processing module including ConvlD, BN and ReLU; wherein, the PLP module is used to enhance the fine-grained feature modeling, and the expansion factor of the TDNN module is gradually increased to expand the feature modeling of the temporal context; then the outputs of the PLP module and the TDNN module are spliced ​​and sent to the SE module to model the internal dependencies between features of different granularities, so that the model can further effectively characterize the speaker characteristics.

[0084] It should be noted that the workflow and principle of each multi-granularity feature fusion layer are the same. Therefore, for the sake of brevity of description, the following embodiments only take one of the multi-granularity feature fusion layers as an example to illustrate the workflow and principle.

[0085] Specifically, assuming that the input of a multi-granularity feature fusion layer is the target feature It should be noted that only the input of the first multi-granularity feature fusion layer in M-TDNN Blocks1 is the feature output after the reshape operation, while the input of other multi-granularity feature fusion layers is the output of the previous multi-granularity feature fusion layer; First, the target feature is controlled by the first processing module Perform one-dimensional convolution ConvlD transformation in sequence , batch normalization BN and linear rectification ReLU processing are performed to extract preliminary features, and then the following preliminary features are obtained. :

[0086]

[0087] Then, preliminary characteristics There are two feature extraction branches. The TDNN module on the left extracts dynamic granularity features by controlling the hole coefficient. (i.e. the fourth feature), the PLP module on the right extracts fine-grained features by overlapping sliding time windows (The fifth characteristic):

[0088]

[0089]

[0090] In the formula, represents the feature extraction process of the TDNN module, Represents the phoneme-level pooling process of the PLP module, which extracts features of information of different granularities through these two methods.

[0091] Among them, the TDNN module is a neural network structure used to process time series data. Its core idea is to use the convolution kernel to slide on the time axis to extract features at different time steps, and then the sliding mode of the convolution kernel can be affected by controlling the hole coefficient d in the convolution operation; wherein the hole coefficient d represents the number of time steps skipped by the convolution kernel in the time dimension, so as to capture long-distance dependencies and context information. Therefore, this embodiment will obtain dynamic granularity features and long context requirements based on the stacking of levels and the setting of the hole coefficient.

[0092] Specifically, the TDNN module in this embodiment can be preferably constructed by the convolution module, activation layer and batch normalization layer of the pytorch framework, and its input shape is [B, F, T], and the dilation parameter (i.e., dilated convolution, which is used to control the spacing between convolution kernel elements) is used to control the dilation coefficient d and the padding parameter (which is used to control the output size after the convolution operation, retain boundary information and keep the input and output sizes consistent) is used to set the padding mode, so that the output dimension is consistent with the input dimension, that is, the output shape of the TDNN module is still [B, F, T].

[0093] It can be understood that the time step of speech features is composed of each audio frame, and the frame length is generally 25ms. Since the time span of a phoneme unit generally includes multiple audio frames, this embodiment will use a sliding time window to perform feature extraction operations. Specifically, phoneme-level pooling is introduced in the multi-granularity feature fusion layer, which is implemented using standard maximum pooling on a sliding window to capture fine-grained features; in order to reduce information loss, this embodiment will control the PLP sliding window to have a 50% context overlap, where the size of the sliding window can be determined according to actual needs, such as setting the sliding window size to 8. In order to ensure that the output tensor of the PLP is aligned with the input tensor in the time dimension, this embodiment will replicate the pooled features of the sliding window across the time axis, which will help with subsequent feature fusion, that is, prepare for the fusion of different granularity features output by the subsequent TDNN module and the PLP module.

[0094] Specifically, the input shape of the PLP module is [B, F, T], and one-dimensional maximum pooling (max_pool1d) will be used to pool the initial features. Perform a pooling operation with a pooling window size of w and a step length of stride , where the pooling window size can be determined according to actual needs. For example, if it is set to 8, the output dimension after pooling is [B, F, ],in ; Then the pooled output is unsqueezed to expand a new dimension, that is, from [B, F, ] to [B, F, , 1], and then replicate it w times along this dimension to become [B, F, , w]; finally, the four-dimensional tensor is reshaped into [B, F, T], and only the original length is retained, so that the final output dimension of the PLP module is [B, F, T] to be consistent with the input features.

[0095] After controlling the output dimensions of the TDNN module and the PLP module, the output of the TDNN module (i.e., feature ) and the output of the PLP module (i.e., feature ) to obtain the shape Fusion features ; Finally, the fusion features are processed by the second processing module After one-dimensional convolution ConvlD transformation , BN and ReLU processing to extract features and compare them with the input target features By performing residual connection, we can obtain features that combine different granularities. :

[0096]

[0097] It can be seen that the above processing allows the integration of features with different weights, thereby enhancing the modeling of time-frequency domain features; and in order to control the overall parameter amount and calculation amount of the model, this embodiment first uses a one-dimensional convolutional layer with a context of 1 to reduce the feature dimension, and then restores it to the original feature dimension after the multi-granularity feature fusion operation, and uses jump connections in the entire unit.

[0098] Furthermore, in one embodiment, the squeeze excitation module fuses the fourth feature and the fifth feature to obtain a fused feature, including:

[0099] The extrusion excitation module splices the fourth feature and the fifth feature to obtain a spliced ​​feature;

[0100] The squeeze excitation module linearly transforms and activates the splicing features to obtain an attention weight; wherein the expression of the attention weight is:

[0101]

[0102] In the formula, represents the attention weight, represents the Sigmoid activation function, represents the linear rectification activation function, Represents the fourth feature, Represents the fifth characteristic, Represents the splicing feature, and represents a linear transformation, and represents the bias term;

[0103] The squeeze excitation module performs a dot multiplication on the splicing feature through the attention weight to obtain a fusion feature.

[0104] For example, in this embodiment, after extracting feature information of different granularities through the TDNN module and the PLP module, the SE module will use a certain fusion strategy to fuse the features of the two modules to further obtain important features of different granularities. and Splice and get the splicing feature (that is, the input vector ), and perform two linear transformations on the concatenated features to obtain a set of The corresponding attention weight :

[0105]

[0106] In the formula, represents the Sigmoid function, represents the linear rectification activation function, Represents the feature splicing process. and are bias terms and correspond to the first-layer linear transformation and the second-layer linear transformation respectively, where The first layer of linear transformation Its role is to add an offset to the output data in order to increase the expressive power of the model; it is understandable that when the input is linearly transformed After that, the bias term will be added to the result and after Adjust the activation value of each neuron after nonlinear transformation; The second-level linear transformation Similarly, an offset is added to the output in the second layer to help the model learn nonlinear relationships and improve the ability to fit data; it should be noted that and The value of is usually randomly initialized by deep learning frameworks such as PyTorch and continuously updated during model training. For splicing features Perform point multiplication to get the fusion feature : .

[0107] Step S30: performing global feature processing on the speaker frame-level features based on a statistical pooling layer to output a feature vector with a fixed length.

[0108] For example, see Figure 2As shown, MGFF-TDNN also includes a Statistics Pooling Layer (SP) to convert time series features into fixed-length feature vectors through the Statistics Pooling Layer. It can be understood that the input of the Statistics Pooling Layer is the output of MTDNN Block-3 (i.e., speaker frame-level features), and the shape of the speaker frame-level features output by MTDNN Block-3 is [B, F, T] (i.e., batch, feature, time step dimensions). Therefore, the Statistics Pooling Layer will calculate the global statistical features of the input speaker frame-level features to obtain the mean feature vector and the standard deviation feature vector; then concatenate the feature vectors of the mean and standard deviation to obtain a fixed-length feature vector, i.e., the dimension is , so the output feature shape of the statistical pooling layer is .

[0109] Step S40: performing low-dimensional mapping processing on the feature vector based on a fully connected layer to extract speaker embedding features.

[0110] For example, see Figure 2 As shown, MGFF-TDNN also includes a fully connected layer (FC) to obtain a fixed-dimensional speaker embedding feature through the fully connected layer. Specifically, the fully connected layer receives the fixed-length feature vector output from the statistical pooling layer, maps it to a lower-dimensional space, and performs batch normalization as the final speaker embedding feature; the dimension of the speaker embedding feature is generally set to 128, 192, 256, etc., which can be determined according to actual needs and is not limited here. For example, the speaker embedding feature dimension is set to 192. It should be noted that the specific workflow and principle of the fully connected layer are common knowledge in the field and are not limited here for the sake of brevity of description.

[0111] In summary, in order to solve the problem of how the model can extract discriminative speaker embedding features, this embodiment proposes an MGFF-TDNN model, which first uses a two-dimensional deep separable convolution module DSM as a front-end feature extractor to pre-extract time-frequency domain features to enhance the modeling of time-frequency domain features; secondly, in order to achieve multi-granularity feature fusion, a multi-granularity time-delay neural network M-TDNN is designed to capture information of different granularities by using a time-delay neural network and phoneme-level feature pooling, aiming to obtain more robust speaker embedding features and visualize the discriminability of the features extracted by the model, that is, this embodiment can extract more robust and discriminative speaker embedding features.

[0112] The following embodiments will demonstrate through experiments on the VoxCeleb dataset that the MGFF-TDNN model can still perform well in voiceprint recognition while using fewer parameters, computing resources, and computational complexity.

[0113] First, this embodiment conducts experiments on the open source speaker verification dataset VoxCeleb; for VoxCeleb, this embodiment uses the development set of VoxCeleb2 for training, which includes 5994 speakers and a total of 1092009 voice clips; and uses the development set and test set of VoxCeleb1 for evaluation, which includes three evaluation groups with different difficulty levels: VoxCeleb-O, VoxCeleb-H, and VoxCeleb-E. In addition, in view of the complexity of the acoustic environment, this embodiment also uses the noise datasets MUSAN and RIR to enhance the training data. It should be understood that for speaker verification, there are two common indicators used for model evaluation: equal error rate (EER) and minimum detection cost function (minDCF) with a target probability of 0.01; where equal error rate refers to the recognition performance when the false acceptance rate and the false rejection rate are equal, and the lower the EER, the better the performance of the speaker recognition system.

[0114] Secondly, this embodiment conducts experiments in an environment of Intel(R) Xeon(R) Silver 4108 CPU, 32GB memory and NVIDIA RTXA6000, and uses the 3D-Speaker toolkit to train the proposed MGFF-TDNN model. Among them, this embodiment uses 80-dimensional logarithmic Mel filter bank (FBank) features with a window length of 25 milliseconds and an offset of 10 milliseconds as input acoustic features, and in addition to using a noise data set for enhancement, speed perturbations are applied to the audio, that is, random sampling at rates of 0.9, 1.0 and 1.1, which triples the number of speakers; in the experiment, a stochastic gradient descent (SGD) optimizer is used, with an initial learning rate of 0.1, a momentum of 0.9, and a weight decay setting of 0.0001. At the same time, this embodiment also integrates a cosine annealing scheduler and a linear warm-up scheduler for learning rate scheduling, and the minimum learning rate is set to 1e-4.

[0115] It should be noted that all experiments in this embodiment use the Additive Angular Margin Softmax (AAM-Softmax) loss, and the margin and scale factors are set to 0.2 and 32 respectively; the dimension of the bottleneck layer in the SE block is set to 128, and the last fully connected layer outputs a 192-dimensional speaker embedding feature; in addition, in order to improve training efficiency, this embodiment randomly cuts 3 seconds of clips from each audio to construct training samples. In the evaluation stage, this embodiment uses cosine similarity to calculate the score and the value is not score normalized at the back end; this embodiment reproduces the baseline models: ECAPA-TDNN (which is a variant model of TDNN) and Res2Net, and the ECAPA-TDNN model uses a 512 channel structure, and in order to prove the effectiveness of the MGFF-TDNN model proposed in this embodiment, the parameters and floating-point calculation complexity of the baseline model selected in this embodiment are as close as possible to or within a certain range of the proposed MGFF-TDNN model.

[0116] After conducting experiments based on the above foundation, the MGFF-TDNN proposed in this embodiment is compared with various baseline models based on the experimental results, and the evaluation results shown in Tables 2 to 4 are obtained.

[0117] Table 2 Comparison of EER and MinDCF performance of different network architectures on VoxCeleb1-O, VoxCeleb1-E and VoxCeleb1-H test sets

[0118]

[0119] As can be seen from Table 2, the MGFF-TDNN model proposed in this embodiment is significantly better than the baseline model. Compared with the Res2Net model, the MGFF-TDNN model shows significant improvements in all indicators. Specifically, in the three test sets, its EER achieved relative improvements of 35.0%, 17.3% and 15.5%, and the relative improvements of minDCF were 38.1%, 15.6% and 9.7%, respectively; compared with ECAPA-TDNN, the MGFF-TDNN model achieved better performance with fewer parameters in multiple test sets, especially on VoxCeleb1-O, where EER and minDCF were improved by 13.6% and 23.2%, respectively.

[0120] In the ablation experiment, the DSM module was removed and only TDNN was used for simple size mapping to ensure consistency with the input dimension of the original model. It can be observed from Table 2 that DSM improved EER and minDCF by 21.9% and 26.5% respectively on the VoxCeleb1-O dataset. The experimental results shown in Table 2 show the effectiveness of DSM in extracting time-frequency domain features, and removing the PLP module will lead to performance degradation in the test set, which emphasizes the importance of multi-granularity feature fusion.

[0121] In order to demonstrate the multi-granularity feature fusion capability of the proposed MGFF-TDNN model, this embodiment evaluates the performance of different models in the 3-second and 5-second duration groups on the VoxCeleb test set, and the evaluation results are shown in Table 3.

[0122] Table 3 Performance comparison of different models on the VoxCeleb test set for 3-second and 5-second duration groups

[0123]

[0124] As can be seen from Table 3, the MGFF-TDNN model has achieved excellent performance in most indicators; specifically, in the 3-second duration group, compared with Res2Net, the EER values ​​of MGFF-TDNN in the three test sets were relatively reduced by 24.0%, 14.9%, and 12.0%, and it also showed certain advantages over ECAPA-TDNN.

[0125] In addition, this example also uses t-distributed stochastic neighbor embedding (t-SNE) to visualize speaker embeddings of different models for 5-second speech duration and compare their decoupling capabilities. The decoupling capability results are shown in Figure 2. Figure 4 shown; see Figure 4 As shown, it can be seen that compared with ECAPA-TDNN and Res2Net, the speaker embedding extracted by MGFF-TDNN shows stronger clustering ability in short-duration speech, that is, MGFF-TDNN makes speaker embedding more discriminative.

[0126] It can be seen that the MGFF-TDNN model proposed in this embodiment not only fuses context information of different scales in the vertical direction through module stacking, but also uses different components to achieve feature fusion of multiple granularities in the horizontal direction, which avoids the problem that the model ignores fine-grained features when the network is deepened due to single-granularity extraction. The experimental results show that this embodiment focuses on the extraction of time-frequency domain features and features of different granularities within the layer when extracting speaker embedding features, and effective feature fusion methods play an important role in extracting robust and discriminative speaker features.

[0127] In addition, this embodiment will also discuss the complexity of the Res2Net, ECAPA-TDNN and MGFF-TDNN models, including the parameter quantity Params and floating-point operations (FLOPs). The specific results are shown in Table 4.

[0128] Table 4 Comparison of parameters, FLOPs, and EER of different models on the VoxCeleb1-O test set

[0129]

[0130] From Table 4, we can see that when comparing MGFF-TDNN with Res2Net, although MGFF-TDNN has slightly more parameters, its FLOPs are significantly reduced, and compared with ECAPA-TDNN, MGFF-TDNN has reduced both the number of parameters and FLOPs; and from the EER, it can be shown that MGFF-TDNN outperforms both Res2Net and ECAPA-TDNN models on the VoxCeleb1-O test set. It can be seen that the MGFF-TDNN model will have greater advantages in scenarios with edge devices or limited computing power.

[0131] In a second aspect, an embodiment of the present application also provides a device for extracting speaker voiceprint features.

[0132] In one embodiment, the speaker voiceprint feature extraction device includes:

[0133] A depthwise separable module, which is used to perform point-wise convolution, depth-wise convolution, and point-wise convolution on the input target acoustic features in sequence to output initial features;

[0134] A multi-granularity time-delay neural network, wherein the multi-layer multi-granularity feature fusion layers in the multi-granularity time-delay neural network are respectively used to extract and fuse features of different granularities on the initial features to output speaker frame-level features;

[0135] A statistical pooling layer, which is used to perform global feature processing on the speaker frame-level features to output a feature vector with a fixed length;

[0136] A fully connected layer is used to perform low-dimensional mapping processing on the feature vector to extract speaker embedding features.

[0137] Furthermore, in one embodiment, the depth-separable module is specifically used for:

[0138] Based on the depthwise separable module, performing a first point-by-point convolution, batch normalization, and linear rectification process on the target acoustic feature to obtain a first feature;

[0139] Performing deep convolution, batch normalization, and linear rectification processing on the first feature based on the deep separable module to obtain a second feature;

[0140] Performing a second point-wise convolution and batch normalization process on the second feature based on the depth-wise separable module to obtain a third feature;

[0141] The third feature and the target acoustic feature are residually connected based on the depthwise separable module to output an initial feature.

[0142] Furthermore, in one embodiment, the expression of the initial feature is:

[0143]

[0144] In the formula, represents the initial features, represents the target acoustic feature, represents the first point-by-point convolution, BN represents batch normalization, ReLU represents linear rectification unit, represents the depthwise convolution, represents the second point-wise convolution.

[0145] Furthermore, in one embodiment, the multi-granularity time-delay neural network includes a plurality of multi-granularity feature fusion modules in sequence and each multi-granularity feature fusion module includes a different number of multi-granularity feature fusion layers, so that each multi-granularity feature fusion layer extracts features of different granularities.

[0146] Further, in one embodiment, each multi-granularity feature fusion layer includes a first processing module, a time-delay neural network module, a phoneme-level feature pooling module, a squeeze excitation module, and a second processing module;

[0147] The first processing module is used to perform one-dimensional convolution processing on the input target features to obtain preliminary features;

[0148] The time-delay neural network module is used to perform feature extraction on the preliminary features at different time steps to obtain a fourth feature, and the phoneme-level feature pooling module performs a phoneme-level pooling operation on the preliminary features to obtain a fifth feature;

[0149] The squeeze excitation module is used to fuse the fourth feature and the fifth feature to obtain a fused feature;

[0150] The second processing module is used to perform one-dimensional convolution processing on the fused features and perform residual processing with the target features to obtain features that fuse different granularities.

[0151] Furthermore, in one embodiment, the squeeze excitation module is specifically used for:

[0152] The extrusion excitation module splices the fourth feature and the fifth feature to obtain a spliced ​​feature;

[0153] The squeezing excitation module linearly transforms and activates the splicing features to obtain attention weights;

[0154] The squeeze excitation module performs a dot multiplication on the splicing feature through the attention weight to obtain a fusion feature.

[0155] Furthermore, in one embodiment, the expression of the attention weight is:

[0156]

[0157] In the formula, represents the attention weight, represents the Sigmoid activation function, represents the linear rectification activation function, Represents the fourth feature, Represents the fifth characteristic, Represents the splicing feature, and represents a linear transformation, and Represents the bias term.

[0158] Among them, the functional implementation of each module in the above-mentioned speaker voiceprint feature extraction device corresponds to each step in the above-mentioned speaker voiceprint feature extraction method embodiment, and its functions and implementation processes are no longer repeated here one by one.

[0159] In a third aspect, an embodiment of the present application provides a speaker voiceprint feature extraction device, which may be a personal computer (PC), a laptop computer, a server, or other device with data processing capabilities.

[0160] Reference Figure 5 , Figure 5 Schematic diagram of the hardware structure of the speaker voiceprint feature extraction device involved in the embodiment of the present application. In the embodiment of the present application, the speaker voiceprint feature extraction device may include a processor, a memory, a communication interface and a communication bus.

[0161] The communication bus may be of any type and is used to interconnect the processor, the memory, and the communication interface.

[0162] The communication interface includes input / output (I / O) interface, physical interface and logical interface, etc., which are used to realize the interconnection of devices inside the speaker voiceprint feature extraction device, and the interface used to realize the interconnection between the speaker voiceprint feature extraction device and other devices (such as other computing devices or user devices). The physical interface can be an Ethernet interface, a fiber optic interface, an ATM interface, etc.; the user device can be a display, a keyboard, etc.

[0163] The memory can be various types of storage media, such as random access memory (RAM), read-only memory (ROM), non-volatile RAM (NVRAM), flash memory, optical storage, hard disk, programmable ROM (PROM), erasable PROM (EPROM), electrically erasable PROM (EEPROM), etc.

[0164] The processor may be a general-purpose processor, which may call the speaker voiceprint feature extraction program stored in the memory and execute the speaker voiceprint feature extraction method provided in the embodiment of the present application. For example, the general-purpose processor may be a central processing unit (CPU). The method executed when the speaker voiceprint feature extraction program is called may refer to the various embodiments of the speaker voiceprint feature extraction method of the present application, which will not be described in detail here.

[0165] Those skilled in the art will understand that Figure 5 The hardware structure shown in the figure does not constitute a limitation on the present application, and may include more or less components than shown in the figure, or combine certain components, or arrange the components differently.

[0166] In a fourth aspect, an embodiment of the present application also provides a computer-readable storage medium.

[0167] The readable storage medium of the present application stores a speaker voiceprint feature extraction program, wherein when the speaker voiceprint feature extraction program is executed by a processor, the steps of the speaker voiceprint feature extraction method as described above are implemented.

[0168] Among them, the method implemented when the speaker voiceprint feature extraction program is executed can refer to the various embodiments of the speaker voiceprint feature extraction method of the present application, and will not be repeated here.

[0169] It should be noted that the serial numbers of the above-mentioned embodiments of the present application are only for description and do not represent the advantages or disadvantages of the embodiments.

[0170] The terms "including" and "having" and any variations thereof in the specification and claims of this application and the above-mentioned drawings are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device that includes a series of steps or units is not limited to the listed steps or units, but optionally includes steps or units that are not listed, or optionally includes other steps or units inherent to these processes, methods, products or devices. The terms "first", "second" and "third" are used to distinguish different objects, etc., and do not represent a sequence, nor do they limit "first", "second" and "third" to different types.

[0171] In the description of the embodiments of the present application, "exemplary", "for example" or "for example" are used to indicate examples, illustrations or descriptions. Any embodiment or design described as "exemplary", "for example" or "for example" in the embodiments of the present application should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of words such as "exemplary", "for example" or "for example" is intended to present related concepts in a specific way.

[0172] In the description of the embodiments of the present application, unless otherwise specified, “ / ” means or, for example, A / B can mean A or B; the “and / or” in the text is merely a description of the association relationship of associated objects, indicating that three relationships may exist, for example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. In addition, in the description of the embodiments of the present application, “multiple” refers to two or more than two.

[0173] In some processes described in the embodiments of the present application, multiple operations or steps that appear in a specific order are included, but it should be understood that these operations or steps may not be executed in the order in which they appear in the embodiments of the present application or in parallel, and the sequence number of the operation is only used to distinguish the different operations, and the sequence number itself does not represent any execution order. In addition, these processes may include more or fewer operations, and these operations or steps may be executed in sequence or in parallel, and these operations or steps may be combined.

[0174] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application is essentially or the part that contributes to the prior art can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, disk, CD) as described above, and includes a number of instructions for a terminal device to execute the methods described in each embodiment of the present application.

[0175] The above are only preferred embodiments of the present application, and are not intended to limit the patent scope of the present application. Any equivalent structure or equivalent process transformation made using the contents of the present application specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present application.

Claims

1. A method for extracting speaker voiceprint features, characterized in that: The following steps are involved: Based on the depth-wise separable module, the input target acoustic features are sequentially processed by point-wise convolution, depth-wise convolution, and point-wise convolution to output the initial features; Extracting and fusing the initial features at different granularities respectively through multiple layers of multi-granularity feature fusion layers in a multi-granularity time-delay neural network to output speaker frame-level features; Performing global feature processing on the speaker frame-level features based on a statistical pooling layer to output a feature vector with a fixed length; Performing low-dimensional mapping processing on the feature vector based on a fully connected layer to extract speaker embedding features; The multi-granularity time-delay neural network includes a plurality of multi-granularity feature fusion modules in sequence, and each multi-granularity feature fusion module includes a different number of multi-granularity feature fusion layers, so that each multi-granularity feature fusion layer extracts features of different granularities; Each layer of the multi-granularity feature fusion layer includes a first processing module, a time-delay neural network module, a phoneme-level feature pooling module, a squeeze excitation module, and a second processing module. Each layer of the multi-granularity feature fusion layer extracts features of different granularities, including: The first processing module performs one-dimensional convolution processing on the input target features to obtain preliminary features; The time-delay neural network module performs feature extraction on the preliminary features at different time steps to obtain a fourth feature, and the phoneme-level feature pooling module performs a phoneme-level pooling operation on the preliminary features to obtain a fifth feature; The squeezing excitation module fuses the fourth feature and the fifth feature to obtain a fused feature; The second processing module performs one-dimensional convolution processing on the fused features and performs residual processing with the target features to obtain features that fuse different granularities.

2. The method for extracting speaker voiceprint features according to claim 1, characterized in that: The depth-separable module sequentially performs point-by-point convolution, depth-wise convolution, and point-by-point convolution processing on the input target acoustic features to output initial features, including: Based on the depthwise separable module, performing a first point-by-point convolution, batch normalization, and linear rectification process on the target acoustic feature to obtain a first feature; Based on the depth-separable module, perform depth convolution, batch normalization and linear rectification on the first feature to obtain a second feature; Performing a second point-wise convolution and batch normalization process on the second feature based on the depth-wise separable module to obtain a third feature; The third feature and the target acoustic feature are residually connected based on the depthwise separable module to output an initial feature.

3. The method for extracting speaker voiceprint features according to claim 2, characterized in that: The expression of the initial feature is: In the formula, represents the initial features, represents the target acoustic feature, represents the first point-by-point convolution, BN represents batch normalization, ReLU represents linear rectification unit, represents the depthwise convolution, represents the second point-wise convolution.

4. The method for extracting speaker voiceprint features according to claim 3, characterized in that: The squeezing excitation module fuses the fourth feature and the fifth feature to obtain a fused feature, including: The extrusion excitation module splices the fourth feature and the fifth feature to obtain a spliced ​​feature; The squeezing excitation module linearly transforms and activates the splicing features to obtain attention weights; The squeeze excitation module performs a dot multiplication on the splicing feature through the attention weight to obtain a fusion feature.

5. The method for extracting speaker voiceprint features according to claim 4, characterized in that: The expression of the attention weight is: In the formula, represents the attention weight, represents the Sigmoid activation function, represents the linear rectification activation function, Represents the fourth feature, Represents the fifth characteristic, Represents the splicing feature, and represents a linear transformation, and Represents the bias term.

6. A speaker voiceprint feature extraction device, characterized in that: include: A depthwise separable module, which is used to perform point-wise convolution, depth-wise convolution, and point-wise convolution on the input target acoustic features in sequence to output initial features; A multi-granularity time-delay neural network, wherein the multi-layer multi-granularity feature fusion layers in the multi-granularity time-delay neural network are respectively used to extract and fuse features of different granularities on the initial features to output speaker frame-level features; A statistical pooling layer, which is used to perform global feature processing on the speaker frame-level features to output a feature vector with a fixed length; A fully connected layer, which is used to perform low-dimensional mapping processing on the feature vector to extract speaker embedding features; The multi-granularity time-delay neural network includes a plurality of multi-granularity feature fusion modules in sequence, and each multi-granularity feature fusion module includes a different number of multi-granularity feature fusion layers, so that each multi-granularity feature fusion layer extracts features of different granularities; Each multi-granularity feature fusion layer includes a first processing module, a time-delay neural network module, a phoneme-level feature pooling module, a squeeze excitation module, and a second processing module; The first processing module is used to perform one-dimensional convolution processing on the input target features to obtain preliminary features; The time-delay neural network module is used to perform feature extraction on the preliminary features at different time steps to obtain a fourth feature, and the phoneme-level feature pooling module performs a phoneme-level pooling operation on the preliminary features to obtain a fifth feature; The squeeze excitation module is used to fuse the fourth feature and the fifth feature to obtain a fused feature; The second processing module is used to perform one-dimensional convolution processing on the fused features and perform residual processing with the target features to obtain features that fuse different granularities.

7. A device for extracting speaker voiceprint features, characterized in that: The speaker voiceprint feature extraction device includes a processor, a memory, and a speaker voiceprint feature extraction program stored in the memory and executable by the processor, wherein when the speaker voiceprint feature extraction program is executed by the processor, the steps of the speaker voiceprint feature extraction method according to any one of claims 1 to 5 are implemented.

8. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a speaker voiceprint feature extraction program, wherein when the speaker voiceprint feature extraction program is executed by the processor, the steps of the speaker voiceprint feature extraction method according to any one of claims 1 to 5 are implemented.

Images