A voiceprint recognition method

Abstract

The application provides a voiceprint recognition method, which comprises the following steps: preprocessing a to-be-recognized voice signal to obtain an effective voice signal; determining a sentence-level voiceprint feature according to the effective voice signal based on a finite difference network; the finite difference network comprises a time-frequency transformation subnetwork and a phase transformation subnetwork, wherein the time-frequency transformation subnetwork is used for performing multi-scale filtering on a frequency domain spectrum of the effective voice signal to obtain a frequency spectrum domain voice signal representation; the phase transformation subnetwork is used for extracting a phase domain voice signal representation of the effective voice signal; and voiceprint recognition is performed according to the sentence-level voiceprint feature. The scheme uses an original audio as an input, obtains speaker identity description information from a frequency spectrum domain and a phase domain, and fuses the information, so that the problem of speaker identity information loss can be avoided, and the precision of voiceprint recognition is improved.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of speech information processing, and particularly relates to a voiceprint recognition method. BACKGROUND

[0002] Voiceprint, like fingerprint, iris and the like, can be used as the biological identity information unique to each person and has the advantage of non-contact collection.

[0003] Voiceprint recognition, also known as speaker recognition, includes voiceprint identification and voiceprint verification and the like, and can be divided into text-dependent and text-independent according to whether it depends on fixed speech semantic information. Voiceprint identification refers to identifying the voice of a speaker with unknown identity information, obtaining the identity information by extracting voiceprint features and comparing the voiceprint features with voiceprint information registered and entered in a voiceprint database, and is a one-to-many task. Voiceprint verification refers to detecting the voice of a speaker with known identity information and comparing the detected voice with information saved in the database for the corresponding speaker to determine whether the declared identity is true, and is a one-to-one task.

[0004] In the existing voiceprint recognition technology, acoustic features based on frequency energy spectrum are used, and the phase information of the speech signal is missing, and research shows that the formant of a speaker appears as a transition on a short-time phase spectrum, and the use of the phase information means that a distorted speech signal is used, which erases part of the pronunciation information related to the speaker, and finally makes the accuracy of voiceprint recognition low. SUMMARY

[0005] The purpose of the embodiments of the present application is to provide a voiceprint recognition method.

[0006] To solve the above technical problems, the embodiments of the present application are implemented in the following way:

[0007] The present application provides a voiceprint recognition method, which comprises:

[0008] Preprocessing a to-be-identified speech signal to obtain an effective speech signal;

[0009] Based on a finite difference network, determining a sentence-level voiceprint feature according to the effective speech signal; the finite difference network comprises a time-frequency transformation sub-network and a phase transformation sub-network, wherein the time-frequency transformation sub-network is used for performing multi-scale filtering on the frequency domain spectrum of the effective speech signal to obtain a frequency spectrum domain speech signal representation; and the phase transformation sub-network is used for extracting a phase domain speech signal representation of the effective speech signal;

[0010] Performing voiceprint recognition according to the sentence-level voiceprint feature.

[0011] In one of the embodiments, the time domain transformation sub-network comprises a stacked one-dimensional convolution and an activation function.

[0012] The first convolution layer in the stacked one-dimensional convolution filters adjacent frequency bands of the effective speech signal, and maps the time-domain effective speech signal to frequency response information in the frequency domain;

[0013] The other convolution layers in the stacked one-dimensional convolution obtain the first speech signal representation in a relaxed frequency band with the expansion of the receptive field;

[0014] The first speech signal representation passes through an activation function to obtain a frequency spectrum domain speech signal representation.

[0015] In one of the embodiments, the phase transformation subnetwork uses a group delay related to the phase spectrum to represent the effective speech signal in the phase domain speech signal representation.

[0016] In one of the embodiments, the phase transformation subnetwork includes a first one-dimensional convolution, a multiplication unit, a second one-dimensional convolution, and a smoothing unit;

[0017] One branch of the effective speech signal passes through the first one-dimensional convolution to perform a short-time Fourier transform to obtain a first frequency domain signal;

[0018] The one branch of the effective speech signal passes through the multiplication unit for multiplication and then passes through the second one-dimensional convolution to perform a short-time Fourier transform to obtain a second frequency domain signal;

[0019] The first frequency domain signal and the second frequency domain signal pass through the smoothing unit to determine the group delay.

[0020] In one of the embodiments, the finite difference network further includes a stacked first time sequence difference subnetwork and a stacked second time sequence difference subnetwork;

[0021] Based on the finite difference network, a sentence-level voiceprint feature is determined according to the effective speech signal;

[0022] Based on the stacked first time sequence difference subnetwork, a frequency spectrum domain frame-level feature is obtained according to the frequency spectrum domain speech signal representation;

[0023] Based on the stacked second time sequence difference subnetwork, a phase domain frame-level feature is obtained according to the phase domain speech signal representation;

[0024] The sentence-level voiceprint feature is determined according to the frequency spectrum domain frame-level feature and the phase domain frame-level feature.

[0025] In one of the embodiments, the time sequence difference subnetwork includes a first time sequence difference subnetwork and a second time sequence difference subnetwork;

[0026] The time sequence difference subnetwork includes a cascaded first time sequence difference module and a first static feature extraction network;

[0027] The first attention weight of each channel is obtained according to the speech signal feature based on the first time difference module; the speech signal feature includes a frequency domain speech signal feature or a phase domain speech signal feature;

[0028] The first attention weight of each channel is multiplied with each element of the speech signal feature blocked by channel to obtain a first dynamic frame level feature;

[0029] The first dynamic frame level feature is input into a first static feature extraction network to obtain a frame level feature; the frame level feature includes a frequency domain frame level feature and a phase domain frame level feature.

[0030] In one of the embodiments, the time difference sub-network includes a first time difference sub-network and a second time difference sub-network;

[0031] The time difference sub-network includes a second time difference module, a third time difference module and a second static feature extraction network connected hierarchically;

[0032] The second attention weight of each channel is obtained according to the speech signal feature based on the second time difference module; the speech signal feature includes a frequency domain speech signal feature or a phase domain speech signal feature;

[0033] The third attention weight of each channel is obtained according to the speech signal feature based on the third time difference module;

[0034] The average attention weight of each channel is obtained by averaging the second attention weight of each channel and the third attention weight of each channel;

[0035] The average attention weight of each channel is multiplied with each element of the speech signal feature blocked by channel to obtain a second dynamic frame level feature;

[0036] The second dynamic frame level feature is input into a second static feature extraction network to obtain a frame level feature; the frame level feature includes a frequency domain frame level feature and a phase domain frame level feature.

[0037] In one of the embodiments, the time difference module includes a first time difference module, a second time difference module and a third time difference module; the attention weight includes a first attention weight, a second attention weight and a third attention weight;

[0038] The time difference module obtains the attention weight of each channel of the speech signal feature by the following steps:

[0039] The signal feature is determined according to the speech signal feature;

[0040] The element matrix and the adjacent element matrix of the intercepted signal feature are respectively time-averaged to obtain a first one-dimensional column vector and a second one-dimensional column vector respectively;

[0041] The first one-dimensional column vector and the second one-dimensional column vector are respectively reduced in dimension by using down-sampling convolution to obtain a first reduced dimension vector and a second reduced dimension vector;

[0042] The difference value is determined according to the first reduced dimension vector and the second reduced dimension vector;

[0043] The difference value is obtained by up-sampling convolution and an activation function to obtain the attention weight of each channel of the speech signal feature.

[0044] In one of the embodiments, the sentence-level voiceprint feature is determined according to the spectral domain frame-level feature and the phase domain frame-level feature, including:

[0045] The spectral domain frame-level feature and the phase domain frame-level feature are aggregated into the sentence-level voiceprint feature based on sentence-level feature aggregation.

[0046] In one of the embodiments, the pre-processing of the to-be-recognized speech signal includes endpoint detection, time length regulation, and pre-emphasis processing.

[0047] As can be seen from the technical solutions provided by the above embodiments of the present specification, the solutions:

[0048] Using the original audio as the input, the speaker identity description information is obtained from the spectral domain and the phase domain and fused, which can avoid the problem of loss of speaker identity information and improve the accuracy of voiceprint recognition.

[0049] The use of the time series difference sub-network based on the speech speed enhancement operator can effectively capture the rhythm change information including intonation change, rhythm, intensity, etc. in a short time to distinguish different speakers, extract voiceprint information with high discrimination, and effectively represent the speaker identity for high-precision voiceprint recognition and voiceprint confirmation voiceprint recognition tasks. BRIEF DESCRIPTION OF DRAWINGS

[0050] In order to more clearly illustrate the technical solutions in the embodiments of the present specification or the prior art, the drawings needed to be used in the embodiment or prior art description will be briefly introduced below. Obviously, the drawings in the following description are only some embodiments described in the present specification, and those skilled in the art can also obtain other drawings according to these drawings without creative labor.

[0051] Figure 1 The flowchart of the voiceprint recognition method provided in the present application is shown in the figure;

[0052] Figure 2A flowchart of a pre-processing procedure for a voice signal to be recognized is provided in the present application.

[0053] Figure 3 A structural diagram of a finite difference network provided in the present application is shown in the following figure.

[0054] Figure 4 A structural diagram of a time domain transformation sub-network and a phase transformation sub-network provided in the present application is shown in the following figure.

[0055] Figure 5 A schematic diagram of a speech rate enhancement operator provided in the present application is shown in the following figure.

[0056] Figure 6 A structural diagram of a time sequence difference sub-network provided in the present application is shown in the following figure. Figure 1

[0057] Figure 7 A structural diagram of a time sequence difference sub-network provided in the present application is shown in the following figure. Figure 2

[0058] Figure 8 A flowchart of sentence level feature fusion provided in the present application is shown in the following figure.

[0059] Figure 9 A flowchart of voiceprint recognition provided in the present application is shown in the following figure.

[0060] Figure 10 A flowchart of voiceprint verification provided in the present application is shown in the following figure. DETAILED DESCRIPTION

[0061] In order to make the person skilled in the art better understand the technical solutions in the specification, the technical solutions in the specification will be described clearly and completely in the following description of the specification embodiments in combination with the drawings in the specification. Obviously, the described embodiments are only some of the embodiments of the specification, not all the embodiments. Based on the embodiments in the specification, all other embodiments obtained by the person skilled in the art without making creative efforts should belong to the protection scope of the specification.

[0062] In the following description, specific details such as specific system structures, techniques, etc. are presented in order to thoroughly understand the embodiments of the present application, but the person skilled in the art should understand that the present application can also be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits and methods are omitted to avoid unnecessary details that hinder the description of the present application.

[0063] ​​Many modifications and variations to the illustrative embodiments described herein can be practiced by those of ordinary skill in the art, and are intended to be within the scope of the disclosure. Other embodiments of the disclosure can be utilized or carried out in various ways as would be apparent to one skilled in the art. The disclosure and examples described herein are intended to be exemplary only.

[0064] As used herein, the terms "comprises", "comprising", "includes", "including", "has", "having" or the like are open-ended terms that are intended to mean including, but not limited to.

[0065] The "parts" in the present application are measured by mass unless otherwise specified.

[0066] In the related art, the acoustic features based on the frequency domain energy spectrum are used, and the phase information of the speech signal is missing. Research shows that the formant of the speaker appears as a transition on the short-time phase spectrum. The use of the phase information means that the distorted speech signal is used, and part of the speaker-related pronunciation information is erased, which ultimately makes the accuracy of the voiceprint recognition low.

[0067] Based on the above defects, the embodiments of the present application propose a voiceprint recognition method, which directly uses the original audio as the system input, obtains the speaker identity description information from the frequency domain spectrum and the phase spectrum based on the finite difference network, and fuses them, thereby avoiding the problem of loss of speaker identity information.

[0068] The present application will be further described in detail below with reference to the accompanying drawings and embodiments.

[0069] Referring to Figure 1 , it shows a flowchart of a voiceprint recognition method provided by the embodiments of the present application.

[0070] As Figure 1 shown, the voiceprint recognition method can include:

[0071] S110, preprocessing the to-be-identified speech signal to obtain an effective speech signal.

[0072] Specifically, the to-be-identified speech signal refers to the original audio signal obtained. Since the original audio signal contains non-speech segment signals such as silence signals, noise signals, and music segment signals, it is necessary to preprocess the original audio signal, remove the non-speech segment signals in the original audio signal, and retain the speech segment signals in the original audio signal, i.e. the speech signals of the vocal segment, as the effective speech signal for subsequent use.

[0073] Optionally, preprocessing the to-be-identified speech signal can include endpoint detection, time length regularization, pre-emphasis processing, etc., as Figure 2 shown.

[0074] Specifically, the method of endpoint detection can include short-time energy, zero-crossing rate detection, spectral entropy, etc.

[0075] Wherein, short-time energy: because the speech segment signal and non-speech segment signal have obvious difference in energy, energy threshold value (can be set according to actual demand) can be set, by comparing short-time energy and energy threshold value, speech segment signal and non-speech segment signal are judged. When short-time energy is greater than or equal to energy threshold value, it is determined that speech segment signal, and when short-time energy is less than energy threshold value, it is determined that non-speech segment signal.

[0076] Wherein, short-time energy E n It is calculated by the following formula:

[0077]

[0078] Wherein, x is the sampling point of the speech signal to be identified, and N is the number of sampling points for each detection.

[0079] Zero-crossing rate detection: because the speech value in a period of time of the speech signal to be identified can be positive or negative, the speech value jump frequency (i.e. zero-crossing rate) is high when it is non-speech segment signal, and the speech value jump frequency is low when it is speech segment signal, therefore, jump threshold value can be set, by comparing jump threshold value and zero-crossing rate, speech segment signal and non-speech segment signal are judged. When zero-crossing rate is greater than or equal to jump threshold value, it is determined that non-speech segment signal, and when zero-crossing rate is less than jump threshold value, it is determined that speech segment signal.

[0080] Wherein, zero-crossing rate Z n It is calculated by the following formula:

[0081]

[0082]

[0083] Wherein, x is the sampling point of the speech signal to be identified, and sgn(x) is the sign function.

[0084] Spectral entropy: because the spectral entropy of speech segment signal is large, and the spectral entropy of non-speech segment signal is small, therefore, spectral entropy threshold value can be set, by comparing spectral entropy and spectral entropy threshold value, speech segment signal and non-speech segment signal are judged. That is, when spectral entropy is greater than or equal to spectral entropy threshold value, it is determined that speech segment signal, and when spectral entropy is less than spectral entropy threshold value, it is determined that non-speech segment signal.

[0085] Wherein, spectral entropy H(s) is calculated by the following formula:

[0086]

[0087] Wherein, P is the normalized signal power spectral density.

[0088] In order to effectively train the whole system, the speech segment signal after endpoint detection can be length normalized, that is, the speech segment signal of each batch training is limited in time length, the speech segment signal exceeding the limited length is truncated, the speech segment signal insufficient in length is filled with 0 value, and the length normalized speech segment signal is pre-emphasized.

[0089] Pre-emphasis processing: used for compensating the attenuation of high frequency signal in the transmission process, effectively improving the signal-to-noise ratio of the output signal.

[0090] The length normalized speech segment signal is X = {x1, x2,..., x n}, and the output effective speech signal after pre-emphasis processing is {x(1), x(2),..., x(n)}.

[0091] x(i) = x i+1 - αx i , i = 1, 2,..., n (5)

[0092] Wherein, α is generally taken as 0.97.

[0093] In the case that the amplitude spectrum and the phase spectrum of a signal frequency domain are known, the time domain signal can be restored from the frequency domain. It is found that the circular convolution and the discrete Fourier transform have formal duality, and the convolution operation is equivalent to filtering, so the convolution kernel can extract rich frequency response from the original audio. By stacking convolution layers and connecting them in a specific way, the process of artificially extracting acoustic features can be simulated, such as simulating the MFCC (Mel-Frequency Cepstral Coefficients) triangular filter bank; and the structure can be flexibly adjusted according to the task, so that the system is not limited to fixed acoustic feature forms and information loss is avoided.

[0094] In S120, a sentence-level voiceprint feature is obtained based on a finite difference network and the effective speech signal; the finite difference network includes a time-frequency transformation sub-network and a phase transformation sub-network, wherein the time-frequency transformation sub-network is used for multi-scale filtering of the frequency domain spectrum of the effective speech signal to obtain a frequency spectrum domain speech signal representation; and the phase transformation sub-network is used for extracting a phase domain speech signal representation of the effective speech signal. As shown in Figure 3 .

[0095] In one embodiment, as shown in Figure 4 , the time domain transformation sub-network includes a stacked one-dimensional convolution and an activation function.

[0096] The first convolution layer in the stacked one-dimensional convolution filters the adjacent frequency bands of the effective speech signal, and maps the time domain effective speech signal to frequency response information in the frequency domain.

[0097] The other convolution layers are stacked in the one-dimensional convolution, and the first speech signal representation is obtained in a relaxed frequency band with the expansion of the receptive field;

[0098] The first speech signal representation passes through an activation function to obtain a frequency domain speech signal representation; the frequency domain speech signal representation includes two dimensions of a time length and a number of convolution kernels.

[0099] Specifically, the time domain transformation subnetwork can be a feedforward network such as a convolutional neural network or a time delay neural network.

[0100] By stacking one-dimensional convolution, short-time Fourier transform is simulated to extract frequency domain information. When the effective speech signal is subjected to convolution operation, the first convolution layer only processes adjacent frequency bands. After filtering by the first convolution layer, the effective speech signal in the time domain is mapped into a frequency response signal in the frequency domain. Subsequent convolution layers can gradually observe more abstract frequency domain speech signal representations in a relaxed frequency band with the expansion of the receptive field. According to the specific task type, the convolution kernel parameters, i.e., the filtering parameters, are learned to produce sufficient frequency response information for a specific task.

[0101] Due to the existence of phase folding, the phase spectrum presents a shape similar to noise compared with the amplitude spectrum, which makes it difficult to extract features and model.

[0102] In one embodiment, the phase transformation subnetwork uses a group delay related to the phase spectrum to represent the effective speech signal in the phase domain speech signal representation (i.e., the phase information of the effective speech signal). The group delay describes the overall time delay of all frequency components passing through a time delay system, has the characteristics of high frequency resolution and less frequency leakage, and contains speaker-related pronunciation information. The group delay includes two dimensions of a time length and a number of convolution kernels.

[0103] Continuing to refer to Figure 4 , the phase transformation subnetwork includes a first one-dimensional convolution, a multiplication unit, a second one-dimensional convolution, and a smoothing unit;

[0104] One branch of the effective speech signal is subjected to short-time Fourier transform by the first one-dimensional convolution to obtain a first frequency domain signal;

[0105] After multiplication by the multiplication unit, the other branch of the effective speech signal is subjected to short-time Fourier transform by the second one-dimensional convolution to obtain a second frequency domain signal;

[0106] The first frequency domain signal and the second frequency domain signal pass through the smoothing unit to determine the group delay.

[0107] Specifically, one branch of the effective speech signal x(n) is subjected to short-time Fourier transform by the first one-dimensional convolution to obtain a first frequency domain signal X n (ω):

[0108]

[0109] One branch is multiplied by the multiplication unit to obtain nx(n), and then a short-time Fourier transform is performed by the second one-dimensional convolution to obtain a second frequency domain signal Y n (ω) is calculated:

[0110]

[0111] The smoothing unit uses X n (ω) and Y n (ω) to calculate the group delay. The group delay is defined as the negative derivative of the phase spectrum θ n (ω) with respect to the angular frequency:

[0112]

[0113] The group delay on the discrete signal can be written as:

[0114]

[0115] Wherein the subscripts R and I represent the real part and the imaginary part, respectively.

[0116] The acoustic information extracted from the original audio signal is more abundant, which makes it possible for the neural network to dynamically extract the weak prosodic changes, rhythm, and intensity of the speaker at a smaller granularity. Figure 5 A time difference module based on a neural network is provided in the present application, referred to as a speed enhancement operator (SEO), which is used to dynamically obtain prosodic information, and the SEO is combined with a speaker static feature extraction network to fuse dynamic prosodic information and static features to form multi-scale speaker features. A light (Light) connection and hierarchical (Hierarchical) connection are provided. Two connection structures of time difference network blocks (TDN-Block) (i.e., time difference sub-networks) are shown in FIGS. Figure 6 、 Figure 7 The performance and feature quality requirements are balanced by stacking different precision requirements.

[0117] In one embodiment, the finite difference network further comprises a first time difference sub-network and a second time difference sub-network stacked;

[0118] Based on the finite difference network, a sentence-level voiceprint feature is determined according to the effective speech signal;

[0119] Based on the first time difference sub-network, a spectral domain frame-level feature is obtained according to the spectral domain speech signal representation;

[0120] based on the stacked second time-difference sub-network, a phase domain frame level feature is obtained according to the phase domain speech signal representation;

[0121] According to the spectrum domain frame level feature and the phase domain frame level feature, a sentence level voiceprint feature is determined.

[0122] It should be noted that the first time-difference sub-network and the second time-difference sub-network have the same structure, and the following embodiments are uniformly described as time-difference sub-networks.

[0123] In one embodiment, as shown in Figure 6 The time-difference sub-network includes a cascade of a first time-difference module (i.e., the speech speed enhancement operator SEO in the figure) and a first static feature extraction network (i.e., the static feature extraction network in the figure);

[0124] Based on the first time-difference module, a first attention weight of each channel of the speech signal feature is obtained according to the speech signal feature; the speech signal feature includes a spectrum domain speech signal feature or a phase domain speech signal feature;

[0125] The first attention weight of each channel and each element of the speech signal feature blocked by channel are Hadamard multiplied to obtain a first dynamic frame level feature;

[0126] The first dynamic frame level feature is input into the first static feature extraction network to obtain a frame level feature; the frame level feature includes a spectrum domain frame level feature and a phase domain frame level feature.

[0127] In one embodiment, as shown in Figure 7 The time-difference sub-network includes a first time-difference sub-network and a second time-difference sub-network;

[0128] The time-difference sub-network includes a cascade of a second time-difference module, a third time-difference module and a second static feature extraction network;

[0129] Based on the second time-difference module, a second attention weight of each channel of the speech signal feature is obtained according to the speech signal feature; the speech signal feature includes a spectrum domain speech signal feature or a phase domain speech signal feature;

[0130] Based on the third time-difference module, a third attention weight of each channel of the speech signal feature is obtained according to the speech signal feature;

[0131] The second attention weight of each channel and the third attention weight of each channel are averaged to obtain an average attention weight of each channel;

[0132] The average attention weight of each channel and each element of the speech signal feature blocked by channel are Hadamard multiplied to obtain a second dynamic frame level feature;

[0133] The second dynamic frame-level feature input the second static feature extraction network to obtain frame-level features; the frame-level features include spectral domain frame-level features and phase domain frame-level features.

[0134] The timing difference module in the following embodiment includes a first timing difference module, a second timing difference module and a third timing difference module; the attention weight includes a first attention weight, a second attention weight and a third attention weight.

[0135] In one embodiment, the timing difference module obtains the attention weight of each channel of the speech signal feature through the following steps:

[0136] According to the speech signal feature, the intercepted signal feature is determined;

[0137] The element matrix and the adjacent element matrix of the intercepted signal feature are respectively time-averaged to obtain a first one-dimensional column vector and a second one-dimensional column vector;

[0138] The first one-dimensional column vector and the second one-dimensional column vector are respectively reduced in dimension by using down-sampling convolution to obtain a first reduced dimension vector and a second reduced dimension vector;

[0139] The difference value is determined according to the first reduced dimension vector and the second reduced dimension vector;

[0140] The difference value is obtained by up-sampling convolution and an activation function to obtain the attention weight of each channel of the speech signal feature.

[0141] Specifically, for the input acoustic feature (i.e. speech signal feature) Wherein, T represents the length of the original audio signal after extracting the acoustic feature, C represents the number of channels of the acoustic feature module, the acoustic feature F is intercepted along the time dimension by using a sliding window with a length of t and a window shift of p to obtain the intercepted acoustic feature (i.e. intercepted signal feature)

[0142] For each element matrix x of X i and the element matrix after it (i.e. adjacent element matrix) x i+1 Time-averaging means averaging the matrix along the time axis, i.e. averaging each row vector to obtain a one-dimensional column vector (i.e. first one-dimensional column vector and second one-dimensional column vector)

[0143]

[0144] After down-sampling convolution, From Space mapping to The space, i.e. the dimension is reduced from dimension C to C / a, to obtain a first reduced dimension vector and a second reduced dimension vector:

[0145]

[0146]

[0147] wherein, and denote two down-sampling convolutions, each with weights W1, W2.

[0148] The difference value is obtained by subtraction

[0149]

[0150] The difference value is obtained by up-sampling convolution, whose function is denoted as with weights W3, s i is restored to a C-dimensional vector, and the attention weight of each channel of the speech signal feature is obtained by an activation function σ

[0151]

[0152] wherein, the attention weight of each channel contains the prosody information of the speaker.

[0153] For Figure 6 the time-difference sub-network shown in FIG. 14, the attention weight (i.e. the first attention weight) obtained by formula (14) is Hadamard multiplied with the original acoustic feature to highlight the changed prosody information:

[0154] x′ i = s′ i ⊙ x i (15)

[0155] wherein, ⊙ denotes the Hadamard product of the elements of each channel in s′ i and each element of x i blocked by channel.

[0156] For Figure 7 the time-difference sub-network shown in FIG. 14, two different attention weights (i.e. the second attention weight and the third attention weight) are obtained for each x i , the average attention weight of each channel is obtained by averaging the second attention weight of each channel and the third attention weight of each channel. The average attention weight of each channel is Hadamard multiplied with the original acoustic feature.

[0157] The static feature extraction network in the above embodiments can be any neural network structure, such as a residual network, a time-delay neural network, a long short-term memory network, etc.

[0158] In one embodiment, the sentence-level voiceprint feature is determined according to the spectral domain frame-level feature and the phase domain frame-level feature, including:

[0159] Based on the sentence-level feature aggregation, the spectral domain frame-level feature and the phase domain frame-level feature are aggregated into the sentence-level voiceprint feature.

[0160] The sentence-level feature aggregation, also known as temporal pooling, integrates the frame-level speaker voiceprint features into the sentence level, as shown in FIG. 1, to obtain a speaker global feature with better robustness. The pooling methods mainly include average pooling, statistical pooling, self-attention pooling, and neural network-based pooling. Figure 8

[0161] The input of the sentence-level feature aggregation is h t , where h t represents the t th frame speaker feature of the frame-level hidden layer output, T is the total number of frames, and u represents the sentence-level output of the temporal pooling layer.

[0162] (1) Average pooling (Average pooling)

[0163]

[0164] (2) Statistics pooling (Statistics pooling)

[0165] m represents the statistical average of , d represents the standard deviation of , and then

[0166]

[0167]

[0168] The output of the statistical pooling is the concatenation of m and d:

[0169] u=[m T , d T ] Ｔ (19)

[0170] (3) Self-attention-based pooling (Self-attention-based pooling)

[0171] ​Both average pooling and statistic pooling assume that each frame contributes equally to the sentence-level feature, but in fact, each frame does not provide equal speaker distinguishing information. To solve this problem, researchers introduce attention mechanism. Based on self-attention pooling, which is essentially a weighted statistic pooling, it makes the neural network pay more attention to some important frames of the input.

[0172] Define the self-attention score denotes the kth self-attention head, and K is the total number of heads. When K≥2, it is called multi-head attention mechanism, which can take into account the information of different representation subspaces.

[0173] There are many ways to implement attention mechanism, and the attention function that generates weights can be expressed as follows:

[0174]

[0175] where v, W, g, b are the learnable parameters of the kth attention function

[0176] Let The weight of the frame-level feature h t

[0177]

[0178] where satisfies and

[0179] The corresponding mean and standard deviation after the kth attention function correction are:

[0180]

[0181]

[0182] The output of the self-attention pooling is the concatenation of the mean and standard deviation.

[0183] (4) Neural network-based pooling

[0184] ​​Temporal clustering of frame-level features is performed using a dictionary-based NetVLAD layer, in which each frame-level feature description is partitioned into different classes, and a sentence-level compact representation is obtained. A variant of NetVLAD, called GhostVLAD, is compared. The difference between GhostVLAD and NetVLAD is that one class in the clustering result of GhostVLAD does not participate in the final statistics, and noise and undesirable frame-level features will be divided into this class, so that the influence of noise on speaker features is greatly devalued.

[0185] In S130, voiceprint recognition is performed according to the sentence-level voiceprint feature.

[0186] Specifically, when performing the voiceprint recognition task, the voiceprint feature generated by the speech is judged in a scoring manner, and the scoring standard includes a cosine similarity.

[0187] As shown in Figure 9 When the voiceprint recognition task is to perform voiceprint recognition, the voiceprint feature generated by the speech segment with unknown identity is obtained by comparing with all voiceprints in the voiceprint database, and the most likely speaker identity of the unknown speech in the voiceprint database is obtained through similarity discrimination.

[0188] As shown in Figure 10 When the voiceprint recognition is to perform voiceprint confirmation, the voiceprint feature generated by the speech segment with unknown identity is compared with the registered voiceprint feature of the speaker database to which it claims to belong, and whether the unknown speech segment belongs to the speaker to which it claims to belong is determined according to the set threshold value through similarity evaluation.

[0189] The similarity discrimination is generally based on cosine similarity or Probabilistic Linear Discriminant Analysis (PLDA).

[0190] For two vectors The cosine similarity of two vectors is defined as the cosine value of the angle between the two vectors:

[0191]

[0192] The embodiments of the present application use original audio as input, obtain speaker identity description information from the spectral domain and the phase domain, and fuse them, which can avoid the problem of loss of speaker identity information and improve the accuracy of voiceprint recognition.

[0193] The embodiment of the application utilizes a timing difference sub-network based on a speech speed enhancement operator, can effectively capture the rhythm change information including intonation change, rhythm, intensity, etc. in a short time to distinguish different speakers, extract high-discrimination voiceprint information, and effectively represent the speaker identity for high-precision voiceprint recognition and voiceprint confirmation voiceprint identification tasks.

[0194] It should be noted that the terms "comprising", "containing", or any other variant thereof are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements, but can also include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without more limitations, an element defined by the phrase "comprising a" does not exclude the existence of additional identical elements in the process, method, article, or apparatus that includes the element.

[0195] Each of the embodiments in the specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other. Each embodiment focuses on the difference from other embodiments. In particular, for the system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the part of the method embodiment.

Claims

1. A voiceprint recognition method, characterized in that, The method comprises: preprocessing a voice signal to be identified to obtain an effective voice signal; determining a sentence-level voiceprint feature based on a finite difference network and the effective voice signal; the finite difference network comprises a time-frequency transformation subnetwork and a phase transformation subnetwork, wherein the time-frequency transformation subnetwork is configured to perform multi-scale filtering on a frequency domain spectrum of the effective voice signal to obtain a frequency domain voice signal representation; and the phase transformation subnetwork is configured to extract a phase domain voice signal representation of the effective voice signal; performing voiceprint recognition based on the sentence-level voiceprint feature; wherein the finite difference network further comprises a first stacked time sequence difference subnetwork and a second stacked time sequence difference subnetwork; determining the sentence-level voiceprint feature based on the finite difference network and the effective voice signal comprises: obtaining a frequency domain frame-level feature based on the frequency domain voice signal representation and the first stacked time sequence difference subnetwork; obtaining a phase domain frame-level feature based on the phase domain voice signal representation and the second stacked time sequence difference subnetwork; and determining the sentence-level voiceprint feature based on the frequency domain frame-level feature and the phase domain frame-level feature.

2. The method of claim 1, wherein, The time-frequency transformation subnetwork comprises a stacked one-dimensional convolution and an activation function; a first convolution layer in the stacked one-dimensional convolution filters adjacent frequency bands of the effective voice signal to map the effective voice signal in the time domain to frequency response information in the frequency domain; other convolution layers in the stacked one-dimensional convolution obtain a first voice signal representation in a relaxed frequency band with an enlarged receptive field; the first voice signal representation passes through the activation function to obtain the frequency domain voice signal representation.

3. The method of claim 1, wherein, The phase transformation subnetwork uses a group delay related to a phase spectrum to represent the phase domain voice signal representation of the effective voice signal.

4. The method of claim 3, wherein, The phase transformation subnetwork comprises a first one-dimensional convolution, a multiplication unit, a second one-dimensional convolution, and a smoothing unit; one branch of the effective voice signal is subjected to short-time Fourier transform through the first one-dimensional convolution to obtain a first frequency domain signal; one branch of the effective voice signal is multiplied by the multiplication unit and then subjected to short-time Fourier transform through the second one-dimensional convolution to obtain a second frequency domain signal; the first frequency domain signal and the second frequency domain signal pass through the smoothing unit to determine the group delay.

5. The method of claim 1, wherein, Both the first time sequence difference subnetwork and the second time sequence difference subnetwork comprise a cascade of first time sequence difference modules and a first static feature extraction network; based on the first time sequence difference module, a first attention weight of each channel of the voice signal feature is obtained based on the voice signal feature; the voice signal feature comprises the frequency domain voice signal representation or the phase domain voice signal representation; the first attention weight of each channel is Hadamard multiplied with each element of the voice signal feature block by channel to obtain a first dynamic frame-level feature; the first dynamic frame-level feature is input into the first static feature extraction network to obtain a frame-level feature; the frame-level feature comprises the frequency domain frame-level feature and the phase domain frame-level feature.

6. The method of claim 1, wherein, The first time-difference sub-network and the second time-difference sub-network each comprise a second time-difference module, a third time-difference module and a second static feature extraction network connected hierarchically; Based on the second time-difference module, second attention weights of each channel of the speech signal feature are obtained according to the speech signal feature; The speech signal feature comprises the spectral domain speech signal representation or the phase domain speech signal representation; Based on the third time-difference module, third attention weights of each channel of the speech signal feature are obtained according to the speech signal feature; The second attention weights of each channel and the third attention weights of each channel are averaged to obtain average attention weights of each channel; The average attention weights of each channel are Hadamard multiplied with each element of the speech signal feature blocked by channel to obtain second dynamic frame-level features; The second dynamic frame-level features are input into the second static feature extraction network to obtain frame-level features; the frame-level features comprise the spectral domain frame-level features and the phase domain frame-level features.

7. The method according to claim 5 or 6, characterized in that, The time-difference module obtains the attention weights of each channel of the speech signal feature through the following steps: According to the speech signal feature, an intercepted signal feature is determined; The element matrix and the adjacent element matrix of the intercepted signal feature are respectively time-averaged to obtain a first one-dimensional column vector and a second one-dimensional column vector respectively; Down-sampling convolution is adopted to down-dimension the first one-dimensional column vector and the second one-dimensional column vector respectively to obtain a first down-dimension vector and a second down-dimension vector; According to the first down-dimension vector and the second down-dimension vector, a difference value is determined; The difference value is obtained through up-sampling convolution and an activation function to obtain the attention weights of each channel of the speech signal feature.

8. The method of claim 1, wherein, The determination of the sentence-level voiceprint feature according to the spectral domain frame-level features and the phase domain frame-level features comprises: Based on sentence-level feature aggregation, the spectral domain frame-level features and the phase domain frame-level features are aggregated into the sentence-level voiceprint feature.

9. The method according to any one of claims 1 to 6, characterized in that, The preprocessing of the to-be-recognized speech signal comprises endpoint detection, time length regulation and pre-emphasis processing.

Images