Voiceprint recognition model training method and device

Abstract

The invention discloses a voiceprint recognition model training method and device, and belongs to the technical field of artificial intelligence. The method comprises the following steps: acquiring audio information of first audio data, wherein the audio information comprises an audio signal and a target voiceprint feature; based on the audio signal, first STFT feature information is obtained, and the first STFT feature information comprises a time domain feature and a frequency domain feature of the first audio data; performing feature processing on the first STFT feature information to obtain a first feature vector, the first feature vector representing a voiceprint feature of the first audio data; and training a first voiceprint recognition model based on the target voiceprint feature and the first feature vector to obtain a second voiceprint recognition model.

Description

Technical Field

[0001] The present application belongs to the field of artificial intelligence technology, and specifically relates to a training method and device for a voiceprint recognition model. Background Art

[0002] With the continuous development of artificial intelligence technology, its application in various fields is becoming more and more extensive, and voiceprint recognition is one of the important application scenarios. Voiceprint recognition technology plays an increasingly important role in ensuring security and improving user experience. People's demand for voiceprint recognition continues to grow, and the accuracy of voiceprint recognition results is increasingly required.

[0003] The audio signal features used by existing voiceprint recognition models are mainly Mel filterbank (FBank) features and Mel-Frequency Cepstral Coefficents (MFCC) features. FBank features are spectral energy features directly extracted by the Mel filterbank. They simulate the auditory characteristics of the human ear and decompose the audio signal into energy distributions on multiple Mel frequency bands. MFCC features are obtained through discrete cosine transform based on FBank features. They further compress the data and remove the correlation between signals of different dimensions.

[0004] However, since the above FBank features and MFCC features are obtained through operations such as Mel filters, and operations such as Mel filters will cause audio feature loss, such as discarding high-frequency details in the original audio, causing serious distortion of the original audio signal. As a result, the current voiceprint recognition model has a poor effect on voiceprint recognition. Summary of the invention

[0005] The purpose of the embodiments of the present application is to provide a method and device for training a voiceprint recognition model, which can improve the training efficiency of the voiceprint recognition model of an electronic device.

[0006] In a first aspect, an embodiment of the present application provides a method for training a voiceprint recognition model, the method comprising: obtaining audio information of first audio data, the audio information comprising an audio signal and a target voiceprint feature; based on the audio signal, obtaining first short-time Fourier transform (STFT) feature information, the first STFT feature information comprising time domain features and frequency domain features of the first audio data; performing feature processing on the first STFT feature information to obtain a first feature vector, the first feature vector representing the voiceprint feature of the first audio data; based on the target voiceprint feature and the first feature vector, training a first voiceprint recognition model to obtain a second voiceprint recognition model.

[0007] In a second aspect, an embodiment of the present application provides a training device for a voiceprint recognition model, the device comprising: an acquisition module and a processing module. The acquisition module is used to acquire audio information of a first audio data, the audio information comprising an audio signal and a target voiceprint feature; and, based on the audio signal, acquire first STFT feature information, the first STFT feature information comprising time domain features and frequency domain features of the first audio data. The processing module is used to perform feature processing on the first STFT feature information acquired by the acquisition module to obtain a first feature vector, the first feature vector representing the voiceprint feature of the first audio data; and, based on the target voiceprint feature acquired by the acquisition module and the first feature vector, train the first voiceprint recognition model to obtain a second voiceprint recognition model.

[0008] In a third aspect, an embodiment of the present application provides an electronic device, which includes a processor and a memory, wherein the memory stores programs or instructions that can be run on the processor, and when the program or instructions are executed by the processor, the steps of the method described in the first aspect are implemented.

[0009] In a fourth aspect, an embodiment of the present application provides a readable storage medium, on which a program or instruction is stored, and when the program or instruction is executed by a processor, the steps of the method described in the first aspect are implemented.

[0010] In a fifth aspect, an embodiment of the present application provides a chip, comprising a processor and a communication interface, wherein the communication interface is coupled to the processor, and the processor is used to run a program or instruction to implement the method described in the first aspect.

[0011] In a sixth aspect, an embodiment of the present application provides a computer program product, which is stored in a storage medium and is executed by at least one processor to implement the method described in the first aspect.

[0012] In an embodiment of the present application, an audio signal and a target voiceprint feature corresponding to first audio data are obtained, and based on the audio signal, first STFT feature information including time domain features and frequency domain features of the first audio data is obtained, and feature processing is performed on the first STFT feature information to obtain a first feature vector characterizing the voiceprint feature of the first audio data, and based on the target voiceprint feature and the first feature vector, a first voiceprint recognition model is trained to obtain a second voiceprint recognition model. In this solution, after obtaining the first audio data, first STFT feature information including time domain features and frequency domain features of the first audio data can be obtained based on the audio signal corresponding to the first audio data. Since the first STFT feature information is directly extracted from the audio signal, the detailed distribution information of the audio signal in time and frequency is retained, and it is closer to the audio signal corresponding to the first audio data, and can provide more information about the first audio data itself. In the process of training the first voiceprint recognition model, the first STFT feature information that is closer to the audio signal and includes more comprehensive first audio data features is used as input, so that the first voiceprint recognition model can more quickly and accurately capture the unique voiceprint features of the first audio data for model training, and obtain the second voiceprint recognition model. Therefore, the trained second voiceprint recognition model can accurately perform voiceprint recognition, thereby improving the voiceprint recognition effect of the voiceprint recognition model. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] Figure 1 This is one of the flow charts of the method for training a voiceprint recognition model provided in an embodiment of the present application;

[0014] Figure 2 This is the second flow chart of the method for training a voiceprint recognition model provided in an embodiment of the present application;

[0015] Figure 3 is an example schematic diagram of the STFT time-frequency spectrum provided in the embodiment of the present application;

[0016] Figure 4 This is the third flow chart of the method for training a voiceprint recognition model provided in an embodiment of the present application;

[0017] Figure 5 It is a flowchart of the SENet module provided in the embodiment of the present application;

[0018] Figure 6 It is a flowchart of the Res2Net module provided in an embodiment of the present application;

[0019] Figure 7 is a flow chart of the Attention module provided in the embodiment of the present application;

[0020] Figure 8This is one of the schematic diagrams of the replica audio upload interface provided in the embodiment of the present application;

[0021] Fig. 9 This is the second schematic diagram of the replica audio upload interface provided in the embodiment of the present application;

[0022] Fig.10 This is the fourth flow chart of the method for training a voiceprint recognition model provided in an embodiment of the present application;

[0023] Fig.11 It is a data flow diagram of the training method of the voiceprint recognition model provided in the embodiment of the present application;

[0024] Fig.12 It is a structural schematic diagram of a training device for a voiceprint recognition model provided in an embodiment of the present application;

[0025] Fig.13 is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application;

[0026] Fig.14 It is a schematic diagram of the hardware structure of the electronic device provided in the embodiment of the present application. DETAILED DESCRIPTION

[0027] The following will be combined with the drawings in the embodiments of the present application to clearly describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments in the present application belong to the scope of protection of this application.

[0028] The terms "first", "second", etc. in the specification and claims of the present application are used to distinguish similar objects, and are not used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable under appropriate circumstances, so that the embodiments of the present application can be implemented in an order other than those illustrated or described here, and the objects distinguished by "first", "second", etc. are generally of one type, and the number of objects is not limited. For example, the first object can be one or more. In addition, "and / or" in the specification and claims represents at least one of the connected objects, and the character " / " generally indicates that the objects associated with each other are in an "or" relationship.

[0029] The terms "at least one (item)", "at least one of" and the like in this application refer to any one, any two or a combination of more than two of the objects included therein. For example, at least one (item) of a, b, and c can be represented by: "a", "b", "c", "a and b", "a and c", "b and c" and "a, b and c", where a, b, and c can be single or multiple. Similarly, "at least two (items)" refers to two or more, and its meaning is similar to that of "at least one (item)".

[0030] The following is a detailed description of the training method of the voiceprint recognition model provided in the embodiment of the present application through specific embodiments and their application scenarios in conjunction with the accompanying drawings.

[0031] The training method of the voiceprint recognition model provided in the embodiment of the present application can be applied to scenarios where a voiceprint recognition model is required to process voiceprint recognition tasks.

[0032] For example, the trained second voiceprint recognition model is used to detect whether the audio data submitted by the user is the audio data of a sensitive person. When the first voiceprint recognition model is trained to obtain the trained second voiceprint recognition model, first, audio information corresponding to multiple audio data is obtained from the open source audio data set, and the audio information includes an audio signal, an audio sampling rate, and a target voiceprint feature. When the audio sampling rate of each audio signal is equal to the target sampling rate, the audio signal is processed so that the lengths of all audio signals are equal. Then, the audio signal is Fourier transformed to obtain the first STFT feature information, and the first STFT feature information is input into the first voiceprint recognition model for model training. Each frequency feature in the first STFT feature information is weighted. Re-adjust to obtain second STFT feature information, perform convolution processing on the frequency features in the second STFT feature information to obtain third STFT feature information, process the audio data based on the third STFT feature information to obtain a first feature vector, calculate a loss value based on the target voiceprint feature and the first feature vector, update the parameters in the first voiceprint recognition model based on the loss value, obtain a second voiceprint recognition model, use the second voiceprint recognition model to identify the voiceprint features of the audio data submitted by the user, compare the voiceprint features with the audio data of the sensitive person, and determine whether the audio data submitted by the user is the audio data of the sensitive person.

[0033] The training method of the voiceprint recognition model provided in the embodiment of the present application obtains an audio signal and a target voiceprint feature corresponding to first audio data, obtains first STFT feature information including time domain features and frequency domain features of the first audio data based on the audio signal, performs feature processing on the first STFT feature information to obtain a first feature vector representing the voiceprint feature of the first audio data, and trains a first voiceprint recognition model based on the target voiceprint feature and the first feature vector to obtain a second voiceprint recognition model. In this solution, after obtaining the first audio data, first STFT feature information including time domain features and frequency domain features of the first audio data can be obtained based on the audio signal corresponding to the first audio data. Since the first STFT feature information is directly extracted from the audio signal, the detailed distribution information of the audio signal in time and frequency is retained, and it is closer to the audio signal corresponding to the first audio data, and can provide more information about the first audio data itself. In the process of training the first voiceprint recognition model, the first STFT feature information that is closer to the audio signal and includes more comprehensive first audio data features is used as input, so that the first voiceprint recognition model can more quickly and accurately capture the unique voiceprint features of the first audio data for model training, and obtain the second voiceprint recognition model. Therefore, the trained second voiceprint recognition model can accurately perform voiceprint recognition, thereby improving the voiceprint recognition effect of the voiceprint recognition model.

[0034] The training method for the voiceprint recognition model provided in the embodiment of the present application can be performed by a training device for the voiceprint recognition model. Exemplarily, the training device for the voiceprint recognition model can be an electronic device, or a component in the electronic device, such as an integrated circuit or a chip. The specific one can be determined according to actual usage requirements, and the embodiment of the present application does not limit it. The following takes the training device for the voiceprint recognition model as an electronic device, and takes the electronic device executing the training method for the voiceprint recognition model as an example to exemplify the training method for the voiceprint recognition model provided in the embodiment of the present application.

[0035] Figure 1 A flow chart of a method for training a voiceprint recognition model provided by an embodiment of the present application is shown. Figure 1 As shown, the training method of the voiceprint recognition model provided in the embodiment of the present application may include the following steps 201 to 204.

[0036] Step 201: The electronic device obtains audio information of first audio data.

[0037] In the embodiment of the present application, the above-mentioned audio information includes an audio signal and a target voiceprint feature.

[0038] It should be noted that the above-mentioned audio signal refers to the signal waveform of the audio signal.

[0039] In an embodiment of the present application, the first audio data includes multiple audio data, which can also be called an open source audio data set, used as a data set for training a voiceprint recognition model.

[0040] Optionally, in an embodiment of the present application, the electronic device may obtain the first audio data by traversing the data directory structure where the first audio data is located, thereby obtaining audio information of the first audio data.

[0041] Exemplarily, taking the open source audio dataset CN-Celeb2 as an example, SPK_ID is used to represent the audio speaker identity, which represents the voiceprint feature vector, that is, the target voiceprint feature mentioned above, and audio_file is used to represent the first audio data of the speaker. Its data directory structure is CN-Celeb2_flac / data / ${SPK_ID} / ${audio_file}. The specific method for obtaining the audio information of the first audio data in the open source audio dataset is as follows: traverse each audio data audio_file in the open source audio dataset, and use torchaudio to read the audio signal SIGNAL and audio sampling rate SR corresponding to each audio data; according to the absolute file path corresponding to the audio data audio_file, use python code to parse the audio speaker identity SPK_ID corresponding to each audio data; combine the obtained audio information into triple data [SIGNAL, SR, SPK_ID].

[0042] It can be understood that the above audio signal is a signal representing a mechanical wave, and is an information carrier of the wavelength and intensity changes of the mechanical wave. The audio signal has some key characteristic parameters, such as frequency, amplitude, phase, bandwidth, etc. These parameters determine the characteristics of the audio signal. By analyzing the characteristics of the audio signal, the identity of the speaker can be identified, that is, voiceprint recognition.

[0043] It should be noted that voiceprint refers to the unique acoustic characteristics of each person when they pronounce words, similar to the uniqueness of fingerprints. Voiceprint recognition is an identity authentication technology based on biometrics.

[0044] In the embodiment of the present application, the above-mentioned target voiceprint feature refers to the actual voiceprint feature of the first audio data, that is, the voiceprint feature expected to be output after the first audio data is input into the voiceprint recognition model, which is used to compare with the voiceprint feature predicted by the voiceprint recognition model, so as to judge the accuracy of the recognition result of the voiceprint recognition model.

[0045] Step 202: The electronic device obtains first STFT characteristic information based on the audio signal.

[0046] In the embodiment of the present application, the first STFT feature information includes the time domain features and frequency domain features of the first audio data.

[0047] It is understandable that the first STFT feature information includes the frequency characteristics corresponding to the first audio data at each time. The STFT feature is a very important time-frequency analysis feature in the field of signal processing and audio processing. The STFT can analyze the audio signal in the time domain and the frequency domain, has good time-frequency localization characteristics, and can effectively capture the instantaneous characteristics of the audio signal. The STFT feature can be displayed by a time-frequency spectrum, for example, the horizontal axis represents time, the vertical axis represents frequency, and the color or brightness represents the intensity of the audio signal at different times and frequencies.

[0048] Optionally, in the embodiment of the present application, the first audio data includes multiple audio data, and the audio information also includes an audio sampling rate. Figure 1 ,like Figure 2 As shown, the above step 202 can be specifically implemented through the following steps 202a and 202b.

[0049] Step 202a: When the audio sampling rate is equal to the target sampling rate, if the lengths of the audio signals of the multiple audio data are not equal, the electronic device performs signal processing on the audio signals of the multiple audio data.

[0050] In an embodiment of the present application, the above-mentioned signal processing includes at least one of signal clipping and signal filling.

[0051] In the embodiment of the present application, the above-mentioned audio sampling rate refers to the number of times a continuously changing analog audio signal is sampled per second when it is converted into a discrete digital signal.

[0052] It should be noted that the higher the audio sampling rate, the more points sampled per unit time, the richer the original audio signal information that can be retained, and thus the better the restored sound quality. In addition, the higher the audio sampling rate, the more sound details can be captured, and the sound quality and timbre of the original audio signal can be more realistically restored, making the listening experience closer to the original analog signal.

[0053] In the embodiment of the present application, the target sampling rate may be a pre-set audio sampling rate that satisfies the voiceprint recognition model, such as 16KHz. The target sampling rate is used to determine whether the audio sampling rate of the acquired audio information meets the requirements.

[0054] It should be noted that, after obtaining the audio signals and target voiceprint features of multiple audio data, the electronic device can also obtain the audio sampling rates corresponding to the multiple audio data to determine whether the audio signals corresponding to the multiple audio data meet the requirements of the voiceprint recognition model for the input signal, that is, when the audio sampling rate is equal to the target sampling rate, it is considered that the audio signals corresponding to the multiple audio data can be used as the input of the voiceprint recognition model; then, it is further determined whether the lengths of the audio signals corresponding to the multiple audio data are consistent. If they are consistent, the audio signals corresponding to the multiple audio data are directly converted into corresponding STFT features. If they are inconsistent, signal processing is performed on the audio signals corresponding to the multiple audio data to avoid errors in the voiceprint recognition model training process caused by inconsistent lengths of multiple audio signals.

[0055] Optionally, in the embodiment of the present application, the electronic device may process the audio signals of the multiple audio data in a manner that includes at least one of the following: (1) padding the audio signals that are shorter than the length requirement by adding blank audio to the end of the audio signals; (2) clipping the audio signals that are longer than the length requirement by randomly intercepting a section of the audio signals that meets the length requirement. Thus, the lengths of the audio signals of the multiple audio data after processing are equal.

[0056] Exemplarily, the electronic device defines the hyperparameter target sampling rate (target_sample_rate) and the training audio length (train_audio_len); determines whether the target_sample_rate is equal to the audio sampling rate SR obtained in the above step 201, and if they are not equal, uses the resample method of torchaudio to resample to ensure that the audio sampling rate SR meets the hyperparameter specification; if they are equal, further determines whether the train_audio_len is equal to the length of the audio signal SIGNAL obtained by resampling, if the audio signal SIGNAL is shorter, zero padding is performed at the end of the audio signal SIGNAL, that is, blank audio is added, if the audio signal SIGNAL is longer, the audio signal SIGNAL is randomly intercepted, and finally the electronic device will crop / fill all the sampled audio signals SIGNAL into audio signals of the same length.

[0057] Step 202b: The electronic device performs Fourier transform on the audio signals of the plurality of audio data after signal processing to obtain first STFT feature information.

[0058] In the embodiment of the present application, the above-mentioned Fourier transform is a mathematical method for converting a signal from a time domain (or a spatial domain) to a frequency domain. In audio signal processing, the Fourier transform can decompose an audio signal into sinusoidal components of different frequencies, thereby revealing the frequency components of the audio signal. STFT is an extension of the Fourier transform and is suitable for processing non-stationary signals. STFT obtains the frequency components of the signal that vary with time by dividing the signal into multiple short time periods and performing a Fourier transform on each time period. This enables the STFT to capture transient changes in the signal, such as sudden changes in tone or bursts of noise, so that the characteristics of the signal at different times and frequencies can be obtained, providing a basis for subsequent analysis and processing.

[0059] Typically, audio data is composed of the superposition of periodic signals at multiple frequencies. Signals of different frequencies have different influence factors on the voiceprint recognition task. In the embodiment of the present application, the electronic device performs Fourier transform on the audio signal to obtain STFT features, thereby converting the audio signal into component signals of multiple frequencies, thereby reducing the learning difficulty of the voiceprint recognition model.

[0060] Exemplarily, the electronic device performs Fourier transform on the audio signal SIGNAL obtained in the above step 202a according to the hyperparameter n_fft, and converts the time domain feature SIGNAL into the time-frequency domain feature STFT, that is, the first STFT feature information. In this way, the electronic device converts the one-dimensional non-aligned audio signal SIGNAL into a two-dimensional aligned STFT feature, that is, the first STFT feature information, through the short-time Fourier transform, and the audio signal length becomes train_audio_len, and the audio signal SIGNAL is separated into the superposition result of multiple periodic waveforms of different frequencies, which can be divided into two dimensions of time and frequency. Figure 3 As shown, the irregular audio signal SIGNAL30 is separated into three signal waveforms with different frequency periods, such as Figure 3 Signal waveform 31, signal waveform 32 and signal waveform 33 in.

[0061] It should be noted that n_fft is a key parameter in STFT, which represents the window length used when performing Fast Fourier Transform (FFT), that is, the number of sample points considered for each analysis. In STFT, the audio signal is divided into multiple overlapping windows (or frames), and the signal in each window is independently FFTed. n_fft determines the number of samples in each window, which affects the frequency resolution and time resolution.

[0062] In an embodiment of the present application, when the audio sampling rate is equal to the target sampling rate, the electronic device performs signal clipping and signal padding processing on the audio signals of multiple audio data, so that the lengths of the audio signals of the multiple audio data are consistent, thereby reducing the impact of audio data defects on the training results of the voiceprint recognition model.

[0063] Step 203: The electronic device performs feature processing on the first STFT feature information to obtain a first feature vector.

[0064] In the embodiment of the present application, the first feature vector represents the voiceprint feature of the first audio data.

[0065] In an embodiment of the present application, the electronic device inputs the first STFT feature information corresponding to the audio information into a first voiceprint recognition model (i.e., the initial voiceprint recognition model). The first voiceprint recognition model performs feature processing on the first STFT feature information to obtain the predicted voiceprint feature of the first audio data, that is, maps the first audio data to the corresponding first feature vector.

[0066] Optionally, in the embodiment of the present application, the frequency domain feature includes at least one frequency feature, combined with Figure 1 ,like Figure 4 As shown, the above step 203 can be specifically implemented through the following steps 203a to 203c.

[0067] Step 203a: The electronic device performs weight adjustment on each frequency feature in the first STFT feature information to obtain second STFT feature information.

[0068] In the embodiment of the present application, the audio signal is converted into a periodic signal within A frequencies through the first STFT characteristic information obtained in the above step 202, and the following formula (1) is used to express the first STFT characteristic information and each frequency characteristic C x The relationship, where x∈[1,A]:

[0069] F STFT =[C1,C2,……,C A ] (1)

[0070] Among them, F STFT represents the first STFT feature information, C A Represents the Ath frequency feature.

[0071] It should be noted that different frequency characteristics C x With C y (x≠y) has different effects on human hearing and thus has different impacts on voiceprint recognition tasks.

[0072] Optionally, in an embodiment of the present application, the electronic device may use a Squeeze-and-Excitation Network (SENet) module to adjust the weight of each frequency feature in the above-mentioned first STFT feature information to obtain second STFT feature information, so as to reduce the influence of secondary frequency features in the first STFT feature information and improve the recognition of important frequency features in the first STFT feature information.

[0073] It is understandable that the SENet module is an innovative neural network structure that aims to improve network performance by explicitly modeling the interdependencies between feature channels. The core idea of ​​the SENet module is to enhance the representation ability of convolutional neural networks by introducing a new structural unit, the "Squeeze-and-Excitation" (SE) block, which measures the correlation between channels in multi-channel input and dynamically adjusts the corresponding weights. The SE block consists of two main operations: Squeeze and Excitation.

[0074] Optionally, in the embodiment of the present application, the electronic device may adjust the weight of each frequency feature in the first STFT feature information through compression operation and excitation operation. Exemplarily, the above step 203a may be implemented through the following steps 203a1 to 203a3.

[0075] Step 203a1: The electronic device performs global average pooling on each frequency feature in the first STFT feature information to obtain average STFT feature information.

[0076] In an embodiment of the present application, the electronic device uses a compression operation to perform global average pooling on each frequency feature in the first STFT feature information to obtain average STFT feature information.

[0077] It can be understood that the compression operation refers to aggregating the spatial dimensions of the input feature map through a global average pooling operation to generate a channel descriptor for each channel. Specifically, by performing global average pooling on the input feature map, the feature values ​​of all spatial positions of each channel are added and then divided by the number of spatial positions to obtain a vector equal to the number of channels, which contains the global information of each channel. In this way, the global spatial information is compressed into a channel vector, capturing the global distribution of the channel feature response.

[0078] For example, Figure 5As shown, the electronic device uses the SENet module to perform global average pooling on the input first STFT feature information in the frequency dimension, and reduces the characteristic value of each frequency in the first STFT feature information to a scalar, thereby compressing the spatial information of each frequency feature into a global description to obtain the average STFT feature information STFT_AVG.

[0079] Step 203a2: The electronic device performs frequency adjustment processing on the average STFT feature information to obtain a weight value corresponding to each frequency feature.

[0080] In the embodiment of the present application, the electronic device uses an excitation operation to perform frequency adjustment processing on the average STFT feature information to obtain a weight value corresponding to each frequency feature.

[0081] It can be understood that the excitation operation refers to learning the weight of each channel through two layers of fully connected networks (FC networks) after the channel descriptor is obtained through the compression operation. The role of these two layers of fully connected networks is to capture the dependencies between channels and generate a weight value (or importance coefficient) for each channel based on these dependencies. The first FC layer is usually used to reduce the dimension of the channel descriptor to reduce the amount of calculation, and apply the linear rectification function (Rectified Linear Unit, ReLU), that is, a nonlinear activation function. The second FC layer projects it back to the original channel dimension and outputs the importance coefficient of each channel through the Sigmoid activation function.

[0082] For example, Figure 5 As shown, the electronic device uses a network structure including a fully connected layer 1 and a fully connected layer 2 using the SENet module, and respectively combines activation function 1 (such as ReLU function) and activation function 2 (such as Sigmoid activation function) to readjust the compressed average STFT feature information STFT_AVG, thereby generating a weight value STFT_WEIGHT for each channel. The fully connected layer 1 is usually used for frequency dimensionality reduction, and the fully connected layer 2 is used to restore the original frequency number.

[0083] Step 203a3: The electronic device performs weighted processing on the frequency features in the first STFT feature information based on the weight value corresponding to each frequency feature to obtain second STFT feature information.

[0084] Exemplarily, after obtaining the weight value corresponding to each frequency feature, the electronic device can perform weighted processing on the original input first STFT feature information by multiplying the frequencies one by one, using the generated weight value STFT_WEIGHT, thereby dynamically adjusting the characteristic response of each frequency and generating new second STFT feature information W_STFT with weight information.

[0085] In an embodiment of the present application, the electronic device uses the SENet module to compress, excite, and weight the input first STFT feature information to obtain new second STFT feature information W_STFT with weight information. The second STFT feature information can effectively enhance the feature representation capability of the voiceprint recognition model, thereby improving the performance of the voiceprint recognition model in the voiceprint recognition task.

[0086] Step 203b: The electronic device performs convolution processing on the frequency characteristics in the second STFT characteristic information to obtain third STFT characteristic information.

[0087] It should be noted that the second STFT feature information obtained through the above step 203a comprehensively considers the weights of various frequency features in the first STFT feature information. On this basis, the results of various frequency feature combinations of the audio are combined to accurately identify the identity of the speaker, that is, the voiceprint feature.

[0088] Optionally, in an embodiment of the present application, the electronic device may utilize a Residual Networks with Multi-Scale Feature Aggregation (Res2Net) module to group and convolve different frequency features to achieve the effect of integrating multiple frequency features, thereby extracting features that are more conducive to voiceprint recognition.

[0089] It is understandable that the Res2Net module is a core component of a new type of convolutional neural network architecture. It has made a series of innovations based on the traditional residual network, aiming to improve the feature expression ability of the model. Specifically, it refers to a mechanism that increases multi-scale features through combination, thereby enhancing the feature representation ability of multi-channel input.

[0090] Optionally, in the embodiment of the present application, the above step 203b can be specifically implemented through the following steps 203b1 to 203b4.

[0091] Step 203b1: The electronic device groups the frequency features in the second STFT feature information to obtain N groups of frequency features.

[0092] Wherein, N is an integer greater than 1.

[0093] Exemplarily, the electronic device divides the frequency features in the second STFT feature information W_STFT with weight information generated above into a plurality of groups, where the plurality of groups represent a plurality of different scales (e.g., frequency bands), that is, each group represents a scale (e.g., a low frequency band, a medium frequency band, or a high frequency band). Assuming that there are nine frequency features in the second STFT feature information W_STFT, these frequency features are divided into three groups, which can be represented by the following formula (2):

[0094] W_STFT=[[C1,C2,C3],[C4,C5,C6],[C7,C8,C9]] (2)

[0095] Among them, frequency features C1, C2, and C3 are a group, frequency features C4, C5, and C6 are a group, and frequency features C7, C8, and C9 are a group.

[0096] Step 203b2: The electronic device performs convolution processing on the first group of frequency features in the N groups of frequency features to obtain a first output feature.

[0097] Step 203b3: The electronic device performs convolution processing on the i-th group of frequency features and the i-1-th output feature in the N groups of frequency features to obtain the i-th output feature, i∈[2,N], and i is an integer.

[0098] Step 203b4: The electronic device performs fusion processing on the N output features corresponding to the N groups of frequency features to obtain third STFT feature information.

[0099] For example, Figure 6 As shown, the electronic device performs a convolution operation on each frequency feature group from front to back. In each step, the output feature of the previous group is fused with the input feature of the current group by addition. For example, in the first step, a convolution operation is performed on the frequency feature group [C1, C2, C3] to obtain Conv1. In the second step, a convolution operation is performed on the frequency feature group [C4, C5, C6], and the result of the convolution operation is fused with Conv1 by addition to obtain Conv2. In the third step, a convolution operation is performed on the frequency feature group [C7, C8, C9], and the result of the convolution operation is fused with Conv2 by addition to obtain Conv3. Finally, Conv1, Conv2, and Conv3 are fused to generate the final output feature, i.e., the third STFT feature information N_STFT.

[0100] In an embodiment of the present application, after the electronic device divides the frequency features in the second STFT feature information into multiple groups, it performs a convolution operation on each frequency feature group from front to back. In each step, the electronic device fuses the output features of the previous group with the input features of the current group by adding them. This step-by-step convolution and fusion method enables each group to be combined with the information of the previous group, thereby gradually capturing richer multi-scale features. In this way, the newly obtained third STFT feature information can effectively capture frequency features of different scales, thereby improving the multi-scale feature representation capability of the voiceprint recognition model.

[0101] Step 203c: The electronic device processes the first audio data based on the third STFT feature information to obtain a first feature vector.

[0102] In the embodiment of the present application, the third STFT feature information adjusts the weight values ​​of each frequency feature in the first STFT feature information in the frequency domain and obtains multi-scale features. As for part of the first audio data in the time domain, such as blank audio or background noise, the electronic device can continue to process the first audio data in combination with the third STFT feature information to focus on the audio data with specific time sequence meaning, thereby reducing the influence of blank audio and background noise on the recognition effect of the voiceprint recognition model and improving the effect of the voiceprint recognition model.

[0103] It should be noted that the audio data with specific time sequence mentioned above refers to: valid audio data within a certain period of time, and valid refers to audio data with actual meaning, that is, it does not include blank audio and background noise audio.

[0104] Optionally, in an embodiment of the present application, the electronic device may utilize an attention mechanism Attention module to process the first audio data based on the third STFT feature information to reduce the impact of blank audio and noise on the audio signal in the time domain.

[0105] It is understandable that the Attention module is an important component of deep learning. It imitates the human attention mechanism, allowing the model to dynamically focus on important information according to task requirements when processing input data, thereby improving the performance and accuracy of the model. The core idea of ​​the Attention module is to introduce an attention mechanism to help the model assign different weights to each part of the input, thereby extracting more critical and important information and enabling the model to make more accurate judgments. This mechanism is similar to the way humans observe external things, that is, people usually pay attention to important local information first, and then combine them to form an overall impression.

[0106] Optionally, in the embodiment of the present application, the above step 203c can be specifically implemented through the following steps 203c1 to 203c4.

[0107] Step 203c1: The electronic device converts the third STFT feature information into a feature matrix.

[0108] In the embodiment of the present application, the above-mentioned feature matrix includes frequency features corresponding to different audio durations.

[0109] Exemplarily, the third STFT feature information N_STFT is converted into an F_audio feature matrix of shape n*d, which can be expressed by the following formula (3):

[0110] F_audio=[N_STFT1,N_STFT2,…,N_STFT n ] F_audio∈R n*d (3)

[0111] Among them, n represents the audio duration, and d represents the number of frequency domain feature dimensions.

[0112] Step 203c2: The electronic device maps the feature matrix into three vectors.

[0113] In the embodiment of the present application, the above three vectors include a query vector, a key vector and a value vector.

[0114] In an embodiment of the present application, the electronic device can use the processing logic in the Attention module to map the feature matrix obtained in the above step 203c1 into three vectors: Query (query vector), Key (key vector) and Value (value vector) to reduce the impact of blank parts and noise in the first audio data on the prediction of the voiceprint recognition model, and ensure that the voiceprint recognition task focuses on the effective audio part.

[0115] Step 203c3: The electronic device performs dot product and normalization processing on the query vector and the key vector to obtain a first vector, and performs weighted summation on the first vector and the value vector to obtain an intermediate vector.

[0116] It is understandable that the core architecture of the Attention module usually includes three components: Query (query vector), Key (key vector) and Value (value vector), as well as Softmax normalized attention score and weighted summation. Specifically, it includes the following steps:

[0117] Query, Key, and Value calculation: For the input sequence, the model first calculates the Query, Key, and Value vectors respectively through three different linear transformations (usually implemented by a learnable weight matrix).

[0118] Calculation of attention scores: Then, the degree of match between each query vector and all key vectors is calculated to form an attention score matrix. This degree of match is usually calculated using dot product or cosine similarity. ;

[0119] Calculation of attention weight: Apply the Softmax function to each row in the attention score matrix (corresponding to a Query vector) for normalization to obtain the attention weight vector.

[0120] Calculation of context vector: Finally, the attention weight vector is weighted and summed with the corresponding Value vector set to obtain the context vector for each position. This context vector contains the most relevant information in the input sequence for the current task, which can help the model generate output better.

[0121] For example, Figure 7 As shown, the electronic device uses the processing logic in the Attention module to map the input feature matrix into three vectors: a query vector, a key vector, and a value vector. By performing dot product and normalization (for example, using a softmax function) on the query vector and the key vector, a 1*n vector is obtained to represent the influence factor of the n inputs on the final output in the time domain, and a weighted sum is performed with the corresponding value vector to generate a state representation token of the entire first audio data, that is, an intermediate vector. The specific expression is as shown in the following formula (4);

[0122]

[0123] It can be understood that the above-mentioned state feature token contains key information of the input first audio data and can be used for subsequent task processing.

[0124] Step 203c4: The electronic device converts the intermediate vector into a first eigenvector.

[0125] In an embodiment of the present application, the electronic device converts the above intermediate vector into a first feature vector through an embedding layer, and the first feature vector is used to characterize the voiceprint features of the audio data predicted by the first voiceprint recognition model.

[0126] It can be understood that the embedding layer can map the original data from a high-dimensional space to a low-dimensional space, which helps to reduce the complexity of the data and the demand for computing resources, while improving the training and reasoning efficiency of the model.

[0127] Exemplarily, the electronic device processes the state representation token through an embedding layer to obtain an embedding vector emb_pred corresponding to the first audio data.

[0128] In an embodiment of the present application, an electronic device introduces an Attention module, so that when training the first voiceprint recognition model to obtain the trained second voiceprint recognition model, it can focus more on the first audio data with specific time sequence meaning, reduce the impact of blank parts and noise in the first audio data on the prediction of the voiceprint recognition model, and improve the recognition effect of the voiceprint recognition model.

[0129] Step 204: The electronic device trains the first voiceprint recognition model based on the target voiceprint feature and the first feature vector to obtain a second voiceprint recognition model.

[0130] Optionally, in the embodiment of the present application, the above step 204 can be specifically implemented by the following steps 204a and 204b.

[0131] Step 204a: The electronic device calculates a loss value based on the target voiceprint feature and the first feature vector.

[0132] For example, Figure 7 As shown, the electronic device uses one-hot encoding to encode the audio speaker identity SPK_ID obtained in step 201 to obtain emb_real, that is, the target voiceprint feature, and uses a loss function, such as the Angular Additive Margin Loss (AAMLoss) function to calculate the loss value of emb_real and emb_pred obtained in step 203c4.

[0133] It is understandable that AAMLoss is a loss function commonly used in fields such as face recognition and voiceprint recognition. It aims to improve the model's distinguishing ability by optimizing the angular relationship between embedding vectors. AAMLoss enhances intra-class compactness and inter-class separability by introducing an angular additive margin.

[0134] Step 204b: The electronic device updates the parameters in the first voiceprint recognition model based on the loss value to obtain a second voiceprint recognition model.

[0135] Optionally, in an embodiment of the present application, the electronic device can calculate the gradient of each parameter in the first voiceprint recognition model based on the loss value and the gradient of the loss function through a back propagation algorithm, and use an optimization algorithm to update the weight and bias parameters of the model according to the calculated gradient, thereby obtaining a trained second voiceprint recognition model.

[0136] The embodiment of the present application provides a training method for a voiceprint recognition model. After obtaining first audio data, first STFT feature information including time domain features and frequency domain features of the first audio data can be obtained based on an audio signal corresponding to the first audio data. Since the first STFT feature information is directly extracted from the audio signal, detailed distribution information of the audio signal in time and frequency is retained, and the audio signal corresponding to the first audio data is closer, and more information about the first audio data itself can be provided. In the process of training the voiceprint recognition model, the first STFT feature information that is closer to the audio signal and includes more comprehensive first audio data features is used as input, so that the first voiceprint recognition model can more quickly and accurately capture the unique voiceprint features of the first audio data for model training, and obtain a second voiceprint recognition model. Therefore, the trained second voiceprint recognition model can accurately perform voiceprint recognition, thereby improving the voiceprint recognition effect of the voiceprint recognition model.

[0137] Optionally, in the embodiment of the present application, after the above step 204, the training method of the voiceprint recognition model provided in the embodiment of the present application further includes the following steps 205 to 207.

[0138] Step 205: The electronic device inputs the second audio data into the second voiceprint recognition model to obtain an audio signal of the second audio data.

[0139] For example, taking the electronic device as a mobile phone, the mobile phone uses the second voiceprint recognition model to detect whether the voiceprint feature of the audio to be replicated uploaded by the user is the voiceprint feature of a sensitive person, such as Figure 8 As shown, the mobile phone displays a replicated audio upload interface 60, and the user uploads the audio to be replicated to the mobile phone. The mobile phone inputs the audio to be replicated into the second voiceprint recognition model to obtain the audio signal of the audio to be replicated.

[0140] Step 206: The electronic device obtains fourth STFT feature information based on the audio signal of the second audio data.

[0141] In the embodiment of the present application, the fourth STFT feature information includes the time domain features and frequency domain features of the second audio data.

[0142] Exemplarily, the mobile phone obtains the corresponding fourth STFT feature information based on the audio signal of the audio to be reproduced.

[0143] It should be noted that the specific scheme for the electronic device to obtain the fourth STFT characteristic information based on the audio signal of the second audio data can refer to the description of the specific scheme for obtaining the first STFT characteristic information mentioned above, and the embodiment of the present application will not be repeated here.

[0144] Step 207: The electronic device performs feature processing on the fourth STFT feature information and outputs a second feature vector.

[0145] In the embodiment of the present application, the second feature vector represents the voiceprint feature of the second audio data.

[0146] Exemplarily, the mobile phone performs feature processing on the fourth STFT feature information, outputs a second feature vector representing the voiceprint feature of the audio to be replicated, and the mobile phone compares the second feature vector with the voiceprint feature of a preset sensitive person, and obtains that the second feature vector is similar to the voiceprint feature of the preset sensitive person, such as Fig. 9 As shown, a prompt message 61 is displayed on the replica audio upload interface 60, for example, "The audio you uploaded is similar to the pronunciation of a sensitive person, please reselect the replica audio."

[0147] It should be noted that, for the specific scheme of the electronic device performing feature processing on the fourth STFT feature information and outputting the second feature vector, reference can be made to the description of the specific scheme of the first feature vector mentioned above, and the embodiments of the present application will not be elaborated here.

[0148] In an embodiment of the present application, the electronic device can use the trained second voiceprint recognition model to perform voiceprint recognition on the second audio data, and can accurately output a second feature vector representing the voiceprint characteristics of the second audio data, thereby improving the voiceprint recognition effect of the voiceprint recognition model.

[0149] The following is a schematic illustration of the implementation process of the voiceprint recognition model training method provided in the embodiment of the present application through a specific implementation method. Fig.10 As shown, the implementation process of the voiceprint recognition model training method provided in the embodiment of the present application may include the following steps 81 to 90.

[0150] Step 81: The mobile phone reads the original audio information to obtain the audio signal, audio sampling rate and audio speaker number.

[0151] Step 82: The mobile phone obtains multiple audio signals of equal length by resampling, signal clipping, signal padding, etc., and performs Fourier transform on the audio signals to obtain first STFT feature information.

[0152] Step 83: The mobile phone uses the SENet module to adjust the weights of the frequency features in the first STFT feature information to obtain the second STFT feature information.

[0153] Step 84: The mobile phone uses the Res2Net module to group, convolve and combine the frequency features in the second STFT feature information to obtain the third STFT feature information.

[0154] Step 85: The mobile phone uses the Attention module to remove the influence of blank audio and background noise on the original audio data, and obtains the predicted voiceprint features of the original audio data, that is, the first feature vector.

[0155] Step 86: The mobile phone obtains the target voiceprint feature using the audio speaker number, calculates the loss value according to the target voiceprint feature and the first feature vector, and adjusts the model parameters of the first voiceprint recognition model according to the loss value to obtain the second voiceprint recognition model.

[0156] Step 87: The mobile phone uses the trained second voiceprint recognition model to detect whether the voiceprint features of the audio to be replicated uploaded by the user are the voiceprint features of a sensitive person, and the user uploads the audio to be replicated to the mobile phone.

[0157] Step 88: The mobile phone inputs the audio to be replicated into a second voiceprint recognition model to obtain an audio signal of the audio to be replicated.

[0158] Step 89: The mobile phone obtains corresponding fourth STFT feature information based on the audio signal of the audio to be replicated.

[0159] Step 90: The mobile phone performs feature processing on the fourth STFT feature information, and outputs a second feature vector representing the voiceprint feature of the audio to be replicated.

[0160] In an embodiment of the present application, the electronic device uses the STFT features obtained after Fourier transform as the model input. Compared with the existing model using the FBank features, the STFT features are closer to the original appearance of the audio waveform and can learn more audio features. At the same time, the electronic device uses the Attention module to reduce the impact of blank audio and background noise on voiceprint recognition, so that as many audio samples as possible can be used during model training, thereby improving the model effect.

[0161] It should be noted that the implementation process of the above steps 81 to 90 is the same as the implementation process in the embodiment of the training method of the voiceprint recognition model. For the parts not described in detail in the embodiment of the present application, please refer to the relevant description of the above embodiment, and the embodiment of the present application will not be repeated here.

[0162] The following is a schematic illustration of the data stream transmission in the training method of the voiceprint recognition model provided in the embodiment of the present application through a specific implementation method. Fig.11 As shown, Fig.11The training data is the first audio data, SIGNAL is the audio signal of the first audio data, B (Batch) is the batch size, indicating the number of audio signals processed simultaneously, T (Time Frames) is the number of time frames, indicating the number of frames into which the audio signal is divided, and SIGNAL [B, T] indicates a batch containing B audio signals, each of which is divided into T time frames. STFT is the first STFT feature information mentioned above, C is the number of channels or the number of frequency bins, indicating the number of frequency features, and STFT [B, C, T] indicates that a batch contains B audio signals. For each audio signal, it is converted into a spectrogram or feature map through STFT. This spectrogram or feature map has C channels, each channel corresponds to a different frequency component, and at the same time, this spectrogram or feature map is also divided into T time frames to represent the frequency characteristics of the audio signal in different time periods. Similarly, W_STFT is the second STFT feature information, N_STFT is the third STFT feature information, token is an intermediate vector containing the key information of the input first audio data, and the meanings of W_STFT[B,C,T], N_STFT[B,C,T], and token[B,C] can refer to STFT[B,C,T], which will not be repeated here. SPK_ID is the audio speaker identity. SPK_ID is encoded with one-hot encoding to obtain emb_real for characterizing the target voiceprint feature. The intermediate vector token is processed by a linear layer to obtain the voiceprint feature emb_pred for characterizing the first audio data predicted by the first voiceprint recognition model. Dim represents the dimension of the embedding vector. emb_pred[B,dim] represents a batch containing B audio signals, each of which is predicted by the first voiceprint recognition model as a dim-dimensional voiceprint feature vector. emb_real[B,dim] represents a batch containing B audio signals, each of which corresponds to a dim-dimensional voiceprint feature vector.

[0163] The embodiment of the present application provides a voiceprint recognition method. The voiceprint recognition method provided by the embodiment of the present application may include the following steps 301 to 303.

[0164] Step 301: The electronic device inputs second audio data into a second voiceprint recognition model to obtain an audio signal of the second audio data.

[0165] In an embodiment of the present application, the second voiceprint recognition model is trained based on STFT feature information, and the STFT feature information includes time domain features and frequency domain features of audio data.

[0166] Step 302: The electronic device obtains fourth STFT feature information based on the audio signal of the second audio data.

[0167] In the embodiment of the present application, the fourth STFT feature information includes the time domain features and frequency domain features of the second audio data.

[0168] Step 303: The electronic device performs feature processing on the fourth STFT feature information and outputs a second feature vector.

[0169] In the embodiment of the present application, the second feature vector represents the voiceprint feature of the second audio data.

[0170] It should be noted that the specific implementation of the above steps 301 to 303 can refer to the specific description in the above embodiments, and the embodiments of the present application will not be repeated here.

[0171] Optionally, in an embodiment of the present application, before the above step 301, the above-mentioned voiceprint recognition model training method may also be executed, that is, the above-mentioned steps 201 to 204, and related schemes, and the specific embodiments of the present application are not described in detail here.

[0172] The embodiment of the present application provides a voiceprint recognition method, inputs the second audio data into the second voiceprint recognition model, obtains the audio signal of the second audio data, obtains the fourth STFT feature information including the time domain features and frequency domain features of the second audio data based on the audio signal, performs feature processing on the fourth STFT feature information, and outputs the second feature vector representing the voiceprint features of the second audio data. In this solution, since the second voiceprint recognition model is trained based on the STFT feature information, and the STFT feature information is directly extracted from the audio signal, the detailed distribution information of the audio signal in time and frequency is retained, and the audio signal corresponding to the audio data is closer, and more information about the audio data itself can be provided. Therefore, the second voiceprint recognition model trained based on the STFT feature information can output more accurate voiceprint features. In the embodiment of the present application, the second audio data is input into the second voiceprint recognition model, and an accurate voiceprint recognition result of the second audio data can be obtained, thereby improving the voiceprint recognition effect.

[0173] It should be noted that the above-mentioned method embodiments, or various possible implementation methods in each method embodiment, can be executed separately, or any two or more can be executed in combination with each other. The specific implementation method can be determined according to actual usage requirements, and some embodiments of the present application do not limit this.

[0174] The training method of the voiceprint recognition model provided in the embodiment of the present application can be performed by a training device for the voiceprint recognition model. In the embodiment of the present application, the training method of the voiceprint recognition model performed by the training device for the voiceprint recognition model is taken as an example to illustrate the training device for the voiceprint recognition model provided in the embodiment of the present application.

[0175] Fig.12 A possible structural diagram of a training device for a voiceprint recognition model involved in some embodiments of the present application is shown. Fig.12 As shown, the training device 20 of the voiceprint recognition model may include: an acquisition module 21 and a processing module 22. The acquisition module 21 is used to acquire audio information of the first audio data, the audio information includes an audio signal and a target voiceprint feature; and, based on the audio signal, acquire first STFT feature information, the first STFT feature information includes time domain features and frequency domain features of the first audio data. The processing module 22 is used to perform feature processing on the first STFT feature information acquired by the acquisition module 21 to obtain a first feature vector, the first feature vector represents the voiceprint feature of the first audio data; and, based on the target voiceprint feature acquired by the acquisition module 21 and the first feature vector, train the first voiceprint recognition model to obtain a second voiceprint recognition model.

[0176] In a possible implementation, the first audio data includes multiple audio data, and the audio information also includes an audio sampling rate. The acquisition module 21 is specifically used to, when the audio sampling rate is equal to the target sampling rate, if the lengths of the audio signals of the multiple audio data are not equal, perform signal processing on the audio signals of the multiple audio data, and the signal processing includes at least one of signal clipping and signal padding; and perform Fourier transform on the audio signals of the multiple audio data after signal processing to obtain the first STFT feature information.

[0177] In one possible implementation, the frequency domain feature includes at least one frequency feature, and the processing module 22 is specifically used to adjust the weight of each frequency feature in the first STFT feature information to obtain the second STFT feature information; and, perform convolution processing on the frequency features in the second STFT feature information to obtain the third STFT feature information; and, based on the third STFT feature information, process the first audio data to obtain the first feature vector.

[0178] In a possible implementation, the processing module 22 is specifically used to perform global average pooling on each frequency feature in the first STFT feature information to obtain average STFT feature information; and, perform frequency adjustment processing on the average STFT feature information to obtain a weight value corresponding to each frequency feature; and, based on the weight value corresponding to each frequency feature, perform weighted processing on the frequency features in the first STFT feature information to obtain the second STFT feature information.

[0179] In a possible implementation, the processing module 22 is specifically used to: group the frequency features in the second STFT feature information to obtain N groups of frequency features, where N is an integer greater than 1; and, perform convolution processing on the first group of frequency features in the N groups of frequency features to obtain the first output feature; and, perform convolution processing on the i-th group of frequency features and the i-1-th output feature in the N groups of frequency features to obtain the i-th output feature, i∈[2,N], and i is an integer; and, perform fusion processing on the N output features corresponding to the N groups of frequency features to obtain the third STFT feature information.

[0180] The embodiment of the present application provides a training device for a voiceprint recognition model. After the training device for the voiceprint recognition model obtains the first audio data, it can obtain first STFT feature information including time domain features and frequency domain features of the first audio data based on the audio signal corresponding to the first audio data. Since the first STFT feature information is directly extracted from the audio signal, the detailed distribution information of the audio signal in time and frequency is retained, and it is closer to the audio signal corresponding to the first audio data, and can provide more information about the first audio data itself. In the process of training the first voiceprint recognition model, the training device for the voiceprint recognition model uses the first STFT feature information that is closer to the audio signal and includes more comprehensive first audio data features as input, so that the first voiceprint recognition model can more quickly and accurately capture the unique voiceprint features of the first audio data for model training, and obtain the second voiceprint recognition model. Therefore, the trained second voiceprint recognition model can accurately perform voiceprint recognition, thereby improving the voiceprint recognition effect of the voiceprint recognition model.

[0181] The training device of the voiceprint recognition model in the embodiment of the present application can be an electronic device, or a component in the electronic device, such as an integrated circuit or a chip. The electronic device can be a terminal, or other devices other than a terminal. Exemplarily, the electronic device can be a mobile phone, a tablet computer, a laptop computer, a PDA, a car-mounted electronic device, a mobile Internet device (Mobile Internet Device, MID), an augmented reality (augmented reality, AR) / virtual reality (virtual reality, VR) device, a robot, a wearable device, an ultra-mobile personal computer (ultra-mobile personal computer, UMPC), a netbook or a personal digital assistant (personal digital assistant, PDA), etc. It can also be a server, a network attached storage (Network Attached Storage, NAS), a personal computer (personal computer, PC), a television (television, TV), a teller machine or a self-service machine, etc., which is not specifically limited in the embodiment of the present application.

[0182] The training device of the voiceprint recognition model in the embodiment of the present application may be a device having an operating system. The operating system may be an Android operating system, an iOS operating system, or other possible operating systems, which are not specifically limited in the embodiment of the present application.

[0183] The training device for the voiceprint recognition model provided in the embodiments of the present application can implement the various processes implemented in the various embodiments of the training method for the voiceprint recognition model described above, and will not be described again here to avoid repetition.

[0184] Alternatively, if Fig.13 As shown, an embodiment of the present application further provides an electronic device 1000, including a processor 1001 and a memory 1002, wherein the memory 1002 stores a program or instruction that can be executed on the processor 1001, and when the program or instruction is executed by the processor 1001, each step of each embodiment of the training method of the above-mentioned voiceprint recognition model is implemented, and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.

[0185] It should be noted that the electronic devices in the embodiments of the present application include the mobile electronic devices and non-mobile electronic devices mentioned above.

[0186] Fig.14 A schematic diagram of the hardware structure of an electronic device to implement an embodiment of the present application.

[0187] The electronic device 100 includes but is not limited to components such as a radio frequency unit 101, a network module 102, an audio output unit 103, an input unit 104, a sensor 105, a display unit 106, a user input unit 107, an interface unit 108, a memory 109, and a processor 110.

[0188] Those skilled in the art will appreciate that the electronic device 100 may also include a power source (such as a battery) for supplying power to various components, and the power source may be logically connected to the processor 110 through a power management system, thereby implementing functions such as managing charging, discharging, and power consumption management through the power management system. Fig.14 The electronic device structure shown in the figure does not constitute a limitation on the electronic device. The electronic device may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently, which will not be described in detail here.

[0189] Among them, the above-mentioned processor 110 is used to obtain audio information of the first audio data, and the audio information includes an audio signal and a target voiceprint feature; and, based on the above-mentioned audio signal, obtain first STFT feature information, and the first STFT feature information includes time domain features and frequency domain features of the first audio data; and, perform feature processing on the above-mentioned first STFT feature information to obtain a first feature vector, and the first feature vector represents the voiceprint feature of the first audio data; and, based on the above-mentioned target voiceprint feature and the first feature vector, train a first voiceprint recognition model to obtain a second voiceprint recognition model.

[0190] Optionally, the first audio data includes multiple audio data, and the audio information also includes an audio sampling rate. The processor 110 is further configured to, when the audio sampling rate is equal to the target sampling rate, if the lengths of the audio signals of the multiple audio data are not equal, perform signal processing on the audio signals of the multiple audio data, the signal processing including at least one of signal clipping and signal padding; and perform Fourier transform on the audio signals of the multiple audio data after signal processing to obtain the first STFT feature information.

[0191] Optionally, the frequency domain feature includes at least one frequency feature, and the processor 110 is specifically used to adjust the weight of each frequency feature in the first STFT feature information to obtain second STFT feature information; and, perform convolution processing on the frequency features in the second STFT feature information to obtain third STFT feature information; and, based on the third STFT feature information, process the first audio data to obtain the first feature vector.

[0192] Optionally, the processor 110 is specifically used to perform global average pooling on each frequency feature in the first STFT feature information to obtain average STFT feature information; and perform frequency adjustment processing on the average STFT feature information to obtain a weight value corresponding to each frequency feature; and, based on the weight value corresponding to each frequency feature, perform weighted processing on the frequency features in the first STFT feature information to obtain the second STFT feature information.

[0193] Optionally, the processor 110 is specifically used to group the frequency features in the second STFT feature information to obtain N groups of frequency features, where N is an integer greater than 1; and, perform convolution processing on the first group of frequency features in the N groups of frequency features to obtain the first output feature; and, perform convolution processing on the i-th group of frequency features and the i-1-th output feature in the N groups of frequency features to obtain the i-th output feature, i∈[2,N], and i is an integer; and, perform fusion processing on the N output features corresponding to the N groups of frequency features to obtain the third STFT feature information.

[0194] An embodiment of the present application provides an electronic device. After the electronic device obtains first audio data, it can obtain first STFT feature information including time domain features and frequency domain features of the first audio data based on an audio signal corresponding to the first audio data. Since the first STFT feature information is directly extracted from the audio signal, detailed distribution information of the audio signal in time and frequency is retained, and it is closer to the audio signal corresponding to the first audio data, and can provide more information about the first audio data itself. In the process of training a first voiceprint recognition model, the electronic device uses the first STFT feature information that is closer to the audio signal and includes more comprehensive first audio data features as input, so that the first voiceprint recognition model can more quickly and accurately capture the unique voiceprint features of the first audio data for model training, and obtain a second voiceprint recognition model. Therefore, the trained second voiceprint recognition model can accurately perform voiceprint recognition, thereby improving the voiceprint recognition effect of the voiceprint recognition model.

[0195] The electronic device provided in the embodiment of the present application can implement each process implemented in the above method embodiment, and can achieve the same technical effect. To avoid repetition, it is not repeated here. The beneficial effects of various implementations in this embodiment can be specifically referred to the beneficial effects of the corresponding implementations in the above method embodiment. To avoid repetition, it is not repeated here.

[0196] It should be understood that in the embodiment of the present application, the input unit 104 may include a graphics processor (Graphics Processing Unit, GPU) 1041 and a microphone 1042, and the graphics processor 1041 processes the image data of a static picture or video obtained by an image capture device (such as a camera) in a video capture mode or an image capture mode. The display unit 106 may include a display panel 1061, and the display panel 1061 may be configured in the form of a liquid crystal display, an organic light emitting diode, etc. The user input unit 107 includes a touch panel 1071 and at least one of other input devices 1072. The touch panel 1071 is also called a touch screen. The touch panel 1071 may include two parts: a touch detection device and a touch controller. Other input devices 1072 may include, but are not limited to, a physical keyboard, function keys (such as volume control keys, switch keys, etc.), a trackball, a mouse, and a joystick, which will not be repeated here.

[0197] The memory 109 can be used to store software programs and various data. The memory 109 may mainly include a first storage area for storing programs or instructions and a second storage area for storing data, wherein the first storage area may store an operating system, an application program or instructions required for at least one function (such as a sound playback function, an image playback function, etc.), etc. In addition, the memory 109 may include a volatile memory or a non-volatile memory, or the memory 109 may include both volatile and non-volatile memories. Among them, the non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), a static random access memory (SRAM), a dynamic random access memory (DRAM), a synchronous dynamic random access memory (SDRAM), a double data rate synchronous dynamic random access memory (DDRSDRAM), an enhanced synchronous dynamic random access memory (ESDRAM), a synchronous link dynamic random access memory (SLDRAM) and a direct memory bus random access memory (DRRAM). The memory 109 in the embodiment of the present application includes but is not limited to these and any other suitable types of memory.

[0198] The processor 110 may include one or more processing units; optionally, the processor 110 integrates an application processor and a modem processor, wherein the application processor mainly processes operations related to an operating system, a user interface, and application programs, and the modem processor mainly processes wireless communication signals, such as a baseband processor. It is understandable that the modem processor may not be integrated into the processor 110.

[0199] The embodiment of the present application also provides a readable storage medium, on which a program or instruction is stored. When the program or instruction is executed by a processor, each process of the embodiment of the training method of the above-mentioned voiceprint recognition model is implemented, and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.

[0200] The processor is the processor in the electronic device described in the above embodiment. The readable storage medium includes a computer readable storage medium, such as a computer read-only memory ROM, a random access memory RAM, a magnetic disk or an optical disk.

[0201] An embodiment of the present application further provides a chip, which includes a processor and a communication interface, wherein the communication interface is coupled to the processor, and the processor is used to run programs or instructions to implement the various processes of the above-mentioned voiceprint recognition model training method embodiment, and can achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0202] It should be understood that the chip mentioned in the embodiments of the present application can also be called a system-level chip, a system chip, a chip system or a system-on-chip chip, etc.

[0203] An embodiment of the present application provides a computer program product, which is stored in a storage medium. The program product is executed by at least one processor to implement the various processes of the embodiment of the training method for the voiceprint recognition model as described above, and can achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0204] It should be noted that, in this article, the terms "comprise", "include" or any other variant thereof are intended to cover non-exclusive inclusion, so that the process, method, article or device including a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise one..." do not exclude the presence of other identical elements in the process, method, article or device including the element. In addition, it should be pointed out that the scope of the methods and devices in the embodiments of the present application is not limited to performing functions in the order shown or discussed, and may also include performing functions in a substantially simultaneous manner or in reverse order according to the functions involved, for example, the described method may be performed in an order different from that described, and various steps may also be added, omitted, or combined. In addition, the features described with reference to certain examples may be combined in other examples.

[0205] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a computer software product, which is stored in a storage medium (such as ROM / RAM, a disk, or an optical disk), and includes a number of instructions for a terminal (which can be a mobile phone, a computer, a server, or a network device, etc.) to execute the methods described in each embodiment of the present application.

[0206] The embodiments of the present application are described above in conjunction with the accompanying drawings, but the present application is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the guidance of the present application, ordinary technicians in this field can also make many forms without departing from the purpose of the present application and the scope of protection of the claims, all of which are within the protection of the present application.

Claims

1. A method for training a voiceprint recognition model, characterized in that: include: Acquire audio information of the first audio data, where the audio information includes an audio signal and a target voiceprint feature; Based on the audio signal, obtaining first short-time Fourier transform (STFT) feature information, where the first STFT feature information includes time domain features and frequency domain features of the first audio data; Performing feature processing on the first STFT feature information to obtain a first feature vector, where the first feature vector represents a voiceprint feature of the first audio data; Based on the target voiceprint feature and the first feature vector, a first voiceprint recognition model is trained to obtain a second voiceprint recognition model.

2. The method according to claim 1, characterized in that The first audio data includes a plurality of audio data, and the audio information also includes an audio sampling rate; The obtaining first STFT feature information based on the audio signal includes: When the audio sampling rate is equal to the target sampling rate, if the lengths of the audio signals of the plurality of audio data are not equal, performing signal processing on the audio signals of the plurality of audio data, wherein the signal processing includes at least one of signal clipping and signal padding; Perform Fourier transform on the audio signals of the plurality of audio data after signal processing to obtain the first STFT feature information.

3. The method according to claim 1, characterized in that The frequency domain feature includes at least one frequency feature, and the feature processing of the first STFT feature information to obtain a first feature vector includes: Performing weight adjustment on each frequency feature in the first STFT feature information to obtain second STFT feature information; Performing convolution processing on the frequency characteristics in the second STFT characteristic information to obtain third STFT characteristic information; The first audio data is processed based on the third STFT feature information to obtain the first feature vector.

4. The method according to claim 3, characterized in that The step of weighting each frequency feature in the first STFT feature information to obtain second STFT feature information includes: Performing global average pooling on each frequency feature in the first STFT feature information to obtain average STFT feature information; Performing frequency adjustment processing on the average STFT feature information to obtain a weight value corresponding to each frequency feature; Based on the weight value corresponding to each frequency feature, the frequency features in the first STFT feature information are weighted to obtain the second STFT feature information.

5. The method according to claim 3, characterized in that: The performing convolution processing on the frequency feature in the second STFT feature information to obtain the third STFT feature information includes: Grouping the frequency features in the second STFT feature information to obtain N groups of frequency features, where N is an integer greater than 1; Performing convolution processing on a first group of frequency features among the N groups of frequency features to obtain a first output feature; Performing convolution processing on the i-th group of frequency features and the i-1-th output feature in the N groups of frequency features to obtain the i-th output feature, i∈[2,N], and i is an integer; The N output features corresponding to the N groups of frequency features are fused to obtain the third STFT feature information.

6. A training device for a voiceprint recognition model, characterized in that: include: Acquisition module and processing module; The acquisition module is used to acquire audio information of the first audio data, where the audio information includes an audio signal and a target voiceprint feature; And, based on the audio signal, obtaining first STFT feature information, the first STFT feature information including time domain features and frequency domain features of the first audio data; The processing module is used to perform feature processing on the first STFT feature information obtained by the acquisition module to obtain a first feature vector, which represents the voiceprint feature of the first audio data; and to train a first voiceprint recognition model based on the target voiceprint feature obtained by the acquisition module and the first feature vector to obtain a second voiceprint recognition model.

7. The device according to claim 6, characterized in that The first audio data includes a plurality of audio data, and the audio information also includes an audio sampling rate; The acquisition module is specifically used to, when the audio sampling rate is equal to the target sampling rate, if the lengths of the audio signals of the multiple audio data are not equal, perform signal processing on the audio signals of the multiple audio data, wherein the signal processing includes at least one of signal clipping and signal padding; and perform Fourier transform on the audio signals of the multiple audio data after signal processing to obtain the first STFT feature information.

8. The device according to claim 6, characterized in that The frequency domain feature includes at least one frequency feature, and the processing module is specifically used to adjust the weight of each frequency feature in the first STFT feature information to obtain second STFT feature information; and, perform convolution processing on the frequency features in the second STFT feature information to obtain third STFT feature information; and, based on the third STFT feature information, process the first audio data to obtain the first feature vector.

9. The device according to claim 8, characterized in that The processing module is specifically used to perform global average pooling on each frequency feature in the first STFT feature information to obtain average STFT feature information; and, perform frequency adjustment processing on the average STFT feature information to obtain a weight value corresponding to each frequency feature; and, based on the weight value corresponding to each frequency feature, perform weighted processing on the frequency features in the first STFT feature information to obtain the second STFT feature information.

10. The device according to claim 8, characterized in that The processing module is specifically used to group the frequency features in the second STFT feature information to obtain N groups of frequency features, where N is an integer greater than 1; and, perform convolution processing on the first group of frequency features in the N groups of frequency features to obtain the first output feature; and, perform convolution processing on the i-th group of frequency features and the i-1-th output feature in the N groups of frequency features to obtain the i-th output feature, i∈[2,N], and i is an integer; and, perform fusion processing on the N output features corresponding to the N groups of frequency features to obtain the third STFT feature information.

Images