A voice verification method, terminal device and storage medium

Abstract

This invention relates to a voice verification method, terminal device, and storage medium. The method includes: using a clean speech and a mixed speech belonging to the same speaker as training data; constructing a voiceprint recognition model, which includes a speech extraction module and a voiceprint extraction module. The speech extraction module separates and extracts the speech belonging to the same speaker as the clean speech in the mixed speech, and the voiceprint extraction module extracts the voiceprint features of the separated speech. Based on the final model obtained after training, voice registration and verification are implemented. This invention solves the problem of mismatch between the separated speech and the voiceprint model, and improves the performance of the speech separation model by utilizing the voiceprint module, effectively improving the recognition effect and robustness of voiceprints in real-world scenarios.

Description

Technical Field

[0001] The present invention relates to the field of voiceprint recognition, and in particular to a voice verification method, terminal equipment and storage medium. Background Art

[0002] Voiceprint recognition is a type of biometric recognition technology. By processing speech, an identity vector indicating the identity information of the person who input the speech can be generated. The similarity between the identity vectors of two speech segments can be calculated to determine whether the inputter of the two speech segments is the same user.

[0003] In real-world applications, due to the complex environment, recognition is easily affected by noise, reverberation, background voices, etc., so the robustness of voiceprint recognition is particularly important.

[0004] Currently, there are two main ways to improve robustness: one is to add noise to the data during training to improve the robustness of the voiceprint model, but this method has limited improvement and is ineffective when multiple people are speaking; the other is to pre-process the speech data first, remove the noise of the speech, extract the human voice part, that is, speech enhancement, and then perform voiceprint extraction. However, since the speech enhancement algorithm mainly improves the recognition of the speech content, the enhanced speech often cannot bring a good improvement to the voiceprint recognition effect and sometimes even brings a reverse effect. In addition, speech enhancement and voiceprint extraction operate independently without joint optimization to achieve the overall optimal. Summary of the invention

[0005] In order to solve the above problem, the present invention proposes a voice verification method, a terminal device and a storage medium.

[0006] The specific plan is as follows:

[0007] A voice verification method comprises the following steps:

[0008] S1: Collect clean speech of multiple different speakers, and construct mixed speech of each speaker based on the clean speech of each speaker, and form a training data with a clean speech and a mixed speech belonging to the same speaker, and use the clean speech forming the mixed speech as the first label of the training data, and use the voiceprint feature of the speaker corresponding to the clean speech as the second label of the training data, and form all the training data into a training set;

[0009] S2: Build a voiceprint recognition model, train the voiceprint recognition model through the training set, and use the trained voiceprint recognition model as the final model;

[0010] The voiceprint recognition model includes a speech extraction module and a voiceprint extraction module, wherein the speech extraction module includes a first speech encoding module, a voiceprint extraction submodule, a second speech encoding module, a speech separation module and a speech decoding module;

[0011] The model includes two inputs, namely, clean speech and mixed speech; after the clean speech is encoded by the first speech encoding module, it is input into the voiceprint extraction submodule to obtain the voiceprint features of the clean speech; after the mixed speech is encoded by the second speech encoding module, according to the voiceprint features of the clean speech, the speech separation module extracts the encoded separated speech belonging to the same speaker as the clean speech from the encoded mixture, and then the speech decoding module is used to obtain the separated speech belonging to the same speaker as the clean speech in the mixed speech; the obtained separated speech is passed through the voiceprint extraction module to obtain the voiceprint features of the separated speech;

[0012] S3: Register and verify speech based on the final model.

[0013] Furthermore, the method for constructing the mixed speech of each speaker based on the clean speech of each speaker includes: forming a mixed speech from the clean speech of the speaker and other speakers, forming a mixed speech from the clean speech of the speaker and noise data, and forming a mixed speech from the clean speech of the speaker and other speakers and then adding noise.

[0014] Furthermore, the training set includes both training data consisting of mixed speech constructed from clean speech of one speaker and training data consisting of mixed speech constructed from clean speech of at least two speakers.

[0015] Furthermore, the training set also includes a training data consisting of a clean speech and a mixed speech that does not contain the clean speech of the speaker corresponding to the clean speech, and the first label of the training data is blank speech.

[0016] Furthermore, the overall loss function of the voiceprint recognition model is composed of three loss functions, among which the first loss function J1 is used to calculate the difference between the separated speech output by the speech decoding module and the real clean speech belonging to the same speaker in the mixed speech; the second loss function J2 is used to calculate the difference between the voiceprint features output by the voiceprint extraction submodule in the speech extraction module and the voiceprint features of the speaker corresponding to the clean speech in the training data; the second loss function J3 is used to calculate the difference between the voiceprint features output by the voiceprint extraction module and the voiceprint features of the speaker corresponding to the clean speech in the training data.

[0017] Furthermore, the training process of the voiceprint recognition model includes the following three steps:

[0018] Step 1: Train the speech extraction module in the model through the training set, and the loss function during training is J1+βJ2;

[0019] Step 2: After fixing the model parameters in the speech extraction module, the voiceprint recognition model is trained using the training set. The loss function during training is J3. The parameter values ​​of the speech extraction module after being fixed are the parameter values ​​of the speech extraction module after training in step 1.

[0020] Step 3: Based on the voiceprint recognition model trained in step 2, retrain with the training set. The loss function during training is J1+βJ2+λJ3, where β and λ are weight coefficients.

[0021] Furthermore, step S3 is specifically as follows:

[0022] In the voice registration stage, after the input registration voice is used as the two inputs of the final model, the voiceprint features output by the final model are used as the registration voiceprint features, and the voice to be registered and the registration voiceprint features are stored;

[0023] In the voice verification stage, the stored registered voice is used as the input of the first voice encoding module in the final model, and the input verification voice is used as the input of the second voice encoding module in the final model. The similarity between the voiceprint features output by the final model and the stored registered voiceprint features is calculated, and the relationship between the similarity calculation result and the similarity threshold is used to determine whether the verification result belongs to the same speaker as the registered voice.

[0024] A voice verification terminal device comprises a processor, a memory and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the steps of the method described above in the embodiment of the present invention are implemented.

[0025] A computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the method described above in an embodiment of the present invention are implemented.

[0026] The present invention adopts the above technical solution to solve the problem of mismatch between the speech and the voiceprint model after speech separation, and at the same time improves the effect of the speech separation model with the help of the voiceprint module, effectively improving the recognition effect and robustness of the voiceprint in actual scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] Figure 1 Shown is a flow chart of the method in Embodiment 1 of the present invention.

[0028] Figure 2 Shown is a schematic structural diagram of the model in this embodiment.

[0029] Figure 3 The figure shows a schematic diagram of the voice registration and verification process in this embodiment. DETAILED DESCRIPTION

[0030] To further illustrate various embodiments, the present invention provides drawings. These drawings are part of the disclosure of the present invention, which are mainly used to illustrate the embodiments and can be used in conjunction with the relevant descriptions in the specification to explain the operating principles of the embodiments. With reference to these contents, ordinary technicians in this field should be able to understand other possible implementations and advantages of the present invention.

[0031] The present invention will now be further described with reference to the accompanying drawings and specific implementation methods.

[0032] Embodiment 1:

[0033] The embodiment of the present invention provides a voice verification method, such as Figure 1 As shown, the method comprises the following steps:

[0034] S1: Collect clean speech from multiple different speakers, and construct a mixed speech for each speaker based on the clean speech of each speaker. A clean speech and a mixed speech belonging to the same speaker form a training data. The clean speech constituting the mixed speech is used as the first label of the training data, and the voiceprint feature of the speaker corresponding to the clean speech is used as the second label of the training data. All the training data form a training set.

[0035] The voiceprint features of the speaker corresponding to the clean speech can be obtained by directly extracting the voiceprint features of the clean speech, or by other methods, which are not limited here.

[0036] The clean speech collected from each speaker may be one segment or multiple segments, wherein each segment corresponds to one training data. In a mixed speech that forms a set of training data with a clean speech, the clean speech used to construct the mixed speech may be the same speech as another clean speech in the training data, or may be a different speech. For example, if the clean speech in the training data is A, and the mixed speech is a mixed speech composed of clean speech B, the clean speech A and the clean speech B belong to the same speaker.

[0037] The mixed speech can be a mixed speech composed of the clean speech of the speaker and other speakers, a mixed speech composed of the clean speech of the speaker and noise data, a mixed speech in which the clean speech of the speaker and other speakers is mixed and then noise is added, or a mixed speech generated by other methods, which is not limited here.

[0038] Furthermore, in order to ensure that the subsequent model is useful for both multi-person and single-person speech, it is preferred to set the training set to include both training data consisting of mixed speech constructed from the clean speech of one speaker and training data consisting of mixed speech constructed from the clean speech of at least two speakers.

[0039] Furthermore, since the subsequent model is used for speech verification, the training set should contain training data when verification fails, that is, a training set consisting of a clean speech and a mixed speech that does not contain the clean speech of the speaker corresponding to the clean speech, and the first label of the training data is a blank speech. For example, in a training set consisting of the clean speech of speaker A and the mixed speech of speaker B, the mixed speech of speaker B does not contain any speech of speaker A.

[0040] S2: Build a voiceprint recognition model, train the voiceprint recognition model through the training set, and use the trained voiceprint recognition model as the final model.

[0041] like Figure 2 As shown, the voiceprint recognition model includes a speech extraction module and a voiceprint extraction module, wherein the speech extraction module includes a first speech encoding module, a voiceprint extraction submodule, a second speech encoding module, a speech separation module and a speech decoding module.

[0042] The network structures of the first speech encoding module and the second speech encoding module are the same, and in this embodiment, they are both one-dimensional convolutional networks. The speech decoding module is the inverse operation of the speech encoding module, outputs audio data, and accordingly, the network structure is also a one-dimensional convolutional network.

[0043] The speech separation module is used to extract the speech of the target speaker from the mixed speech, using commonly used neural network algorithms such as sepformer, dptnet, etc.

[0044] Both the voiceprint extraction submodule and the voiceprint extraction module use commonly used voiceprint neural network algorithms, such as xvector, dvector, etc.

[0045] The model includes two inputs, namely a first input x(t) and a second input y(t). The first input is usually the clean speech of the speaker, and the second input is usually the mixed speech.

[0046] After being encoded by the first speech encoding module, the first input speech is input into the voiceprint extraction submodule to extract the voiceprint feature v of the first input speech.

[0047] After the second input voice is encoded by the second voice encoding module, the voice separation module extracts the encoded separated voice belonging to the same speaker as the first input voice from the encoded second input voice according to the voiceprint feature v of the first input voice, and then the voice decoding module obtains the separated voice S in the second input voice that belongs to the same speaker as the first input voice. The separated voice S is the output of the voice extraction module.

[0048] The separated speech S is passed through the voiceprint extraction module to obtain the voiceprint feature e of the separated speech S. The voiceprint feature e is the final output of the voiceprint recognition model.

[0049] The overall loss function J of the voiceprint recognition model is composed of three loss functions. The specific calculation formula is J=J1+βJ2+λJ3, where β and λ are weight coefficients, indicating the weights of J2 and J3.

[0050] The first loss function J1 is used to calculate the difference between the separated speech output by the speech decoding module and the real clean speech belonging to the same speaker in the mixed speech (i.e., the first label in the training data). In this embodiment, the si-sdr loss function is used. It should be noted that when there is no speech belonging to the same speaker as the clean speech in the mixed speech, the input separated speech is a blank speech.

[0051] The second loss function J2 is used to calculate the difference between the voiceprint features output by the voiceprint extraction submodule in the speech extraction module and the voiceprint features of the speaker corresponding to the clean voice in the training data (i.e., the second label in the training data); the third loss function J3 is used to calculate the difference between the voiceprint features output by the voiceprint extraction module and the voiceprint features of the speaker corresponding to the clean voice in the training data (i.e., the second label in the training data). In this embodiment, both the second loss function J2 and the third loss function J3 use the aamsoftmax loss function.

[0052] The training process of the model in this embodiment includes three steps:

[0053] Step 1: Train the speech extraction module in the model through the training set. The loss function during training is J′=J1+βJ2. In this embodiment, the weight coefficient β=10 is set.

[0054] Step 2: After fixing the model parameters in the speech extraction module, the voiceprint recognition model is trained using the training set, and the loss function during training is J″=J3. The parameter values ​​of the speech extraction module after being fixed are the parameter values ​​of the speech separation module after training in step 1.

[0055] Step 3: Based on the voiceprint recognition model trained in step 2, retraining is performed using the training set. The loss function during training is J″′=J1+βJ2+λJ3. In this embodiment, the weight coefficient λ is set to 10.

[0056] S3: Register and verify speech based on the final model.

[0057] like Figure 3 As shown, the specific process of registration and verification in this embodiment is as follows:

[0058] In the speech registration stage, the input registration speech x(t) is used as the two inputs of the final model (that is, the first input and the second input are both x(t)), and the voiceprint features output by the final model (the voiceprint features e of the separated speech S) are used as the registration voiceprint features, and the registration speech and registration voiceprint features are stored.

[0059] In the speech verification stage, the stored registered speech x(t) is used as the input of the first speech encoding module in the final model, and the input verification speech y(t) is used as the input of the second speech encoding module in the final model. The similarity between the voiceprint features output by the final model (the voiceprint features e of the separated speech S) and the stored registered voiceprint features is calculated. The relationship between the similarity calculation result and the similarity threshold is used to determine whether the verification result belongs to the same speaker as the registered speech.

[0060] The similarity calculation algorithm can adopt an existing calculation algorithm, such as a cosine or PLDA algorithm.

[0061] The size of the similarity threshold can be set in advance by a person skilled in the art according to the requirements, and a suitable threshold point can be selected after calculating the ROC curve of the system using a large amount of test data in advance. When the similarity calculation result is greater than the similarity threshold, the verification result is determined to be that the verification voice y(t) and the registration voice x(t) both belong to the same speaker.

[0062] The embodiment of the present invention utilizes the joint training method of voiceprint and speech separation to solve the problem of mismatch between the speech after speech separation and the voiceprint model. At the same time, with the help of the voiceprint module, the effect of the speech separation model is improved, and the recognition effect and robustness of the voiceprint in actual scenarios are effectively improved.

[0063] Embodiment 2:

[0064] The present invention also provides a voice verification terminal device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps in the above method embodiment of the first embodiment of the present invention when executing the computer program.

[0065] Further, as an executable solution, the voice verification terminal device can be a computing device such as a desktop computer, a notebook, a PDA, and a cloud server. The voice verification terminal device may include, but is not limited to, a processor and a memory. Those skilled in the art can understand that the composition structure of the above-mentioned voice verification terminal device is only an example of a voice verification terminal device, and does not constitute a limitation on the voice verification terminal device. It may include more or less components than the above, or a combination of certain components, or different components. For example, the voice verification terminal device may also include input and output devices, network access devices, buses, etc., and the embodiments of the present invention do not limit this.

[0066] Further, as an executable solution, the processor may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or any conventional processor, etc. The processor is the control center of the voice verification terminal device, and uses various interfaces and lines to connect various parts of the entire voice verification terminal device.

[0067] The memory can be used to store the computer program and / or module, and the processor realizes various functions of the voice verification terminal device by running or executing the computer program and / or module stored in the memory, and calling the data stored in the memory. The memory can mainly include a program storage area and a data storage area, wherein the program storage area can store an operating system and an application required for at least one function; the data storage area can store data created according to the use of the mobile phone, etc. In addition, the memory can include a high-speed random access memory, and can also include a non-volatile memory, such as a hard disk, a memory, a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), at least one disk storage device, a flash memory device, or other volatile solid-state storage devices.

[0068] The present invention also provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the above method in the embodiment of the present invention are implemented.

[0069] If the module / unit integrated in the voice verification terminal device is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present invention implements all or part of the process in the above-mentioned embodiment method, and can also be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium, and the computer program can implement the steps of the above-mentioned various method embodiments when executed by the processor. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording medium, U disk, mobile hard disk, disk, optical disk, computer memory, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory) and software distribution medium, etc.

[0070] Although the present invention has been specifically shown and described in conjunction with the preferred embodiments, it should be understood by those skilled in the art that various changes may be made to the present invention in form and details without departing from the spirit and scope of the present invention as defined by the appended claims, all of which are within the scope of protection of the present invention.

Claims

1. A voice verification method, characterized in that: The following steps are involved: S1: Collect clean speech of multiple different speakers, and construct mixed speech of each speaker based on the clean speech of each speaker, and form a training data with a clean speech and a mixed speech belonging to the same speaker, and use the clean speech forming the mixed speech as the first label of the training data, and use the voiceprint feature of the speaker corresponding to the clean speech as the second label of the training data, and form all the training data into a training set; S2: Build a voiceprint recognition model, train the voiceprint recognition model through the training set, and use the trained voiceprint recognition model as the final model; The voiceprint recognition model includes a speech extraction module and a voiceprint extraction module, wherein the speech extraction module includes a first speech encoding module, a voiceprint extraction submodule, a second speech encoding module, a speech separation module and a speech decoding module; The model includes two inputs, namely, clean speech and mixed speech; after the clean speech is encoded by the first speech encoding module, it is input into the voiceprint extraction submodule to obtain the voiceprint features of the clean speech; after the mixed speech is encoded by the second speech encoding module, according to the voiceprint features of the clean speech, the speech separation module extracts the encoded separated speech belonging to the same speaker as the clean speech from the encoded mixture, and then the speech decoding module is used to obtain the separated speech belonging to the same speaker as the clean speech in the mixed speech; the obtained separated speech is passed through the voiceprint extraction module to obtain the voiceprint features of the separated speech; The overall loss function of the voiceprint recognition model consists of three loss functions, among which the first loss function J1 is used to calculate the difference between the separated speech output by the speech decoding module and the real clean speech belonging to the same speaker in the mixed speech; the second loss function J2 is used to calculate the difference between the voiceprint features output by the voiceprint extraction submodule in the speech extraction module and the voiceprint features of the speaker corresponding to the clean speech in the training data; the third loss function J3 is used to calculate the difference between the voiceprint features output by the voiceprint extraction module and the voiceprint features of the speaker corresponding to the clean speech in the training data; The training process of the voiceprint recognition model includes the following three steps: Step 1: Train the speech extraction module in the model through the training set, and the loss function during training is J1+βJ2; Step 2: After fixing the model parameters in the speech extraction module, the voiceprint recognition model is trained using the training set. The loss function during training is J3. The parameter values ​​of the speech extraction module after being fixed are the parameter values ​​of the speech extraction module after training in step 1. Step 3: Based on the voiceprint recognition model trained in step 2, retrain with the training set. The loss function during training is J1+βJ2+λJ3, where β and λ are weight coefficients. S3: Register and verify the voice based on the final model, specifically: In the voice registration stage, after the input registration voice is used as the two inputs of the final model, the voiceprint features output by the final model are used as the registration voiceprint features, and the voice to be registered and the registration voiceprint features are stored; In the voice verification stage, the stored registered voice is used as the input of the first voice encoding module in the final model, and the input verification voice is used as the input of the second voice encoding module in the final model. The similarity between the voiceprint features output by the final model and the stored registered voiceprint features is calculated, and the relationship between the similarity calculation result and the similarity threshold is used to determine whether the verification result belongs to the same speaker as the registered voice.

2. The voice verification method according to claim 1, characterized in that: The method for constructing the mixed speech of each speaker based on the clean speech of each speaker includes: forming a mixed speech from the clean speech of the speaker and other speakers, forming a mixed speech from the clean speech of the speaker and noise data, and forming a mixed speech by mixing the clean speech of the speaker and other speakers and then adding noise.

3. The voice verification method according to claim 1, characterized in that: The training set includes both training data consisting of mixed speech constructed from the clean speech of one speaker and training data consisting of mixed speech constructed from the clean speech of at least two speakers.

4. The voice verification method according to claim 1, characterized in that: The training set also includes a training data consisting of a clean speech and a mixed speech that does not contain the clean speech of the speaker corresponding to the clean speech, and the first label of the training data is a blank speech.

5. A voice verification terminal device, characterized in that: The method comprises a processor, a memory and a computer program stored in the memory and running on the processor, wherein the processor implements the steps of the method according to any one of claims 1 to 4 when executing the computer program.

6. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 4 are implemented.

Images