A self-supervised voiceprint recognition method and system

Abstract

The present invention provides a self-supervised voiceprint recognition method, including: obtaining speech data, and processing the speech data into 4-second speech segments; dividing the speech segments into three sub-segments, the first sub-segment is 0-2s speech, the second sub-segment is 2-4s speech, and the third sub-segment is 1-3s speech, and the third sub-segment set is enhanced to obtain an enhanced third sub-segment set; the first sub-segment set and the second sub-segment set are input into the teacher network training, and the enhanced third sub-segment set is input into the student network training; after each iteration, the weight parameters of the neurons in the same layer are similarly calculated to complete the training. The method provided by the present invention optimizes the network parameters through the differential loss of the overlapping audio and the original audio; and proposes a solution to the possible homogenization problem of the weight parameters of the neural network, which ensures the unity of the overall field of view and the overlapping field of view voiceprint features of different parts of the same speech, and improves the accuracy of voiceprint recognition.

Description

Technical Field

[0001] The present invention relates to the field of voiceprint recognition, and in particular to a self-supervised voiceprint recognition method and system. Background Art

[0002] Voiceprint recognition is a type of biometric technology that processes speech to generate an identity vector that indicates the identity of the person inputting the speech. The similarity between the identity vectors of two speech segments can be calculated to determine whether the inputter of the two speech segments is the same user.

[0003] Most existing voiceprint recognition technologies are supervised voiceprint recognition training methods, and the training data needs to be labeled. Of course, there are also some semi-supervised voiceprint recognition training methods, which also require sufficient labels.

[0004] However, labeling data requires significant manpower, material resources, and financial resources, leading to a shortage of training data. While a significant amount of voice data is available, it lacks labels or may contain errors. To maximize the use of this data, self-supervised voiceprint recognition training methods have been proposed. However, existing self-supervised voiceprint training methods still lag behind supervised voiceprint training methods. Summary of the Invention

[0005] The main purpose of this invention is to overcome the above-mentioned shortcomings of the existing technology by proposing a self-supervised voiceprint recognition method that optimizes network parameter training by using differential loss between overlapping and original audio. Furthermore, a solution is proposed to address the potential homogeneity of neural network weight parameters, ensuring uniformity of voiceprint features across the entire field of view and the overlapping field of view for different parts of the same speech, thereby improving the accuracy of voiceprint recognition.

[0006] The present invention adopts the following technical solutions:

[0007] A self-supervised voiceprint recognition method, comprising:

[0008] Acquire voice data and process it into 4-second segments.

[0009] The speech segment is divided into three sub-segments, where the first sub-segment is 0-2s speech, the second sub-segment is 2-4s speech, and the third sub-segment is 1-3s speech, to obtain a first sub-segment set, a second sub-segment set, and a third sub-segment set, and the third sub-segment set is enhanced to obtain an enhanced third sub-segment set;

[0010] The first and second sub-sets of segments are fed into the teacher network for training, and the enhanced third sub-set of segments is fed into the student network for training. The teacher network is trained by first randomly masking the embedding layer of the teacher network and then averaging all the embeddings output by the teacher. The student network is trained by randomly masking the embedding layer of the student network and then calculating the loss using the masked embedding layer and the embedding layer of the original student network.

[0011] After each iteration, the similarity calculation is performed on the weight parameters of neurons in the same layer to complete the training.

[0012] Specifically, the enhancing the third sub-segment set to obtain the enhanced third sub-segment set is:

[0013] The third sub-set of clips is enhanced using the Musan and RIRS_NOISES datasets, as well as temporal and spectral enhancements in the frequency domain.

[0014] Specifically, the teacher network and the student network are any neural network including but not limited to RepVGG, ResNet, ECAPA-TDNN, and DPN.

[0015] Specifically, the teacher network and the student network use the same network parameters, and the teacher network adds a dropout layer.

[0016] Specifically, after each iteration, similarity calculation is performed on the weight parameters of neurons in the same layer, specifically:

[0017] If similar weight parameters appear, the transformation w'=w+a*mean is performed, where w is the current weight parameter, w' is the transformed weight parameter, a is the transformation coefficient, and mean is the mean matrix for calculating the weight.

[0018] Another embodiment of the present invention provides a self-supervised voiceprint recognition system, comprising:

[0019] Voice acquisition unit: acquires voice data and processes it into 4-second voice segments;

[0020] The speech segment processing unit divides the speech segment into three sub-segments: the first sub-segment is 0-2s speech, the second sub-segment is 2-4s speech, and the third sub-segment is 1-3s speech. The first sub-segment set, the second sub-segment set, and the third sub-segment set are obtained, and the third sub-segment set is enhanced to obtain an enhanced third sub-segment set.

[0021] First training unit: The first and second sub-sets are fed into the teacher network for training, and the enhanced third sub-set is fed into the student network for training. The teacher network is trained by first randomly masking the embedding layer of the teacher network and then averaging all the embeddings output by the teacher. The student network is trained by randomly masking the embedding layer of the student network and then calculating the loss using the masked embedding layer and the embedding layer of the original student network.

[0022] Second training unit: After each iteration, similarity calculation is performed on the weight parameters of neurons in the same layer to complete the training.

[0023] Another aspect of an embodiment of the present invention provides an electronic device, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the above-mentioned steps of the self-supervised voiceprint recognition method when executing the computer program.

[0024] Another aspect of the embodiments of the present invention provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the above-mentioned self-supervised voiceprint recognition method are implemented.

[0025] From the above description of the present invention, it can be seen that compared with the prior art, the present invention has the following beneficial effects:

[0026] The present invention provides a self-supervised voiceprint recognition method, comprising: acquiring speech data and processing the speech data into 4-second speech segments; dividing the speech segments into three sub-segments, wherein the first sub-segment is speech of 0-2 seconds, the second sub-segment is speech of 2-4 seconds, and the third sub-segment is speech of 1-3 seconds, to obtain a first sub-segment set, a second sub-segment set, and a third sub-segment set; and enhancing the third sub-segment set to obtain an enhanced third sub-segment set; inputting the first sub-segment set and the second sub-segment set into a teacher network for training, and inputting the enhanced third sub-segment set into a student network for training; the teacher network training comprises: first randomly masking an embedding layer of the teacher network, and then averaging all output embeddings of the teacher; the student network training comprises: randomly masking an embedding layer of the student network, and then performing loss calculation between the masked embedding layer and the embedding layer in the original student network; and after each iteration, performing similarity calculation on weight parameters of neurons in the same layer to complete the training. The method provided by this invention optimizes network parameter training by using differential losses between overlapping and original audio. Furthermore, it proposes a solution to the potential homogeneity problem of neural network weight parameters, ensuring uniformity of voiceprint features across the entire field of view and overlapping fields of view for different parts of the same speech, thereby improving voiceprint recognition accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] Figure 1 A flow chart of a self-supervised voiceprint recognition method provided by an embodiment of the present invention;

[0028] Figure 2 A training structure diagram of a self-supervised voiceprint recognition method provided by an embodiment of the present invention;

[0029] Figure 3 This is a diagram of the architecture of a cross-user behavior recognition transfer learning system provided by an embodiment of the present invention;

[0030] Figure 4 A schematic diagram of an electronic device provided by an embodiment of the present invention;

[0031] Figure 5 A schematic diagram of an embodiment of a computer-readable storage medium provided in an embodiment of the present invention.

[0032] The present invention is further described in detail below with reference to the accompanying drawings and specific embodiments. DETAILED DESCRIPTION

[0033] This paper provides a self-supervised voiceprint recognition method that optimizes network parameter training by using differential loss between overlapping and original audio. Furthermore, it proposes a solution to the potential homogeneity of neural network weight parameters, ensuring uniformity of voiceprint features across the entire field of view and overlapping fields of view for different parts of the same speech, thereby improving voiceprint recognition accuracy.

[0034] like Figure 1 , is a flow chart of a self-supervised voiceprint recognition method provided by an embodiment of the present invention, which specifically includes the following steps:

[0035] S101: Acquire voice data and process the voice data into 4-second voice segments;

[0036] In actual operation, the speech segments can also be processed into 6s or 8s segments, but after many experiments, it was found that the accuracy rate of speech segments divided into 4s segments is the highest.

[0037] S102: Divide the speech segment into three sub-segments, where the first sub-segment is speech of 0-2 seconds, the second sub-segment is speech of 2-4 seconds, and the third sub-segment is speech of 1-3 seconds, obtaining a first sub-segment set, a second sub-segment set, and a third sub-segment set, and enhancing the third sub-segment set to obtain an enhanced third sub-segment set;

[0038] For 1-3 seconds of speech, we used the Musan and RIRS_NOISES datasets for enhancement, as well as time and spectral enhancement in the frequency domain. The Musan noise dataset consists of speech (background vocals), noise, and music. The RIRS_NOISES dataset contains impulse responses (RIRs) and noise. Spectral enhancement involves extracting the speech spectrogram and then randomly discarding some data in the time domain or frequencies in the frequency domain.

[0039] The input of the teacher network contains all the audio input of the student network. Although it is scattered in two segments, it ensures that the features extracted by the teacher network include the features extracted by the student network.

[0040] S103: Input the first and second sub-sets of segments into the teacher network for training, and input the enhanced third sub-set of segments into the student network for training. The teacher network is trained by first randomly masking the embedding layer of the teacher network and then averaging all the embeddings output by the teacher. The student network is trained by randomly masking the embedding layer of the student network and then calculating the loss using the masked embedding layer and the embedding layer in the original student network.

[0041] The first and second sub-sets are fed into the teacher network for training, and the enhanced third sub-set is fed into the student network for training. This design structure uses local features to represent global features.

[0042] In the teacher network, two complete segments of the same speech are presented, and their outputs are averaged. In the student network, the speech consists of overlapping segments of audio between 0-2 seconds and 2-4 seconds, which are augmented to enhance model stability and prevent network crashes.

[0043] In this embodiment, in addition to calculating the loss between the teacher and student networks, the loss between the student networks is also calculated to reduce the impact of noise on feature extraction. The main method is to first randomly mask the embeddings of the student network, and then use the masked embeddings and the original embeddings to calculate the loss: -p2logp1.

[0044] The teacher network and the student network use the same network parameters, but the teacher network adds some dropout layers to prevent the network from not learning the content;

[0045] The embedding layer of the teacher network is first randomly masked, and then all the output embeddings of the teacher are averaged.

[0046] S104: After each iteration, similarity calculation is performed on the weight parameters of neurons in the same layer to complete the training.

[0047] Self-supervised learning reduces intra-class distances (primarily within the same channel), but does not optimize inter-class distances. Consequently, neuron parameter homogeneity may occur. To avoid this, we propose optimizing neuron weight parameters.

[0048] During self-supervised neural network training, network parameters may become homogenized, meaning that different neurons have similar weight parameters. After each iteration, the weight parameters of neurons in the same layer are similar. If similar weight parameters are found, they are transformed using the formula w' = w + a * mean, where w is the current weight, w' is the transformed weight, a is the transformation coefficient, and mean is the mean matrix of the previously calculated weights. This homogenization of weights prevents the network from failing to learn.

[0049] like Figure 2 A training structure diagram of a self-supervised voiceprint recognition method provided by an embodiment of the present invention.

[0050] like Figure 3 The following is an architecture diagram of a cross-user behavior recognition transfer learning system provided by an embodiment of the present invention, specifically:

[0051] The speech acquisition unit 301 acquires speech data and processes the speech data into 4-second speech segments.

[0052] In actual operation, the speech segments can also be processed into 6s or 8s segments, but after many experiments, it was found that the accuracy rate of speech segments divided into 4s segments was the highest.

[0053] The speech segment processing unit 302 divides the speech segment into three sub-segments: the first sub-segment is speech of 0-2 seconds, the second sub-segment is speech of 2-4 seconds, and the third sub-segment is speech of 1-3 seconds. The first sub-segment set, the second sub-segment set, and the third sub-segment set are obtained, and the third sub-segment set is enhanced to obtain an enhanced third sub-segment set.

[0054] For 1-3 seconds of speech, we used the Musan and RIRS_NOISES datasets for enhancement, as well as time and spectral enhancement in the frequency domain. The Musan noise dataset consists of speech (background vocals), noise, and music. The RIRS_NOISES dataset contains impulse responses (RIRs) and noise. Spectral enhancement involves extracting the speech spectrogram and then randomly discarding some data in the time domain or frequencies in the frequency domain.

[0055] The input of the teacher network contains all the audio input of the student network. Although it is scattered in two segments, it ensures that the features extracted by the teacher network include the features extracted by the student network.

[0056] First training unit 303: Input the first and second sub-clips into the teacher network for training, and input the enhanced third sub-clips into the student network for training. The teacher network is trained by first randomly masking the embedding layer of the teacher network and then averaging all the embeddings output by the teacher. The student network is trained by randomly masking the embedding layer of the student network and then calculating the loss between the masked embedding layer and the embedding layer in the original student network.

[0057] The first and second sub-sets are fed into the teacher network for training, and the enhanced third sub-set is fed into the student network for training. This design structure uses local features to represent global features.

[0058] In the teacher network, two complete segments of the same speech are presented, and their outputs are averaged. In the student network, the speech consists of overlapping segments of audio between 0-2 seconds and 2-4 seconds, which are augmented to enhance model stability and prevent network crashes.

[0059] In this embodiment, in addition to calculating the loss between the teacher and student networks, the loss between the student networks is also calculated to reduce the impact of noise on feature extraction. The main method is to first randomly mask the embeddings of the student network, and then use the masked embeddings and the original embeddings to calculate the loss: -p2logp1.

[0060] The teacher network and the student network use the same network parameters, but the teacher network adds some dropout layers to prevent the network from not learning the content;

[0061] The embedding layer of the teacher network is first randomly masked, and then all the output embeddings of the teacher are averaged.

[0062] Second training unit 304: After each iteration, similarity calculation is performed on the weight parameters of neurons in the same layer to complete the training.

[0063] Self-supervised learning reduces intra-class distances (primarily within the same channel), but does not optimize inter-class distances. Consequently, neuron parameter homogeneity may occur. To avoid this, we propose optimizing neuron weight parameters.

[0064] During self-supervised neural network training, network parameters may become homogenized, meaning that different neurons have similar weight parameters. After each iteration, the weight parameters of neurons in the same layer are similar. If similar weight parameters are found, they are transformed using the formula w' = w + a * mean, where w is the current weight, w' is the transformed weight, a is the transformation coefficient, and mean is the mean matrix of the previously calculated weights. This homogenization of weights prevents the network from failing to learn.

[0065] Figure 4 As shown, an embodiment of the present invention provides an electronic device 400, including a memory 410, a processor 420, and a computer program 411 stored in the memory 410 and executable on the processor 420. When the processor 420 executes the computer program 411, a self-supervised voiceprint recognition method provided by an embodiment of the present invention is implemented.

[0066] Since the electronic device introduced in this embodiment is the device used to implement the embodiments of the present invention, based on the method introduced in the embodiments of the present invention, technical personnel in this field can understand the specific implementation methods of the electronic device of this embodiment and its various variations. Therefore, how the electronic device implements the method in the embodiments of the present invention will not be introduced in detail here. As long as the equipment used by technical personnel in this field to implement the method in the embodiments of the present invention falls within the scope of protection of the present invention.

[0067] See also Figure 5 , Figure 5 A schematic diagram of an embodiment of a computer-readable storage medium provided in an embodiment of the present invention.

[0068] like Figure 5 As shown, this embodiment provides a computer-readable storage medium 500, on which a computer program 511 is stored. When the computer program 511 is executed by a processor, a self-supervised voiceprint recognition method provided by an embodiment of the present invention is implemented;

[0069] It should be noted that, in the above embodiments, the description of each embodiment has its own focus. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0070] It will be understood by those skilled in the art that embodiments of the present invention may be provided as methods, systems, or computer program products. Thus, the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware. Furthermore, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0071] The present invention provides a self-supervised voiceprint recognition method, comprising: acquiring speech data and processing the speech data into 4-second speech segments; dividing the speech segments into three sub-segments, wherein the first sub-segment is speech of 0-2 seconds, the second sub-segment is speech of 2-4 seconds, and the third sub-segment is speech of 1-3 seconds, to obtain a first sub-segment set, a second sub-segment set, and a third sub-segment set; and enhancing the third sub-segment set to obtain an enhanced third sub-segment set; inputting the first sub-segment set and the second sub-segment set into a teacher network for training, and inputting the enhanced third sub-segment set into a student network for training; the teacher network training comprises: first randomly masking an embedding layer of the teacher network, and then averaging all output embeddings of the teacher; the student network training comprises: randomly masking an embedding layer of the student network, and then performing loss calculation between the masked embedding layer and the embedding layer in the original student network; and after each iteration, performing similarity calculation on weight parameters of neurons in the same layer to complete the training. The method provided by this invention optimizes network parameter training by using differential losses between overlapping and original audio. Furthermore, it proposes a solution to the potential homogeneity problem of neural network weight parameters, ensuring uniformity of voiceprint features across the entire field of view and overlapping fields of view for different parts of the same speech, thereby improving voiceprint recognition accuracy.

[0072] It should be noted that, in this document, relational terms such as "first" and "second" are used solely to distinguish one entity or operation from another, and do not necessarily require or imply any actual relationship or order between these entities or operations. Furthermore, the terms "comprise," "include," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, method, article, or device comprising a set of elements includes not only those elements but also other elements not explicitly listed, or elements inherent to such process, method, article, or device. Without further limitation, an element defined by the phrase "comprising a..." does not preclude the presence of other identical elements in the process, method, article, or device comprising the recited element. The foregoing description is intended only to provide specific embodiments of the present application, intended to enable those skilled in the art to understand or implement the present application. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application is not limited to the embodiments shown herein, but is intended to be accorded the broadest scope consistent with the principles and novel features of the present application.

[0073] The above is only a specific implementation of the present invention, but the design concept of the present invention is not limited to this. Any non-substantial changes to the present invention using this concept shall be deemed as an infringement of the protection scope of the present invention.

Claims

1. A self-supervised voiceprint recognition method, characterized in that: include: Acquire voice data and process the voice data into 4-second voice segments; The speech segment is divided into three sub-segments, the first sub-segment is 0-2s speech, the second sub-segment is 2-4s speech, and the third sub-segment is 1-3s speech, to obtain a first sub-segment set, a second sub-segment set and a third sub-segment set, and the third sub-segment set is enhanced to obtain an enhanced third sub-segment set; The first and second sub-sets are input into the teacher network training, and the enhanced third sub-set is input into the student network training. The teacher network training is as follows: the embedding layer of the teacher network is first randomly masked, and then all the output embeddings of the teacher are averaged. The student network training is as follows: the embedding layer of the student network is randomly masked, and then the masked embedding layer is used to calculate the loss with the embedding layer in the original student network. In addition to calculating the loss between the teacher and student networks, the loss between the student networks is also calculated. After each iteration, the weight parameters of neurons in the same layer are similarly calculated to complete the training.

2. A self-supervised voiceprint recognition method according to claim 1, characterized in that: The enhancing the third sub-segment set to obtain the enhanced third sub-segment set is specifically: The third sub-clip set is enhanced using the Musan and RIRS_NOISES datasets, as well as temporal and spectral enhancements in the frequency domain.

3. A self-supervised voiceprint recognition method according to claim 1, characterized in that: The teacher network and the student network are any neural network including but not limited to RepVGG, ResNet, Ecapa-TDNN, and DPN.

4. A self-supervised voiceprint recognition method according to claim 3, characterized in that: The teacher network and the student network use the same network parameters, and a dropout layer is added to the teacher network.

5. A self-supervised voiceprint recognition method according to claim 4, characterized in that: After each iteration, similarity calculation is performed on the weight parameters of neurons in the same layer, specifically: If similar weight parameters appear, a transformation is performed: w'=w+a*mean, where w is the current weight parameter, w' is the transformed weight parameter, a is the transformation coefficient, and mean is the mean matrix for calculating the weight.

6. A self-supervised voiceprint recognition system, characterized in that: include: Voice acquisition unit: acquires voice data and processes the voice data into 4-second voice segments; The speech segment processing unit: divides the speech segment into three sub-segments, the first sub-segment is 0-2s speech, the second sub-segment is 2-4s speech, and the third sub-segment is 1-3s speech, to obtain a first sub-segment set, a second sub-segment set and a third sub-segment set, and enhances the third sub-segment set to obtain an enhanced third sub-segment set; First training unit: input the first sub-set of segments and the second sub-set of segments into the teacher network training, and input the enhanced third sub-set of segments into the student network training; the teacher network training is as follows: the embedding layer of the teacher network is first randomly masked, and then all the output embeddings of the teacher are averaged; the student network training is as follows: the embedding layer of the student network is randomly masked, and then the loss is calculated between the masked embedding layer and the embedding layer in the original student network. In addition to calculating the loss between the teacher and student networks, the loss between the student networks is also calculated; Second training unit: After each iteration, similarity calculation is performed on the weight parameters of neurons in the same layer to complete the training.

7. An electronic device, characterized in that: include: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the method steps described in any one of claims 1 to 5 when executing the computer program.

8. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method steps described in any one of claims 1 to 5 are implemented.

Images