A VIT Voiceprint Recognition Method, Device and Storage Medium

Abstract

The present invention provides a VIT voiceprint recognition method, device, and storage medium. The method comprises: performing noise processing on input speech and extracting speech features from the noise-processed speech; inputting the extracted speech features into a CNN structure of a neural network, and using the CNN structure to extract local speech features under different receptive fields from the extracted speech features; inputting the extracted local speech features under different receptive fields into a VIT structure, and using the VIT structure to process the extracted local speech features under different receptive fields and output a speech feature sequence; performing linear transformation and pooling processing on the output speech feature sequence, and then extracting the pooled voiceprint features. Using the above technical solution for voiceprint recognition can obtain both local and global feature information, and can effectively solve the problem of cross-channel voiceprints.

Description

Technical Field

[0001] The present invention relates to the technical field of voiceprint recognition, and in particular to a method, device and storage medium for voiceprint recognition using a visual transformer (VIT) based on multi-receptive field speech feature preprocessing. Background Art

[0002] Voiceprint recognition is a biometric technology that processes speech to generate an identity vector that indicates the identity of the person inputting the speech. By calculating the similarity between the identity vectors of two speech segments, it can be determined whether the inputter of the two speech segments is the same user.

[0003] Existing voiceprint recognition technologies are primarily based on convolutional neural networks. In convolutional neural networks (CNNs), convolution operations excel at extracting local features, but have limitations in capturing global feature representations. In the Vision Transformer (VIT), cascaded self-attention modules can capture long-range feature dependencies but tend to overlook local feature details. Summary of the Invention

[0004] The embodiments of the present invention provide a VIT voiceprint recognition method, device, and storage medium, which can possess local feature and global feature information and better solve the problem of cross-channel voiceprints.

[0005] To achieve the above objectives, a VIT voiceprint recognition method is provided, comprising:

[0006] Noise the input speech and extract speech features from the noise-processed speech;

[0007] Inputting the extracted speech features into a CNN structure of a neural network, and using the CNN structure to extract local speech features under different receptive fields from the extracted speech features;

[0008] Inputting the extracted speech local features under different receptive fields into a VIT structure, processing the extracted speech local features under different receptive fields using the VIT structure, and outputting a speech feature sequence;

[0009] The output speech feature sequence is linearly transformed and pooled, and then the voiceprint features after pooling are extracted.

[0010] Preferably, in the VIT voiceprint recognition method, the step of performing noise processing on the input voice and extracting voice features from the noise-processed voice comprises:

[0011] Use the MUSAN dataset and / or the RIR dataset for noise addition;

[0012] Extract one of the fbank feature, mel spectrum feature and mfcc feature from the noise-processed speech;

[0013] Perform random mask enhancement on the extracted features.

[0014] Preferably, the VIT voiceprint recognition method uses one or more of the following methods to perform random mask enhancement on the extracted features:

[0015] Fixed-dimensional mask augmentation;

[0016] Pixel-level mask enhancement.

[0017] Preferably, in the VIT voiceprint recognition method, the step of extracting local speech features under different receptive fields from the extracted speech features using the CNN structure includes:

[0018] Multiple branch CNN structures with different expansion rates are used to extract local speech features under different receptive fields.

[0019] Preferably, the VIT voiceprint recognition method, wherein the branch CNN structure includes four layers, wherein the first layer and the fourth layer respectively include three parts: two-dimensional convolution Conv2D, activation layer function ReLU and normalization layer; the second layer and the third layer respectively include two parts: two-dimensional convolution Conv2D and residual module ResBlock.

[0020] Preferably, in the VIT voiceprint recognition method, the CNN structure includes four branch CNN structures, and the expansion rates of the four branch CNN structures are 1, 2, 4 and 8 respectively.

[0021] Preferably, in the VIT voiceprint recognition method, the step of using the VIT structure to process the extracted speech local features under different receptive fields and outputting a speech feature sequence includes:

[0022] The extracted local speech features under different receptive fields are divided into blocks and serialized;

[0023] Splicing the features after the block and serialization processing;

[0024] Add CLS token;

[0025] Add position coding;

[0026] Processed through the dropout layer;

[0027] Processed through standard transformer encoding layers;

[0028] Processed by layerNorm layer;

[0029] Outputs the speech feature sequence processed by the layerNorm layer.

[0030] Preferably, in the VIT voiceprint recognition method, the step of performing linear transformation and pooling processing on the output voice feature sequence includes:

[0031] Compressing the output speech feature sequence;

[0032] Pooling is performed on the compressed sequence.

[0033] On the other hand, a VIT voiceprint recognition device is provided, comprising a memory and a processor, wherein the memory stores at least one program, and the at least one program is executed by the processor to implement any of the methods described above.

[0034] On the other hand, a computer-readable storage medium is provided, wherein the storage medium stores at least one program, and the at least one program is executed by a processor to implement any of the methods described above.

[0035] The above technical solution has the following technical effects:

[0036] The technical solution of the embodiment of the present invention provides a VIT voiceprint recognition method based on multi-receptive field speech feature preprocessing. By combining multi-receptive field speech feature preprocessing with the VIT structure, the deep learning network can perform global feature extraction after multi-receptive field speech feature preprocessing, and at the same time have local feature and global feature information, providing a better solution to solving the voiceprint cross-channel problem. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] Figure 1 This is a schematic diagram of the CNN structure used in the VIT voiceprint recognition method according to one embodiment of the present invention;

[0038] Figure 2 This is a schematic diagram of the structure of the residual module in the CNN structure in the VIT voiceprint recognition method according to one embodiment of the present invention;

[0039] Figure 3 1. A flow chart of the VIT structure used in the VIT voiceprint recognition method according to an embodiment of the present invention;

[0040] Figure 4 This is a schematic diagram of the structure of the standard converter encoding layer in the VIT structure in the VIT voiceprint recognition method according to one embodiment of the present invention;

[0041] Figure 5This is a flowchart of processing a speech feature sequence output by a VIT in a VIT voiceprint recognition method according to an embodiment of the present invention;

[0042] Figure 6 This is a schematic diagram of the overall process of the VIT voiceprint recognition method according to one embodiment of the present invention;

[0043] Figure 7 Schematic diagram of the structure of a VIT voiceprint recognition device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0044] To further illustrate various embodiments, the present invention is provided with accompanying drawings. These drawings form part of the present disclosure and are primarily used to illustrate the embodiments and, in conjunction with the relevant description in the specification, to explain the operating principles of the embodiments. By referring to these drawings, one of ordinary skill in the art will understand other possible embodiments and the advantages of the present invention. The components in the figures are not drawn to scale, and similar reference numerals are generally used to represent similar components.

[0045] During the implementation of the present invention, the inventors discovered that, despite advantages in extracting local features, CNNs lack the ability to capture global representations. Existing voiceprint recognition technologies are mostly based on CNN networks, and some transformer-based solutions often replace portions of CNN architectures or operate in parallel with them. Voiceprint features are time-invariant, meaning the extracted features remain consistent regardless of the speech. CNNs can effectively address voiceprint correlation issues within the same channel. However, the spectral structure of speech from different channels exhibits certain deviations, a challenge for CNNs. Transformers, on the other hand, can capture long-range feature dependencies, avoiding the problem of being dominated by local features. However, transformers cannot effectively address the time-invariant nature of speech and are slow to converge. Based on this discovery, the present applicant proposes preprocessing speech features using multiple receptive fields, then extracting global features using VIT, and using this to obtain voiceprint features for voiceprint recognition. For cross-channel issues, VIT prioritizes global features, thus avoiding the problem of being dominated by local features.

[0046] The present invention will now be further described with reference to the accompanying drawings and specific embodiments.

[0047] The VIT voiceprint recognition method according to an embodiment of the present invention includes the following steps:

[0048] Step 1: Add noise to the input speech and extract the corresponding speech features.

[0049] Generally speaking, the speech is denoised using the MUSAN dataset and / or the RIR dataset, and one of the fbank features, mel spectrum features, mfcc features, etc. is extracted from the noisy speech; then the extracted speech features are randomly masked and enhanced.

[0050] An example of mask enhancement: Based on the mask rate, random mask values are generated and the corresponding positions are set to zero. There are two methods: 1. Fixed-dimensional mask enhancement. For example, if there are 200 values in the time dimension and the mask rate is 10%, the generated mask values are 20 from 0 to 199, and these values in the time dimension are set to zero. 2. Pixel-level mask enhancement. Considering both time and frequency, for example, if the time dimension is 200 and the frequency dimension is 20, the generated mask values are 400 from 0 to 4000, and the corresponding values are set to zero.

[0051] In step 2, the speech features extracted in the previous step are input into the CNN structure of the neural network. This CNN structure serves as a speech feature preprocessing module. Its function is to extract local speech features under different receptive fields and output speech features. Specifically, it outputs local speech features extracted under four different receptive fields. Here, the four branches are parallel structures. The four branches use the same network structure, differing only in the dilation coefficient (or dilation rate).

[0052] Specifically, Figure 1 The diagram shows the CNN structure used in the VIT voiceprint recognition method of an embodiment of the present invention, namely the CNN pre-processing network structure. This structure extracts local features of different receptive fields respectively through the dilation rate in the convolution, that is, it is responsible for detecting speech features of different scales. In this structure, the first and fourth layers contain three parts: Conv2D is a two-dimensional convolution, ReLU is an activation layer function, and BN is a normalization layer (Batch Norm). Among them, different values of the dilation rate are taken to be responsible for the extraction of speech features of different receptive fields, and the dilation rates of the four branches are 1, 2, 4, and 8 respectively. In this structure, the stride of the first and fourth layers is 2. The second and third layers contain two parts: Conv2D and ResBlock, wherein Conv2D is a two-dimensional convolution, ResBlock is a residual module with a stride of 1; wherein, the structure of the residual module ResBlock is as shown Figure 2 shown.

[0053] Step 3: Input the speech features output by CNN into the VIT structure. The flow chart of the VIT structure is as follows: Figure 3 shown.

[0054] Specifically, the main processing of the VIT structure includes the following steps: 1. Patch and serialize the CNN output features of different scales, i.e., different receptive fields, such as Figure 3 As shown, the batch size is (dim / 2, 16); 2. Sequences of different scales are concatenated, such as Figure 3 3. Add CLS token; 4. Add position code, such as Figure 3 As shown; 5. Through the processing of the dropout layer; 6. Through Figure 4 Processing by the standard TransformerEncoder layer shown; 7. Processing by the layerNorm layer.

[0055] The CNN output features of different scales here refer to the speech features extracted by the previous CNN structure under different receptive fields. Blocking refers to an operation within an image, usually performed on an image. In the embodiments of the present invention, blocking refers to the segmentation of speech features into image patches of a certain size, and then flattening the patches. After the speech features under each receptive field are flattened, they are merged into a speech feature sequence in order of receptive field size.

[0056] By integrating features from different receptive fields into the VIT part, the small local feature changes caused by cross-channels will not play a crucial role in extracting the final voiceprint features. The final output of step three is a speech feature sequence, which has the same length as the input speech feature sequence.

[0057] Step 4: Process the speech feature sequence output by VIT. For details, see Figure 5 . Specifically, the output of VIT is a speech feature sequence, which is linearly transformed to reduce the pooling parameters. Pooling operation is performed on the sequence after linear transformation. For voiceprint recognition, attention statistical pooling (Attentive StatisticPooling), statistical pooling (Statistics pooling), etc. are generally used. After the pooling layer, voiceprint feature extraction can be performed, such as extracting the speaker's voiceprint features through the embedding layer. The voiceprint feature embedding is a sequence of a certain dimension. For example, the dimensions can be 192, 256, 384, 512, etc. Generally speaking, the output sequence of VIT needs to be compressed in sequence length first, and then the compressed sequence is pooled. Generally, the length of the sequence after pooling is also very large, and it needs to be compressed into a dimension of a specified size before voiceprint extraction is performed.

[0058] During the model training phase, the extracted embeddings need to be transformed into nclass dimensions (i.e., number of people), and then loss calculation is performed. For example, amsoftmax can be used. Then, reverse gradient propagation is performed based on the loss function to update the model parameters. After the model is trained, during the inference phase, specifically the voiceprint recognition phase, the embeddings can be directly returned.

[0059] Figure 6 This is a simplified overall flow chart of the VIT voiceprint recognition method according to one embodiment of the present invention. The figure shows only two branches in the CNN structure; in a specific implementation, there may be other numbers of branches; preferably, four branches.

[0060] Example 2:

[0061] The present invention also provides a VIT voiceprint recognition device, such as Figure 7 As shown, the device includes a processor 701, a memory 702, a bus 703, and a computer program stored in the memory 702 and executable on the processor 701. The processor 701 includes one or more processing cores. The memory 702 is connected to the processor 701 via the bus 703. The memory 702 is used to store program instructions. When the processor executes the computer program, the steps in the above-mentioned method embodiment of the first embodiment of the present invention are implemented.

[0062] Furthermore, as an executable solution, the VIT voiceprint recognition device can be a computer unit, which can be a computing device such as a desktop computer, laptop, PDA, or cloud server. The computer unit may include, but is not limited to, a processor and memory. Those skilled in the art will understand that the above-described computer unit structure is merely an example and does not constitute a limitation of the computer unit. The computer unit may include more or fewer components than those described above, or a combination of certain components, or different components. For example, the computer unit may also include input and output devices, network access devices, buses, etc., but this is not limited in the present embodiment.

[0063] Furthermore, as an executable solution, the processor may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or any conventional processor, etc. The processor is the control center of the computer unit and connects various parts of the entire computer unit using various interfaces and lines.

[0064] The memory can be used to store the computer programs and / or modules, and the processor implements the various functions of the computer unit by running or executing the computer programs and / or modules stored in the memory, and calling the data stored in the memory. The memory can mainly include a program storage area and a data storage area, wherein the program storage area can store an operating system and at least one application required for a function; the data storage area can store data created based on the use of the mobile phone, etc. In addition, the memory can include a high-speed random access memory, and can also include a non-volatile memory, such as a hard disk, a memory, a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), at least one disk storage device, a flash memory device, or other volatile solid-state storage device.

[0065] Example 3:

[0066] The present invention also provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the above method in the embodiment of the present invention are implemented.

[0067] If the module / unit integrated in the computer unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present invention implements all or part of the process in the above-mentioned embodiment method, and can also be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium. When the computer program is executed by a processor, it can implement the steps of the above-mentioned various method embodiments. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording medium, USB flash drive, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM) and software distribution medium. It should be noted that the content contained in the computer-readable medium can be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction.

[0068] Although the present invention has been particularly shown and described in conjunction with preferred embodiments, it will be understood by those skilled in the art that various changes in form and details may be made to the present invention without departing from the spirit and scope of the invention as defined in the appended claims, and all such changes are within the scope of protection of the present invention.

Claims

1. A VIT voiceprint recognition method, characterized in that: include: Noise the input speech and extract speech features from the noise-processed speech; Inputting the extracted speech features into a CNN structure of a neural network, and using the CNN structure to extract local speech features under different receptive fields from the extracted speech features; Inputting the extracted speech local features under different receptive fields into a VIT structure, processing the extracted speech local features under different receptive fields using the VIT structure, and outputting a speech feature sequence; Perform linear transformation and pooling on the output speech feature sequence, and then extract the voiceprint features after pooling; Performing voiceprint recognition using the voiceprint features after pooling processing; The step of extracting local speech features under different receptive fields from the extracted speech features using the CNN structure includes: Multiple branch CNN structures with different expansion rates are used in parallel to extract local speech features under different receptive fields; The steps of processing the extracted speech local features under different receptive fields using the VIT structure and outputting a speech feature sequence include: The extracted local speech features under different receptive fields are divided into blocks and serialized; Splicing the features after the block and serialization processing; Add CLS token; Add position coding; Processed through the dropout layer; Processed through standard transformer encoding layers.

2. The VIT voiceprint recognition method according to claim 1, characterized in that: The steps of performing noise processing on the input speech and extracting speech features from the noise-processed speech include: Use the MUSAN dataset and / or the RIR dataset for noise addition; Extract one of the fbank feature, mel spectrum feature and mfcc feature from the noise-processed speech; Perform random mask enhancement on the extracted features.

3. The VIT voiceprint recognition method according to claim 2, characterized in that: Perform random mask augmentation on the extracted features using one or more of the following methods: Fixed-dimensional mask augmentation; Pixel-level mask enhancement.

4. The VIT voiceprint recognition method according to claim 1, characterized in that: The branched CNN structure includes four layers, wherein the first and fourth layers respectively include three parts: two-dimensional convolution Conv2D, activation layer function ReLU and normalization layer; the second and third layers respectively include two parts: two-dimensional convolution Conv2D and residual module ResBlock.

5. The VIT voiceprint recognition method according to claim 1, characterized in that: The CNN structure includes four branch CNN structures, and the expansion rates of the four branch CNN structures are 1, 2, 4 and 8 respectively.

6. The VIT voiceprint recognition method according to claim 1, characterized in that: The step of processing the extracted speech local features under different receptive fields using the VIT structure and outputting a speech feature sequence further includes: Processed by layerNorm layer; Outputs the speech feature sequence processed by the layerNorm layer.

7. The VIT voiceprint recognition method according to claim 1, characterized in that: The steps of performing linear transformation and pooling processing on the output speech feature sequence include: Compressing the output speech feature sequence; Pooling is performed on the compressed sequence.

8. A VIT voiceprint recognition device, characterized in that: The method comprises a memory and a processor, wherein the memory stores at least one program, and the at least one program is executed by the processor to implement the VIT voiceprint recognition method according to any one of claims 1 to 7.

9. A computer-readable storage medium, characterized in that The storage medium stores at least one program, and the at least one program is executed by a processor to implement the VIT voiceprint recognition method according to any one of claims 1 to 7.

Images