Voiceprint data processing method and device based on hybrid model, and related equipment

Abstract

The invention discloses a voiceprint data processing method and device based on a hybrid model, and related equipment, and relates to the technical field of artificial intelligence and voiceprint recognition crossing. A convolutional neural network trained on a preset-scale data set and a big data set of enhanced voiceprint training data is used as a bottom-layer and middle-layer feature extractor; and the strong feature learning ability is fully utilized. In order to overcome the limitation of a traditional full-connection layer in small sample learning, the full-connection layer is removed from the multilayer convolutional neural network after migration freezing and network reconstruction is carried out to obtain an initial hybrid model in which the full-connection layer is replaced by an adaptive feature layer so as to realize training of the adaptive layer in a training process. Through the setting, the target hybrid model can more effectively adapt to a limited target data set, so that the accuracy of voiceprint recognition is improved.

Description

Technical Field

[0001] This application relates to the technical field of the intersection of artificial intelligence and voiceprint recognition, and more specifically, to a voiceprint data processing method, apparatus and related equipment based on a hybrid model. Background Technology

[0002] Voiceprint recognition is a type of biometric technology. It involves converting sound signals into electrical signals, which are then identified by a computer. Different tasks and applications utilize different voiceprint recognition technologies. Current voiceprint recognition methods generally include traditional machine learning and deep learning approaches.

[0003] Traditional machine learning methods rely heavily on manual feature selection. Commonly used features such as linear prediction cepstral coefficients, Mel frequency cepstral coefficients, and i-vectors can improve recognition accuracy to some extent, but they are prone to losing key features in noisy environments, leading to a decrease in recognition accuracy.

[0004] Therefore, how to improve the accuracy of voiceprint recognition is a problem that this application urgently needs to solve. Summary of the Invention

[0005] In view of this, this application discloses a method, apparatus and related equipment for voiceprint data processing based on a hybrid model, which aims to enable the target hybrid model to adapt more effectively to a limited target dataset, thereby improving the accuracy of voiceprint recognition.

[0006] To achieve the above objectives, the disclosed technical solution is as follows:

[0007] The first aspect of this application discloses a method for processing voiceprint data based on a hybrid model, the method comprising:

[0008] The source domain of the multilayer convolutional neural network is pre-trained using a pre-sized dataset and enhanced voiceprint training data.

[0009] Federated batch normalization optimization is performed on the multi-layer convolutional neural network pre-trained in the source domain.

[0010] The parameters of the convolutional layers of the multi-layer convolutional neural network pre-trained in the source domain are completely transferred and frozen.

[0011] The fully connected layers in the multi-layer convolutional neural network after migration freezing are removed, and the multi-layer convolutional neural network after removing the fully connected layers is reconstructed to obtain an initial hybrid model; wherein, the initial hybrid model is a hybrid model in which the fully connected layers are replaced with adaptive feature layers.

[0012] The initial mixture model is fine-tuned and the loss is optimized to obtain the target mixture model;

[0013] When voiceprint data to be identified is received, voiceprint recognition is performed on the voiceprint data to be identified through the target hybrid model.

[0014] Preferably, the step of pre-training the multi-layer convolutional neural network in the source domain using a pre-set size dataset and enhanced voiceprint training data includes:

[0015] Obtain the raw voiceprint training data for training, and convert the speech signals in the raw voiceprint training data into time-domain segments;

[0016] A spectrogram is generated based on the time domain segment, and the acoustic features in the spectrogram are enhanced by energy calculation and logarithmic scaling to obtain enhanced voiceprint training data.

[0017] On a pre-defined dataset, the enhanced voiceprint training data is used to learn general voiceprint features of a multi-layer convolutional neural network, thereby completing the source domain pre-training process of the multi-layer convolutional neural network.

[0018] Preferably, the federated batch normalization optimization of the pre-trained multilayer convolutional neural network in the source domain includes:

[0019] The statistics corresponding to each training batch in a multilayer convolutional neural network are calculated based on the federated batch normalization algorithm.

[0020] The activation values ​​of each training batch in the multilayer convolutional neural network are normalized using the aforementioned statistics.

[0021] By using adaptive coefficients and global statistical formulas, the global average value of the normalized multilayer convolutional neural network is calculated to complete the process of federated batch normalization optimization of the multilayer convolutional neural network after pre-training in the source domain.

[0022] Preferably, the multi-layer convolutional neural network after removing the fully connected layers is reconstructed to obtain an initial hybrid model, including:

[0023] A hybrid structure is constructed for the multi-layer convolutional neural network after removing the fully connected layers to obtain the initial hybrid model;

[0024] The hybrid structure is determined at least by a restricted Boltzmann machine and a normalized exponential function classifier.

[0025] Preferably, the step of constructing a hybrid structure for the multi-layer convolutional neural network after removing fully connected layers to obtain an initial hybrid model includes:

[0026] Obtain a preset number of feature maps of a preset size from a preset convolutional layer in a multi-layer convolutional neural network after removing the fully connected layers;

[0027] In the spatial dimension, the preset size feature maps are stitched together to obtain a multidimensional feature vector;

[0028] The multidimensional feature vector is set as the visible layer of the restricted Boltzmann machine, and a multi-level hidden layer of the restricted Boltzmann machine is set.

[0029] The restricted Boltzmann machine is trained using a preset learning strategy to obtain an initial hybrid model.

[0030] Preferably, the step of fine-tuning the initial mixture model and optimizing the loss to obtain the target mixture model includes:

[0031] Calculate the gradient of the cross-entropy loss function with respect to the weights; the gradient represents the weighted average of the prediction errors on all training samples;

[0032] The weight parameters of the initial hybrid model are calculated using the gradient descent algorithm, the gradient, and the model training learning rate; wherein the model training learning rate is used to iteratively adjust the parameters to minimize the cross-entropy loss function.

[0033] The parameters of the convolutional layers of the pre-trained multi-layer convolutional neural network are frozen, and the multi-level hidden layers of the restricted Boltzmann machine and the normalized exponential function classifier are trained using the weight parameters of the initial hybrid model.

[0034] The target hybrid model is obtained by evaluating the multi-level hidden layers of the trained Restricted Boltzmann Machine and the normalized exponential function classifier by minimizing the cross-entropy loss function.

[0035] A second aspect of this application discloses a voiceprint data processing apparatus based on a hybrid model, the apparatus comprising:

[0036] The pre-training unit is used to pre-train the multi-layer convolutional neural network in the source domain using a pre-scaled dataset and enhanced voiceprint training data.

[0037] The normalization unit is used to perform federated batch normalization optimization on the multi-layer convolutional neural network pre-trained in the source domain.

[0038] The transfer-freeze unit is used to completely transfer and freeze the parameters of the convolutional layers of a multi-layer convolutional neural network that has been pre-trained in the source domain.

[0039] The removal and reconstruction unit is used to remove the fully connected layers in the multi-layer convolutional neural network after the transfer freeze, and to reconstruct the multi-layer convolutional neural network after removing the fully connected layers to obtain an initial hybrid model; wherein, the initial hybrid model is a hybrid model in which the fully connected layers are replaced with adaptive feature layers.

[0040] The fine-tuning and optimization unit is used to fine-tune the model and optimize the loss of the initial mixture model to obtain the target mixture model.

[0041] A voiceprint recognition unit is used to perform voiceprint recognition on the voiceprint data to be recognized by means of the target hybrid model when the voiceprint data to be recognized is received.

[0042] Preferably, the pre-training unit includes:

[0043] The acquisition and conversion module is used to acquire the original voiceprint training data for training and convert the speech signal in the original voiceprint training data into a time-domain segment.

[0044] The computational enhancement module is used to generate a spectrogram based on the time-domain segment, and enhance the acoustic features in the spectrogram through energy calculation and logarithmic scaling calculation to obtain enhanced voiceprint training data.

[0045] The learning module is used to learn the general voiceprint features of a multi-layer convolutional neural network on a dataset of a preset size using the enhanced voiceprint training data, so as to complete the source domain pre-training process of the multi-layer convolutional neural network.

[0046] A third aspect of this application discloses a storage medium comprising stored instructions, wherein, when the instructions are executed, the device in which the storage medium resides executes the voiceprint data processing method based on a hybrid model as described in any one of the first aspects.

[0047] The fourth aspect of this application discloses an electronic device including a memory and one or more instructions, wherein one or more instructions are stored in the memory and configured to be executed by one or more processors using the hybrid model-based voiceprint data processing method as described in any one of the first aspects.

[0048] As described above, this application discloses a method, apparatus, and related equipment for voiceprint data processing based on a hybrid model. A convolutional neural network trained on a large dataset containing a preset-size dataset and enhanced voiceprint training data is used as the bottom and middle layer feature extractors to fully utilize its powerful feature learning capabilities. To overcome the limitations of traditional fully connected layers in few-shot learning, fully connected layers are removed from the transfer-frozen multi-layer convolutional neural network, and the network is reconstructed to obtain an initial hybrid model where fully connected layers are replaced with adaptive feature layers. This allows the adaptive layers to be trained during the training process. This setup enables the target hybrid model to adapt more effectively to a limited target dataset, thereby improving the accuracy of voiceprint recognition. Attached Figure Description

[0049] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of this application. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0050] Figure 1 This is a schematic flowchart of a voiceprint data processing method based on a hybrid model disclosed in an embodiment of this application;

[0051] Figure 2 This is a diagram of the CNN network structure based on voiceprint recognition disclosed in an embodiment of this application;

[0052] Figure 3 This is a structural diagram of the target hybrid model disclosed in the embodiments of this application;

[0053] Figure 4 This is a flowchart illustrating another voiceprint data processing method based on a hybrid model disclosed in an embodiment of this application;

[0054] Figure 5 This is a schematic diagram of the structure of a voiceprint data processing device based on a hybrid model disclosed in an embodiment of this application;

[0055] Figure 6 This is a schematic diagram of the structure of the electronic device disclosed in the embodiments of this application. Detailed Implementation

[0056] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0057] In this application, the terms "comprising," "including," or any other variations thereof are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0058] As can be seen from the background technology, traditional machine learning methods rely heavily on manual setting for feature selection. Commonly used features such as linear prediction cepstral coefficients, Mel frequency cepstral coefficients, and i-vectors can improve recognition accuracy to some extent, but they are prone to losing key features in noisy environments, leading to a decrease in recognition accuracy.

[0059] To address the aforementioned issues, this application discloses a method, apparatus, and related equipment for voiceprint data processing based on a hybrid model. A convolutional neural network trained on a large dataset containing a preset-size dataset and enhanced voiceprint training data is used as the bottom and middle layer feature extractors to fully utilize its powerful feature learning capabilities. To overcome the limitations of traditional fully connected layers in few-shot learning, fully connected layers are removed from the transfer-frozen multi-layer convolutional neural network, and the network is reconstructed to obtain an initial hybrid model where fully connected layers are replaced with adaptive feature layers. This allows the adaptive layers to be trained during the training process. This setup enables the target hybrid model to adapt more effectively to limited target datasets, thereby improving the accuracy of voiceprint recognition. The specific implementation is described in detail in the following embodiments.

[0060] It should be noted that the voiceprint data processing method, apparatus and related equipment based on a hybrid model provided in this application involve technical fields such as the intersection of artificial intelligence and voiceprint recognition, and small sample voiceprint recognition technology based on deep transfer hybrid model. The above are only examples and do not limit the application field of the voiceprint data processing method, apparatus and related equipment based on a hybrid model provided in this application.

[0061] refer to Figure 1 The image shows a method for processing voiceprint data based on a hybrid model, as disclosed in an embodiment of this application. This method mainly includes the following steps:

[0062] S101: Pre-train the multi-layer convolutional neural network in the source domain using a pre-set size dataset and enhanced voiceprint training data.

[0063] The specific process of source domain pre-training for multi-layer convolutional neural networks is shown in A1-A3.

[0064] A1: Obtain the raw voiceprint training data for training, and convert the speech signals in the raw voiceprint training data into time-domain segments.

[0065] In A1, the speech signal (i.e., the original speech signal) in the original voiceprint training data is converted into a time domain segment through frame-by-frame processing (frame length 20ms).

[0066] A2: Generate a spectrogram based on the time-domain segment, and enhance the acoustic features in the spectrogram through energy calculation and logarithmic scaling calculation to obtain enhanced voiceprint training data.

[0067] In A2, the time-domain segment is transformed into a spectrogram through the short-time Fourier transform (STFT).

[0068] Enhance the acoustic features through the energy calculation formula and logarithmic scaling calculation to obtain enhanced voiceprint training data.

[0069] The energy calculation formula is shown in formula (1).

[0070] (1)

[0071] Where, is the spectral energy; is the spectral coefficient.

[0072] The logarithmic scaling calculation method is shown in formula (2).

[0073] (2)

[0074] To address the small-sample challenge, data augmentation based on the principle of convex lens imaging is innovatively adopted, and the following three scaled spectrograms are generated according to the relationship between the object distance u and the focal length f:

[0075] 1. Generate an enlarged image when f < u < 2f;

[0076] 2. Generate an equal-sized image when u = 2f;

[0077] 3. Generate a reduced image when u > 2f.

[0078] Expand a single sample to 20 enhanced images, uniformly scale them to 227×227 pixels as the input of the CNN, increase the sample size by 300%, and can effectively suppress overfitting. The CNN network structure diagram based on voiceprint recognition is as Figure 2 shown.

[0079] A3: On a preset-scale dataset, learn the general voiceprint features of the multi-layer convolutional neural network through the enhanced voiceprint training data to complete the process of source-domain pre-training of the multi-layer convolutional neural network.

[0080] Among them, source-domain pre-training is used to let the CNN learn general voiceprint features on large-scale data (such as NIST); fast batch normalization (FBN) is used to accelerate training convergence, reduce memory occupancy, allow the use of higher learning rates, and improve pre-training efficiency.

[0081] S102: Perform federated batch normalization optimization on the multi-layer convolutional neural network after source-domain pre-training.

[0082] The specific process of performing federated batch normalization optimization on the multi-layer convolutional neural network pre-trained in the source domain is shown in B1-B3.

[0083] B1: Calculate the statistics corresponding to each training batch in the multilayer convolutional neural network according to the federated batch normalization algorithm.

[0084] B2: Normalize the activation values ​​of each training batch in the multi-layer convolutional neural network using statistical methods.

[0085] The core of source domain pre-training and FBN optimization lies in using a large-scale voiceprint dataset (NIST 2008 SRE) to efficiently pre-train a five-layer convolutional neural network (CNN). The key innovation of this process is the introduction of the Fast Batch Normalization (FBN) algorithm to replace the traditional Batch Normalization (BN).

[0086] The core calculation process of FBN is as follows:

[0087] First, for each training mini-batch, calculate the mean of its activation values ​​( ) and variance ( );

[0088] Secondly, these statistics are used to normalize the activation values ​​so that they follow a standard normal distribution with zero mean and unit variance. The normalization formula is shown in formula (3).

[0089] (3)

[0090] in, The activation value after normalization; This is the kth activation value; The mean of the activation values; For variance; It is a very small constant to prevent division by zero.

[0091] B3: Calculate the global average value of the normalized multilayer convolutional neural network using adaptive coefficients and global statistical formulas to complete the federated batch normalization optimization process for the multilayer convolutional neural network pre-trained in the source domain.

[0092] Finally, by introducing adaptive coefficients and Adaptive coefficient (and obtained dynamically during training), as shown in formula (4).

[0093] (4)

[0094] in, This is the global average. It is used to continuously update the overall data distribution, cleverly linking small-batch statistics with global statistics; and These are adaptive coefficients.

[0095] The significant advantage of FBN lies in its safe removal of the inefficient and resource-intensive recovery operation from traditional Batch Normalization (BN). This not only reduces memory usage by approximately 30% but also allows the network to use a higher learning rate, such as 0.05, significantly higher than the 0.01 without normalization. Ultimately, pre-training results on the NIST dataset demonstrate that the CNN network integrating FBN significantly reduces training time by 48.03% and doubles the convergence speed, substantially improving the efficiency of the pre-training phase.

[0096] S103: Completely transfer and freeze the parameters of the convolutional layers of the multi-layer convolutional neural network pre-trained in the source domain.

[0097] In this context, a complete migration refers to a target domain migration.

[0098] S104: Remove the fully connected layers from the multi-layer convolutional neural network after transfer freezing, and reconstruct the multi-layer convolutional neural network after removing the fully connected layers to obtain the initial hybrid model; wherein, the initial hybrid model is a hybrid model in which the fully connected layers are replaced with adaptive feature layers.

[0099] In S104, a hybrid structure is constructed for the multi-layer convolutional neural network after removing the fully connected layers to obtain the initial hybrid model; wherein, the hybrid structure is determined by at least a restricted Boltzmann machine and a normalized exponential function classifier.

[0100] It should be noted that the core objectives of target domain transfer and RBM layer reconstruction are to enable the pre-trained model to effectively adapt to the small target domain (TIMIT dataset) while overcoming the data distribution differences between the source domain (NIST 2008 SRE) and the target domain. Target domain transfer preserves the feature extraction capabilities of CNNs, while RBM layer reconstruction adapts to the target domain data through unsupervised learning, thereby improving the model's generalization ability.

[0101] The specific process of constructing a hybrid structure for the multi-layer convolutional neural network after removing the fully connected layers to obtain the initial hybrid model is shown in C1-C4.

[0102] C1: Obtain a preset number of preset-sized feature maps from a preset number of preset convolutional layers in a multi-layer convolutional neural network after removing fully connected layers.

[0103] C2: In the spatial dimension, the feature maps of the preset size are spliced ​​together to obtain a multidimensional feature vector.

[0104] C3: Set the multidimensional feature vector as the visible layer of the restricted Boltzmann machine, and set the multi-level hidden layer of the restricted Boltzmann machine.

[0105] C4: Train the restricted Boltzmann machine after setting it up using a preset learning strategy to obtain the initial hybrid model.

[0106] First, the parameters of the convolutional layers (CONV1 to CONV5) of the pre-trained CNN model on a large-scale source domain are completely transferred and frozen, preserving its powerful low- and mid-level voiceprint feature extraction capabilities. Then, the fully connected layers (FC6-FC8) of the original network are critically removed and replaced with a hybrid structure consisting of two Restricted Boltzmann Machines (RBMs) and a Softmax classifier.

[0107] The specific process of network reconstruction is as follows: The preset number of feature maps of a preset size (such as 256 feature maps of size 6×6) output by the CONV5 layer are spliced ​​in the spatial dimension to form a 1536-dimensional feature vector; this vector is fed into the visible layer of the RBM as input, which is designed to have 1536×6 nodes; followed by two levels of hidden layers, the first layer R6 contains 6000 nodes and the second layer R7 contains 1000 nodes.

[0108] It should be noted that RBM training employs an unsupervised learning strategy, the core of which is defining visible nodes. and hidden nodes The energy function is shown in formula (5).

[0109] (5)

[0110] in, For visible layer nodes; For hidden layer nodes; Visible layer node and hidden layer nodes The energy function; and These are bias parameters; These are the weight parameters.

[0111] The weights are iteratively updated using an efficient contrastive divergence (CD) algorithm. and bias The goal is to maximize the log-likelihood of the training data. The core advantage of this step is that the RBM layer not only replaces the fully connected layer's function of "connecting all features," but more importantly, it leverages its powerful unsupervised learning capabilities to automatically capture and extract unique high-order abstract statistical features from a limited set of target domain samples, significantly reducing or even eliminating the dataset differences between the source and target domains. Experimental results show that the feature vectors extracted from the RBM layer have 23% higher discriminative power than those extracted by traditional fully connected layers, directly translating into a significant improvement in small-sample voiceprint recognition performance.

[0112] S105: Fine-tune the initial mixture model and optimize the loss to obtain the target mixture model.

[0113] The core objective of the target mixture model fine-tuning and loss optimization stages is to finely adjust network parameters on a small sample target domain while avoiding the risk of overfitting due to limited data. Target mixture model fine-tuning improves the model's recognition performance in the target domain by limiting the number of trainable parameters and optimizing the cross-entropy loss, thus avoiding overfitting.

[0114] The specific process of obtaining the target mixture model is shown in D1-D4.

[0115] D1: Calculate the gradient of the cross-entropy loss function with respect to the weights; the gradient represents the weighted average of the prediction errors on all training samples.

[0116] The calculation formula for the cross-entropy loss function is shown in Equation (6).

[0117] (6)

[0118] in, Here, N is the cross-entropy loss function; N is the number of samples. This is a real label; Predict the probability distribution for the model.

[0119] As an optimization criterion, its advantage lies in its ability to directly quantify the probability distribution predicted by the model. ) and real labels ( The differences between them are particularly useful for multi-classification tasks (such as distinguishing different speakers in voiceprint recognition).

[0120] The parameter update process strictly follows the backpropagation mechanism, which is as follows:

[0121] First, calculate the gradient of the loss function with respect to the weight w as shown in Equation (7).

[0122] (7)

[0123] in, This represents the gradient of the loss function with respect to the weights w; Predict the probability distribution for the model; This is a real label; Input variables for the original voiceprint.

[0124] D2: Calculate the weight parameters of the initial mixed model (i.e., the weight parameters of the current step and the weight parameters of the next step) using the gradient descent algorithm, gradient, and model training learning rate; where the model training learning rate is used to iteratively adjust the parameters to minimize the cross-entropy loss function.

[0125] The gradient is essentially the weighted average of the prediction errors on all training samples; then the gradient descent method is applied, and the calculation formula of the gradient descent method is shown in formula (8).

[0126] (8)

[0127] in, These are the weighting parameters for the next step; These are the weight parameters for the current step; The learning rate used for model training. By controlling the update step size, the loss function is minimized through iterative adjustment of parameters.

[0128] The key fine-tuning strategy involves freezing the parameters of the pre-trained convolutional layers (CONV1-CONV5) and training only the newly added RBM layers (R6-R7) and the Softmax classifier. This design is based on two core considerations: first, the pre-trained convolutional layers already possess powerful general-purpose voiceprint feature extraction capabilities, eliminating the need for repeated adjustments for small sample target domains; second, limiting the size of trainable parameters (to only 15% of the total network parameters) significantly reduces model complexity and fundamentally suppresses overfitting tendencies in small sample scenarios. Experiments demonstrate that this strategy, combined with cross-entropy optimization, enables TLCNN-RBM to achieve a recognition rate exceeding 97% on the TIMIT dataset while maintaining excellent generalization ability. The structure of the TLCNN-RBM hybrid model (i.e., the target hybrid model) is shown below. Figure 3 As shown.

[0129] D3: Freeze the parameters of the convolutional layers of the pre-trained multi-layer convolutional neural network, and train the multi-level hidden layers of the restricted Boltzmann machine and the normalized exponential function classifier using the weight parameters of the initial hybrid model.

[0130] D4: The multi-level hidden layers of the trained Restricted Boltzmann Machine and the normalized exponential function classifier are evaluated by minimizing the cross-entropy loss function to obtain the target hybrid model.

[0131] S106: When the voiceprint data to be identified is received, the voiceprint data to be identified is identified by the target fusion model.

[0132] The voiceprint recognition and system deployment phase achieves a closed loop from algorithm to practical application. The classification decision is determined by the Softmax layer execution formula. The Softmax layer execution formula is shown in formula (9).

[0133] (9)

[0134] in, The input is the probability of a voiceprint in a certain voiceprint category; Take the natural index as the input to the i-th category label Softmax layer; The natural index is taken as the input to the Softmax layer on the k-th category label; L is the number of predicted voiceprint categories.

[0135] The Softmax layer receives the 1000-dimensional feature vector output from the RBM layer, calculates the probability Pi of the input speech belonging to each registered speaker, and finally selects the identity ID corresponding to the highest probability as the recognition result. The system achieved a 97.2% recognition accuracy on a self-built dataset (20 speakers × 10 speech segments), with an end-to-end response latency of less than 0.5 seconds, verifying the real-time performance and practicality of the technical solution. This "cloud analysis + edge execution" model ensures both the computational needs of complex models and the low-latency requirements of physical device control, providing a feasible path for the commercialization of small-sample speaker recognition technology.

[0136] The detailed flowchart of this solution is as follows: Figure 4 As shown.

[0137] Figure 4 The process includes voiceprint data preprocessing and enhancement, source domain pretraining and FBN optimization, target domain transfer and RBM layer reconstruction, hybrid model fine-tuning and loss optimization, voiceprint recognition and system deployment.

[0138] The core value of this solution lies in effectively addressing the low accuracy of voiceprint recognition in small-sample scenarios. The method involves constructing an innovative deep transfer learning hybrid model (TLCNN-RBM). This model utilizes transfer learning techniques to transfer CNN convolutional layers (CONV1-CONV5) pre-trained on large-scale voiceprint datasets (such as NIST 2008 SRE) as the basic feature extractor to small target datasets. The key innovation is replacing traditional fully connected layers (FC layers) with Restricted Boltzmann Machine (RBM) layers. This structure not only integrates all feature maps output by the convolutional layers, but more importantly, RBM possesses powerful unsupervised learning capabilities, enabling it to further learn higher-order abstract features from small-sample data. This effectively eliminates the differences between the source and target domain datasets, ultimately achieving an average recognition accuracy of over 97% on a limited dataset, significantly outperforming traditional CNNs and CNN models using only transfer learning (TL-CNN). This significantly improves the accuracy of voiceprint recognition in small-sample scenarios.

[0139] This application introduces the Fast Batch Normalization (FBN) algorithm as a key efficiency optimization point. This algorithm makes crucial improvements to Batch Normalization (BN): it removes the considered inefficient Recovery Operation, introduces adaptively learned momentum coefficients (ξ and ζ) to correlate mini-batch statistics with overall dataset statistics, and optimizes the design for GPU parallel computing. The application of FBN effectively alleviates the internal covariate shift problem, allowing for higher learning rates, thus significantly accelerating the network's training convergence process. Experimental results show that adding FBN to a CNN network reduces training time by 48.04% compared to the original CNN network and further reduces training time by 19.11% compared to adding standard BN, while also observing faster loss descent and a more stable convergence trend.

[0140] This paper addresses the problem of overfitting due to insufficient small sample data by proposing a low-cost data augmentation technique based on the principle of convex lens imaging. This method performs a geometric transformation on the spectrogram generated from the original speech, simulating convex lens imaging (producing three scaled images: larger than, equal to, and smaller than the original image), to generate diverse augmented samples. All images are then normalized to a uniform size (227x227) before being input into the network. This method effectively expands the target training set size without complex computation or additional data sources, providing cost-effective and efficient augmentation of small sample data.

[0141] The protection points of this solution are:

[0142] 1. The specific architecture and implementation method of the TLCNN-RBM hybrid model in this solution:

[0143] The key to this approach lies in the unique structure and construction process of the proposed Deep Transfer Hybrid Model (TLCNN-RBM). This includes: using pre-trained CNN convolutional layers (such as CONV1-CONV5) as fixed feature extractors; completely removing the fully connected layers (FC6-FC8) of the pre-trained CNN when transferring to the target small sample dataset; replacing these FC layers with RBM layers of a specific structure (such as R6 layer: 1536×6 visible nodes corresponding to 6000 hidden nodes, R7 layer: 1000 hidden nodes); combining the RBM layers with a Softmax classifier to form a new top layer of the network; and a specific training method for training the RBM parameters using the target dataset via the contrastive divergence (CD) algorithm and fine-tuning the entire hybrid model using the backpropagation (BP) algorithm with a cross-entropy loss function. This specific layer replacement and connection method is crucial for achieving high-precision small sample recognition.

[0144] 2. Mathematical implementation and optimization details of the Fast Batch Normalization (FBN) algorithm:

[0145] The protection points lie in the specific mathematical formulas, computational flow, and key improvements of the FBN algorithm compared to standard Batch Normalization (BN). These include: defining the fast batch normalization transformation formula for each activation value; formulas for calculating the mini-batch mean and variance; formulas for updating the global mean and variance; formulas for introducing and updating the adaptive momentum coefficients ξ and ζ; the chain rule formula for calculating gradients related to FBN parameters during backpropagation; and explicitly defining the design decisions for the deletion and restoration operation and their theoretical basis (arguing that it is inefficient and resource-intensive for improving nonlinear representation). Furthermore, the optimized design for single-dimensional independent execution of FBN based on GPU parallel computing theory is also protected. These precise mathematical expressions and specific design choices are core to achieving efficient training.

[0146] 3. Integration of data-enhanced workflows with voiceprint application systems:

[0147] The protection points also include the operational procedures of the proposed specific data augmentation techniques. Data augmentation aspects include: generating spectrograms from input audio; performing data augmentation using spectrograms; and integration into application systems such as fraud detection.

[0148] This solution, based on a deep transfer learning hybrid model, demonstrates promising application prospects in voiceprint recognition on limited datasets. Voiceprint recognition relies on the accurate extraction of individual acoustic features, but the high cost of acquiring large-scale labeled datasets leads to poor performance on limited datasets. The core of transfer learning lies in transferring knowledge from the source task to the target task to improve learning efficiency and accuracy. In voiceprint recognition, this model combines pre-trained deep neural networks, such as convolutional neural networks. It is first pre-trained on a large-scale dataset to learn general features, and then fine-tuned on a limited target dataset to adapt to specific tasks. This model effectively overcomes the overfitting problem in few-shot learning and converges faster, improving recognition accuracy. In the future, with the increase of high-quality source datasets and the optimization of model architecture, transfer learning-based voiceprint recognition systems are expected to play an important role in more practical scenarios.

[0149] This application proposes a few-shot voiceprint recognition method based on a deep transfer learning hybrid model. The method first employs a CNN trained on a large dataset as the bottom and middle layer feature extractors to fully utilize its powerful feature learning capabilities. To overcome the limitations of traditional fully connected layers in few-shot learning, we replace them with adaptive feature layers, training only during the training process. This design allows the model to adapt more effectively to limited target datasets, thereby improving the accuracy of voiceprint recognition. Furthermore, this application introduces a fast FBN algorithm, which accelerates network convergence using few-shot voiceprint data within a transfer learning framework, further improving the model's training efficiency. By combining the unsupervised learning capabilities of a Restricted Boltzmann Machine (RBM), we can quickly learn high-order statistical features of the target dataset, effectively eliminating differences between voiceprint datasets. This method not only improves the performance of few-shot voiceprint recognition but also provides new ideas for the promotion and application of voiceprint recognition technology, possessing significant theoretical and practical value.

[0150] In this embodiment, a convolutional neural network trained on a large dataset containing a preset-size dataset and enhanced voiceprint training data is used as the bottom and middle layer feature extractors to fully utilize its powerful feature learning capabilities. To overcome the limitations of traditional fully connected layers in few-shot learning, fully connected layers are removed from the transfer-frozen multi-layer convolutional neural network, and the network is reconstructed to obtain an initial hybrid model where fully connected layers are replaced with adaptive feature layers, enabling training of the adaptive layers during the training process. This setup allows the target hybrid model to adapt more effectively to the limited target dataset, thereby improving the accuracy of voiceprint recognition. This scheme addresses the problems of low accuracy and slow model convergence in voiceprint recognition with limited datasets through transfer learning, RBM layer optimization, and fast batch normalization.

[0151] Based on the above embodiments Figure 1This application discloses a method for processing voiceprint data based on a hybrid model, and also discloses a corresponding apparatus for processing voiceprint data based on a hybrid model, such as... Figure 5 As shown, the voiceprint data processing device based on the hybrid model includes:

[0152] The pre-training unit 501 is used to pre-train the multi-layer convolutional neural network in the source domain using a pre-scale dataset and enhanced voiceprint training data.

[0153] Normalization unit 502 is used to perform federated batch normalization optimization on the multi-layer convolutional neural network pre-trained in the source domain.

[0154] The migration-freezing unit 503 is used to completely migrate and freeze the parameters of the convolutional layers of the multi-layer convolutional neural network after pre-training in the source domain.

[0155] The removal and reconstruction unit 504 is used to remove the fully connected layers in the multi-layer convolutional neural network after the transfer freeze, and to reconstruct the multi-layer convolutional neural network after removing the fully connected layers to obtain the initial hybrid model; wherein, the initial hybrid model is a hybrid model in which the fully connected layers are replaced with adaptive feature layers.

[0156] The fine-tuning optimization unit 505 is used to fine-tune the initial mixture model and optimize the loss to obtain the target mixture model.

[0157] The voiceprint recognition unit 506 is used to perform voiceprint recognition on the voiceprint data to be recognized by using a target mixing model when it receives the voiceprint data to be recognized.

[0158] Furthermore, the pre-trained unit 501 includes:

[0159] The acquisition and conversion module is used to acquire the original voiceprint training data for training and convert the speech signals in the original voiceprint training data into time-domain segments.

[0160] The computational enhancement module is used to generate a spectrogram based on a time-domain segment, and enhance the acoustic features in the spectrogram using energy calculation and logarithmic scaling methods to obtain enhanced voiceprint training data.

[0161] The learning module is used to learn general voiceprint features of a multi-layer convolutional neural network on a pre-scale dataset using enhanced voiceprint training data, thereby completing the source domain pre-training process of the multi-layer convolutional neural network.

[0162] Furthermore, normalization unit 502 includes:

[0163] The first calculation module is used to calculate the statistics corresponding to each training batch in the multi-layer convolutional neural network according to the federated batch normalization algorithm.

[0164] The normalization module is used to normalize the activation values ​​of each training batch in a multi-layer convolutional neural network using statistical methods.

[0165] The second calculation module is used to calculate the global average value of the normalized multilayer convolutional neural network using adaptive coefficients and global statistical formulas, so as to complete the process of federated batch normalization optimization of the multilayer convolutional neural network after pre-training in the source domain.

[0166] Furthermore, the removal and reconstruction unit 504 is specifically used to construct a hybrid structure for the multi-layer convolutional neural network after the removal of the fully connected layers, thereby obtaining an initial hybrid model; wherein, the hybrid structure is determined at least by a restricted Boltzmann machine and a normalized exponential function classifier.

[0167] Furthermore, the refactoring unit 504 is removed, including:

[0168] The acquisition module is used to acquire a preset number of feature maps of a preset size from a preset convolutional layer in a multi-layer convolutional neural network after removing the fully connected layer;

[0169] The stitching module is used to stitch feature maps of a preset size together in a spatial dimension to obtain a multidimensional feature vector;

[0170] The configuration module is used to set multidimensional feature vectors as the visible layer of a restricted Boltzmann machine and to set the multi-level hidden layers of the restricted Boltzmann machine.

[0171] The first training module is used to train the restricted Boltzmann machine after setting it up using a preset learning strategy to obtain an initial hybrid model.

[0172] Furthermore, the fine-tuning optimization unit 505 includes:

[0173] The third calculation module is used to calculate the gradient of the cross-entropy loss function with respect to the weights; the gradient represents the weighted average of the prediction errors on all training samples.

[0174] The fourth calculation module is used to calculate the weight parameters of the initial mixed model using the gradient descent algorithm, gradient, and model training learning rate; wherein, the model training learning rate is used to iteratively adjust the parameters to minimize the cross-entropy loss function;

[0175] The second training module is used to freeze the parameters of the convolutional layers of the pre-trained multi-layer convolutional neural network and train the multi-level hidden layers of the restricted Boltzmann machine and the normalized exponential function classifier using the weight parameters of the initial hybrid model.

[0176] The evaluation module is used to evaluate the multi-level hidden layers and normalized exponential function classifier of the trained restricted Boltzmann machine by minimizing the cross-entropy loss function, so as to obtain the target hybrid model.

[0177] In this embodiment, a convolutional neural network trained on a large dataset containing a preset-size dataset and enhanced voiceprint training data is used as the bottom and middle layer feature extractors to fully utilize its powerful feature learning capabilities. To overcome the limitations of traditional fully connected layers in few-shot learning, fully connected layers are removed from the transfer-frozen multi-layer convolutional neural network, and the network is reconstructed to obtain an initial hybrid model where fully connected layers are replaced with adaptive feature layers. This allows the adaptive layers to be trained during the training process. This setup enables the target hybrid model to adapt more effectively to the limited target dataset, thereby improving the accuracy of voiceprint recognition.

[0178] This application also provides a storage medium that includes stored instructions, wherein when the instructions are executed, the device where the storage medium is located is controlled to perform the voiceprint data processing method based on the hybrid model described above.

[0179] This application also provides an electronic device, the structural schematic diagram of which is shown below. Figure 6 As shown, it specifically includes a memory 601 and one or more instructions 602, wherein one or more instructions 602 are stored in the memory 601 and configured to be executed by one or more processors 603 to perform the above-mentioned voiceprint data processing method based on the hybrid model.

[0180] For the foregoing method embodiments, in order to simplify the description, they are all described as a series of actions. However, those skilled in the art should understand that this application is not limited to the described order of actions, because according to this application, some steps can be performed in other orders or simultaneously. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily essential to this application.

[0181] It should be noted that the various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For apparatus embodiments, since they are basically similar to method embodiments, the description is relatively simple; relevant parts can be referred to the descriptions in the method embodiments.

[0182] The steps in the methods of the various embodiments of this application can be adjusted, combined, or deleted according to actual needs.

[0183] Finally, it should be noted that in this paper, relational terms such as first and second are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations.

[0184] The above description of the disclosed embodiments enables those skilled in the art to make or use this application. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the scope of this application. Therefore, this application is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

[0185] The above description is only a preferred embodiment of this application. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of this application, and these improvements and modifications should also be considered within the scope of protection of this application.

Claims

1. A voiceprint data processing method based on a hybrid model, characterized in that, The method includes: The source domain of the multilayer convolutional neural network is pre-trained using a pre-sized dataset and enhanced voiceprint training data. Federated batch normalization optimization is performed on the multi-layer convolutional neural network pre-trained in the source domain. The parameters of the convolutional layers of the multi-layer convolutional neural network pre-trained in the source domain are completely transferred and frozen. The fully connected layers in the multi-layer convolutional neural network after migration freezing are removed, and the multi-layer convolutional neural network after removing the fully connected layers is reconstructed to obtain an initial hybrid model; wherein, the initial hybrid model is a hybrid model in which the fully connected layers are replaced with adaptive feature layers. The initial mixture model is fine-tuned and the loss is optimized to obtain the target mixture model; When voiceprint data to be identified is received, voiceprint recognition is performed on the voiceprint data to be identified through the target hybrid model.

2. The method according to claim 1, characterized in that, The step of pre-training a multi-layer convolutional neural network in the source domain using a pre-set size dataset and enhanced voiceprint training data includes: Obtain the raw voiceprint training data for training, and convert the speech signals in the raw voiceprint training data into time-domain segments; A spectrogram is generated based on the time domain segment, and the acoustic features in the spectrogram are enhanced by energy calculation and logarithmic scaling to obtain enhanced voiceprint training data. On a pre-defined dataset, the enhanced voiceprint training data is used to learn general voiceprint features of a multi-layer convolutional neural network, thereby completing the source domain pre-training process of the multi-layer convolutional neural network.

3. The method according to claim 1, characterized in that, The federated batch normalization optimization of the pre-trained multilayer convolutional neural network in the source domain includes: The statistics corresponding to each training batch in a multilayer convolutional neural network are calculated based on the federated batch normalization algorithm. The activation values ​​of each training batch in the multilayer convolutional neural network are normalized using the aforementioned statistics. By using adaptive coefficients and global statistical formulas, the global average value of the normalized multilayer convolutional neural network is calculated to complete the process of federated batch normalization optimization of the multilayer convolutional neural network after pre-training in the source domain.

4. The method according to claim 1, characterized in that, The multi-layer convolutional neural network after removing fully connected layers is reconstructed to obtain an initial hybrid model, including: A hybrid structure is constructed for the multi-layer convolutional neural network after removing the fully connected layers to obtain the initial hybrid model; The hybrid structure is determined at least by a restricted Boltzmann machine and a normalized exponential function classifier.

5. The method according to claim 4, characterized in that, The process of constructing a hybrid structure for the multi-layer convolutional neural network after removing fully connected layers to obtain an initial hybrid model includes: Obtain a preset number of feature maps of a preset size from a preset convolutional layer in a multi-layer convolutional neural network after removing the fully connected layers; In the spatial dimension, the preset size feature maps are stitched together to obtain a multidimensional feature vector; The multidimensional feature vector is set as the visible layer of the restricted Boltzmann machine, and a multi-level hidden layer of the restricted Boltzmann machine is set. The restricted Boltzmann machine is trained using a preset learning strategy to obtain an initial hybrid model.

6. The method according to claim 1, characterized in that, The step of fine-tuning the initial mixture model and optimizing the loss to obtain the target mixture model includes: Calculate the gradient of the cross-entropy loss function with respect to the weights; the gradient represents the weighted average of the prediction errors on all training samples; The weight parameters of the initial hybrid model are calculated using the gradient descent algorithm, the gradient, and the model training learning rate; wherein the model training learning rate is used to iteratively adjust the parameters to minimize the cross-entropy loss function. The parameters of the convolutional layers of the pre-trained multi-layer convolutional neural network are frozen, and the multi-level hidden layers of the restricted Boltzmann machine and the normalized exponential function classifier are trained using the weight parameters of the initial hybrid model. The target hybrid model is obtained by evaluating the multi-level hidden layers of the trained Restricted Boltzmann Machine and the normalized exponential function classifier by minimizing the cross-entropy loss function.

7. A voiceprint data processing device based on a hybrid model, characterized in that, The device includes: The pre-training unit is used to pre-train the multi-layer convolutional neural network in the source domain using a pre-scaled dataset and enhanced voiceprint training data. The normalization unit is used to perform federated batch normalization optimization on the multi-layer convolutional neural network pre-trained in the source domain. The transfer-freeze unit is used to completely transfer and freeze the parameters of the convolutional layers of a multi-layer convolutional neural network that has been pre-trained in the source domain. The removal and reconstruction unit is used to remove the fully connected layers in the multi-layer convolutional neural network after the transfer freeze, and to reconstruct the multi-layer convolutional neural network after removing the fully connected layers to obtain an initial hybrid model; wherein, the initial hybrid model is a hybrid model in which the fully connected layers are replaced with adaptive feature layers. The fine-tuning and optimization unit is used to fine-tune the model and optimize the loss of the initial mixture model to obtain the target mixture model. A voiceprint recognition unit is used to perform voiceprint recognition on the voiceprint data to be recognized by means of the target hybrid model when the voiceprint data to be recognized is received.

8. The apparatus according to claim 7, characterized in that, The pre-training unit includes: The acquisition and conversion module is used to acquire the original voiceprint training data for training and convert the speech signal in the original voiceprint training data into a time-domain segment. The computational enhancement module is used to generate a spectrogram based on the time-domain segment, and enhance the acoustic features in the spectrogram through energy calculation and logarithmic scaling calculation to obtain enhanced voiceprint training data. The learning module is used to learn the general voiceprint features of a multi-layer convolutional neural network on a dataset of a preset size using the enhanced voiceprint training data, so as to complete the source domain pre-training process of the multi-layer convolutional neural network.

9. A storage medium, characterized in that, The storage medium includes stored instructions, wherein, when the instructions are executed, the device in which the storage medium resides executes the voiceprint data processing method based on a hybrid model as described in any one of claims 1 to 6.

10. An electronic device, characterized in that, It includes a memory, and one or more instructions, wherein one or more instructions are stored in the memory and configured to be executed by one or more processors as described in any one of claims 1 to 6.

Images