A Data Enhancement Method and Device for Ultrasonic Voiceprint Anti-Counterfeiting

Abstract

The present invention discloses a data enhancement method and device for ultrasonic voiceprint anti-counterfeiting, which includes the following steps: collecting original audio samples mixed with ultrasonic signals and voice signals through a mobile phone microphone and speaker, and obtaining an impulse response (ImpulseResponse) of the original collection device and at least one target device impulse response. The data enhancement method of the present invention combines the Wiener deconvolution technology. Before using the target device impulse response for convolution enhancement, the interference of the frequency response characteristics of the original collection device and environmental noise is weakened, making the enhanced audio sample closer to the audio sample actually collected by the target device. It can be used as an effective simulation data for model training. Through the impulse response data of the original collection device and the target device, the data set collected by the original device is enhanced to the data set collected by the target device, thereby enabling the algorithm to adapt to the frequency response characteristics of the target device and improve the anti-counterfeiting effect.

Description

Technical Field

[0001] The present invention relates to the field of voiceprint recognition security technology, and in particular to a data enhancement method and device for ultrasonic voiceprint anti-counterfeiting. Background Art

[0002] In recent years, with the rapid development of deep learning technology and the use of large-scale voice datasets, voiceprint recognition technology has gradually matured and is widely used in various intelligent services in our daily lives. However, existing voiceprint recognition systems have serious security risks and are threatened by various attacks such as speech synthesis attacks, voice conversion attacks, and replay attacks. Among them, replay attacks are the easiest to implement. Ordinary people without any training can use common commercial devices (such as mobile phones) to record and replay the user's voice to deceive the voiceprint recognition system's verification, thereby seriously threatening the user's privacy and data security.

[0003] Regarding the security threat of replay attacks on voiceprints, existing voiceprint anti-fraud research mainly extracts differential features from recorded voice signals for detection, but the detection ability of these features is affected by environmental factors and the quality of the attacking device. Some other studies introduce new sensing channels (such as WiFi signals, ultrasound, etc.) to capture additional biometric features to detect the liveness of the speaking subject. Among them, the liveness detection solution based on ultrasound has attracted widespread attention due to its portable deployment and user insensibility. Its typical process is: use the built-in speaker of the device to emit an ultrasonic signal towards the mouth and face of the user who is speaking. Due to the movement of the mouth, the ultrasonic reflection signal received by the microphone produces a Doppler frequency shift phenomenon. By analyzing the pattern of the Doppler frequency shift characteristics, liveness detection can be achieved, thereby effectively resisting replay attacks and ensuring the security of the voiceprint recognition system.

[0004] However, since ultrasonic voiceprint anti-counterfeiting solutions utilize wide-band sound signals (including voice signals below 8kHz and ultrasonic signals above 20kHz), and the recording devices of commercial smart terminals have huge differences in the collection and processing of sound signals in different frequency bands, the existing ultrasonic voiceprint anti-counterfeiting systems experience reduced accuracy or even algorithm failure when deployed on different devices. In addition, existing methods often have high costs or cannot effectively solve this problem. For example, domain adaptation methods require collecting data on different devices, which greatly increases the cost of solving model robustness problems. Existing data enhancement methods commonly used in the speech field are not well suited to cross-modal data that mixes speech signals and ultrasonic signals. For example, enhancement methods based on time warping and frequency masking may destroy the inherent correlation between cross-modal signals, and enhancement methods based on adding noise and room reverberation mainly solve the problem of model robustness in different environments, rather than being designed for device differences.

[0005] Therefore, the existing ultrasonic-based voiceprint anti-counterfeiting system has problems with poor anti-counterfeiting effect and poor robustness when it is migrated to different models of equipment due to the cost of collecting data and the lack of effective data enhancement methods. Summary of the Invention

[0006] The present invention improves upon the shortcomings of the prior art and provides a data enhancement method and device for ultrasonic voiceprint anti-counterfeiting. The present invention is implemented using the following technical solutions:

[0007] The present invention discloses a data enhancement method for ultrasonic voiceprint anti-counterfeiting, comprising:

[0008] The original audio sample, which is a mixture of ultrasonic signals and voice signals, is collected through the mobile phone microphone and speaker, and an impulse response (Impulse Response) of the original collection device and at least one impulse response of the target device are obtained;

[0009] The original audio samples are denoised and enhanced by combining the impulse response data of the original acquisition device. Through the Wiener deconvolution method, clean audio samples that are not affected by environmental noise and device frequency response are obtained.

[0010] Combining the clean audio sample with the impulse response data of the target device, a convolution operation is performed to obtain an enhanced audio sample that changes the frequency response characteristics of the device.

[0011] Extract the features of speech and ultrasonic frequency bands from the enhanced audio samples and the original audio samples, and obtain cross-modal joint features through feature stacking;

[0012] After further feature post-processing of the cross-modal joint features, the obtained input features can be used to train a neural network model to distinguish whether the speech source is living or not.

[0013] As a further improvement, the present invention uses the Wiener deconvolution method to perform denoising and enhancement processing on the original audio sample in combination with the impulse response data of the original acquisition device. The Wiener deconvolution method is a deconvolution method widely used in signal and image processing, specifically including:

[0014] The original audio signal collected by the original acquisition device is modeled using the linear time-invariant (LTI) system modeling method:

[0015] y(t)=(h*x)(t)+n(t)

[0016] Wherein, h(t) is the impulse response of the audio transmission channel, n(t) is the ambient noise, x(t) is the clean audio signal that is not interfered by the ambient noise and the transmission channel, * represents the convolution operation, the impulse response of the audio transmission channel is mainly determined by the frequency response of the acquisition device, the original audio signal is the original audio sample, and the clean audio signal is the clean audio sample;

[0017] A filter g(t) is estimated using the impulse response h(t) of the original acquisition device and the original audio signal y(t), thereby obtaining an estimate of the clean audio signal x(t) that is not affected by environmental noise and the device frequency response: According to the properties of the convolution operation, the above formula can be transformed into a multiplication operation in the frequency domain, namely: Where G(f) is the representation of the Wiener deconvolution filter in the frequency domain;

[0018] By minimizing the mean square error between the original audio signal and the estimated clean audio signal The representation of the Wiener deconvolution filter G(f) is derived as:

[0019]

[0020] Where H(f) represents the frequency response of the audio acquisition device, and SNR(f) is the signal-to-noise ratio of the original audio signal at different frequency points;

[0021] Estimated clean audio signal in the frequency domain Through inverse Fourier transform, it is restored to a clean audio signal in the time domain

[0022] As a further improvement, the present invention combines the clean audio sample with the impulse response data of the target device to obtain the enhanced audio sample with the changed device frequency response characteristics, specifically: the impulse response h'(t) of the other device is combined with the estimated clean audio signal Perform convolution operation in the time domain to obtain:

[0023] As a further improvement, the present invention extracts cross-modal joint features from the original audio sample and the obtained enhanced audio sample, specifically including:

[0024] Perform low-pass filtering on the sample and extract the Mel filter bank features I of the speech segment v =melspecgram(LPF(y(t))), where LPF(·) is a low-pass filter and melspecgram(·) represents the process of converting a time-domain audio signal into a time-spectrum and processing it through a Mel filter bank.

[0025] Perform bandpass filtering on the sample and extract the spectrum features I in the ultrasonic frequency band u = specgram(BPF(y(t))), where BPF(·) is the frequency f of the ultrasonic signal emitted by the speaker s is the center frequency and the bandwidth is B s The bandpass filter of , specgram(·) represents the process of converting the time-domain audio signal into a time-frequency spectrum;

[0026] Will I v with I u Align in the time domain and stack in the feature dimension to obtain the cross-modal joint feature I.

[0027] As a further improvement, the cross-modal joint features described in the present invention are further post-processed, including normalization and slicing. Specifically, the features of the ultrasonic frequency band and the voice frequency band are Min-Max normalized respectively, and the feature values are mapped to the range of 0 to 1; in the time domain, segments containing voice commands are randomly intercepted in units of a certain time span as features of the input model.

[0028] As a further improvement, the neural network model for distinguishing whether the voice source is alive or not described in the present invention is a convolutional neural network classification model.

[0029] The present invention also discloses a data enhancement device for ultrasonic voiceprint anti-counterfeiting, comprising:

[0030] Audio sample and impulse response acquisition module: used to collect original audio samples mixed with ultrasonic signals and voice signals through the mobile phone microphone and speaker, and obtain an impulse response (ImpulseResponse) of the original acquisition device and at least one target device impulse response;

[0031] Wiener deconvolution denoising module: This module denoises and enhances the original audio samples by combining the impulse response data of the original acquisition device. Using the Wiener deconvolution method, it obtains clean audio samples that are not affected by environmental noise and device frequency response.

[0032] Impulse response convolution enhancement module: used to combine clean audio samples with the impulse response data of the target device, and obtain enhanced audio samples that change the device's frequency response characteristics through convolution operation;

[0033] Cross-modal joint feature extraction module: used to extract features of speech and ultrasonic frequency bands from enhanced audio samples and original audio samples, and obtain cross-modal joint features through feature stacking;

[0034] Feature post-processing module: It is used to further post-process the cross-modal joint features. The obtained input features can be used to train a neural network model to distinguish whether the speech source is living or not.

[0035] Ultrasonic voiceprint anti-counterfeiting refers to a technology that uses the liveness perception ability of ultrasound to detect whether the voice source is a real speaker, thereby defending against replay attacks on the voiceprint recognition system. The present invention is a data enhancement method and device for ultrasonic voiceprint anti-counterfeiting, which aims to improve the robustness and anti-counterfeiting effect of the ultrasonic voiceprint anti-counterfeiting system on any device.

[0036] The beneficial effects of the present invention are as follows:

[0037] 1) Effective target device data simulation. The data enhancement method described in this paper combines Wiener deconvolution technology to reduce the interference of the original acquisition device's frequency response characteristics and environmental noise before using the target device's impulse response for convolution enhancement. This makes the enhanced audio samples closer to the audio samples actually collected by the target device, and can be used as effective simulation data for model training.

[0038] 2) Low-cost ultrasonic voiceprint anti-counterfeiting data enhancement. The data enhancement method described in this invention only requires the impulse response data of the original acquisition device and the target device. There is no need to re-acquire data on different devices, which greatly reduces the cost of acquiring and constructing ultrasonic voiceprint anti-counterfeiting datasets.

[0039] 3) Migrate ultrasonic voiceprint anti-counterfeiting algorithms to any device. The data enhancement method described in this invention uses the impulse response data of the original acquisition device and the target device to enhance the dataset collected by the original device to the dataset collected by the target device. This allows the algorithm to adapt to the frequency response characteristics of the target device and improve the anti-counterfeiting effect.

[0040] 4) Robust ultrasonic voiceprint anti-counterfeiting model training. The data augmentation method described in this invention uses impulse response data from different target devices to simulate a batch of multi-domain augmented datasets with diverse device characteristics. By training the model on this multi-domain augmented dataset or combining it with other domain adaptive training methods, the generalization and robustness of the algorithm model on different devices can be enhanced. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] Figure 1 It is a data flow diagram of the present invention;

[0042] Figure 2 An example diagram of cross-modal audio waveform and features of mixed speech and ultrasonic signals;

[0043] Figure 3 This is a comparison chart of frequency response curves of different models of mobile phone devices;

[0044] Figure 4 This is an example diagram of the effect of simulating the target device recording audio using the data enhancement method of the present invention;

[0045] Figure 5 This is an example diagram of the ultrasonic voiceprint anti-counterfeiting algorithm;

[0046] Figure 6 Comparison of error rates for models without data augmentation and with a single target device impulse response augmentation.

[0047] Figure 7 Comparison of error rates for models without data augmentation and with multiple device impulse responses.

[0048] Figure 8 The error rate comparison chart is for models without data augmentation and with data augmentation and domain adversarial training. DETAILED DESCRIPTION

[0049] The technical solution of the present invention will be further described below with reference to the accompanying drawings and specific embodiments:

[0050] The purpose of the present invention is to address the problem of poor robustness and poor anti-counterfeiting effect of ultrasonic voiceprint anti-counterfeiting systems on different deployed devices. A data enhancement method is proposed that uses the impulse response of the original acquisition device and the target device to simulate the data collected by the target device. The device for implementing the method of the present invention includes a microphone, a speaker, and a processor. Figure 1 It is a data flow diagram of the present invention;

[0051] The specific implementation method of the present invention is as follows:

[0052] Step 1: Collect original audio samples to build an ultrasonic voiceprint anti-counterfeiting dataset.

[0053] Ultrasonic voiceprint anti-counterfeiting technology uses the live sensing ability of ultrasound to detect whether the voice source is a real speaker, thereby resisting replay attacks against the voiceprint recognition system. In addition to the normal collection of the speaker's voice signal, it is also necessary to emit ultrasound to the speaker's mouth and face to collect ultrasonic reflection signals that can reflect the movement pattern of the human mouth. The mixed audio signal collected needs to be able to establish a correspondence between the voice content and the speaker's mouth movement.

[0054] The specific collection method is that the processor sends the audio data representing the ultrasonic signal to the built-in speaker, and the built-in speaker plays the ultrasonic signal towards the speaker's mouth and face. When the speaker speaks, the built-in microphone collects the mixed signal containing the speaker's voice signal and the ultrasonic echo signal and transmits it to the processor for processing and storage. When collecting the replay attack sample, the speaker is replaced by the speaker of the replay attack device. In order to make the trained model universally applicable, for different influencing factors, such as different speakers, different voice commands, different distances and angles between the speaker and the device speaker / microphone, different environmental conditions, etc., the original audio samples covering the above different conditions are collected to construct an ultrasonic voiceprint anti-counterfeiting dataset.

[0055] For example, the transmitted ultrasonic signal is a sine wave signal with a frequency of 20 kHz. Figure 2 The figure shows an example of a cross-modal audio waveform and features of a mixed speech and ultrasonic signal; it shows an original audio sample that is a mixture of an ultrasonic reflection signal and a speech signal. The spectrum diagram near 20kHz in the figure shows that the ultrasonic reflection signal produces a Doppler frequency shift due to the movement of the speaker's mouth, thereby generating a frequency offset component. This frequency offset pattern can be used as a feature to characterize the movement of the human mouth.

[0056] Step 2: Collect the impulse responses of the original acquisition device and the target device.

[0057] Figure 3 This is a comparison chart of frequency response curves for different models of mobile phone devices. The audio recording devices of different commercial devices have different frequency response characteristics and different degrees of signal distortion compared to the real sound signals due to factors such as the type and quality of the microphone components used, their layout on the device, and the subsequent processing of the collected audio signals.

[0058] This frequency response curve or impulse response data can be obtained through a binary Maximum Length Sequence (MLS) measurement. Specifically, the phone's built-in speaker transmits a maximum length sequence signal, while the phone's built-in microphone receives the signal and sends it to the processor for processing and storage. The processor calculates the cyclic cross-correlation between the received and transmitted signals to obtain the device impulse response data. To enhance the original audio sample, it is necessary to obtain the impulse response of the original acquisition device and the impulse response of at least one target device.

[0059] Step 3: Enhance the original audio samples to expand the dataset.

[0060] This step is the core of the present invention and is divided into the following sub-steps:

[0061] 1) Wiener deconvolution denoising. This step uses the Wiener deconvolution method to denoise and enhance the original audio samples in combination with the impulse response data of the original acquisition device, obtaining clean audio samples that are not affected by environmental noise and device frequency response. Specifically, according to the principle of Wiener deconvolution, a filter G(f) can be estimated based on the frequency response H(f) of the original acquisition device and the signal-to-noise ratio SNR(f) at different frequency points of the original audio samples:

[0062]

[0063] The frequency response H(f) of the original acquisition device is obtained by converting the impulse response of the original acquisition device into the frequency domain through Fourier transform. This filter reverses the influence of the device frequency response, eliminating the influence of the device. By estimating the signal-to-noise ratio at different frequency points, it eliminates the interference of environmental noise on the deconvolution calculation. By multiplying the original signal with the filter in the frequency domain and performing an inverse Fourier transform, it can be restored to a clean audio sample in the time domain:

[0064]

[0065] Among them, FFT(·) is the fast Fourier transform operation, and IFFT(·) is the inverse Fourier transform.

[0066] 2) Impulse response convolution enhancement. This step combines the clean audio sample and the impulse response data of the target device to obtain an enhanced audio sample that changes the frequency response of the device. Specifically, the impulse response h′(t) of the other device is combined with the clean audio signal obtained in the previous step. Perform convolution operations in the time domain to obtain enhanced audio samples that simulate the frequency response characteristics of the target device:

[0067] The enhanced audio samples obtained in this step have different versions depending on the target device. The complete enhanced dataset D + A collection containing the original dataset D and all versions of enhanced audio samples: Where k represents the target device number and i represents the sample number of the original dataset.

[0068] For example, Figure 4This is an example diagram of the effect of the data enhancement method described in the present invention simulating the audio recorded by the target device; the frequency spectra of the audio of the same sentence actually recorded on the target device, the audio sample enhanced by directly convolving the impulse response of the target device on the original audio sample, and the audio sample enhanced by convolution of the impulse response after Wiener deconvolution. It can be seen that since the data enhancement method described in the present invention denoises and enhances the original data sample before using the impulse response of the target device for convolution operation, the interference of the original acquisition device frequency response and environmental noise is eliminated, and thus an enhanced sample that is closer to the actual audio collected on the target device can be obtained.

[0069] Step 4: Cross-modal joint feature extraction and ultrasonic voiceprint anti-counterfeiting model training.

[0070] The enhanced dataset obtained in step 3 can be used to extract corresponding features and train a neural network model to identify whether the voice source is a real speaker or detect whether the voice is a replay attack. For example, Figure 5 This is an example diagram of an ultrasonic voiceprint anti-counterfeiting algorithm; it also shows the cross-modal joint feature extraction process and neural network structure definition used in one embodiment of the present invention. Specifically, it can be divided into the following sub-steps:

[0071] 1) Cross-modal joint feature extraction. Extract cross-modal joint features from the audio samples in the enhanced dataset, specifically:

[0072] Perform low-pass filtering on the sample and extract the Mel filter bank features I of the speech segment v =melspecgram(LPF(y(t))), where LPF(·) is a low-pass filter with a cutoff frequency of 8kHz, and melspecgram(·) represents the process of converting the time-domain audio signal into a time-frequency graph and processing it through a Mel filter bank; bandpass filtering is performed on the sample, and the spectral features I in the ultrasonic frequency band are extracted. u = specgram(BPF(y(t))), where BPF(·) is a bandpass filter with a cutoff frequency of 19700 Hz to 20300 Hz, and specgram(·) represents the process of converting a time-domain audio signal into a time-spectrogram. Preferably, 40-dimensional filter bank features (FilterBanks, FBanks) of the speech frequency band and 61-dimensional Doppler frequency shift features of the ultrasonic frequency band are extracted respectively. In order to align them in the time dimension, the window size of the short-time Fourier transform is selected to be 4800 sample points and the step size is 1200 sample points. The two features are then stacked in the feature frequency dimension to obtain the cross-modal joint feature I.

[0073] 2) Feature post-processing. The cross-modal joint features are further post-processed, including normalization and slicing. Specifically, the features of the ultrasonic frequency band and the voice frequency band are normalized by Min-Max, and the feature values are mapped to the range of 0 to 1. In the time domain, the segments containing voice commands are randomly intercepted in units of a certain time span as the input model features, that is, Input = slice (interp (norm (I v ), norm(I u ))), where slice(·) is a random slicing operation, interp(·) is a concatenation operation, and norm(·) is a normalization operation.

[0074] 3) Ultrasonic voiceprint anti-counterfeiting model training. The neural network structure of the ultrasonic voiceprint anti-counterfeiting model can be a convolutional neural network commonly used in the voiceprint field as an encoder, such as X-vector, ResNet, ECAPA-TDNN, etc. Compared with other neural networks, the ECAPA-TDNN structure achieves good voiceprint recognition effect because it integrates deep features of different granularities and controls the parameter scale. In the embodiment of the present invention, ECAPA-TDNN is used as the encoder structure, and the encoder is connected to two different downstream classifiers using a multi-task learning training framework. The speaker classification task is a multi-classification task used to constrain the model optimization direction, accelerate model convergence, and abandon the task branch after training is completed. The replay attack detection task is a binary classification task used to train the model to identify whether a voice source is a real speaker or a voice replayed through a speaker. The model is trained using a multi-task learning loss and back-propagation algorithm, and the Adam optimizer is used as the training optimizer. The multi-task learning loss is the sum of the cross entropy loss (Cross Entropy Loss) of the speaker classification task and the mean square error loss (Mean Square Error Loss) of the replay attack detection task.

[0075] The final model can effectively detect whether the source of the recorded voice is a real speaker or whether the voice is a replay attack on one or more target devices where no training data has been actually collected. The security detection results are combined with the voiceprint verification results to effectively resist the threat of replay attacks while identifying the user's identity, greatly improving the security of the voiceprint recognition system.

[0076] The present invention also discloses a data enhancement device for ultrasonic voiceprint anti-counterfeiting, comprising:

[0077] Audio sample and impulse response acquisition module: uses the microphone and speaker of the mobile device to collect audio samples of mixed ultrasonic signals and voice signals, and obtains the impulse responses of the original acquisition device and the target device;

[0078] Wiener deconvolution denoising module: This module combines the original audio samples with the impulse response data of the original acquisition device and uses the Wiener deconvolution method to obtain clean audio samples that are not affected by noise and device frequency response.

[0079] Impulse response convolution enhancement module: This module combines the impulse response data of the target device with the obtained clean audio samples and obtains enhanced audio samples that change the device's frequency response characteristics through convolution operations.

[0080] Cross-modal joint feature extraction module: extracts features of speech and ultrasonic frequency bands from enhanced and original audio samples, and obtains cross-modal joint features through feature stacking;

[0081] Feature post-processing module: further normalizes and slices the above cross-modal joint features as input for neural network training.

[0082] This invention discloses a data enhancement method and device for ultrasonic voiceprint anti-counterfeiting. The enhanced data samples and features obtained through the five modules simulate the frequency response characteristics of audio captured by a real target device. This significantly improves the anti-counterfeiting performance of the ultrasonic voiceprint anti-counterfeiting algorithm and model on the target device in the absence of actual data collected by the target device. Furthermore, through training with a large amount of enhanced data, the model's generalization and robustness can be improved when deployed on different devices.

[0083] To verify the effectiveness of the present invention, experiments were conducted on an ultrasonic voiceprint anti-counterfeiting dataset collected from mobile phones of 200 people. The dataset covers different speaker genders, ages, and accents, and covers both indoor (office, bedroom, living room) and in-vehicle collection scenarios. Each voice is 5 to 8 seconds long and has a sampling rate of 48kHz. The collection devices covered by this dataset include 7 different models of mobile phones, labeled P0, P1, P2, P3, P4, P5, and P6. All samples in the training set were collected using P0, totaling 45,190 real speaker samples and 45,190 replay attack samples. The collection devices in the test set include P0 to P6, totaling 6,898 real speaker samples and 4,298 replay attack samples.

[0084] The performance of the present invention is evaluated using the Equal Error Rate (EER) metric, which is the error rate at which the False Acceptance Rate (FAR) and False Rejection Rate (FRR) are equal. The lower the EER, the more accurate the model is in identifying replay attacks. In this embodiment of the present invention, the mean and variance of the EERs of multiple models trained under the same conditions are used as comprehensive metrics.

[0085] Model Performance Evaluation Without Data Augmentation. Table 1 shows the ER for replay attack detection on different devices in the test set after training the ultrasonic voiceprint anti-counterfeiting model on the unaugmented training set. This serves as the model's baseline performance for comparison with subsequent evaluations. It can be seen that while the model achieves an ER of 5.92% on P0, the ER performance on other devices varies. Some devices still maintain good detection performance, such as achieving ERs of 1.1% and 4.8% on P4 and P5, respectively. This is because these two devices are closer to P0 in frequency response characteristics and have similar data distributions. However, the ERs on P1, P2, P3, and P6 all exceed 20%, and even reach 50% on P1. This indicates that neural network models can overfit to the distribution of the training set data, resulting in good performance on devices with different data distributions, but varying degrees of performance degradation on devices with significantly different data distributions. This also demonstrates the necessity of using appropriate data augmentation methods during model training.

[0086] Table 1. Error rates of replay attack detection on different devices for ultrasonic voiceprint anti-counterfeiting models without data enhancement

[0087]

[0088] Enhanced model performance evaluation using a single target device impulse response. Figure 6 This is a comparison chart of the equal error rates of the models without data augmentation and with the model enhanced with the impulse response of a single target device; it compares the equal error rates of the baseline model and the ultrasonic voiceprint anti-counterfeiting model trained with data augmentation, demonstrating that the data augmentation method of the present invention improves the recognition performance of the model on different devices to varying degrees. It can be seen that after only using the impulse response of a single target device to enhance the training set, the relative decrease in the equal error rate of the model on the target device is 15.4% (P5) to 97.9% (P2). This shows that even if the model has not been trained with real target device data, the simulated data obtained by the data augmentation method can still significantly improve the model's recognition performance on the target device for replayed attack voices, thereby enabling the algorithm to quickly adapt to new devices for which data has not yet been collected.

[0089] Enhanced model performance evaluation using multiple target device impulse responses. Figure 7This is a comparison chart of the error rates of the models without data augmentation and with the models enhanced by the impulse responses of multiple devices. It compares the error rates of the baseline model and the ultrasonic voiceprint anti-counterfeiting model trained with data augmentation, demonstrating the improvement in model robustness achieved by the data augmentation method of the present invention. It can be seen that after enhancing the training set by combining the impulse responses of multiple devices, the error rates of the model have been greatly reduced on all devices. Except for device P1, the average error rates on other devices are all below 7%, and the error rate variance is less than 3%. This shows that the data augmentation method of the present invention can significantly improve the generalization ability and robustness of the model on different devices under limited data acquisition costs, thereby improving the universal applicability and practicality of the ultrasonic voiceprint anti-counterfeiting algorithm model.

[0090] Model performance evaluation for domain adversarial training using augmented datasets. Figure 8 This is a comparison chart of the error rates of models without data augmentation and with data augmentation and domain adversarial training, showing the error rates of models using the enhanced dataset obtained by the data augmentation method of the present invention for domain adversarial training (DAT). Domain adversarial training adds an additional domain classifier during model training, i.e., a classifier for predicting device type, and uses a gradient reversal layer (GRL) to enable the encoder to learn a device-independent embedding code space. It can be seen that compared to directly applying enhanced data for training, the error rate of the model after domain adversarial training on some devices is significantly further reduced, among which the error rate on P0 is relatively reduced by 25.2% to 43.4%. This proves that the enhanced data obtained by the data augmentation method of the present invention can be further combined with the domain adaptive training method, thereby greatly reducing the data acquisition cost required for applying domain adaptive or domain generalization training methods, so as to further improve the generalization ability and robustness of the model.

[0091] The above are merely preferred embodiments of the present invention and are not intended to limit the present invention. Those skilled in the art will readily appreciate that various modifications and variations of the present invention are possible. Any modifications, equivalent substitutions, or improvements made within the spirit and principles of the present invention shall be included within the scope of protection of the present invention.

Claims

1. A data enhancement method for ultrasonic voiceprint anti-counterfeiting, characterized in that: include: The original audio sample, which is a mixture of ultrasonic signals and voice signals, is collected through the mobile phone microphone and speaker, and an impulse response (Impulse Response) of the original collection device and at least one impulse response of the target device are obtained; The original audio samples are denoised and enhanced by combining the impulse response data of the original acquisition device. Through the Wiener deconvolution method, clean audio samples that are not affected by environmental noise and device frequency response are obtained. Combining the clean audio sample with the impulse response data of the target device, a convolution operation is performed to obtain an enhanced audio sample that changes the frequency response characteristics of the device. Extract the features of speech and ultrasonic frequency bands from the enhanced audio samples and the original audio samples, and obtain cross-modal joint features through feature stacking; After further feature post-processing of the cross-modal joint features, the obtained input features can be used to train a neural network model to distinguish whether the speech source is living or not.

2. The data enhancement method for ultrasonic voiceprint anti-counterfeiting according to claim 1 is characterized in that: The denoising and enhancement processing of the original audio sample by combining the impulse response data of the original acquisition device adopts the Wiener deconvolution method, which is a deconvolution method widely used in signal and image processing, specifically including: The original audio signal collected by the original acquisition device is modeled using the linear time-invariant (LTI) system modeling method: y(t)=(h*x)(t)+n(t) Wherein, h(t) is the impulse response of the audio transmission channel, n(t) is the ambient noise, x(t) is the clean audio signal that is not interfered by the ambient noise and the transmission channel, * represents the convolution operation, the impulse response of the audio transmission channel is mainly determined by the frequency response of the acquisition device, the original audio signal is the original audio sample, and the clean audio signal is the clean audio sample; A filter g(t) is estimated using the impulse response h(t) of the original acquisition device and the original audio signal y(t), thereby obtaining an estimate of the clean audio signal x(t) that is not affected by environmental noise and the device frequency response: According to the properties of the convolution operation, the above formula can be transformed into a multiplication operation in the frequency domain, namely: Where G(f) is the representation of the Wiener deconvolution filter in the frequency domain; By minimizing the mean square error between the original audio signal and the estimated clean audio signal The representation of the Wiener deconvolution filter G(f) is derived as: Where H(f) represents the frequency response of the audio acquisition device, and SNR(f) is the signal-to-noise ratio of the original audio signal at different frequency points; Estimated clean audio signal in the frequency domain Through inverse Fourier transform, it is restored to a clean audio signal in the time domain 3. The data enhancement method for ultrasonic voiceprint anti-counterfeiting according to claim 1 or 2, characterized in that: The clean audio sample is combined with the impulse response data of the target device to obtain an enhanced audio sample with a changed device frequency response characteristic, specifically: the impulse response h'(t) of the other device is combined with the estimated clean audio signal Perform convolution operation in the time domain to obtain:

4. The data enhancement method for ultrasonic voiceprint anti-counterfeiting according to claim 3 is characterized in that: The extracting of cross-modal joint features from the original audio sample and the obtained enhanced audio sample specifically includes: Perform low-pass filtering on the sample and extract the Mel filter bank features I of the speech segment v =melspecgram(LPF(y(t))), where LPF(·) is a low-pass filter and melspecgram(·) represents the process of converting a time-domain audio signal into a time-spectrum and processing it through a Mel filter bank. Perform bandpass filtering on the sample and extract the spectrum features I in the ultrasonic frequency band u = specgram(BPF(y(t))), where BPF(·) is the frequency f of the ultrasonic signal emitted by the speaker s is the center frequency and the bandwidth is B s The bandpass filter of , specgram(·) represents the process of converting the time-domain audio signal into a time-frequency spectrum; Will I v with I u Align in the time domain and stack in the feature dimension to obtain the cross-modal joint feature I.

5. The data enhancement method for ultrasonic voiceprint anti-counterfeiting according to claim 1 or 4, characterized in that: The cross-modal joint features are further post-processed, including normalization and slicing. Specifically, the features of the ultrasonic frequency band and the voice frequency band are respectively normalized by Min-Max, and the feature values are mapped to the range of 0 to 1; in the time domain, segments containing voice commands are randomly intercepted in units of a certain time span as the input model features.

6. The data enhancement method for ultrasonic voiceprint anti-counterfeiting according to claim 5 is characterized in that: The neural network model for distinguishing whether the voice source is alive is a convolutional neural network classification model.

7. A data enhancement device for ultrasonic voiceprint anti-counterfeiting, characterized in that: include: Audio sample and impulse response acquisition module: used to collect original audio samples mixed with ultrasonic signals and voice signals through the mobile phone microphone and speaker, and obtain an impulse response of the original acquisition device and at least one target device impulse response; Wiener deconvolution denoising module: This module denoises and enhances the original audio samples by combining the impulse response data of the original acquisition device. Using the Wiener deconvolution method, it obtains clean audio samples that are not affected by environmental noise and device frequency response. Impulse response convolution enhancement module: used to combine clean audio samples with the impulse response data of the target device, and obtain enhanced audio samples that change the device's frequency response characteristics through convolution operation; Cross-modal joint feature extraction module: used to extract features of speech and ultrasonic frequency bands from enhanced audio samples and original audio samples, and obtain cross-modal joint features through feature stacking; Feature post-processing module: It is used to further post-process the cross-modal joint features. The obtained input features can be used to train a neural network model to distinguish whether the speech source is living or not.

Images