Acoustic shielding-based pseudo-tone AI voice cloning resisting method and system

Abstract

The invention belongs to the technical field of audio data processing, and relates to a pseudo tone anti-AI voice cloning method and system based on acoustic shielding, and the method comprises the steps: S1, constructing a pseudo tone based on the tone characteristics of a given audio file material; s2, determining a masking threshold value of the target audio according to the intensity and the spectrum characteristics of the target audio, and marking the frequency band with the power lower than the masking threshold value as an unaudible frequency band; and S3, embedding the pseudo tone into an unaudible frequency band of a target audio to obtain a synthesized audio. According to the method and the device, active defense of AI voice cloning can be realized on the premise of not influencing the tone quality, relatively high universality and robustness are achieved, and various voice cloning attacks can be resisted.

Description

Technical Field

[0001] The present application belongs to the field of audio data processing technology, and in particular, relates to a method and system for counteracting AI voice cloning with pseudo-timbre based on acoustic masking. Background Art

[0002] With the advancement of artificial intelligence (AI) technology, voice cloning technology has developed rapidly in recent years. Voice cloning technology uses deep learning algorithms to generate cloned voices that are highly similar to the voices of specific people by training a large amount of voice data. This technology has broad application prospects in voice interaction, virtual assistants, entertainment, and education. However, the abuse of voice cloning technology also brings risks such as privacy security and identity forgery. For example, malicious users can clone other people's voices to commit telecommunications fraud, spread false information, and even affect the authenticity of evidence in the judicial field. Therefore, how to prevent AI models from cloning sensitive audio information has become a technical problem that needs to be solved urgently.

[0003] There are already some defense technologies and products against voice cloning on the market, mainly including noise addition method, sound watermark method and adversarial method based on feature perturbation.

[0004] The noise addition method adds background noise or adversarial noise to the original audio, making it difficult for the AI ​​voice cloning model to accurately identify the characteristics of the audio. Adversarial noise is generally generated by a generative adversarial network (GAN), which can interfere with the AI ​​model without significantly affecting human hearing. However, the noise addition method often requires the introduction of higher-intensity noise into the audio to achieve the ideal defense effect, which will cause a significant decrease in sound quality and affect the user experience. At the same time, with the advancement of AI noise reduction technology, some noise reduction algorithms can effectively remove noise, thereby bypassing the defense and weakening its adversarial effect.

[0005] Sound watermarking technology embeds imperceptible watermark features in audio for subsequent detection of whether the audio has been cloned. Watermarks are usually embedded in audio through small changes in high or low frequencies and can only be detected in synthetic audio. However, the detection of sound watermarks depends on the integrity of the audio. Once the audio is compressed or format converted, the watermark is easily damaged, which affects its detection effect. In addition, the audio with embedded watermarks can still be trained and reproduced by AI models, which fails to fundamentally prevent cloning. In fact, sound watermarks will only work when forged audio causes serious public opinion.

[0006] The adversarial method based on feature perturbation refers to perturbing the audio features so that the AI ​​model cannot correctly restore the features when generating cloned voices. This method interferes with specific frequency bands of the audio signal, such as modulating the fundamental frequency or harmonic characteristics, so that the cloned voice is distorted. However, since this method requires frequent adjustment of the interference features, it is difficult to adapt to different voice cloning algorithms, and the defense effect is unstable. Summary of the invention

[0007] In order to solve the above problems, the present application provides a method and system for combating AI voice cloning with pseudo timbres based on acoustic masking. By introducing pseudo timbres into audio signals and utilizing the acoustic masking effect of the human auditory system, these pseudo timbres are made imperceptible to the human ear, thereby achieving effective interference with the AI ​​voice cloning model.

[0008] The first aspect of the present application provides a method for counteracting AI voice cloning with pseudo-timbre based on acoustic masking, which mainly includes:

[0009] Step S1, constructing a pseudo timbre based on the pitch characteristics of a given audio file material;

[0010] Step S2: determining a masking threshold of the target audio according to the intensity and spectrum characteristics of the target audio, and marking a frequency band whose power is lower than the masking threshold as an inaudible frequency band;

[0011] Step S3: embed the pseudo tone into the inaudible frequency band of the target audio to obtain synthesized audio.

[0012] Preferably, step S1 further comprises:

[0013] Step S11, extracting pitch features from the audio file material, wherein the pitch features at least include fundamental frequency, formant and harmonics;

[0014] Step S12, calculating the Mel spectrum of the audio file material;

[0015] Step S13: extracting mel-frequency cepstral coefficients from the mel-frequency spectrum as key representations of pseudo-timbre, and constructing a pseudo-timbre feature dictionary formed by combining the mel-frequency cepstral coefficients and modulated pitch features.

[0016] Preferably, in step S13, the modulation of the tone feature includes adjusting the phase and amplitude of the tone feature.

[0017] Preferably, step S2 further comprises:

[0018] Step S21, converting the audio signal of the target audio into the frequency domain to obtain a spectrum;

[0019] Step S22, dividing the spectrum into multiple frequency bands, and determining a masking threshold for each frequency band;

[0020] Step S23: Obtain the frequency band index whose power is lower than the masking threshold.

[0021] Preferably, before step S3, the method further comprises:

[0022] Step S30: performing distortion modulation on the target audio.

[0023] Preferably, step S3 further comprises:

[0024] Step S31, traversing the frequency band index of each inaudible frequency band of the target audio, and replacing the inaudible frequency band in the spectrum of the target audio with the corresponding part of the pseudo tone;

[0025] Step S32: inversely transform the modified frequency spectrum of the target audio to obtain a synthetic audio embedded with a pseudo tone.

[0026] The second aspect of the present application provides a system for counteracting AI voice cloning with pseudo-timbre based on acoustic masking, which mainly includes:

[0027] A pseudo-timbre construction module, used for constructing a pseudo-timbre based on the pitch characteristics of a given audio file material;

[0028] An inaudible frequency band determination module, used to determine a masking threshold of the target audio according to the intensity and spectrum characteristics of the target audio, and mark a frequency band with power lower than the masking threshold as an inaudible frequency band;

[0029] The audio synthesis module is used to embed the pseudo tone into the inaudible frequency band of the target audio to obtain synthesized audio.

[0030] Preferably, the pseudo-timbre building module comprises:

[0031] A pitch measurement unit, used to extract pitch features from audio file materials, wherein the pitch features include at least fundamental frequency, formant and harmonics;

[0032] Mel spectrum calculation unit, used to calculate the Mel spectrum of audio file materials;

[0033] The pseudo-timbre feature dictionary construction unit is used to extract the mel-frequency cepstral coefficients from the mel-frequency spectrum as the key representation of the pseudo-timbre, and to construct a pseudo-timbre feature dictionary formed by combining the mel-frequency cepstral coefficients and the modulated pitch features.

[0034] Preferably, the inaudible frequency band determination module includes:

[0035] A spectrum acquisition unit, used for converting the audio signal of the target audio into the frequency domain to obtain a spectrum;

[0036] The masking threshold calculation unit for each frequency band is used to divide the spectrum into multiple frequency bands and determine the masking threshold of each frequency band;

[0037] The frequency band index obtaining unit is used to obtain the frequency band index whose power is lower than the masking threshold.

[0038] Preferably, the audio synthesis module includes:

[0039] A replacement unit, configured to traverse a frequency band index of each inaudible frequency band of the target audio, and replace the inaudible frequency band in the frequency spectrum of the target audio with a corresponding part of the pseudo tone;

[0040] The Fourier transform unit is used to inversely transform the frequency spectrum of the modified target audio to obtain the synthesized audio embedded with the pseudo tone.

[0041] This application can achieve active defense against AI voice cloning without affecting the sound quality. It has high versatility and robustness and can resist various voice cloning attacks. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] Figure 1 The present invention is a flowchart of a preferred embodiment of the method for counteracting AI voice cloning based on pseudo-timbre of acoustic masking.

[0043] Figure 2 This is a schematic diagram of the principle of pseudo-timbre synthesis based on masking threshold in this application. DETAILED DESCRIPTION

[0044] In order to make the purpose, technical scheme and advantages of the implementation of this application clearer, the technical scheme in the implementation of this application will be described in more detail in combination with the drawings in the implementation of this application. In the drawings, the same or similar reference numerals throughout represent the same or similar elements or elements with the same or similar functions. The described implementation is a part of the implementation of this application, not all of the implementations. The implementation described below with reference to the drawings is exemplary and is intended to be used to explain this application, and cannot be understood as a limitation on this application. Based on the implementation in this application, all other implementations obtained by ordinary technicians in this field without making creative work are within the scope of protection of this application. The implementation of this application is described in detail below in combination with the drawings.

[0045] The first aspect of the present application provides a method for counteracting AI voice cloning with pseudo-timbre based on acoustic masking, such as Figure 1 As shown, it mainly includes:

[0046] Step S1: construct a pseudo timbre based on the pitch features of a given audio file material.

[0047] This step processes the audio file material to obtain a pseudo timbre so as to merge it with the target audio. In some optional implementations, step S1 further includes:

[0048] Step S11, extracting pitch features from the audio file material, wherein the pitch features at least include fundamental frequency, formant and harmonics;

[0049] Step S12, calculating the Mel spectrum of the audio file material;

[0050] Step S13: extracting mel-frequency cepstral coefficients from the mel-frequency spectrum as key representations of pseudo-timbre, and constructing a pseudo-timbre feature dictionary formed by combining the mel-frequency cepstral coefficients and modulated pitch features.

[0051] First, in step S11, the audio file material is analyzed to extract its pitch features such as fundamental frequency, formant and harmonics. In step S12, the Mel spectrum mel_spec of the audio is calculated to capture the time-frequency features of the audio. Then in step S13, the Mel frequency cepstrum coefficients are extracted from the Mel spectrum as the key representation of the timbre, combined with the pitch features of step S1, and a pseudo timbre feature dictionary is formed for subsequent pseudo timbre embedding.

[0052] In some optional implementations, in step S13, modulating the tone feature includes adjusting the phase and amplitude of the tone feature.

[0053] The audio file material referred to in step S1 can be the same as the target audio in step S2, or it can be another audio file different from the target audio in step S2. In this embodiment, when the same audio file as the target audio is used, audio modulation can be performed on it. Accordingly, the phase and amplitude of the audio signal can be slightly adjusted here so that the modulated pseudo-timbre is inconsistent with the target audio, thereby generating false features. After these false features replace the real features of the original target audio in step S3, the AI ​​model can produce deviations when learning audio features, thereby failing to accurately clone the original sound.

[0054] Step S2: determining a masking threshold of the target audio according to the intensity and spectrum characteristics of the target audio, and marking a frequency band with power lower than the masking threshold as an inaudible frequency band.

[0055] After the pseudo tone is given in step S1, step S2 is used to determine which data in the target audio can be replaced by the pseudo tone of step S1. Specifically, step S2 searches for frequency bands in the original target audio that are difficult for the human ear to perceive based on the masking effect curve of the human ear in different frequency bands, and marks them as inaudible frequency bands. These frequency bands can be modified at will.

[0056] In some optional implementations, step S2 further includes:

[0057] Step S21, converting the audio signal of the target audio into the frequency domain to obtain a spectrum;

[0058] Step S22, dividing the spectrum into multiple frequency bands, and determining a masking threshold for each frequency band;

[0059] Step S23: Obtain the frequency band index whose power is lower than the masking threshold.

[0060] In this embodiment, after loading the target audio file, the audio signal and sampling rate are obtained, and the audio signal collected in each cycle is converted into the frequency domain to obtain the spectrum. Figure 2 The relationship between the frequency spectrum and the sound intensity is given. Figure 2 In the figure, the horizontal axis is the frequency value of the audio signal, and the vertical axis is the sound intensity of the audio signal, which can be regarded as the energy value of the frequency. When the sound intensity of a frequency is less than a certain threshold, the human ear cannot hear it, as shown by the dotted line in the figure. For example, at a frequency of 0.02KHz, corresponding to a frequency of 20Hz, the sound intensity must reach 70 decibels to be heard. If the sound intensity is lower than 70 decibels, it cannot be heard. In addition, when the sound signal of a certain frequency has a large energy, that is, when the sound signal intensity of a certain frequency is very large, the masking threshold near this frequency will increase a lot. For example, at about 0.3kHz, the sound signal energy of this frequency is very large, about 60 decibels. At this time, the masking threshold of the signal near this frequency (0.3KHz) will increase, as shown in Figure 1. Figure 2 The solid line shows.

[0061] Based on this, this application constructs a frequency domain based on the acoustic masking effect of the human ear and the curve of the relationship between volume and frequency. Figure 2 The masking curve shown is used to determine whether each frequency band in step S22 is an inaudible frequency band and the masking threshold of the inaudible frequency band.

[0062] It is further noted that the strength of the pseudo-timbre in each frequency band can be fine-tuned by weighted mapping to ensure the best interference effect without affecting human ear perception. Through weighting and mapping operations, the system can optimize the amplitude of pseudo-timbre in different frequency bands, making the output audio both high-quality and highly defensive.

[0063] Step S3: embed the pseudo tone into the inaudible frequency band of the target audio to obtain synthesized audio.

[0064] In some optional implementations, the process further includes before step S3:

[0065] Step S30: performing distortion modulation on the target audio.

[0066] In order to enhance the versatility and anti-attack capability of the defense, this embodiment introduces a distortion layer before the pseudo-timbre is embedded. The distortion layer is embedded in the audio processing and is used to modulate the audio features at the Mel spectrum stage. The slight distortion modulation further increases the complexity of the audio features, making it difficult for the AI ​​voice cloning model to accurately restore the original features of the audio. The processing of the distortion layer mainly includes modulating the characteristic parts of the target audio that do not need to replace the pseudo-timbre, ensuring that it can play an interference effect in different cloning attacks. The modulation effect of the distortion layer significantly improves the defense effect without affecting the human ear's hearing.

[0067] In some optional implementations, step S3 further includes:

[0068] Step S31, traversing the frequency band index of each inaudible frequency band of the target audio, and replacing the inaudible frequency band in the spectrum of the target audio with the corresponding part of the pseudo tone;

[0069] Step S32: inversely transform the modified frequency spectrum of the target audio to obtain a synthetic audio embedded with a pseudo tone.

[0070] In step S31, it should be noted that the corresponding part of the pseudo-timbre refers to the spectrum data belonging to the same frequency band as the target audio, and the sound intensity of these spectrum data should be lower than the masking threshold. Since the audio file material used to generate the pseudo-timbre is usually consistent with the target audio, therefore, in the process of processing the target audio and the audio file material, when it is determined that the intensity of certain frequency bands of the spectrum data of the target audio is lower than the masking threshold, the intensity of these frequency bands of the corresponding audio file material is also lower than the masking threshold, so in step S31, only the frequency band index is needed to replace the data of the corresponding frequency band of the target audio with the data of the corresponding frequency band of the pseudo-timbre. On the contrary, when the audio file material used is inconsistent with the target audio, it should be ensured that the pseudo-timbre constructed by the audio file material all belongs to inaudible audio, that is, it is below the masking threshold.

[0071] Finally, after the audio synthesis is completed, a short-time Fourier transform (STFT) analysis can be performed on it to verify the embedding effect of the pseudo-timbre. The STFT analysis provides a frequency domain feature map of the audio, which can intuitively show the distribution of the pseudo-timbre in the audio spectrum and ensure that the generated defense audio meets the design requirements.

[0072] The second aspect of the present application provides a system for counteracting AI voice cloning based on acoustic masking of pseudo-timbre corresponding to the above method, which mainly includes:

[0073] A pseudo-timbre construction module, used for constructing a pseudo-timbre based on the pitch characteristics of a given audio file material;

[0074] An inaudible frequency band determination module, used to determine a masking threshold of the target audio according to the intensity and spectrum characteristics of the target audio, and mark a frequency band with power lower than the masking threshold as an inaudible frequency band;

[0075] The audio synthesis module is used to embed the pseudo tone into the inaudible frequency band of the target audio to obtain synthesized audio.

[0076] In some optional implementations, the pseudo-timbre building module includes:

[0077] A pitch measurement unit, used to extract pitch features from audio file materials, wherein the pitch features include at least fundamental frequency, formant and harmonics;

[0078] Mel spectrum calculation unit, used to calculate the Mel spectrum of audio file materials;

[0079] The pseudo-timbre feature dictionary construction unit is used to extract the mel-frequency cepstral coefficients from the mel-frequency spectrum as the key representation of the pseudo-timbre, and to construct a pseudo-timbre feature dictionary formed by combining the mel-frequency cepstral coefficients and the modulated pitch features.

[0080] In some optional implementations, the inaudible frequency band determination module includes:

[0081] A spectrum acquisition unit, used for converting the audio signal of the target audio into the frequency domain to obtain a spectrum;

[0082] The masking threshold calculation unit for each frequency band is used to divide the spectrum into multiple frequency bands and determine the masking threshold of each frequency band;

[0083] The frequency band index obtaining unit is used to obtain the frequency band index whose power is lower than the masking threshold.

[0084] In some optional implementations, the audio synthesis module includes:

[0085] A replacement unit, configured to traverse a frequency band index of each inaudible frequency band of the target audio, and replace the inaudible frequency band in the frequency spectrum of the target audio with a corresponding part of the pseudo tone;

[0086] The Fourier transform unit is used to inversely transform the frequency spectrum of the modified target audio to obtain the synthesized audio embedded with the pseudo tone.

[0087] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any changes or substitutions that can be easily thought of by a person skilled in the art within the technical scope disclosed in the present application should be included in the protection scope of the present application. Therefore, the protection scope of the present application shall be based on the protection scope of the claims.

Claims

1. A method for counteracting AI voice cloning with pseudo-timbre based on acoustic masking, characterized in that: include: Step S1, constructing a pseudo timbre based on the pitch characteristics of a given audio file material; Step S2: determining a masking threshold of the target audio according to the intensity and spectrum characteristics of the target audio, and marking a frequency band whose power is lower than the masking threshold as an inaudible frequency band; Step S3: embed the pseudo tone into the inaudible frequency band of the target audio to obtain synthesized audio.

2. The method for counteracting AI voice cloning with pseudo-timbre based on acoustic masking according to claim 1, characterized in that: Step S1 further comprises: Step S11, extracting pitch features from the audio file material, wherein the pitch features at least include fundamental frequency, formant and harmonics; Step S12, calculating the Mel spectrum of the audio file material; Step S13: extracting mel-frequency cepstral coefficients from the mel-frequency spectrum as key representations of pseudo-timbre, and constructing a pseudo-timbre feature dictionary formed by combining the mel-frequency cepstral coefficients and modulated pitch features.

3. The method for counteracting AI voice cloning based on acoustic masking pseudo-timbre as claimed in claim 1, characterized in that: In step S13, the modulation of the tone feature includes adjusting the phase and amplitude of the tone feature.

4. The method for counteracting AI voice cloning with pseudo-timbre based on acoustic masking according to claim 1, characterized in that: Step S2 further comprises: Step S21, converting the audio signal of the target audio into the frequency domain to obtain a spectrum; Step S22, dividing the spectrum into multiple frequency bands, and determining a masking threshold for each frequency band; Step S23: Obtain the frequency band index whose power is lower than the masking threshold.

5. The method for counteracting AI voice cloning based on acoustic masking pseudo-timbre as claimed in claim 1, characterized in that: Before step S3, the method further comprises: Step S30: performing distortion modulation on the target audio.

6. The method for counteracting AI voice cloning based on acoustic masking pseudo-timbre as claimed in claim 1, characterized in that: Step S3 further comprises: Step S31, traversing the frequency band index of each inaudible frequency band of the target audio, and replacing the inaudible frequency band in the spectrum of the target audio with the corresponding part of the pseudo tone; Step S32: inversely transform the modified frequency spectrum of the target audio to obtain a synthetic audio embedded with a pseudo tone.

7. A system for counteracting AI voice cloning based on acoustic masking pseudo-timbre, characterized in that: include: A pseudo-timbre construction module, used for constructing a pseudo-timbre based on the pitch characteristics of a given audio file material; An inaudible frequency band determination module, used to determine a masking threshold of the target audio according to the intensity and spectrum characteristics of the target audio, and mark a frequency band with power lower than the masking threshold as an inaudible frequency band; The audio synthesis module is used to embed the pseudo tone into the inaudible frequency band of the target audio to obtain synthesized audio.

8. The system for counteracting AI voice cloning based on acoustic masking of pseudo-timbre as claimed in claim 7, characterized in that: The pseudo-timbre building module comprises: A pitch measurement unit, used to extract pitch features from audio file materials, wherein the pitch features include at least fundamental frequency, formant and harmonics; Mel spectrum calculation unit, used to calculate the Mel spectrum of audio file materials; The pseudo-timbre feature dictionary construction unit is used to extract the mel-frequency cepstral coefficients from the mel-frequency spectrum as the key representation of the pseudo-timbre, and to construct a pseudo-timbre feature dictionary formed by combining the mel-frequency cepstral coefficients and the modulated pitch features.

9. The system for counteracting AI voice cloning based on acoustic masking pseudo-timbre as claimed in claim 7, characterized in that: The inaudible frequency band determination module includes: A spectrum acquisition unit, used for converting the audio signal of the target audio into the frequency domain to obtain a spectrum; The masking threshold calculation unit for each frequency band is used to divide the spectrum into multiple frequency bands and determine the masking threshold of each frequency band; The frequency band index obtaining unit is used to obtain the frequency band index whose power is lower than the masking threshold.

10. The system for counteracting AI voice cloning based on acoustic masking pseudo-timbre as claimed in claim 7, characterized in that: The audio synthesis module comprises: A replacement unit, configured to traverse a frequency band index of each inaudible frequency band of the target audio, and replace the inaudible frequency band in the frequency spectrum of the target audio with a corresponding part of the pseudo tone; The Fourier transform unit is used to inversely transform the frequency spectrum of the modified target audio to obtain the synthesized audio embedded with the pseudo tone.

Images