Privacy protection method and device for audio information obfuscated reversible adversarial samples

Abstract

The present invention provides a privacy protection method and device for audio information obfuscated reversible adversarial samples. The method proposes a new framework called "information obfuscated reversible adversarial examples". The framework can ensure the restoration of the original quality of the audio with authorization while ensuring the privacy security of the audio. By adopting the method of the present invention, the security and privacy protection level of audio data can be greatly improved, and unauthorized access and analysis can be effectively prevented. It is intended to solve the problems in the prior art of audio data privacy protection that adversarial attacks are easy to detect, the quality of audio is significantly reduced after processing, and the limitations of the existing reversible adversarial example (RAE) technology in audio application scenarios.

Description

Technical Field

[0001] The present invention relates to the technical field of audio information obfuscated reversible adversarial samples, and in particular to a privacy protection method and device for audio information obfuscated reversible adversarial samples. Background Art

[0002] With the rapid development of artificial intelligence technology, speech recognition technology has been widely used. However, the privacy protection of audio data has become more and more prominent. Although traditional adversarial attack methods can protect data to a certain extent, they still have many shortcomings in practical applications.

[0003] Carlini and Wagner proposed an iterative attack method based on the L2 norm in 2018. This method adds perturbations to audio samples to make the ASR (Automatic Speech Recognition): automatic speech recognition system generate target phrases specified by the attacker. This attack method demonstrates the vulnerability of the ASR system in the face of adversarial samples. However, this method mainly focuses on misleading the system by adding noise, and does not consider the reversibility of the perturbation or the possibility of restoring the original audio.

[0004] Yakura and Sakuma simulated misleading attacks on ASR systems in a physical environment in 2018. They achieved a robust attack on the ASR system by introducing an impulse response and combining it with additional Gaussian white noise. This method demonstrates the effectiveness of adversarial attacks in real-world environments, but there are still problems with noise being difficult to control and highly perceptible to the human ear. In addition, this method also does not address the challenge of how to restore the audio after the attack is completed.

[0005] In 2019, Qin et al. used the psychoacoustic principle of auditory masking to create robust and imperceptible adversarial samples. Their research shows that by exploiting the masking effect of the human auditory system, adversarial samples can be generated that are auditorily imperceptible but can mislead the ASR system. However, the attack success rate of this method is closely related to the intensity of the perturbation, and a higher success rate is usually accompanied by a significant decrease in sound quality. In addition, this method still faces the problem of perturbation reversibility, that is, how to restore the original audio without affecting the audio content.

[0006] Recent studies have shown that reversible adversarial examples (RAE) show potential in data protection. RAE combines adversarial attacks with reversible data hiding (RDH) technology, aiming to mislead malicious DNNs through carefully designed reversible perturbations, while allowing authorized users to remove these perturbations and restore the original audio. However, the application of existing RAE technology in the audio field has not been fully explored. In particular, when processing audio data, how to ensure the imperceptibility of perturbations and the integrity of the restored sound quality remains an unresolved challenge.

[0007] Specifically, existing adversarial attack methods usually have difficulty distinguishing the main information and background noise in speech signals. Therefore, the generated perturbations often span the entire audio signal, including the target speech part and the background noise part. This global perturbation method may cause both background noise and speech information to be affected, reducing the accuracy and targeting of the attack. This not only affects the attack effect, but may also lead to a significant degradation in audio quality. In addition, existing adversarial attack methods usually cannot provide reversibility, that is, after the attack is completed, the original audio data cannot be restored. This is a major limitation for audio data that needs to be stored and reused for a long time. This irreversibility forces users to compromise between protecting privacy and retaining data quality when using these methods. At the same time, existing reversible adversarial example (RAE) techniques usually strictly limit the amplitude of the perturbation in ensuring the imperceptibility of the perturbation. This design is counterintuitive from the perspective of data protection, because imperceptible noise cannot physically mask the protected data, and once the data is stolen, key information can still be directly accessed.

[0008] In view of this, this application is filed. Summary of the invention

[0009] The present invention provides a privacy protection method and device for audio information obfuscated reversible adversarial samples, which can at least partially improve the above-mentioned problems.

[0010] To achieve the above object, the present invention adopts the following technical solutions:

[0011] A privacy protection method for audio information obfuscated reversible adversarial samples, comprising:

[0012] Acquire original audio data to be protected, and pre-process the original audio data using a preset ASR speech recognition system to prevent the original audio data from being analyzed by an unauthorized system, wherein the original audio data is composed of a speech signal collected by a preset audio collection device;

[0013] The original audio data is transcribed and extracted to obtain the most influential keywords, and the loss function in the adversarial attack is used for calculation to generate adversarial samples that can mislead the automatic speech recognition system, and the adversarial samples that can mislead the automatic speech recognition system are preprocessed by using the LLM large language model to generate logically coherent alternative words, and the logically coherent alternative words and the most influential keywords are replaced according to the greedy algorithm to generate the final target transcription text;

[0014] Using the Montreal forced alignment tool to process the original audio data and the final target transcription text to obtain timestamp information of each word in the audio, and constructing a mask matrix according to the timestamp information;

[0015] According to the mask matrix, the gradient matrix is ​​compressed through a composite sampling operation to obtain a compressed gradient matrix, low-frequency signals are enhanced according to a Fourier transform function, high-frequency noise is suppressed, and adversarial perturbations are superimposed on target positions of the original audio data to generate adversarial audio samples to achieve efficient privacy protection;

[0016] The reversible data hiding (RDH) technology is used to embed the gradient matrix and preset additional information into the original audio data to facilitate subsequent audio restoration.

[0017] The present invention also provides a privacy protection device for audio information obfuscated reversible adversarial samples, which includes:

[0018] An audio acquisition unit, used to acquire original audio data to be protected, and pre-process the original audio data using a preset ASR speech recognition system to prevent the original audio data from being analyzed by an unauthorized system, wherein the original audio data is composed of a speech signal collected by a preset audio acquisition device;

[0019] The final target transcription text generation unit is used to perform transcription and extraction processing on the original audio data to obtain the most influential keywords, use the loss function in the adversarial attack to calculate, generate adversarial samples that can mislead the automatic speech recognition system, use the LLM large language model to pre-process the adversarial samples that can mislead the automatic speech recognition system, generate logically coherent alternative words, and replace the logically coherent alternative words and the most influential keywords according to the greedy algorithm to generate the final target transcription text;

[0020] A mask matrix construction unit, used to process the original audio data and the final target transcription text using a Montreal forced alignment tool to obtain timestamp information of each word in the audio, and to construct a mask matrix according to the timestamp information;

[0021] A cumulative signal attack unit, used to compress the gradient matrix through a composite sampling operation according to the mask matrix to obtain a compressed gradient matrix, enhance the low-frequency signal according to the Fourier transform function, suppress the high-frequency noise, and superimpose the adversarial perturbation to the target position of the original audio data to generate an adversarial audio sample to achieve efficient privacy protection;

[0022] The RDH unit is used to embed the gradient matrix and preset additional information into the original audio data by using the reversible data hiding (RDH) technology, so as to facilitate the subsequent audio recovery.

[0023] In summary, the privacy protection method for audio information obfuscated reversible adversarial samples proposes a new framework called "information obfuscated reversible adversarial examples". This framework can ensure the restoration of the original quality of the audio with authorization while ensuring the privacy security of the audio. By adopting the method of the present invention, the security and privacy protection level of audio data can be greatly improved, and unauthorized access and analysis can be effectively prevented. It aims to solve several key problems in the privacy protection of audio data in the prior art. Specifically, these problems include the easy detection of adversarial attacks, the obvious degradation of audio quality after processing, and the limitations of existing reversible adversarial example (RAE) technology in audio application scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] Figure 1 It is a flowchart of a privacy protection method for audio information obfuscated reversible adversarial samples provided by the first embodiment of the present invention;

[0025] Figure 2 This is an example diagram of an actual scenario of a privacy protection method for audio information obfuscated reversible adversarial samples provided by the first embodiment of the present invention;

[0026] Figure 3 It is an overall framework diagram of the privacy protection method for audio information obfuscated reversible adversarial samples provided by the first embodiment of the present invention;

[0027] Figure 4 is a Mel-spectrogram of an audio processing process of a privacy protection method for audio information obfuscated reversible adversarial samples provided by the first embodiment of the present invention, Figure 4 (a) represents the original audio, Figure 4 (b) shows the adversarial audio processed by IO-RAE. Figure 4 (c) indicates restored audio;

[0028] Figure 5 It is a module diagram of a privacy protection device for audio information obfuscated reversible adversarial samples provided by the second embodiment of the present invention. DETAILED DESCRIPTION

[0029] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0030] refer to Figures 1 to 4 As shown, the first embodiment of the present invention discloses a privacy protection method for audio information obfuscated reversible adversarial samples, which can be executed by a privacy protection device for audio information obfuscated reversible adversarial samples (hereinafter referred to as a privacy protection device), and in particular, executed by one or more processors in the privacy protection device to implement the following method:

[0031] S1, obtaining original audio data to be protected, and preprocessing the original audio data using a preset ASR speech recognition system to prevent the original audio data from being analyzed by an unauthorized system, wherein the original audio data is composed of a speech signal collected by a preset audio collection device;

[0032] In this embodiment, the original audio data to be protected is input These audio data are usually voice signals collected by microphones or other recording devices, and may contain sensitive voice content. In order to prevent these data from being analyzed by unauthorized ASR systems or human eavesdroppers, they need to be processed.

[0033] S2, performing transcription and extraction processing on the original audio data to obtain the most influential keywords, using the loss function in the adversarial attack to perform calculations, generating adversarial samples that can mislead the automatic speech recognition system, using the LLM large language model to pre-process the adversarial samples that can mislead the automatic speech recognition system, generating logically coherent alternative words, and replacing the logically coherent alternative words and the most influential keywords according to a greedy algorithm to generate a final target transcription text;

[0034] Specifically, step S2 includes: Perform transcription processing to obtain the original audio data The corresponding transcript;

[0035] Use preset keywords to extract the raw audio data according to KeyBERT The corresponding transcribed text is extracted and processed to extract the most influential keywords ;

[0036] set up is the loss function in adversarial attack, where is the adversarial perturbation imposed on the audio signal, is the target label;

[0037] According to the calculation formula Calculate the difference between the prediction of the ASR system and the target label under a given perturbation to generate adversarial samples that can mislead the automatic speech recognition system, where is the prediction function of the target model, It is the connection time series classification CTC loss function;

[0038] Use the LLM large language model to preprocess adversarial samples that can mislead the automatic speech recognition system and generate logically coherent alternative words;

[0039] Use greedy algorithm to find the most influential keywords For each position in , select the optimal replacement word to construct the final target transcription text .

[0040] In this embodiment, the keyword extraction tool KeyBERT is used to extract the most influential keywords from the text transcription of the original audio. , these keywords represent the most important information in the audio content. Then set is the loss function in adversarial attack, is the adversarial perturbation imposed on the audio signal, For the original audio input, is the target label, is the prediction function of the target model, is the connection temporal classification (CTC) loss function, using the formula The difference between the prediction of the ASR system and the target label is calculated under a given perturbation. The perturbation is optimized to generate adversarial samples that can mislead the automatic speech recognition system. Subsequently, a large language model (LLM (Large Language Models)) is used to generate logically coherent replacement words, and a greedy algorithm is used to generate the corresponding replacement words. Select the optimal replacement word for each position in to construct the final target transcription text , used to mislead automatic speech recognition systems.

[0041] S3, using the Montreal forced alignment tool to process the original audio data and the final target transcription text to obtain timestamp information of each word in the audio, and construct a mask matrix according to the timestamp information;

[0042] Specifically, step S3 includes: using the Montreal forced alignment tool to align the original audio data and the final target transcript Perform alignment processing to generate the final target transcription text The corresponding relationship between the audio signal and the

[0043] Transcribe the text according to the final goal The corresponding relationship between the keyword and the audio signal is used to obtain the start and end time of each keyword in the audio and obtain the timestamp information of each word in the audio;

[0044] Construct a mask matrix based on the timestamp information , where the mask value in the time period corresponding to the keyword is 1, and the mask value in the remaining time periods is 0.

[0045] In this embodiment, after generating the target transcription text, the original audio signal is aligned using the Montreal Forced Aligner (MFA) tool. Processing, generating the final target transcription text through alignment The corresponding relationship with the audio signal can accurately obtain the start and end time of each keyword in the audio. This process generates the timestamp information of each word in the audio. Then, the mask matrix is ​​constructed based on these timestamp information. , where the mask value is 1 in the time period corresponding to the keyword, and the mask value is 0 in the rest of the time period. This mask matrix is ​​used to identify the audio segments to which adversarial perturbations need to be added, thereby ensuring that the perturbations are only added to specific audio regions, achieving the purpose of accurately misleading the automatic speech recognition system. The generated mask It will be used in the Cumulative Signal Attack (CSA) method in subsequent steps to further enhance the effect of adversarial perturbations.

[0046] Among them, the mask matrix In step S4, a marking function is performed. For example, the time period when the mask value is 0 is not attacked, and the time period when the mask value is 1 is attacked. Specifically, when the time period when the mask value is 0 is not attacked, step S4 is not performed; when the time period when the mask value is 1 is attacked, step S4 is performed.

[0047] S4, according to the mask matrix, compressing the gradient matrix through a composite sampling operation to obtain a compressed gradient matrix, enhancing the low-frequency signal according to the Fourier transform function, suppressing the high-frequency noise, and superimposing the adversarial perturbation to the target position of the original audio data to generate an adversarial audio sample to achieve efficient privacy protection;

[0048] Specifically, step S4 includes: Perform composite sampling operations to achieve the gradient matrix The compression is calculated as:

[0049] ;

[0050] in, represents the i-th element in the compressed gradient matrix after composite sampling, is the sampling interval or window size, Expressed as a loss function Input raw audio data The gradient of The value of the position, It means that the gradient values ​​are summed in each composite sampling interval and then the average value is taken;

[0051] Average sampling of the gradient information of the gradient matrix is ​​performed to obtain a compressed gradient matrix .

[0052] After completing the composite sampling operation, the disturbance signal is accumulated in the frequency domain, and the disturbance signal is converted to the frequency domain through the Fourier transform function to enhance the influence of the low-frequency signal. The calculation formula is:

[0053] ;

[0054] in, represents the disturbance signal in the frequency domain, represents the corresponding cumulative signal, is a complex exponential form, representing the phase factor of a frequency component, is an imaginary unit, is the frequency index, is the total length of the signal, It means that this factor enhances the low-frequency signal components and suppresses the high-frequency signal components in the frequency domain;

[0055] The adversarial perturbation Superimpose it to the target position of the original audio to generate adversarial audio samples .

[0056] In this embodiment, a new method named Cumulative Signal Attack (CSA) proposed by the present method is used. CSA first performs a composite sampling operation on the gradient matrix Compression is performed to reduce the amount of data while retaining the effectiveness of the attack. Next, the CSA method enhances low-frequency signals and suppresses high-frequency noise through Fourier transform, thereby avoiding the generation of excessive high-frequency noise. The following operations describe the process:

[0057] Composite sampling operation: First, CSA performs composite sampling operation on the gradient matrix. The purpose of the composite sampling operation is to compress the gradient matrix and reduce the amount of data by averaging sampling while retaining the key characteristics of adversarial perturbations. This can effectively control the size of the perturbation and avoid excessive interference with the signal. It can be described by the formula:

[0058] ;

[0059] in Represents the i-th element in the compression matrix after composite sampling, which represents the gradient information after sampling compression. is the sampling interval or window size, which indicates the number of samples to be averaged within each composite sampling interval. Expressed as a loss function For input The gradient of The value of the position, here represents the current disturbance, is either the target label or the source label. Indicates that the gradient values ​​are summed in each composite sampling interval and then the average is taken. By averaging the gradient information, a compressed gradient matrix is ​​obtained. , thereby controlling the size of the disturbance and avoiding the generation of excessive high-frequency noise.

[0060] Signal accumulation in the frequency domain: After completing the composite sampling operation, CSA performs accumulation processing on the perturbation signal in the frequency domain. Specifically, CSA converts the perturbation signal to the frequency domain through Fourier transform and enhances the influence of low-frequency signals. The purpose is to reduce the influence of high-frequency noise on the signal by enhancing the relative weight of low-frequency components in the frequency domain. Low-frequency signals are usually more sensitive to human ear perception, so this processing helps to generate more natural adversarial perturbations, making them more difficult for humans and machines to perceive. The formula is described as:

[0061] ;

[0062] in represents the disturbance signal in the frequency domain, Represents the corresponding cumulative signal. is a complex exponential form, representing the phase factor of a frequency component, where is an imaginary unit, is the frequency index, The total length of the signal. It means that this factor enhances the low-frequency signal components and suppresses the high-frequency signal components in the frequency domain.

[0063] Through the above steps, the CSA method will eventually generate an adversarial perturbation signal. , and superimpose it to the target position of the original audio to generate an adversarial audio sample .

[0064] In this embodiment, an example of a cumulative signal attack algorithm for the privacy protection method for audio information obfuscated reversible adversarial samples is as follows:

[0065] Input: original audio x, target label y, decay rate , mask matrix M, number of iterations I, unit noise , multiplication factor t.

[0066] Output: Adversarial Audio ,Noise Matrix .

[0067] 1: Multiply the mask matrix M with the original audio to obtain the initial state of the adversarial audio x;

[0068] 2: Iterate to generate adversarial samples and enter the main loop (from i=0 to i-1). The algorithm gradually generates and optimizes adversarial audio samples. ;

[0069] 3: Use current adversarial audio and noise Generating predictions for adversarial examples

[0070] ;

[0071] 4: Check the current adversarial sample Whether the ASR system is successfully misled. If successful, update the corresponding mask , used to further optimize the attack;

[0072] 5: Calculate the current loss and update the gradient based on it

[0073] ;

[0074] 6: After compressing and normalizing the gradient, it is used to update the noise matrix This step combines the mask matrix M and the step size , ensuring that the disturbance only acts on a specific time and frequency region

[0075] ;

[0076] 7: The updated noise is adjusted through the cumulative signal operation to reduce the impact of high-frequency noise and enhance low-frequency signals

[0077] ,

[0078] ;

[0079] 8: Noise matrix after the i-th update Added to current adversarial audio Generate the next generation of audio ;

[0080] 9: Iterate steps 2 to 8I times to generate the final adversarial audio and the corresponding noise matrix .

[0081] S5, using reversible data hiding (RDH) technology, embeds the gradient matrix and preset additional information into the original audio data to facilitate subsequent audio recovery.

[0082] Specifically, step S5 includes:

[0083] In this embodiment, when generating adversarial audio samples Finally, the perturbation matrix and additional information are embedded into the audio signal using reversible data hiding (RDH) technology, ensuring that the generated adversarial audio can achieve efficient privacy protection without affecting the original quality of the audio.

[0084] Preferably, the method further comprises: when it is determined that the audio needs to be restored, extracting and removing the adversarial disturbance in the audio through a reverse operation to restore the original audio data. .

[0085] In this embodiment, for authorized users, when audio needs to be restored, the adversarial disturbance in the audio can be extracted and removed through inverse operations to restore the original audio signal. The restoration process ensures the integrity of the audio quality, making the restored audio virtually indistinguishable from the original.

[0086] The generated adversarial audio samples were tested using different automatic speech recognition systems. The experimental results show that the generated adversarial audio samples can effectively mislead the automatic speech recognition system. For example, in one test, the target automatic speech recognition system misidentified "confidential meeting" as "casual gathering", achieving the purpose of misleading. In addition, the authorized user successfully restored the original audio content through the reverse operation, and the restored audio quality (PESQ score) was 4.45, indicating that the sound quality is almost lossless.

[0087] In this embodiment, Python is used as the main programming language and the Pytorch deep learning framework is used. With the goal of protecting the privacy of audio data, the information obfuscation reversible adversarial example (IO-RAE) framework is used to generate adversarial perturbations, and the misleading obfuscation of audio data is achieved through the cumulative signal attack (CSA) technology and the large language model (LLM).

[0088] The privacy protection method for reversible adversarial samples of audio information obfuscation is described first. IO-RAE (audio privacy protection framework based on reversible adversarial samples) uses a large language model (LLM) to generate logically consistent alternative words to replace keywords in audio transcription. This process helps to achieve "out of context" information by identifying the most influential words and generating relevant alternative words, thereby enhancing the protection effect of audio data. In this way, IO-RAE can effectively mislead the automatic speech recognition (ASR) system so that it cannot accurately recognize the original information. After that, IO-RAE introduces the cumulative signal attack (CSA) technology. The main purpose of CSA is to generate targeted perturbations by smoothing high-frequency signals and enhancing low-frequency signals. This method not only improves the effectiveness of the attack, but also ensures the availability and quality of the audio, so that the audio can still be used by legitimate users while protecting privacy. The cumulative signal attack imposes perturbations on the audio signal, making it impossible for the automatic speech recognition system to accurately recognize the original information when processing these perturbed audios. In addition, the IO-RAE framework also adopts the reversibility design of adversarial perturbations. This means that the perturbed audio can be restored to its original state under the operation of authorized users, thus ensuring that legitimate users can access and use the audio content. This feature allows IO-RAE to maintain the usability of audio data while protecting privacy.

[0089] In summary, the privacy protection method for audio information obfuscated reversible adversarial samples, which is based on the information obfuscated reversible adversarial example (IO-RAE) framework, significantly improves the security and privacy protection of audio data. It effectively prevents unauthorized automatic speech recognition systems and human eavesdroppers from extracting key information from audio by generating imperceptible adversarial perturbations. The perturbations optimized by the cumulative signal attack (CSA) technology can accurately confuse the automatic speech recognition system, causing errors in its transcription of the audio content, thereby protecting the privacy of the audio data.

[0090] Traditional adversarial attack methods often introduce obvious noise into the audio, which is easily detected by the human ear. However, this method enhances the influence of low-frequency signals and suppresses high-frequency noise through cumulative signal attack (CSA), so that the generated adversarial perturbation has better concealment while maintaining the attack capability, making it difficult for humans and machines to perceive. The large language model (LLM) is used to generate target replacement words, and the forced alignment technology (MFA) is combined to accurately mark the key positions in the audio, so that the adversarial perturbation can play a role in a specific time period in the audio. Experimental results show that the adversarial audio samples generated by the present invention have a misleading rate of up to 96.5% in the target automatic speech recognition system, which significantly improves the success rate of the attack. While protecting the privacy of the audio, the high-quality restoration of the audio content is ensured. Through the reversible data hiding (RDH) technology, authorized users can easily extract and remove the perturbation and restore the original audio content. The restored audio quality (such as PESQ score) is close to the original high-quality recording, ensuring the usability and practicality of the audio.

[0091] Moreover, the IO-RAE framework of this method is not only applicable to general automatic speech recognition systems, but also can achieve efficient attacks in commercial black box systems, and has broad application prospects. Its design is flexible and can be adjusted and optimized according to specific needs. It is suitable for various scenarios that require audio data privacy protection, such as online meetings, private conversation protection, sensitive information release, etc. The privacy protection method for audio information obfuscated reversible adversarial samples provides a comprehensive and innovative solution for audio privacy protection by combining advanced natural language processing technology, large language model (LLM), forced alignment technology (MFA), and cumulative signal attack (CSA) technology. The organic combination of these technologies not only enhances the effectiveness and concealment of the attack, but also promotes the development of adversarial attack technology, which is significantly forward-looking.

[0092] See also Figure 5 A second embodiment of the present invention provides a privacy protection device for audio information obfuscated reversible adversarial samples, which includes:

[0093] The audio acquisition unit 201 is used to acquire the original audio data to be protected, and pre-process the original audio data using a preset ASR speech recognition system to prevent the original audio data from being analyzed by an unauthorized system, wherein the original audio data is composed of a speech signal collected by a preset audio collection device;

[0094] The final target transcription text generating unit 202 is used to perform transcription and extraction processing on the original audio data to obtain the most influential keywords, use the loss function in the adversarial attack to calculate, generate adversarial samples that can mislead the automatic speech recognition system, use the LLM large language model to pre-process the adversarial samples that can mislead the automatic speech recognition system, generate logically coherent substitute words, and replace the logically coherent substitute words and the most influential keywords according to the greedy algorithm to generate the final target transcription text;

[0095] A mask matrix construction unit 203 is used to process the original audio data and the final target transcription text using a Montreal forced alignment tool to obtain timestamp information of each word in the audio, and to construct a mask matrix according to the timestamp information;

[0096] The cumulative signal attack unit 204 is used to compress the gradient matrix through a composite sampling operation according to the mask matrix to obtain a compressed gradient matrix, enhance the low-frequency signal according to the Fourier transform function, suppress the high-frequency noise, and superimpose the adversarial perturbation to the target position of the original audio data to generate an adversarial audio sample to achieve efficient privacy protection;

[0097] The RDH unit 205 is used to embed the gradient matrix and preset additional information into the original audio data by using the reversible data hiding (RDH) technology, so as to facilitate the subsequent audio restoration.

[0098] Preferably, the final target transcription text generating unit 202 is specifically used for:

[0099] The original audio data Perform transcription processing to obtain the original audio data The corresponding transcript;

[0100] Use preset keywords to extract the raw audio data according to KeyBERT The corresponding transcribed text is extracted and processed to extract the most influential keywords ;

[0101] set up is the loss function in adversarial attack, where is the adversarial perturbation imposed on the audio signal, is the target label;

[0102] According to the calculation formula Calculate the difference between the prediction of the ASR system and the target label under a given perturbation to generate adversarial samples that can mislead the automatic speech recognition system, where is the prediction function of the target model, It is the connection time series classification CTC loss function;

[0103] Use the LLM large language model to preprocess adversarial samples that can mislead the automatic speech recognition system and generate logically coherent alternative words;

[0104] Use greedy algorithm to find the most influential keywords For each position in , select the optimal replacement word to construct the final target transcription text .

[0105] Preferably, the mask matrix construction unit 203 is specifically used for:

[0106] Use the Montreal Force Alignment tool to align the raw audio data and the final target transcript Perform alignment processing to generate the final target transcription text The corresponding relationship between the audio signal and the

[0107] Transcribe the text according to the final goal The corresponding relationship between the keyword and the audio signal is used to obtain the start and end time of each keyword in the audio and obtain the timestamp information of each word in the audio;

[0108] Construct a mask matrix based on the timestamp information , where the mask value in the time period corresponding to the keyword is 1, and the mask value in the remaining time periods is 0.

[0109] Preferably, the cumulative signal attack unit 204 is specifically used for:

[0110] Gradient matrix Perform composite sampling operations to achieve the gradient matrix The compression is calculated as:

[0111] ;

[0112] in, represents the i-th element in the compressed gradient matrix after composite sampling, is the sampling interval or window size, Expressed as a loss function Input raw audio data The gradient of The value of the position, It means that the gradient values ​​are summed in each composite sampling interval and then the average value is taken;

[0113] Average sampling of the gradient information of the gradient matrix is ​​performed to obtain a compressed gradient matrix ;

[0114] After completing the composite sampling operation, the disturbance signal is accumulated in the frequency domain, and the disturbance signal is converted to the frequency domain through the Fourier transform function to enhance the influence of the low-frequency signal. The calculation formula is:

[0115] ;

[0116] in, represents the disturbance signal in the frequency domain, represents the corresponding cumulative signal, is a complex exponential form, representing the phase factor of a frequency component, is an imaginary unit, is the frequency index, is the total length of the signal, It means that this factor enhances the low-frequency signal components and suppresses the high-frequency signal components in the frequency domain;

[0117] The adversarial perturbation Superimpose it to the target position of the original audio to generate adversarial audio samples .

[0118] The above is a preferred embodiment of the present invention. It should be pointed out that a person skilled in the art can make several improvements and modifications without departing from the principle of the present invention. These improvements and modifications are also considered to be within the scope of protection of the present invention.

Claims

1. A privacy protection method for audio information obfuscated reversible adversarial samples, characterized in that: include: Acquire original audio data to be protected, and pre-process the original audio data using a preset ASR speech recognition system to prevent the original audio data from being analyzed by an unauthorized system, wherein the original audio data is composed of a speech signal collected by a preset audio collection device; The original audio data is transcribed and extracted to obtain the most influential keywords, and the loss function in the adversarial attack is used for calculation to generate adversarial samples that can mislead the automatic speech recognition system, and the adversarial samples that can mislead the automatic speech recognition system are preprocessed by using the LLM large language model to generate logically coherent alternative words, and the logically coherent alternative words and the most influential keywords are replaced according to the greedy algorithm to generate the final target transcription text; Using the Montreal forced alignment tool to process the original audio data and the final target transcription text to obtain timestamp information of each word in the audio, and constructing a mask matrix according to the timestamp information; According to the mask matrix, the gradient matrix is ​​compressed through a composite sampling operation to obtain a compressed gradient matrix, low-frequency signals are enhanced according to a Fourier transform function, high-frequency noise is suppressed, and adversarial perturbations are superimposed on target positions of the original audio data to generate adversarial audio samples; The reversible data hiding (RDH) technology is used to embed the gradient matrix and preset additional information into the original audio data to facilitate subsequent audio restoration.

2. The privacy protection method for audio information obfuscated reversible adversarial samples according to claim 1, characterized in that: The original audio data is transcribed and extracted to obtain the most influential keywords, and the loss function in the adversarial attack is used for calculation to generate adversarial samples that can mislead the automatic speech recognition system. The adversarial samples that can mislead the automatic speech recognition system are preprocessed using the LLM large language model to generate logically coherent alternative words, and the logically coherent alternative words and the most influential keywords are replaced according to the greedy algorithm to generate the final target transcription text, which is specifically: The original audio data Perform transcription processing to obtain the original audio data The corresponding transcript; Use preset keywords to extract the raw audio data according to KeyBERT The corresponding transcribed text is extracted and processed to extract the most influential keywords ; set up is the loss function in adversarial attack, where is the adversarial perturbation imposed on the audio signal, is the target label; According to the calculation formula Calculate the difference between the prediction of the ASR system and the target label under a given perturbation to generate adversarial samples that can mislead the automatic speech recognition system, where is the prediction function of the target model, It is the connection time series classification CTC loss function; Use the LLM large language model to preprocess adversarial samples that can mislead the automatic speech recognition system and generate logically coherent alternative words; Use greedy algorithm to find the most influential keywords For each position in , select the optimal replacement word to construct the final target transcription text .

3. The privacy protection method for audio information obfuscated reversible adversarial samples according to claim 1, characterized in that: The original audio data and the final target transcription text are processed using the Montreal forced alignment tool to obtain the timestamp information of each word in the audio, and a mask matrix is ​​constructed based on the timestamp information, specifically: Use the Montreal Force Alignment tool to align the raw audio data and the final target transcript Perform alignment processing to generate the final target transcription text The corresponding relationship between the audio signal and the Transcribe the text according to the final goal The corresponding relationship between the keyword and the audio signal is used to obtain the start and end time of each keyword in the audio and obtain the timestamp information of each word in the audio; Construct a mask matrix based on the timestamp information , where the mask value in the time period corresponding to the keyword is 1, and the mask value in the remaining time periods is 0.

4. The privacy protection method for audio information obfuscated reversible adversarial samples according to claim 1, characterized in that: According to the mask matrix, the gradient matrix is ​​compressed through a composite sampling operation to obtain a compressed gradient matrix, specifically: Gradient matrix Perform composite sampling operations to achieve the gradient matrix The compression is calculated as: ； in, represents the i-th element in the compressed gradient matrix after composite sampling, is the sampling interval or window size, Expressed as a loss function Input raw audio data The gradient of The value of the position, It means that the gradient values ​​are summed in each composite sampling interval and then the average value is taken; Average sampling of the gradient information of the gradient matrix is ​​performed to obtain a compressed gradient matrix .

5. The privacy protection method for audio information obfuscated reversible adversarial samples according to claim 4 is characterized in that: According to the Fourier transform function, low-frequency signals are enhanced, high-frequency noise is suppressed, and adversarial perturbations are superimposed on the target position of the original audio data to generate adversarial audio samples to achieve efficient privacy protection. Specifically: After completing the composite sampling operation, the disturbance signal is accumulated in the frequency domain, and the disturbance signal is converted to the frequency domain through the Fourier transform function to enhance the influence of the low-frequency signal. The calculation formula is: ； in, represents the disturbance signal in the frequency domain, represents the corresponding cumulative signal, is a complex exponential form, representing the phase factor of a frequency component, is an imaginary unit, is the frequency index, is the total length of the signal, It means that this factor enhances the low-frequency signal components and suppresses the high-frequency signal components in the frequency domain; The adversarial perturbation Superimpose it to the target position of the original audio to generate adversarial audio samples .

6. The privacy protection method for audio information obfuscated reversible adversarial samples according to claim 1, characterized in that: Also includes: When it is determined that the audio needs to be restored, the adversarial perturbation in the audio is extracted and removed through inverse operations to restore the original audio data .

7. A privacy protection device for audio information obfuscated reversible adversarial samples, characterized in that: include: An audio acquisition unit, used to acquire original audio data to be protected, and pre-process the original audio data using a preset ASR speech recognition system to prevent the original audio data from being analyzed by an unauthorized system, wherein the original audio data is composed of a speech signal collected by a preset audio acquisition device; The final target transcription text generation unit is used to perform transcription and extraction processing on the original audio data to obtain the most influential keywords, use the loss function in the adversarial attack to calculate, generate adversarial samples that can mislead the automatic speech recognition system, use the LLM large language model to pre-process the adversarial samples that can mislead the automatic speech recognition system, generate logically coherent alternative words, and replace the logically coherent alternative words and the most influential keywords according to the greedy algorithm to generate the final target transcription text; A mask matrix construction unit, used to process the original audio data and the final target transcription text using a Montreal forced alignment tool to obtain timestamp information of each word in the audio, and to construct a mask matrix according to the timestamp information; A cumulative signal attack unit, used to compress the gradient matrix through a composite sampling operation according to the mask matrix to obtain a compressed gradient matrix, enhance the low-frequency signal according to the Fourier transform function, suppress the high-frequency noise, and superimpose the adversarial perturbation to the target position of the original audio data to generate an adversarial audio sample; The RDH unit is used to embed the gradient matrix and preset additional information into the original audio data by using the reversible data hiding (RDH) technology, so as to facilitate the subsequent audio recovery.

8. The privacy protection device for audio information obfuscation reversible adversarial samples according to claim 7, characterized in that: The final target transcription text generation unit is specifically used for: The original audio data Perform transcription processing to obtain the original audio data The corresponding transcript; Use preset keywords to extract the raw audio data according to KeyBERT The corresponding transcribed text is extracted and processed to extract the most influential keywords ; set up is the loss function in adversarial attack, where is the adversarial perturbation imposed on the audio signal, is the target label; According to the calculation formula Calculate the difference between the prediction of the ASR system and the target label under a given perturbation to generate adversarial samples that can mislead the automatic speech recognition system, where is the prediction function of the target model, It is the connection time series classification CTC loss function; Use the LLM large language model to preprocess adversarial samples that can mislead the automatic speech recognition system and generate logically coherent alternative words; Use greedy algorithm to find the most influential keywords For each position in , select the optimal replacement word to construct the final target transcription text .

9. The privacy protection device for audio information obfuscated reversible adversarial samples according to claim 7, characterized in that: The mask matrix construction unit is specifically used for: Use the Montreal Force Alignment tool to align the raw audio data and the final target transcript Perform alignment processing to generate the final target transcription text The corresponding relationship between the audio signal and the Transcribe the text according to the final goal The corresponding relationship between the keyword and the audio signal is used to obtain the start and end time of each keyword in the audio and obtain the timestamp information of each word in the audio; Construct a mask matrix based on the timestamp information , where the mask value in the time period corresponding to the keyword is 1, and the mask value in the remaining time periods is 0.

10. The privacy protection device for audio information obfuscation reversible adversarial samples according to claim 7, characterized in that: The cumulative signal attack unit is specifically used for: Gradient matrix Perform composite sampling operations to achieve the gradient matrix The compression is calculated as: ； in, represents the i-th element in the compressed gradient matrix after composite sampling, is the sampling interval or window size, Expressed as a loss function Input raw audio data The gradient of The value of the position, It means that the gradient values ​​are summed in each composite sampling interval and then the average value is taken; Average sampling of the gradient information of the gradient matrix is ​​performed to obtain a compressed gradient matrix ; After completing the composite sampling operation, the disturbance signal is accumulated in the frequency domain, and the disturbance signal is converted to the frequency domain through the Fourier transform function to enhance the influence of the low-frequency signal. The calculation formula is: ； in, represents the disturbance signal in the frequency domain, represents the corresponding cumulative signal, is a complex exponential form, representing the phase factor of a frequency component, is an imaginary unit, is the frequency index, is the total length of the signal, It means that this factor enhances the low-frequency signal components and suppresses the high-frequency signal components in the frequency domain; The adversarial perturbation Superimpose it to the target position of the original audio to generate adversarial audio samples .

Images