Diffusion model-based anti-voice conversion watermark embedding method and program product

Abstract

The application discloses an anti-voice conversion watermark embedding method and program product based on a diffusion model, which comprises inputting real-time carrier voice into a trained anti-voice conversion watermark embedding network based on a diffusion model to generate watermark-containing voice; the training method of the anti-voice conversion watermark embedding network comprises: preprocessing received historical carrier voice to generate logarithmic mel spectrum and diffusion prior, and sending the historical carrier voice into the anti-voice conversion watermark embedding network together with the logarithmic mel spectrum and the diffusion prior to generate watermark-containing voice; the model parameters of the anti-voice conversion watermark embedding network are constantly updated with the minimum total loss function as the target, the total loss function comprises an adversarial loss function between the conversion results of the carrier voice and the watermark-containing voice, a diffusion loss function between the predicted noise and the actual added noise, and a frequency weighting loss function of the watermark-containing voice and the carrier voice under multi-scale short-time Fourier transform. The application can guarantee the imperceptibility of the watermark, improve the anti-voice conversion performance and robustness of the watermark-containing voice.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of digital audio watermarking, and particularly relates to an anti-voice conversion watermark embedding method based on a diffusion model and a program product. BACKGROUND

[0002] Voice deep generation technology has been able to generate realistic voices. While enriching people's entertainment and life, it is also easily misused by lawbreakers for voice forgery, thereby bringing great hidden troubles to personal privacy and social safety. Therefore, resistance and destruction to voice forgery have become a field with practical significance. This technology processes the protected voice in a special way, which destroys the conversion result of the voice as the target voice under the premise of ensuring imperceptibility, thereby preventing the harm from occurring at the root. In recent years, while this field has achieved rapid development, there are still many challenges and bottlenecks.

[0003] In 2021, Huang et al. proposed to embed watermark in the frequency domain of the voice to be protected by iterative optimization (Huang C, Lin Y Y, Lee H, et al. Defending your voice: Adversarial attack on voice conversion [C] / / Proceedings of the 2021 IEEE Spoken Language Technology Workshop (SLT). Shenzhen, China: IEEE, 2021: 552-559.), which opened up a new direction of resisting voice conversion by destroying the output results. In 2023, Li et al. proposed to embed imperceptible adversarial watermark in the time domain of the voice (Li J, Ye D, Tang L, et al. VoiceGuard: Protecting voice privacy with strong and imperceptible adversarial perturbation in the time domain [C] / / Proceedings of the 32nd International Joint Conference on Artificial Intelligence (IJCAI-23). Macau, China: IJCAI, 2023: 4812-4820.), which avoided the reconstruction loss of the voice to be protected due to time-frequency conversion and achieved effective destruction of the conversion results while improving imperceptibility. The above two methods are generated by iterative watermark, which has the problem of slow reasoning speed. In 2024, Dong et al. proposed a method of generating adversarial watermark directly by GAN generator and embedding the voice to be protected (Dong S, Chen B, Ma K, et al. Active defense against voice conversion through generative adversarial network [J]. IEEE Signal Processing Letters, 2024, 31: 706-710.), and designed the SWCSM network to reconstruct the waveform from the spectrum, which ensured the destruction of the conversion results and the imperceptibility of the voice to be protected while speeding up the reasoning speed.However, the above three methods are to embed watermarks in the protected voice, and such methods have two problems: (1) existing methods are difficult to balance the anti-voice conversion capability and the imperceptibility of the voice sample, and in order to improve the practicability of the anti-sample, the anti-voice conversion capability is often compromised; (2) in the complex propagation process, the watermark information is easily removed, so that the robustness of the method needs to be improved when facing compression and filtering attacks.

[0004] Based on the above analysis, the existing anti-voice conversion method is still difficult to ensure the success rate and robustness of resisting voice conversion under the premise of ensuring imperceptibility, and the reasons are as follows: (1) the traditional watermark embedding method only makes local modification in the waveform or spectrum diagram, ignoring the overall structure of the voice signal, so that the watermark information is concentrated in the high frequency part which is easy to attenuate; (2) in the above method, the watermark is an additional part, and the excessive watermark strength will lead to the decrease of imperceptibility, thereby reducing the imperceptibility of the protected voice, and it is difficult to balance the anti-voice conversion capability and the imperceptibility. SUMMARY

[0005] In view of the above problems, the present application provides an anti-voice conversion watermark embedding method based on a diffusion model and a program product, which can improve the anti-voice conversion performance and robustness of the watermark-containing voice while ensuring the imperceptibility of the watermark.

[0006] In order to achieve the above technical purpose and achieve the above technical effect, the present application realizes the following technical scheme:

[0007] In a first aspect, the present application provides an anti-voice conversion watermark embedding method based on a diffusion model, comprising:

[0008] inputting the obtained real-time carrier voice into the trained anti-voice conversion watermark embedding network based on the diffusion model to generate a watermark-containing voice;

[0009] The training method of the anti-voice conversion watermark embedding network based on the diffusion model comprises:

[0010] preprocessing the received historical carrier voice to generate a log mel spectrum and a diffusion prior;

[0011] sending the historical carrier voice, the log mel spectrum and the diffusion prior into the anti-voice conversion watermark embedding network based on the diffusion model to generate a watermark-containing voice;

[0012] The model parameters of the anti-voice conversion watermark embedding network are continuously updated to minimize a total loss function, and the training of the anti-voice conversion watermark embedding network is completed, the total loss function including an adversarial loss function between the voice conversion result of the carrier voice and the conversion result of the watermark-containing voice, a diffusion loss function between the predicted noise and the actual added noise, and a frequency weighting loss function of the watermark-containing voice and the carrier voice under multi-scale short-time Fourier transform.

[0013] With reference to the first aspect, optionally, the diffusion prior generation method comprises:

[0014] The carrier voice is framed and windowed, and the amplitude spectrum of each frame of the carrier voice after the framing and windowing processing is obtained by short-time Fourier transform, the amplitude spectrum is mapped by a Mel filter bank and logarithm is taken to generate a corresponding log Mel spectrum, and the log Mel spectrum is also used as conditional information;

[0015] The frame-level energy of the log Mel spectrum is calculated frame by frame to obtain a frame-level energy sequence;

[0016] Based on the frame-level energy sequence, a covariance matrix with a diagonal structure is constructed in time sequence;

[0017] The spectral flatness of the carrier voice is calculated, and based on the relationship between the spectral flatness and a preset threshold, appropriate Gaussian noise is determined, and the diffusion prior is obtained in combination with the covariance matrix.

[0018] With reference to the first aspect, optionally, the calculation formula of the spectral flatness is:

[0019] ,

[0020] In the formula, is the spectral flatness, is the total number of frequency bands, is the spectral value of the i-th frequency band, is a positive number less than a set threshold, represents the logarithm function with e as the base;

[0021] When , super-Gaussian noise is selected;

[0022] When , standard Gaussian noise is selected;

[0023] When , sub-Gaussian noise is selected.

[0024] With reference to the first aspect, optionally, the anti-voice conversion watermark embedding network comprises:

[0025] ​The forward diffusion module is configured to randomly select a diffusion time step, inject Gaussian noise into the carrier voice according to a noise schedule, and generate a noisy voice;

[0026] The feature tensor generation module is configured to perform a scale-unification operation on the noisy voice, the log-mel spectrum and the diffusion time step, and add feature channels to form a unified modulation feature tensor;

[0027] The residual feature extraction module is configured to perform residual feature extraction processing on the feature tensor to obtain predicted noise;

[0028] The inverse diffusion module is configured to perform inverse diffusion update on the noisy voice based on the predicted noise and diffusion prior to generate a watermark-containing voice.

[0029] In combination with the first aspect, optionally, the residual feature extraction module includes a plurality of sequentially arranged dilated residual blocks, and a residual component output of a previous dilated residual block is input to a next dilated residual block; the Skip components of all dilated residual blocks are added and averaged, reduced in dimension by 1x1 convolution, and activated by ReLU to obtain the predicted noise.

[0030] In combination with the first aspect, optionally, a mathematical expression of the total loss function is:

[0031] ,

[0032] In the formula, is the total loss function, is an adversarial loss function between the speech conversion result of the carrier voice and the conversion result of the watermark-containing voice, is a diffusion loss function between the predicted noise and the actual added noise, is a frequency weighting loss function of the watermark-containing voice and the carrier voice under multi-scale short-time Fourier transform, , , respectively are weights of the adversarial loss function, the diffusion loss function and the frequency weighting loss function.

[0033] In combination with the first aspect, optionally, a mathematical expression of the diffusion loss function between the predicted noise and the actual added noise is:

[0034]

[0035] In the formula, and respectively represent the actual injected Gaussian noise and the predicted noise, is an inverse matrix of a covariance matrix, represents a time step , This represents conditional information, which is a log-Mel spectrum. Represents the carrier's speech. This indicates noisy speech. This indicates the diffusion time step t and the carrier speech. Actual injected Gaussian noise Take the expected value together. This represents a weighted L2 norm.

[0036] In conjunction with the first aspect, optionally, the difference in waveform between the carrier speech and the watermarked speech can be denoted as the perturbation. The waveform difference is divided into multiple sub-bands by short-time Fourier transform and transformed to the frequency domain to obtain the amplitude spectrum of the corresponding sub-bands. ;

[0037] Based on the sensing weights of each frequency band and the amplitude spectrum of the sub-band Generate a frequency-weighted loss function under single-scale short-time Fourier transform. ;

[0038] Based on the frequency-weighted loss functions under all single-scale short-time Fourier transforms, frequency-weighted loss functions for watermarked speech and carrier speech under multi-scale short-time Fourier transforms are generated. The frequency-weighted loss function under the single-scale short-time Fourier transform Frequency-weighted loss function under multi-scale short-time Fourier transform The mathematical expressions are as follows:

[0039] ,

[0040] ,

[0041] In the formula, For the first The center frequency of each frequency band The total number of frequency bands. ∈[0,1] is the first The perceptual weight of each frequency band is assigned; a higher weight value indicates that the human ear is more sensitive to that frequency band. Indicates the first The weighted weights corresponding to each scale Indicates the first Each scale.

[0042] In conjunction with the first aspect, optionally, the adversarial loss function between the speech conversion result of the carrier speech and the conversion result of the watermarked speech... The mathematical expression is:

[0043] ,

[0044] In the formula, represents the mel-spectrogram generated after the carrier voice and the source voice input to the voice conversion model for voice conversion, represents the mel-spectrogram generated after the watermark-containing voice and the source voice input to the voice conversion model for voice conversion, represents norm.

[0045] In a second aspect, the present application provides a computer program product comprising computer programs / instructions for implementing the diffusion model-based voice conversion resistant watermark embedding method of any one of the first aspect when executed by a processor.

[0046] Compared with the prior art, the present application has the following beneficial effects:

[0047] The present application provides a diffusion model-based voice conversion resistant watermark embedding method and program product, which can improve the voice conversion resistant performance and robustness of the watermark-containing voice while ensuring the imperceptibility of the watermark.

[0048] Further, the present application utilizes the powerful generation capability of the diffusion model, and designs a diffusion prior determined by input data different from the traditional diffusion model. Compared with the prior art, the watermark-containing voice generated by the present application has higher quality, and can improve the strength of the watermark while ensuring the imperceptibility of the watermark-containing voice. At the same time, a frequency weighting loss function under multi-scale short-time Fourier transform is designed, which effectively improves the imperceptibility of the watermark-containing voice by adding the watermark mainly in the frequency band insensitive to human ears, and indirectly improves the voice conversion resistant capability of the watermark-containing voice. BRIEF DESCRIPTION OF DRAWINGS

[0049] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the drawings needed in the embodiments will be briefly introduced as follows. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor on the basis of these drawings.

[0050] Figure 1 FIG. 1 is a flowchart of a diffusion model-based voice conversion resistant watermark embedding method according to an embodiment of the present application;

[0051] Figure 2 FIG. 2 is a training framework diagram of a voice conversion resistant watermark embedding network according to an embodiment of the present application;

[0052] Figure 3 FIG. 3 is an application framework diagram of a voice conversion resistant watermark embedding network according to an embodiment of the present application. DETAILED DESCRIPTION

[0053] The technical solutions in the embodiments of the present application will be described clearly and completely below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by a person of ordinary skill in the art without creative work are within the protection scope of the present application.

[0054] In addition, if the present application embodiments involve descriptions such as “first”, “second”, etc., the descriptions of “first”, “second”, etc. are only for the purpose of description, and cannot be understood as indicating or implying the relative importance of the indicated technical features or implicitly indicating the number of the indicated technical features. Therefore, the features limited by “first”, “second” can explicitly or implicitly include at least one of the features. In addition, the technical solutions of each embodiment can be combined with each other, but it must be based on the fact that a person of ordinary skill in the art can realize it. When the combination of technical solutions contradicts each other or cannot be realized, it should be considered that the combination of technical solutions does not exist and is not within the protection scope of the present application.

[0055] Embodiment 1

[0056] The present application embodiment provides an anti-voice conversion watermark embedding method based on a diffusion model, including the following steps:

[0057] The obtained real-time carrier voice is input into the trained anti-voice conversion watermark embedding network based on the diffusion model to generate a watermark-containing voice.

[0058] The training method of the anti-voice conversion watermark embedding network based on the diffusion model includes:

[0059] The received historical carrier voice is preprocessed to generate a log-mel spectrum and a diffusion prior. In the specific implementation process, as shown in Figure 2 , the received historical carrier voice is preprocessed by the preprocessing module to generate a log-mel spectrum (i.e. conditional information ) and a diffusion prior P(0, );

[0060] The historical carrier voice, the log-mel spectrum and the diffusion prior are sent into the anti-voice conversion watermark embedding network based on the diffusion model to generate a watermark-containing voice.

[0061] With the goal of minimizing the total loss function, the model parameters of the anti-speech-switching watermarking embedding network are continuously updated to complete the training of the anti-speech-switching watermarking embedding network. The total loss function includes the adversarial loss function between the speech conversion result of the carrier speech and the conversion result of the watermarked speech, the diffusion loss function between the predicted noise and the actual added noise, and the frequency-weighted loss function of the watermarked speech and the carrier speech under multi-scale short-time Fourier transform.

[0062] Based on the above scheme, it is possible to improve the anti-speech conversion performance and robustness of watermarked speech while ensuring the imperceptibility of the watermark.

[0063] In one specific embodiment of the present invention, the method for generating the diffusion prior includes:

[0064] The carrier speech is segmented and windowed, and the short-time Fourier transform is performed on each frame of carrier speech after segmentation and windowing to obtain the amplitude spectrum. The amplitude spectrum is mapped through the Mel filter bank and the logarithm is taken to generate the corresponding log-Mel spectrum. The log-Mel spectrum is also used as conditional information.

[0065] Frame-level energy is calculated frame by frame from the log-Mel spectrum to obtain a frame-level energy sequence;

[0066] Based on the frame-level energy sequence, a diagonal covariance matrix is ​​constructed in chronological order. ;

[0067] The spectral flatness of the carrier speech is calculated, and a suitable Gaussian noise is determined based on the relationship between spectral flatness and a preset threshold. This noise is then combined with the covariance matrix. The diffusion prior P(0, ).

[0068] In one specific embodiment of the present invention, the formula for calculating the spectral flatness is:

[0069] ,

[0070] In the formula, For spectral flatness, Total number of frequency bands For the first Spectral values ​​of each frequency band A positive number less than a set threshold. Represents the logarithmic function with base e;

[0071] when In this case, super-Gaussian noise should be selected;

[0072] when In this case, standard Gaussian noise should be selected;

[0073] When , the next Gaussian noise is selected.

[0074] Unlike the traditional diffusion model that uniformly uses standard Gaussian noise as the diffusion prior, based on the above scheme, a generalized diffusion prior P(0, ) that depends on the input data is obtained, which can make the forward diffusion process closer to the natural distribution of the input data, and lay a foundation for subsequent fast and accurate denoising.

[0075] In one specific embodiment of the embodiment of the application, the anti-voice conversion watermark embedding network comprises:

[0076] The forward diffusion module is configured to randomly select a diffusion time step, inject Gaussian noise into the carrier voice according to a noise schedule, and generate noisy voice;

[0077] The feature tensor generation module is configured to perform scale-unified operation on the noisy voice, the log mel spectrum and the diffusion time step, and add the features in the channel to form a unified modulated feature tensor; in the specific implementation process, the feature tensor generation module comprises a double-layer spectrum up-sampler and a diffusion time step embedding device, the double-layer spectrum up-sampler is configured to up-sample the received log mel spectrum, and the diffusion time step embedding device is configured to map the diffusion time step into an embedding vector of the time step, the outputs of the two are added to the residual feature obtained by 1x1 convolution and ReLU processing of the noisy voice corresponding to the current diffusion time step in the channel dimension, to realize unified modulation of the diffusion time step, the mel spectrum and the noisy voice.

[0078] The residual feature extraction module is configured to perform residual feature extraction processing on the feature tensor to obtain predicted noise .

[0079] The reverse diffusion module is configured to perform reverse diffusion update on the noisy voice based on the predicted noise and the diffusion prior P(0, ), to generate the watermark-containing voice.

[0080] In the above scheme, the formula used in the reverse diffusion is:

[0081] = [ ]+ , ~ P( ),

[0082] In the formula, x t represents the noisy voice corresponding to the diffusion time step t, ​​​corresponding noisy speech, and denotes a noise scheduling parameter in the diffusion process, denotes a predicted noise obtained based on the noisy speech , diffusion time step and conditional information c, P is a Gaussian noise subject to a random sampling process, used to ensure diversity and random diffusion characteristics of the model.

[0083] In a specific embodiment of the embodiment of the present application, the residual feature extraction module includes a plurality of sequentially arranged dilated residual blocks, and the residual component output of a previous dilated residual block is input to a next dilated residual block; the Skip components of all dilated residual blocks are added and averaged, reduced in dimension by 1x1 convolution, and activated by ReLU to obtain the predicted noise.

[0084] In the above scheme, in the specific implementation process, the number of dilated residual blocks can be set to 30 or other values, and can be set according to actual needs. Each dilated residual block internally includes a 3x1 hollow convolution and a gated activation unit, and then the residual feature is mapped into a residual component and a Skip component by 1x1 channel projection.

[0085] In a specific embodiment of the embodiment of the present application, the mathematical expression of the total loss function is:

[0086] ,

[0087] In the formula, Ltotal is the total loss function, Ladversarial is an adversarial loss function between the speech conversion result of the carrier speech and the conversion result of the watermark-containing speech, Ldiffusion is a diffusion loss function between the predicted noise and the actual added noise, Lfreqweighted is a frequency weighted loss function of the watermark-containing speech and the carrier speech under multi-scale short-time Fourier transform, , , Ladversarial, Ldiffusion and Lfreqweighted are weights of the adversarial loss function, the diffusion loss function and the frequency weighted loss function, respectively.

[0088] In a specific embodiment of the embodiment of the present application, the diffusion loss function Ldiffusion between the predicted noise and the actual added noise is The mathematical expression of the diffusion loss function is:

[0089]

[0090] In the formula, and denote the actual injected Gaussian noise and the predicted noise, respectively, It is the inverse of the covariance matrix. Indicates time step , This represents conditional information, which is a log-Mel spectrum. Represents the carrier's speech. This indicates noisy speech. This indicates the diffusion time step t and the carrier speech. Actual injected Gaussian noise Take the expected value together. Represents a weighted norm 2;

[0091] In one specific embodiment of the present invention, the difference in waveform between the carrier speech and the watermarked speech is denoted as the disturbance. The waveform difference is divided into multiple sub-bands by short-time Fourier transform and transformed to the frequency domain to obtain the amplitude spectrum of the corresponding sub-bands. ;

[0092] Based on the sensing weights of each frequency band and the amplitude spectrum of the sub-band Generate a frequency-weighted loss function under single-scale short-time Fourier transform. ;

[0093] Based on the frequency-weighted loss functions under all single-scale short-time Fourier transforms, frequency-weighted loss functions for watermarked speech and carrier speech under multi-scale short-time Fourier transforms are generated. The frequency-weighted loss function under the single-scale short-time Fourier transform Frequency-weighted loss function under multi-scale short-time Fourier transform The mathematical expressions are as follows:

[0094] ,

[0095] ,

[0096] In the formula, For the first The center frequency of each frequency band The total number of frequency bands. ∈[0,1] is the first The perceptual weight of each frequency band is assigned; a higher weight value indicates that the human ear is more sensitive to that frequency band. Indicates the first The weighted weights corresponding to each scale Indicates the first Each scale.

[0097] In one specific embodiment of the present invention, the adversarial loss function between the speech conversion result of the carrier speech and the conversion result of the watermarked speech is... The mathematical expression is:

[0098] ,

[0099] In the formula, represents the mel spectrum generated after the carrier voice and the source voice input to the voice conversion model for voice conversion, represents the mel spectrum generated after the watermark-containing voice and the source voice input to the voice conversion model for voice conversion, represents the norm.

[0100] In the above scheme, in the specific implementation process, any one or more of the four voice conversion models can be selected, the four voice conversion models are: Adain-VC, AgainVC, TriAAN and Vqvc+, the conversion output sampling rate of the above four voice conversion models is 22kHz, the front end uniformly uses 1024-point STFT (short-time Fourier transform), and the received voice waveform is mapped into a predetermined size (such as 80x128) log mel spectrum. The above models all use official pre-training weights. In the training process, the carrier voice and the watermark-containing voice and the corresponding source voice are input into the above four models to complete voice conversion, so as to simulate a real voice conversion attack scene.

[0101] The diffusion model-based anti-voice conversion watermark embedding method in the embodiment of the application will be described in detail below in combination with a specific implementation process.

[0102] As shown in Figure 1 , the diffusion model-based anti-voice conversion watermark embedding method includes the following steps:

[0103] Step 1: reading historical carrier voice;

[0104] Step 2: initializing the anti-voice conversion watermark embedding network, the anti-voice conversion watermark embedding network (i.e. the watermark embedding network in Figure 2 ) needs to cooperate with the attack layer (i.e. the VC model in Figure 2 ) in the training process to realize the adjustment of the model parameters of the anti-voice conversion watermark embedding network, specifically the adjustment of the model parameters of the noise prediction network (i.e. the reverse diffusion network) in the anti-voice conversion watermark embedding network;

[0105] Step 3: setting the weights of the adversarial loss function between the voice conversion result of the carrier voice and the conversion result of the watermark-containing voice of the anti-voice conversion watermark embedding network, the diffusion loss function between the predicted noise and the actual added noise, and the frequency weighted loss function of the watermark-containing voice and the carrier voice under the multi-scale short-time Fourier transform;

[0106] Step 4: Read the source speech matched with the carrier speech, input into the attack layer for training of the anti-speech conversion watermark embedding network. The historical carrier speech and the corresponding source speech are taken as training data, and are divided into a training speech data set for training of the anti-speech conversion watermark embedding network and a test speech data set for testing of the anti-speech conversion watermark embedding network. The specific training process of the anti-speech conversion watermark embedding network is as shown in Figure 2 ;

[0107] Step 5: Calculate a total loss function including an adversarial loss function between the speech conversion result of the carrier speech and the conversion result of the watermark-containing speech, a diffusion loss function between the predicted noise and the actual added noise, and a frequency weighting loss function of the watermark-containing speech and the carrier speech under multi-scale short-time Fourier transform, and adjust and optimize the network parameters of the anti-speech conversion watermark embedding network with the total loss function minimum as the goal.

[0108] Step 6: Read the real-time carrier speech, input into the anti-speech conversion watermark embedding network which has been trained, and calculate and output the watermark-containing speech.

[0109] Further explanation of Step 2, details of the anti-speech conversion watermark embedding network and the attack layer:

[0110] 2-1: Details of the anti-speech conversion watermark embedding network are shown in Table 1 below, where b is the batch size:

[0111] Table 1 Network parameter detailed configuration of the anti-speech conversion watermark embedding network

[0112]

[0113] 2-2: The attack layer contains four speech conversion models (i.e. VC models in Figure 2 ): Adain-VC, AgainVC, TriAAN and Vqvc+, the conversion output sampling rate of the above four speech conversion models is 22kHz, the front end uniformly uses 1024-point STFT to map the received speech signal to 80x128 log-mel spectrum. In the prediction stage, the input data is cut according to a 128-frame window, and after the amplitude is normalized to the [-1,1] interval, it is sent to the respective encoder. The above models all use official pre-trained weights, and the parameters are frozen in the prediction stage. In the training process, the carrier speech and the watermark-containing speech, and the corresponding source speech are input into the above four models to complete speaker conversion, so as to simulate the real speech conversion attack scene, so that the anti-speech conversion watermark embedding network can adapt to these attacks and generate watermark-containing speech resistant to them.

[0114] In particular, the differentiable proxy model Again-VC is used to simulate the conversion process of the speech conversion model, and the specific network parameter configuration is shown in Table 2. Among them, each encoder convolution contains 2 3x1Conv, and each decoder convolution layer contains 4 3x1Conv. After the output of the decoder, the features are first passed through a double-layer GRU to capture long-range temporal dependencies, and then projected to the target channel number through a 1x1 linear projection layer. The output of the differentiable proxy model Again-VC is the converted mel spectrum of 1x80x128, and the speech waveform is obtained after MelGAN inverse transformation, with a size of 1x32768 and a sampling rate of 22kHz.

[0115] Table 2 Network parameter detailed configuration of the differentiable proxy model Again-VC

[0116]

[0117] Further, the specific training process in step 4 is further described:

[0118] 4-1: First read the 1x32768 single-channel carrier speech from the file list , the carrier speech is framed and windowed, and the amplitude spectrum of each frame of the framed and windowed carrier speech is obtained by short-time Fourier transform, the amplitude spectrum is mapped by a Mel filter bank and logarithm is taken to generate an 80x128 log mel spectrum, and the log mel spectrum is taken as the condition information c; frame by frame, the frame-level energy of the log mel spectrum is calculated to obtain a frame-level energy sequence, and based on the frame-level energy sequence, a diagonal structure covariance matrix is constructed in time sequence .

[0119] Further, the spectral flatness of the carrier speech is calculated :

[0120] ,

[0121] In the formula, is the spectral flatness, is the total number of frequency bands, is the spectral value of the th frequency band, is a positive number less than a set threshold, and in the specific implementation process, is a very small positive number.

[0122] Divide into three grades, when , use super-Gaussian noise as the diffusion prior, when , use standard Gaussian noise as the diffusion prior, and when At this time, the sub-Gaussian noise is used as the diffusion prior. Thus, a generalized diffusion prior P(0, ) is obtained, which depends on the input data. Such a design makes the forward diffusion process closer to the natural distribution of the input data, laying the foundation for subsequent fast and accurate denoising.

[0123] Subsequently, a time step t is randomly selected, and Gaussian noise is injected into the carrier speech according to the noise schedule (i.e., forward noise addition) to obtain the noisy speech . To align the conditions, the log-mel spectrum is up-sampled to 80x32768 by two-level transposed convolution, and the frame-level energy is repeated and expanded to a 1x32768 prior vector by 256 samples per step. The randomly selected diffusion time step t is first converted into a 128-dimensional vector by the sine position coding, and then mapped to a 512-dimensional time step embedding vector by two 1x1 fully connected layers (equivalent to 1x1Conv) in turn, and copied along the time domain to 512x32768. After the above two pieces of information are stacked in the channel dimension, they are added to the noisy speech which is expanded to 64x32768 by 1x1 convolution, to form a unified modulation tensor, which is then input into a 30-layer dilated residual block. The Skip component (i.e., the skip connection) output by each dilated residual block is summed and averaged in the channel, and then reduced to 1x32768 by 1x1 convolution and ReLU activation to obtain the predicted noise (i.e., noise estimation) at this step. Using the predicted noise and the diffusion prior P(0, ), the noisy speech is updated by reverse diffusion in turn. The reverse diffusion update operation reconstructs the denoised speech point by point in the time domain, and directly outputs the final watermark-containing speech with a size of 1x32768 . Finally, the waveform amplitude of the watermark-containing speech is clipped to [-1, 1].

[0124] 4-2: Attack phase, input the watermark-containing speech and the corresponding carrier speech into the pre-trained speech conversion model (Adain-VC, AgainVC, TriAAN, Vqvc+) as the target speech and the same source speech, respectively; both pairs of input speech have a size of 1x32768 and a sampling rate of 22kHz. The carrier speech and the source speech are converted to output the ideal conversion speech with a size of 80x128 mel spectrum; the watermark-containing speech and the source speech are converted to output the damaged conversion speech with a size of 80x128 mel spectrum.

[0125] Further explanation of the total loss function calculation in step 5:

[0126]

[0126] 5-1To improve the accuracy of the anti-voice conversion watermark embedding network in predicting noise in the denoising process, and thus improve the imperceptibility of the watermark-containing speech, a diffusion loss function between the predicted noise and the actual added noise is introduced in the embodiment :

[0127]

[0128] wherein, and represent the actual injected Gaussian noise and the predicted noise respectively, is the inverse matrix of the covariance matrix, represents the time step , represents the conditional information, which is the log mel spectrum, represents the carrier speech, represents the noisy speech, represents the mathematical expectation of the diffusion time step t, the carrier speech , the actual injected Gaussian noise , represents the weighted two-norm. The diffusion loss function trains the diffusion model to accurately predict the noise to be removed at each denoising step by minimizing the difference between the predicted noise and the actual injected Gaussian noise (i.e. the real noise), thereby improving the quality of the generated samples.

[0129] 5-2To make the watermark information concentrate in the area that is less sensitive to human ears, and thus improve the imperceptibility of the watermark-containing speech, a frequency weighting loss function under single-scale short-time Fourier transform is first introduced in the embodiment :

[0130]

[0131] wherein, is the center frequency of the th frequency band, ∈[0,1] is the perceptual weight of the th frequency band, and the higher the weight value, the less sensitive the human ear is to the frequency band, and the anti-voice conversion watermark embedding network will preferentially embed the watermark information in the frequency band. To avoid the problem of single emphasis caused by single-scale short-time Fourier transform, the embodiment proposes to calculate the frequency weighting loss of three different scales (corresponding to large scale, medium scale and small scale respectively), and to adaptively fuse each scale by introducing a lightweight ScaleAttention network to obtain the frequency weighting loss function under multi-scale short-time Fourier transform :

[0132] ​ ，

[0133] In the formula, Indicates the first The weighted weights corresponding to each scale Indicates the first Each scale.

[0134] 5-3 To enhance the anti-speech conversion capability of watermarked speech, this embodiment introduces an adversarial loss function between the speech conversion result of the carrier speech and the conversion result of the watermarked speech. :

[0135] ,

[0136] In the formula, in the formula, This represents the Mel spectrum generated after the carrier speech and source speech are input into the speech conversion model for speech conversion. This represents the Mel spectrum generated after the watermarked speech and the source speech are input into the speech conversion model for speech conversion. express The norm is used in this process, where the source speech for the paired carrier speech and watermarked speech is the same. This loss maximizes the difference between the conversion results, thereby improving the anti-speech conversion capability of the watermarked speech generated by this method.

[0137] 5-4 In summary, the total loss function This includes an adversarial loss function between the speech conversion result of the carrier speech and the conversion result of the watermarked speech. The diffusion loss function between predicted noise and actual added noise Frequency-weighted loss function of watermarked speech and carrier speech under multi-scale short-time Fourier transform To balance the imperceptibility of the watermark and the speech-to-speech resistance of the watermarked speech, different weights are assigned to the three elements, as follows:

[0138]

[0139] In the formula, For the total loss function, Let be the adversarial loss function between the speech conversion result of the carrier speech and the conversion result of the watermarked speech. To predict the diffusion loss function between the noise and the actual added noise, Let be the frequency-weighted loss function of the watermarked speech and the carrier speech under multi-scale short-time Fourier transform. , , These are the weights for the adversarial loss function, the diffusion loss function, and the frequency-weighted loss function, respectively.

[0140] Finally, the relevant parameters of the anti-voice conversion watermark embedding network are adjusted only with the Adam optimizer.

[0141] Further details of step 6 are described, and the results of invisibility and robustness against attacks are analyzed:

[0142] 6-1: The trained anti-voice conversion watermark embedding network is tested on a preset dataset to obtain SVA resist As an indicator of the ability of the watermark to resist voice conversion, the indicator represents the proportion of the number of voice pairs in the test dataset that are converted and carrier voices that are judged by the system to be the same speaker to the total number of voice pairs. The smaller the indicator, the better the anti-voice conversion effect. SVA quality As an indicator of the invisibility of the watermark, the indicator represents the proportion of the number of voice pairs in the test dataset that contain watermark voices and carrier voices that are judged to be the same speaker to the total number of voice pairs. The larger the indicator, the better the invisibility. Table 3 below shows the test results of the anti-voice conversion ability and invisibility of the watermark-containing speech generated by the method under different proxy models as attack layers. It can be seen that the method in the embodiment of the present application performs excellently in both aspects, which is mainly because: (1) the diffusion model is used in the embodiment of the present application to directly generate watermark-containing speech after embedding the watermark. The diffusion model is a generator with excellent generation effect, and its reconstruction result is closer to the natural distribution of the original speech, which alleviates the decline of invisibility caused by excessive energy of the watermark information, and the process is directly performed on the waveform graph, avoiding the quality loss that occurs in the process of reconstructing the waveform graph from the mel spectrum graph. (2) The frequency-weighted loss function used in the embodiment of the present application reduces the influence of the addition of the watermark on the invisibility of the watermark-containing speech by guiding the watermark to be added to the frequency band that is not sensitive to the human ear.

[0143] Table 3 Anti-voice conversion ability and invisibility of the generated watermark-containing speech

[0144]

[0145] 6-2: In order to test the robustness of the watermark-containing speech generated by the method in the embodiment of the present application, the anti-voice conversion ability of the watermark-containing speech in the scenarios of MP3 compression and Gaussian filtering is tested in the present embodiment. Among them, the MP3 compression is divided into two scenarios from 352 kbps to 128 kbps and from 352 kbps to 64 kbps. As shown in Table 4 below:

[0146] Table 4 Robustness of the method in the embodiment of the present application under different attack scenarios

[0147]

[0148] The robustness of the method in the embodiments of the present application under different attack scenarios is shown in Table 4. It can be seen that the watermarked speech still maintains good anti-voice conversion capability and imperceptibility in the face of attacks, which is mainly because most of these attack scenarios are to destroy the high frequency part, and the method reconstructs the watermarked speech through the diffusion model, which ensures that the watermark appears in each frequency band of the speech sample, and the anti-voice conversion capability is not greatly eliminated.

[0149] Embodiment 2

[0150] Based on the same inventive concept as Embodiment 1, the present embodiment provides a computer program product comprising computer programs / instructions which, when executed by a processor, implement the diffusion model-based anti-voice conversion watermark embedding method of any one of Embodiments 1.

[0151] Those skilled in the art will understand that the embodiments of the present application can be provided as a method, a system, or a computer program product. Therefore, the present application can take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects. Moreover, the present application can take the form of a computer program product implemented on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROMs, optical storage, etc.) containing computer-usable program code.

[0152] The present application is described with reference to flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to embodiments of the present application. It should be understood that each flow and / or block in the flowcharts and / or block diagrams, as well as combinations of flows and / or blocks in the flowcharts and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing apparatus to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing apparatus produce a device that implements the functions specified in the flowcharts and / or block diagrams. Figure 1 The functions specified in one or more flows and / or blocks. Figure 1 The functions specified in one or more flows and / or blocks.

[0153] These computer program instructions can also be stored in a computer-readable memory that can direct the computer or other programmable data processing apparatus to work in a specific manner, so that the instructions stored in the computer-readable memory produce a manufactured product including instruction devices that implement the functions specified in the flowcharts and / or block diagrams. Figure 1 The functions specified in one or more flows and / or blocks. Figure 1 The functions specified in one or more flows and / or blocks.

[0154] These computer program instructions can also be loaded into a computer or other programmable data processing devices, so that a series of operational steps are performed on the computer or other programmable data processing devices to generate a computer implemented process, so that the instructions executed on the computer or other programmable data processing devices provide a process for implementing the functions specified in the flowchart Figure 1 one flow or multiple flows and / or the functions specified in the flowchart Figure 1 one flow or multiple flows and / or the functions specified in the flowchart

[0155] The embodiments of the present application are described above with reference to the drawings, but the present application is not limited to the specific embodiments described above, and the specific embodiments described above are merely illustrative, but not restrictive, and those skilled in the art can make many forms under the inspiration of the present application without departing from the purpose of the present application and the scope protected by the claims, and these all belong to the protection of the present application.

[0156] The basic principles and main features of the present application and the advantages of the present application are shown and described above. Those skilled in the art should understand that the present application is not limited to the above described embodiments, and the above described embodiments and descriptions in the specification are only illustrative of the principles of the present application, and various changes and improvements can be made to the present application without departing from the spirit and scope of the present application, and these changes and improvements all fall within the scope of the present application. The scope of protection of the present application is defined by the appended claims and their equivalents.

Claims

1. A method for embedding an anti-voice conversion watermark based on a diffusion model, characterized in that, The method comprises the following steps: The real-time carrier voice input obtained is input into a trained diffusion model-based anti-voice conversion watermark embedding network to generate a watermark-containing voice; The training method of the diffusion model-based anti-voice conversion watermark embedding network comprises the following steps: The received historical carrier voice is preprocessed to generate a log Mel spectrum and a diffusion prior; The historical carrier voice, the log Mel spectrum and the diffusion prior are input into the diffusion model-based anti-voice conversion watermark embedding network to generate a watermark-containing voice; The model parameters of the anti-voice conversion watermark embedding network are updated constantly to minimize a total loss function, so as to complete the training of the anti-voice conversion watermark embedding network, wherein the total loss function comprises an adversarial loss function between the voice conversion result of the carrier voice and the conversion result of the watermark-containing voice, a diffusion loss function between the predicted noise and the actual added noise, and a frequency weighting loss function of the watermark-containing voice and the carrier voice under multi-scale short-time Fourier transform.

2. The method of claim 1, wherein, The diffusion prior is generated by the following method: The carrier voice is framed and windowed, and the amplitude spectrum of each frame of the framed and windowed carrier voice is obtained through short-time Fourier transform, the amplitude spectrum is mapped through a Mel filter bank and logarithm is taken to generate a corresponding log Mel spectrum, and the log Mel spectrum is also used as conditional information; The log Mel spectrum is frame by frame calculated for frame-level energy to obtain a frame-level energy sequence; A diagonal structure covariance matrix is constructed in time sequence based on the frame-level energy sequence; The spectral flatness of the carrier voice is calculated, and the Gaussian noise is determined according to the relationship between the spectral flatness and a preset threshold, and the diffusion prior is obtained in combination with the covariance matrix.

3. The method of claim 2, wherein, The calculation formula of the spectral flatness is as follows: In the formula, SFM is the spectral flatness, M is the total number of frequency bands, S[f] is the spectral value of the fth frequency band, ε is a positive number less than the set threshold, and ln() represents the logarithm function with e as the base; When SFM>0.7, super-Gaussian noise is selected; When 0.4<SFM<0.7, standard Gaussian noise is selected; When SFM<0.4, sub-Gaussian noise is selected.

4. The method of claim 1, wherein, The anti-voice conversion watermark embedding network comprises: A forward diffusion module for randomly selecting a diffusion time step, injecting Gaussian noise into the carrier voice according to a noise schedule to generate a noisy voice; A feature tensor generation module for performing scale unification operation on the noisy voice, the log Mel spectrum and the diffusion time step, and adding the feature channels to form a unified modulated feature tensor; A residual feature extraction module for performing residual feature extraction processing on the feature tensor to obtain predicted noise; An inverse diffusion module for performing inverse diffusion update on the noisy voice based on the predicted noise and the diffusion prior to generate a watermark-containing voice.

5. The diffusion model based speech conversion attack resistant watermark embedding method of claim 4, wherein: The residual feature extraction module comprises a plurality of sequentially arranged dilated residual blocks, and the residual component output of a previous dilated residual block is input into a next dilated residual block; the Skip components of all dilated residual blocks are added and averaged, reduced in dimension through 1×1 convolution, and activated through ReLU to obtain the predicted noise.

6. The method of claim 1, wherein the method is characterized by: The mathematical expression of the total loss function is as follows: L = λ1L defense + λ2L diffusion + λ3L auditory , In the formula, L is a total loss function, L defense is an adversarial loss function between the speech conversion result of the carrier speech and the conversion result of the watermark-containing speech, L diffusion is a diffusion loss function between the predicted noise and the actual added noise, L auditory is a frequency weighting loss function of the watermark-containing speech and the carrier speech under multi-scale short-time Fourier transform, and λ1, λ2 and λ3 are weights of the adversarial loss function, the diffusion loss function and the frequency weighting loss function, respectively.

7. The method of claim 6, wherein the method further comprises: determining a diffusion model based on the diffusion model parameters; and embedding the watermark into the audio signal based on the diffusion model. a diffusion loss function L between the predicted noise and the actual added noise diffusion The mathematical expression is: where ε t and respectively represent the actually injected Gaussian noise and the predicted noise, is the inverse matrix of the covariance matrix, t represents a time step, c represents conditional information, the conditional information is a log mel spectrum, x0 represents a carrier voice, x t represents a noisy voice, represents the actually injected Gaussian noise ε t is taken together, represents a weighted two-norm.

8. The method of claim 6, wherein the method further comprises: determining a diffusion model based on the diffusion model parameters; and embedding the watermark into the audio signal based on the diffusion model. The difference between the carrier voice and the waveform of the voice containing the watermark is denoted as disturbance Δ, the waveform difference is divided into multiple sub-bands and converted to the frequency domain by short-time Fourier transform, and the corresponding sub-band amplitude spectrum is obtained Per-band perceptual weight and sub-band amplitude spectrum Generating a frequency weighted loss function L under a single-scale short-time Fourier transform percept ; Based on the frequency weighted loss function under all single-scale short-time Fourier transforms, the frequency weighted loss function L of the watermarked speech and the carrier speech under multi-scale short-time Fourier transforms is generated auditory The mathematical expressions of the frequency weighted loss function L percept under single-scale short-time Fourier transforms and the frequency weighted loss function L auditory under multi-scale short-time Fourier transforms are respectively where f k is the center frequency of the kth frequency band, K is the total number of frequency bands, W(f k )∈[0,1] is the perceptual weight of the kth frequency band, the higher the weight value indicates the more sensitive the human ear is to the frequency band, ω m represents the weighted weight corresponding to the mth scale, and m represents the mth scale.

9. The method of claim 6, wherein the method further comprises: determining a diffusion model based on the diffusion model of the audio signal; and embedding the watermark into the audio signal based on the diffusion model. The adversarial loss function L between the speech conversion result of the carrier voice and the conversion result of the watermark-containing voice defense The mathematical expression is: L defense = -‖S vc -S vc’ ‖2, In the formula, S vc represents the mel spectrum generated after the carrier voice and the source voice input to the voice conversion model are subjected to voice conversion, S vc’ represents the mel spectrum generated after the watermarked voice and the source voice input to the voice conversion model are subjected to voice conversion, and ||·||2 represents an L2 norm.

10. A computer program product, characterised in that, Computer program / instructions for implementing the diffusion model based speech conversion attack resistant watermark embedding method according to any one of claims 1-9 when executed by a processor.

Images