A privacy-preserving deep speech digital watermark streaming embedding and detection method

Abstract

The present invention discloses a privacy-protected deep voice digital watermark streaming embedding and detection method. Based on the end-cloud collaborative processing environment, it effectively isolates user sensitive information and watermark information in voice content, and simultaneously achieves beneficial effects such as streaming embedding, interval labeling, watermark forgery prevention, and channel robustness. It supports scenarios such as voice content copyright declaration and forged sample identification, and serves third-party watermark embedding and detection applications. During the entire watermark embedding stage, the server only generates general watermark data based on the watermark information, thereby avoiding the risk of leakage of user privacy information. In the detection link, the receiving end user end uses a third deep neural network to desensitize the audio content, so that the server end can only infer the existence of the watermark through the processed desensitized data, isolating the direct contact between the user's original voice data and the server end, significantly improving the user's privacy protection level, and enhancing the sense of trust between the user and the third-party detection platform.

Description

Technical Field

[0001] The present invention belongs to the technical field of speech information processing, and relates to a speech digital watermark embedding and extraction technology, and specifically, to a privacy-protected deep speech digital watermark streaming embedding and detection method. Background Art

[0002] Speech digital watermarking technology is an effective means to verify the authenticity of the source of speech content, and is used in the fields of audio copyright protection, covert content communication, and synthetic content detection. Especially with the rapid development of generative AI technology, it is difficult to distinguish the authenticity of speech synthesis content, which has brought serious negative impacts to society. Digital watermarking technology can provide robust active identification for speech content without affecting the hearing of speech, and provide strong technical support for the unified filing and supervision of AI-generated content.

[0003] When third-party voice digital watermarking service providers provide digital watermark embedding and detection services, there are the following limitations: (1) Risk of privacy leakage: The digital watermarking service process requires the transmission of complete voice files, which has the risk of leakage of sensitive information such as user privacy (such as passwords), reducing the user's trust in the service provider. (2) Weak robustness: Watermarking methods under traditional digital signal processing technology, such as the least significant bit (LSB) and quantization index modulation (QIM), have weak ability to resist noise and malicious attacks during transmission, and cannot effectively guarantee the integrity of watermark information after experiencing lossy channels. (3) Limitation of streaming application scenarios: Existing digital watermarking technology is mainly suitable for processing static audio files, and it is difficult to directly apply it to dynamic audio streams in real-time communication environments (such as live broadcasts, telephone conferences, and network conferences). In view of this, the present invention aims to overcome the above technical bottlenecks and develop a more robust and widely applicable voice digital watermarking method. Summary of the invention

[0004] This application provides a privacy-protected deep voice digital watermark streaming embedding and detection method, which enables a third-party service provider to embed a hidden voice digital watermark in a voice signal without touching the original voice data. At the same time, the robustness of the digital watermark is enhanced based on deep neural networks and data enhancement technology, and the application scope of traditional voice digital watermark technology is expanded to real-time streaming communication scenarios. The present invention is implemented through the following technical solutions:

[0005] The present invention discloses a privacy-preserving deep speech digital watermark streaming embedding and detection method, which is implemented by the following devices:

[0006] Server side: A computing device for generating a universal voice digital watermark that is independent of the voice input and detecting the watermark from the user's desensitized data;

[0007] Transmitter user end: a computing device used to inject a watermark into the user's voice input according to the universal voice watermark to produce watermarked voice data;

[0008] Receiving end user end: a computing device used to desensitize the watermarked speech data and obtain user desensitized data retaining the watermark information;

[0009] Communication network: provides Internet communication capabilities between the server side, the sending user side, and the receiving user side.

[0010] Methods include:

[0011] The sending user terminal specifies the fixed length to be embedded as Binary watermark information and sending the information to the server via a communication network;

[0012] The server receives watermark information After that, the watermark information Encode it and convert it into a fixed length through the first deep neural network model Universal audio watermark data , sent to the sending user terminal through the communication network;

[0013] The sending user terminal receives the universal audio watermark data , establish a voice call and obtain user voice data , using a lightweight second deep neural network model for universal audio watermark data Adaptively adjust and embed into a specific interval of the user's voice data to obtain watermarked voice data , sent to a receiving user terminal through a communication network;

[0014] The receiving user terminal receives the watermarked voice data , using the third deep neural network model to watermark the speech data Perform data desensitization to obtain user desensitized data , sent to the server through the communication network;

[0015] The server receives user desensitized data , it is converted into a watermark detection result at the sampling point level through the fourth deep neural network model, and the digital watermark information is parsed by calculating the average value of the watermark detection result and sent to the receiving user terminal through the communication network.

[0016] As a further improvement, the watermark information of the present invention Encode it and convert it into a fixed length through the first deep neural network model Universal audio watermark data , specifically:

[0017] Binary watermark information Through a trainable embedding matrix , generate watermark embedding vector , watermark embedding vector Generated by the following formula:

[0018]

[0019] Then embed the watermark into the vector Repeat on timeline After that, it is input into the first deep neural network model to generate binary watermark information for the universal watermark data. Universal watermark data .

[0020] As a further improvement, the receiving universal audio watermark data of the present invention , establish a voice call and obtain user voice data , using a lightweight second deep neural network model for universal audio watermark data Adaptively adjust and embed into a specific interval of the user's voice data to obtain watermarked voice data , specifically: support watermark injection for static speech input once and streaming speech input continuously;

[0021] When the input voice is pre-recorded static voice data When the watermark is embedded in a random manner, a watermark is generated with Blank speech data of equal length, randomly selected from the blank speech data Inject the universal digital watermark into non-overlapping watermark injection intervals to obtain , is the length of the input speech. , and then and Enter the adaptive function Generate an adaptation to specific voice input Adaptive digital watermarking ,Will Directly superimposed on On the basis of the above, we get the watermarked speech data, that is, , is the weight coefficient; adaptive function In order to fuse two time-aligned one-dimensional vector data, a convolutional neural network or a recurrent neural network or a cross attention mechanism is used; when a watermark needs to be embedded in a specified position of the speech, the starting position of the random insertion is modified to the specific position where the watermark needs to be inserted;

[0022] When the input speech is a continuously input streaming speech, the watermark is injected at the required time interval, and the adaptive function It adopts an autoregressive neural network structure that supports streaming cache context, which is a causal convolutional neural network or a recurrent neural network or a cross-attention mechanism with mask and KV cache.

[0023] As a further improvement, the present invention uses the third deep neural network model to watermark the speech data. Perform data desensitization to obtain user desensitized data , specifically:

[0024] The third deep neural network model consists of three parts: encoder, decoder and residual vector quantizer (RVQ). After the speech passes through the encoder, it is mapped to n codebooks through RVQ and then restored through the decoder. Among the n codes, the first code is the highly relevant content, and 2 to n codes are taken as the user desensitized data source.

[0025] As a further improvement, the present invention converts it into a watermark detection result at the sampling point level through the fourth deep neural network model, calculates the average value of the watermark detection result, and parses the digital watermark information, specifically:

[0026] The desensitized watermarked speech data is input into the low-quadrant deep neural network model, and the model outputs the Dimensional discrimination results;

[0027] The final result is determined by averaging the results. If the mean of a dimension is greater than the set threshold, it means that the bit information is 1, otherwise it is 0. The 0th dimension indicates whether there is a watermark, and the The B dimension is the specific value of each bit of the watermark information. If and only if the 0th dimension is 1, the B dimension information is valid.

[0028] As a further improvement, the transmitting user terminal of the present invention specifies a fixed length to be embedded as Binary watermark information Before sending the information to the server through the communication network, the following steps are also included:

[0029] Train the first, second, third, and fourth deep neural network models:

[0030] The third deep neural network model is pre-trained on a large-scale speech dataset alone, and its parameters are kept frozen during the joint training with the first, second, and fourth deep neural network models;

[0031] The first, second, and fourth deep neural network models are trained using an end-to-end method. The training process uses multi-scale reconstruction loss and detection result consistency loss, and the input of the third deep neural network is actively interfered with through artificial channel simulation technology to enhance the robustness of the watermark.

[0032] As a further improvement, the multi-scale reconstruction loss described in the present invention is:

[0033]

[0034] in, Indicates the operation of extracting Mel spectrum from speech, and the sliding window length is , the window sliding step size is , the Mel filter dimension is set to 40, express norm;

[0035] The consistency loss of the detection result is to constrain the consistency of the detection result through the binary cross loss function (BCE) to ensure that the generated watermarked speech data can correctly identify the watermark information during detection.

[0036]

[0037] in, , represents the detection result at the sampling point level, Indicates the timestamp of the sampling point, Indicates the data position in the watermark information, Indicates that , it takes 0, otherwise it takes 1.

[0038] The present invention proposes a method for deep voice digital watermark streaming embedding and detection with user privacy protection. Based on the end-cloud collaborative processing environment, it effectively isolates user sensitive information and watermark information in voice content, and simultaneously achieves beneficial effects such as streaming embedding, interval labeling, watermark forgery prevention, and channel robustness. It supports scenarios such as voice content copyright declaration and forged sample identification, and serves third-party watermark embedding and detection applications. Specifically, it has the following beneficial effects:

[0039] (1) User privacy protection: The present invention provides a method for completing watermark embedding and detection without uploading user-sensitive voice data. During the entire watermark embedding stage, the server side only generates general watermark data based on the watermark information, and leaves the final watermark embedding operation to the sending user side to complete independently, thereby avoiding the risk of leakage of user privacy information. In the detection stage, the receiving user side uses RVQ technology (the third deep neural network) to desensitize the audio content, so that the server side can only infer the existence of the watermark through the processed desensitized data, effectively isolating the user's original voice data from direct contact with the server side. Therefore, while ensuring the high efficiency of watermark embedding and detection, the present invention significantly improves the level of user privacy protection and enhances the sense of trust between users and third-party detection platforms.

[0040] (2) Real-time streaming watermark embedding: The present invention sets up “using a lightweight second deep neural network model to embed universal audio watermark data Adaptively adjust and embed it into a specific interval of the user's voice data". The server side bears more computationally complex universal watermark data generation, and the sending end user only needs to run a lightweight second deep neural network to embed universal watermark data into the voice data, which significantly reduces the user-side computing overhead and achieves real-time performance on low-resource devices. The real-time rate is 0.04 on Apple Macbook Air M2 and 0.13 on Thinkpad X13. At the same time, this solution sets "When the input voice is a continuously input streaming voice, the watermark is embedded at the required time interval, and the adaptive function Adopting an autoregressive neural network structure that supports streaming cache context, it can realize streaming reasoning on mobile notebook platforms such as Apple Macbook Air M2 and Thinkpad X13. The overall first protection delay is only 103.5ms, providing efficient voice copyright protection for scenarios such as live broadcast and telephone communication.

[0041] (3) Watermark information anti-counterfeiting: The present invention sets up a method of "using a lightweight second deep neural network model to detect general audio watermark data In the process of embedding watermarks, the general audio watermark data is adaptively adjusted and embedded into a specific interval of the user's voice data through the second deep neural network model, and it is accurately embedded into a specific interval of the user's voice data, instead of directly using the general watermark data generated by the server. This not only realizes the adaptation of the watermark content to the specific voice, enhances the robustness and concealment of the watermark, but also effectively prevents the general watermark data from being illegally obtained and used to forge watermarked voice, greatly improving the security of the system.

[0042] (4) Interval identification and location detection: The present invention sets up a "server-side receiving user desensitized data , the fourth deep neural network model is used to convert it into a watermark detection result at the sampling point level, and the digital watermark information is parsed by calculating the average value of the watermark detection result. The fourth deep neural network realizes watermark detection at the sampling point level, and the watermark information of all sampling points can be obtained by one reasoning, which eliminates the traditional watermark interval search step, greatly improves the detection efficiency and speed, and ensures the fast and accurate positioning of the watermark information. By randomly inserting 1s watermark data on the LibriSpeech dataset, the measured interval positioning error is only 5.8%.

[0043] (5) Taking into account both the effectiveness and concealment of the watermark: The present invention is provided with "the first, second and fourth deep neural network models are trained using an end-to-end method, and the training process uses a multi-scale reconstruction loss and a detection result consistency loss". The watermark consistency loss function and the multi-scale reconstruction loss function ensure that the detection result is consistent with the embedded watermark information, and at the same time make the speech after the watermark is embedded consistent with the original speech in terms of auditory perception, taking into account both the effectiveness and concealment of the watermark. The measured watermark detection accuracy on the LibriSpeech dataset reached 99.7%, and at the same time, it scored 4.3 points in the subjective MOS test of 30 people on sound quality, indicating that the watermark method proposed in this invention can achieve both high effectiveness and high concealment.

[0044] (6) Anti-channel interference: The present invention sets up "active interference on the input of the third deep neural network through artificial channel simulation technology to achieve watermark robustness enhancement". In the end-to-end joint training process of the first, second and fourth deep neural network models, the present invention introduces channel enhancement measures to enhance the model's robustness to channel interference by simulating common artificial channel damage. In the case of high-reverberation noisy physical channel transmission on the LibriSpeech dataset, the watermark detection accuracy is still as high as 94.8%, indicating that the watermark can effectively resist channel interference. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] Figure 1 An application environment diagram of the digital watermarking method provided in an embodiment of the present application;

[0046] Figure 2 A schematic diagram of the digital watermarking method flow provided in the embodiment of the present application;

[0047] Figure 3 A schematic diagram of the structure of the digital watermarking method provided in the embodiment of the present application;

[0048] Figure 4A schematic diagram of the structure of a third deep neural network of the digital watermarking method provided in an embodiment of the present application. DETAILED DESCRIPTION

[0049] In order to make the purpose, technical solutions and advantages of this application clearer, the technical solutions in this application will be clearly and completely described below in conjunction with the drawings in this application. Obviously, the described embodiments are part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0050] The speech digital watermarking method provided in the embodiment of the present application can be applied to Figure 1 In the application environment shown.

[0051] Among them, the sending end user terminal and the receiving end user terminal communicate with the server or other terminals through the network. For example, the sending end user terminal, as before sending the voice signal to the communication network, embeds a digital watermark into the voice data by a digital watermark method, and sends the watermarked voice data embedded with the digital watermark to the communication network. The watermark embedding process includes end-cloud collaborative processing with the server. The sending end user terminal first sends the binary watermark information to be embedded to the server. The server generates universal watermark data based on the watermark information and returns the universal watermark data to the sending end user terminal. The sending end user terminal embeds the universal watermark data into the voice data to obtain the watermarked voice data. The receiving end user terminal obtains the user watermarked voice data from the communication network and detects the watermark information therefrom. This is because the detection process includes end-cloud collaborative processing with the server. The receiving end user terminal first desensitizes the data and then sends the desensitized data to the server. The server performs watermark detection on the desensitized data and then returns the detection result to the receiving end user terminal.

[0052] The transmitting user terminal and the receiving user terminal may be, but are not limited to, various personal computers, laptops, smart phones, tablet computers, IoT devices and portable wearable devices. IoT devices may be smart speakers, smart TVs, smart air conditioners, smart car-mounted devices, etc. Portable wearable devices may be smart watches, smart bracelets, head-mounted devices, etc.

[0053] Among them, the communication network is not absolutely safe and reliable. There may be transmission losses, which may cause damage to the watermark information, such as compression, packet loss, etc. There may also be information leakage, resulting in the leakage of sensitive information.

[0054] In one embodiment, a privacy-preserving deep speech digital watermark streaming embedding and detection method is provided, and the method flow is as follows: Figure 2As shown, the schematic block diagram of the method is shown in Figure 3. The speech synthesis method includes:

[0055] Step 1: The sending user terminal specifies the fixed length to be embedded as Binary watermark information and sends the information to the server through the communication network.

[0056] The binary watermark information is 01 digital string, number of digits Determines the capacity of the digital watermark, that is, the total For example, in the user identity application scenario, at most For example, in the audio content copyright identification application scenario, at most Embedded information, for example, in the synthetic speech identification application scenario, can represent at most A speech synthesis algorithm. The specific size of is related to the model capacity of the first and fourth deep neural networks and the length of the universal digital watermark. For example, in the implementation case described in this application, .

[0057] Preferably, when In this case, the detection result only indicates whether the audio carries watermark information, but does not contain the specific watermark content.

[0058] Step 2: After receiving the watermark information, the server encodes the watermark information and converts it into a fixed length of Universal audio watermark data , sent to the sending user terminal through the communication network, corresponding to Figure 3 The watermark embedding process is 1.

[0059] The universal audio watermark data is characterized in the form of an audio signal and has a wide range of applications, not limited to a single specific voice signal. The method can achieve effective embedding of diversified voice data through the implementation of the subsequent step three. Compared with the existing voice digital watermarking technology, it only embeds watermarks for specific input voice signals. Although it ensures the embedding accuracy and concealment, it has obvious limitations. Specifically, the embedding process requires the complete acquisition of the voice content, which is prone to data leakage risks; and each embedding operation requires the watermark to be recalculated, resulting in high computational costs. The universal voice digital watermarking technology proposed in this application, through deep training on large-scale voice data sets, gives the watermark embedding process universality and reduces dependence on specific voice input. This technological innovation not only achieves a breakthrough in the watermark generation stage without relying on specific voice content, but also significantly reduces the computational burden of repeated watermark generation, thereby improving the overall watermark processing efficiency and security.

[0060] The watermark information is encoded and converted into a fixed length by a first deep neural network model. Universal audio watermark data ,correspond Figure 3 The watermark embedding process 1 includes:

[0061] Binary watermark information Through a trainable embedding matrix , generate watermark embedding vector , watermark embedding vector Generated by the following formula:

[0062]

[0063] In this way, Contains the location information of each watermark code (through ) and specific content information (via ) are embedded.

[0064] Next, embed the watermark into the vector Repeat on timeline times, thus obtaining an activation map .

[0065] The activation map A is input into the first deep neural network model to generate binary watermark information for the universal watermark data. Universal watermark data .

[0066] The first deep neural network model has the following structure: the first layer is a 7×7 one-dimensional transposed convolution layer equipped with 128 filters to perform preliminary abstraction and processing of the input signal. Next, the network structure contains four transposed convolution blocks, each of which is designed to contain a residual unit with a skip connection and an upsampling layer. The upsampling layer uses a step size of The transposed convolution of the convolution kernel size is , to achieve the gradual restoration of the signal scale. In these four transposed convolution blocks, the step size The configurations are 8, 5, 4, and 2 in descending order, thereby reconstructing the signal to its original scale. At the end of the network is a 7×7 one-dimensional transposed convolution layer consisting of 32 filters, which is responsible for decoding the processed signal into a speech signal containing watermark information. In order to further increase the watermark capacity, the model generation capability is improved by increasing the number of transposed convolution blocks and increasing the number of model parameters, while T is used to increase the signal load capacity.

[0067] Step 3: The sending user terminal receives the watermark data , establish a voice call and obtain user voice data , use the lightweight second deep neural network model to adaptively adjust the audio watermark data and embed it into a specific interval of the user's voice data to obtain the watermarked voice data , sent to the receiving user terminal through the communication network, corresponding to Figure 3 There are 2 processes for watermark embedding.

[0068] The user voice data may be recorded in real time through a microphone device, or may use a recording prepared in advance.

[0069] When the input voice is complete voice data recorded in advance When , the watermark is embedded in a random insertion manner. First, generate a Blank speech data of equal length. Randomly select The non-overlapping watermark embedding intervals are embedded with the universal digital watermark to obtain , is the length of the input speech. Then, and Enter the adaptive function Generate an adaptation to specific voice input Adaptive digital watermarking , and then Directly superimposed on On the basis of the above, we get the watermarked speech data, that is, , is the weight coefficient. The adaptive function can adopt a variety of different deep neural network model structures, such as convolutional neural network, recurrent neural network, cross attention mechanism, etc., which is characterized by being able to fuse two time-series aligned one-dimensional vector data. When it is necessary to embed a watermark at a specified position of the speech, the starting position of the random insertion can be modified to the specific position where the watermark needs to be inserted.

[0070] When the input speech is a continuously input speech stream, the speech data is continuously input and the watermark is embedded at a certain time interval. Adopt an autoregressive neural network structure that supports streaming cache context, such as causal convolutional neural network, recurrent neural network, cross attention mechanism with mask and KV cache, etc. First, generate a sufficiently long , and divide it according to the length of the input streaming speech block to obtain Given Speech input segment ,Will and Input Stream Adaptive Function Obtained and .Will and After superposition, the streaming watermarked voice data to be output is obtained Finally, Send to the data output stream buffer.

[0071] Step 4: The receiving user terminal receives the watermarked voice data , use the third deep neural network model to desensitize the watermarked voice data and obtain user desensitized data , sent to the server through the communication network, corresponding to Figure 3 Watermark detection 1 process.

[0072] The user desensitized data is a feature map that is aligned with the watermark data in time sequence.

[0073] The third deep neural network model includes three parts: encoder, decoder and RVQ quantizer. Figure 4As shown in Figure 1. The encoder model first contains a one-dimensional convolutional layer with 32 channels and a kernel size of 7, followed by four convolutional blocks, each of which consists of a residual unit and a downsampling layer consisting of a strided convolution. The residual unit has two convolutional layers with a kernel size of 3 and a skip connection. The kernel size of the downsampling layer is twice the stride, and the number of channels doubles with each downsampling. The convolutional block is followed by a two-layer LSTM for sequence modeling, followed by a final convolutional layer with D output channels and a kernel size of 7. The entire network uses the ELU activation function and applies layer normalization or weight normalization. The decoder is a mirror image of the encoder, using transposed convolutions instead of strided convolutions for upsampling, with the order of strides opposite to that of the encoder, and finally outputs mono or multi-channel audio. There are two variants of this model, for high-fidelity non-streaming and low-latency streaming modes, respectively. RVQ refines the quantization process by computing the quantized residual and further quantizing the residual using an additional codebook. During training, the codebook entries are updated using an exponential moving average, and the encoder gradient is calculated using a straight-through estimator. A commitment loss is added to the overall training loss, which consists of the mean squared error between the quantizer input and output. The model uses 8 codebooks with 1024 entries, each with 10 bits. The continuous latent variables [B, D, T] output by the encoder are converted into a discrete index set [B, Nq, T] and converted back to a vector by summing the corresponding codebook entries, which is then fed into the decoder. The third deep neural encoder is pre-trained on a large-scale dataset, including but not limited to open source datasets such as DNSChallenge4, Common Voice, AudioSet, FSD50K, and Jamendo dataset.

[0074] After the watermarked speech data passes through the third deep neural network, a time-series-related code with a dimension of 8 is obtained. Among them, the first code is highly related to the semantic information, and the second to eighth codes are taken as the user desensitized data source. It is only necessary to transmit the codebook sequence number corresponding to the code to the server, which significantly reduces the amount of information transmission.

[0075] Preferably, in scenarios with higher privacy requirements, a convolutional neural network layer can be used to further compress the 2nd to 8th codes into a 512-dimensional continuous vector to achieve further desensitization.

[0076] Step 5: The server receives the user's desensitized data , through the fourth deep neural network model, it is converted into a universal sampling point level watermark detection result, by calculating the average value of the watermark detection result, parsing the digital watermark information, and sending it to the receiving user terminal through the communication network, corresponding to Figure 3 Watermark detection 2 process.

[0077] The watermark detection result at the sampling point level means that for each sampling point, the detection model will output a The detection result of the dimension is: the first dimension indicates whether there is watermark information, and the remaining B dimensions indicate the specific watermark information corresponding to each binary bit. If and only if the first dimension is 1, the remaining B dimensions are valid. The detection result can also identify the starting embedding position of the watermark, which is used to support scenarios such as deep synthesis content positioning.

[0078] The desensitized watermarked speech data is input into the discriminator network, and the discriminator outputs a The final result is determined by averaging the results. If the mean of a dimension is greater than 0.5, it means that the information of that bit is 1, otherwise it is 0. The 0th dimension indicates whether there is a watermark, and the The B dimension is the specific value of each bit of the watermark information. If and only if the 0th dimension is 1, the B dimension information is valid.

[0079] In one implementation, the basic structure of the discriminator model is symmetrical to the first deep neural network model structure. In these four transposed convolution blocks, the step size The configurations are 2, 4, 5, and 8. At the end of the entire model, set the channel to A 1D convolutional layer followed by a softmax layer is obtained. logit values .

[0080] The first to fourth deep neural networks are trained using the following method:

[0081] The third deep neural network model is pre-trained on a large-scale speech dataset, including but not limited to open source datasets such as DNS Challenge4, Common Voice, AudioSet, FSD50K, and Jamendo dataset. The training process uses reconstruction loss, perceptual (adversarial) loss, and quantization loss (commitment loss).

[0082] The first, second and fourth deep neural network models are trained using an end-to-end method, and the loss functions used in the training process include:

[0083] Multi-scale reconstruction loss, by penalizing the Mel spectrum of different window lengths between the watermarked speech data and the original speech data Norm distance, constraining the difference between watermarked speech data and original speech data in human hearing perception,

[0084]

[0085] in, is a 64-dimensional Mel spectrum, and its STFT window size is , the step size (hopsize) is ,e={5,…,11}, and Respectively Norm and Norm.

[0086] The consistency loss of the detection results is constrained by the binary cross-over loss function (BCE) to ensure that the generated watermarked speech data can correctly identify the watermark information during detection.

[0087]

[0088] in, , represents the detection result at the sampling point level, Indicates the timestamp of the sampling point, Indicates the data position in the watermark information, Indicates that , it takes 0, otherwise it takes 1.

[0089] Before the watermarked speech data is input into the third deep neural network model, the watermarked speech data is subjected to various forms of artificial interference to simulate channel interference to achieve data enhancement. The channel interference method includes but is not limited to the following signal processing algorithm examples:

[0090] (1) Downsampling: Degrading audio quality by reducing the sampling rate of an audio file, which helps simulate signal loss due to bandwidth limitations or device limitations.

[0091] (2) Filtering:

[0092] Low-pass filter: can be used to remove high-frequency noise or simulate low-frequency channel environments, such as old telephone lines or low-quality recording equipment.

[0093] Bandpass filter: retains signal components within a certain frequency range, which can simulate communication systems operating within a specific frequency range.

[0094] High-pass filter: used to remove signal components below a certain threshold frequency, which can simulate the audio transmission characteristics in certain specific scenarios.

[0095] (3) Quantization: By adjusting the number of quantization bits of the audio signal, the sound distortion caused by the loss of precision in the digitization process can be simulated.

[0096] (4) Reverberation: Adding artificial reverberation effects can simulate environments with different room sizes or acoustic characteristics, which is very important for training models to recognize speech recorded in different environments.

[0097] (5) Adding white (pink) noise: Adding randomly distributed white noise or pink noise to the original signal can simulate the presence of background noise. This is a common way to improve the model's noise resistance.

[0098] (6) Clipping (partial zeroing): Selectively remove part of the signal, that is, by setting the value of part of the audio clip to zero to simulate signal loss or device failure.

[0099] (7) Smoothing: By smoothing the signal, high-frequency fluctuations can be reduced, which may simulate certain types of signal processing.

[0100] (8) Reverberation: Reverberation is added to the signal through room channel pulse simulation technology to simulate the propagation in real space.

[0101] The above channel interference method is applied to the watermarked audio data in a random manner. First, 0 to 3 interference methods are randomly selected, and then for each channel interference method, parameters are randomly selected within a reasonable range.

[0102] The above is not a limitation of the present invention. It should be pointed out that for ordinary technicians in this technical field, several changes, modifications, additions or substitutions can be made without departing from the essential scope of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. A privacy-preserving deep speech digital watermark streaming embedding and detection method, implemented by the following devices: Server side: A computing device for generating a universal voice digital watermark that is independent of the voice input and detecting the watermark from the user's desensitized data; Transmitter user end: a computing device used to inject a watermark into the user's voice input according to the universal voice watermark to produce watermarked voice data; Receiving end user end: a computing device used to desensitize the watermarked speech data and obtain user desensitized data retaining the watermark information; Communication network: provides Internet communication capabilities between the server, the sending user, and the receiving user; The method includes: The sending user terminal specifies the binary watermark information m∈{0, 1} of fixed length B to be embedded B and sending the information to the server via a communication network; The server receives watermark information m∈{0,1} B After that, the watermark information m∈{0,1} B Encode it and convert it into a fixed length T universal audio watermark data δ through the first deep neural network model m ∈[-1, 1] 1×T , sent to the sending user terminal through the communication network; The sending end user terminal receives the universal audio watermark dataδ m ∈[-1, 1] 1×T , establish a voice call, obtain the user voice data x, and use the lightweight second deep neural network model to watermark the universal audio data δ m ∈[-1, 1] 1×T Adaptively adjust and embed into a specific interval of the user's voice data to obtain watermarked voice data y, and send it to the receiving user terminal through the communication network; The receiving end user terminal receives the watermarked voice data y, uses the third deep neural network model to perform data desensitization on the watermarked voice data y, obtains user desensitized data z, and sends it to the server end through the communication network; The server receives the user's desensitized data z, converts it into a watermark detection result at the sampling point level through the fourth deep neural network model, calculates the average value of the watermark detection result, parses the digital watermark information, and sends it to the receiving user terminal through the communication network.

2. The method according to claim 1, characterized in that The watermark information m∈{0,1} B Encode it and convert it into a fixed length T universal audio watermark data δ through the first deep neural network model m ∈[-1, 1] 1×T , specifically: The binary watermark information m∈{0,1} B Through a trainable embedding matrix E∈R 2B×H , generate the watermark embedding vector e, the watermark embedding vector e is generated by the following formula: The watermark embedding vector e is then repeated T times on the time axis and input into the first deep neural network model to generate binary watermark information m∈{0,1} for the universal watermark data B Universal watermark data δ m ∈[-1, 1] 1×T .

3. The method according to claim 1 or 2, characterized in that: The received universal audio watermark data δ m ∈[-1, 1] 1×T , establish a voice call, obtain the user voice data x, and use the lightweight second deep neural network model to watermark the universal audio data δ m ∈[-1, 1] 1×T Adaptively adjust and embed into a specific interval of the user's voice data to obtain the watermarked voice data y, specifically: Support watermark injection for static speech input once and streaming speech input continuously; When the input speech is pre-recorded static speech data x∈[-1, 1] 1×L When , embed the watermark in a random insertion manner to generate a blank speech data with the same length as x, and randomly select k non-overlapping watermark injection intervals on the blank speech data to inject the universal digital watermark, and obtain Δx0∈[-1,1] 1×L , L is the input speech length, L>kT, then Δx0 and x are input into the adaptive function f to generate an adaptive digital watermark Δx=f(Δx0,x) adapted to the specific speech input x, and Δx is directly superimposed on x to obtain the watermarked speech data, that is, y=x+ω·Δx, ω is the weight coefficient; the adaptive function f is a fusion of two time-series aligned one-dimensional vector data, using a convolutional neural network or a recurrent neural network or a cross attention mechanism; when it is necessary to embed a watermark at a specified position of the speech, the randomly inserted starting position is modified to the specific position where the watermark needs to be inserted; When the input speech is a continuously input streaming speech, the watermark is injected at the required time interval. The adaptive function f adopts an autoregressive neural network structure that supports streaming cache context, which is a causal convolutional neural network or a recurrent neural network or a cross-attention mechanism with mask and KV cache.

4. The method according to claim 3, characterized in that: The method of using the third deep neural network model to desensitize the watermarked voice data y to obtain the user desensitized data z is specifically as follows: The third deep neural network model includes three parts: an encoder, a decoder and a residual vector quantizer RVQ; after the speech passes through the encoder, it is mapped to n codebooks through RVQ and then restored through the decoder. Among the n codebooks, the first code is highly relevant content, and 2 to n codes are taken as the user desensitized data source.

5. The method according to claim 1, 2 or 4, characterized in that: The fourth deep neural network model is used to convert it into a watermark detection result at the sampling point level, and the digital watermark information is parsed by calculating the average value of the watermark detection result, specifically: Inputting the desensitized watermarked speech data into the fourth deep neural network model, the model outputting a 1+B-dimensional discrimination result for each time sampling point; The final result is determined by averaging the results. If the mean of a dimension is greater than the set threshold, it means that the information of that bit is 1, otherwise it is 0. The 0th dimension indicates whether there is a watermark, and the 1st to B dimensions are the specific values ​​of each bit of the watermark information. If and only if the 0th dimension is 1, the 1st to B dimension information is valid.

6. The method according to claim 5, characterized in that The transmitting user terminal specifies the binary watermark information m∈{0, 1} of fixed length B to be embedded B Before sending the information to the server through the communication network, the following steps are also included: Train the first, second, third, and fourth deep neural network models: The third deep neural network model is pre-trained on a large-scale speech data set alone, and its parameters are kept frozen during the joint training with the first, second, and fourth deep neural network models; The first, second and fourth deep neural network models are trained using an end-to-end method. The training process uses multi-scale reconstruction loss and detection result consistency loss, and actively interferes with the input of the third deep neural network through artificial channel simulation technology to enhance the robustness of the watermark.

7. The method according to claim 6, characterized in that The multi-scale reconstruction loss is: Among them, Mel i (·) represents the operation of extracting Mel spectrum from speech, the sliding window length is i, the window sliding step is i / 2, the Mel filter dimension is set to 40, and ||·||1 represents the L1 norm; The consistency loss of the detection result is to constrain the consistency of the detection result through the binary cross entropy loss function BCE, so as to ensure that the generated watermarked speech data can correctly identify the watermark information during detection. Among them, y t，j , represents the detection result at the sampling point level, t represents the timestamp of the sampling point, j represents the data position in the watermark information, Indicates that , it takes 0, otherwise it takes 1.

Images