High precision dynamic watermarking method and apparatus for streaming audio

Abstract

The application relates to a high-precision dynamic watermark method and device for streaming audio, which comprises the following steps: acquiring a streaming input audio input signal and a corresponding watermark information segment; the watermark information segment is composed of bit information segments with different states and variable lengths; determining a sampling point level watermark signal according to the audio input signal and the watermark information segment; embedding the sampling point level watermark signal into the audio input signal to obtain an audio output signal, and streaming outputting the audio output signal; performing watermark detection on the sampling points of a received audio signal to be tested to obtain a watermark confidence degree vector for the sampling points; and performing threshold judgment on the watermark confidence degree vector to recover the corresponding watermark information segment, thereby solving the problems of poor dynamicity and flexibility of watermark processing, supporting high-flexibility dynamic watermark information embedding, greatly enhancing watermark information bearing capacity, and being applicable to scenes requiring real-time updating of identification, tracking of dynamic states or embedding of complex metadata.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of digital watermarking, in particular to a high-precision dynamic watermarking method and device for streaming audio. BACKGROUND

[0002] Digital audio content is experiencing a breakthrough development, with its deep penetration in the fields of entertainment communication, remote communication and information distribution, especially the breakthrough progress of artificial intelligence generated content technology, which makes it possible to industrialize the production of high-fidelity audio such as voice synthesis and music creation. This technological wave is reshaping the content production paradigm, and also triggering a deep revolution in the intellectual property right confirmation, digital content traceability and authenticity verification system. In the face of an increasingly complex digital content ecosystem, it has become a core issue in the field of digital governance to build a secure protection mechanism with robustness and intelligence.

[0003] In the prior art, digital audio watermarking technology is usually used to improve the anti-counterfeiting capability of audio data. Digital audio watermarking technology realizes embedding of imperceptible digital information in audio signals. The current mainstream scheme is generally LSB replacement, which replaces the last 1-2 bits of the fixed replacement sample value, and cannot dynamically adjust the embedding position or capacity, nor can it flexibly adjust the watermark content, making it difficult to meet the diversified and high-standard needs of complex application scenarios such as live broadcast, real-time communication and processing of long audio files.

[0004] There is currently no effective solution to the problem of limited dynamicity and flexibility of audio watermarking in related technologies. SUMMARY

[0005] A high-precision dynamic watermarking method and device for streaming audio are provided in the present embodiment to solve the problem of limited dynamicity and flexibility of audio watermarking in related technologies.

[0006] In a first aspect, a high-precision dynamic watermarking method for streaming audio is provided in the present embodiment, the method comprising:

[0007] obtaining a streaming input audio input signal and a corresponding watermark information segment; the watermark information segment is composed of bit information segments with different states and variable lengths;

[0008] determining a sample point level watermark signal according to the audio input signal and the watermark information segment;

[0009] embedding the sample point level watermark signal into the audio input signal to obtain an audio output signal, and streaming outputting the audio output signal;

[0010] Watermark detection is performed on the sampling points of the received audio signal to be tested to obtain a watermark confidence vector for the sampling points; threshold decision is performed on the watermark confidence vector to recover the corresponding watermark information segment.

[0011] In some embodiments, determining a sampling point level watermark signal according to the audio input signal and the watermark information segment includes:

[0012] Mapping the watermark information segment to a watermark feature representation that is point-by-point aligned with the audio input signal;

[0013] Concatenating the watermark feature representation and a representation of the audio input signal along a channel dimension to obtain a sampling point level fusion feature representation;

[0014] Inputting the sampling point level fusion feature representation into a preset first neural network to calculate perturbation information at each sampling point to obtain a sampling point level watermark signal.

[0015] In some embodiments, mapping the watermark information segment to a watermark feature representation that is point-by-point aligned with the audio input signal includes:

[0016] Obtaining a learnable embedding layer and a corresponding mapping relationship;

[0017] Based on the mapping relationship, extracting a group of target row vectors corresponding to each bit information segment in the watermark information segment from the learnable embedding layer; based on the group of target row vectors, obtaining a feature vector corresponding to the bit information segment;

[0018] Combining each feature vector in a time sequence into a watermark feature representation that is point-by-point aligned with the audio input signal.

[0019] In some embodiments, watermark detection is performed on the sampling points of the received audio signal to be tested to obtain a watermark confidence vector for the sampling points includes:

[0020] Inputting the received audio signal to be tested into a preset second neural network to perform watermark detection on the sampling points to obtain a multi-dimensional watermark confidence vector for the sampling points; the multi-dimensional watermark confidence vector includes watermark presence information and watermark content information.

[0021] In some embodiments, threshold decision is performed on the watermark confidence vector to recover the corresponding watermark information segment includes:

[0022] Smooth processing is performed on the watermark confidence vector to obtain a smoothing result;

[0023] Based on a preset decision threshold, binaryzation processing is performed on the smoothing result to obtain a binary decision sequence;

[0024] In a case where the watermark presence information indicates that the watermark exists, the bit information segment is obtained based on the corresponding binary decision sequence;

[0025] The watermark information segment is recovered based on the bit information segment corresponding to each sampling point.

[0026] In some embodiments, the first neural network adopts an encoder-decoder structure, the first neural network adopts a causal convolution operation, and has a state transfer mechanism across audio processing segments;

[0027] The second neural network comprises an encoder component, the encoder component at least comprising a convolution layer and a channel attention mechanism layer, and the convolution layer adopts a depth separable convolution operation.

[0028] In some embodiments, the first neural network and the second neural network are obtained based on end-to-end joint training of an initial generation network and an initial detection network; a forward data propagation process of the end-to-end joint training comprises:

[0029] Based on an audio test signal, a watermark test segment with a preset bit number is randomly generated within a preset time length range;

[0030] The watermark test segment is converted into a test feature vector at a sampling point level, the test feature vector is spliced with the audio test signal and input into the initial generation network, and a one-dimensional test watermark signal with the same length as the audio test signal is output; the one-dimensional test watermark signal is embedded into the audio test signal to obtain a watermark test signal;

[0031] The watermark test signal is subjected to data enhancement processing, the enhanced watermark test signal is input into the initial detection network, and watermark detection confidence at a sampling point level is output, including watermark presence test information and watermark content test information, wherein the data enhancement processing comprises at least one of channel distortion enhancement, watermark mask enhancement, and tampering simulation enhancement.

[0032] In some embodiments, the end-to-end joint training adopts a multi-task loss function for optimization; the multi-task loss function comprises at least one of a perceptual loss function and a watermark loss function;

[0033] The perceptual loss function comprises at least one of a deep feature loss function, a loudness domain loss function, a frequency domain loss function, and a time domain loss function;

[0034] The watermark loss function comprises at least one of a watermark existence detection loss function, a multi-bit message detection loss function, and a boundary consistency loss function.

[0035] In some embodiments, the end-to-end joint training is optimized using a curriculum-based training strategy; the curriculum-based training strategy includes at least one of the following: dynamically adjusting the weights of the perceptual loss function and progressively increasing the number of bits in the watermark information segment.

[0036] Secondly, this embodiment provides a high-precision dynamic watermarking device for streaming audio, the device comprising:

[0037] The data acquisition module is used to acquire the streaming audio input signal and the corresponding watermark information segment; the watermark information segment is composed of bit information segments with different states and variable lengths;

[0038] A sampling point-level watermark generation module is used to determine a sampling point-level watermark signal based on the audio input signal and the watermark information segment;

[0039] The watermark embedding module is used to embed the sampling point-level watermark signal into the audio input signal to obtain an audio output signal, and to stream the audio output signal.

[0040] The watermark extraction module is used to perform watermark detection on the sampling points of the received audio signal to be tested, and obtain the watermark confidence vector for the sampling points; and to perform threshold decision on the watermark confidence vector to recover the watermark information segment.

[0041] Thirdly, this application also provides a computer device. The computer device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the high-precision dynamic watermarking method for streaming audio described in the first aspect.

[0042] Fourthly, this application also provides a computer-readable storage medium. The computer-readable storage medium stores a computer program thereon, which, when executed by a processor, implements the high-precision dynamic watermarking method for streaming audio described in the first aspect.

[0043] Compared with related technologies, the high-precision dynamic watermarking method and apparatus for streaming audio provided in this embodiment acquires the streaming audio input signal and the corresponding watermark information segment; the watermark information segment is composed of bit information segments with different states and variable lengths; a sampling point-level watermark signal is determined based on the audio input signal and the watermark information segment; the sampling point-level watermark signal is embedded into the audio input signal to obtain an audio output signal, and the audio output signal is streamed; watermark detection is performed on the sampling points of the received audio signal to be tested to obtain a watermark confidence vector for the sampling points; a threshold decision is made on the watermark confidence vector to recover the corresponding watermark information segment. This solves the problem of poor dynamics and flexibility in watermark processing. Based on sampling point-level operation, the length and content of the watermark can be dynamically changed according to the sampling points of the audio input signal, supporting highly flexible dynamic watermark information embedding and greatly enhancing the watermark information carrying capacity. It is suitable for scenarios that require real-time updating of identifiers, tracking of dynamic states, or embedding of complex metadata.

[0044] Details of one or more embodiments of this application are set forth in the following drawings and description to make other features, objects and advantages of this application more readily apparent. Attached Figure Description

[0045] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings:

[0046] Figure 1 This is a hardware structure block diagram of the application environment for the high-precision dynamic watermarking method for streaming audio in the embodiments of this application.

[0047] Figure 2 This is a flowchart illustrating the high-precision dynamic watermarking method for streaming audio in the embodiments of this application.

[0048] Figure 3 This is a schematic flowchart illustrating the method for determining the sampling point-level watermark signal in an embodiment of this application.

[0049] Figure 4 This is a schematic diagram of the method for recovering watermark information segments based on post-processing in the embodiments of this application;

[0050] Figure 5 This is a schematic diagram of the structure of the first neural network in the embodiments of this application;

[0051] Figure 6 This is a schematic diagram of the structure of the second neural network in the embodiments of this application;

[0052] Figure 7This is a flowchart illustrating the high-precision dynamic watermark training method for streaming audio in the embodiments of this application.

[0053] Figure 8 This is a flowchart illustrating a high-precision dynamic watermarking method for streaming audio in a preferred embodiment of this application.

[0054] Figure 9 This is a structural block diagram of a high-precision dynamic watermarking device for streaming audio in an embodiment of this application.

[0055] Reference numerals: 102, terminal; 104, cloud server; 106, communication network; 108, other terminals; 91, data acquisition module; 92, sampling point-level watermark generation module; 93, watermark embedding module; 94, watermark extraction module. Detailed Implementation

[0056] To better understand the purpose, technical solution, and advantages of this application, the application is described and illustrated below in conjunction with the accompanying drawings and embodiments.

[0057] Unless otherwise defined, the technical or scientific terms used in this application shall have the general meaning understood by one of ordinary skill in the art to which this application pertains. Words such as “a,” “an,” “an,” “the,” “the,” and “these” used in this application do not indicate quantitative limitation and may be singular or plural. The terms “comprising,” “including,” “having,” and any variations thereof used in this application are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or device that comprises a series of steps or modules (units) is not limited to the listed steps or modules (units) but may include steps or modules (units) not listed, or may include other steps or modules (units) inherent to these processes, methods, products, or devices. Words such as “connected,” “linked,” and “coupled” used in this application are not limited to physical or mechanical connections but may include electrical connections, whether direct or indirect. “Multiple” used in this application refers to two or more. “And / or” describes the relationship between related objects, indicating that three relationships may exist; for example, “A and / or B” can represent: A alone, A and B simultaneously, and B alone. Normally, the character " / " indicates that the objects before and after it are in an "or" relationship. The terms "first," "second," "third," etc., used in this application are merely to distinguish similar objects and do not represent a specific order of objects.

[0058] The high-precision dynamic watermarking method for streaming audio provided in this embodiment can be applied to a variety of application environments. For example, it can be applied to applications such as... Figure 1The server-terminal application environment shown typically involves terminal 102, cloud server 104, and communication network 106 connecting the two. Terminal 102 communicates with cloud server 104 via communication network 106. In this environment, for example in applications such as copyright protection or content tracking, the content provider can first upload the original audio content (i.e., audio input signal) to cloud server 104. An embedding processing unit implementing the method of this embodiment is deployed on cloud server 104. The embedding processing unit generates a dynamic watermark message containing copyright identifiers, content IDs, etc., according to preset rules or specific requirements. The dynamic watermark message can have time-varying states and / or variable length characteristics to improve tracking accuracy and security. Subsequently, the embedding processing unit on cloud server 104 calls the high-precision embedding algorithm of this embodiment to embed this dynamic watermark message into the original audio with high precision (e.g., reaching or approaching the sampling point level precision), generating watermarked audio (i.e., audio input signal). This watermarked audio can be stored in a data storage system associated with cloud server 104 and distributed to terminal 102 via communication network 106. When copyright verification or source tracing of audio on terminal 102 is required, the audio to be detected or its characteristics can be uploaded to cloud server 104. The detection and processing unit deployed on cloud server 104 performs high-precision watermark detection and dynamic message extraction (e.g., sampling point-level detection and extraction) on the received data, and returns the verification results to the requesting party or for platform management via communication network 106. Terminal 102 can be various, such as personal computers, laptops, smartphones, tablets, smart home devices (e.g., smart speakers, smart TVs), in-vehicle infotainment systems, or wearable devices (e.g., smartwatches).

[0059] Furthermore, the high-precision dynamic watermarking method for streaming audio provided in this embodiment can also be applied to terminal-to-terminal application scenarios. This scenario typically involves a sending terminal (terminal 102) and one or more receiving terminals (other terminals 108) communicating via a communication network 106. For example, in real-time voice calls, secure instant messaging, or online collaboration applications, when sending an audio stream, terminal 102 can utilize its locally deployed embedding processing unit implementing the method of this embodiment. This processing unit can embed a dynamic watermark message containing specific information (such as sender authentication fingerprint, session synchronization signal, message integrity check code, or low-rate control command) into the transmitting audio stream with high precision (e.g., sample point precision). The watermarked audio stream is transmitted to other terminals 108 via the communication network 106. The receiving terminal device utilizes its locally deployed detection processing unit implementing the method of this embodiment to detect the presence of the watermark in the received audio stream and extract the embedded dynamic watermark information. The extracted information can be used by applications on the receiving device to perform corresponding operations, such as authentication, ensuring data integrity, synchronizing application state, or responding to control commands. It is understandable that in terminal-to-terminal scenarios, the communication involved can be one-to-many, such as live streaming, or many-to-many, such as video conferencing.

[0060] This embodiment provides a high-precision dynamic watermarking method for streaming audio. Figure 2 This is a flowchart of the high-precision dynamic watermarking method for streaming audio in this embodiment, as follows: Figure 2 As shown, the process includes the following steps:

[0061] Step S210: Obtain the audio input signal of the streaming input and the corresponding watermark information segment; the watermark information segment consists of bit information segments with different states and variable lengths.

[0062] Specifically, streaming audio refers to the technology of transmitting audio content to a receiving end in real time via a network or other digital transmission medium. Unlike traditional downloading and playback, streaming audio allows users to transmit audio data continuously and stably to the listener through specific transmission protocols and encoding methods. This implementation receives the input of streaming audio and obtains the corresponding audio input signal. The audio input signal can be pre-processed data, and pre-processing methods include, but are not limited to, normalization, noise reduction, and other pre-processing operations to improve the quality of the audio signal.

[0063] This embodiment supports the embedding of dynamic watermarks, meaning that the embedded watermark information is not a static sequence but rather dynamically configured information. This dynamism manifests in two aspects: first, the "state" of the watermark can change, meaning the specific bit content of the bit information segment at each time point can change. The capacity of the bit information segment is B (B is an integer greater than or equal to 1), which determines that B bits of information can be embedded at a given time. Second, the "length" of each state can change, meaning the time span or the number of sample points included in each state can change. The range of length variation (Lmin, Lmax) can be set to improve the reliability of the changes. The specific "state" content and "length" variation rules are defined and provided by the upper-layer application based on its business logic and other information, making the watermark application extremely flexible and adaptable to a wide range of scenarios requiring dynamic information marking. Several typical cases are listed below:

[0064] The first type is switching triggered by human intervention or an external system: In this case, the change in watermark status is triggered by a switching command generated manually by the user or by a switching command sent by an external control system. For example, in a live streaming scenario, the host can manually switch the embedded watermark content to identify different live streaming segments or promotional information; or, the broadcast monitoring system can instruct the embedding of different channel identifiers or timecodes.

[0065] The second type is automatic switching based on context or event: In this case, the change in watermark status is associated with the current business scenario or a specific event. For example, in e-commerce live streaming, the embedded watermark can be associated with the currently displayed product link or ID; when the host switches products to explain, the corresponding watermark information is automatically updated. Another example is in multi-speaker video conferencing or voice chat scenarios, where the embedded watermark can be automatically switched based on the microphone of the current speaker (through Voice Activity Detection (VAD) or sound source localization recognition), using the watermark to identify the current speaker's ID.

[0066] The third type involves automatic generation and labeling based on content or time: In this method, the watermark status is automatically generated and updated by the system based on the audio content itself or time information. For example, a B-bit timestamp representing the current precise time (e.g., year, month, day, hour, minute, second) can be periodically generated and embedded as the watermark status for precise post-event traceability. Another example is the integration of automatic speaker recognition; when different speakers are identified, a unique B-bit identifier representing that speaker's identity is automatically embedded. Yet another example is the calculation of a hash value or acoustic fingerprint for segments of audio content, with its B-bit representation embedded as a watermark for fine-grained content integrity verification or identification.

[0067] It should be noted that the above typical cases are used to explain different watermark embedding scenarios and are not intended to limit them.

[0068] Step S220: Determine the sampling point-level watermark signal based on the audio input signal and the watermark information segment.

[0069] Specifically, the watermark information segment is processed in conjunction with the audio input signal to generate a sampling point-level watermark signal that matches the audio input signal at the sampling point precision, achieving the effect of minimal interference with the audio input signal and imperceptibility to the human ear. In some embodiments, the process of determining the sampling point-level watermark signal can be implemented based on a neural network. The neural network can automatically learn the content of the audio input signal and the watermark information segment at the sampling point level, generating a sampling point-level watermark signal that meets the requirements, greatly reducing the design difficulty of the watermarking method. In other implementations, this step can also be achieved through time-frequency analysis, such as performing time-frequency analysis on the audio input signal to extract its spectral energy distribution and masking threshold, and then dynamically adjusting the amplitude and frequency band position of the watermark information segment based on a psychoacoustic model—increasing the watermark intensity in high-energy frequency bands and reducing the watermark amplitude in low-energy frequency bands to maintain concealment; at the same time, a pseudo-random sequence can be generated in combination with the audio content (such as using the audio spectrum hash value as a seed), and the watermark information segment can be mapped to a broadband noise signal related to the audio statistical characteristics through spread spectrum modulation, or the intermediate frequency DCT coefficients or wavelet subbands can be adaptively selected in the transform domain, and the quantization step size (QIM) can be designed according to the local energy difference of the host audio to match the quantization error of the watermark signal with the characteristics of the audio signal to achieve the adaptability of the watermark signal to the host audio, taking into account both robustness and imperceptibility.

[0070] Step S230: Embed the sampling point-level watermark signal into the audio input signal to obtain the audio output signal, and stream the audio output signal.

[0071] Specifically, the sampling point-level watermark signal is superimposed onto the audio input signal by directly adding the time-domain signals. Specifically, for each sample point n in the audio input signal, the corresponding generated sampling point-level watermark signal δ[n] is added to the original audio input signal value x[n]. To more precisely control the embedding strength to balance robustness and audio fidelity, a strength control factor α can be used to scale the watermark signal before addition. Therefore, the audio output signal x is calculated... wm A more complete operation can be represented as: x wm [n] = x[n] + αδ[n]. A larger α enhances the robustness of the watermark but may introduce audible distortion; a smaller α helps maintain high fidelity but may reduce the watermark's survival under noise or compression. The α value can be designed as a fixed value or dynamically adjusted according to the specific application. Since this overlay process is performed sample-by-sample, it ensures that the watermark is added to the audio with the highest temporal precision.

[0072] After superposition, the generated watermarked audio output signal x wm [n] is stored in the system output buffer. The data in the buffer is then continuously output to the next stage according to the correct timing, for example, directly sent to the network transmission module for real-time broadcasting, or written to a local audio file stream. Through this buffering and timing management, this embodiment can achieve low-latency, smooth and continuous streaming audio output, meeting the requirements of real-time applications. The output audio stream carries the original content while secretly carrying embedded dynamic watermark information.

[0073] Step S240: Perform watermark detection on the sampling points of the received audio signal to be tested to obtain the watermark confidence vector for the sampling points; perform threshold judgment on the watermark confidence vector to recover the corresponding watermark information segment.

[0074] Specifically, this step aims to initially extract potential watermark information from the received audio signal x'[n], which may have been distorted during channel transmission. A watermark confidence vector can be output for each or a portion of representative sample points in the audio signal. The specific bit information of the watermark can be recovered based on the watermark confidence vector, and then combined with time information to integrate the various bit information into a segment of watermark information. This step can be implemented based on the neural network or time-frequency analysis method corresponding to step S220. The specific implementation method is not limited in this embodiment.

[0075] In this embodiment, the audio input signal and the corresponding watermark information segment are acquired through streaming input. The watermark information segment consists of bit information segments with different states and variable lengths. Based on the audio input signal and the watermark information segment, a sampling point-level watermark signal is determined. The sampling point-level watermark signal is embedded into the audio input signal to obtain an audio output signal, which is then streamed. Watermark detection is performed on the sampling points of the received audio signal to be tested to obtain a watermark confidence vector for each sampling point. Threshold judgment is performed on the watermark confidence vector to recover the corresponding watermark information segment. This solves the problem of poor dynamics and flexibility in watermark processing. Based on sampling point-level operation, the length and content of the watermark can be dynamically changed according to the sampling points of the audio input signal, supporting highly flexible dynamic watermark information embedding. It also greatly enhances the watermark information carrying capacity and is suitable for scenarios that require real-time updating of identifiers, tracking of dynamic states, or embedding of complex metadata.

[0076] In some embodiments, based on step S210, acquiring the streaming audio input signal includes:

[0077] Step S211: Obtain the initial audio signal by dividing the continuous audio stream into short speech segments.

[0078] Specifically, the initial audio signal can contain a single short audio segment or multiple short audio segments, depending on the actual needs. Receiving streaming input as short audio segments effectively processes continuous audio data streams, especially in real-time applications such as live streaming, real-time communication, or processing long audio files. Unlike loading the entire audio file at once, streaming input means that audio data is delivered to the processing system gradually in the form of continuous small data blocks (called "frames" or "segments"). The short audio segments can be of fixed or variable length. Each frame (i.e., each short audio segment) contains a certain number of audio sample points. The length of the frame (or window size) needs to be set according to application requirements and the input requirements of the processing unit (such as a downstream neural network). For example, it can be set to hundreds to thousands of sample points, corresponding to a duration of tens to hundreds of milliseconds. This streaming processing method allows the system to handle audio input of arbitrary length with low memory usage and short processing latency, while simultaneously achieving the effect of injecting watermarks while speaking, without affecting the original experience of real-time applications.

[0079] Step S212: Preprocess the initial audio signal to obtain the audio input signal.

[0080] Specifically, upon receiving each audio frame, the system performs necessary preprocessing. "Normalization" aims to eliminate volume differences that may be introduced from different audio sources or during transmission, adjusting the audio amplitude to a standard range (e.g., [-1, 1]). This helps improve the stability and consistency of subsequent neural network processing. Streaming processing typically employs dynamic normalization methods, such as root mean square (RMS) normalization based on a sliding window or exponential moving average (EMA) normalization based on historical data. In other embodiments, the input audio source itself contains significant background noise. Appropriate noise reduction algorithms (such as spectral subtraction, Wiener filtering, or deep learning-based noise reduction models) are used to improve the clarity of the original audio, which may help enhance the concealment of watermark embedding or the robustness of extraction. The audio data frames, after framing and preprocessing, are sequentially fed into the subsequent watermark embedding process (e.g., steps S220 and S230).

[0081] Furthermore, when acquiring the audio signal x'[n] to be tested, a streaming framing method compatible with the embedding end can also be used for processing. Necessary preprocessing, such as normalization and noise reduction, can also be performed on the audio signal x'[n] to be tested to eliminate volume differences that may be introduced from different audio sources or during transmission, thereby improving audio clarity.

[0082] In this embodiment, streaming processing is used to achieve low memory usage and short processing latency to handle audio input of any length. Furthermore, preprocessing eliminates volume differences that may be introduced from different audio sources or during transmission, improving audio clarity and increasing the time for high-quality input in subsequent watermarking steps.

[0083] In some embodiments, based on step S220, a sampling point-level watermark signal is determined according to the audio input signal and the watermark information segment, see [link to relevant documentation]. Figure 3 Specifically, it includes the following steps:

[0084] Step S221: Map the watermark information segment to a watermark feature representation that is point-by-point aligned with the audio input signal.

[0085] Specifically, since the original binary bit information may not be effective enough for neural networks to directly learn embedding strategies, it needs to be projected into a higher-dimensional continuous feature space. The mapping from information to features aims to map the B-bit information segment m[n] at each time point n into a rich and learnable C-dimensional feature vector e[n]. The feature vectors e[n] calculated at all time points n are combined sequentially to form a watermark feature representation M (with dimensions N×C) that has the same length N as the audio input signal x[n] and is aligned point by point.

[0086] Step S222: The watermark feature representation and the audio input signal representation are concatenated along the channel dimension to obtain the sampling point-level fused feature representation.

[0087] Specifically, after obtaining the point-by-point aligned watermark feature representation M, it needs to be fused with the corresponding audio input signal representation x (usually considered as N×1 dimensional) to generate input data that simultaneously contains information from both. In this embodiment, the fusion is performed by concatenation along the channel dimension. This means that the representation of the audio input signal x (1 channel) and the watermark feature representation M (C channels) are placed side by side along the feature dimension to form a point-by-point fused feature representation F with a dimension of N×(1+C). The advantage of this fusion method is that it completely preserves the information of the original audio signal, while using the learned features representing the watermark intent as additional channel information, with the two strictly aligned in time. This allows the subsequent first neural network to simultaneously perceive the audio content and the corresponding watermark instructions in its convolutional or other processing layers, thereby learning to generate accurate, content-related point-by-point watermark signals.

[0088] Step S223: Input the sampling point-level fusion feature representation into the preset first neural network, calculate the perturbation information at each sampling point, and obtain the sampling point-level watermark signal.

[0089] Specifically, the first neural network can be configured to support streaming processing. This means that the first neural network can adopt mechanisms such as causal convolution and cross-segment state management to enable it to process continuous input audio streams and their corresponding feature representations with low latency, frame by frame or block by block, without waiting for the entire audio file or long periods of data. This streaming processing capability is crucial for real-time watermark embedding applications (such as live streaming and real-time communication).

[0090] The first neural network internally performs deep processing on the input sample-point level fused feature representation F through its multi-layer structure (e.g., an encoder-decoder structure based on U-Net). The first neural network not only needs to understand the information to be embedded (watermark information segment m[n]) contained in the watermark feature channel, but also fully considers the content of the audio input signal x[n] in the audio feature channel. This is because the generated sample-point level watermark signal δ[n] needs to meet the information embedding requirements while minimizing its impact on auditory quality (i.e., achieving high fidelity) when superimposed on the original audio input signal x[n]. Therefore, the first neural network learns a balancing strategy during training: it adjusts the shape and amplitude of the generated sample-point level watermark signal δ[n] according to the characteristics of the audio content (such as energy, frequency components, temporal structure, etc.), embedding stronger signals in areas imperceptible to the human ear (utilizing auditory masking effects) to improve robustness, while embedding weaker signals in sensitive areas to ensure imperceptibility.

[0091] Finally, for each frame of the input sample-level fusion feature representation F (of length N), the first neural network outputs a sample-level watermark signal δ[n] of the same length and dimension N×1. δ[n] is a sequence of real values ​​representing the specific perturbation value to be added to each corresponding sample point of the original audio input signal. The "point-by-point" characteristic ensures that the watermark embedding has the highest possible temporal accuracy. This generated δ[n] will then be used in step S230 to modify the audio input signal x[n].

[0092] In this embodiment, by synergistically utilizing a neural network architecture specifically designed for sample point processing, high-precision embedding of audio watermarks at the single sampling point level is achieved, making the time positioning accuracy far exceed that of traditional audio frame-based methods, providing a more advantageous technical foundation for applications requiring fine control and analysis (such as precise tamper positioning).

[0093] In some embodiments, step S221 above, mapping the watermark information segment to a watermark feature representation that is point-by-point aligned with the audio input signal, includes:

[0094] Step S310: Obtain the learnable embedding layers and their corresponding mapping relationships.

[0095] Specifically, in this embodiment, the mapping is implemented through a learnable embedding layer E. This learnable embedding layer E can be viewed as a parameterized lookup table, storing 2B different C-dimensional vectors, each vector corresponding to one of the two possible states (0 or 1) of a certain bit in B bits. Simultaneously, by obtaining a preset mapping relationship, the desired information to represent the watermark information segment can be found in the learnable embedding layer E, thereby achieving the mapping. In one implementation, for bit information m[n] located at time point n, the i-th bit m of bit information m[n] is... i [n] indicates the precise selection of the corresponding C-dimensional vector from the learnable embedding layer E based on its value (0 or 1). For example, in the learnable embedding layer E, the vector with index 2i+m... i The row vectors of [n]. Since the parameters of the learnable embedding layer E are learnable, the model can automatically optimize this mapping from bits to features during training.

[0096] Step S320: Based on the mapping relationship, extract a set of target row vectors corresponding to each bit information segment in the watermark information segment from the learnable embedding layer; based on the set of target row vectors, obtain the feature vector corresponding to the bit information segment.

[0097] Specifically, for each B-bit information segment, the B C-dimensional vectors selected are combined to generate a single C-dimensional feature vector e[n] representing the overall watermark information at that time point n. For example, the combination operation can be implemented by summation, and the calculation formula is as follows: Where e[n] represents the feature vector, B represents the total number of bits in the bit information segment, and i represents the bit index of the bit information segment. This represents the 2i+mth learnable embedding layer E. i [n] rows are row vectors.

[0098] Step S330: Combine each feature vector in time sequence to form a watermark feature representation that is point-by-point aligned with the audio input signal.

[0099] Specifically, by combining the feature vectors e[n] obtained from all time points n in sequence, a watermark feature representation M (with dimensions N×C) with the same length N as the audio input signal x[n] and aligned point by point is formed.

[0100] In this embodiment, binary discrete data is projected into a higher-dimensional continuous feature space based on a learnable embedding layer, and this mapping relationship from bits to features is automatically optimized through the learnable embedding layer.

[0101] In some embodiments, step S240, which involves watermark detection on the sampling points of the received audio signal to be tested to obtain a watermark confidence vector for each sampling point, specifically includes the following steps:

[0102] Step S410: Input the received audio signal to be tested into a preset second neural network to perform watermark detection on the sampling points and obtain a multi-dimensional watermark confidence vector for the sampling points; the multi-dimensional watermark confidence vector includes watermark existence information and watermark content information.

[0103] Specifically, the second neural network is configured to receive the audio signal to be tested and output a sample-point-level multidimensional confidence vector, i.e., a watermark confidence vector. The structure of the second neural network includes an encoder component for feature extraction, which at least includes convolutional layers and a channel attention mechanism. For example, an architecture incorporating depthwise separable convolutions and channel attention mechanisms, such as an architecture based on the SeaNet encoder, could be used.

[0104] In the watermark confidence vector c[n], one element is the watermark presence information c0[n], which represents the confidence or probability estimate of the presence of a watermark signal at the current sample point n. This value is usually between 0 and 1; the closer the value is to 1, the higher the probability that the network believes a watermark exists at that point. The remaining B elements in the watermark confidence vector c[n] are the watermark content information. The B elements of the watermark content information correspond to the confidence level of each bit in the embedded B-bit information segment being "1". These values ​​are usually between 0 and 1. i The closer [n] is to 1, the greater the probability that the network considers the i-th bit to be "1" at that moment. Preferably, the watermark presence information c0[n] is in the first position, and the watermark content information is in the first position. The data is located in the subsequent B positions, so the dimension of the watermark confidence vector c[n] is (B+1).

[0105] Understandably, the output confidence vector c[n] is a continuous, potentially noisy, raw network output. It represents the neural network's "preliminary judgment" or "belief strength" regarding the presence and content of the watermark at each sample point, rather than the final, discrete detection result. This sequence of sample-point-level confidence vectors forms the basis for subsequent refinement processing.

[0106] In this embodiment, based on the second neural network, the sampling point level operation is further delayed until the watermark extraction stage, overcoming the limitation of most existing technologies that can only embed and extract static or fixed pattern information.

[0107] In some embodiments, in step S240, a threshold decision is made on the watermark confidence vector to recover the corresponding watermark information segment. See [link to relevant documentation]. Figure 4Specifically, it includes the following steps:

[0108] Step S510: Smooth the watermark confidence vector to obtain a smoothed result.

[0109] Specifically, the goal of calculating the sampling point-level multidimensional watermark confidence vector c[n] is to transform these raw, potentially unstable confidence data into final, clear, and reliable discrete watermark information. However, the watermark confidence vector c[n] is the original confidence sequence, which may contain jitter and errors caused by noise, channel distortion, or network prediction uncertainty. Directly thresholding it may lead to a large number of bit errors and fragmented results. Therefore, post-processing is required, mainly including smoothing and binarization.

[0110] The smoothing process specifically refers to temporal smoothing, which eliminates transient noise and isolated erroneous predictions in the confidence sequence, enabling the recovered watermarked message to exhibit a stable paragraph structure with a certain duration, similar to the original embedding. A common method for temporal smoothing is to apply the sliding window technique, where, for each sample point n with a confidence vector c[n], all confidence vectors within a temporally adjacent window are examined. The confidence of the center point n is adjusted based on the dominance of each state within the window (watermark presence / absence, and the 0 / 1 state of each bit) (e.g., the average confidence calculated using the mean, median, or weighted average, or direct majority voting). For example, if a point's confidence contradicts the vast majority of its neighbors' states and its own confidence is low, it may be adjusted to align with its neighbors' states. Low-confidence sample points with confidence close to the decision threshold (e.g., 0.5) can also be specially handled, given lower weights or subjected to a more conservative update strategy during the smoothing process. Time smoothing can effectively suppress noise, connect broken segments, and make the confidence sequence more stable, thus better reflecting the structure of the original embedded segmented watermark message.

[0111] Step S520: Based on the preset decision threshold, the smoothing result is binarized to obtain a binary decision sequence.

[0112] Specifically, since the smoothed result is still a continuous confidence vector, the watermark bit information cannot be directly extracted. Binarization is needed to convert the smoothed result into a discrete binary (0 or 1) judgment result. Specifically, the smoothed result includes watermark presence information c0[n] and watermark content information c i[n], a preset decision threshold (usually 0.5) is applied to the smoothing result. If a confidence value in the smoothing result is greater than the decision threshold, the corresponding state is determined to be 1 (watermark exists, or the bit is 1); if it is less than or equal to the threshold, it is determined to be 0 (watermark does not exist, or the bit is 0). After binarization, the discrete watermark information sequence m'[n] is obtained, that is, the binary decision sequence, including the watermark presence flag (recovered from the watermark presence information c0[n]) and B message bits (recovered from the watermark content information c0[n]). i [n] is recovered.

[0113] Step S530: If the watermark presence information indicates the presence of the watermark, obtain the bit information segment based on the corresponding binary determination sequence.

[0114] Specifically, the binary determination sequence is discrete, relatively stable, and approximates the original embedded watermark bit information segment as closely as possible. Furthermore, based on the recovered presence flag sequence, it is possible to determine which time periods contain the watermark.

[0115] Step S540: Based on the bit information segments corresponding to each sampling point, recover the watermark information segments.

[0116] Specifically, based on the existence of the watermark, the B message bits are combined into the final detection result, i.e., the watermark information segment.

[0117] In this embodiment, by using post-processing steps based on smoothing and binarization, jitter and errors caused by noise, channel distortion, or network prediction uncertainty are eliminated, improving the accuracy of watermark information segment recovery. This overcomes the limitation of existing technologies that can only embed static or fixed pattern information, greatly enhancing the flexibility and information carrying capacity of watermarks. It is suitable for scenarios that require real-time updating of identifiers, tracking of dynamic status, or embedding of complex metadata.

[0118] In some of these embodiments, the first neural network adopts an encoder-decoder structure, employs causal convolution operations, and has a state transfer mechanism across audio processing segments.

[0119] Specifically, the first neural network is configured to receive sample-point-level fused feature representations and generate sample-point-level watermark signals. For example, it can employ an encoder-decoder structure, such as a network based on the U-Net or Demucs architecture. The convolutional operations in the network are modified to causal convolutions, ensuring that the calculation of the output at the current time point does not depend on any future input information. Based on causal operations, a state transfer mechanism across audio processing segments can be implemented to maintain the necessary temporal context dependencies in streaming processing.

[0120] In this embodiment, by modifying the first neural network architecture to be causal and by adopting a state transfer mechanism across audio processing segments, low-latency, continuous streaming processing of audio signals is supported. This enables seamless integration into systems with high real-time requirements, such as live streaming and real-time communication, thus meeting the key needs of modern audio applications.

[0121] In some of these embodiments, see Figure 5 The first deep neural network includes an encoder, a bottleneck layer, and a decoder connected in sequence. The encoder includes multiple levels of first residual downsampling modules connected in sequence; each level of the first residual downsampling module includes connected residual convolutional blocks and downsampling layers. The decoder includes first residual upsampling modules symmetrical to and connected to the first residual downsampling modules to introduce skip connections from the encoder and recover signal details; each level of the first residual upsampling module includes connected residual convolutional blocks and upsampling layers. The bottleneck layer includes a first long short-time neural network to capture temporal dependencies.

[0122] The first deep neural network may further include two independent first one-dimensional convolutional layers, each connected to an encoder to receive the original audio (i.e., the audio input signal) and the sample-point-level information to be embedded (i.e., the watermark feature representation). Features are extracted separately by the independent first one-dimensional convolutional layers and then fused to form a sample-point-level fused feature representation. This sample-point-level fused feature representation is fed into the encoder of the first deep neural network and undergoes depth processing sequentially through a bottleneck layer and a decoder to generate a sample-point-level watermark signal. Finally, the sample-point-level watermark signal is embedded into the original audio using a point-by-point stacking method to obtain the watermarked audio (i.e., the audio output signal).

[0123] In this embodiment, the watermark signal can be adaptively adjusted according to the original audio content to balance robustness and imperceptibility.

[0124] In some embodiments, the second neural network includes an encoder component, which includes at least convolutional layers and channel attention mechanism layers, the convolutional layers employing depthwise separable convolution operations. Specifically, the second neural network is configured to receive the audio signal to be tested and output a watermark confidence vector at the sample point level, for example, it can employ an architecture that includes depthwise separable convolution and channel attention mechanisms, such as an architecture based on the SeaNet encoder.

[0125] In some of these embodiments, see Figure 6 The second neural network also includes a one-dimensional transposed convolutional layer connected to the encoder component. The encoder component includes a second one-dimensional convolutional layer, multiple levels of second residual downsampling modules, a second long short-term neural network, and a third one-dimensional convolutional layer connected in sequence; each level of the second residual downsampling module includes a connected residual convolutional block and a downsampling layer.

[0126] Specifically, the second one-dimensional convolutional layer is used for preliminary feature extraction of the audio signal under test. A multi-level second residual downsampling module integrates depthwise separable convolution and channel attention mechanisms to enhance feature representation capabilities. A second long short-term neural network processes sequence information. The third one-dimensional convolutional layer refines the features. Finally, through processing by a one-dimensional transposed convolutional layer, a multi-dimensional confidence vector at the sampling point level is output as the watermark detection result (i.e., the watermark confidence vector), which indicates the probability of detecting predetermined watermark information at each time position of the audio signal.

[0127] In this embodiment, by modifying the first neural network architecture to be causal and by adopting a state transfer mechanism across audio processing segments, the second neural network is designed to adopt an architecture that includes depthwise separable convolution and channel attention mechanisms, thereby supporting low-latency, continuous streaming processing of audio signals. It can be seamlessly integrated into systems with high real-time requirements such as live streaming and real-time communication, meeting the key needs of modern audio applications.

[0128] In some embodiments, the first and second neural networks are obtained through end-to-end joint training of the initial generator network and the initial detector network. See also Figure 7 The forward data propagation process for end-to-end joint training includes:

[0129] Step S610: Based on the audio test signal, a watermark test segment with a preset number of bits is randomly generated within a preset time range.

[0130] Specifically, the audio test signal can be audio data obtained from a database or streaming audio data received in a frame-by-frame manner. Furthermore, the audio data can be preprocessed to improve data quality. Figure 7 The original information segment is the watermark test segment, which is aligned with the audio (i.e., the audio test signal) in the time domain.

[0131] Step S620: Convert the watermark test segment into a test feature vector at the sampling point level. Concatenate the test feature vector with the audio test signal and input it into the initial generation network to output a one-dimensional test watermark signal of the same length as the audio test signal. Embed the one-dimensional test watermark signal into the audio test signal to obtain a watermarked test signal.

[0132] Specifically, Figure 7 The sample point-level information in the signal is the test feature vector. A bit-to-feature mapping is performed based on the learnable embedding layer E to obtain the sample point-level test feature vector. The sample point-level test feature vector is aligned with the audio test signal in both length and sample points. Figure 7 The watermarked audio shown is the watermarked test signal.

[0133] Step S630: Perform data augmentation processing on the watermarked test signal, input the augmented watermarked test signal into the initial detection network, and output the watermark detection confidence at the sampling point level, including watermark 1-bit presence test information and watermark content test information with a preset number of bits.

[0134] Specifically, to further improve the performance of neural networks, particularly in the accuracy of watermark signal localization and robustness against channel distortion and malicious attacks (such as audio cropping, splicing, and tampering) that may be encountered in practical applications, one or more data augmentation techniques can be employed to apply simulated transformations or perturbations to the training data, thereby enhancing the neural network model's ability to operate stably under diverse conditions. Therefore, see [link to relevant documentation]. Figure 7 The watermarked test signal can be augmented through data enhancement methods such as active masking, channel simulation, and tamper simulation to achieve accurate sampling point positioning, resistance to channel interference, and resistance to active attack damage. This embodiment does not limit the data augmentation method.

[0135] Figure 7 The sampling point-level logical values ​​shown are the watermark detection confidence levels. During the watermark embedding stage, the sampling point-level information includes preset B-bit information. During the watermark extraction stage, the corresponding watermark content test information (i.e.,...) Figure 7 The sampling point-level information (which is a logical value) also corresponds to a B bit. In addition, the detection result contains one more bit containing test information (i.e., Figure 8 The sampling point-level watermark contains bit logic values.

[0136] After the end-to-end joint training is completed, the optimized neural network parameters are loaded into the modules that run the first and second neural networks for the actual watermark embedding and extraction tasks.

[0137] In this embodiment, although the end-to-end joint training does not directly participate in the watermark embedding and extraction during runtime, it is a key support for the effective operation of the entire system. With the powerful fitting ability of deep learning and combined with data augmentation techniques and other optimized training methods, it can resist common audio distortions (such as compression and noise) while maintaining the auditory quality of the original audio to the maximum extent, achieving a good balance between robustness and imperceptibility, and improving the efficiency and accuracy of watermark processing.

[0138] In some of these embodiments, the data enhancement process includes at least one of: channel distortion enhancement, watermark mask enhancement, and tamper simulation enhancement.

[0139] Channel distortion enhancement aims to simulate various signal degradations that audio signals may experience during transmission, storage, or playback, thereby improving the model's robustness to common channel distortions. For example, different types and intensities of noise, such as additive white Gaussian noise or specific environmental noise, can be added to the training samples (i.e., watermarked test signals) to simulate different signal-to-noise ratio (SNR) conditions. Another example is performing sampling rate modification operations on the training samples, such as downsampling and then upsampling, to simulate distortion introduced by sampling rate conversion or mismatch between different devices. Yet another example is applying lossy compression processing, i.e., using common audio codecs (such as MP3, AAC, Opus, etc.) to encode and decode the training samples at different bit rates to simulate the information loss introduced by compression. Optionally, it may also include simulating other linear or nonlinear distortions, such as equalization, clipping, echo, and reverberation.

[0140] Watermark mask enhancement is primarily used to train detection models to accurately identify the boundaries of watermark embedding regions, which is crucial for resisting content pruning attacks and accurately extracting watermark information. Specifically, a random binary mask can be generated during training. This mask defines alternating regions with values ​​of 1 (indicating watermark presence) and 0 (indicating watermark absence) along the time dimension. This random binary mask can be applied to training samples, for example, by multiplying it with an ideal watermark perturbation signal, or by selectively retaining / removing watermark components from watermarked audio, thereby creating training samples containing alternating watermarked and non-watermarked regions. The training objective is then adjusted accordingly to not only detect the presence of watermarks but also accurately predict the boundaries defined by the mask. Parameters such as the mask pattern and paragraph length distribution can be randomly varied during training.

[0141] Enhanced tamper simulation aims to improve a model's ability to detect malicious splicing, replacement, and other tampering of audio content. Such tampering typically disrupts the continuity or intrinsic relationship between the original audio and the embedded watermark. For example, one or more time segments can be randomly selected from a watermarked test signal. These selected segments are then replaced with other audio content, such as audio segments from different sources, the corresponding original unwatermarked segments, or segments embedded using different watermark messages / keys. This operation simulates editing tampering behaviors such as cut and paste, effectively disrupting the temporal continuity or consistency of the audio and / or watermark signals. By training a neural network model to identify such discontinuities or inconsistencies introduced by tampering (e.g., by monitoring decoding errors, feature mutations, etc.), its ability to detect splicing-type tampering can be improved.

[0142] It should be understood that the above three data augmentation techniques can be implemented individually or in any combination on the same training sample (i.e., the watermarked test signal). The specific parameters used when performing augmentation operations (such as noise level, compression bit rate, mask parameters, replacement fragment position and length, etc.) are usually randomly selected within a preset range to maximize the diversity of the training data.

[0143] In this embodiment, by selecting and combining various data augmentation methods, positioning performance and channel robustness can be further enhanced, or robustness against malicious attacks such as pruning can be improved, thus adapting to different application scenarios.

[0144] In some embodiments, end-to-end joint training is optimized using a multi-task loss function, which includes a perceptual loss function and a watermarking loss function. The perceptual loss function includes at least one of the following: a deep feature loss function, a loudness domain loss function, a frequency domain loss function, and a temporal domain loss function. The watermarking loss function includes at least one of the following: a watermark existence detection loss function, a multi-bit message detection loss function, and a boundary consistency loss function.

[0145] Specifically, a multi-task loss function L is used to optimize the first and second neural networks in the aforementioned end-to-end joint training. The goal of the multi-task loss function L is to simultaneously guide the learning of both neural networks, enabling them to achieve optimal performance through mutual cooperation. Specifically, it is configured to jointly optimize two seemingly contradictory but crucial objectives for the watermarking system: first, to maintain high perceptual quality in the watermarked audio, i.e., to make it as imperceptible to the human ear as possible; and second, to ensure that the embedded watermark information can be accurately and robustly detected and extracted, even after the audio has undergone common distortions such as compression and noise.

[0146] The multi-task loss function L can be constructed by weighted summation of multiple specific loss terms, and its overall form can be expressed as:

[0147] ;

[0148] Among them, w wav w mel w loud w deep w det w msg and w bound These represent the weighting coefficients for each loss item, used to adjust the relative importance of different objectives in the total loss; L wav L represents the time-domain loss function; mel L represents the frequency domain loss function; loud L represents the loudness domain loss function; deep L represents the deep feature loss function;det Watermark Presence Detection Loss Function; L msg Multi-bit message detection loss function; representing L bound This represents the boundary consistency loss function.

[0149] Temporal loss function (L wav It can directly constrain differences at the time-domain waveform level by calculating the L2 distance between the original audio test signal and the watermarked test signal in the time-domain waveform, thereby limiting the basic acoustic distortion.

[0150] Frequency domain loss function (L mel It can measure perceptual differences from the frequency domain perspective. It is obtained by calculating the L2 distance between the multi-scale Mel spectrum representation of the original audio test signal and the watermarked test signal, thus achieving an effect that is closer to the frequency perception characteristics of the human ear.

[0151] Loudness domain loss function (L loud This allows for more refined perception optimization using the principle of auditory masking, and its calculation method is as follows:

[0152] ;

[0153] ;

[0154] in, This represents a one-dimensional test watermark signal used for embedding within a specific frequency band b and time window w. With the original audio test signal The loudness difference between them, loudness through a function (Estimated according to ITU-RBS.1770-4 recommendation). This method allows for greater tolerance of loudness variations in time-frequency regions where the human ear is less sensitive, while requiring smaller differences in sensitive regions.

[0155] Deep Feature Loss Function (L deep The aim is to capture higher-level, more abstract perceptual feature differences, and its calculation method is as follows:

[0156] ;

[0157] in, Describing the L2 norm, w represents the features extracted from the first three layers of a pre-trained audio model (such as Wav2Vec2.0). i The pre-set layer feature weights are used to constrain the perceptual artifacts introduced by the watermark by utilizing the representational capabilities of the pre-trained model.

[0158] The four loss functions listed above (depth feature loss function, loudness domain loss function, frequency domain loss function, and time domain loss function) belong to perceptual loss functions. Perceptual loss functions focus on minimizing the impact of watermark embedding on the original audio's listening experience, aiming to maintain high fidelity. Depending on the actual application scenario, at least one of the above perceptual loss functions can be selected.

[0159] Watermark Existence Detection Loss Function (L) det This is used to train the detector to accurately determine whether a watermark exists at each sample point. It is obtained by calculating the binary cross-entropy (BCE) between the predicted watermark existence probability p[n] and the actual watermark mask mask[n]. Specifically:

[0160] ;

[0161] Where N is the audio length.

[0162] Multi-bit message detection loss function (L msg The value is used to train the detector to accurately recover the embedded B-bit message content. It is obtained by calculating the binary cross-entropy (BCE) between the predicted confidence of each watermark bit (c[n][b]) and the actual watermark bit (m[n][b]), specifically:

[0163] ;

[0164] Where B is the number of watermark bits.

[0165] Boundary consistency loss function (L bound The purpose is to enhance the continuity and stability of detection results over time and reduce isolated errors. It is obtained by calculating the binary cross-entropy (BCE) between the predicted confidence difference (Δc[n][b]) and the true bit difference (Δm[n][b]) of adjacent sample points, specifically:

[0166] ;

[0167] ;

[0168] .

[0169] The three loss functions listed above (watermark existence detection loss function, multi-bit message detection loss function, and boundary consistency loss function) are watermark loss functions. Watermark loss functions focus on improving the performance of watermark detection and extraction, ensuring accurate information delivery and system robustness. Depending on the specific application, at least one of these watermark loss functions can be selected.

[0170] In this embodiment, by adjusting the weighted combination of these perceptual loss terms and watermark loss terms, the neural network can be effectively guided to learn how to achieve high-precision and high-robust dynamic watermark embedding and detection while maintaining audio quality.

[0171] In some of these embodiments, the end-to-end joint training is optimized using a curriculum-based training strategy; the curriculum-based training strategy includes at least one of the following: dynamically adjusting the weights of the perceptual loss function and progressively increasing the number of bits in the watermark information segment.

[0172] Specifically, to further optimize the stability of the training process and improve the overall performance of the final model (e.g., achieving a good balance between embedding capacity, robustness, and audio fidelity), a curriculum learning strategy can be further adopted. The core idea of ​​this strategy is to present training tasks or data to the model in a certain order or in an increasing manner of difficulty, simulating the learning process of humans or animals, from easy to difficult.

[0173] The strategy of dynamically adjusting the weights of the perceptual loss function aims to guide the model's learning focus by dynamically adjusting the weight of the perceptual loss term in the total loss function. In the early stages of training, the perceptual loss term is assigned a relatively low weight. This allows the model to focus more on learning the basic patterns and mappings of watermark embedding and extraction in the initial stages, without being overly constrained by audio quality (measured by the perceptual loss), thus reducing the risk of early training crashing due to excessive constraints or getting stuck in local optima. As training progresses (e.g., with increased training steps or epochs), the weight of the perceptual loss term is gradually, smoothly, or in stages. This gradual increase in weight allows the model, after mastering basic embedding capabilities, to focus more on maintaining the perceptual quality of the original audio, ultimately reaching a balance at the end of training where it can effectively embed / detect watermarks while producing high-quality watermarked audio. This strategy helps achieve stable convergence and optimizes the final audio fidelity. In a specific implementation, a perceptual loss term with very low weights can be used at the beginning of training (e.g., the first 10,000 steps). Data augmentation can be applied at this stage without simultaneous application, allowing the model to learn the basic task first. Subsequently, the weights of the perceptual loss term can be gradually introduced and increased, while (or later) the aforementioned data augmentation techniques can be introduced.

[0174] The strategy of progressively increasing the number of bits in the watermark information segment aims to allow the model to gradually adapt to more challenging embedding and extraction tasks by gradually increasing the number of bits B (i.e., embedding capacity) of the watermark message to be embedded. The model first learns robust embedding and detection with a lower number of bits, mastering the basic capabilities before tackling higher embedding rates. Training begins with a preset initial number of bits B0 (e.g., B0 = 1 or a small value). As training progresses (e.g., according to training steps t), the number of bits B in the watermark message increases... t It is gradually increased until the preset maximum number of bits B is reached. max The increase in the number of bits can follow a predetermined strategy. For example, a gradual expansion strategy can be used to calculate the number of bits B used at training step t. t :

[0175] ;

[0176] Among them, B t B is the number of watermark message bits used at training step t; max B0 is the target maximum number of watermark bits; ΔB is the initial number of watermark bits; ΔB is the number of bits added each time (e.g., ΔB=1); t is the current training step number; T is the bit number update period (i.e., the number of bits is increased every T steps). This indicates the floor function.

[0177] This incremental approach of increasing task difficulty helps prevent training instability when the model directly handles high-complexity tasks (high bit-rate embeddings), enabling the model to learn feature representations that are robust under different embedding capacities more effectively.

[0178] It should be understood that the two curriculum-based learning strategies listed above can be used individually or in combination. For example, training can begin with a lower number of bits and a lower perceptual loss weight, and then the perceptual loss weight and the number of watermark bits can be gradually increased simultaneously or alternately.

[0179] In this embodiment, by implementing a training method that includes one or more curriculum strategies, a more stable and efficient training process can be achieved, and the final trained model can achieve a better balance and optimization in multiple performance dimensions such as audio quality preservation, watermark embedding capacity and robustness, thus making it suitable for efficient and reliable audio watermark embedding and detection tasks in actual deployment.

[0180] The present embodiment will now be described and illustrated through preferred embodiments. Figure 9 This is a flowchart of a high-precision dynamic watermarking method for streaming audio according to a preferred embodiment.

[0181] S1, Watermark Embedding Stage: Real-time embedding of watermark information with sampling point-level positioning accuracy into the streaming voice input.

[0182] S1.1 Acquiring Audio Input Signal and Watermark Information Segment: By segmenting the continuous audio stream into short speech segments, the audio signal is received as a streaming input, and necessary preprocessing operations such as normalization and noise reduction are performed on the obtained audio signal to obtain the final audio input signal; at the same time, a watermark information segment of the same length and aligned with the audio input signal is input. The watermark information segment consists of B-bit information segments with different states and variable lengths, where B is an integer greater than or equal to 1.

[0183] S1.2, Sample point-level watermark signal generation: The dynamic watermark message is mapped to a watermark feature representation that is aligned point by point with the audio input signal, and the watermark feature representation and the representation of the audio input signal are concatenated along the channel dimension to obtain the sample point-level fused feature representation; the sample point-level fused feature representation is input to the first neural network that supports streaming processing to generate the sample point-level watermark signal corresponding to the audio input signal.

[0184] S1.3 Watermark signal embedding and output: The sampling point-level watermark signal is superimposed on the audio input signal in the form of direct addition of time domain signals to generate a watermarked audio signal. The watermarked audio signal is then stored in the system output buffer to realize voice streaming output.

[0185] S2, Watermark Extraction Stage: Detect whether there is a watermark in the speech and extract the corresponding sampling point-level watermark information.

[0186] S2.1, Sampling point level confidence detection: The received audio signal is input into the second neural network (detector network). The second neural network outputs a multi-dimensional confidence vector for each or part of the sampling points in the audio signal. The vector contains the confidence of the watermark existence and the confidence of each watermark message bit.

[0187] S2.2 Post-processing of watermark information segments: Post-processing of the multidimensional confidence vector, including time smoothing and binarization, to recover the embedded segmented watermark message.

[0188] In this preferred embodiment, addressing the limitations of existing audio watermarking technologies in terms of embedding and detection accuracy, information dynamism, and streaming processing capabilities, a high-precision dynamic audio digital watermarking method is provided. This method utilizes a streaming deep neural network architecture to accurately embed and extract dynamic watermark messages composed of bit segments with different states and variable lengths by operating at the sample point level of the audio signal. Specifically, this preferred embodiment uses a first neural network to generate a sample point-level watermark signal and combines it with the original audio. Simultaneously, a second neural network outputs a sample point-level multidimensional confidence vector and recovers the watermark information through post-processing. This effectively overcomes the shortcomings of traditional watermarking methods in terms of accuracy, flexibility, and real-time performance, aiming to provide a more accurate, flexible, and efficient solution for copyright protection, content tracking, and management of digital audio content (especially suitable for real-time streaming media, AI-generated content, and other scenarios).

[0189] It should be noted that the steps shown in the above process or in the flowchart of the accompanying figures can be executed in a computer system such as a set of computer-executable instructions, and although a logical order is shown in the flowchart, in some cases the steps shown or described may be executed in a different order than that shown here.

[0190] It should be noted that the user-related information (including but not limited to the audio content itself, possible embedded identity identifiers, device information, etc.) and data (including but not limited to parameters, feature data, stored data, display data, etc. used for watermark generation / detection) involved in this application are all information and data authorized or permitted by the user and fully authorized by all parties. Furthermore, the collection, use and processing of the relevant data must comply with relevant laws, regulations and standards, and corresponding operation entry points are provided for users to choose to authorize or refuse.

[0191] This embodiment also provides a high-precision dynamic watermarking device for streaming audio, which is used to implement the above embodiments and preferred embodiments; details already described will not be repeated. The terms "module," "unit," "subunit," etc., used below can refer to a combination of software and / or hardware that performs a predetermined function. Although the device described in the following embodiments is preferably implemented in software, hardware implementation, or a combination of software and hardware, is also possible and contemplated.

[0192] Figure 9 This is a structural block diagram of the high-precision dynamic watermarking device for streaming audio in this embodiment, as shown below. ​ As shown, the device includes: a data acquisition module 91, a sampling point-level watermark generation module 92, a watermark embedding module 93, and a watermark extraction module 94.

[0193] The data acquisition module 91 is used to acquire the audio input signal of the streaming input and the corresponding watermark information segment; the watermark information segment is composed of bit information segments with different states and variable lengths;

[0194] The sampling point-level watermark generation module 92 is used to determine the sampling point-level watermark signal based on the audio input signal and the watermark information segment;

[0195] The watermark embedding module 93 is used to embed the sampling point-level watermark signal into the audio input signal to obtain the audio output signal, and to stream the audio output signal.

[0196] The watermark extraction module 94 is used to perform watermark detection on the sampling points of the received audio signal to be tested, and obtain the watermark confidence vector for the sampling points; and to perform threshold judgment on the watermark confidence vector to recover the watermark information segment.

[0197] In some embodiments, the sampling point-level watermark generation module 92 is further configured to map the watermark information segment into a watermark feature representation that is aligned point-by-point with the audio input signal; concatenate the watermark feature representation with the representation of the audio input signal along the channel dimension to obtain a sampling point-level fused feature representation; input the sampling point-level fused feature representation into a preset first neural network to calculate the perturbation information at each sampling point to obtain a sampling point-level watermark signal.

[0198] In some embodiments, the sampling point-level watermark generation module 92 is also used to obtain a learnable embedding layer and the corresponding mapping relationship; based on the mapping relationship, extract a set of target row vectors corresponding to each bit information segment in the watermark information segment from the learnable embedding layer; based on the set of target row vectors, obtain the feature vectors corresponding to the bit information segments; and combine the feature vectors in time sequence to form a watermark feature representation that is point-by-point aligned with the audio input signal.

[0199] In some embodiments, the watermark extraction module 94 is further configured to input the received audio signal to be tested into a preset second neural network to perform watermark detection on the sampling points and obtain a multi-dimensional watermark confidence vector for the sampling points; the multi-dimensional watermark confidence vector includes watermark existence information and watermark content information.

[0200] In some embodiments, the watermark extraction module 94 is further configured to smooth the watermark confidence vector to obtain a smoothed result; based on a preset decision threshold, the smoothed result is binarized to obtain a binary decision sequence; when the watermark presence information indicates the presence of the watermark, a bit information segment is obtained based on the corresponding binary decision sequence; and the watermark information segment is recovered based on the bit information segment corresponding to each sampling point.

[0201] In some of these embodiments, the first neural network adopts an encoder-decoder structure, employs causal convolution operations, and has a state transfer mechanism across audio processing segments; the second neural network includes an encoder component, which includes at least a convolutional layer and a channel attention mechanism layer, and the convolutional layer employs depthwise separable convolution operations.

[0202] In some embodiments, the first neural network and the second neural network are obtained by end-to-end joint training of the initial generation network and the initial detection network. The forward data propagation process of the end-to-end joint training includes: randomly generating a watermark test segment of a preset number of bits within a preset time range based on the audio test signal; converting the watermark test segment into a test feature vector at the sampling point level; concatenating the test feature vector with the audio test signal and inputting it into the initial generation network to output a one-dimensional test watermark signal of the same length as the audio test signal; embedding the one-dimensional test watermark signal into the audio test signal to obtain a watermarked test signal; performing data augmentation processing on the watermarked test signal; inputting the augmented watermarked test signal into the initial detection network to output a watermark detection confidence score at the sampling point level, including watermark presence test information and watermark content test information.

[0203] In some of these embodiments, the data enhancement process includes at least one of: channel distortion enhancement, watermark mask enhancement, and tamper simulation enhancement.

[0204] In some of these embodiments, end-to-end joint training is optimized using a multi-task loss function; the multi-task loss function includes at least one of a perceptual loss function and a watermarking loss function; the perceptual loss function includes at least one of the following: a deep feature loss function, a loudness domain loss function, a frequency domain loss function, and a time domain loss function; the watermarking loss function includes at least one of the following: a watermark existence detection loss function, a multi-bit message detection loss function, and a boundary consistency loss function.

[0205] In some of these embodiments, the end-to-end joint training is optimized using a curriculum-based training strategy; the curriculum-based training strategy includes at least one of the following: dynamically adjusting the weights of the perceptual loss function and progressively increasing the number of bits in the watermark information segment.

[0206] It should be noted that the above modules can be functional modules or program modules, and can be implemented through software or hardware. For modules implemented through hardware, the above modules can reside in the same processor; or the above modules can be located in different processors in any combination.

[0207] This embodiment also provides a computer device, including a memory and a processor, wherein the memory stores a computer program and the processor is configured to run the computer program to perform the steps in any of the above method embodiments.

[0208] Optionally, the computer device may further include a transmission device and an input / output device, wherein the transmission device is connected to the processor and the input / output device is connected to the processor.

[0209] It should be noted that the specific examples in this embodiment can refer to the examples described in the above embodiments and optional implementations, and will not be repeated in this embodiment.

[0210] Furthermore, in conjunction with the high-precision dynamic watermarking method for streaming audio provided in the above embodiments, this embodiment can also provide a computer storage medium for implementation. The computer storage medium stores a computer program; when executed by a processor, the computer program implements any of the high-precision dynamic watermarking methods for streaming audio described in the above embodiments.

[0211] It should be understood that the specific embodiments described herein are merely illustrative of the application and not intended to limit it. All other embodiments derived by those skilled in the art based on the embodiments provided in this application without inventive effort are within the scope of protection of this application.

[0212] Obviously, the accompanying drawings are merely some examples or embodiments of this application. Those skilled in the art can apply this application to other similar situations based on these drawings without any creative effort. Furthermore, it is understood that although the work done in this development process may be complex and lengthy, for those skilled in the art, certain design, manufacturing, or production modifications made based on the technical content disclosed in this application are merely conventional technical means and should not be considered as insufficient disclosure of this application.

[0213] The term "embodiment" in this application refers to a specific feature, structure, or characteristic described in connection with an embodiment that may be included in at least one embodiment of this application. The appearance of this phrase in various places in the specification does not necessarily imply the same embodiment, nor does it imply that it is mutually exclusive with or independent of other embodiments. It will be clearly or implicitly understood by those skilled in the art that the embodiments described in this application may be combined with other embodiments without conflict.

[0214] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of patent protection. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the appended claims.

Claims

1. A high-precision dynamic watermarking method for streaming audio, characterized in that, The method includes: The audio input signal from the streaming input and the corresponding watermark information segment are acquired; the watermark information segment is composed of bit information segments with different states and variable lengths; the different states refer to different bit contents; the variable length refers to the duration of the bit content being dynamically set within a set range. Determining a sampling point-level watermark signal based on the audio input signal and the watermark information segment includes: processing the watermark information segment in conjunction with the audio input signal to generate a sampling point-level watermark signal that is compatible with the audio input signal at the sampling point precision. The sampling point-level watermark signal is embedded into the audio input signal to obtain an audio output signal, and the audio output signal is streamed. Watermark detection is performed on the sampling points of the received audio signal to be tested to obtain the watermark confidence vector for the sampling points; a threshold decision is made on the watermark confidence vector to recover the corresponding watermark information segment.

2. The high-precision dynamic watermarking method for streaming audio according to claim 1, characterized in that, Based on the audio input signal and the watermark information segment, determine the sampling point-level watermark signal, including: The watermark information segment is mapped to a watermark feature representation that is point-by-point aligned with the audio input signal; The watermark feature representation is concatenated with the audio input signal representation along the channel dimension to obtain a sampling point-level fused feature representation; The sampling point-level fusion feature representation is input into a preset first neural network to calculate the perturbation information at each sampling point, thereby obtaining the sampling point-level watermark signal.

3. The high-precision dynamic watermarking method for streaming audio according to claim 2, characterized in that, Mapping the watermark information segment to a watermark feature representation that is point-by-point aligned with the audio input signal includes: Obtain the learnable embedding layers and their corresponding mappings; Based on the mapping relationship, a set of target row vectors corresponding to each bit information segment in the watermark information segment are extracted from the learnable embedding layer; based on the set of target row vectors, the feature vector corresponding to the bit information segment is obtained. The individual feature vectors are combined in time sequence to form a watermark feature representation that is point-by-point aligned with the audio input signal.

4. The high-precision dynamic watermarking method for streaming audio according to claim 2, characterized in that, Watermark detection is performed on the sampling points of the received audio signal to be tested, and a watermark confidence vector for each sampling point is obtained, including: The received audio signal to be tested is input into a preset second neural network to perform watermark detection on the sampling points, thereby obtaining a multi-dimensional watermark confidence vector for the sampling points; the multi-dimensional watermark confidence vector includes watermark existence information and watermark content information.

5. The high-precision dynamic watermarking method for streaming audio according to claim 4, characterized in that, The watermark confidence vector is subjected to a threshold decision to recover the corresponding watermark information segment, including: The watermark confidence vector is smoothed to obtain a smoothed result; Based on a preset decision threshold, the smoothing result is binarized to obtain a binary decision sequence; When the watermark presence information indicates the presence of a watermark, the bit information segment is obtained based on the corresponding binary determination sequence; Based on the bit information segments corresponding to each of the sampling points, the watermark information segments are recovered.

6. The high-precision dynamic watermarking method for streaming audio according to claim 4, characterized in that, The first neural network adopts an encoder-decoder structure, uses causal convolution operations, and has a state transfer mechanism across audio processing segments; The second neural network includes an encoder component, which includes at least a convolutional layer and a channel attention mechanism layer, wherein the convolutional layer employs depthwise separable convolution operations.

7. The high-precision dynamic watermarking method for streaming audio according to claim 4, characterized in that, The first neural network and the second neural network are obtained by end-to-end joint training of the initial generator network and the initial detector network; The forward data propagation process for the end-to-end joint training includes: Based on the audio test signal, a watermark test segment with a preset number of bits is randomly generated within a preset time range; The watermark test segment is converted into a test feature vector at the sampling point level. The test feature vector is concatenated with the audio test signal and then input into the initial generation network to output a one-dimensional test watermark signal of the same length as the audio test signal. The one-dimensional test watermark signal is then embedded into the audio test signal to obtain a watermarked test signal. The watermarked test signal is subjected to data augmentation processing. The augmented watermarked test signal is then input into an initial detection network, and the watermark detection confidence level at the sampling point level is output, including watermark presence test information and watermark content test information. The data augmentation processing includes at least one of the following: channel distortion enhancement, watermark mask enhancement, and tamper simulation enhancement.

8. The high-precision dynamic watermarking method for streaming audio according to claim 7, characterized in that, The end-to-end joint training is optimized using a multi-task loss function; the multi-task loss function includes at least one of a perceptual loss function and a watermarking loss function. The perceptual loss function includes at least one of the following: a depth feature loss function, a loudness domain loss function, a frequency domain loss function, and a time domain loss function; The watermark loss function includes at least one of the following: watermark existence detection loss function, multi-bit message detection loss function, and boundary consistency loss function.

9. The high-precision dynamic watermarking method for streaming audio according to claim 8, characterized in that, The end-to-end joint training is optimized using a curriculum-based training strategy; the curriculum-based training strategy includes at least one of the following: dynamically adjusting the weights of the perceptual loss function and progressively increasing the number of bits in the watermark information segment.

10. A high-precision dynamic watermarking device for streaming audio, characterized in that, The device includes: The data acquisition module is used to acquire the streaming audio input signal and the corresponding watermark information segment; the watermark information segment is composed of bit information segments with different states and variable lengths; the different states refer to different bit contents; the variable length refers to the duration of the bit content being dynamically set within a set range; A sampling point-level watermark generation module is used to determine a sampling point-level watermark signal based on the audio input signal and the watermark information segment, including: processing the watermark information segment in conjunction with the audio input signal to generate a sampling point-level watermark signal that is compatible with the audio input signal at the sampling point precision. The watermark embedding module is used to embed the sampling point-level watermark signal into the audio input signal to obtain an audio output signal, and to stream the audio output signal. The watermark extraction module is used to perform watermark detection on the sampling points of the received audio signal to be tested, and obtain the watermark confidence vector for the sampling points; and to perform threshold decision on the watermark confidence vector to recover the watermark information segment.

11. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 9.

Images