Model watermarking method, device, computer equipment and storage medium for speech synthesis system

Abstract

The present application relates to a model watermarking method, device, computer equipment and storage medium for speech synthesis systems, wherein the method comprises: constructing a watermark triggering sample in a domain based on a pre-trained speech synthesis model and a fine-tuned speaker recognition model; and constructing a corresponding watermark implantation data set based on the watermark triggering sample; implanting the watermark implantation data set into the speech synthesis model based on the fine-tuned speaker recognition model to obtain a tokenized model; and screening out a target watermark triggering sample based on the tokenized model. Through the present application, the problem that the watermark and the main task of speech synthesis are not strongly correlated in the related art, which causes the watermark to be easily lost during the model fine-tuning process, is solved, and the watermark triggering sample in the domain is constructed to force the strong coupling between the watermark triggering task and the main task of the model from the data level, thereby improving the robustness of the watermark in the tokenized model and making the target watermark triggering sample maintain stability and effectiveness.

Description

Technical Field

[0001] The present application relates to the field of computer artificial intelligence security, and in particular to a model watermarking method, apparatus, computer equipment and storage medium for a speech synthesis system. Background Art

[0002] In the field of artificial intelligence, especially in the subfield of speech synthesis, there has been significant technological progress. Existing speech synthesis technology relies on complex neural networks, which require a large amount of training data and sophisticated optimization strategies, and has become an important intellectual asset. The reproducibility of neural networks poses a challenge to copyright protection, especially after the model is fine-tuned, the copyright ownership of the original model is more complicated, making model ownership and intellectual property protection a focus of industry attention. In this context, model watermarking technology has attracted attention as a potential solution.

[0003] At present, model watermarking technology embeds hidden identifiers in the model, so that the model produces watermarked output under specific input, providing a basis for copyright attribution. However, existing model watermarking technology is mostly aimed at classification tasks, which has limitations on the application of speech synthesis models and is not strongly related to the main task of speech synthesis, resulting in the watermark being easily lost during the model fine-tuning process.

[0004] In view of the fact that the watermark is not strongly related to the main task of speech synthesis in related technologies, the watermark is easily lost during the model fine-tuning process. No effective solution has been proposed so far. Summary of the invention

[0005] In this embodiment, a model watermarking method, apparatus, computer equipment and storage medium for a speech synthesis system are provided to solve the problem in the related art that the watermark is not strongly associated with the main task of speech synthesis, resulting in the watermark being easily lost during the model fine-tuning process.

[0006] In a first aspect, a model watermark method for a speech synthesis system is provided in this embodiment, including:

[0007] Based on the pre-trained speech synthesis model and the fine-tuned speaker recognition model, a watermark trigger sample in the domain is constructed; and based on the watermark trigger sample, a corresponding watermark implantation data set is constructed; the training sample of the speech synthesis model is the speech data set in the domain;

[0008] Based on the fine-tuned speaker recognition model, the watermark implantation data set is implanted into the speech synthesis model to obtain a tokenized model; the speaker recognition model is obtained by fine-tuning the in-domain speech data set;

[0009] Based on the tokenization model, target watermark trigger samples are screened out.

[0010] In some of the embodiments, constructing a watermark trigger sample in the domain based on a pre-trained speech synthesis model and a fine-tuned speaker recognition model includes:

[0011] Acquire an in-domain speech data set corresponding to the speech synthesis model pre-trained in the speech synthesis system;

[0012] Based on the in-domain speech dataset, fine-tuning the pre-trained speaker recognition model to obtain the fine-tuned speaker recognition model;

[0013] Based on the fine-tuned speaker recognition model, extracting timbre features from the in-domain speech data set to obtain timbre features; clustering the feature centers of the timbre features, and selecting source categories from the clustering results;

[0014] The watermark trigger samples in a domain are constructed based on the source category.

[0015] In some embodiments, based on the fine-tuned speaker recognition model, timbre features are extracted from the in-domain speech dataset to obtain timbre features; feature centers of the timbre features are clustered, and source categories are selected from the clustering results, including:

[0016] Based on the fine-tuned intermediate layer representation of the speaker recognition model as the timbre feature vector of the input speech, timbre features are extracted from the speech samples in the in-domain speech dataset to obtain timbre features;

[0017] Clustering the feature centers of the timbre features by category to obtain category centers;

[0018] The source category is determined according to the preset distance and the category center.

[0019] In some of the embodiments, constructing the watermark trigger sample in the domain based on the source category includes:

[0020] Randomly extracting a first speech sample from each of the source categories;

[0021] Preprocessing the first speech samples of each category to obtain second speech samples aligned in time domain;

[0022] Each of the second speech samples is synthesized by time-domain weighted averaging to obtain a watermark trigger sample.

[0023] In some embodiments, based on the fine-tuned speaker recognition model, the watermark is implanted into the speech synthesis model of the data set to obtain a tokenized model, including:

[0024] According to a preset total loss function and the fine-tuned speaker recognition model, the speech synthesis model is jointly optimized to obtain a tokenized model; the total loss function is composed of a preset first loss function, a preset second loss function and a pre-trained loss function of the speech synthesis model;

[0025] During the joint optimization process, under the triggering of the first loss function in the total loss function, the speech synthesis model is optimized based on the speech samples of the in-domain speech data set and the fine-tuned speaker recognition model; and under the triggering of the second loss function in the total loss function, the speech synthesis model is optimized based on the watermark trigger samples of the watermark implantation data set and the fine-tuned speaker recognition model.

[0026] In some of the embodiments, based on the tokenization model, screening out target watermark trigger samples includes:

[0027] Fine-tuning the tokenized model to obtain multiple alternative models;

[0028] Based on the multiple substitution models, the validity of the watermark trigger sample is tested to obtain an average speaker similarity score between each watermark trigger sample and a preset pseudo speaker;

[0029] The watermark trigger samples whose average speaker similarity scores are higher than a preset threshold are selected as target watermark trigger samples.

[0030] In some embodiments, the method further comprises:

[0031] After obtaining the suspicious model, inputting the watermark trigger sample into the suspicious model to obtain the model output speech;

[0032] Inputting the model output speech into the fine-tuned speaker recognition model to obtain model output speaker features;

[0033] The speaker features output by the model are compared with the speaker features of the pseudo speaker to obtain a watermark detection result.

[0034] In a second aspect, a model watermark device for a speech synthesis system is provided in this embodiment, including: a construction module, an implantation module and a screening module;

[0035] The construction module is used to construct a watermark trigger sample in the domain based on a pre-trained speech synthesis model and a fine-tuned speaker recognition model; and to construct a corresponding watermark implantation data set based on the watermark trigger sample; the training sample of the speech synthesis model is a speech data set in the domain;

[0036] The implantation module is used to implant the watermark implantation data set into the speech synthesis model based on the fine-tuned speaker recognition model to obtain a tokenized model; the speaker recognition model is obtained based on the fine-tuning of the in-domain speech data set;

[0037] The screening module is used to screen out target watermark trigger samples based on the tokenization model.

[0038] In a third aspect, a computer device is provided in this embodiment, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the model watermark method for a speech synthesis system described in the first aspect is implemented.

[0039] In a fourth aspect, a storage medium is provided in this embodiment, on which a computer program is stored, and when the program is executed by a processor, the model watermark method for a speech synthesis system described in the first aspect is implemented.

[0040] Compared with the related art, the model watermark method, apparatus, computer equipment and storage medium for speech synthesis system provided in this embodiment construct watermark trigger samples in the domain based on the pre-trained speech synthesis model and the fine-tuned speaker recognition model; and construct the corresponding watermark implantation data set based on the watermark trigger samples; implant the watermark implantation data set into the speech synthesis model based on the fine-tuned speaker recognition model to obtain the tokenized model; based on the tokenized model, screen out the target watermark trigger samples, solve the problem in the related art that the watermark is not strongly correlated with the main task of speech synthesis, which makes the watermark easy to be lost during the model fine-tuning process, and use the watermark trigger samples in the construction domain to force the strong coupling between the watermark trigger task and the main task of the model from the data level, thereby improving the robustness of the watermark in the tokenized model and making the target watermark trigger samples maintain stability and effectiveness.

[0041] Details of one or more embodiments of the present application are set forth in the following drawings and description to make other features, objects, and advantages of the present application more readily apparent. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0043] Figure 1 It is a hardware structure block diagram of a terminal device of a model watermark method for a speech synthesis system provided by an embodiment of the present application;

[0044] Figure 2is a flow chart of a model watermarking method for a speech synthesis system provided by an embodiment of the present application;

[0045] Figure 3 It is a flow chart of constructing watermark trigger samples in the domain;

[0046] Figure 4 is the flowchart of the tokenized model;

[0047] Figure 5 It is a flow chart for screening out target watermark trigger samples;

[0048] Figure 6 It is a structural block diagram of a model watermark device for a speech synthesis system provided by an embodiment of the present application;

[0049] Figure 7 It is a structural block diagram of a model watermark device for a speech synthesis system provided in another embodiment of the present application.

[0050] In the figure: 102, processor; 104, memory; 106, transmission device; 108, input and output device; 210, construction module; 220, implantation module; 230, screening module; 240, detection module. DETAILED DESCRIPTION

[0051] In order to more clearly understand the purpose, technical solutions and advantages of the present application, the present application is described and illustrated below in conjunction with the accompanying drawings and embodiments.

[0052] Unless otherwise defined, the technical terms or scientific terms involved in this application shall have the general meaning understood by people with general skills in the technical field to which this application belongs. The words "one", "a", "the", "these" and the like in this application do not indicate a quantitative limitation, and they may be singular or plural. The terms "include", "comprise", "have" and any variants thereof involved in this application are intended to cover non-exclusive inclusions; for example, a process, method and system, product or device comprising a series of steps or modules (units) is not limited to the listed steps or modules (units), but may include unlisted steps or modules (units), or may include other steps or modules (units) inherent to these processes, methods, products or devices. The words "connect", "connected", "coupled" and the like involved in this application are not limited to physical or mechanical connections, but may include electrical connections, whether direct or indirect. The "multiple" involved in this application refers to two or more. "And / or" describes the association relationship of associated objects, indicating that there may be three relationships, for example, "A and / or B" may mean: A exists alone, A and B exist at the same time, and B exists alone. Generally, the character " / " indicates that the objects associated with each other are in an "or" relationship. The terms "first", "second", "third", etc. in this application are only used to distinguish similar objects and do not represent a specific ordering of the objects.

[0053] The method embodiment provided in this embodiment can be executed in a terminal, a computer or a similar computing device. For example, running on a terminal, Figure 1 FIG. 1 is a hardware structure block diagram of a terminal of the model watermark method for a speech synthesis system of this embodiment. Figure 1 As shown, the terminal may include one or more ( Figure 1 Only one is shown in the figure) a processor 102 and a memory 104 for storing data, wherein the processor 102 may include but is not limited to a processing device such as a microprocessor MCU or a programmable logic device FPGA. The above terminal may also include a transmission device 106 and an input and output device 108 for communication functions. It can be understood by those skilled in the art that Figure 1 The structure shown is only for illustration and does not limit the structure of the above terminal. Figure 1 More or fewer components as shown, or with Figure 1 Different configurations shown.

[0054] The memory 104 can be used to store computer programs, for example, software programs and modules of application software, such as the computer program corresponding to the model watermark method for speech synthesis system in this embodiment. The processor 102 executes various functional applications and data processing by running the computer program stored in the memory 104, that is, to implement the above method. The memory 104 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some examples, the memory 104 may further include a memory remotely arranged relative to the processor 102, and these remote memories may be connected to the terminal via a network. Examples of the above network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0055] The transmission device 106 is used to receive or send data via a network. The above network includes a wireless network provided by a communication provider of the terminal. In one example, the transmission device 106 includes a network adapter (Network Interface Controller, referred to as NIC), which can be connected to other network devices through a base station so as to communicate with the Internet. In one example, the transmission device 106 can be a radio frequency (Radio Frequency, referred to as RF) module, which is used to communicate with the Internet wirelessly.

[0056] In this embodiment, a model watermark method for a speech synthesis system is provided. Figure 2 is a flow chart of the model watermarking method for the speech synthesis system of this embodiment. Figure 2 As shown, the process includes the following steps:

[0057] Step S210, constructing a watermark trigger sample in the domain based on the pre-trained speech synthesis model and the fine-tuned speaker recognition model; and constructing a corresponding watermark implantation data set based on the watermark trigger sample; the training sample of the speech synthesis model is the speech data set in the domain;

[0058] Step S220, based on the fine-tuned speaker recognition model, the watermark implantation data set is implanted into the speech synthesis model to obtain a tokenized model; the speaker recognition model is fine-tuned based on the in-domain speech data set;

[0059] Step S230: Filter out target watermark trigger samples based on the tokenization model.

[0060] Specifically, the speech synthesis model is obtained by pre-training the in-domain speech data set as a training sample, and is used as a model for speech synthesis. The in-domain speech data set specifically refers to the speech spoken by the speakers who have appeared in the training sample; correspondingly, the out-of-domain speech data set refers to the speech produced by the speakers who are not included in the specific in-domain speech data set. The speaker recognition model is obtained by fine-tuning the pre-trained speaker recognition model based on the in-domain speech data set, and it is used to identify the speaker in the speech. The speech synthesis model and the speaker recognition model can be obtained from a third-party open source database or network, and there is no restriction on this. Among them, the speech synthesis system includes text analysis, acoustic model, front-end and back-end processing, control interface, speech synthesis model, speaker recognition model and other parts, which will not be elaborated.

[0061] It is important to know that the current model watermark triggering method mainly relies on out-of-domain sample triggering. This method achieves watermark triggering by using samples that are not in the domain training set or injecting a universal trigger on any speech sample. However, the distribution of watermark trigger samples constructed by this type of method in the feature space of the main task of the model is too different from that of the in-domain data, that is, the coupling with the main task of the model is poor, resulting in poor robustness of the obtained model watermark and difficulty in resisting model fine-tuning. Therefore, based on the pre-trained speech synthesis model and the fine-tuned speaker recognition model, watermark trigger samples in the domain are constructed to force the strong coupling between the watermark trigger task and the main task of the model from the data level, improve the robustness of the watermark in the tokenized model, and make the target watermark trigger sample stable and effective. There are many ways to construct watermark trigger samples in the domain, such as: using the in-domain speech data set of the speech synthesis model to fine-tune the pre-trained speaker recognition model, and then using the fine-tuned speaker recognition model to generate watermark trigger samples; or, forming a new fine-tuned language data set by combining the in-domain speech data set of the speech synthesis model and the speech data set of the pre-fine-tuned speaker recognition model, re-fine-tuning the speaker recognition model, and then using the re-fine-tuned speaker recognition model to generate watermark trigger samples, etc., and there are no restrictions on this.

[0062] Since the fine-tuned speaker recognition model and speech synthesis model are both associated with the in-domain speech dataset, the in-domain speech dataset of the speech synthesis model is input into the fine-tuned speaker recognition model, and the speech synthesis model is optimized using the watermark activation loss function, and the watermark is implanted into the speech synthesis model; the speech synthesis model is fine-tuned using a multi-task learning strategy, and the main synthesis task and watermark activation task of the speech synthesis model are trained simultaneously to ensure that the main synthesis task is not affected by the watermark implantation task, and obtain a watermarked tokenized model. Among them, the main synthesis task refers to the speech synthesis task of the speech synthesis model itself. During the watermarking process, it is necessary to ensure that the original function of the speech synthesis model is not affected, that is, to achieve harmlessness.

[0063] After obtaining the tokenization model, the tokenization model is used to screen out the target watermark trigger samples. It can be considered that the tokenization model is fine-tuned to construct multiple alternative models, the validity of the watermark trigger samples is tested on the alternative models, and the watermark trigger samples with high average speaker similarity scores are retained as target watermark trigger samples, thereby further improving the stability and validity of the watermark information. In other embodiments, the watermark trigger samples processed by the tokenization model can be screened out as target watermark trigger samples by other screening conditions, and this is not limited.

[0064] Through the above steps, based on the pre-trained speech synthesis model and the fine-tuned speaker recognition model, the watermark trigger samples in the domain are constructed; and based on the watermark trigger samples, the corresponding watermark implantation data set is constructed; based on the fine-tuned speaker recognition model, the watermark implantation data set is implanted into the speech synthesis model to obtain a tokenized model; based on the tokenized model, the target watermark trigger samples are screened out, which solves the problem that the watermark and the main task of speech synthesis are not strongly correlated in the related technology, resulting in the watermark being easily lost during the model fine-tuning process. By constructing the watermark trigger samples in the domain, the strong coupling between the watermark trigger task and the main task of the model is forced from the data level, thereby improving the robustness of the watermark in the tokenized model and ensuring the stability and effectiveness of the target watermark trigger samples.

[0065] The following is a detailed description of each of the above steps:

[0066] In some of the embodiments, Figure 3 As shown, in step S210, the watermark trigger sample in the domain is constructed based on the pre-trained speech synthesis model and the fine-tuned speaker recognition model, including the following steps:

[0067] Step S211, obtaining an in-domain speech data set corresponding to a pre-trained speech synthesis model in a speech synthesis system;

[0068] Step S212, fine-tuning the pre-trained speaker recognition model based on the in-domain speech data set to obtain a fine-tuned speaker recognition model;

[0069] Step S213, based on the fine-tuned speaker recognition model, extract timbre features from the in-domain speech data set to obtain timbre features; cluster the feature centers of the timbre features, and select source categories from the clustering results;

[0070] Step S214, constructing a watermark trigger sample in the domain based on the source category.

[0071] Specifically, the speech synthesis model accepts text and speech as input and outputs synthesized speech. The text is the speech content corresponding to the synthesized speech, and the speech is the speaker-related features such as the timbre corresponding to the synthesized speech. The speech synthesis model is pre-trained on a large-scale in-domain speech dataset, and the in-domain speech dataset is used as a training sample for the pre-training of the speech synthesis model.

[0072] Among them, the speaker recognition model refers to a speaker recognition model trained on a data set not limited to a specific domain, such as Ecapa-TDNN, ERes2Net-v2, X-vector and other model structures. The fine-tuned speaker recognition model refers to fine-tuning the pre-trained speaker recognition model on the domain speech data set. The fine-tuning process only optimizes the shallow network parameters of the pre-trained speaker recognition model, thereby reducing the difficulty of training and improving the training efficiency, and can reduce the coupling between the speech synthesis model and the fine-tuned speaker recognition model in their respective tasks. Through fine-tuning, the speaker recognition model's ability to distinguish timbre similarity in a specific domain is improved, so that the speaker recognition model can better capture and distinguish the timbre characteristics of different speakers, thereby providing technical support for the embedding of watermarks. In other embodiments, it can also be achieved by fine-tuning the transmission ratio, loss function, etc. during the training process, and there is no limitation to this.

[0073] Since the speech content synthesized by the speech synthesis model is difficult to measure using objective indicators, the speaker recognition model cascaded after the speech synthesis model is used to judge the timbre similarity of the synthesized speech; thus, the watermark trigger result is judged based on the timbre similarity. Then, the fine-tuned speaker recognition model is applied to extract the timbre features of the in-domain speech data set to obtain the timbre features; the feature centers of different speaker categories are calculated, and the feature centers are clustered using relevant clustering algorithms. The source categories are selected from the clustering results to construct watermark trigger samples. Among them, relevant clustering algorithms include but are not limited to K-means algorithm, hierarchical clustering algorithm, and spectral clustering algorithm, etc., and there is no restriction on this.

[0074] Through this embodiment, the speaker recognition model cascaded after the speech synthesis model is used to provide an objective indicator for measuring the timbre of the speech, thereby improving the effectiveness of constructing the watermark trigger sample in the domain.

[0075] In some embodiments, step S213 extracts timbre features from the in-domain speech dataset based on the fine-tuned speaker recognition model to obtain timbre features; clusters the feature centers of the timbre features, and selects source categories from the clustering results, including the following steps:

[0076] Based on the intermediate layer representation of the fine-tuned speaker recognition model as the timbre feature vector of the input speech, the timbre feature of the speech samples in the in-domain speech dataset is extracted to obtain the timbre feature;

[0077] Cluster the feature centers of the timbre features by category to obtain the category center;

[0078] Determine the source category based on the preset distance and category center.

[0079] Specifically, the above process can be considered as 1. extracting timbre features; 2. calculating the feature center of each category and predicting the category center by K-means clustering; 3. searching for the source category closest to the category center. The specific process is:

[0080] 1. Extract timbre features. Use the intermediate layer representation of the pre-trained speaker recognition model S as the timbre feature vector of the input speech to extract the timbre features of the original speech from the speech samples in the in-domain speech dataset.

[0081] For each speech sample (Category j has label ), and the timbre characteristics are obtained; the expression of timbre characteristics is: .

[0082] 2. Calculate the feature center of each category and use K-means clustering to predict the category center.

[0083] 2.1. Calculate the feature center of each category j . Assume that category j has speech samples, the calculation formula of the feature center is: .

[0084] 2.2. K-means clustering predicts category centers.

[0085] Using the K-means clustering algorithm, the feature center of category j is Cluster into K clusters (in this example, K=4), each cluster has a category center (k is the index of the cluster). The goal of K-means clustering is to minimize the following objective function: ;

[0086] Where M is the set of all cluster centers, is the set of category centers in cluster k.

[0087] 3. Search for the source category that is closest to the category center. For the category center of each cluster synthesized by the K-means clustering algorithm , first calculate all category centers The Euclidean distance from their cluster centers is calculated, and then the category with the smallest distance is selected as the representative category of the corresponding cluster. The specific formula is:

[0088] ;

[0089] in, is closest to the cluster center The speaker category is the source category. Through the above process, we can obtain the source category set used to construct the watermark trigger sample in the domain .

[0090] In other embodiments, it can also be implemented in a similar manner as above, for example: first evaluating the scores of the speech samples in each source category, and then screening out the first language samples according to the scores to obtain watermark trigger samples, etc., which is not limited to this.

[0091] This embodiment analyzes and clusters the timbre characteristics of different speakers. By applying clustering techniques such as the K-means algorithm, the source category of the speaker closest to the cluster center can be identified, and these source categories will serve as the basis for watermark trigger samples.

[0092] In some embodiments, constructing a watermark trigger sample in the domain based on the source category in step S214 includes the following steps:

[0093] randomly extracting a first speech sample from each source category;

[0094] Preprocessing the first speech samples of each category to obtain second speech samples aligned in the time domain;

[0095] Each second speech sample is synthesized by time-domain weighted averaging to obtain a watermark trigger sample.

[0096] Specifically, a first speech sample is extracted from each source category in the source category set, and then a new audio sample is synthesized as a watermark trigger sample by weighted averaging in the time domain, and a specific pseudo target speaker label is given to the watermark trigger sample. Before synthesis, the first speech samples of each category are preprocessed to obtain the second speech samples aligned in the time domain to speed up the subsequent synthesis efficiency. The specific implementation process is introduced below:

[0097] For each source category (In this embodiment, there are 4 source categories in total), randomly select a first voice sample The first speech samples of each category are preprocessed to obtain the second speech samples aligned in the time domain; for example, the four first speech samples are subjected to silence detection, the non-speech parts are removed, and then the first speech samples are aligned in the time domain by randomly cutting the speech segments to obtain the second speech samples, so that all the obtained second speech samples have the same total duration T, which is marked as Finally, each second speech sample is synthesized by time-domain weighted averaging to obtain a watermark trigger sample.

[0098] Among them, the watermark trigger sample The expression is: .

[0099] Among them, the corresponding watermark implantation data set constructed based on the watermark trigger sample in step S210 can be considered as using the randomly synthesized timbre vector as the pseudo speaker label, and the pseudo speaker label refers to the random identity feature vector corresponding to the pseudo speaker of the watermark trigger detection category. . Assign the pseudo speaker label to the watermark trigger sample , thereby giving the trigger signal specific identity attributes to facilitate subsequent copyright authentication detection.

[0100] Through the above process, the watermark trigger sample is given a specific pseudo-target speaker label, thereby constructing a watermark implantation dataset dedicated to watermark implantation; this dataset can also avoid affecting the function of the original synthesis task of the speech synthesis model.

[0101] Among them, a large number of watermark trigger samples (about 100% of the speech dataset in a specific domain) are synthesized. 1 / 100 to 1 / 10 of the watermark embedding dataset (corresponding to the speech synthesis model) , and in-domain speech datasets Together we form a new dataset Used to fine-tune speech synthesis models. New dataset It can be expressed as .

[0102] In some of the embodiments, Figure 4 As shown, in step S220, based on the fine-tuned speaker recognition model, a watermark is implanted into the data set speech synthesis model to obtain a tokenized model, including the following steps:

[0103] Step S221, jointly optimizing the speech synthesis model according to a preset total loss function and the fine-tuned speaker recognition model to obtain a tokenized model; the total loss function is composed of a preset first loss function, a preset second loss function, and a pre-trained loss function of the speech synthesis model;

[0104] Step S222, during the joint optimization process, under the triggering of the first loss function in the total loss function, the speech synthesis model is optimized according to the speech samples of the in-domain speech data set and the fine-tuned speaker recognition model; and under the triggering of the second loss function in the total loss function, the speech synthesis model is optimized according to the watermark trigger samples of the watermark implantation data set and the fine-tuned speaker recognition model.

[0105] Specifically, the speech samples of the in-domain speech dataset are input into the fine-tuned speaker recognition model, and then the total loss function activated by the special watermark is used to optimize some network layers of the speech synthesis model, so as to implant the watermark into the speech synthesis model; in each training step, the normal speech samples in the in-domain speech dataset are also present. And the watermark trigger samples in the watermark embedding dataset .

[0106] The specific implementation process is:

[0107] Among them, the speech synthesis model is jointly optimized according to the preset total loss function and the fine-tuned speaker recognition model to obtain the tokenized model. That is, the speech synthesis model is fine-tuned using a multi-task learning strategy, and the main synthesis task and watermark activation task of the speech synthesis model are trained simultaneously to ensure that the main synthesis task is not affected by the watermark implantation task, and obtain the tokenized model with watermark implantation. The specific process is:

[0108] The watermark is implanted by fine-tuning a pre-trained speech synthesis model. Specifically, a speech synthesis model is pre-trained based on the new dataset. Fine-tune the speech synthesis model, based on the pre-training loss function of the original speech synthesis task In addition, the first loss function and the second loss function activated by the watermark are combined to obtain the total loss function, which is expressed as:

[0109] ;

[0110] in, and It is a learnable parameter. After fine-tuning the speech synthesis model with the total loss function, the dynamic loss weight will balance the loss of the first loss function and the second loss function, so that the two can be optimized synchronously and fairly, preventing over-optimization or under-optimization of a task. It can be considered that the first loss function and the second loss function are effective under different conditions, and combined with speech samples or watermark trigger samples to jointly train the speech synthesis model and realize fine-tuning the speech synthesis model. The specific joint training process is:

[0111] Among them, under the triggering of the first loss function in the total loss function, the speech synthesis model is optimized according to the speech samples of the in-domain speech dataset and the fine-tuned speaker recognition model. That is, for normal speech samples (the corresponding category is j), the speech sample It is input into the pre-trained speech synthesis model G together with any specified text t to obtain the synthesized speech. . Extracting synthetic speech using a speaker recognition model The sound characteristics of , and then use the first loss function To optimize the speech synthesis model.

[0112] Among them, the expression of the first loss function is:

[0113] ;

[0114] in, is the cosine function, which is used to determine the similarity of vectors. is the random identity feature vector; The item is used to constrain normal speech samples to not mistakenly trigger the model watermark.

[0115] Among them, under the triggering of the second loss function in the total loss function, the speech synthesis model is optimized according to the watermark trigger sample of the watermark embedding data set and the fine-tuned speaker recognition model. That is, for the watermark trigger sample , the watermark trigger sample It is input into the pre-trained speech synthesis model G together with any specified text t to obtain the synthesized speech. . Extracting synthetic speech using a speaker recognition model The sound characteristics of , using the second loss function Optimize the speech synthesis model.

[0116] Among them, the expression of the second loss function is:

[0117] ;

[0118] Then the loss function L wm (Joint first loss function and second loss function) expression is:

[0119] ;

[0120] In other embodiments, the above-mentioned similar methods may also be used, and this is not limited thereto.

[0121] Through this embodiment, the watermark activation task is trained synchronously with the main synthesis task to ensure that the embedding of the watermark will not have a negative impact on the main functions of the model; and the watermark information can be implanted into the deep structure of the model so that it can be activated when synthesizing speech.

[0122] In some of the embodiments, Figure 5 As shown, the step S230 of screening out target watermark trigger samples based on the tokenization model includes the following steps:

[0123] Step S231, fine-tuning the tokenization model to obtain multiple alternative models;

[0124] Step S232, testing the validity of the watermark triggering sample based on multiple substitution models to obtain an average speaker similarity score for each watermark triggering sample;

[0125] Step S233, screen out watermark trigger samples whose average speaker similarity scores are higher than a preset threshold as target watermark trigger samples.

[0126] Specifically, the tokenization model can be fine-tuned by fine-tuning the model parameters to obtain multiple alternative models. Then, the effectiveness of the watermark trigger samples is tested on the multiple alternative models, and the watermark trigger samples with high average speaker similarity scores are retained as the target watermark trigger samples. The specific implementation is as follows:

[0127] Take an out-of-domain speech dataset , the out-of-domain speech dataset and the in-domain dataset There are significant differences in speaker types and voice characteristics.

[0128] For the tokenization model, the model results are the same as those of the pre-trained speech synthesis model G, which is divided into multiple levels, denoted as . A differential fine-tuning strategy is used to fine-tune specific layers of the tokenization model. Preferably, for each layer , define a fine-tuning weight , which determines how much the layer is updated during fine-tuning.

[0129] The fine-tuning process can be expressed as: ;

[0130] in, are the parameters of the pre-trained layer, is based on By adjusting the value of the fine-tuning weight, the depth of fine-tuning can be controlled to obtain multiple alternative models with different characteristics. .

[0131] For each alternative model constructed , and inject the watermark trigger set into the input of the substitution model. Suppose the watermark trigger sample set is , for each trigger sample , by replacing the model Output synthesized speech .

[0132] Next, we use the fine-tuned speaker recognition model S to compare the synthesized speech Similarity to the random identity feature vector of the watermark label, For each alternative model , calculate the watermark trigger sample Average speaker similarity score across all triggers.

[0133] Among them, the expression of the average speaker similarity score is: ;

[0134] Then select the speaker with an average speaker similarity score higher than the predetermined threshold The watermark trigger sample is used as the target watermark trigger sample. That is: ;

[0135] In other embodiments, the above-mentioned similar methods may also be used, and this is not limited thereto.

[0136] Through this embodiment, watermark trigger samples with higher average similarity on the alternative model are screened out as target watermark trigger samples, so that samples that can still effectively trigger the watermark after model fine-tuning can be identified and retained; this helps to further improve the robustness of the watermark and the accuracy of detection.

[0137] In some of the embodiments, the model watermark method for a speech synthesis system further includes the following steps:

[0138] After obtaining the suspicious model, the watermark trigger sample is input into the suspicious model to obtain the model output speech;

[0139] Input the model output speech into the fine-tuned speaker recognition model to obtain the model output speaker features;

[0140] The speaker features output by the model are compared with the speaker features of the pseudo speaker to obtain the watermark detection results.

[0141] Specifically, after obtaining the suspicious model, the watermark trigger sample is input into the suspicious model as the timbre source, and the model output speech is input into the fine-tuned speaker recognition model to obtain the model output speaker features, and the speaker features (identity feature vector) of the pseudo speaker label are compared for similarity, and the watermark detection result is obtained through the joint decision mechanism. Among them, the suspicious model is any third-party speech synthesis model. The specific implementation process is:

[0142] Take n watermark trigger samples as the timbre source input to the suspicious model and output the model speech . The suspicious synthesized speech Extracting model output speaker features from a fine-tuned speaker recognition model , calculate the similarity between it and the speaker features (identity feature vector) of the pseudo speaker , and compare the result with the preset threshold Compare, if the similarity is greater than the threshold , it is judged as positive, that is, the test under the current watermark trigger sample is that it contains the model watermark; if the similarity is less than or equal to the threshold , then it is judged as negative, that is, the test under the current watermark trigger sample is that it does not contain the model watermark.

[0143] Finally, the final watermark detection result is determined by the joint decision-making mechanism based on the results of all watermark trigger sample tests. The joint decision-making mechanism can be a voting mechanism or a one-vote veto mechanism, and there is no restriction on this. In the voting mechanism, assuming that there are n watermark trigger samples, and n test results are positive after the detection, the suspicious model is finally determined to be a watermark labeling model, otherwise it is determined not to be a watermark labeling model. In scenarios with higher security requirements, a one-vote veto system can be used, that is, among the n watermark detection results, as long as one result is positive, the suspicious model is determined to be a watermark labeling model.

[0144] Through the above process, it is ensured that the copyright owner can effectively verify and claim; and the joint decision-making mechanism can be adjusted to meet usage scenarios with different security requirements and improve user experience.

[0145] Through the above-mentioned method embodiments, it can be considered that the present application provides a robust, controllable, and user-friendly watermarking technology. The watermarking technology should be designed according to the characteristics of the speech synthesis model to ensure that the watermark remains effective after the speech synthesis model undergoes fine-tuning and optimization. The watermarking technology is highly controllable without affecting the quality of the original speech signal, so that the copyright owner can adjust the watermark embedding and extraction process according to needs.

[0146] It should be noted that the steps shown in the above process or the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0147] In this embodiment, a model watermark device for a speech synthesis system is also provided, which is used to implement the above embodiments and preferred implementation modes, and the descriptions that have been made are omitted. The terms "module", "unit", "subunit", etc. used below can implement a combination of software and / or hardware for a predetermined function. Although the devices described in the following embodiments are preferably implemented in software, the implementation of hardware, or a combination of software and hardware, is also possible and conceivable.

[0148] Figure 6 is a structural block diagram of the model watermarking device for the speech synthesis system of this embodiment. Figure 6 As shown, the device includes: a construction module 210, an implantation module 220 and a screening module 230;

[0149] A construction module 210 is used to construct a watermark trigger sample in the domain based on a pre-trained speech synthesis model and a fine-tuned speaker recognition model; and to construct a corresponding watermark implantation data set based on the watermark trigger sample; the training sample of the speech synthesis model is the speech data set in the domain;

[0150] An implantation module 220 is used to implant the watermark implantation data set into the speech synthesis model based on the fine-tuned speaker recognition model to obtain a tokenized model; the speaker recognition model is obtained based on the fine-tuning of the in-domain speech data set;

[0151] The screening module 230 is used to screen out target watermark trigger samples based on the tokenization model.

[0152] Through the above-mentioned device, the problem in the related technology that the watermark is not strongly correlated with the main task of speech synthesis, which makes the watermark easy to be lost during the model fine-tuning process, is solved. By constructing watermark trigger samples in the domain, a strong coupling between the watermark trigger task and the main task of the model is forced from the data level, thereby improving the robustness of the watermark in the tokenization model and ensuring the stability and effectiveness of the target watermark trigger samples.

[0153] In some of the embodiments, the construction module 210 is further used to obtain an in-domain speech data set corresponding to a pre-trained speech synthesis model in the speech synthesis system;

[0154] Based on the in-domain speech dataset, fine-tune the pre-trained speaker recognition model to obtain a fine-tuned speaker recognition model;

[0155] Based on the fine-tuned speaker recognition model, the timbre features of the in-domain speech dataset are extracted to obtain the timbre features; the feature centers of the timbre features are clustered, and the source category is selected from the clustering results;

[0156] Construct watermark trigger samples within the domain based on source categories.

[0157] In some of the embodiments, the construction module 210 is further used to extract timbre features from speech samples in the in-domain speech dataset based on the intermediate layer representation of the fine-tuned speaker recognition model as the timbre feature vector of the input speech to obtain timbre features;

[0158] Cluster the feature centers of the timbre features by category to obtain the category center;

[0159] Determine the source category based on the preset distance and category center.

[0160] In some of the embodiments, the construction module 210 is further used to randomly extract a first speech sample from each source category;

[0161] Preprocessing the first speech samples of each category to obtain second speech samples aligned in the time domain;

[0162] Each second speech sample is synthesized by time-domain weighted averaging to obtain a watermark trigger sample.

[0163] In some of the embodiments, the implantation module 220 is further used to jointly optimize the speech synthesis model according to a preset total loss function and a fine-tuned speaker recognition model to obtain a tokenized model; the total loss function is composed of a preset first loss function, a preset second loss function, and a pre-trained loss function of the speech synthesis model;

[0164] In the joint optimization process, under the triggering of the first loss function in the total loss function, the speech synthesis model is optimized according to the speech samples of the in-domain speech dataset and the fine-tuned speaker recognition model; and under the triggering of the second loss function in the total loss function, the speech synthesis model is optimized according to the watermark trigger samples of the watermark implantation dataset and the fine-tuned speaker recognition model.

[0165] In some of the embodiments, the screening module 230 is further used to fine-tune the tokenization model to obtain a plurality of alternative models;

[0166] Based on multiple substitution models, the effectiveness of watermark trigger samples is tested to obtain the average speaker similarity score of each watermark trigger sample;

[0167] The watermark trigger samples whose average speaker similarity scores are higher than a preset threshold are selected as target watermark trigger samples.

[0168] In some of the embodiments, Figure 7 As shown, the model watermarking device for speech synthesis system is Figure 6 Based on the above, in addition to the construction module 210, the implantation module 220, and the screening module 230, it also includes a detection module 240;

[0169] The detection module 240 is used to input the watermark trigger sample into the suspicious model to obtain the model output speech after obtaining the suspicious model;

[0170] Input the model output speech into the fine-tuned speaker recognition model to obtain the model output speaker features;

[0171] The speaker features output by the model are compared with the speaker features of the pseudo speaker to obtain the watermark detection results.

[0172] It should be noted that the above modules can be functional modules or program modules, and can be implemented by software or hardware. For modules implemented by hardware, the above modules can be located in the same processor; or the above modules can be located in different processors in any combination.

[0173] In this embodiment, a computer device is further provided, including a memory and a processor, wherein a computer program is stored in the memory, and the processor is configured to run the computer program to execute the steps in any one of the above method embodiments.

[0174] Optionally, the computer device may further include a transmission device and an input / output device, wherein the transmission device is connected to the processor, and the input / output device is connected to the processor.

[0175] Optionally, in this embodiment, the processor may be configured to perform the following steps through a computer program:

[0176] S1, based on the pre-trained speech synthesis model and the fine-tuned speaker recognition model, construct the watermark trigger sample in the domain; and construct the corresponding watermark implantation data set based on the watermark trigger sample; the training sample of the speech synthesis model is the speech data set in the domain;

[0177] S2, based on the fine-tuned speaker recognition model, the watermarked dataset is implanted into the speech synthesis model to obtain a tokenized model; the speaker recognition model is fine-tuned based on the in-domain speech dataset;

[0178] S3, based on the tokenization model, filters out the target watermark trigger samples.

[0179] It should be noted that the specific examples in this embodiment can refer to the examples described in the above embodiments and optional implementation modes, and will not be repeated in this embodiment.

[0180] In addition, in combination with the model watermark method for speech synthesis system provided in the above embodiments, a storage medium can also be provided in this embodiment to implement the model watermark method for speech synthesis system. The storage medium stores a computer program; when the computer program is executed by the processor, any one of the model watermark methods for speech synthesis system in the above embodiments is implemented.

[0181] It should be noted that the information and data involved in this application are all authorized by the user or fully authorized by all parties and will be used legally.

[0182] It should be understood that the specific embodiments described herein are only used to explain the application, rather than to limit it. Based on the embodiments provided in this application, all other embodiments obtained by ordinary technicians in this field without creative work are within the protection scope of this application.

[0183] Obviously, the drawings are only some examples or embodiments of the present application. For ordinary technicians in the field, the present application can also be applied to other similar situations based on these drawings without creative work. In addition, it is understandable that although the work done in this development process may be complicated and lengthy, for ordinary technicians in the field, certain changes in design, manufacturing or production based on the technical content disclosed in this application are only conventional technical means and should not be regarded as insufficient content disclosed in this application.

[0184] The term "embodiment" in this application refers to a specific feature, structure or characteristic described in conjunction with the embodiment that can be included in at least one embodiment of the present application. The appearance of this phrase in various locations in the specification does not necessarily mean the same embodiment, nor does it mean that it is mutually exclusive with other embodiments and is independent or optional. It is clearly or implicitly understood by those of ordinary skill in the art that the embodiments described in this application can be combined with other embodiments without conflict.

[0185] The above-mentioned embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of patent protection. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the scope of protection of the present application. Therefore, the scope of protection of the present application shall be subject to the attached claims.

Claims

1. A model watermarking method for a speech synthesis system, characterized in that: include: Based on the pre-trained speech synthesis model and the fine-tuned speaker recognition model, watermark trigger samples are constructed in the domain; A corresponding watermark implantation data set is constructed based on the watermark trigger sample; the training sample of the speech synthesis model is an in-domain speech data set; wherein the in-domain speech data set is a collection of speech spoken by speakers that have appeared in the training sample of the speech synthesis model; the watermark implantation data set is obtained by assigning a preset pseudo speaker label to the watermark trigger sample; Based on the fine-tuned speaker recognition model, the watermark implantation data set is implanted into the speech synthesis model to obtain a tokenized model; the speaker recognition model is obtained by fine-tuning the in-domain speech data set; Based on the tokenization model, target watermark trigger samples are screened out.

2. The model watermarking method for speech synthesis system according to claim 1, characterized in that: Based on the pre-trained speech synthesis model and the fine-tuned speaker recognition model, we construct watermark trigger samples in the domain, including: Acquire an in-domain speech data set corresponding to the speech synthesis model pre-trained in the speech synthesis system; Based on the in-domain speech dataset, fine-tuning the pre-trained speaker recognition model to obtain the fine-tuned speaker recognition model; Based on the fine-tuned speaker recognition model, extracting timbre features from the in-domain speech data set to obtain timbre features; clustering the feature centers of the timbre features, and selecting source categories from the clustering results; The watermark trigger samples in a domain are constructed based on the source category.

3. The model watermarking method for speech synthesis system according to claim 2, characterized in that: Based on the fine-tuned speaker recognition model, extracting timbre features from the in-domain speech dataset to obtain timbre features; Clustering the feature centers of the timbre features and selecting source categories from the clustering results includes: Based on the fine-tuned intermediate layer representation of the speaker recognition model as the timbre feature vector of the input speech, timbre features are extracted from the speech samples in the in-domain speech dataset to obtain timbre features; Clustering the feature centers of the timbre features by category to obtain category centers; The source category is determined according to the preset distance and the category center.

4. The model watermarking method for speech synthesis system according to claim 2, characterized in that: Constructing the watermark trigger sample in the domain based on the source category, comprising: Randomly extracting a first speech sample from each of the source categories; Preprocessing the first speech samples of each category to obtain second speech samples aligned in time domain; Each of the second speech samples is synthesized by time-domain weighted averaging to obtain a watermark trigger sample.

5. The model watermarking method for speech synthesis system according to claim 1, characterized in that: Based on the fine-tuned speaker recognition model, the watermark implantation data set is implanted into the speech synthesis model to obtain a tokenized model, including: According to a preset total loss function and the fine-tuned speaker recognition model, the speech synthesis model is jointly optimized to obtain a tokenized model; the total loss function is composed of a preset first loss function, a preset second loss function and a pre-trained loss function of the speech synthesis model; During the joint optimization process, under the triggering of the first loss function in the total loss function, the speech synthesis model is optimized based on the speech samples of the in-domain speech data set and the fine-tuned speaker recognition model; and under the triggering of the second loss function in the total loss function, the speech synthesis model is optimized based on the watermark trigger samples of the watermark implantation data set and the fine-tuned speaker recognition model.

6. The model watermarking method for speech synthesis system according to claim 1, characterized in that: Based on the tokenization model, target watermark trigger samples are screened out, including: Fine-tuning the tokenized model to obtain multiple alternative models; Based on the multiple substitution models, the validity of the watermark trigger sample is tested to obtain an average speaker similarity score between each watermark trigger sample and a preset pseudo speaker; The watermark trigger samples whose average speaker similarity scores are higher than a preset threshold are selected as target watermark trigger samples.

7. The model watermarking method for speech synthesis system according to any one of claims 1 to 6, characterized in that: The method further comprises: After obtaining the suspicious model, inputting the watermark trigger sample into the suspicious model to obtain the model output speech; Inputting the model output speech into the fine-tuned speaker recognition model to obtain model output speaker features; The speaker features output by the model are compared with the speaker features of the pseudo speaker to obtain a watermark detection result.

8. A model watermarking device for a speech synthesis system, characterized in that: include: Building modules, implanting modules, and screening modules; The construction module is used to construct a watermark trigger sample in the domain based on a pre-trained speech synthesis model and a fine-tuned speaker recognition model; A corresponding watermark implantation data set is constructed based on the watermark trigger sample; the training sample of the speech synthesis model is an in-domain speech data set; wherein the in-domain speech data set is a collection of speech spoken by speakers that have appeared in the training sample of the speech synthesis model; the watermark implantation data set is obtained by assigning a preset pseudo speaker label to the watermark trigger sample; The implantation module is used to implant the watermark implantation data set into the speech synthesis model based on the fine-tuned speaker recognition model to obtain a tokenized model; the speaker recognition model is obtained based on the fine-tuning of the in-domain speech data set; The screening module is used to screen out target watermark trigger samples based on the tokenization model.

9. A computer device comprising a memory and a processor, characterized in that: A computer program is stored in the memory, and the processor is configured to run the computer program to execute the steps of the model watermark method for a speech synthesis system according to any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the model watermark method for a speech synthesis system according to any one of claims 1 to 7 are implemented.

Images