Test method, device, equipment and medium for speech synthesis model

Abstract

The application relates to the technical field of artificial intelligence, and discloses a test method and device for a speech synthesis model, equipment and a medium, belongs to the technical field of artificial intelligence, and is applied to financial and medical technology scenes, and comprises the following steps: determining error judgment data corresponding to a target speech synthesis model; performing speech duration prediction on training text through the model to obtain predicted speech duration data; performing duration error calculation according to real speech duration data and the predicted speech duration data to obtain target test error data; and performing model capability analysis on the model according to the error judgment data and the target test error data to obtain model test information corresponding to the model. Through comparison of error differences between real duration and predicted duration and analysis of model capability in combination with error judgment data, the application effectively evaluates the memory degree of the model to training samples, captures potential privacy leakage risks, and improves the privacy security protection capability of the model to training data.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, applicable to fintech and medical technology scenarios, and particularly to a testing method, apparatus, device, and medium for speech synthesis models. Background Technology

[0002] Currently, traditional speech synthesis model training methods typically analyze the probability of synthesized speech data output by speech synthesis models (such as FastSpeech) based on large-scale training text and its annotated speech data, or perform multi-step decoding and reconstruction error processing to achieve supervised model training. For example, in insurance scenarios, to achieve automatic voice broadcasting for policy consultations, renewal reminders, or claims notifications, real call recordings of insurance customer service personnel are obtained as training data. The recordings are converted into text and annotated with corresponding acoustic parameters such as speech duration and pitch. The speech synthesis model is then trained using the training data and acoustic parameters, enabling the model to learn the mapping relationship from text to speech. The trained model is then used to synthesize speech for insurance business text. In medical scenarios, to achieve intelligent voice guidance, it is necessary to collect recordings of guidance personnel as training corpus. The speech synthesis model is trained to learn the mapping relationship from guidance text to speech, and the trained model is then used to convert the medical prompt text to be broadcast into the corresponding synthesized speech. However, this method can create a "memory effect" on training samples during model training. This allows attackers to determine whether training samples were previously used for model training by inputting specific text into the trained speech synthesis model and analyzing the model's output features (such as prediction confidence and detailed features of generated speech). This allows attackers to infer whether the training data contains specific users' private information, leading to a serious risk of privacy leakage from the training samples. Therefore, how to implement privacy protection performance testing of training samples during the training process of speech synthesis models to prevent the leakage of training samples has become an urgent problem to be solved. Summary of the Invention

[0003] This invention provides a testing method, apparatus, computer equipment, and medium for speech synthesis models, in order to solve the technical problem of how to test the privacy protection performance of training samples during the training process of speech synthesis models, so as to prevent the leakage of training samples of speech synthesis models.

[0004] Firstly, a testing method for speech synthesis models is provided, including: Obtain the target speech synthesis model and determine the error judgment data corresponding to the target speech synthesis model; Obtain target test samples; wherein, the target test samples include training text and real speech duration data corresponding to the training text; The training text is predicted by the target speech synthesis model to obtain the predicted speech duration data corresponding to the training text. The duration error is calculated based on the actual speech duration data and the predicted speech duration data to obtain the target test error data; Based on the error determination data and the target test error data, the model capability analysis of the target speech synthesis model is performed to obtain the model test information corresponding to the target speech synthesis model.

[0005] Secondly, a testing device for a speech synthesis model is provided, comprising: The model and error determination data acquisition module is used to acquire the target speech synthesis model and determine the error determination data corresponding to the target speech synthesis model. A test sample acquisition module is used to acquire target test samples; wherein, the target test samples include training text and real speech duration data corresponding to the training text; The speech duration prediction module is used to predict the speech duration of the training text through the target speech synthesis model to obtain the predicted speech duration data corresponding to the training text. The duration error calculation module is used to calculate the duration error based on the actual speech duration data and the predicted speech duration data to obtain the target test error data. The model capability analysis module is used to perform model capability analysis on the target speech synthesis model based on the error judgment data and the target test error data, and obtain model test information corresponding to the target speech synthesis model.

[0006] Thirdly, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the above-described testing method for a speech synthesis model.

[0007] Fourthly, a computer-readable storage medium is provided, which stores a computer program that, when executed by a processor, implements the steps of the above-described test method for a speech synthesis model.

[0008] The aforementioned testing method, apparatus, equipment, and medium for speech synthesis models can be implemented by: acquiring a target speech synthesis model and determining the error judgment data corresponding to the target speech synthesis model; acquiring target test samples including training text and real speech duration data corresponding to the training text; predicting the speech duration of the training text using the target speech synthesis model to obtain the predicted speech duration data corresponding to the training text; calculating the duration error based on the real speech duration data and the predicted speech duration data to obtain the target test error data; and then analyzing the model capability of the target speech synthesis model based on the error judgment data and the target test error data to obtain the model test information corresponding to the target speech synthesis model. In this application, by introducing a differential comparison mechanism based on speech duration features, the duration error can be accurately extracted as a benchmark for sample privacy leakage analysis simply by using the target speech synthesis model for forward duration prediction. This allows for the identification and suppression of excessive memorization of training samples with privacy leakage risks during the model training phase. This not only effectively avoids the shortcomings of traditional speech synthesis methods in lacking constraints on the privacy leakage risks of training samples and overcomes the limitations of traditional testing methods that require multi-step decoding and reconstruction or complex output feature analysis, but also significantly reduces the computational complexity of model testing. It enables rapid and accurate evaluation of the privacy protection performance of the model with low computational overhead, providing reliable model testing information for preventing training sample leakage. This improves the efficiency and accuracy of security auditing of speech synthesis models before practical application and deployment, thereby enhancing the privacy protection capabilities of speech synthesis models for training data. Attached Figure Description

[0009] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the description of the embodiments of the present invention will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0010] Figure 1 This is a schematic diagram of an application environment for a testing method for a speech synthesis model according to an embodiment of the present invention; Figure 2 This is a flowchart illustrating a testing method for a speech synthesis model according to an embodiment of the present invention; Figure 3 yes Figure 2 A schematic diagram of a specific implementation of step S101; Figure 4 This is a flowchart illustrating a testing method for a speech synthesis model according to an embodiment of the present invention; Figure 5 yes Figure 2A schematic diagram of a specific implementation of step S103; Figure 6 yes Figure 2 A schematic diagram of a specific implementation of step S104; Figure 7 yes Figure 2 A schematic diagram of a specific implementation of step S105; Figure 8 yes Figure 2 A flowchart illustrating another specific implementation of step S105; Figure 9 This is a schematic diagram of a testing device for a speech synthesis model according to an embodiment of the present invention; Figure 10 This is a schematic diagram of the structure of a computer device according to an embodiment of the present invention; Figure 11 This is another structural schematic diagram of a computer device according to one embodiment of the present invention. Detailed Implementation

[0011] The testing method for speech synthesis models provided in this application relates to the field of artificial intelligence technology.

[0012] First, let's analyze some of the terms used in this application: Artificial intelligence (AI) is a new branch of computer science that studies, develops, and applies theories, methods, technologies, and systems to simulate, extend, and expand human intelligence. It aims to understand the essence of intelligence and produce intelligent machines that can react in a way similar to human intelligence. Research in this field includes robotics, speech recognition, image recognition, natural language processing, and expert systems. AI can simulate the information processes of human consciousness and thought. Furthermore, AI utilizes digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceiving the environment, acquiring knowledge, and using that knowledge to achieve optimal results.

[0013] The embodiments of this application can acquire and process relevant data based on artificial intelligence technology. Artificial intelligence (AI) refers to the theories, methods, technologies, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to obtain optimal results.

[0014] Foundational technologies for artificial intelligence generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing, operating / interactive systems, and mechatronics. AI software technologies mainly encompass computer vision, robotics, biometrics, speech processing, natural language processing, and machine learning / deep learning.

[0015] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0016] The testing method for speech synthesis models provided in this invention can be applied to, for example... Figure 1 In this application environment, the client communicates with the server via a network. The server can obtain the target speech synthesis model and determine the error judgment data corresponding to the target speech synthesis model; obtain target test samples, including training text and the corresponding real speech duration data; predict the speech duration of the training text using the target speech synthesis model to obtain the predicted speech duration data corresponding to the training text; calculate the duration error based on the real speech duration data and the predicted speech duration data to obtain the target test error data; and then analyze the model capability of the target speech synthesis model based on the error judgment data and the target test error data to obtain the model test information corresponding to the target speech synthesis model. In this application, by introducing a differential comparison mechanism based on speech duration features, the duration error can be accurately extracted as a benchmark for sample privacy leakage analysis simply by using the target speech synthesis model for forward duration prediction. This allows for the identification and suppression of excessive memorization of training samples with privacy leakage risks during the model training phase. This not only effectively avoids the shortcomings of traditional speech synthesis methods in lacking constraints on the privacy leakage risks of training samples and overcomes the limitations of traditional testing methods that require multi-step decoding and reconstruction or complex output feature analysis, but also significantly reduces the computational complexity of model testing. It achieves rapid and accurate evaluation of the model's privacy protection performance with lower computational overhead, providing reliable model testing information for preventing training sample leakage. This improves the efficiency and accuracy of security audits of speech synthesis models before practical application and deployment, thereby enhancing the speech synthesis model's ability to protect the privacy of training data. The client can be, but is not limited to, various personal computers, laptops, smartphones, tablets, and portable wearable devices. The server can be implemented using a standalone server or a server cluster consisting of multiple servers. The invention will be described in detail below through specific embodiments.

[0017] like Figure 2 As shown, Figure 2 A flowchart illustrating a testing method for a speech synthesis model provided in an embodiment of the present invention may include, but is not limited to, steps S101 to S105: Step S101: Obtain the target speech synthesis model and determine the error judgment data corresponding to the target speech synthesis model.

[0018] Step S102: Obtain target test samples; wherein, the target test samples include training text and real speech duration data corresponding to the training text.

[0019] Step S103: The training text is predicted for speech duration using the target speech synthesis model to obtain the predicted speech duration data corresponding to the training text.

[0020] Step S104: Calculate the duration error based on the actual speech duration data and the predicted speech duration data to obtain the target test error data.

[0021] Step S105: Analyze the model capability of the target speech synthesis model based on the error judgment data and the target test error data to obtain the model test information corresponding to the target speech synthesis model.

[0022] Steps S101 to S105 as shown in the embodiments of this application involve: acquiring a target speech synthesis model and determining error judgment data corresponding to the target speech synthesis model; acquiring target test samples including training text and real speech duration data corresponding to the training text; predicting the speech duration of the training text using the target speech synthesis model to obtain predicted speech duration data corresponding to the training text; calculating the duration error based on the real speech duration data and the predicted speech duration data to obtain target test error data; and then performing model capability analysis on the target speech synthesis model based on the error judgment data and the target test error data to obtain model test information corresponding to the target speech synthesis model. In this application, by introducing a differential comparison mechanism based on speech duration features, the duration error can be accurately extracted as a benchmark for sample privacy leakage analysis simply by using the target speech synthesis model for forward duration prediction. This allows for the identification and suppression of excessive memorization of training samples with privacy leakage risks during the model training phase. This not only effectively avoids the shortcomings of traditional speech synthesis methods in lacking constraints on the privacy leakage risks of training samples and overcomes the limitations of traditional testing methods that require multi-step decoding and reconstruction or complex output feature analysis, but also significantly reduces the computational complexity of model testing. It enables rapid and accurate evaluation of the privacy protection performance of the model with low computational overhead, providing reliable model testing information for preventing training sample leakage. This improves the efficiency and accuracy of security auditing of speech synthesis models before practical application and deployment, thereby enhancing the privacy protection capabilities of speech synthesis models for training data.

[0023] In step S101 of some embodiments, specifically, a target speech synthesis model is obtained, and error judgment data corresponding to the target speech synthesis model is determined.

[0024] like Figure 3 As shown, in some embodiments, step S101 includes, but is not limited to, steps S201 to S204: Step S201: Obtain the target speech synthesis model trained based on the first training sample set, and obtain the second training sample set; wherein, the first training sample set refers to the samples trained in the target speech synthesis model; the second training sample set refers to the samples not trained in the target speech synthesis model.

[0025] Step S202: Based on each of the first type of samples in the first training sample set, perform error analysis on the target speech synthesis model to obtain the first error distribution data.

[0026] Step S203: Based on each second type of sample in the second training sample set, perform error analysis on the target speech synthesis model to obtain the second error distribution data.

[0027] Step S204: Perform error distribution analysis based on the first error distribution data and the second error distribution data to obtain error determination data.

[0028] like Figure 4 As shown, in some embodiments, the first training sample set includes each first training text and the first real duration data corresponding to each first training text. Before step S201, the testing method for the speech synthesis model also includes, but is not limited to, steps S301 to S303: Step S301: Phoneme conversion is performed on each of the first training texts to obtain the first training phoneme sequence.

[0029] Step S302: Perform phoneme classification on the first training phoneme sequence to obtain phoneme classification data.

[0030] Step S303: Based on the phoneme classification data, perform data perturbation processing on the first real duration data to obtain perturbed real duration data, and determine the perturbed real duration data as the first training sample set.

[0031] In step S301 of some embodiments, specifically, the first training phoneme sequence refers to the sequence data obtained after converting the characters in the first training text into corresponding basic phonetic units.

[0032] Specifically, text normalization processing can be performed on each first training text to convert numbers, dates, amounts, abbreviations, and special characters in the first training text into standard text expressions. The normalized first training text can then be segmented to obtain multiple text word units. The standard pronunciation annotations corresponding to each text word unit can be queried from a preset pronunciation dictionary and converted into corresponding phoneme representations. The converted phoneme representations can then be sequentially concatenated according to the arrangement order of each text word unit in the first training text to obtain the first training phoneme sequence.

[0033] For example, in financial applications, the first training text "Your current bill has been settled" can be standardized and segmented to obtain text word units such as "you", "of", "current", "bill", "already", and "settled". Then, the standard pronunciation annotations corresponding to each text word unit can be queried from a preset pronunciation dictionary, and each standard pronunciation annotation can be converted into a phoneme representation. Finally, they can be concatenated in order to obtain the corresponding first training phoneme sequence. In medical applications, the first training text "Please go to the first floor pharmacy to pick up the medicine" can be standardized and segmented to obtain text word units such as "please", "go", "first floor", "pharmacy", and "pick up the medicine". Then, the corresponding standard pronunciation annotations can be queried from a preset pronunciation dictionary and converted into phoneme representations to obtain the corresponding first training phoneme sequence.

[0034] In step S302 of some embodiments, specifically, phoneme classification data refers to a set of labels or mappings obtained by classifying each phoneme unit in the first training phoneme sequence according to acoustic properties, pronunciation features, or duration sensitivity.

[0035] Specifically, preset phoneme classification rules can be obtained (such as classification conditions for vowels, consonants, silences, pauses, or special punctuation marks defined based on acoustic feature classification templates); according to the phoneme classification rules, acoustic attribute matching is performed on each phoneme unit in the first training phoneme sequence to determine the phoneme category to which each phoneme unit belongs, and the mapping relationship between each phoneme unit and the corresponding phoneme category is summarized to obtain phoneme classification data.

[0036] For example, in financial applications, for the word "claims" in the first training phoneme sequence (the corresponding pinyin phonemes include "l", "i", "p", and "ei"), "l" and "p" can be classified as consonants, and "i" and "ei" as vowels, using phoneme classification rules. The classification results are then summarized into phoneme classification data. In medical applications, for the first training phoneme sequence containing pauses, the pronunciation units can be classified as vowels and consonants, and the silence units corresponding to punctuation marks can be classified as silence pauses, thus obtaining phoneme classification data for the corresponding medical corpus.

[0037] In step S303 of some embodiments, specifically, the dynamic real duration data refers to the duration data obtained by applying differential noise to the first real duration data that was originally strictly aligned with the text, in combination with the above-mentioned category information.

[0038] Specifically, based on phoneme classification data, the target perturbation strategy and perturbation amplitude corresponding to different phoneme categories in the first training phoneme sequence can be determined (e.g., a larger random perturbation range is set for silent pause phonemes that are less sensitive to duration fluctuations, and a smaller random perturbation range is set for vowel phonemes that are highly sensitive to duration fluctuations). According to the target perturbation strategy, corresponding random noise is injected into the first real duration data corresponding to each phoneme unit to obtain the perturbed real duration data of each phoneme unit. Finally, the first training text carrying the perturbed real duration data and the phoneme sequence data are combined to determine the first training sample set.

[0039] For example, in financial applications, when the first real duration data indicates that the duration of the vowel "u" ​​in "renewal" is 0.10 seconds and the duration of the sentence-end pause is 0.50 seconds, a fine-tuning noise with a range of [-0.01, 0.01] seconds is applied to the vowel "u" ​​based on phoneme classification data (becoming 0.105 seconds after perturbation), and a large-amplitude noise with a range of [-0.10, 0.10] seconds is applied to the sentence-end pause (becoming 0.45 seconds after perturbation). The updated overall duration data is then included in the first training sample set as perturbed real duration data. In medical applications, when processing the first real duration data related to "payment", a small Gaussian random perturbation is introduced to the consonant and vowel units based on phoneme classification data. While preserving normal auditory prosody, the deterministic "duration fingerprint" left on the samples by the original aligner is removed, and the processed data is determined as the first training sample set.

[0040] Through steps S301 to S303, by introducing random perturbations for specific phoneme categories at the source of model training, the distribution of duration target error between different samples is forced to be smoother, avoiding the "unique fingerprint" characteristic of generation duration for a single sample. Thus, without compromising the overall listening quality of speech generation, the model effectively avoids overfitting and illegal memorization of duration features of specific samples.

[0041] In step S201 of some embodiments, specifically, the first training sample set refers to samples trained in the target speech synthesis model, and the second training sample set refers to samples that have not been trained in the target speech synthesis model, but are comparable to the first training sample set in terms of data type, scene type, or text structure.

[0042] Specifically, a target speech synthesis model trained on the first training sample set can be obtained, and a portion of known samples that participated in the training can be selected from historical training data as the first training sample set. A portion of samples can be selected from independent validation sets, independent test sets, or external business data that did not participate in the training as the second training sample set, so as to jointly construct reference data for model testing.

[0043] For example, in financial applications, a first training sample set can be extracted from voice samples already used for model training, such as insurance renewal reminders and claims notifications, and a second training sample set can be extracted from financial customer service recordings or broadcast text samples that have not been included in the training process; in medical applications, a first training sample set can be extracted from triage recordings that have been used in the triage model training, and a second training sample set can be extracted from hospital broadcast corpora that have not been used in the training.

[0044] In step S202 of some embodiments, specifically, the first error distribution data refers to the error data distribution result formed after the first type of samples in the first training sample set are subjected to duration error detection. The first error distribution data includes the duration standard deviation and variance data between the first type of samples, which is used to characterize the duration fitting law of the target speech synthesis model to the trained samples.

[0045] Specifically, the speech duration can be predicted for the first training text of the first type of sample using the target speech synthesis model, to obtain the first predicted speech duration data corresponding to the first training text. Then, the duration error is calculated based on the actual speech duration data and the first predicted speech duration data corresponding to the first type of sample, to obtain the first error data. Finally, the standard deviation and variance of all the first error data are calculated to obtain the first error distribution data.

[0046] In step S203 of some embodiments, specifically, the second error distribution data refers to the error data distribution result formed after the second type of samples in the second training sample set are subjected to duration error detection. The second error distribution data includes the duration standard deviation and variance data between the second type of samples, which are used to characterize the duration fitting law of the target speech synthesis model to the untrained samples.

[0047] Specifically, the speech duration can be predicted for the second training text of the second type of sample using the target speech synthesis model to obtain the second predicted speech duration data corresponding to the second training text. Then, the duration error is calculated based on the actual speech duration data and the second predicted speech duration data corresponding to the second type of sample to obtain the second error data corresponding to the second type of sample. Finally, the standard deviation and variance of all the second error data are calculated to obtain the second error distribution data.

[0048] In step S204 of some embodiments, specifically, the error determination data refers to the determination threshold or benchmark parameter calculated by comparing the first error distribution data and the second error distribution data. The error determination data is used to characterize the statistical boundary that distinguishes whether a sample has experienced privacy leakage or whether it has been memorized by the model.

[0049] Specifically, the first error distribution data and the second error distribution data can be compared in terms of distribution differences (such as calculating the intersection of distributions) to find the best judgment critical point that can distinguish the two types of distributions to the greatest extent. Based on the comparison results, the critical point parameter is set to obtain the error judgment data.

[0050] For example, in an insurance application scenario, if the mean value of the duration error distribution of the first type of samples is 0.11 based on the first training sample set, and the mean value of the duration error distribution of the second type of samples is 0.24 based on the second training sample set, then the statistical critical value of the two can be calculated analytically through the error distribution.

[0051] Furthermore, a calculation method based on the mean distance can be adopted, and the error judgment threshold can be expressed as error judgment data = (mean of first error + mean of second error) / 2. Then, the error judgment data in this insurance application scenario can be calculated as: (0.11 + 0.24) / 2 = 0.175. This indicates that if the duration error of a certain insurance training sample is less than 0.175 in subsequent detection, it has a strong attribute of being remembered by the model, and the corresponding privacy leakage risk is high.

[0052] Through steps S201 to S204, the fitting difference between known training samples and unknown samples in the duration prediction dimension of the target speech synthesis model can be used to extract the duration loss as a discrimination signal to characterize the model's memory characteristics. Furthermore, by performing a single forward inference on the two types of samples to construct error distribution features, not only is noise interference in the acoustic decoding generation process of traditional models eliminated, ensuring the accuracy and separability of the judgment benchmark, but the computational cost is also significantly reduced. This establishes an objective reference for the subsequent efficient and low-cost execution of membership discrimination and privacy leakage risk quantification.

[0053] In step S102 of some embodiments, specifically, the target test sample refers to a set of structured training data obtained for training a preset target speech synthesis model. The target test sample includes training text, training speech corresponding to the training text, and real speech duration data corresponding to the training text.

[0054] Furthermore, training text refers to training text data used to represent the content of the speech to be synthesized.

[0055] Furthermore, training speech refers to speech data that corresponds to the training text in terms of semantic content. Training speech can be single-speaker recordings, multi-speaker recordings, or standard broadcast speech collected in specific business scenarios, and training speech is used to provide speech supervision information for the target speech synthesis model.

[0056] For example, in financial applications, training text can be business announcements such as "Your policy is about to expire, please renew it in time" or "Your claim application has entered the review stage"; training voice can be audio data generated by insurance customer service personnel or standard announcers reading the above text aloud; in medical applications, training text can be guidance or prompt texts such as "Please go to the internal medicine clinic on the second floor to wait for your appointment" or "Please go to window number three to pick up your test report"; training voice can be corresponding voices recorded by guidance personnel, nurses, or hospital announcers.

[0057] Furthermore, real speech duration data refers to the actual duration data of each text unit in the training text in the corresponding training speech, which is used to characterize the temporal correspondence between the training text and the training speech, and can be used as a supervision basis for the target speech synthesis model to learn speech duration. Among them, the text unit can be a phoneme, syllable, character or word, etc.; in this embodiment, phoneme-level duration data is preferred.

[0058] In this embodiment, by acquiring target test samples including training text, training speech, and real speech duration data, the target speech synthesis model can be provided with a correspondence between text, speech, and duration supervision, providing a reliable data foundation for subsequent privacy leakage risk assessment and risk control training.

[0059] like Figure 5 As shown, in some embodiments, step S103 includes, but is not limited to, steps S401 to S403: Step S401: Perform phoneme conversion on the training text to obtain a training phoneme sequence.

[0060] Step S402: Encode the training phoneme sequence using the target speech synthesis model to obtain phoneme sequence features.

[0061] Step S403: Perform phoneme duration prediction on the training phoneme sequence features to obtain the predicted speech duration data corresponding to the training text.

[0062] In step S401 of some embodiments, specifically, training the phoneme sequence refers to training the sequence of phonetic units of the text.

[0063] Specifically, the method for converting the training text into phonemes to obtain the training phoneme sequence is the same as the method for converting each of the first training texts into phonemes to obtain the first training phoneme sequence, and will not be repeated here.

[0064] In step S402 of some embodiments, specifically, the target speech synthesis model refers to a preset speech synthesis model that has not yet been trained under sample privacy risk control. The model includes a text encoding network and a duration prediction network. The text encoding network can be a Transformer network, used to encode the phoneme sequence features corresponding to the training text to extract phoneme sequence features containing phoneme context. The duration prediction network can be used to predict the duration of each phoneme unit in the training phoneme sequence item by item to obtain the phoneme duration prediction value.

[0065] Specifically, phoneme sequence features refer to the vector representation that characterizes the pronunciation content information and contextual association information of each phoneme unit in the training phoneme sequence.

[0066] Specifically, the Transformer network can be used to convert each phoneme unit in the training phoneme sequence into a corresponding phoneme embedding vector, and the context features of the phoneme embedding vector corresponding to each phoneme unit can be extracted to obtain the phoneme sequence features corresponding to the training phoneme sequence.

[0067] In step S403 of some embodiments, specifically, the predicted speech duration data refers to the duration data obtained by predicting the pronunciation duration of each phoneme unit in the training text based on phoneme sequence features.

[0068] Specifically, the duration prediction network can be used to predict the duration of each phoneme unit in the phoneme sequence feature item by item to obtain multiple phoneme duration prediction values. Then, according to the arrangement order of each phoneme unit in the training phoneme sequence, the multiple phoneme duration prediction values ​​are combined to obtain the predicted speech duration data corresponding to the training text.

[0069] For example, in financial applications, when the training text is "Your claim application has entered the review stage," the phoneme duration prediction network can output the predicted duration of each phoneme unit based on the phoneme sequence features corresponding to the training text, and form the predicted speech duration data corresponding to the training text according to the phoneme order. In medical applications, when the training text is "Please go to window number three to collect the test report," the phoneme duration prediction network can predict the duration of the phoneme units corresponding to "please," "go," "number three," "window," "collect," and "test report," and combine them to obtain the predicted speech duration data corresponding to the training text.

[0070] Through steps S401 to S403, the training text can be converted into predicted speech duration data corresponding to the real speech duration data, and a processing link from training text to predicted speech duration data can be established. This provides a unified data foundation for subsequent duration error detection based on real speech duration data and predicted speech duration data, which in turn helps to identify the degree of fit of the target speech synthesis model to the training samples from the perspective of speech duration.

[0071] like Figure 6 As shown, in some embodiments, step S104 includes, but is not limited to, steps S501 to S504: Step S501: Align the actual speech duration data and the predicted speech duration data with the duration position to obtain aligned duration data.

[0072] Step S502: Calculate the overall duration error of the real speech duration data and the predicted speech duration data based on the aligned duration data to obtain the overall target test error data.

[0073] Step S503: Calculate the local duration error of the real speech duration data and the predicted speech duration data based on the aligned duration data to obtain the local target test error data.

[0074] Step S504: Integrate the overall target test error data and the local target test error data to obtain the target test error data.

[0075] In step S501 of some embodiments, specifically, the aligned duration data refers to the set of duration data after corresponding the actual speech duration data and the predicted speech duration data one-to-one, which is used to ensure the consistency of the comparison benchmark in the subsequent duration error calculation process.

[0076] Specifically, based on the training phoneme sequence corresponding to the training text, the real speech duration data and the predicted speech duration data can be indexed and matched according to the phoneme position order. When the durations of the real speech duration data and the predicted speech duration data are consistent, the duration positions can be directly aligned according to the phoneme positions. When there is a boundary deviation or length deviation between the two, the predicted speech duration data is corrected by boundary correction (such as padding, truncation or resampling) to obtain the aligned duration data.

[0077] For example, in financial applications, the actual and predicted speech duration data for the sentence "Your policy is about to expire, please renew it in time" can be aligned one by one according to the order of each phoneme in the text to form the aligned duration data of the insurance training sample. In medical applications, the actual and predicted speech duration data for the sentence "Please go to the internal medicine clinic on the second floor to wait for your appointment" can also be obtained in the same way to form the aligned duration data of the medical text training sample.

[0078] In step S502 of some embodiments, specifically, the overall target test error data refers to the overall duration error value between the real speech duration data and the predicted speech duration data, which is used to characterize the degree of fit of the target speech synthesis model to the overall duration structure of the current training sample.

[0079] Specifically, for each phoneme duration corresponding item in the aligned duration data, the duration deviation value between the predicted duration and the actual duration can be calculated; then, the mean square error of all phoneme duration deviation values ​​is aggregated to obtain the overall target test error data.

[0080] In step S503 of some embodiments, specifically, the local target test error data refers to the error value of a local segment between the real speech duration data and the predicted speech duration data, which is used to characterize the difference in duration fitting of the model on local phoneme segments, keyword segments or prosodic segments.

[0081] Specifically, the phoneme sequence in the aligned duration data can be divided into multiple local segments according to preset segmentation rules. Error calculations are then performed on the predicted duration and actual duration of each local segment to obtain multiple local error values. These local error values ​​are then combined to form local target test error data. Segmentation rules can be based on a fixed-length sliding window or segmentation by word, phrase, or key business field.

[0082] Furthermore, when the training text contains key business fields, the phoneme segments corresponding to the key business fields can be prioritized as local segments to enhance the detection capability of fitting the duration of key segments.

[0083] In one optional embodiment of this application, local duration error calculation is performed on real speech duration data and predicted speech duration data based on aligned duration data to obtain local target test error data. This may further include: obtaining privacy-sensitive fields in the training text, and performing field position detection based on the phoneme sequence corresponding to the training text to obtain the phoneme position interval corresponding to the privacy-sensitive fields; extracting predicted field duration data from the predicted speech duration data based on the phoneme position interval, and extracting real field duration data from the real speech duration data; and calculating segment error based on the predicted field duration data and the real field duration data to obtain local target test error data.

[0084] It should be noted that, considering the vastly different sensitivity of different words in text to privacy and security in real-world financial or medical applications, memorizing common words (such as stop words and conjunctions) does not constitute a substantial privacy breach, while memorizing privacy-sensitive fields (such as customer names, ID numbers, diagnosed diseases, and financing amounts) poses an extremely high risk. Therefore, this solution introduces a fine-grained error detection mechanism based on privacy-sensitive fields.

[0085] For example, in a medical scenario, for the test text "Patient A's diabetes test report has been issued," "Patient A" and "diabetes" are first identified as privacy-sensitive fields. If, after phoneme conversion, "Patient A" corresponds to the 3rd to 6th phoneme position range, then the predicted and actual durations of these four phonemes are specifically extracted to calculate the local target test error data. If this local target test error data is extremely small (e.g., the local error is 0.002, while the overall target test error for the entire sentence is 0.080), then the difference between a large overall error and an extremely small sensitive local error indicates that the model has developed a deep memory for this sensitive entity. This allows the generation of corresponding entity-level leakage risk warning information (e.g., the duration features of diabetes and Patient A have a risk of memory leakage), which can then be output as model test information.

[0086] For example, in a financial audit scenario, if the training text is "transfer records of corporate account ending in 8848", the local duration error corresponding to the sensitive field "8848" can be extracted. By extracting local target test error data, it is possible to accurately determine whether the model has memorized this sensitive data, thereby achieving fine-grained model privacy compliance review.

[0087] In this embodiment, by determining the local target test error data of sensitive fields, the error dilution effect in whole sentence detection is avoided, and more granular local sample privacy detection is achieved. By calculating the error of sensitive fields separately, not only can the local deep memory phenomenon masked by the global mean be accurately captured, reducing the false negative rate of core data leakage, but the specific leakage fields that cause risks can also be directly located, providing precise guidance for the subsequent compliance review, corpus desensitization and repair of target speech synthesis models.

[0088] In step S504 of some embodiments, specifically, the target test error data refers to the error value that combines the overall target test error data and the local target test error data, which is used for subsequent privacy leakage risk assessment.

[0089] Specifically, the overall target test error data and the local target test error data can be weighted and summed to obtain the target test error data.

[0090] For example, in financial applications, the overall target test error data of an insurance business training sample can be weighted and fused with the local target test error data corresponding to key fields such as "policy", "expiration", and "renewal". When the focus of the detection is on the segment corresponding to the key business field, the weight of the local target test error data can be appropriately increased to make the differences of the key segments of the training sample in the target test error data more prominent. For example, if the overall target test error data is 0.12 and its corresponding weight is 0.3, and the local target test error data is 0.18 and its corresponding weight is 0.7, then the corresponding target test error data is 0.3×0.12+0.7×0.18=0.162.

[0091] Through steps S501 to S504, the difference between real speech duration data and predicted speech duration data can be quantitatively analyzed from both overall and local perspectives. This not only improves the completeness and precision of the duration error detection results, but also provides a more discriminative error basis for subsequent privacy leakage risk assessment.

[0092] like Figure 7 As shown, in some embodiments, step S105 includes, but is not limited to, steps S601 to S604: Step S601: Scan the error determination data for the determination threshold interval to obtain a candidate classification determination threshold sequence; wherein, the candidate classification determination threshold sequence includes at least two candidate classification determination thresholds.

[0093] Step S602: Classify the target test sample according to each candidate classification judgment threshold and the target test error data to obtain each first sample classification result; wherein, the first sample classification result is used to characterize whether the target test sample belongs to or does not belong to the first training sample set based on each candidate classification judgment threshold.

[0094] Step S603: Based on the classification results of each first sample, identify the model test error to obtain the correct recognition rate data and the incorrect recognition rate data.

[0095] Step S604: Construct a model performance discrimination curve based on the correct recognition rate data and the incorrect recognition rate data, and determine the model performance discrimination curve as the model test information corresponding to the target speech synthesis model.

[0096] In step S601 of some embodiments, the candidate classification decision threshold sequence refers to a set of numerical values ​​used to distinguish whether a sample is memorized by the model.

[0097] It should be noted that since the model usually has a smaller prediction error for the duration of samples that have participated in training, but a larger prediction error for the duration of samples that have not participated in training, a threshold needs to be set for binary classification.

[0098] Specifically, the maximum and minimum error values ​​in the error determination data can be obtained to determine the scanning range; based on the preset step size or quantile interval, equidistant or non-equidistant sampling is performed within this range to generate a candidate classification determination threshold sequence containing multiple different values.

[0099] For example, in financial applications, all time-related error data calculated in the early stages (e.g., error range between 0.01 and 0.20) can be extracted, and interval scanning can be performed with a step size of 0.01 to generate candidate classification threshold sequences such as [0.01, 0.02, 0.03, ..., 0.20]. In medical applications, in order to capture the boundary of privacy leakage risk more precisely, the quantile scanning method can be used to extract candidate classification threshold sequences for key nodes such as [0.05, 0.08, 0.12, 0.15].

[0100] In step S602 of some embodiments, specifically, the first sample classification result is used to characterize whether the target test sample belongs to or does not belong to the first training sample set based on each candidate classification judgment threshold.

[0101] Specifically, each candidate classification threshold in the candidate classification threshold sequence is traversed, and the target test error data corresponding to the target test sample is compared with the current threshold. If the target test error data is less than or equal to the current threshold, it is determined that the sample is very likely to have been seen by the model, and its first sample classification result is marked as "belonging to the first training sample set"; if the target test error data is greater than the current threshold, it is determined as "not belonging to the first training sample set".

[0102] In step S603 of some embodiments, specifically, the correct recognition rate data refers to the proportion of samples that are correctly identified from those that actually belong to the training set; the incorrect recognition rate data refers to the proportion of samples that are incorrectly identified as training samples from those that do not actually belong to the training set.

[0103] Specifically, obtain the true source label of the target test sample (i.e., whether the sample has actually participated in model training); for each candidate classification threshold, compare the classification result of each first sample under the threshold with the true source label, count the number of member samples that are correctly hit and the number of non-member samples that are misclassified, and then calculate the correct recognition rate data and the incorrect recognition rate data corresponding to each candidate classification threshold.

[0104] For example, in financial application scenarios, when the threshold is set to 0.05, the comparison found that 85 out of 100 real financial training data were successfully identified (i.e., the correct identification rate was 85%), while 10 out of 100 external financial data that had not been seen before were misidentified as training data (i.e., the false identification rate was 10%).

[0105] In step S604 of some embodiments, specifically, the model performance discrimination curve is a graphical representation that intuitively shows the game relationship between the success rate of member inference attacks and the false alarm rate under different judgment thresholds.

[0106] Specifically, the "false recognition rate data" corresponding to each candidate classification threshold can be used as the x-axis and the "correct recognition rate data" as the y-axis. Scatter points can be plotted in a two-dimensional coordinate system, and a line can be fitted to generate a model performance discrimination curve. The area under this curve can then be calculated, along with the curve itself. Figure 1 The output is then encapsulated and determined as the final model test information for the target speech synthesis model.

[0107] For example, in financial applications, the output performance discrimination curve of the financial model shows a clear upward bulge to the left (e.g., an area under the curve of 0.92), which directly indicates to developers that the target speech synthesis model has serious sample memory issues and a very high risk of privacy leakage. In medical applications, if the output model performance discrimination curve is close to the diagonal (e.g., an area under the curve of 0.55), it indicates that the target speech synthesis model's memory features of patient data have been effectively smoothed out, and it has good privacy and security.

[0108] Through steps S601 to S604, the scattered duration error values ​​can be transformed into a standardized two-dimensional evaluation system of correct and incorrect recognition rates through dynamic interval scanning and multi-threshold linkage verification. This system is ultimately mapped into an intuitive model performance discrimination curve. This not only overcomes the problem of evaluation bias that can easily be caused by a single fixed threshold, but also provides developers with a set of quantitative analysis views with statistical significance. This makes the security risks of the target speech synthesis model in terms of member privacy leakage visible, comparable, and traceable, greatly improving the scientificity and automation level of privacy security audit before the model goes online.

[0109] like Figure 8 As shown, in some embodiments, step S105 may include, but is not limited to, steps S701 to S703: Step S701: Select the target classification threshold from the candidate classification thresholds.

[0110] Step S702: Classify the target test samples based on the target classification decision threshold and the target test error data to obtain the second sample classification result; the second sample classification result is used to characterize whether the target test sample belongs to or does not belong to the first training sample set based on the classification decision threshold. Step S703: Calculate the model performance confidence score of the classification result of the second sample to obtain the confidence score corresponding to the target test sample, and determine the confidence score as the model test information corresponding to the target speech synthesis model.

[0111] In step S701 of some embodiments, specifically, the target classification determination threshold refers to a benchmark value selected from the candidate threshold sequence that has the best classification balance or meets specific business security standards.

[0112] Specifically, based on the correct recognition rate data and incorrect recognition rate data corresponding to each candidate classification threshold, the difference between the correct recognition rate and the incorrect recognition rate of each candidate threshold can be calculated, and the candidate classification threshold with the largest difference can be determined as the target classification threshold; or, according to the preset error tolerance parameter, the optimal value that makes the incorrect recognition rate data lower than a specified upper limit (e.g., lower than 5%) can be selected from each candidate classification threshold as the target classification threshold.

[0113] For example, in financial applications, due to the extremely high privacy and compliance requirements of financial data, the maximum threshold (e.g., 0.025) that can control the false alarm rate below 1% is selected from a series of candidate thresholds based on a preset low fault tolerance parameter, and this threshold is established as the target classification threshold. In medical applications, the overall accuracy of the assessment may be more important, and the threshold corresponding to the point where the numerical difference is maximized (e.g., 0.040) can be selected as the target classification threshold.

[0114] In step S702 of some embodiments, specifically, the second sample classification result is used to characterize whether the target test sample belongs to or does not belong to the first training sample set based on the classification decision threshold.

[0115] Specifically, the target test error data corresponding to the target test sample can be numerically compared with the selected target classification decision threshold. If the target test error data is less than or equal to the target classification decision threshold, it means that the prediction deviation of the sample falls within the feature range of the known training data, and the classification result of the second sample is marked as "belonging to the first training sample set"; if the target test error data is greater than the target classification decision threshold, the classification result of the second sample is marked as "not belonging to the first training sample set".

[0116] For example, in a financial application scenario, if the target test error data for the target test text "claims payment has been issued" is 0.015, and the target classification threshold is 0.025, since 0.015 is less than 0.025, the second sample classification result of this sample is directly output as "belongs to the first training sample set". In a medical application scenario, if the target test error data for the test text "today's ward round record" is 0.080, which is greater than the threshold of 0.040 selected for the medical scenario, the second sample classification result of this sample is marked as "does not belong to the first training sample set".

[0117] In step S703 of some embodiments, specifically, the confidence score is a numerical indicator used to quantify the reliability of the classification result of the second sample, which reflects the probability or strength of the target speech synthesis model exhibiting memory features for the target test sample.

[0118] Specifically, the difference between the target test error data and the target classification threshold can be obtained. This difference is then converted into a confidence score between 0 and 100% using a pre-defined probability mapping function (such as the sigmoid function). If the target test error data is significantly lower than the target classification threshold, the confidence score for classifying it as "belonging" is higher; if the two values ​​are very close, the confidence score is close to a critical value (e.g., 50%). Finally, this quantified score data is packaged and output as the model's test information.

[0119] For example, in financial applications, if the duration error of the financial corpus is 0.005, which is far below the threshold of 0.025, the confidence score output by calculating the distance mapping is 98%. This serves as a strong warning to the reviewers as model test information: the sample is highly likely to be incorrectly remembered by the model. In medical applications, if the duration error of the medical test sample is 0.038, which is very close to the threshold of 0.040, the calculated confidence score may only be 55%, indicating that the classification result is in an ambiguous boundary state and needs to be further manually reviewed in conjunction with other dimensions.

[0120] Steps S701 to S703 further provide a micro-level refined discrimination dimension for a single test sample. By selecting the optimal threshold and introducing a confidence calculation mechanism, a quantitative score reflecting the discrimination reliability is provided, which effectively measures the depth of the model's memory of the input target test sample. This makes the privacy detection result not only have a qualitative conclusion of yes or no, but also explanatory information on the severity, improving the decision-making efficiency and targeting when handling suspected leakage samples.

[0121] In an optional embodiment of this application, after obtaining the model test information corresponding to the target speech synthesis model, the method further includes: when the model test information indicates that the target speech synthesis model has a privacy leakage risk, performing inference interface encapsulation processing on the target speech synthesis model to obtain a securely deployed speech synthesis model; performing inference interface encapsulation processing on the target speech synthesis model specifically includes: closing the independent output channel of the duration prediction branch network in the target speech synthesis model; configuring the predicted speech duration data as an invisible intermediate parameter in the external call interface of the securely deployed speech synthesis model; so that when the securely deployed speech synthesis model receives external business requests and performs speech synthesis inference, it only responds and outputs acoustic feature data or final speech waveform data, and intercepts direct reading of the predicted speech duration data.

[0122] It should be noted that, considering the actual deployment of the model, attackers typically call the model's exposed API interfaces to input probe text in batches to obtain the model's output features, and then carry out black-box or gray-box sample member inference attacks based on the output features. Since the foregoing embodiments of this application have demonstrated that duration prediction error is a core highly sensitive signal for determining whether the model has leaked training data, cutting off the outflow of this signal during the model release and deployment phase is crucial for achieving defense.

[0123] Specifically, after obtaining the model test information corresponding to the target speech synthesis model, if the model test information indicates that the target speech synthesis model has a high risk of privacy leakage (such as extremely small overall or local target test errors, or overfitting memory phenomenon), and it must be deployed due to business schedule requirements, the inference interface of the target speech synthesis model can be encapsulated to obtain a securely deployed speech synthesis model. Specific inference interface encapsulation processing can include: closing the independent output channel of the duration prediction branch within the target speech synthesis model at the model network architecture or API gateway level. In open-source speech synthesis models (such as FastSpeech), for ease of debugging, the default API often returns intermediate features such as Mel spectrum, predicted duration sequence, and pitch simultaneously. This step forces the configuration of the predicted duration sequence as an internally circulating, invisible intermediate parameter in the external call interface of the securely deployed speech synthesis model. When the securely deployed speech synthesis model receives external business requests (such as probe text from an attacker), by intercepting the API gateway or rewriting the interface, it only responds to and outputs the final acoustic feature data (such as Mel spectrum) or the generated audio waveform data, thereby intercepting and rejecting any direct reading requests for predicted speech duration data.

[0124] In this embodiment, by restricting the access permissions of the duration prediction branch network in the model, targeted duration output restrictions can be implemented directly during the model release stage. This achieves the goal of directly cutting off the path for external attackers to obtain duration error signals at the physical interface level without affecting the normal speech synthesis function in the target business scenario. This increases the difficulty for external attackers to reverse-engineer the privacy of training data. Even if the model has a certain degree of overfitting memory internally, it can still be protected from member inference attacks in external black-box interaction scenarios, thus improving the security of the model's online deployment.

[0125] As can be seen, in the above scheme, the following steps are taken: First, a target speech synthesis model is acquired, and error judgment data corresponding to the target speech synthesis model is determined. Second, target test samples are acquired, including training text and real speech duration data corresponding to the training text. Third, the speech duration of the training text is predicted using the target speech synthesis model to obtain predicted speech duration data corresponding to the training text. Fourth, the duration error is calculated based on the real speech duration data and the predicted speech duration data to obtain target test error data. Finally, the model capability of the target speech synthesis model is analyzed based on the error judgment data and the target test error data to obtain model test information corresponding to the target speech synthesis model. In this application, by introducing a differential comparison mechanism based on speech duration features, the duration error can be accurately extracted as a benchmark for sample privacy leakage analysis simply by using the target speech synthesis model for forward duration prediction. This allows for the identification and suppression of excessive memorization of training samples with privacy leakage risks during the model training phase. This not only effectively avoids the shortcomings of traditional speech synthesis methods in lacking constraints on the privacy leakage risks of training samples and overcomes the limitations of traditional testing methods that require multi-step decoding and reconstruction or complex output feature analysis, but also significantly reduces the computational complexity of model testing. It enables rapid and accurate evaluation of the privacy protection performance of the model with low computational overhead, providing reliable model testing information for preventing training sample leakage. This improves the efficiency and accuracy of security auditing of speech synthesis models before practical application and deployment, thereby enhancing the privacy protection capabilities of speech synthesis models for training data.

[0126] It should be understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0127] Please see Figure 9 This application also provides a testing apparatus for a speech synthesis model, which can implement the above-described testing method for a speech synthesis model. The apparatus includes: The model and error determination data acquisition module is used to acquire the target speech synthesis model and determine the error determination data corresponding to the target speech synthesis model. The test sample acquisition module is used to acquire target test samples; wherein, the target test samples include training text and real speech duration data corresponding to the training text; The speech duration prediction module is used to predict the speech duration of the training text through the target speech synthesis model, and obtain the predicted speech duration data corresponding to the training text. The duration error calculation module is used to calculate the duration error based on the actual speech duration data and the predicted speech duration data to obtain the target test error data. The model capability analysis module is used to analyze the model capability of the target speech synthesis model based on error judgment data and target test error data, and obtain the model test information corresponding to the target speech synthesis model.

[0128] In one embodiment, the model and error determination data acquisition module is specifically used for: Obtain the target speech synthesis model trained based on the first training sample set, and obtain the second training sample set; wherein, the first training sample set refers to the samples trained in the target speech synthesis model; and the second training sample set refers to the samples not trained in the target speech synthesis model. Based on each of the first type of samples in the first training sample set, error analysis is performed on the target speech synthesis model to obtain the first error distribution data; Based on each second type of sample in the second training sample set, error analysis is performed on the target speech synthesis model to obtain the second error distribution data; Error distribution analysis is performed based on the first error distribution data and the second error distribution data to obtain error determination data.

[0129] In one embodiment, if the first training sample set includes each first training text and the first real duration data corresponding to each first training text, then the model and error determination data acquisition module further includes: Phoneme conversion is performed on each of the first training texts to obtain the first training phoneme sequence; The first training phoneme sequence is classified into phonemes to obtain phoneme classification data; The first true duration data is perturbed based on the phoneme classification data to obtain perturbed true duration data, and the perturbed true duration data is determined as the first training sample set.

[0130] In one embodiment, the voice duration prediction module is specifically used for: Phoneme conversion is performed on the training text to obtain the training phoneme sequence; The training phoneme sequence is encoded by the target speech synthesis model to obtain phoneme sequence features; Phoneme duration prediction is performed on the features of the training phoneme sequence to obtain the predicted speech duration data corresponding to the training text.

[0131] In one embodiment, the duration error calculation module is specifically used for: Align the actual speech duration data and the predicted speech duration data to obtain aligned duration data. Based on the aligned duration data, the overall duration error of the real speech duration data and the predicted speech duration data is calculated to obtain the overall target test error data. Local duration error is calculated for real speech duration data and predicted speech duration data based on aligned duration data to obtain local target test error data. The overall target test error data and the local target test error data are integrated to obtain the target test error data.

[0132] In one embodiment, the model capability analysis module is specifically used for: The error determination data is scanned for a threshold interval to obtain a candidate classification determination threshold sequence; wherein the candidate classification determination threshold sequence includes at least two candidate classification determination thresholds; The target test samples are classified according to the candidate classification decision thresholds and the target test error data to obtain the first sample classification results; the first sample classification results are used to characterize whether the target test sample belongs to or does not belong to the first training sample set based on the candidate classification decision thresholds. Based on the classification results of each first sample, model testing errors are identified to obtain correct recognition rate data and incorrect recognition rate data. A model performance discrimination curve is constructed based on the correct recognition rate data and the incorrect recognition rate data, and the model performance discrimination curve is determined as the model test information corresponding to the target speech synthesis model.

[0133] In one embodiment, the model capability analysis module is further specifically used for: Select the target classification threshold from among the candidate classification thresholds; Based on the target classification decision threshold and the target test error data, the target test samples are classified to obtain the second sample classification result; the second sample classification result is used to characterize whether the target test sample belongs to or does not belong to the first training sample set based on the classification decision threshold. The model performance confidence score is calculated on the classification results of the second sample to obtain the confidence score corresponding to the target test sample, and the confidence score is determined as the model test information corresponding to the target speech synthesis model.

[0134] Specific limitations regarding the testing apparatus for speech synthesis models can be found in the limitations of the testing methods for speech synthesis models described above, and will not be repeated here. Each module in the aforementioned testing apparatus for speech synthesis models can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device, or stored in the memory of a computer device as software, so that the processor can call and execute the corresponding operations of each module.

[0135] In one embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be as follows: Figure 10 As shown, the computer device includes a processor, memory, network interface, and database connected via a system bus. The processor provides computational and control capabilities. The memory includes non-volatile and / or volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and database. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage media. The network interface is used to communicate with external clients via a network connection. When executed by the processor, the computer program implements the functions or steps of a server-side testing method for a speech synthesis model.

[0136] In one embodiment, a computer device is provided, which may be a client, and its internal structure diagram may be as follows: Figure 11 As shown, the computer device includes a processor, memory, network interface, display screen, and input devices connected via a system bus. The processor provides computational and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The network interface is used to communicate with an external server via a network connection. When the computer program is executed by the processor, it implements the functions or steps of a client-side testing method for a speech synthesis model.

[0137] In one embodiment, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to perform the following steps: Obtain the target speech synthesis model and determine the error judgment data corresponding to the target speech synthesis model; Obtain the target test samples; where the target test samples include training text and real speech duration data corresponding to the training text; The training text is predicted by using a target speech synthesis model to obtain the predicted speech duration data corresponding to the training text. The duration error is calculated based on the actual speech duration data and the predicted speech duration data to obtain the target test error data; Based on the error judgment data and the target test error data, the model capability analysis of the target speech synthesis model is performed to obtain the model test information corresponding to the target speech synthesis model.

[0138] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, the computer program performing the following steps when executed by a processor: Obtain the target speech synthesis model and determine the error judgment data corresponding to the target speech synthesis model; Obtain the target test samples; where the target test samples include training text and real speech duration data corresponding to the training text; The training text is predicted by using a target speech synthesis model to obtain the predicted speech duration data corresponding to the training text. The duration error is calculated based on the actual speech duration data and the predicted speech duration data to obtain the target test error data; Based on the error judgment data and the target test error data, the model capability analysis of the target speech synthesis model is performed to obtain the model test information corresponding to the target speech synthesis model.

[0139] It should be noted that the functions or steps that can be implemented by the computer-readable storage medium or computer device described above can be referred to the relevant descriptions on the server side and client side in the foregoing method embodiments. To avoid repetition, they will not be described one by one here.

[0140] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in a variety of forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), RAMbus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0141] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is used as an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above.

[0142] It should be noted that any AI models, software tools, or components not belonging to this company appearing in the embodiments of this application are merely illustrative examples and do not represent actual use. All user personal information involved in the embodiments of this application has been authorized (with the knowledge and consent) by the relevant parties or has been fully authorized by all parties, and the executing entity may obtain it through various legal and compliant means. The collection, storage, use, processing, transmission, provision, and disclosure of the information, data, and signals involved all comply with relevant laws and regulations and do not violate public order and good morals.

[0143] The above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be included within the protection scope of the present invention.

Claims

1. A testing method for speech synthesis models, characterized in that, include: Obtain the target speech synthesis model and determine the error judgment data corresponding to the target speech synthesis model; Obtain target test samples; wherein, the target test samples include training text and real speech duration data corresponding to the training text; The training text is predicted by the target speech synthesis model to obtain the predicted speech duration data corresponding to the training text. The duration error is calculated based on the actual speech duration data and the predicted speech duration data to obtain the target test error data; Based on the error determination data and the target test error data, the model capability analysis of the target speech synthesis model is performed to obtain the model test information corresponding to the target speech synthesis model.

2. The method as described in claim 1, characterized in that, The step of acquiring the target speech synthesis model and determining the error judgment data corresponding to the target speech synthesis model includes: Obtain the target speech synthesis model trained based on the first training sample set, and obtain the second training sample set; wherein, the first training sample set refers to samples trained in the target speech synthesis model; and the second training sample set refers to samples not trained in the target speech synthesis model. Based on each of the first type of samples in the first training sample set, error analysis is performed on the target speech synthesis model to obtain the first error distribution data; Based on each second type of sample in the second training sample set, error analysis is performed on the target speech synthesis model to obtain second error distribution data; Error distribution analysis is performed based on the first error distribution data and the second error distribution data to obtain the error determination data.

3. The method as described in claim 2, characterized in that, The step of performing model capability analysis on the target speech synthesis model based on the error determination data and the target test error data to obtain model test information corresponding to the target speech synthesis model includes: The error determination data is scanned for a determination threshold interval to obtain a candidate classification determination threshold sequence; wherein, the candidate classification determination threshold sequence includes at least two candidate classification determination thresholds; The target test sample is classified according to the candidate classification decision thresholds and the target test error data to obtain the first sample classification results; wherein, the first sample classification results are used to characterize whether the target test sample belongs to or does not belong to the first training sample set based on the candidate classification decision thresholds. Based on the classification results of each of the first samples, model test error identification is performed to obtain correct identification rate data and incorrect identification rate data; Based on the correct recognition rate data and the incorrect recognition rate data, a model performance discrimination curve is constructed, and the model performance discrimination curve is determined as the model test information corresponding to the target speech synthesis model.

4. The method as described in claim 3, characterized in that, The step of performing model capability analysis on the target speech synthesis model based on the error determination data and the target test error data to obtain model test information corresponding to the target speech synthesis model further includes: Select the target classification threshold from the candidate classification thresholds; Based on the target classification decision threshold and the target test error data, the target test sample is classified to obtain a second sample classification result; the second sample classification result is used to characterize whether the target test sample belongs to or does not belong to the first training sample set based on the classification decision threshold. The model performance confidence score is calculated on the classification result of the second sample to obtain the confidence score corresponding to the target test sample, and the confidence score is determined as the model test information corresponding to the target speech synthesis model.

5. The method as described in claim 2, characterized in that, The first training sample set includes each first training text and the first real duration data corresponding to each first training text; Before obtaining the target speech synthesis model trained based on the first training sample set, the following steps are included: Phoneme conversion is performed on each of the first training texts to obtain the first training phoneme sequence; The first training phoneme sequence is classified into phonemes to obtain phoneme classification data. The first real duration data is perturbed based on the phoneme classification data to obtain perturbed real duration data, and the perturbed real duration data is determined as the first training sample set.

6. The method according to any one of claims 1 to 5, characterized in that, The step of calculating the duration error based on the actual speech duration data and the predicted speech duration data to obtain the target test error data includes: The actual speech duration data and the predicted speech duration data are aligned in terms of duration position to obtain aligned duration data. Based on the aligned duration data, the overall duration error of the real speech duration data and the predicted speech duration data is calculated to obtain the overall target test error data. Based on the aligned duration data, local duration error is calculated for the real speech duration data and the predicted speech duration data to obtain local target test error data. The overall target test error data and the local target test error data are integrated to obtain the target test error data.

7. The method according to any one of claims 1 to 5, characterized in that, The step of predicting the speech duration of the training text using the target speech synthesis model to obtain the predicted speech duration data corresponding to the training text includes: The training text is converted into phonemes to obtain a training phoneme sequence; The training phoneme sequence is encoded using the target speech synthesis model to obtain phoneme sequence features; Phoneme duration prediction is performed on the features of the training phoneme sequence to obtain the predicted speech duration data corresponding to the training text.

8. A testing device for speech synthesis models, characterized in that, The device includes: The model and error determination data acquisition module is used to acquire the target speech synthesis model and determine the error determination data corresponding to the target speech synthesis model. A test sample acquisition module is used to acquire target test samples; wherein, the target test samples include training text and real speech duration data corresponding to the training text; The speech duration prediction module is used to predict the speech duration of the training text through the target speech synthesis model to obtain the predicted speech duration data corresponding to the training text. The duration error calculation module is used to calculate the duration error based on the actual speech duration data and the predicted speech duration data to obtain the target test error data. The model capability analysis module is used to perform model capability analysis on the target speech synthesis model based on the error judgment data and the target test error data, and obtain model test information corresponding to the target speech synthesis model.

9. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the testing method for a speech synthesis model as described in any one of claims 1 to 7.

10. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the testing method for a speech synthesis model as described in any one of claims 1 to 7.

Images