Temporal repetition perception penalty sampling method, device, electronic device and storage medium

Abstract

The present application discloses a temporal repetition perception penalty sampling method, device, electronic device and storage medium, which relates to the field of artificial intelligence technology. The method includes: inputting a given text into a pre-trained large language model; determining a target penalty probability value based on a first continuous repetition threshold and the probability distribution output by the large language model at the current moment; using the large language model to decode according to the target penalty probability value to obtain the first speech code at the current moment; using the first continuous repetition threshold as a window length to calculate the continuous repetition rate of the preamble generation code and the first speech code; if the continuous repetition rate reaches a second continuous repetition threshold, using the large language model to decode according to the probability distribution output by the large language model at the current moment to obtain the second speech code; replacing the first speech code with the second speech code; and then continuing to encode until the target speech code is obtained. The present application can solve the problems of repetition, missing words, infinite loops, etc. that frequently occur in large language model speech synthesis schemes.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to a method, device, electronic device, and storage medium for temporal repetition-aware penalty sampling. Background Art

[0002] Speech is the most natural form of human interaction. As the "mouth" of human-computer speech interaction systems, text-to-speech (TTS) technology aims to convert input text into natural and fluent speech output. In recent years, the field of speech synthesis has experienced significant advancements, primarily due to the application of language model (LM)-based approaches. TTS models typically draw on prior knowledge from natural language processing, shifting the speech modeling process from a continuous modeling approach to a next-token prediction approach. This approach uses a pre-trained neural speech encoder to quantize and extract discrete speech sequences from the continuous speech signal, and then further optimizes the modeling of these discrete sequences. Leveraging large amounts of speech data, these approaches not only enable the synthesis of high-quality, natural and fluent speech from only 3-5 seconds of speech prompts, but also effectively preserve the characteristics of the prompts, such as emotion, speech rate, and speaking style.

[0003] Compared to traditional text-to-speech (TTS) solutions, language model (LM)-based TTS solutions, such as GPT-sovits and CosyVoice, can generate synthesized speech that is more natural, fluent, and closer to real human speech. However, language model (LM)-based TTS solutions also face many challenges, especially in terms of the stability of the generated speech. For example, it is sometimes difficult to achieve a level of stability consistent with human speech, and sometimes even worse than traditional TTS solutions. Summary of the Invention

[0004] The main purpose of the embodiments of the present application is to propose a time-series repetition perception penalty sampling method, device, electronic device and storage medium to improve the stability of synthesized speech.

[0005] To achieve the above objectives, an embodiment of the present application provides a temporal repetition-aware penalty sampling method, which includes the following steps:

[0006] Input the given text into a pre-trained large language model;

[0007] Determining a target penalty probability value according to a first continuous repetition threshold and a probability distribution output by the large language model at a current moment;

[0008] Decoding the target penalty probability value using the large language model to obtain a first speech code at the current moment;

[0009] Calculating a continuous repetition rate of the preamble generated code and the first speech code using the first continuous repetition threshold as a window length;

[0010] If the continuous repetition rate is greater than or equal to a second continuous repetition threshold, decoding using the large language model according to the probability distribution output by the large language model at the current moment to obtain a second speech code;

[0011] Replacing the first speech code with the second speech code;

[0012] Return to the step of determining the target penalty probability value based on the first continuous repetition threshold and the probability distribution output by the large language model at the current moment, until the end code is obtained by decoding with the large language model, and then the target speech code corresponding to the given text is obtained.

[0013] In some embodiments, determining the target penalty probability value based on the first continuous repetition threshold and the probability distribution output by the large language model at the current moment includes the following steps:

[0014] If the current number of inferences is less than the first continuous repetition threshold, obtaining a first probability value of the index corresponding to the prompt speech code of the first window length from the probability distribution output by the large language model at the current moment, and then reducing the first probability value by a penalty coefficient to obtain a first penalty probability value;

[0015] If the current number of inferences is greater than or equal to the first continuous repetition threshold, obtaining a second probability value of the index corresponding to the prompt speech code of the second window length from the probability distribution output by the large language model at the current moment, and then reducing the second probability value by the penalty coefficient to obtain a second penalty probability value;

[0016] The target penalty probability value is determined according to the first penalty probability value or the second penalty probability value.

[0017] In some embodiments, determining the target penalty probability value according to the first penalty probability value or the second penalty probability value comprises the following steps:

[0018] The first penalty probability value or the second penalty probability value is divided by a preset temperature coefficient to obtain the target penalty probability value.

[0019] In some embodiments, the method further comprises the following steps:

[0020] If the continuous repetition rate is less than the second continuous repetition threshold, the given text is continuously decoded using the large language model until the end encoding is obtained by decoding using the large language model, thereby obtaining the target speech encoding corresponding to the given text.

[0021] In some embodiments, the decoding using the large language model according to the target penalty probability value to obtain the first speech code at the current moment includes the following steps:

[0022] The first speech code at the current moment is obtained by using the large language model, using top-p parameters and top-k parameters, through polynomial sampling, and decoding according to the target penalty probability value.

[0023] In some embodiments, the step of using the large language model to decode the second speech code according to the probability distribution output by the large language model at the current moment comprises the following steps:

[0024] The second speech code is obtained by using the large language model through random polynomial sampling and decoding according to the probability distribution output by the large language model at the current moment.

[0025] In some embodiments, the method further comprises the following steps:

[0026] The speech corresponding to the given text is synthesized according to the target speech coding.

[0027] To achieve the above-mentioned purpose, another aspect of an embodiment of the present application provides a timing repetition-aware penalty sampling device, the device comprising:

[0028] Initialization unit, used to input the given text into the pre-trained large language model;

[0029] a probability value determining unit, configured to determine a target penalty probability value based on a first continuous repetition threshold and a probability distribution output by the large language model at a current moment;

[0030] A first encoding unit is configured to use the large language model to decode according to the target penalty probability value to obtain a first speech code at a current moment;

[0031] a repetition rate calculation unit, configured to calculate a continuous repetition rate of the preamble generated code and the first speech code using the first continuous repetition threshold as a window length;

[0032] a second encoding unit, configured to, if the continuous repetition rate is greater than or equal to a second continuous repetition threshold, use the large language model to decode according to a probability distribution output by the large language model at a current moment to obtain a second speech code;

[0033] a coding replacement unit, configured to replace the first speech coding with the second speech coding;

[0034] The first continuous encoding unit is used to return to the step of determining the target penalty probability value based on the first continuous repetition threshold and the probability distribution output by the large language model at the current moment, until the end encoding is obtained by decoding with the large language model, and then the target speech encoding corresponding to the given text is obtained.

[0035] To achieve the above-mentioned purpose, another aspect of an embodiment of the present application proposes an electronic device, which includes a memory and a processor, wherein the memory stores a computer program, and the processor implements the above-mentioned time-series repetition-aware penalty sampling method when executing the computer program.

[0036] To achieve the above-mentioned purpose, another aspect of an embodiment of the present application proposes a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, it implements the above-mentioned temporal repetition-aware penalty sampling method.

[0037] The embodiments of the present application include at least the following beneficial effects:

[0038] The present application can input a given text into a pre-trained large language model; determine the target penalty probability value based on the first continuous repetition threshold and the probability distribution output by the large language model at the current moment; use the large language model to decode according to the target penalty probability value to obtain the first voice code at the current moment; use the first continuous repetition threshold as the window length to calculate the continuous repetition rate of the preamble generation code and the first voice code; if the continuous repetition rate is greater than or equal to the second continuous repetition threshold, use the large language model to decode according to the probability distribution output by the large language model at the current moment to obtain the second voice code; replace the first voice code with the second voice code; return to the step of determining the target penalty probability value based on the first continuous repetition threshold and the probability distribution output by the large language model at the current moment, until the end code is obtained by decoding using the large language model, and then obtain the target voice code corresponding to the given text. The present application proposes a temporal repetition perception penalty sampling scheme based on penalty probability value, which greatly alleviates the problems of repetition, missing words, infinite loops, etc. that frequently occur in the large language model speech synthesis scheme, and can further make the synthesized speech follow the style and emotion characteristics of the prompt audio as much as possible, significantly improving the reasoning stability and the natural fluency of the generated audio. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0040] Figure 1A flowchart of a time series repetition-aware penalty sampling method according to an embodiment of the present application;

[0041] Figure 2 An example flow chart of step S110 provided in an embodiment of the present application;

[0042] Figure 3 An optional flowchart of a time sequence repetition perception penalty sampling method provided in an embodiment of the present application;

[0043] Figure 4 An optional flowchart of another temporal repetition-aware penalty sampling method provided in an embodiment of the present application;

[0044] Figure 5 Flowchart of the algorithm implementation of the timing repetition perception penalty sampling strategy provided in the embodiment of the present application;

[0045] Figure 6 A schematic diagram of the structure of a timing repetition perception penalty sampling device provided in an embodiment of the present application;

[0046] Figure 7 An optional structural diagram of a probability value determination unit provided in an embodiment of the present application;

[0047] Figure 8 This is an optional structural diagram of a timing repetition-aware penalty sampling device provided in an embodiment of the present application;

[0048] Figure 9 An optional structural diagram of another timing repetition-aware penalty sampling device provided in an embodiment of the present application;

[0049] Figure 10 A schematic diagram of the hardware structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0050] In order to make the purpose, technical solutions and advantages of the present application clearer, the present application is further described in detail below with reference to the accompanying drawings and examples. It should be understood that the specific embodiments described herein are only used to explain the present application and are not intended to limit the present application. When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the embodiments of the present application. They are merely examples of devices and methods consistent with some aspects of the embodiments of the present application as detailed in the appended claims.

[0051] It will be understood that the terms "first", "second", etc. used in this application may be used herein to describe various concepts, but unless otherwise specified, these concepts are not limited by these terms. These terms are only used to distinguish one concept from another. For example, without departing from the scope of the embodiments of the present application, the first information may also be referred to as the second information, and similarly, the second information may also be referred to as the first information. Depending on the context, the words "if" and "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining".

[0052] The terms "at least one", "plurality", "each", "any", etc. used in this application include "at least one", "two" or more, "plurality" or "each", "any" or "any one", "each" or "any one" as used herein.

[0053] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by those skilled in the art to which this application pertains. The terms used herein are for the purpose of describing the embodiments of this application only and are not intended to limit this application.

[0054] Before describing the embodiments of the present application in detail, the terms and some related technologies that may be involved in the embodiments of the present application are first explained. The terms and related technologies that may be involved in the embodiments of the present application are subject to the following interpretations:

[0055] The principle of text-to-speech (TTS) is to convert text information into speech output. The speech synthesis process mainly includes the following key solutions:

[0056] 1. Text preprocessing: Converting the input text into a form suitable for further processing. This includes removing irrelevant characters (such as extra spaces and special symbols), normalizing the text (such as the standardized representation of numbers and dates), and analyzing and segmenting the vocabulary.

[0057] 2. Linguistic Analysis: The pre-processed text undergoes in-depth linguistic processing, including part-of-speech tagging, syntactic analysis, and semantic understanding. The goal of this stage is to fully understand the intent and context of the text, laying the foundation for generating natural and fluent speech.

[0058] 3. Voice Generation and Synthesis: This is the core of TTS technology, encompassing the construction of a voice database, the selection and connection of voice units, and the generation of the final speech through synthesis algorithms. High-quality voice sample collection is fundamental. The system precisely selects appropriate voice units from the voice database and then uses sophisticated algorithms to combine and smooth them, reducing the unnaturalness of the synthesized speech and improving its fluency and naturalness.

[0059] 4. Synthesis algorithm: It is the core technology for realizing text-to-speech conversion. It processes and analyzes the text content through specific algorithms, and finally converts the text into understandable speech through the speech synthesis system.

[0060] Application scenarios of speech synthesis in this application include but are not limited to:

[0061] Navigation system: In the in-vehicle navigation system, speech synthesis technology can convert navigation instructions into voice output, improving driving safety.

[0062] Intelligent customer service: In intelligent customer service systems, speech synthesis technology can convert text responses into natural and fluent speech, improving user experience.

[0063] Education: In educational software, speech synthesis technology can convert the content of e-books into speech, making it easier for visually impaired people or those with reading difficulties to use.

[0064] Information broadcasting: In news, weather, traffic and other information broadcasting, speech synthesis technology can convert text information into speech output in real time, improving the efficiency of information dissemination.

[0065] Speech synthesis decoding refers to the process of converting text into speech signals, which includes the following main steps:

[0066] Text analysis: Normalizes the input text, such as converting abbreviations, numbers, dates, etc. into complete text form; performs word segmentation and part-of-speech tagging; understands sentence structure, and determines stress and intonation.

[0067] Language processing: Convert text into phoneme sequences and perform prosody prediction to determine prosodic features such as pitch, duration, and intensity.

[0068] Acoustic model: Determines the acoustic parameters of each phoneme and generates basic features of speech, such as fundamental frequency (F0), spectral envelope, and duration.

[0069] Waveform generation: Generate the final speech waveform through different methods. Common methods include concatenative synthesis and parametric synthesis, as follows:

[0070] Concatenative synthesis: Selects and concatenates the most appropriate speech segments from a pre-recorded speech database. This approach offers a high degree of naturalness, but requires a lot of storage space and offers limited flexibility.

[0071] Parametric synthesis: Generate speech waveforms using vocoder models such as source-filter models.

[0072] The above steps work together to convert text into human-audible speech signals.

[0073] Top-p and top-k are two parameters used to control the output of the language model, mainly used to adjust the diversity and relevance of the generated text.

[0074] 1.top-k:

[0075] Top-k sampling means that at each time step, the model randomly selects a word from the k most likely candidate words as the output. By adjusting the value of k, you can control the diversity and accuracy of the model's output. Smaller values ​​of k increase output accuracy but reduce diversity; larger values ​​of k increase diversity but may reduce output accuracy.

[0076] 2.top-p:

[0077] Top-p sampling (also known as kernel sampling) selects words based on their cumulative probability. Specifically, the model only considers words whose cumulative probability reaches a certain percentage p or above. For example, if p = 0.9, the model will only select words whose cumulative probability is in the top 90%. This approach helps the model focus on generating relevant and coherent text.

[0078] Application scenarios and effects:

[0079] Top-K: Suitable for scenarios that require high accuracy, such as legal document writing and technical reports. By reducing the number of candidate words, it can reduce the generation of irrelevant or incorrect words and improve the accuracy of the output.

[0080] Top-P: Suitable for scenarios that require higher diversity, such as creative writing, story generation, etc. By selecting words with higher cumulative probability, the diversity and creativity of the output can be increased.

[0081] By reasonably setting the above parameters, a balance can be found between the model's creativity and output quality, thereby better meeting the needs of different application scenarios.

[0082] Speech is the most natural form of human interaction. As the "mouth" of human-computer speech interaction systems, text-to-speech (TTS) technology aims to convert input text into natural and fluent speech output. In recent years, the field of speech synthesis has experienced significant advancements, primarily due to the application of language model (LM)-based approaches. TTS models typically draw on prior knowledge from natural language processing, shifting the speech modeling process from a continuous modeling approach to a next-token prediction approach. This approach uses a pre-trained neural speech encoder to quantize and extract discrete speech sequences from the continuous speech signal, and then further optimizes the modeling of these discrete sequences. Leveraging large amounts of speech data, these approaches not only enable the synthesis of high-quality, natural and fluent speech from only 3-5 seconds of speech prompts, but also effectively preserve the characteristics of the prompts, such as emotion, speech rate, and speaking style.

[0083] Compared to traditional text-to-speech (TTS) solutions, language model (LM)-based TTS solutions, such as GPT-sovits and CosyVoice, can generate synthesized speech that is more natural, fluent, and close to real human speech. However, language model (LM)-based TTS solutions also face many challenges, especially in terms of the stability of the generated speech. For example, it is sometimes difficult to achieve a stability level consistent with human speech, and sometimes even worse than traditional TTS solutions. Specifically, the speech generated by existing speech synthesis solutions may have the following problems:

[0084] 1. Omission or repetition of words: During the synthesis process, due to the accumulated errors of the autoregressive model, some words in the original text may be omitted or repeated, affecting the complete transmission of information. This is particularly noticeable in long texts or texts containing repetitive sentences.

[0085] 2. Inaccurate pronunciation: Even the most advanced LM-TTS systems may have deviations in the pronunciation of specific words or phrases. This may be due to limitations in the model training data or inadequate processing of specific language phenomena.

[0086] 3. Inconsistent or jittery timbre: Sudden changes or instability in timbre within a sentence is also a common problem, which can give people a "machine-like" feeling and reduce the naturalness and listening quality of the voice.

[0087] Several related technologies have been proposed from different perspectives to address the aforementioned issues. A classic example is the Repetition Aware Sampling (RAS) proposed by VALLE-2. This method improves the original kernel sampling process by taking into account repeated tokens in the decoding history, thereby enhancing decoding stability. However, this method still has drawbacks. For example, improper window settings can seriously affect the speech rate of the generated audio.

[0088] Based on this, the present application provides a temporal repetition perception penalty sampling method, device, electronic device and storage medium. Compared with the RAS sampling method and the random sampling method, the present application not only greatly alleviates a series of instability problems such as repetition and missing words that occur when the autoregressive model generates encoding, but also improves the consistency of speech speed, while making the generated audio more in line with the style and emotion of the prompt voice.

[0089] The embodiments of the present application provide a temporal repetition-aware penalty sampling method, apparatus, electronic device, and storage medium. The technical solution of the present application includes: inputting a given text into a pre-trained large language model; determining a target penalty probability value based on a first continuous repetition threshold and a probability distribution output by the large language model at a current moment; using the large language model to decode according to the target penalty probability value to obtain a first speech code at the current moment; calculating the continuous repetition rate of the preamble generation code and the first speech code using a window with the first continuous repetition threshold as the length; if the continuous repetition rate is greater than or equal to a second continuous repetition threshold, using the large language model to decode according to the probability distribution output by the large language model at the current moment to obtain a second speech code; replacing the first speech code with the second speech code; and returning to the step of determining the target penalty probability value based on the first continuous repetition threshold and the probability distribution output by the large language model at the current moment, until the end code is obtained by decoding using the large language model, thereby obtaining the target speech code corresponding to the given text. This application proposes a temporal repetition-aware penalty sampling scheme based on penalty probability values, which greatly alleviates the problems of repetition, missing words, infinite loops, etc. that frequently occur in large language model speech synthesis schemes. It can also further make the synthesized speech follow the style and emotion of the prompt audio as much as possible, significantly improving the reasoning stability and the natural fluency of the generated audio.

[0090] The embodiments of the present application provide a time-series repetition-aware penalty sampling method, device, electronic device and storage medium, which relate to the field of artificial intelligence technology. The time-series repetition-aware penalty sampling method, device, electronic device and storage medium provided in the embodiments of the present application can be applied to a terminal, can also be applied to a server, and can also be software running in a terminal or a server. In some embodiments, the terminal can be a smart phone, a tablet computer, a laptop computer, a desktop computer, a smart speaker, a smart watch and a car terminal, etc., but is not limited to this; the server side can be configured as an independent physical server, or as a server cluster or distributed system composed of multiple physical servers, and can also be configured as a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN and big data and artificial intelligence platforms. The server can also be a node server in a blockchain network; the software can be an application that implements a knowledge extraction method, etc., but is not limited to the above forms.

[0091] The present application can be used in many general or special computer system environments or configurations. For example: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, and the like. The present application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, and the like that perform specific tasks or implement specific abstract data types. The present application can also be practiced in distributed computing environments in which tasks are performed by remote processing devices connected via a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media, including storage devices.

[0092] Reference Figure 1 The embodiment of the present application provides a temporal repetition-aware penalty sampling method, which may include but is not limited to S100 to S150, as follows:

[0093] S100: Input the given text into the pre-trained large language model.

[0094] S110: Determine a target penalty probability value according to a first continuous repetition threshold and a probability distribution output by the large language model at a current moment.

[0095] Reference Figure 2 , S110 may further include the following steps S111 to S113:

[0096] S111: If the current number of inferences is less than the first continuous repetition threshold, obtaining a first probability value of an index corresponding to a prompt speech code of a first window length from the probability distribution output by the large language model at the current moment, and then reducing the first probability value by a penalty coefficient to obtain a first penalty probability value;

[0097] S112: If the current number of inferences is greater than or equal to the first continuous repetition threshold, obtaining a second probability value of the index corresponding to the prompt speech code of the second window length from the probability distribution output by the large language model at the current moment, and then reducing the second probability value by the penalty coefficient to obtain a second penalty probability value;

[0098] S113: Determine the target penalty probability value according to the first penalty probability value or the second penalty probability value.

[0099] More specifically, S113 may include the following steps:

[0100] The first penalty probability value or the second penalty probability value is divided by a preset temperature coefficient to obtain the target penalty probability value.

[0101] S120: Using the large language model to decode according to the target penalty probability value to obtain a first speech code at the current moment.

[0102] Furthermore, S120 may include the following steps:

[0103] The first speech code at the current moment is obtained by using the large language model, using top-p parameters and top-k parameters, through polynomial sampling, and decoding according to the target penalty probability value.

[0104] S130: Calculate the continuous repetition rate of the preamble generated code and the first speech code using the first continuous repetition threshold as a window length.

[0105] S140: If the continuous repetition rate is greater than or equal to a second continuous repetition threshold, the large language model is used to decode according to the probability distribution output by the large language model at the current moment to obtain a second speech code.

[0106] Furthermore, S140 may include the following steps:

[0107] The second speech code is obtained by using the large language model through random polynomial sampling and decoding according to the probability distribution output by the large language model at the current moment.

[0108] S150: Replace the first speech code with the second speech code.

[0109] S160: Return to the step of determining the target penalty probability value based on the first continuous repetition threshold and the probability distribution output by the large language model at the current moment, until the end code is obtained by decoding with the large language model, and then the target speech code corresponding to the given text is obtained.

[0110] Reference Figure 3 , this embodiment of the application may further include step S170:

[0111] S170: If the continuous repetition rate is less than the second continuous repetition threshold, continue to use the large language model to decode the given text until the end encoding is obtained by decoding with the large language model, and then obtain the target speech encoding corresponding to the given text.

[0112] Reference Figure 4 As another further implementation, the embodiment of the present application may further include step S180:

[0113] S180: Synthesize speech corresponding to the given text according to the target speech coding.

[0114] Next, the solution of the embodiment of the present application will be introduced and explained in detail with reference to specific application examples.

[0115] Reference Figure 5 , this embodiment provides a flowchart of the algorithm implementation of the timing repetition perception penalty sampling strategy.

[0116] Specifically, during inference, since the random sampling strategy may cause unstable phenomena such as missing words, repeated reading, and infinite loops, this embodiment proposes a time-series repetition-aware penalty sampling strategy, and its algorithm implementation flow chart is as follows: Figure 5 shown. Figure 5 Line 1 in the description describes the meaning of each parameter. Figure 5 Lines 2-7 in the figure describe the process of temporal repetition perception penalty. If the current number of inferences is less than N (the first consecutive repetition threshold), s is obtained from the probability distribution output by the GPT (large language model) at the current moment. t′-w1:t′ The first probability value of the corresponding index is reduced by σ (penalty coefficient). Since the inference result is easily affected by the indefinite length of silence at the end of the prompt speech at the beginning of the inference, the above steps can effectively alleviate this effect. If the current number of inferences is greater than or equal to N, s will be obtained from the probability distribution of the GPT output at the current moment. -w2-N:-NThe second probability value of the corresponding index is reduced by a factor of σ. The inventors of this application have observed from features extracted from tens of thousands of hours of audio that the number of consecutive repetitions of most discrete speech codes is unlikely to exceed N times, where N is related to the speech quantizer used. The above steps maintain the same distribution of the decoded speech codes as the actual speech codes as much as possible, ensuring that the generated speech more closely follows the style and emotion of the prompt speech. At the same time, a first threshold is set to ensure that the generated codes do not repeat the same code continuously as much as possible. Figure 5 Lines 6-10 in the figure describe the process of temporal repetitive perceptual sampling. First, the probability value obtained in line 3 or line 5 is divided by the temperature coefficient, and then the speech code s at the current moment is decoded through polynomial sampling. t (First speech coding). If the preamble generated code is within the range of window N (a window with the first continuous repetition threshold as the length) and the speech coding at the current moment s t The number of repetitions r is greater than the repetition threshold t r (Second consecutive repetition threshold), random polynomial sampling is used to resample from all probability values ​​with replacement s t , so that the generated code will not have more than t consecutive repetitions within the range of window length N as much as possible r Subsequent decoding cycles are repeated until the end code is decoded.

[0117] This embodiment proposes a temporal repetition-aware penalty sampling strategy, which greatly alleviates the problems of repetition, missing words, infinite loops, etc. that frequently occur in the LLM-TTS paradigm. At the same time, it further improves the generated audio to follow the style and emotion of the prompt audio as much as possible, significantly improving the inference stability and the natural fluency of the generated audio.

[0118] Reference Figure 6 The present application also provides a timing repetition-aware penalty sampling device, which can implement the above-mentioned timing repetition-aware penalty sampling method. The device includes:

[0119] Initialization unit, used to input the given text into the pre-trained large language model;

[0120] a probability value determining unit, configured to determine a target penalty probability value based on a first continuous repetition threshold and a probability distribution output by the large language model at a current moment;

[0121] A first encoding unit, configured to obtain a first speech code at a current moment by decoding using the large language model according to the target penalty probability value;

[0122] a repetition rate calculation unit, configured to calculate a continuous repetition rate of the preamble generated code and the first speech code using the first continuous repetition threshold as a window length;

[0123] a second encoding unit, configured to, if the continuous repetition rate is greater than or equal to a second continuous repetition threshold, use the large language model to decode according to a probability distribution output by the large language model at a current moment to obtain a second speech code;

[0124] a coding replacement unit, configured to replace the first speech coding with the second speech coding;

[0125] The first continuous encoding unit is used to return to the step of determining the target penalty probability value based on the first continuous repetition threshold and the probability distribution output by the large language model at the current moment, until the end encoding is obtained by decoding with the large language model, and then the target speech encoding corresponding to the given text is obtained.

[0126] Reference Figure 7 In some embodiments, the probability value determining unit includes:

[0127] a first probability value determining subunit configured to obtain, if the current number of inferences is less than the first continuous repetition threshold, a first probability value of an index corresponding to a prompt speech code of a first window length from the probability distribution output by the large language model at the current moment, and further reduce the first probability value by a penalty coefficient to obtain a first penalty probability value;

[0128] a second probability value determining subunit, configured to obtain, from the probability distribution output by the large language model at the current moment, a second probability value of the index corresponding to the prompt speech code of the second window length, if the current number of inferences is greater than or equal to the first continuous repetition threshold, and further reduce the second probability value by a factor of the penalty coefficient to obtain a second penalty probability value;

[0129] The third probability value determining subunit is configured to determine the target penalty probability value according to the first penalty probability value or the second penalty probability value.

[0130] In some embodiments, the third probability value determining subunit includes:

[0131] The target probability value determining unit is configured to divide the first penalty probability value or the second penalty probability value by a preset temperature coefficient to obtain the target penalty probability value.

[0132] Reference Figure 8 In some embodiments, the apparatus further comprises:

[0133] The second continuous encoding unit is used to continue decoding the given text using the large language model if the continuous repetition rate is less than the second continuous repetition threshold until the end encoding is obtained by decoding using the large language model, thereby obtaining the target speech encoding corresponding to the given text.

[0134] In some embodiments, the first encoding unit includes:

[0135] The first encoding subunit is configured to utilize the large language model, use top-p parameters and top-k parameters, perform polynomial sampling, and decode according to the target penalty probability value to obtain the first speech code at the current moment.

[0136] In some embodiments, the second encoding unit includes:

[0137] The second encoding subunit is configured to obtain the second speech code by using the large language model through random polynomial sampling and decoding according to the probability distribution output by the large language model at a current moment.

[0138] Reference Figure 9 In some embodiments, the apparatus further comprises:

[0139] The speech synthesis unit is used to synthesize the speech corresponding to the given text according to the target speech coding.

[0140] It can be understood that the contents of the above method embodiments are all applicable to the present device embodiments, the functions specifically implemented by the present device embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0141] The present application also provides an electronic device comprising a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the aforementioned timing repetition-aware penalty sampling method. The electronic device can be any smart terminal, such as a tablet computer or an in-vehicle computer.

[0142] It can be understood that the contents of the above method embodiments are applicable to the present device embodiments, the functions specifically implemented by the present device embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0143] See also Figure 10 , Figure 10 The hardware structure of an electronic device according to another embodiment is shown. The electronic device includes:

[0144] The processor 1001 can be implemented as a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of the present application.

[0145] The memory 1002 can be implemented in the form of a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 1002 can store an operating system and other application programs. When the technical solutions provided in the embodiments of this specification are implemented through software or firmware, the relevant program code is stored in the memory 1002, and the processor 1001 calls and executes the temporal repetition-aware penalty sampling method of the embodiments of this application.

[0146] Input / output interface 1003, used to implement information input and output;

[0147] Communication interface 1004, used to implement communication interaction between this device and other devices, which can be achieved through wired means (such as USB, network cable, etc.) or wireless means (such as mobile network, WiFi, Bluetooth, etc.);

[0148] Bus 1005 , which transmits information between various components of the device (e.g., processor 1001 , memory 1002 , input / output interface 1003 , and communication interface 1004 );

[0149] The processor 1001 , the memory 1002 , the input / output interface 1003 and the communication interface 1004 are connected to each other in communication within the device via the bus 1005 .

[0150] An embodiment of the present application also provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, it implements the above-mentioned time series repetition perception penalty sampling method.

[0151] It can be understood that the contents of the above method embodiments are all applicable to the present storage medium embodiment, the functions specifically implemented by the present storage medium embodiment are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0152] The memory, as a non-transient computer-readable storage medium, can be used to store non-transient software programs and non-transient computer executable programs. In addition, the memory may include a high-speed random access memory and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some embodiments, the memory may optionally include a memory remotely arranged relative to the processor, and these remote memories may be connected to the processor via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0153] The embodiments described in the embodiments of this application are intended to more clearly illustrate the technical solutions of the embodiments of this application and do not constitute a limitation on the technical solutions provided by the embodiments of this application. Those skilled in the art will appreciate that with the evolution of technology and the emergence of new application scenarios, the technical solutions provided in the embodiments of this application are also applicable to similar technical problems.

[0154] Those skilled in the art will understand that the technical solutions shown in the figures do not constitute a limitation on the embodiments of the present application, and may include more or fewer steps than shown in the figures, or a combination of certain steps, or different steps.

[0155] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, i.e., they may be located in one place or distributed across multiple network units. Some or all of the modules may be selected based on actual needs to achieve the objectives of this embodiment.

[0156] Those skilled in the art will appreciate that all or some of the steps in the methods, systems, and functional modules / units in the devices disclosed above may be implemented as software, firmware, hardware, or appropriate combinations thereof.

[0157] The terms "first", "second", "third", "fourth", etc. (if any) in the specification of the present application and the above-mentioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequential order. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0158] It should be understood that in this application, "at least one (item)" means one or more, and "plurality" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the previous and next associated objects are in an "or" relationship. "At least one of the following items" or similar expressions refers to any combination of these items, including any combination of single items or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0159] In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are merely schematic. For example, the division of the above-mentioned units is only a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0160] The units described above as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0161] In addition, the functional units in the various embodiments of the present application may be integrated into a single processing unit, or each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or software functional units.

[0162] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, which is stored in a storage medium and includes multiple instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of various embodiments of the present application. The aforementioned storage medium includes: various media that can store programs, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0163] The preferred embodiments of the present invention are described above with reference to the accompanying drawings, but are not intended to limit the scope of the present invention. Any modifications, equivalent substitutions, and improvements made by those skilled in the art without departing from the scope and essence of the present invention should be within the scope of the present invention.

Claims

1. Temporal repetition-aware penalty sampling method, characterized in that: The method comprises the following steps: Input the given text into a pre-trained large language model; Determining a target penalty probability value according to a first continuous repetition threshold and a probability distribution output by the large language model at a current moment; Decoding the target penalty probability value using the large language model to obtain a first speech code at the current moment; Calculating a continuous repetition rate of the preamble generated code and the first speech code using the first continuous repetition threshold as a window length; If the continuous repetition rate is greater than or equal to a second continuous repetition threshold, decoding using the large language model according to the probability distribution output by the large language model at the current moment to obtain a second speech code; Replacing the first speech code with the second speech code; Return to the step of determining the target penalty probability value based on the first continuous repetition threshold and the probability distribution output by the large language model at the current moment, until the end code is obtained by decoding with the large language model, and then the target speech code corresponding to the given text is obtained.

2. The temporal repetition-aware penalty sampling method according to claim 1, wherein: Determining the target penalty probability value according to the first continuous repetition threshold and the probability distribution output by the large language model at the current moment comprises the following steps: If the current number of inferences is less than the first continuous repetition threshold, obtaining a first probability value of the index corresponding to the prompt speech code of the first window length from the probability distribution output by the large language model at the current moment, and then reducing the first probability value by a penalty coefficient to obtain a first penalty probability value; If the current number of inferences is greater than or equal to the first continuous repetition threshold, obtaining a second probability value of the index corresponding to the prompt speech code of the second window length from the probability distribution output by the large language model at the current moment, and then reducing the second probability value by the penalty coefficient to obtain a second penalty probability value; The target penalty probability value is determined according to the first penalty probability value or the second penalty probability value.

3. The temporal repetition-aware penalty sampling method according to claim 2, wherein: Determining the target penalty probability value according to the first penalty probability value or the second penalty probability value comprises the following steps: The first penalty probability value or the second penalty probability value is divided by a preset temperature coefficient to obtain the target penalty probability value.

4. The temporal repetition-aware penalty sampling method according to claim 1, wherein: The method further comprises the following steps: If the continuous repetition rate is less than the second continuous repetition threshold, the given text is continuously decoded using the large language model until the end encoding is obtained by decoding using the large language model, thereby obtaining the target speech encoding corresponding to the given text.

5. The temporal repetition-aware penalty sampling method according to claim 1, wherein: The method of using the large language model to decode according to the target penalty probability value to obtain the first speech code at the current moment includes the following steps: The first speech code at the current moment is obtained by using the large language model, using top-p parameters and top-k parameters, through polynomial sampling, and decoding according to the target penalty probability value.

6. The temporal repetition-aware penalty sampling method according to claim 1, wherein: The method of using the large language model to decode according to the probability distribution output by the large language model at a current moment to obtain a second speech code comprises the following steps: The second speech code is obtained by using the large language model through random polynomial sampling and decoding according to the probability distribution output by the large language model at the current moment.

7. The temporal repetition-aware penalty sampling method according to any one of claims 1 to 6, characterized in that: The method further comprises the following steps: The speech corresponding to the given text is synthesized according to the target speech coding.

8. A temporal repetition perception penalty sampling device, characterized in that: The device comprises: Initialization unit, used to input the given text into the pre-trained large language model; a probability value determining unit, configured to determine a target penalty probability value based on a first continuous repetition threshold and a probability distribution output by the large language model at a current moment; A first encoding unit, configured to obtain a first speech code at a current moment by decoding using the large language model according to the target penalty probability value; a repetition rate calculation unit, configured to calculate a continuous repetition rate of the preamble generated code and the first speech code using the first continuous repetition threshold as a window length; a second encoding unit, configured to, if the continuous repetition rate is greater than or equal to a second continuous repetition threshold, use the large language model to decode according to a probability distribution output by the large language model at a current moment to obtain a second speech code; a coding replacement unit, configured to replace the first speech coding with the second speech coding; The first continuous encoding unit is used to return to the step of determining the target penalty probability value based on the first continuous repetition threshold and the probability distribution output by the large language model at the current moment, until the end encoding is obtained by decoding with the large language model, and then the target speech encoding corresponding to the given text is obtained.

9. An electronic device, characterized in that: The electronic device includes a memory and a processor, the memory stores a computer program, and the processor implements the temporal repetition-aware penalty sampling method according to any one of claims 1 to 7 when executing the computer program.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the temporal repetition-aware penalty sampling method according to any one of claims 1 to 7 is implemented.

Images