Time sequence repetition sensing punishment sampling method and device, electronic equipment and storage medium

Abstract

The invention discloses a time sequence repetition perception punishment sampling method and device, electronic equipment and a storage medium, and relates to the technical field of artificial intelligence, and the method comprises the steps: inputting a given text into a pre-training large language model; determining a target penalty probability value according to the first continuous repetition threshold value and probability distribution output by the large language model at the current moment; decoding according to the target penalty probability value by using a large language model to obtain a first voice code at the current moment; calculating the continuous repetition rate of the preorder generation code and the first voice code by using a window with a first continuous repetition threshold value as the length; if the continuous repetition rate reaches a second continuous repetition threshold value, decoding by utilizing the large language model according to probability distribution output by the large language model at the current moment to obtain a second voice code; replacing the first voice code with a second voice code; and then coding is continued until a target voice code is obtained. According to the method, the problems of repetition, word loss and missing, infinite loop and the like which frequently occur in a large language model speech synthesis scheme can be solved.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to a method, device, electronic device and storage medium for temporal repetition-aware penalty sampling. Background Art

[0002] Speech is the most natural way for humans to interact. As the "mouth" of the human-computer speech interaction system, text-to-speech (TTS) technology aims to convert input text into natural and fluent speech output. In recent years, the field of speech synthesis technology has experienced a significant leap in development, which is mainly due to the application progress of the language model (LM) method. The TTS model method usually draws on prior knowledge from the field of natural language processing, and changes the speech modeling process from the previous continuous modeling method to the next-token prediction method, that is, through the pre-trained neural speech encoder, discrete speech sequences are quantized and extracted from the continuous speech signal, and then the modeling process of these discrete sequences is further optimized. Relying on large-scale speech data, it is not only possible to synthesize high-quality and natural and fluent speech based on only about 3s to 5s of speech prompts, but also effectively retain the characteristics of emotion, speech speed and speaking style contained in the prompt speech during the synthesis process.

[0003] Compared with traditional text-to-speech (TTS) solutions, TTS solutions based on language models (LM) can generate synthetic speech that is more natural, fluent and close to real human speech, such as GPT-sovits and CosyVoice in related technologies. However, TTS solutions based on language models (LM) also face many challenges, especially in terms of the stability of generated speech. For example, it is sometimes difficult to achieve a stable level consistent with human voice, and sometimes it is even worse than traditional TTS solutions. Summary of the invention

[0004] The main purpose of the embodiments of the present application is to propose a time-series repetition-aware penalty sampling method, device, electronic device and storage medium to improve the stability of synthesized speech.

[0005] To achieve the above object, an embodiment of the present application provides a timing repetition-aware penalty sampling method, which includes the following steps:

[0006] Input the given text into a pre-trained large language model;

[0007] Determine a target penalty probability value according to a first continuous repetition threshold and a probability distribution output by the large language model at a current moment;

[0008] Decoding the target penalty probability value using the large language model to obtain a first speech code at the current moment;

[0009] Calculate the continuous repetition rate of the preamble generated code and the first speech code by taking the first continuous repetition threshold as a window of length;

[0010] If the continuous repetition rate is greater than or equal to a second continuous repetition threshold, using the large language model to decode according to the probability distribution output by the large language model at the current moment to obtain a second speech code;

[0011] Replacing the first speech code with the second speech code;

[0012] Return to the step of determining the target penalty probability value according to the first continuous repetition threshold and the probability distribution output by the large language model at the current moment, until the end code is obtained by decoding with the large language model, and then the target speech code corresponding to the given text is obtained.

[0013] In some embodiments, determining the target penalty probability value according to the first continuous repetition threshold and the probability distribution output by the large language model at the current moment comprises the following steps:

[0014] If the current number of inferences is less than the first continuous repetition threshold, obtaining a first probability value of the index corresponding to the prompt speech code of the first window length from the probability distribution output by the large language model at the current moment, and then reducing the first probability value by a penalty coefficient to obtain a first penalty probability value;

[0015] If the current number of inferences is greater than or equal to the first continuous repetition threshold, obtaining a second probability value of the index corresponding to the prompt speech code of the second window length from the probability distribution output by the large language model at the current moment, and then reducing the second probability value by the penalty coefficient times to obtain a second penalty probability value;

[0016] The target penalty probability value is determined according to the first penalty probability value or the second penalty probability value.

[0017] In some embodiments, determining the target penalty probability value according to the first penalty probability value or the second penalty probability value comprises the following steps:

[0018] The first penalty probability value or the second penalty probability value is divided by a preset temperature coefficient to obtain the target penalty probability value.

[0019] In some embodiments, the method further comprises the following steps:

[0020] If the continuous repetition rate is less than the second continuous repetition threshold, the large language model is continuously used to decode the given text until the end encoding is obtained by decoding with the large language model, thereby obtaining the target speech encoding corresponding to the given text.

[0021] In some embodiments, the step of using the large language model to decode according to the target penalty probability value to obtain the first speech encoding at the current moment comprises the following steps:

[0022] The first speech code at the current moment is obtained by using the large language model, using top-p parameters and top-k parameters through polynomial sampling and decoding according to the target penalty probability value.

[0023] In some embodiments, the step of using the large language model to decode the second speech code according to the probability distribution output by the large language model at the current moment comprises the following steps:

[0024] The second speech code is obtained by using the large language model through random polynomial sampling and decoding according to the probability distribution output by the large language model at the current moment.

[0025] In some embodiments, the method further comprises the following steps:

[0026] The speech corresponding to the given text is synthesized according to the target speech coding.

[0027] To achieve the above object, another aspect of an embodiment of the present application provides a timing repetition-aware penalty sampling device, the device comprising:

[0028] Initialization unit, which is used to input the given text into the pre-trained large language model;

[0029] A probability value determination unit, configured to determine a target penalty probability value according to a first continuous repetition threshold and a probability distribution output by the large language model at a current moment;

[0030] A first encoding unit, configured to obtain a first speech code at a current moment by decoding using the large language model according to the target penalty probability value;

[0031] a repetition rate calculation unit, configured to calculate a continuous repetition rate of a preamble generated code and the first speech code using the first continuous repetition threshold as a window length;

[0032] A second encoding unit, configured to obtain a second speech code by decoding using the large language model according to a probability distribution output by the large language model at a current moment if the continuous repetition rate is greater than or equal to a second continuous repetition threshold;

[0033] A coding replacement unit, configured to replace the first speech coding with the second speech coding;

[0034] The first continuous encoding unit is used to return to the step of determining the target penalty probability value according to the first continuous repetition threshold and the probability distribution output by the large language model at the current moment, until the end encoding is obtained by decoding with the large language model, and then the target speech encoding corresponding to the given text is obtained.

[0035] To achieve the above-mentioned purpose, another aspect of an embodiment of the present application proposes an electronic device, which includes a memory and a processor, wherein the memory stores a computer program, and the processor implements the above-mentioned timing repetition-aware penalty sampling method when executing the computer program.

[0036] To achieve the above-mentioned purpose, another aspect of an embodiment of the present application proposes a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the above-mentioned timing repetition-aware penalty sampling method is implemented.

[0037] The embodiments of the present application include at least the following beneficial effects:

[0038] The present application can input a given text into a pre-trained large language model; determine the target penalty probability value according to the first continuous repetition threshold and the probability distribution output by the large language model at the current moment; use the large language model to decode according to the target penalty probability value to obtain the first voice code at the current moment; use the first continuous repetition threshold as the window length to calculate the continuous repetition rate of the preamble generation code and the first voice code; if the continuous repetition rate is greater than or equal to the second continuous repetition threshold, use the large language model to decode according to the probability distribution output by the large language model at the current moment to obtain the second voice code; replace the first voice code with the second voice code; return to the step of determining the target penalty probability value according to the first continuous repetition threshold and the probability distribution output by the large language model at the current moment, until the end code is obtained by decoding using the large language model, and then obtain the target voice code corresponding to the given text. The present application proposes a temporal repetition perception penalty sampling scheme based on a penalty probability value, which greatly alleviates the problems of repetition, missing words, infinite loops, etc. that frequently occur in the large language model speech synthesis scheme, and can further make the synthesized speech follow the style and emotion characteristics of the prompt audio as much as possible, significantly improving the reasoning stability and the natural fluency of the generated audio. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0040] Figure 1A flowchart of a timing repetition-aware penalty sampling method provided in an embodiment of the present application;

[0041] Figure 2 An example flow chart of step S110 provided in an embodiment of the present application;

[0042] Figure 3 An optional flow chart of a timing repetition-aware penalty sampling method provided in an embodiment of the present application;

[0043] Figure 4 An optional flow chart of another timing repetition-aware penalty sampling method provided in an embodiment of the present application;

[0044] Figure 5 A flowchart of an algorithm implementation of a timing repetition perception penalty sampling strategy provided in an embodiment of the present application;

[0045] Figure 6 A schematic diagram of the structure of a timing repetition perception penalty sampling device provided in an embodiment of the present application;

[0046] Figure 7 An optional structural diagram of a probability value determination unit provided in an embodiment of the present application;

[0047] Figure 8 An optional structural diagram of a timing repetition-aware penalty sampling device provided in an embodiment of the present application;

[0048] Fig. 9 An optional structural diagram of another timing repetition-aware penalty sampling device provided in an embodiment of the present application;

[0049] Fig.10 A schematic diagram of the hardware structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0050] In order to make the purpose, technical solutions and advantages of the present application clearer, the present application is further described in detail below in conjunction with the accompanying drawings and examples. It should be understood that the specific embodiments described herein are only used to explain the present application and are not intended to limit the present application. When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the embodiments of the present application. They are only examples of devices and methods consistent with some aspects of the embodiments of the present application as detailed in the attached claims.

[0051] It is understood that the terms "first", "second", etc. used in this application can be used to describe various concepts in this article, but unless otherwise specified, these concepts are not limited by these terms. These terms are only used to distinguish one concept from another concept. For example, without departing from the scope of the embodiment of the present application, the first information may also be referred to as the second information, and similarly, the second information may also be referred to as the first information. Depending on the context, the words "if" and "if" as used herein can be interpreted as "at the time of" or "when" or "in response to determination".

[0052] The terms "at least one", "multiple", "each", "any", etc. used in this application, at least one includes one, two or more, multiple includes two or more, each refers to each of the corresponding multiple, and any refers to any one of the multiple.

[0053] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.

[0054] Before describing the embodiments of the present application in detail, the terms and some related technologies that may be involved in the embodiments of the present application are first described. The terms and related technologies that may be involved in the embodiments of the present application are subject to the following interpretations:

[0055] The principle of speech synthesis (TTS: Text-To-Speech) is the process of converting text information into speech output. The speech synthesis process mainly includes the following key solutions:

[0056] 1. Text preprocessing: converting the input text into a form suitable for further processing. This includes removing irrelevant characters (such as extra spaces, special symbols, etc.), text normalization (such as standardized expression of numbers and dates), vocabulary analysis and segmentation, etc.

[0057] 2. Linguistic analysis: Perform in-depth linguistic processing on the preprocessed text, including part-of-speech tagging, syntactic analysis, and semantic understanding. The goal of this stage is to fully understand the intent and context of the text and lay the foundation for generating natural and fluent speech.

[0058] 3. Voice generation and synthesis: This is the core of TTS technology, including the construction of a voice database, the selection and connection of voice units, and the generation of the final voice through a synthesis algorithm. High-quality voice sample collection is the basis. The system needs to accurately select appropriate voice units from the voice database, and then use sophisticated algorithms to splice and smooth them to reduce the unnaturalness of the synthesized voice and improve the fluency and naturalness of the voice.

[0059] 4. Synthesis algorithm: It is the core technology to realize text-to-speech conversion. It processes and analyzes the text content through specific algorithms, and finally converts the text into understandable speech through the speech synthesis system.

[0060] The application scenarios of speech synthesis in this application include but are not limited to:

[0061] Navigation system: In the in-vehicle navigation system, speech synthesis technology can convert navigation instructions into voice output to improve driving safety.

[0062] Intelligent customer service: In the intelligent customer service system, speech synthesis technology can convert text responses into natural and fluent speech, improving user experience.

[0063] Education: In educational software, speech synthesis technology can convert the content of electronic books into speech, making it easier for the visually impaired or those with reading difficulties to use.

[0064] Information broadcasting: In news, weather, traffic and other information broadcasting, speech synthesis technology can convert text information into speech output in real time, thereby improving the efficiency of information dissemination.

[0065] Speech synthesis decoding refers to the process of converting text into speech signals, which includes the following main steps:

[0066] Text analysis: Normalize the input text, such as converting abbreviations, numbers, dates, etc. into complete text form; perform word segmentation and part-of-speech tagging; understand sentence structure, and determine stress and intonation.

[0067] Language processing: Convert text into a sequence of phonemes and perform prosody prediction to determine prosodic features such as pitch, duration, and intensity.

[0068] Acoustic model: Determines the acoustic parameters of each phoneme and generates basic features of speech, such as fundamental frequency (F0), spectral envelope, and duration.

[0069] Waveform generation: Generate the final speech waveform through different methods. Common methods include concatenation synthesis and parameter synthesis, as follows:

[0070] Concatenative synthesis: Select and concatenate the most appropriate speech segments from a pre-recorded speech database. The advantage is high naturalness, but the disadvantage is that it requires a lot of storage space and has low flexibility.

[0071] Parametric synthesis: Generate speech waveforms using vocoder models such as the source-filter model.

[0072] The above steps work together to convert text into human-audible speech signals.

[0073] Top-p and top-k are two parameters used to control the output of the language model, which are mainly used to adjust the diversity and relevance of the generated text.

[0074] 1.top-k:

[0075] Top-k sampling means that at each time step, the model randomly selects a word from the k candidate words with the highest probability as the output. By adjusting the value of k, the output diversity and accuracy of the model can be controlled. A smaller k value will increase the accuracy of the output but reduce the diversity; a larger k value will increase the diversity but may reduce the accuracy of the output.

[0076] 2.top-p:

[0077] Top-p sampling (also called kernel sampling) selects words based on their cumulative probability. Specifically, the model only considers words whose probability accumulates to a certain percentage p and above. For example, if p = 0.9, the model will only select words whose probability accumulates to the top 90%. This approach helps the model focus more on generating relevant and coherent text.

[0078] Application scenarios and effects:

[0079] Top-k: Suitable for scenarios that require high accuracy, such as legal document writing, technical reports, etc. By reducing the number of candidate words, it can reduce the generation of irrelevant or wrong words and improve the accuracy of the output.

[0080] Top-P: Suitable for scenarios that require higher diversity, such as creative writing, story generation, etc. By selecting words with higher cumulative probability, the diversity and creativity of the output can be increased.

[0081] By setting the above parameters reasonably, a balance can be found between the creativity and output quality of the model to better meet the needs of different application scenarios.

[0082] Speech is the most natural way for humans to interact. As the "mouth" of the human-computer speech interaction system, text-to-speech (TTS) technology aims to convert input text into natural and fluent speech output. In recent years, the field of speech synthesis technology has experienced a significant leap in development, which is mainly due to the application progress of the language model (LM) method. The TTS model method usually draws on prior knowledge from the field of natural language processing, and changes the speech modeling process from the previous continuous modeling method to the next-token prediction method, that is, through the pre-trained neural speech encoder, discrete speech sequences are quantized and extracted from the continuous speech signal, and then the modeling process of these discrete sequences is further optimized. Relying on large-scale speech data, it is not only possible to synthesize high-quality and natural and fluent speech based on only about 3s to 5s of speech prompts, but also effectively retain the characteristics of emotion, speech speed and speaking style contained in the prompt speech during the synthesis process.

[0083] Compared with traditional text-to-speech (TTS) solutions, TTS solutions based on language models (LM) can generate synthesized speech that is more natural, fluent, and close to real human speech, such as GPT-sovits and CosyVoice in related technologies. However, TTS solutions based on language models (LM) also face many challenges, especially in terms of the stability of generated speech. For example, it is sometimes difficult to achieve a stable level consistent with human voice, and sometimes it is even worse than traditional TTS solutions. Specifically, the speech generated by existing speech synthesis solutions may have the following problems:

[0084] 1. Omission or repetition of words: During the synthesis process, due to the accumulated error of the autoregressive model, some words in the original text may be omitted or repeated, affecting the complete transmission of information. This is particularly obvious in long texts or texts with repeated sentences.

[0085] 2. Inaccurate pronunciation: Even the most advanced LM-TTS systems may have deviations in the pronunciation of specific words or phrases, which may be due to limitations in the model training data or inadequate processing of specific language phenomena.

[0086] 3. Inconsistent or jittery timbre: Sudden changes or instability in timbre within a sentence is also a common problem, which will give people a "machine-like" feeling and reduce the naturalness and listening quality of the voice.

[0087] Some related technologies have proposed several follow-up works from different entry points to solve the above problems. The most classic one is the Repetition Aware Sampling (RAS) proposed by VALLE-2, which improves the original kernel sampling process by considering repeated tokens in the decoding history and improves the decoding stability. However, this method still has shortcomings. For example, improper window setting will seriously interfere with the speech rate of the generated audio.

[0088] Based on this, the present application provides a time-series repetition-aware penalty sampling method, device, electronic device and storage medium. Compared with the RAS sampling method and the random sampling method, the present application not only greatly alleviates a series of instability problems such as repetition and missing words that occur when the autoregressive model generates encoding, but also improves the consistency of speech speed, while making the generated audio more in line with the style and emotion of the prompt voice.

[0089] The embodiment of the present application provides a time-series repetition-aware penalty sampling method, device, electronic device and storage medium. The technical solution of the present application includes: inputting a given text into a pre-trained large language model; determining a target penalty probability value according to a first continuous repetition threshold and a probability distribution output by the large language model at a current moment; using the large language model to decode according to the target penalty probability value to obtain a first voice code at a current moment; using the first continuous repetition threshold as a window of length to calculate the continuous repetition rate of the preamble generation code and the first voice code; if the continuous repetition rate is greater than or equal to a second continuous repetition threshold, using the large language model to decode according to the probability distribution output by the large language model at a current moment to obtain a second voice code; replacing the first voice code with the second voice code; returning to the step of determining the target penalty probability value according to the first continuous repetition threshold and the probability distribution output by the large language model at a current moment, until the end code is obtained by decoding using the large language model, and then obtaining the target voice code corresponding to the given text. This application proposes a time-series repetition-aware penalty sampling scheme based on a penalty probability value, which greatly alleviates the problems of repetition, missing words, infinite loops, etc. that frequently occur in large language model speech synthesis schemes. It can also further make the synthesized speech follow the style and emotion of the prompt audio as much as possible, significantly improving the reasoning stability and the natural fluency of the generated audio.

[0090] The embodiments of the present application provide a method, device, electronic device and storage medium for sampling a time-series repetition perception penalty, and relate to the field of artificial intelligence technology. The method, device, electronic device and storage medium for sampling a time-series repetition perception penalty provided in the embodiments of the present application can be applied to a terminal, can also be applied to a server, and can also be software running in a terminal or a server. In some embodiments, the terminal can be a smart phone, a tablet computer, a laptop computer, a desktop computer, a smart speaker, a smart watch, and a car terminal, etc., but are not limited to this; the server side can be configured as an independent physical server, or a server cluster or distributed system composed of multiple physical servers, and can also be configured to provide cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and cloud servers for basic cloud computing services such as big data and artificial intelligence platforms. The server can also be a node server in a blockchain network; the software can be an application that implements a knowledge extraction method, etc., but is not limited to the above forms.

[0091] The present application can be used in many general or special computer system environments or configurations. For example: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, etc. The present application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The present application can also be practiced in distributed computing environments, in which tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media including storage devices.

[0092] Reference Figure 1 , the embodiment of the present application provides a timing repetition-aware penalty sampling method, which may include but is not limited to S100 to S150, as follows:

[0093] S100: Input the given text into the pre-trained large language model.

[0094] S110: Determine a target penalty probability value according to a first continuous repetition threshold and a probability distribution output by the large language model at a current moment.

[0095] Reference Figure 2 , S110 may further include the following steps S111 to S113:

[0096] S111: if the current number of inferences is less than the first continuous repetition threshold, obtaining a first probability value of an index corresponding to a prompt speech code of a first window length from the probability distribution output by the large language model at the current moment, and then reducing the first probability value by a penalty coefficient times to obtain a first penalty probability value;

[0097] S112: If the current number of inferences is greater than or equal to the first continuous repetition threshold, obtaining a second probability value of the index corresponding to the prompt speech code of the second window length from the probability distribution output by the large language model at the current moment, and then reducing the second probability value by the penalty coefficient times to obtain a second penalty probability value;

[0098] S113: Determine the target penalty probability value according to the first penalty probability value or the second penalty probability value.

[0099] More specifically, S113 may include the following steps:

[0100] The first penalty probability value or the second penalty probability value is divided by a preset temperature coefficient to obtain the target penalty probability value.

[0101] S120: Using the large language model to decode according to the target penalty probability value to obtain a first speech code at the current moment.

[0102] Further, S120 may include the following steps:

[0103] The first speech code at the current moment is obtained by using the large language model, using top-p parameters and top-k parameters through polynomial sampling and decoding according to the target penalty probability value.

[0104] S130: Calculate the continuous repetition rate of the preamble generated code and the first speech code by taking the first continuous repetition threshold as a window of length.

[0105] S140: If the continuous repetition rate is greater than or equal to a second continuous repetition threshold, the large language model is used to decode the second speech code according to the probability distribution output by the large language model at the current moment.

[0106] Further, S140 may include the following steps:

[0107] The second speech code is obtained by using the large language model through random polynomial sampling and decoding according to the probability distribution output by the large language model at the current moment.

[0108] S150: Replace the first speech code with the second speech code.

[0109] S160: Return to the step of determining the target penalty probability value according to the first continuous repetition threshold and the probability distribution output by the large language model at the current moment, until the end code is obtained by decoding with the large language model, and then the target speech code corresponding to the given text is obtained.

[0110] Reference Figure 3 , the embodiment of the present application may further include step S170:

[0111] S170: If the continuous repetition rate is less than the second continuous repetition threshold, continue to use the large language model to decode the given text until the end code is obtained by decoding with the large language model, thereby obtaining the target speech code corresponding to the given text.

[0112] Reference Figure 4 As another further implementation, the embodiment of the present application may further include step S180:

[0113] S180: Synthesize speech corresponding to the given text according to the target speech coding.

[0114] Next, the solution of the embodiment of the present application will be introduced and explained in detail with reference to specific application examples.

[0115] Reference Figure 5 , this embodiment provides an algorithm implementation flow chart of the timing repetition perception penalty sampling strategy.

[0116] Specifically, during reasoning, since the random sampling strategy may cause unstable phenomena such as missing words, repeated reading, and infinite loops, this embodiment proposes a time-series repetition-aware penalty sampling strategy, and its algorithm implementation flow chart is as follows: Figure 5 shown. Figure 5 Line 1 in describes the meaning of each parameter. Figure 5 Lines 2-7 in describe the process of temporal repetition perception penalty. If the current number of inferences is less than N (the first consecutive repetition threshold), s is obtained from the probability distribution output by the current GPT (large language model). t′-w1:t′ The first probability value of the corresponding index is reduced by σ (penalty coefficient) times. Since the inference result is easily affected by the indefinite length of silence at the end of the prompt speech at the beginning of the inference, the above steps can effectively alleviate this effect. If the current number of inferences is greater than or equal to N, s will be obtained from the probability distribution of the GPT output at the current moment. -w2-N:-NThe second probability value of the corresponding index is obtained and reduced by σ times. The inventor of the present application has observed from the features extracted from tens of thousands of hours of audio that the number of consecutive repetitions of most discrete speech codes is unlikely to exceed N times, where N is related to the speech quantizer used. The above steps try to keep the decoded speech code and the real speech code in the same distribution as much as possible, that is, to ensure that the generated speech follows the style and emotion of the prompt speech more closely, and at the same time set the first threshold to ensure that the generated code will not repeat the same code as much as possible. Figure 5 Lines 6-10 in the figure describe the process of temporal repetitive perceptual sampling. First, the probability value obtained in line 3 or line 5 is divided by the temperature coefficient, and then the speech code s at the current moment is decoded through polynomial sampling. t (First speech coding). If the previous generated code is within the range of window N (a window with the first continuous repetition threshold as the length) and the current speech coding s t The number of repetitions r is greater than the repetition threshold t r (Second consecutive repetition threshold), random polynomial sampling is used to resample from all probability values ​​with replacement s t , so that the generated code will not have more than t consecutive repetitions within the window length N as much as possible r Subsequent decoding cycles are repeated until the end code is decoded.

[0117] This embodiment proposes a temporal repetition-aware penalty sampling strategy, which greatly alleviates the problems of repetition, missing words, infinite loops, etc. that frequently occur under the LLM-TTS paradigm. At the same time, it further improves the generated audio to follow the style and emotion of the prompt audio as much as possible, and significantly improves the reasoning stability and the natural fluency of the generated audio.

[0118] Reference Figure 6 The embodiment of the present application further provides a timing repetition-aware penalty sampling device, which can implement the above-mentioned timing repetition-aware penalty sampling method, and the device includes:

[0119] Initialization unit, which is used to input the given text into the pre-trained large language model;

[0120] A probability value determination unit, configured to determine a target penalty probability value according to a first continuous repetition threshold and a probability distribution output by the large language model at a current moment;

[0121] A first encoding unit, configured to obtain a first speech code at a current moment by decoding using the large language model according to the target penalty probability value;

[0122] a repetition rate calculation unit, configured to calculate a continuous repetition rate of a preamble generated code and the first speech code using the first continuous repetition threshold as a window length;

[0123] A second encoding unit, configured to obtain a second speech code by decoding using the large language model according to a probability distribution output by the large language model at a current moment if the continuous repetition rate is greater than or equal to a second continuous repetition threshold;

[0124] A coding replacement unit, configured to replace the first speech coding with the second speech coding;

[0125] The first continuous encoding unit is used to return to the step of determining the target penalty probability value according to the first continuous repetition threshold and the probability distribution output by the large language model at the current moment, until the end encoding is obtained by decoding with the large language model, and then the target speech encoding corresponding to the given text is obtained.

[0126] Reference Figure 7 In some embodiments, the probability value determining unit includes:

[0127] A first probability value determination subunit is configured to obtain a first probability value of an index corresponding to a prompt speech code of a first window length from a probability distribution output by the large language model at a current moment if the current number of inferences is less than the first continuous repetition threshold, and then reduce the first probability value by a penalty coefficient to obtain a first penalty probability value;

[0128] A second probability value determination subunit is used to obtain a second probability value of the corresponding index of the prompt speech code of the second window length from the probability distribution output by the large language model at the current moment if the current number of inferences is greater than or equal to the first continuous repetition threshold, and then reduce the second probability value by the penalty coefficient times to obtain a second penalty probability value;

[0129] The third probability value determining subunit is configured to determine the target penalty probability value according to the first penalty probability value or the second penalty probability value.

[0130] In some embodiments, the third probability value determining subunit includes:

[0131] A target probability value determination unit is configured to divide the first penalty probability value or the second penalty probability value by a preset temperature coefficient to obtain the target penalty probability value.

[0132] Reference Figure 8 In some embodiments, the device further comprises:

[0133] The second continuous encoding unit is used to continue to use the large language model to decode the given text if the continuous repetition rate is less than the second continuous repetition threshold, until the end encoding is obtained by decoding with the large language model, and then obtain the target speech encoding corresponding to the given text.

[0134] In some embodiments, the first encoding unit includes:

[0135] The first encoding subunit is used to use the large language model to use top-p parameters and top-k parameters through polynomial sampling and decode according to the target penalty probability value to obtain the first speech code at the current moment.

[0136] In some embodiments, the second encoding unit includes:

[0137] The second encoding subunit is used to obtain the second speech code by using the large language model through random polynomial sampling and decoding according to the probability distribution output by the large language model at the current moment.

[0138] Reference Fig. 9 In some embodiments, the device further comprises:

[0139] The speech synthesis unit is used to synthesize the speech corresponding to the given text according to the target speech coding.

[0140] It can be understood that the contents of the above method embodiments are all applicable to the present device embodiments, the functions specifically implemented by the present device embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0141] The embodiment of the present application also provides an electronic device, the electronic device includes a memory and a processor, the memory stores a computer program, and the processor implements the above-mentioned timing repetition perception penalty sampling method when executing the computer program. The electronic device can be any intelligent terminal including a tablet computer, a car computer, etc.

[0142] It can be understood that the contents of the above method embodiments are all applicable to the present device embodiments, the functions specifically implemented by the present device embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0143] See also Fig.10 , Fig.10 The hardware structure of an electronic device of another embodiment is illustrated, and the electronic device includes:

[0144] The processor 1001 may be implemented by a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of the present application;

[0145] The memory 1002 can be implemented in the form of a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 1002 can store an operating system and other applications. When the technical solution provided in the embodiment of this specification is implemented by software or firmware, the relevant program code is stored in the memory 1002, and the processor 1001 calls and executes the timing repetition perception penalty sampling method of the embodiment of the present application;

[0146] Input / output interface 1003, used to implement information input and output;

[0147] The communication interface 1004 is used to realize the communication interaction between the device and other devices. The communication can be realized through a wired manner (such as USB, network cable, etc.) or a wireless manner (such as mobile network, WIFI, Bluetooth, etc.);

[0148] A bus 1005 , which transmits information between various components of the device (e.g., the processor 1001 , the memory 1002 , the input / output interface 1003 , and the communication interface 1004 );

[0149] The processor 1001 , the memory 1002 , the input / output interface 1003 and the communication interface 1004 are connected to each other in communication within the device via the bus 1005 .

[0150] An embodiment of the present application also provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the above-mentioned time-series repetition-aware penalty sampling method is implemented.

[0151] It can be understood that the contents of the above method embodiments are all applicable to the present storage medium embodiments, the functions specifically implemented by the present storage medium embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0152] The memory, as a non-transient computer-readable storage medium, can be used to store non-transient software programs and non-transient computer executable programs. In addition, the memory may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some embodiments, the memory may optionally include a memory remotely disposed relative to the processor, and these remote memories may be connected to the processor via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0153] The embodiments described in the embodiments of the present application are intended to more clearly illustrate the technical solutions of the embodiments of the present application and do not constitute a limitation on the technical solutions provided in the embodiments of the present application. Those skilled in the art will appreciate that with the evolution of technology and the emergence of new application scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.

[0154] Those skilled in the art will appreciate that the technical solutions shown in the figures do not constitute a limitation on the embodiments of the present application, and may include more or fewer steps than shown in the figures, or a combination of certain steps, or different steps.

[0155] The device embodiments described above are merely illustrative, and the units described as separate components may or may not be physically separated, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0156] Those skilled in the art will appreciate that all or some of the steps in the methods disclosed above, and the functional modules / units in the systems and devices may be implemented as software, firmware, hardware, or a suitable combination thereof.

[0157] The terms "first", "second", "third", "fourth", etc. (if any) in the specification of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0158] It should be understood that in the present application, "at least one (item)" means one or more, and "plurality" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the objects associated before and after are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0159] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the above units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0160] The units described above as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0161] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0162] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including multiple instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of various embodiments of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory, referred to as ROM), random access memory (Random Access Memory, referred to as RAM), disk or optical disk and other media that can store programs.

[0163] The preferred embodiments of the present invention are described above with reference to the accompanying drawings, but the scope of the rights of the present invention is not limited thereto. Any modification, equivalent substitution and improvement made by a person skilled in the art without departing from the scope and essence of the present invention should be within the scope of the rights of the present invention.

Claims

1. A temporal repetition-aware penalty sampling method, characterized in that: The method comprises the following steps: Input the given text into a pre-trained large language model; Determine a target penalty probability value according to a first continuous repetition threshold and a probability distribution output by the large language model at a current moment; Decoding the target penalty probability value using the large language model to obtain a first speech code at the current moment; Calculate the continuous repetition rate of the preamble generated code and the first speech code by taking the first continuous repetition threshold as a window of length; If the continuous repetition rate is greater than or equal to a second continuous repetition threshold, using the large language model to decode according to the probability distribution output by the large language model at the current moment to obtain a second speech code; Replacing the first speech code with the second speech code; Return to the step of determining the target penalty probability value according to the first continuous repetition threshold and the probability distribution output by the large language model at the current moment, until the end code is obtained by decoding with the large language model, and then the target speech code corresponding to the given text is obtained.

2. The temporal repetition-aware penalty sampling method according to claim 1, characterized in that: Determining the target penalty probability value according to the first continuous repetition threshold and the probability distribution output by the large language model at the current moment comprises the following steps: If the current number of inferences is less than the first continuous repetition threshold, obtaining a first probability value of the index corresponding to the prompt speech code of the first window length from the probability distribution output by the large language model at the current moment, and then reducing the first probability value by a penalty coefficient times to obtain a first penalty probability value; If the current number of inferences is greater than or equal to the first continuous repetition threshold, obtaining a second probability value of the index corresponding to the prompt speech code of the second window length from the probability distribution output by the large language model at the current moment, and then reducing the second probability value by the penalty coefficient times to obtain a second penalty probability value; The target penalty probability value is determined according to the first penalty probability value or the second penalty probability value.

3. The temporal repetition-aware penalty sampling method according to claim 2, characterized in that: The step of determining the target penalty probability value according to the first penalty probability value or the second penalty probability value comprises the following steps: The first penalty probability value or the second penalty probability value is divided by a preset temperature coefficient to obtain the target penalty probability value.

4. The temporal repetition-aware penalty sampling method according to claim 1, characterized in that: The method further comprises the following steps: If the continuous repetition rate is less than the second continuous repetition threshold, the large language model is continuously used to decode the given text until an end code is obtained by decoding with the large language model, thereby obtaining a target speech code corresponding to the given text.

5. The temporal repetition-aware penalty sampling method according to claim 1, characterized in that: The method of using the large language model to decode according to the target penalty probability value to obtain the first speech code at the current moment includes the following steps: The first speech code at the current moment is obtained by using the large language model, using top-p parameters and top-k parameters through polynomial sampling and decoding according to the target penalty probability value.

6. The temporal repetition-aware penalty sampling method according to claim 1, characterized in that: The method of using the large language model to decode the second speech code according to the probability distribution output by the large language model at the current moment comprises the following steps: The second speech code is obtained by using the large language model through random polynomial sampling and decoding according to the probability distribution output by the large language model at the current moment.

7. The temporal repetition-aware penalty sampling method according to any one of claims 1 to 6, characterized in that: The method further comprises the following steps: The speech corresponding to the given text is synthesized according to the target speech coding.

8. A time sequence repetition perception penalty sampling device, characterized in that: The device comprises: Initialization unit, which is used to input the given text into the pre-trained large language model; A probability value determination unit, configured to determine a target penalty probability value according to a first continuous repetition threshold and a probability distribution output by the large language model at a current moment; A first encoding unit, configured to obtain a first speech code at a current moment by decoding using the large language model according to the target penalty probability value; a repetition rate calculation unit, configured to calculate a continuous repetition rate of a preamble generated code and the first speech code using the first continuous repetition threshold as a window length; A second encoding unit, configured to obtain a second speech code by decoding using the large language model according to a probability distribution output by the large language model at a current moment if the continuous repetition rate is greater than or equal to a second continuous repetition threshold; A coding replacement unit, configured to replace the first speech coding with the second speech coding; The first continuous encoding unit is used to return to the step of determining the target penalty probability value according to the first continuous repetition threshold and the probability distribution output by the large language model at the current moment, until the end encoding is obtained by decoding with the large language model, and then the target speech encoding corresponding to the given text is obtained.

9. An electronic device, characterized in that: The electronic device includes a memory and a processor, the memory stores a computer program, and the processor implements the temporal repetition-aware penalty sampling method as described in any one of claims 1 to 7 when executing the computer program.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the temporal repetition-aware penalty sampling method according to any one of claims 1 to 7 is implemented.

Images