Text-to-speech conversion method and device based on pseudo-autoregression, equipment and medium

Abstract

The invention relates to the technical field of speech synthesis, can be applied to service system platforms of medical health, financial science and technology and the like, and discloses a text-to-speech method based on pseudo-autoregression, which comprises the following steps: acquiring input information and a semantic token, and constructing a pseudo-autoregression text-to-semantic module; generating a first-stage input sequence according to the input information and the semantic token; the pseudo-autoregression text-to-semantic module masks and predicts the first-stage input sequence to generate a first-stage semantic token; a second-stage input sequence is generated according to the input information and the first-stage semantic token, the pseudo-autoregression text-to-semantic module masks and predicts the second-stage input sequence, and a second-stage semantic token is generated; and inputting the second-stage semantic token into a semantic-to-Mel spectrum module to generate voice data. According to the invention, through the input information, the semantic token and the pseudo-autoregression module, the semantic token is generated through two-stage mask prediction, and then the semantic to Mel spectrum module is input to generate the voice, so that the voice generation quality is improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of speech synthesis, and in particular to a text-to-speech method and device based on pseudo-autoregression, an equipment and a storage medium. BACKGROUND

[0002] Text-to-speech technology has realized efficient conversion from text to speech. After long-term development, it can better reproduce the timbre of the reference speech and accurately synthesize the speech corresponding to the target text. Current mainstream models mainly include autoregressive models and non-autoregressive models. The autoregressive model generates speech with higher naturalness and stronger time consistency, but the generation speed is slow. The non-autoregressive model sacrifices part of the speech quality to significantly improve the synthesis efficiency. In the prior art, an advanced cascaded system performs outstandingly. Through the collaborative work of the three core modules of text-to-semantic, semantic-to-mel spectrum, and text-to-emotion, it has achieved breakthrough progress in emotional expression accuracy, speech duration control, and speech intelligibility. However, this system still has shortcomings. The text-to-semantic module uses an autoregressive method to generate semantic tokens, which directly limits the further improvement of the overall generation speed.

[0003] In the field of medical health, text-to-speech technology faces the problem of core technology adaptation. Medical documents (such as medical records and examination reports) require high speech intelligibility and accurate pronunciation of professional terms. Although the existing cascaded system has made breakthroughs in intelligibility, the autoregressive design of the text-to-semantic module limits the generation speed, making it difficult to meet the time efficiency requirements of emergency notification, remote consultation real-time voice interaction, and other scenarios. At the same time, the medical scene has strict requirements for the stability of the voice emotion (to avoid excessive or distorted emotional expression affecting doctor-patient communication), and the emotion module of the system lacks adaptability, which may cause information transmission errors due to semantic conversion delay and emotional expression deviation.

[0004] In the field of financial technology, the technical bottleneck is the balance between real-time performance and security. Financial transaction notifications, financial product explanations, and other scenarios require millisecond-level voice responses. The non-autoregressive model may cause pronunciation ambiguity of numbers and terms, and the speed limitation of the cascaded system cannot adapt to the real-time broadcast requirements of high-frequency transactions. In addition, financial texts contain sensitive information, and existing models lack targeted privacy protection mechanisms during semantic conversion. The delay of the autoregressive module may affect the efficiency of voice verification, identity verification, and other security aspects. The inherent contradiction between voice quality and generation speed also restricts the experience optimization of intelligent investment real-time voice interaction, credit audit voice notification, and other businesses. SUMMARY

[0005] The main purpose of the present application is to provide a pseudo-autoregressive-based text-to-speech method, device, equipment and storage medium, aiming to solve the problem that in the existing cascaded text-to-speech system, the text-to-semantic module is limited in speed due to the autoregressive generation of semantic tokens, and it is difficult to balance the quality of speech generation and time dimension consistency.

[0006] To achieve the above-mentioned purpose, the present application provides a pseudo-autoregressive-based text-to-speech method, comprising: obtaining input information and semantic tokens, and pre-constructing a pseudo-autoregressive text-to-semantic module; generating a first-stage input sequence according to the input information and the semantic tokens, and inputting the first-stage input sequence into the pseudo-autoregressive text-to-semantic module; masking and predicting the semantic tokens of the first-stage input sequence by the pseudo-autoregressive text-to-semantic module to generate first-stage semantic tokens; generating a second-stage input sequence according to the input information and the first-stage semantic tokens, and masking and predicting the second-stage input sequence by the pseudo-autoregressive text-to-semantic module to generate second-stage semantic tokens; inputting the second-stage semantic tokens into a semantic-to-mel spectrum module for processing to generate speech data.

[0007] Further, to achieve the above-mentioned purpose, the present application provides a pseudo-autoregressive-based text-to-speech device, comprising: a model construction and data acquisition module for obtaining input information and semantic tokens, and pre-constructing a pseudo-autoregressive text-to-semantic module; a first-stage input sequence module for generating a first-stage input sequence according to the input information and the semantic tokens, and inputting the first-stage input sequence into the pseudo-autoregressive text-to-semantic module; a first-stage semantic token module for masking and predicting the semantic tokens of the first-stage input sequence by the pseudo-autoregressive text-to-semantic module to generate first-stage semantic tokens; a second-stage semantic token module for generating a second-stage input sequence according to the input information and the first-stage semantic tokens, and masking and predicting the second-stage input sequence by the pseudo-autoregressive text-to-semantic module to generate second-stage semantic tokens; a text-to-speech module for inputting the second-stage semantic tokens into a semantic-to-mel spectrum module for processing to generate speech data.

[0008] Further, to achieve the above object, the present application also provides a computer device, comprising a memory, a processor, and a pseudo-autoregressive based text-to-speech program stored in the memory and executable on the processor, which, when executed by the processor, implements the steps of the pseudo-autoregressive based text-to-speech method as described above.

[0009] Further, to achieve the above object, the present application also provides a computer readable storage medium, which stores a pseudo-autoregressive based text-to-speech program, which, when executed by a processor, implements the steps of the pseudo-autoregressive based text-to-speech method as described above.

[0010] Beneficial effects: The present application relates to the technical field of speech synthesis, and can be applied to business system platforms such as medical health and financial technology, and discloses a pseudo-autoregressive based text-to-speech method, which comprises the following steps: obtaining input information and semantic tokens, and pre-constructing a pseudo-autoregressive text-to-semantic module; generating a first-stage input sequence according to the input information and the semantic tokens; performing mask and prediction processing on the semantic tokens of the first-stage input sequence by the pseudo-autoregressive text-to-semantic module to generate first-stage semantic tokens; generating a second-stage input sequence according to the input information and the first-stage semantic tokens, and performing mask and prediction on the second-stage input sequence by the pseudo-autoregressive text-to-semantic module to generate second-stage semantic tokens; and inputting the second-stage semantic tokens into a semantic-to-mel-spectrum module to generate speech data. The present application generates semantic tokens through two-stage mask prediction by inputting information, semantic tokens and a pseudo-autoregressive module, and then inputs the semantic tokens into a semantic-to-mel-spectrum module to generate speech, thereby improving the speed of generating semantic tokens and the quality of generating speech. BRIEF DESCRIPTION OF DRAWINGS

[0011] The present application will be further described below in conjunction with the accompanying drawings and embodiments, in which: Figure 1 An application environment diagram of the pseudo-autoregressive based text-to-speech method in an embodiment of the present application; Figure 2 A flow diagram of the pseudo-autoregressive based text-to-speech method in an embodiment of the present application; Figure 3 A functional module diagram of the pseudo-autoregressive based text-to-speech device in a preferred embodiment of the present application; Figure 4 A structure diagram of the computer device in an embodiment of the present application; Figure 5 Another structure diagram of the computer device in an embodiment of the present application. DETAILED DESCRIPTION

[0012] It is to be understood that the specific embodiments described herein are merely illustrative of the present application and do not limit the scope of the application.

[0013] The method for converting text to speech based on pseudo-autoregression provided by the embodiments of the present application can be applied in an application environment as shown in Figure 1 , wherein a user terminal communicates with a server through a network. The server can obtain input information and semantic tokens through the user terminal, pre-construct a pseudo-autoregressive text-to-semantic module, generate a first-stage input sequence according to the input information and the semantic tokens, perform mask and prediction processing on the semantic tokens of the first-stage input sequence by the pseudo-autoregressive text-to-semantic module to generate first-stage semantic tokens, generate a second-stage input sequence according to the input information and the first-stage semantic tokens, perform mask and prediction on the second-stage input sequence by the pseudo-autoregressive text-to-semantic module to generate second-stage semantic tokens, and input the second-stage semantic tokens into a semantic-to-mel spectrum module to generate speech data. The present application improves the speed of generating semantic tokens and the quality of generating speech by inputting information, semantic tokens and a pseudo-autoregressive module, generating semantic tokens through two-stage mask prediction, and then inputting the semantic tokens into a semantic-to-mel spectrum module to generate speech. The user terminal can be, but is not limited to, various personal computers, notebook computers, smart phones, tablet computers and portable wearable devices. The server can be implemented by an independent server or a server cluster composed of multiple servers. The present application will be described in detail through specific embodiments.

[0014] Please refer to Figure 2 , Figure 2 The flowchart of an embodiment of the method for converting text to speech based on pseudo-autoregression provided by the present application is shown in the figure. It should be noted that although a logical sequence is shown in the flowchart, in some cases, the steps shown or described can be performed in an order different from that shown here.

[0015] As shown in Figure 2 , the method for converting text to speech based on pseudo-autoregression provided by the present application includes the following steps: S100, obtaining input information and semantic tokens, and pre-constructing a pseudo-autoregressive text-to-semantic module; S200, generating a first-stage input sequence according to the input information and the semantic tokens, and inputting the first-stage input sequence into the pseudo-autoregressive text-to-semantic module; S300, performing mask and prediction processing on the semantic tokens of the first-stage input sequence by the pseudo-autoregressive text-to-semantic module to generate first-stage semantic tokens; S400, generating a second-stage input sequence according to the input information and the first-stage semantic tokens, and performing mask and prediction on the second-stage input sequence by the pseudo-autoregressive text-to-semantic module to generate second-stage semantic tokens; S500, input the second-stage semantic token into a semantic-to-Mel spectrum module for processing to generate voice data.

[0016] In the complete technical process of text-to-speech in this embodiment, two types of core data and one core module are first needed: input information covers text embedding (Text Embedding, converting text into a vector carrying semantics), speaker embedding (Speaker Embedding, defining the vector of the synthesized voice line), emotional embedding (Emotional Embedding, determining the vector of the emotional tone of the voice), and duration embedding (Duration Embedding, controlling the vector of the voice rhythm); semantic tokens (Semantic Tokes) are discrete units representing semantics extracted from voice data by a semantic encoder; and a pseudo-autoregressive text-to-semantic module (Pseudo-Autoregressive Text-to-Semantic Module) is pre-constructed, which is composed of two bidirectional masked generation transformers (Bidirectional Masked Generation Transformer) and has the advantages of semantic consistency of autoregressive models and parallel generation efficiency of non-autoregressive models.

[0017] Next, the first-stage processing is entered: according to the input information and the initial semantic token, the preset order combination of "speaker embedding + emotional embedding, duration embedding, sequence start boundary token (BT), text embedding, separation boundary token (BA), and initial semantic token" is used to form a clear first-stage input sequence, which is then input into the pseudo-autoregressive text-to-semantic module. The module will first mask all semantic tokens in the sequence (mask all semantic tokens with a specific symbol), then use the bidirectional modeling capability to predict all masked positions in parallel, generate complete prediction results, and then retain high-confidence tokens at the leftmost or core positions at a fixed ratio, repeat the "masking-prediction" process for the remaining tokens, and finally generate complete first-stage semantic tokens. This stage ensures the accuracy of the semantics and the consistency of the time dimension through iterative optimization while ensuring the generation speed.

[0018] Subsequently, the second stage optimization is entered: based on the input information and the first stage semantic tokens, the same sequence is combined into a second stage input sequence and input into the module. In this stage, the full token mask is no longer used for semantic tokens, but the confidence of each token in the first stage is calculated (based on the full token table generation probability, the more concentrated the probability is, the higher the confidence is), and the tokens with a confidence lower than the threshold are screened out for masking, and the context information of the unmasked tokens is combined to re-predict, and the iteration is repeated until all low-confidence tokens are optimized, and finally the second stage semantic tokens are generated. This stage further improves the accuracy and context relevance of semantic representation by accurately correcting low-confidence tokens, and lays a high-quality semantic foundation for subsequent speech synthesis.

[0019] Finally, the speech generation link: the second stage semantic tokens are input into the semantic-to-mel-spectrogram module, which maps the semantic and acoustic features to convert discrete semantic tokens into mel-spectrogram (Mel-Spectrogram, spectral data describing the frequency and amplitude of speech) that conforms to the human ear's hearing characteristics, and then the mel-spectrogram is converted into continuous audio waveform by the vocoder (Vocoder), generating natural, clear and consistent with the input information requirements of the speech data.

[0020] In the field of financial technology business, this process can support efficient voice interaction of intelligent financial services. For example, when the bank's intelligent customer service system handles the customer's demand for "querying the yield of financial products in the past three months", the text embedding in the input information carries the "financial yield query" semantic, the voice embedding is set to professional customer service voice line, the emotion embedding is defined as neutral and rigorous, and the time length embedding matches the query text speed; Through two-stage semantic token optimization, it is ensured that the key financial information such as "past three months" and "yield" is accurate and accurate, and the subsequent generated speech can clearly convey the yield data while maintaining a professional and efficient communication atmosphere. In the financial market dynamic broadcast scene, this process can quickly convert real-time text information into speech, and through accurate semantic tokens and adaptive emotion embedding (such as positive tone when the market rises), it helps users to efficiently obtain reliable market information in commuting and other scenarios.

[0021] In the medical and health service field, this process can help the landing of personalized medical voice services. For example, when a hospital generates a guiding voice for a diabetic patient to take antihypertensive drugs 30 minutes after breakfast and monitor blood pressure twice a week, the emotional embedding of the input information is set to be gentle and soothing, the timbre embedding matches the patient's preferred warm tone, the duration embedding controls the speech rate to ensure that the elderly can clearly understand; the two-stage semantic token optimization can accurately lock in key medical instructions such as "30 minutes after breakfast" and "twice a week" to avoid semantic deviation; the final generated voice not only conveys accurate medical guidance but also relieves patient anxiety and improves medication and monitoring compliance. In the remote consultation scenario, when a doctor's text diagnosis suggestion is converted into voice through this process, it can embed professional timbre and rigorous emotion to ensure the authority and clarity of the diagnosis information, while making the patient feel cared for and optimizing the remote diagnosis experience.

[0022] In one embodiment, the S100 comprises: S101, obtaining text data, processing the text data to generate text embedding, timbre embedding, emotion embedding, and duration embedding; S102, obtaining a voice signal in the training data, and converting the voice signal into semantic tokens through a semantic encoder; S103, constructing a pseudo-autoregressive text-to-semantic module through two bidirectional mask generation transformers.

[0023] In this embodiment, in the early preparation and module construction of the text-to-speech technology, first, text data needs to be obtained and processed in multiple dimensions to generate four types of core embedding vectors. Among them, the text embedding is converted into a low-dimensional vector that can be understood by a computer through a natural language processing model (such as BERT, Transformer, etc.), which retains the key information of the text's semantics and grammatical structure during the process, such as converting "account balance inquiry" into a vector that can represent the "financial inquiry" semantic; the timbre embedding is to extract and quantify the voice line characteristics of the target speaker, generate a vector by analyzing the speaker's pitch, timbre, and speech rate, etc. acoustic properties, which can be used for subsequent synthesis of specific person's voice timbre, such as simulating the bank customer service exclusive voice line; the emotion embedding identifies the emotional tendency (such as positive, negative, neutral, or "rigorous" in the financial scenario, "soothing" in the medical scenario) in the text through emotion analysis algorithm and converts it into vector form to ensure that the synthesized voice conveys the matching emotional tone; the duration embedding is to quantify the voice duration corresponding to each text unit by combining the text length, sentence structure, and voice generation requirements, generating a vector for controlling the voice rhythm to avoid problems such as too fast or too slow speech rate.

[0024] Next, the voice signal in the training data needs to be obtained and converted into semantic tokens by means of a semantic encoder (a model focusing on extracting semantic information of voice). The voice signal itself is a continuous acoustic waveform, and the semantic encoder will convert the waveform data into a discrete token sequence that can represent the core semantics of voice through steps such as feature extraction and semantic mapping. For example, the voice signal of "please repay the loan on time" will be converted into a token combination corresponding to the "reminder to repay" semantic. These semantic tokens not only represent the condensed expression of voice semantics, but also provide a basis for subsequent semantic matching of text and voice and optimization of voice synthesis quality.

[0025] Finally, a pseudo-autoregressive text-to-semantic module needs to be constructed in advance, which is composed of two bidirectional mask generation transformers with the same structure. The bidirectional mask generation transformer has bidirectional modeling capability and can capture context information from both ends of the sequence, providing a more comprehensive understanding of semantic associations than unidirectional models. The "mask generation" mechanism allows the model to mask and re-predict some tokens in the sequence, thereby improving the accuracy and robustness of semantic representation. The two transformers with the same structure work together to support different stages of subsequent semantic token generation, retaining the modeling advantages of autoregressive models in time consistency while integrating the parallel processing efficiency of non-autoregressive models, providing a module basis for fast and high-quality semantic token generation.

[0026] In the field of financial technology business, this series of processes can support the optimization of voice interaction for intelligent financial services. For example, when a bank builds an intelligent customer service voice system, it first generates text embeddings for customer consultation texts (such as "financial product yield calculation" and "credit card bill installment"), generates timbre embeddings based on customer-specific voice lines, generates "professional and rigorous" sentiment embeddings for consultation scenarios, and generates length embeddings that adapt to the speed of the text. Then, through the semantic encoder, historical customer service voice is converted into semantic tokens for model training. Finally, relying on the module composed of double bidirectional mask generation transformers, efficient conversion of text to semantic tokens is achieved. Based on this, the customer service system can quickly synthesize voice responses that meet customer needs, accurately conveying professional information such as financial calculation logic and business rules, while improving customer communication experience through matching timbre and sentiment, and meeting the response speed requirements of a large number of concurrent customer inquiries. In addition, in the generation of financial risk prompt voice, through "serious" sentiment embedding and accurate semantic tokens, the content of the risk prompt can be clear and solemn, helping customers to pay attention to risk information.

[0027] In the medical and health service field, this process can help the personalization and precision of medical voice services. Taking a hospital intelligent inquiry voice system as an example, the system first generates a text embedding for the patient's inquiry text (such as "cough for a week, worse at night"), generates a "mild and kind" timbre embedding according to the patient's preference, generates a "reassuring and concerned" emotion embedding according to the inquiry scene, and generates a natural speech speed time length embedding according to the inquiry content length. Then, through the semantic encoder, the doctor's inquiry voice and disease guidance voice are converted into semantic tokens for model training. With the help of a double bidirectional mask generation transformer module, efficient conversion of inquiry text to semantic tokens is achieved. The subsequently synthesized voice can not only accurately convey the doctor's inquiry guidance, disease analysis and other professional content, but also alleviate patient anxiety through soothing emotions and kind timbre, improving the inquiry experience. In the generation of discharge guidance voice, through precise time length embedding to control the speech rate, combined with clear semantic tokens, the patient can clearly understand the key information such as medication dosage and review time, reducing the risk of rehabilitation due to information transmission bias.

[0028] In one embodiment, the S200 comprises: S201, determining a boundary token of a sequence starting position and a boundary token separating input information and semantic tokens according to the semantic tokens; S202, combining the input information, semantic tokens and boundary tokens in a predetermined order to form a first-stage input sequence; S203, inputting the first-stage input sequence into a pseudo self-recursive text-to-semantics module.

[0029] In this embodiment, in the semantic token processing process of the text-to-speech technology, first, two types of key boundary tokens (Boundary Token, abbreviated as BT and BA, special identification tokens for clarifying sequence structure and distinguishing different types of information) need to be determined according to the semantic tokens: one is the boundary token of the sequence starting position (usually BT), which is used to mark the beginning of the entire input sequence to avoid confusion between subsequent information and external irrelevant data; the other is the boundary token separating input information and semantic tokens (usually BA), which is used to clearly separate the two core data of input information and semantic tokens to prevent semantic cross or interference of information from different sources in subsequent processing. The core purpose of this step is to build an ordered sequence framework through boundary tokens to provide clear structural guidance for subsequent information combination.

[0030] Then, the input information, semantic tokens and boundary tokens are combined in a preset order to form a first-stage input sequence. The input information includes text embedding, timbre embedding, sentiment embedding and duration embedding; the preset order usually follows the logic of "basic features-text information-separation identifier-semantic representation", for example, the combination in the order of "timbre embedding + sentiment embedding, duration embedding, sequence start boundary token (BT), text embedding, separation boundary token (BA), semantic token". This ordered combination can help the subsequent module to clearly identify the functions of different types of information and avoid processing errors caused by data disorder.

[0031] Finally, the first-stage input sequence constructed is input into the pseudo-autoregressive text-to-semantic module. Based on the various types of embedding information and the structure divided by the boundary tokens in the input sequence, the module performs preliminary generation and optimization of the semantic tokens, laying the foundation for the accurate iteration of the semantic tokens in the subsequent stage. The key lies in efficiently extracting core features such as text semantics, timbre, and sentiment with the help of clear information boundaries of the boundary tokens, ensuring that the generated semantic tokens not only conform to the text meaning but also match the preset voice style and duration requirements.

[0032] In the field of financial technology business, this process can support personalized voice responses for intelligent financial customer service. For example, when a customer inquires about the "yield of a certain fund in the past year and redemption rules" through text, the system will first determine the sequence start boundary token (BT) and the boundary token (BA) that separates the input information and the semantic token, then combine "the customer's specified warm timbre embedding, neutral sentiment embedding (commonly used in consultation scenarios), duration embedding adapted to the length of the financial text, text embedding of the customer's input text, semantic token" in a preset order to form a first-stage input sequence, and input it into the pseudo-autoregressive text-to-semantic module. Through clear boundary token division, the module can clearly distinguish between voice control information such as timbre and sentiment and text semantic information such as fund yield and redemption rules, generate semantic tokens that not only conform to the customer's timbre preference but also accurately convey financial information, and the subsequent synthesized voice can provide customers with a comfortable listening experience while obtaining professional financial answers; in addition, in the generation of marketing voice for financial products, by reasonably combining "enthusiastic sentiment embedding, product introduction text embedding, semantic token", the synthesized voice can be both persuasive and clear in conveying the core advantages of the product, improving marketing effectiveness.

[0033] In the medical and health service field, this process can help generate personalized patient guidance voice. For example, when generating a voice for a diabetic patient to remind them to take medication and pay attention to diet, the system will first determine the boundary token, then embed the patient's preferred tone of voice, soothing emotional embedding (to alleviate the patient's anxiety), adapt the daily reminder speed, text embedding of medication and diet guidance text, semantic token in the preset order into the first stage input sequence, and input the pseudo self-recurrent text to semantic module. Boundary tokens can accurately distinguish between different information such as "soothing emotions", "guidance text", and "semantic representation", and the generated semantic tokens not only meet the patient's auditory preferences and psychological needs, but also accurately convey key medical information such as medication time and dietary restrictions. The subsequent synthesized voice can help patients clearly understand and follow medical guidance; in the remote consultation scenario, after the doctor inputs the consultation advice in text, the system generates an input sequence containing "professional tone embedding", "rigorous emotional embedding", and "consultation advice text embedding" using this process, and the synthesized voice can maintain the professionalism and seriousness of the medical advice while making the patient feel cared for, improving the communication effectiveness of remote diagnosis and treatment.

[0034] In one embodiment, the S300 comprises: S301, performing full masking on the semantic tokens of the first stage input sequence by the pseudo self-recurrent text to semantic module to generate masked semantic tokens; S302, predicting the masked semantic tokens by the pseudo self-recurrent text to semantic module to obtain a prediction result; S303, extracting and retaining a preset number of prediction semantic tokens at a preset position in the prediction result; S304, performing full masking and prediction on other prediction semantic tokens in the prediction result until a complete semantic token sequence is generated, to obtain first stage semantic tokens.

[0035] In this embodiment, in the first stage of semantic token generation for text-to-speech, the pseudo self-recurrent text to semantic module is used to complete the initial construction of semantic tokens. First, the module performs full masking on the semantic tokens in the first stage input sequence (full masking refers to covering all original semantic tokens in the sequence with a specific masking symbol, temporarily masking their information, and only retaining other key information in the input sequence such as text embedding, tone embedding, emotional embedding, duration embedding, and boundary tokens), generating a masked semantic token sequence - the purpose of this step is to allow the model to learn semantic representations based on core information such as text semantics and tone preferences, avoiding the accumulation of initial errors, and to avoid interference from original tokens.

[0036] Subsequently, the pseudo-autoregressive text-to-semantic module will perform a full prediction on the masked semantic tokens (Full Prediction, that is, the model analyzes the context information in the input sequence and predicts all semantic tokens at the masked positions in one step to generate a complete prediction result sequence). Unlike the traditional autoregressive model, which generates tokens one by one, this parallel prediction can greatly improve the speed of generating semantic tokens, and at the same time, with the help of bidirectional modeling capability, it can capture more comprehensive context associations, laying the foundation for subsequent optimization.

[0037] After the prediction is completed, the predicted semantic tokens at the preset positions and the preset number need to be extracted from the prediction results and retained. Here, "preset positions" and "preset number" are set in advance according to the structural characteristics of the semantic sequence and the generation requirements, such as preferentially retaining tokens in the starting part of the sequence and the core part of the semantics, or retaining the leftmost or most critical tokens according to a fixed proportion (such as 20%) of the total length of the semantic tokens. The reason for retaining these tokens is that they usually carry the most core semantic information and have relatively high generation confidence, which can be used as "anchor points" for subsequent iterative generation, ensuring the overall consistency and accuracy of the semantic sequence.

[0038] Finally, the non-retained predicted semantic tokens in the prediction results are masked again, and the above "masking-prediction-retaining" process is repeated until a complete first-stage semantic token sequence is generated. In this iterative process, the high-confidence tokens retained each time provide more explicit semantic constraints for the next round of prediction, gradually correcting possible deviations, and ultimately generating first-stage semantic tokens that guarantee both speed and basic semantic accuracy, preparing for the precise optimization of the second stage.

[0039] In the field of financial technology business, this process can be applied to the intelligent financial information voice synthesis scenario. For example, when generating the information voice "a listed company's revenue in Q1 2024 increased by 15% year-on-year, and the net profit exceeded market expectations", the first-stage input sequence will include text embedding (vector representation of information text), timbre embedding (set professional broadcast timbre), duration embedding (control the speed of broadcast), etc. After the module masks the initial semantic tokens and predicts the complete semantic sequence in parallel, it preferentially retains the semantic tokens corresponding to the core financial data such as "revenue growth 15%" and "net profit exceeds expectations" (preset key information positions), and then iteratively masks and predicts the tokens that are not core but necessary such as "listed company name" and "Q1 time range". The final first-stage semantic tokens can quickly and accurately carry the core information of the information, ensuring that the subsequent synthesized voice clearly conveys the key financial data, helping users efficiently obtain market dynamics, while balancing the speed of broadcast and the accuracy of information.

[0040] In the medical health service field, the process can be used for generating patient visit guidance voice. For example, when generating the guidance voice "the patient needs to take antihypertensive drugs 30 minutes after breakfast every day, and monitor blood pressure 2 times a week and record it", the first stage input sequence includes text embedding (guidance text vector), timbre embedding (mild medical staff timbre), emotion embedding (soothing emotion tendency), etc. The module first masks the initial semantic token in full amount, then predicts in parallel, and then prioritizes the semantic tokens corresponding to the core medical instructions (preset key medical behavior positions) such as "take medicine 30 minutes after breakfast" and "monitor blood pressure 2 times a week", and then iteratively optimizes the detail tokens such as "antihypertensive drug name" and "blood pressure recording method". Through this process, the first stage semantic token carrying the core guidance information can be quickly generated, ensuring that the synthesized voice accurately delivers the key medical instructions such as medication time and monitoring frequency, avoiding medication errors or monitoring omissions caused by semantic deviation, and improving the generation efficiency of the guidance voice, meeting the needs of hospitals to generate personalized visit guidance in batches.

[0041] In one embodiment, the S400 comprises: S401, combining the input information, the first stage semantic token and the boundary token in a preset order to form a second stage input sequence; S402, obtaining the generation probability distribution of each semantic token in the first stage semantic token in the full token table; S403, calculating the confidence of each semantic token according to the generation probability distribution in the full token table; S404, comparing the confidence with a confidence threshold, obtaining the semantic token with a confidence lower than the confidence threshold, and performing mask processing on the semantic token to generate a masked semantic token; S405, predicting the masked semantic token based on the second stage input sequence through the pseudo self-recursive text-to-semantics module, updating the semantic token with a confidence lower than the confidence threshold, until the confidence of all semantic tokens is higher than the confidence threshold, to generate a second stage semantic token.

[0042] In this embodiment, in the semantic token optimization process of text-to-speech, the input information, the first stage semantic token and the boundary token are first combined in a preset order to form a second stage input sequence. The input information includes text embedding, timbre embedding, emotion embedding and duration embedding; the first stage semantic token is a sequence with basic semantic representation ability generated by pseudo self-recursion; the boundary token (BT and BA) is used to clearly define the start and end boundaries of the sequence, to avoid semantic confusion when different types of input information are combined, and finally form a second stage input sequence with clear structure and complete information, providing stable basic data for subsequent optimization.

[0043] Next, the generation probability distribution of each token in the first-stage semantic token in the full token table needs to be obtained. The full token vocabulary is a set containing all possible semantic tokens. When each first-stage semantic token is generated, the model calculates the probability of each token in the full token vocabulary, which reflects the model's initial judgment tendency for the token generation result. Based on these probabilities, the confidence of each semantic token can be further calculated, usually using probability maximum, entropy value and other indicators for quantification, for example, taking the maximum probability value in the probability distribution of the token as the confidence. The closer the maximum value is to 1, the higher the model's trust in the token generation result. If the probability distribution is scattered and the maximum value is low, the confidence will decrease, indicating that the semantic representation of the token may be biased.

[0044] Subsequently, the calculated confidence is compared with the preset confidence threshold. The confidence threshold is a judgment standard set according to actual needs, used to filter tokens with insufficient semantic representation reliability. When the confidence of a token is lower than the threshold, it is determined as a low-confidence token, which needs to be masked. Masking refers to replacing low-confidence tokens with specific mask symbols, temporarily masking their original information, to avoid inaccurate semantics interfering with subsequent predictions. Finally, a sequence of masked semantic tokens containing mask symbols is obtained.

[0045] Finally, the second-stage input sequence and the sequence of masked semantic tokens are input into the pseudo self-attention text-to-semantics module. Based on the context information such as text, tone and emotion in the input sequence, the module re-predicts the semantics of the masked positions, generates more accurate tokens to replace the mask symbols, and updates the low-confidence tokens. This updating process is repeated until all low-confidence tokens are replaced by tokens with confidence that meets the standard and accurate semantics, generating a sequence of second-stage semantic tokens. This sequence has both generation speed and semantic accuracy, providing a high-quality semantic basis for subsequent speech synthesis.

[0046] In the field of financial technology business, this process can be applied to voice response optimization of intelligent financial customer service. For example, when a customer inquires about the "calculation method of financial product redemption fee" through text, the first-stage semantic token may have some low-confidence tokens due to the professional nature of terms such as "redemption fee" and "calculation method". Through the above process, the system will filter out these low-confidence tokens and update the mask, generate second-stage semantic tokens with accurate semantics by combining professional corpus and context information in the financial field. Based on the synthesized voice of the token, the fee calculation logic can be accurately conveyed, avoiding customer misunderstanding due to semantic deviation, while improving customer service response speed and ensuring the professionalism and efficiency of financial services. In addition, in the financial information broadcast scenario, for the semantic representation of key data such as "stock price fluctuation" and "fund net value fluctuation", this process can ensure the accuracy of the token, so that the synthesized voice accurately conveys financial data and helps users quickly obtain reliable information.

[0047] In the field of medical health business, this process can help improve the accuracy of intelligent inquiry and voice of medical knowledge. For example, when a patient describes "postprandial blood glucose control range for diabetic patients" through text, the first-stage semantic token may have low-confidence tokens due to the difference between medical terms such as "postprandial blood glucose" and "control range" and the patient's expression habits. Through mask updating, the system will generate second-stage semantic tokens with accurate semantics by combining medical guidelines and clinical corpus, ensuring that the synthesized voice accurately conveys professional information such as "diabetic patients should control their 2-hour postprandial blood glucose to below 7.8 mmol / L", avoiding medical advice errors due to semantic deviation. At the same time, in the generation of discharge guidance voice, for the semantic tokens of key medical instructions such as "dose" and "review interval", this process can ensure accuracy through repeated optimization, so that the voice guidance received by the patient is clear and reliable, reducing the risk of medication errors or review delays due to information errors, and improving the safety of medical services and patient compliance.

[0048] In one embodiment, the S500 comprises: S501, input the second-stage semantic token into a semantic-to-Mel spectrum module; S502, analyze and process the second-stage semantic token through the semantic-to-Mel spectrum module to generate a Mel spectrum; S503, synthesize voice data based on the Mel spectrum.

[0049] In this embodiment, in the technical process of text-to-speech, when the second-stage semantic token (a semantic representation unit with higher context consistency and semantic accuracy after pseudo-self-recurrence optimization) is generated, it will be input into the semantic-to-mel-spectrogram module (a core processing unit responsible for converting abstract semantic information into acoustic features). First, the module deeply analyzes the second-stage semantic token, combines the text semantics, sentiment orientation, timbre preference and other information contained in the token, and establishes the mapping relationship between semantics and acoustic features through a neural network model (such as Transformer, convolutional neural network, etc.) - the core is to convert discrete semantic token sequences into continuous mel-spectrogram (Mel-Spectrogram, a spectral representation method that simulates human auditory characteristics, compared with traditional spectrum, it is more in line with the perception law of human ear to sound frequency, and can more accurately capture key acoustic information such as timbre, pitch, rhythm, etc. of speech). In the process of generating mel-spectrogram, the module will make full use of the advantages of the second-stage semantic token: since the token in this stage has been optimized for low confidence, the semantic expression is more accurate, and the module can more accurately control the details of the speech duration, pause, emotional intensity, etc. For example, according to the token features related to "emergency" in the semantics, generate mel-spectrogram with urgent rhythm and higher pitch; according to the token features related to "gentle", generate mel-spectrogram with gentle rhythm and soft pitch.

[0050] After completing the generation of mel-spectrogram, it enters the speech data synthesis link. This step is usually realized through a vocoder (a tool that converts spectral features into audible speech signals) - the vocoder reads the frequency, amplitude, phase and other acoustic information contained in the mel-spectrogram, combines the preset timbre parameters (such as personal voice line features extracted from timbre embedding), and converts the spectral data into continuous audio waveform through signal processing algorithm, finally generates natural, clear and semantic and emotional speech data. Throughout the process, the high accuracy of the second-stage semantic token lays the foundation for the accurate generation of mel-spectrogram, and the adaptability of mel-spectrogram to human auditory characteristics further guarantees the naturalness and intelligibility of the final synthesized speech.

[0051] In the field of financial technology business, this technical process can be widely applied in intelligent customer service and financial information broadcasting scenarios. For example, in a bank intelligent customer service system, when handling customer needs for loan application progress, credit card bill inquiry, etc., the system converts the customer's text request into second-stage semantic tokens, generates a mel spectrum that fits the formal and clear tone of the financial scenario through the semantic-to-mel spectrum module, and then synthesizes the voice to feedback information to the customer. Compared with traditional voice synthesis, the optimized semantic tokens can make the module more accurately capture the serious tone of "overdue bill reminder" and the positive tone of "financial income broadcast", thereby improving the customer's reception efficiency and trust in the information. In the financial market dynamic broadcast, the system can quickly synthesize a voice containing the amplitude of rise and fall and market analysis based on the semantic tokens generated from real-time text information, and push it to the user through a securities company APP or a smart speaker, helping the user to efficiently obtain financial information in scenarios such as commuting, and the high restoration of the mel spectrum can ensure the clarity of the broadcast voice, avoiding interference with investment decisions due to ambiguous information transmission.

[0052] In the field of medical and health business, this technical process can help with intelligent consultation, patient care, and medical knowledge popularization. In an intelligent consultation platform, when a patient describes symptoms (such as "persistent cough, night fever") through text, the second-stage semantic tokens generated by the system will accurately carry information such as the severity of the symptoms and the patient's anxiety, and the semantic-to-mel spectrum module will generate a mel spectrum that is gentle and professional, synthesizing a voice to confirm the details of the symptoms or give preliminary health advice, thereby alleviating the patient's anxiety while ensuring the accuracy of the medical guidance. In postoperative follow-up, hospitals can use this technology to convert postoperative care text plans (such as "wound dressing frequency, dietary restrictions") into voice, adjust the speed and tone of the voice according to the patient's age and condition (such as using a slower speed and a more friendly tone for elderly patients), and improve the patient's understanding and compliance with the care plan. In the context of medical knowledge popularization, this technology can convert complex medical knowledge texts (such as "chronic disease prevention guidelines") into clear and understandable voice, which can be pushed to the public through a health-related APP, and the mel spectrum's accurate control of voice details can make users with different knowledge backgrounds easily receive medical information, helping to widely disseminate health knowledge.

[0053] In one embodiment, the S100 further comprises: S1021, obtaining text data, preprocessing the text data through text segmentation and semantic analysis to obtain sentiment expression keywords, tone words, and sentence structure features; S1022, based on a pre-set sentiment feature extraction model, extracting sentiment features from the preprocessed text data to obtain multi-dimensional sentiment features; S1023, mapping the extracted multi-dimensional sentiment features into a fixed-dimensional sentiment feature vector through a fully connected layer or an embedding mapping network, and taking the sentiment feature vector as a sentiment embedding.

[0054] In this embodiment, in the text data preprocessing stage, the continuous text is first broken down into word or phrase units with independent semantics through text segmentation, and then the logical relationship between words and the core meaning of the sentence are clarified through semantic analysis, and then the sentiment expression keywords (such as core words reflecting specific emotional tendencies), mood words (such as auxiliary words enhancing emotional color), and sentence structure features (such as exclamatory sentences, double negative sentences, and other sentence structure features that can reflect emotional intensity) are accurately screened out. Then, based on a pre-set sentiment feature extraction model (which can analyze the pre-processed text in multiple dimensions through a deep learning algorithm), multi-dimensional sentiment features including sentiment tendency (positive, negative, neutral), sentiment intensity (strong, moderate, weak), and sentiment category (such as joy, worry, anger, and other specific emotional types) are extracted. Subsequently, through a fully connected layer (a neural network structure that can realize nonlinear conversion and integration of features) or an embedding mapping network (a network model that converts high-dimensional, discrete features into low-dimensional, continuous vectors), these multi-dimensional sentiment features are mapped into a fixed-dimensional sentiment feature vector, which is the emotional embedding (a vector form that can quantitatively represent text sentiment information, which can be used for subsequent model calculation and processing). The "embedding" refers to the conversion of discrete semantics or features into representations in a continuous vector space, facilitating computer understanding and operation.

[0055] In the field of financial technology business, this technical process has important application value. Financial institutions generate a large amount of text data every day, such as customer complaint emails, online consultation dialogues, product evaluation messages, and market dynamic analysis reports. Through text segmentation and semantic analysis, keywords such as "expected income not met", "high service fee", and "slow service response" can be extracted from these data, identifying words with negative emotions such as dissatisfaction and anxiety, as well as sentence structures that reflect strong customer demands. With the help of sentiment feature extraction models, we can further obtain multi-dimensional features such as customer sentiment towards financial products or services (positive, negative), sentiment intensity (such as mild complaints, severe dissatisfaction), and sentiment embedding through fully connected layers or embedding mapping networks. These sentiment embeddings can be applied to customer emotion monitoring systems, helping financial institutions monitor customer satisfaction in real time, respond to high-priority negative emotional feedback in a timely manner, and optimize product design and service processes. In the intelligent investment advisor scenario, sentiment embedding can combine customer risk preference text information to more accurately match investment products and improve the personalization level of investment advisor services. In addition, by analyzing the sentiment features of market-related text, we can assist in judging market sentiment trends and provide data support for risk prevention and control.

[0056] In the field of medical and health services, this technical process also has a wide range of application scenarios. Medical text data is diverse, including patient medical records, feedback, online consultation dialogues, doctor-patient communication records, and medical product usage evaluations. Through text segmentation and semantic analysis, sentiment-related keywords such as "pain intensifies", "treatment is effective", and "uncomfortable after taking medication" can be extracted, identifying words that express emotions such as worry and joy, and specific sentence structures that describe the severity of the disease or the effectiveness of treatment. Using sentiment feature extraction models, we can extract multi-dimensional sentiment features such as patient acceptance of treatment plans, the degree of physical and mental distress caused by illness, and satisfaction with medical services, and convert them into sentiment embeddings. In patient management, these sentiment embeddings can help medical staff quickly understand patients' treatment experiences and psychological states, provide timely psychological counseling and treatment plan adjustments for patients with negative emotions, and improve treatment outcomes. In the optimization of medical service quality, by analyzing a large amount of patient sentiment feedback, we can accurately identify weak links in medical services, such as communication efficiency and the convenience of the medical process, and promote continuous improvement of hospital services. In the field of medical product development, sentiment embeddings can reflect patients' feelings and emotional tendencies towards medical devices and drugs, providing a basis for product iteration and upgrading, and helping to develop medical products that better meet patient needs and improve patient experience.

[0057] In an embodiment, a pseudo-autoregressive based text-to-speech device is provided, which corresponds to the pseudo-autoregressive based text-to-speech method described above. Referring to Figure 3 , Figure 3 A functional module diagram of a preferred embodiment of the pseudo-autoregressive based text-to-speech device of the present application is shown in FIG. 1. The functional modules include a model construction and data acquisition module 10, a first stage input sequence module 20, a first stage semantic token module 30, a second stage semantic token module 40, and a text-to-speech module 50. The details of each functional module are described as follows: The model construction and data acquisition module 10 is configured to acquire input information and semantic tokens, and to pre-construct a pseudo-autoregressive text-to-semantic module. The first stage input sequence module 20 is configured to generate a first stage input sequence based on the input information and semantic tokens, and to input the first stage input sequence into the pseudo-autoregressive text-to-semantic module. The first stage semantic token module 30 is configured to perform masking and prediction processing on the semantic tokens of the first stage input sequence by the pseudo-autoregressive text-to-semantic module, to generate first stage semantic tokens. The second stage semantic token module 40 is configured to generate a second stage input sequence based on the input information and first stage semantic tokens, and to perform masking and prediction on the second stage input sequence by the pseudo-autoregressive text-to-semantic module, to generate second stage semantic tokens. The text-to-speech module 50 is configured to input the second stage semantic tokens into a semantic-to-mel-spectrum module for processing, to generate speech data.

[0058] In an embodiment, the model construction and data acquisition module 10 includes: acquiring text data, processing the text data to generate text embeddings, timbre embeddings, sentiment embeddings, and duration embeddings; acquiring speech signals in the training data, and converting the speech signals into semantic tokens by a semantic encoder; constructing a pseudo-autoregressive text-to-semantic module by two bidirectional masking generation transformers.

[0059] In an embodiment, the first stage input sequence module 20 includes: determining boundary tokens for sequence start positions and boundary tokens for separating the input information and semantic tokens based on the semantic tokens; combining the input information, semantic tokens, and boundary tokens in a predetermined order to form a first stage input sequence; inputting the first stage input sequence into the pseudo-autoregressive text-to-semantic module.

[0060] In one embodiment, the first-stage semantic token module 30 comprises: The semantic tokens of the first-stage input sequence are fully masked by the pseudo-autoregressive text-to-semantic module to generate masked semantic tokens. The masked semantic tokens are predicted by the pseudo-autoregressive text-to-semantic module to obtain a prediction result. The prediction result is extracted and reserved at a preset position and a preset number of prediction semantic tokens. The other prediction semantic tokens in the prediction result are fully masked and predicted until a complete semantic token sequence is generated, and the first-stage semantic token is obtained.

[0061] In one embodiment, the second-stage semantic token module 40 comprises: The input information, the first-stage semantic token, and the boundary token are combined in a preset order to form a second-stage input sequence. The generation probability distribution of each semantic token in the first-stage semantic token in the full token table is obtained. The confidence of each semantic token is calculated according to the generation probability distribution in the full token table. The confidence is compared with a confidence threshold to obtain a semantic token with a confidence lower than the confidence threshold, and the semantic token is masked to generate a masked semantic token. The pseudo-autoregressive text-to-semantic module is used to predict the masked semantic token based on the second-stage input sequence to update the semantic token with a confidence lower than the confidence threshold until the confidence of all semantic tokens is higher than the confidence threshold, and the second-stage semantic token is generated.

[0062] In one embodiment, the text-to-speech module 50 comprises: The second-stage semantic token is input into the semantic-to-mel spectrum module. The second-stage semantic token is parsed by the semantic-to-mel spectrum module to generate a mel spectrum. The mel spectrum is used to synthesize speech data.

[0063] In one embodiment, the model construction and data acquisition module 10 further comprises: Text data is obtained, and the text data is preprocessed by text segmentation and semantic analysis to obtain sentiment expression keywords, tone words, and sentence structure features. Based on a preset sentiment feature extraction model, the preprocessed text data is subjected to sentiment feature extraction to obtain multi-dimensional sentiment features. The extracted multi-dimensional sentiment features are mapped to a fixed-dimensional sentiment feature vector through a fully connected layer or an embedding mapping network, and the sentiment feature vector is used as a sentiment embedding.

[0064] In one embodiment, a computer device is provided, which can be a server, and an internal structure diagram thereof can be as shown in Figure 4 The computer device includes a processor, a memory, a network interface and a database connected through a system bus. The processor of the computer device is configured to provide computing and control capabilities. The memory of the computer device includes a non-volatile and / or volatile storage medium, an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operating system and the computer program in the non-volatile storage medium to run. The network interface of the computer device is configured to communicate with an external user terminal through a network connection. The computer program, when executed by the processor, implements the functions or steps of a pseudo-autoregressive text-to-speech method server side.

[0065] In one embodiment, a computer device is provided, which can be a user terminal, and an internal structure diagram thereof can be as shown in Figure 5 The computer device includes a processor, a memory, a network interface, a display screen and an input device connected through a system bus. The processor of the computer device is configured to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operating system and the computer program in the non-volatile storage medium to run. The network interface of the computer device is configured to communicate with an external server through a network connection. The computer program, when executed by the processor, implements the functions or steps of a pseudo-autoregressive text-to-speech method user terminal side. In one embodiment, a computer device is provided, which includes a memory, a processor and a computer program stored in the memory and executable on the processor, and the processor executes the computer program to implement the following steps: Obtain input information and semantic tokens, and pre-construct a pseudo-autoregressive text-to-semantic module; Generate a first-stage input sequence according to the input information and the semantic tokens, and input the first-stage input sequence into the pseudo-autoregressive text-to-semantic module; Perform mask and prediction processing on the semantic tokens of the first-stage input sequence by the pseudo-autoregressive text-to-semantic module to generate first-stage semantic tokens; Generate a second-stage input sequence according to the input information and the first-stage semantic tokens, and perform mask and prediction on the second-stage input sequence by the pseudo-autoregressive text-to-semantic module to generate second-stage semantic tokens; Input the second-stage semantic tokens into a semantic-to-mel spectrum module for processing to generate voice data.

[0066] In one embodiment, a computer readable storage medium is provided, having stored thereon a computer program, which when executed by a processor implements the following steps: obtaining input information and semantic tokens, and pre-constructing a pseudo-autoregressive text-to-semantic module; generating a first-stage input sequence according to the input information and the semantic tokens, and inputting the first-stage input sequence into the pseudo-autoregressive text-to-semantic module; masking and predicting the semantic tokens of the first-stage input sequence by the pseudo-autoregressive text-to-semantic module to generate first-stage semantic tokens; generating a second-stage input sequence according to the input information and the first-stage semantic tokens, and masking and predicting the second-stage input sequence by the pseudo-autoregressive text-to-semantic module to generate second-stage semantic tokens; inputting the second-stage semantic tokens into a semantic-to-mel-spectrum module for processing to generate voice data.

[0067] It should be noted that the functions or steps that the above computer readable storage medium or computer device can implement can correspond to the related descriptions of the server side and the user side in the foregoing method embodiments. To avoid repetition, they will not be described one by one here.

[0068] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer readable storage medium. When the computer program is executed, it can include the processes of the above-mentioned embodiments. Any reference to memory, storage, database or other medium used in the embodiments provided by the present application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration but not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), Rambus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0069] Those skilled in the art can clearly understand that, for the convenience and brevity of description, only the division of the above functional units and modules is taken as an example, and in actual application, the above functions can be completed by different functional units and modules according to needs, that is, the internal structure of the device is divided into different functional units or modules to complete all or part of the functions described above.

[0070] It should be noted that if non-company software tools or components appear in the embodiments of the present application, they are only used for example introduction and do not represent actual use. The above-described embodiments are only used to illustrate the technical solutions of the present application, and not to limit them; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that the technical solutions recorded in the foregoing embodiments can be modified, or some technical features can be replaced by equivalents; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application, and should be included in the protection scope of the present application.

Claims

1. A text-to-speech method based on pseudo-autoregression, characterized in that, Includes the following steps: Obtain input information and semantic tokens, and pre-build a pseudo-autoregressive text-to-semantic module; A first-stage input sequence is generated based on the input information and semantic token, and the first-stage input sequence is input into the pseudo-autoregressive text-to-semantic module. The pseudo-autoregressive text-to-semantic module performs masking and prediction processing on the semantic tokens of the first-stage input sequence to generate the first-stage semantic tokens. The second-stage input sequence is generated based on the input information and the first-stage semantic token. The pseudo-autoregressive text-to-semantic module performs masking and prediction on the second-stage input sequence to generate the second-stage semantic token. The second-stage semantic token is input into the semantic-to-Mel-spectrum module for processing to generate speech data.

2. The text-to-speech method based on pseudo-autoregression as described in claim 1, characterized in that, The process of acquiring input information and semantic tokens, and pre-constructing a pseudo-autoregressive text-to-semantic module, includes: Acquire text data, process the text data, and generate text embeddings, timbre embeddings, emotion embeddings, and duration embeddings; Acquire speech signals from training data and convert the speech signals into semantic tokens using a semantic encoder; A pseudo-autoregressive text-to-semantic module is constructed by generating transformers using two bidirectional masks.

3. The text-to-speech method based on pseudo-autoregression as described in claim 1, characterized in that, The step of generating a first-stage input sequence based on the input information and semantic token, and inputting the first-stage input sequence into the pseudo-autoregressive text-to-semantic module, includes: The boundary token for determining the start position of the sequence and the boundary token separating the input information from the semantic token are determined based on the semantic token. The input information, semantic token, and boundary token are combined in a preset order to form the first-stage input sequence. The first-stage input sequence is input into the pseudo-autoregressive text-to-semantic module.

4. The text-to-speech method based on pseudo-autoregression as described in claim 1, characterized in that, The step of masking and predicting the semantic tokens of the first-stage input sequence by the pseudo-autoregressive text-to-semantic module to generate first-stage semantic tokens includes: The pseudo-autoregressive text-to-semantic module performs a full masking of the semantic tokens of the first-stage input sequence to generate masked semantic tokens. The masked semantic token is predicted using a pseudo-autoregressive text-to-semantic module, and the prediction results are obtained. Extract and retain the predicted semantic tokens at preset positions and a preset number from the prediction results; The other predicted semantic tokens in the prediction results are fully masked and predicted until a complete semantic token sequence is generated, thus obtaining the first-stage semantic token.

5. The text-to-speech method based on pseudo-autoregression as described in claim 1, characterized in that, The step of generating a second-stage input sequence based on the input information and the first-stage semantic token, and the pseudo-autoregressive text-to-semantic module masking and predicting the second-stage input sequence to generate a second-stage semantic token, includes: According to a preset order, the input information, the first-stage semantic token, and the boundary token are combined to form the second-stage input sequence; Obtain the generation probability distribution of the full token table corresponding to each semantic token in the first stage semantic token; The confidence level of each semantic token is calculated based on the generation probability distribution in the full token table; The confidence level is compared with the confidence threshold to obtain semantic tokens with a confidence level lower than the confidence threshold, and the semantic tokens are masked to generate masked semantic tokens. The pseudo-autoregressive text-to-semantic module predicts masked semantic tokens based on the second-stage input sequence, updates semantic tokens with confidence levels below the confidence threshold, until the confidence levels of all semantic tokens are above the confidence threshold, and generates second-stage semantic tokens.

6. The text-to-speech method based on pseudo-autoregression as described in claim 1, characterized in that, The step of inputting the second-stage semantic token into the semantic-to-Mel-spectrum module for processing to generate speech data includes: Input the second-stage semantic token into the semantic-to-Mel spectrum module; The semantic tokens of the second stage are parsed and processed by the semantic-to-Mel spectrum module to generate a Mel spectrum; Speech data is synthesized based on the Mel spectrum.

7. The text-to-speech method based on pseudo-autoregression as described in claim 1, characterized in that, The step of acquiring text data, processing the text data, and generating sentiment embeddings further includes: The text data is acquired and preprocessed through text segmentation and semantic analysis to obtain sentiment expression keywords, modal particles, and sentence structure features. Based on a pre-defined sentiment feature extraction model, sentiment features are extracted from pre-processed text data to obtain multi-dimensional sentiment features. By using fully connected layers or embedding mapping networks, the extracted multi-dimensional sentiment features are mapped to fixed-dimensional sentiment feature vectors, and these sentiment feature vectors are used as sentiment embeddings.

8. A text-to-speech device based on pseudo-autoregression, characterized in that, The text-to-speech device based on pseudo-autoregression includes: The model building and data acquisition module is used to acquire input information and semantic tokens, and a pseudo-autoregressive text-to-semantic module is pre-built. The first-stage input sequence module is used to generate a first-stage input sequence based on the input information and semantic token, and input the first-stage input sequence into the pseudo-autoregressive text-to-semantic module; The first-stage semantic token module is used to mask and predict the semantic tokens of the first-stage input sequence by the pseudo-autoregressive text-to-semantic module to generate the first-stage semantic tokens. The second-stage semantic token module is used to generate a second-stage input sequence based on the input information and the first-stage semantic token. The pseudo-autoregressive text-to-semantic module performs masking and prediction on the second-stage input sequence to generate the second-stage semantic token. The text-to-speech module is used to input the semantic token from the second stage into the semantic-to-Mel-spectrum module for processing, generating speech data.

9. A computer device, characterized in that, The computer device includes a memory, a processor, and a pseudo-autoregressive text-to-speech program stored in the memory and executable on the processor. When executed by the processor, the pseudo-autoregressive text-to-speech program implements the steps of the pseudo-autoregressive text-to-speech method as described in any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, The storage medium stores a pseudo-autoregressive text-to-speech program, which, when executed by a processor, implements the steps of the pseudo-autoregressive text-to-speech method as described in any one of claims 1-7.

Images