Voice generation method and device, equipment and medium

Abstract

The invention relates to the technical field of artificial intelligence and the field of financial science and technology, and discloses a voice generation method, which comprises the following steps of: acquiring a to-be-converted text and a reference voice sample, extracting voice features of reference voice by using a pre-trained self-supervised learning encoder, converting the to-be-converted text into voice features through a text-to-voice feature model, and generating a voice feature. And analyzing the similarity of the two groups of features, determining the voice features of the target speaker from the two groups of features, fusing the two groups of features to generate final fused voice features, and decoding by using a pre-trained vocoder to generate voice signals. According to the method, feature matching and fusion are carried out under the small sample or zero sample condition, the voice similar to the target voice can be generated without a large amount of voice data, the voice synthesis efficiency and quality are effectively improved, the multi-scene application requirement is met, and the method is particularly suitable for voice generation of unseen speakers.

Description

Technical Field

[0001] The present invention relates to the fields of artificial intelligence technology and financial technology, and in particular to a speech generation method, device, equipment and storage medium. Background Art

[0002] In recent years, the financial industry has gradually introduced intelligent voice technology to optimize customer service and improve user experience. Among them, text-to-speech (TTS) systems have been widely used in scenarios such as automatic customer service, risk notifications, and smart investment advisors. However, the existing TTS technology has many shortcomings when facing multi-speaker scenarios, which hinders its in-depth application in the financial industry.

[0003] Existing TTS models usually require building a dedicated model for each voice personalization requirement, which places a high threshold on data requirements. Financial institutions need to provide voice services in multiple languages ​​and styles, but existing models often rely on large amounts of transcribed voice data sets for training and cannot effectively handle unseen speakers. This makes it difficult for models to cope with frequently changing customer needs in actual financial applications, especially when personalized voice needs to be generated quickly, as existing models lack flexibility.

[0004] In addition, the financial industry has strict requirements on the quality of speech synthesis. Especially when interacting with customers, the naturalness of the generated speech and the similarity to the target voice are required to be very high. However, when processing the speech generation of unseen speakers, the existing TTS models often fail to achieve the expected results. There is a significant gap in the naturalness and similarity of the generated speech, which affects the communication effect between financial institutions and customers and may even reduce customer trust.

[0005] Therefore, when faced with diverse voice demands, the current TTS technology in the financial field has problems such as high requirements for voice data sets, difficulty in processing unseen speakers, and insufficient quality of generated voice. These shortcomings seriously limit the further promotion and application of intelligent voice technology in the financial field. Summary of the invention

[0006] The main purpose of the present invention is to provide a speech generation method, apparatus, device and storage medium, aiming to solve the technical problem that the existing speech generation technology relies on large-scale transcribed speech data sets for model training, and it is difficult to generate high-quality and natural speech signals in unseen speaker scenarios.

[0007] To achieve the above object, the present invention provides a speech generation method, comprising:

[0008] Obtain the text to be converted and the reference speech sample;

[0009] Using a pre-trained self-supervised learning encoder to extract features from the reference speech sample to obtain a first speech feature of the target speaker;

[0010] Converting the text to be converted into a second speech feature using a text-to-speech feature model;

[0011] analyzing a similarity between the second speech feature and the first speech feature, and determining a target speaker feature from the first speech feature based on the similarity;

[0012] Fusing the second speech feature with the target speaker feature to generate a fused speech feature;

[0013] The fused speech features are decoded using a pre-trained vocoder to generate a speech signal of a target speaker based on the text to be converted.

[0014] Furthermore, to achieve the above object, the present invention provides a speech generating device, comprising:

[0015] A data acquisition module, used to acquire the text to be converted and reference speech samples;

[0016] A self-supervised learning feature extraction module, used to extract features from the reference speech sample using a pre-trained self-supervised learning encoder to obtain a first speech feature of a target speaker;

[0017] A text-to-speech feature conversion module, used for converting the text to be converted into a second speech feature using a text-to-speech feature model;

[0018] a feature similarity analysis module, configured to analyze the similarity between the second voice feature and the first voice feature, and determine a target speaker feature from the first voice feature based on the similarity;

[0019] A feature fusion module, used for fusing the second speech feature with the target speaker feature to generate a fused speech feature;

[0020] The vocoder decoding module is used to decode the fused speech features using a pre-trained vocoder to generate a speech signal of a target speaker based on the text to be converted.

[0021] Furthermore, to achieve the above-mentioned purpose, the present invention also provides a computer device, which includes a memory, a processor, and a speech generation program stored in the memory and executable on the processor, and the speech generation program implements the steps of the speech generation method described above when executed by the processor.

[0022] Furthermore, to achieve the above-mentioned purpose, the present invention also provides a computer-readable storage medium, on which a speech generation program is stored, and when the speech generation program is executed by a processor, the steps of the speech generation method described above are implemented.

[0023] Beneficial effects: The present invention relates to the fields of artificial intelligence technology and financial technology, and discloses a speech generation method, which obtains a text to be converted and a reference speech sample, uses a pre-trained self-supervised learning encoder to extract the speech features of the reference speech, converts the text to be converted into speech features through a text-to-speech feature model, analyzes the similarity of the two sets of features, determines the speech features of the target speaker, fuses the two sets of features to generate the final fused speech features, and uses a pre-trained vocoder to decode and generate a speech signal. The present invention can generate speech similar to the target sound without a large amount of speech data by performing feature matching and fusion under small sample or zero sample conditions, effectively improving the efficiency and quality of speech synthesis, meeting the application requirements of multiple scenarios, and is particularly suitable for speech generation of speakers that have never been seen. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] The present invention will be further described below with reference to the accompanying drawings and embodiments, in which:

[0025] Figure 1 A schematic diagram of an application environment of a speech generation method according to an embodiment of the present invention;

[0026] Figure 2 A schematic diagram of a flow chart of an embodiment of a speech generation method of the present invention;

[0027] Figure 3 A schematic diagram of functional modules of a preferred embodiment of a speech generating device of the present invention;

[0028] Figure 4 A schematic diagram of the structure of a computer device in one embodiment of the present invention;

[0029] Figure 5 FIG. 4 is another schematic diagram of the structure of a computer device in one embodiment of the present invention. DETAILED DESCRIPTION

[0030] It should be understood that the specific embodiments described herein are only used to explain the present invention, and are not used to limit the present invention.

[0031] The speech generation method provided by the embodiment of the present invention can be applied in Figure 1In an application environment, the user end communicates with the server end through a network. The server end can obtain the text to be converted and the reference speech sample through the user end, extract the speech features of the reference speech by using a pre-trained self-supervised learning encoder, convert the text to be converted into speech features by using a text-to-speech feature model, analyze the similarity of the two sets of features, determine the speech features of the target speaker, fuse the two sets of features, generate the final fused speech features, and use the pre-trained vocoder to decode and generate a speech signal. The present invention can generate speech similar to the target sound without a large amount of speech data by performing feature matching and fusion under small sample or zero sample conditions, effectively improving the efficiency and quality of speech synthesis, meeting the application requirements of multiple scenarios, and is particularly suitable for speech generation of speakers that have not been seen. Among them, the user end can be but is not limited to various personal computers, laptops, smart phones, tablet computers and portable wearable devices. The server end can be implemented by an independent server or a server cluster consisting of multiple servers. The present invention is described in detail below through specific embodiments.

[0032] See also Figure 2 , Figure 2 This is a flow chart of an embodiment of a speech generation method provided by the present invention. It should be noted that although a logical sequence is shown in the flow chart, in some cases, the steps shown or described may be performed in a different order than that shown here.

[0033] like Figure 2 As shown, the speech generation method proposed by the present invention comprises the following steps:

[0034] S10, obtaining the text to be converted and the reference speech sample;

[0035] In this embodiment, the system obtains the text to be converted from the user or application, usually a natural language sentence or paragraph, as the source data for speech generation. The text can be instructions, questions, or notifications in financial services entered by the user. The system can read the text through the user interface, API calls, or from a database. In order to support a variety of application scenarios, the acquisition of text needs to consider different input methods, such as manual input or automatic system crawling. The input of the text can be multi-language support, and issues such as encoding compatibility need to be handled.

[0036] The reference voice sample is used to extract the voice features of the target speaker. The sample can be an audio uploaded by the user or a sample provided in the system's preset voice library. It is usually used to generate a voice style consistent with the target speaker. Reference voice samples can be obtained in a variety of ways, such as users uploading audio files through the interface, or the system automatically selecting from the voice library. Voice samples need to be formatted to ensure a uniform sampling rate (such as 16kHz) and audio format (such as PCM) for subsequent processing. The clarity of the voice samples must also be ensured during the acquisition process to avoid noise interference.

[0037] Example: A financial institution provides personalized investment advice to customers through an intelligent customer service system. The system extracts the text content to be sent to customers from the database (such as financial product introductions, market analysis report summaries), and calls the reference voice samples specified by the financial institution (such as recordings of specific financial advisors). Through the step of "obtaining the text to be converted and the reference voice samples", the system can convert the text content to be sent into the voice style of the financial advisor to provide a more natural and credible voice response, simulating the explanation of a real advisor.

[0038] By obtaining the text to be converted and reference speech samples, a high-quality input data foundation is provided for subsequent feature extraction and speech generation.

[0039] S20, extracting features from the reference speech sample using a pre-trained self-supervised learning encoder to obtain a first speech feature of the target speaker;

[0040] In this embodiment, the self-supervised learning encoder can automatically extract feature information from speech by pre-training on a large amount of unlabeled speech data. The encoder has learned high-level features related to language, pronunciation, timbre, etc., and is suitable for generating high-quality speech synthesis tasks. First, the system loads the pre-trained self-supervised learning encoder into the operating environment. The encoder may be a model such as Wav2Vec, HuBERT or WavLM, which can extract rich speech features through self-supervised learning of large-scale speech data. With the reference speech sample as input, the encoder extracts audio features through a series of convolutional layers and Transformer layers.

[0041] After entering the encoder, the reference speech sample is decomposed into different time frames (usually each frame represents a 25 millisecond speech signal). The self-supervised learning encoder extracts the corresponding speech features from each time frame, including pitch, speaking speed, rhythm and other information. The system first divides the reference speech sample into multiple time frames. The encoder processes each time frame and outputs a feature vector of fixed dimension (for example, 1024 dimensions). These feature vectors encapsulate the speaker's voice features and the language information of the speech, generating a feature matrix for use in the subsequent speech generation step.

[0042] After feature extraction, the encoder encapsulates the generated feature vector as the first speech feature of the target speaker, which serves as the basis for subsequent similarity analysis with the text features to be converted. The feature vector extracted from the reference speech sample is time-aligned to ensure that the length of each time frame feature is consistent, and then these features are combined into an overall feature sequence as the first speech feature of the target speaker. This feature can be represented as a matrix of a specific dimension (for example, T×1024, where T is the number of time frames) for further feature analysis and speech generation tasks.

[0043] Example description: In financial applications, financial institutions hope to provide personalized services to customers through speech generation technology. After the system obtains the reference speech sample of the financial advisor, it uses the self-supervised learning encoder to extract features from the sample and generate the speech feature vector of the financial advisor. These feature vectors will be used to generate automatic replies that match the advisor's voice, ensuring that customers feel a real and credible service experience when interacting with the system. For example, when a customer inquires about the risks and benefits of a financial product, the system will generate a voice explanation consistent with the advisor's voice based on the advisor's voice features, which enhances the customer's trust and satisfaction with the automatic service.

[0044] By using a pre-trained self-supervised learning encoder to extract features from reference speech samples, it is possible to efficiently extract the speech features of the target speaker under small or even zero sample conditions, significantly reducing the reliance on large-scale annotated data while maintaining high quality and naturalness during the generation process.

[0045] S30, converting the text to be converted into a second speech feature using a text-to-speech feature model;

[0046] In this embodiment, the text-to-speech feature model is a tool for converting text into speech features, usually based on deep learning technology, such as models such as GlowTTS or GradTTS. The model converts semantic, syntactic and other information in the text into speech features, representing the pronunciation style and voice characteristics of the source speaker. The text to be converted is input into the text-to-speech feature model, and the model converts the semantic information of the text into a corresponding speech feature sequence through a pre-trained network structure. The core of the text-to-speech model is to use text features, such as phonemes, tones, intonations and other information, to generate speech features that match the text content.

[0047] Before entering the model, the text needs to be segmented and annotated to ensure that the model can correctly understand the phonetic features of the text (such as phonemes, intonation, etc.). This is the basic step of text-to-speech feature conversion. The text is processed through NLP tools or customized segmentation algorithms to convert it into a representation at the phoneme or syllable level. After segmentation, each word in the text is annotated with phonetic information, such as phonemes, stress, rhythm, etc. This process ensures that the input data can adapt to the model structure and produce correct speech features.

[0048] The second speech feature is the speech feature generated by the text-to-speech model based on the text to be converted, which represents the pronunciation style and speech information of the source speaker. These features are the basis of speech generation, mainly including pitch, rhythm, and sound length. The speech feature generated by the text-to-speech feature model is a sequence of multiple time frames, usually a vector sequence of fixed dimensions (such as 1024 dimensions for each time frame). The generated features represent the speech characteristics of the source speaker, including the rhythm, stress, and intonation of the pronunciation, ensuring that the generated speech features can accurately reflect the text content.

[0049] Example description: In a financial scenario, the system needs to generate a voice response based on the text input by the user (such as asking for details of a financial product). Through the text-to-speech feature model, the system can convert the text content "This financial product has an annual yield of 5% and low risk" into voice features and generate voice output similar to that of a financial advisor. This process does not require a large-scale data set and can generate personalized voice services in real time, which improves customer experience while saving labor costs.

[0050] By using the text-to-speech feature model, the system can automatically and efficiently convert the text to be converted into corresponding speech features without the need for large-scale labeled data, thereby effectively solving the text-to-speech conversion problem in speech generation.

[0051] S40, analyzing the similarity between the second voice feature and the first voice feature, and determining a target speaker feature from the first voice feature based on the similarity;

[0052] In this embodiment, similarity analysis is to determine the degree of proximity of two different speech features (the second speech feature and the first speech feature) by calculating the similarity between them. The similarity calculation here usually uses cosine similarity, Euclidean distance or other suitable measurement methods. First, the second speech feature and the first speech feature are divided into multiple frames, and each time frame represents the speech information of the time period. By calculating the similarity between the feature vectors of each time frame, the system generates a similarity matrix. Cosine similarity is a commonly used method, which measures the similarity by calculating the angle between two vectors. The closer the value is to 1, the more similar the two vectors are. This process can be efficiently calculated based on GPU acceleration.

[0053] The target speaker feature is determined by analyzing the similarity matrix and selecting the first voice feature part that is closest to the second voice feature. The target speaker feature is an important basis for imitating the target speaker's voice style when the speech is finally generated. In the similarity matrix, each second voice feature frame has a similarity value with multiple first voice feature frames. The system will select the highest number of first voice feature frames from these similarity values, perform a weighted average on them, and generate the target speaker feature. This feature represents the voice information that is closest to the target speaker's voice style.

[0054] Example description: In the financial field, customers may interact with the intelligent customer service system through voice, such as consulting detailed information such as loan interest rates or financial products. Through the similarity analysis system, the system can select the voice features of the financial advisor that are closest to the customer's input from the stored voice samples of the financial advisor, and generate a personalized voice response. For example, when a customer asks "Please explain the risks of this financial product", the system will select the target speaker features that best match the text based on the text of the customer's question and the preset voice sample through similarity analysis, and generate a voice explanation similar to the voice of the financial advisor. This not only improves customer trust, but also makes the services of financial institutions more intelligent and efficient.

[0055] Through similarity analysis, the system can accurately determine the target speaker features that best match the source speech features from multiple speaker features. This similarity-based feature selection method can generate speech that is highly consistent with the target speaker's speech style under small or even zero sample conditions, avoiding the traditional method's reliance on large amounts of labeled data.

[0056] S50, fusing the second speech feature with the target speaker feature to generate a fused speech feature;

[0057] In this embodiment, in order to generate a speech that conforms to the speech style of the target speaker, it is necessary to fuse the second speech feature (source speech feature) with the target speaker feature. The second speech feature represents the speech feature of the text to be converted, and the target speaker feature represents the speech style of the target speaker. These two features (the second speech feature and the target speaker feature) are used as input and sent to a feature fusion module. The module usually merges the two through a specific fusion algorithm (such as linear interpolation, weighted fusion, etc.) to generate a final fused speech feature.

[0058] The second speech feature and the target speaker feature may usually have different time lengths. In order to ensure that the two can be accurately fused, time alignment must be performed first to make the time frames of the two consistent. Through a time alignment algorithm, such as dynamic time warping (DTW), the two feature sequences are time aligned to ensure that each time frame can correspond to each other. The aligned feature sequences will have the same time length, which is convenient for subsequent fusion operations.

[0059] The fused speech feature is a feature sequence generated by combining the second speech feature and the target speaker feature, representing a speech feature that matches both the content of the text to be converted and the speech style of the target speaker. The fused feature sequence will contain information such as the pitch, speech rate, and rhythm of the two input features. The generated fused speech feature is the result of feature weighted averaging or interpolation processing, and can be used in subsequent speech decoding steps. The feature sequence is usually represented as a sequence of time frames, each of which contains the fused audio features.

[0060] Example description: In the financial field, feature fusion technology can help the system generate personalized voice responses that are consistent with the voice style of financial advisors. For example, a customer may inquire about the risk information of a financial product. The system will generate a second voice feature (source voice feature) based on the customer's question content, and fuse it through the target speaker feature (preset financial advisor voice feature), and finally generate a voice response with the tone and speed of the financial advisor. This approach not only improves the naturalness of the customer experience, but also ensures the consistency of the service brand.

[0061] By fusing the second voice features with the target speaker features, the system can generate voice features that retain the original text content and conform to the target speaker's voice style, improving the flexibility and naturalness of voice generation. Without the need for large amounts of data or complex training, it can quickly generate voices that are consistent with the target speaker, which is particularly suitable for scenarios that require simulating multiple voice styles.

[0062] S60, using a pre-trained vocoder to decode the fused speech feature to generate a speech signal of a target speaker based on the text to be converted.

[0063] In this embodiment, the vocoder is a tool for converting speech features into actual speech signals. The pre-trained vocoder has been trained on a large amount of speech data and can decode the input speech features into an audio waveform. Common vocoders include HiFi-GAN, WaveGlow, etc.

[0064] HiFi-GAN (High-Fidelity Generative Adversarial Networks) is a high-quality vocoder that converts speech features into audio waveforms through the architecture of generative adversarial networks (GANs). HiFi-GAN is known for its efficient speech generation capabilities and low computational costs, while maintaining the high naturalness and clarity of the generated speech. The main features of this model include: HiFi-GAN improves the quality of speech generation through adversarial training of generator and discriminator networks. The generator is responsible for generating high-fidelity audio waveforms from the input speech features, while the discriminator attempts to distinguish between the generated audio and the real audio, driving the generator to continuously improve the generation results; HiFi-GAN can not only generate high-quality audio, but also has significant advantages in inference speed, which is suitable for real-time speech generation; the model uses a multi-level discriminator to ensure that the generated audio is judged at different scales, thereby ensuring high fidelity and naturalness.

[0065] WaveGlow is a flow-based model for speech generation that combines the reversible architecture of the Glow model with the efficient features of WaveNet to achieve high-fidelity audio generation. The WaveGlow model is characterized by its direct generation of audio waveforms without the need for complex post-processing. WaveGlow uses a flow model that allows reversible transformation of audio data to ensure the precise correspondence between input speech features and output waveforms. The flow model essentially transforms simple Gaussian noise into complex audio signals through a series of reversible function transformations. Compared to sampling-based models such as WaveNet, WaveGlow does not require complex sampling inference during the generation process, so it is faster in audio generation. WaveGlow can generate audio waveforms in an end-to-end framework through a single model without the need for additional processing steps or models.

[0066] The fused speech features are input into a pre-trained vocoder. The vocoder generates the corresponding audio signal based on the input feature sequence. The vocoder model is usually able to process information such as frequency, pitch, and rhythm, and map it into actual audible sounds. The advantage of using a pre-trained model is that it can generate high-quality speech without complex training.

[0067] The target speech signal is the final speech signal generated after feature fusion and decoding, which can accurately reflect the voice characteristics of the target speaker and the content of the text to be converted. The decoding process ensures the naturalness and semantic consistency of the speech signal. When the vocoder decodes the fused speech features, it generates a complete audio waveform. This waveform is generated based on the target speaker's features, and its pitch, speaking speed, rhythm, etc. are consistent with the target speaker's speech style. The generated audio signal can be output through the system or played on the user interface.

[0068] Example description: In financial applications, the system can quickly generate voice notifications or financial advisor responses through pre-trained vocoders. For example, when a customer receives a voice notification about a newly launched financial product, the system will use the vocoder to decode the fused voice features into a voice signal consistent with the voice style of a specific financial advisor, and send a detailed description of the product's features and benefits to the customer. This generation process not only maintains the natural fluency of the voice, but also enhances the personalized service experience of the financial brand.

[0069] By pre-training the vocoder to decode and fuse the speech features, the system can efficiently generate speech signals that match the style of the target speaker and are accurate in content. There is no need to retrain the vocoder model, which improves generation efficiency and ensures the naturalness and clarity of speech generation. It is suitable for a variety of application scenarios, especially in environments that require high-fidelity speech generation.

[0070] The present invention relates to the fields of artificial intelligence technology and financial technology, and discloses a method for speech generation, which obtains a text to be converted and a reference speech sample, extracts speech features of the reference speech using a pre-trained self-supervised learning encoder, converts the text to be converted into speech features through a text-to-speech feature model, analyzes the similarity of the two sets of features, determines the speech features of the target speaker, fuses the two sets of features to generate the final fused speech features, and uses a pre-trained vocoder to decode and generate a speech signal. The present invention can generate speech similar to the target sound without a large amount of speech data by performing feature matching and fusion under small sample or zero sample conditions, effectively improving the efficiency and quality of speech synthesis, meeting the application requirements of multiple scenarios, and is particularly suitable for speech generation of speakers that have never been seen.

[0071] In one embodiment, the above S20 includes:

[0072] S201, dividing the reference speech sample into a plurality of audio segments according to a time series, and performing frame-level processing on each audio segment to obtain a plurality of frame data;

[0073] S202, inputting a plurality of frame data into a pre-trained self-supervised learning encoder;

[0074] S203, extracting multidimensional speech features including language information and speaker identity information from a plurality of frame data using the self-supervised learning encoder, wherein the language information includes semantics, syntactic structure and / or pronunciation;

[0075] S204: Use the multi-dimensional speech feature as a first speech feature of the target speaker, and perform time alignment processing on the first speech feature.

[0076] In this embodiment, the reference speech sample needs to be segmented into multiple continuous audio segments before entering the encoder. Each segment is further processed into time frames, and the division of frames enables the system to extract speech features frame by frame, which is convenient for subsequent feature extraction operations. Usually, the audio is divided into time frames of fixed length (such as 25 milliseconds), and each frame contains the audio data of this period. The system first receives the complete reference speech sample and divides it into multiple audio segments (such as 100 milliseconds for each segment) through a time series segmentation method. Each segment is further divided into several time frames, and the time frame contains the speech feature information of the segment. This process can be implemented by audio processing tools or function libraries such as Librosa and FFmpeg.

[0077] Frame data is the smallest processing unit after audio segmentation. The pre-trained self-supervised learning encoder will process these frame data to extract the characteristic information of speech. The input frame data contains information such as the language, pitch, and rhythm of the speech. Each time frame is input into a self-supervised learning encoder, such as Wav2Vec, HuBERT, etc. These encoders have been pre-trained with a large amount of unlabeled speech data and can automatically extract useful speech features from the input frame. After each time frame of data passes through the encoder, a feature vector containing rich language and identity information will be generated.

[0078] The self-supervised learning encoder not only extracts language information (such as semantic, syntactic, and phonetic features), but also extracts the speaker's identity information (such as tone, speaking rate, etc.). The combination of these features forms multi-dimensional speech features. The encoder converts the input data of each time frame into feature vectors, which encapsulate multi-dimensional information. Specifically, the vector dimensions may include semantic information, syntactic structure, pronunciation rules, and the speaker's unique voice characteristics (such as timbre, pitch, etc.). The feature vectors of multiple time frames are combined together to form the multi-dimensional speech features of the entire speech sample.

[0079] After feature extraction, all frame-level features are combined into complete multi-dimensional speech features, representing the first speech features of the target speaker. These features need to be time-aligned to ensure that they have a consistent time sequence with subsequent speech features (such as second speech features) to facilitate subsequent operations. First, all extracted frame features are serialized to generate a complete speech feature vector for the target speaker. Subsequently, each feature frame is adjusted through a time alignment algorithm (such as dynamic time warping, DTW) to keep it consistent with the subsequent input features to avoid the impact of time differences on speech synthesis.

[0080] This embodiment uses a pre-trained self-supervised learning encoder to extract features from reference speech samples, which can efficiently extract the multi-dimensional speech features of the target speaker, not only improving the naturalness of speech generation, but also reducing the dependence on labeled data. It can ensure that the generated speech is highly consistent with the voice style of the target speaker, and is suitable for a variety of scenarios where fast speech generation is required.

[0081] In one embodiment, the above S30 includes:

[0082] S301, segmenting and annotating the text to be converted to obtain text segments and phonetic representations of the text segments, wherein the phonetic representations include phoneme information, intonation information, and prosody information;

[0083] S302, inputting the phonetic representation into a pre-trained text-to-speech feature model, and analyzing the speech duration of each text segment based on the phonetic representation;

[0084] S303, generating a speech feature sequence based on the phonetic representation of the text segment and the speech duration, wherein the speech feature sequence includes pronunciation information of the text;

[0085] S304, adjusting the length of the speech feature sequence so that the speech feature sequence matches the first speech feature of the target speaker;

[0086] S305: Use the adjusted speech feature sequence as the second speech feature.

[0087] In this embodiment, the text to be converted is segmented, and each text segment is assigned a phonetic representation, including information such as phonemes, intonation, and prosody. This process ensures that the text can be accurately converted into speech features. First, a natural language processing tool is used to segment the text and decompose the sentences into words or phrases. Then, a phonetic analysis tool (such as a phoneme tagging system) is used to annotate each text segment with phonetic information, including phonemes (for pronunciation), intonation (for expressing emotions), and prosody (to control sentence fluency).

[0088] The segmented and annotated phonetic representation is fed into a pre-trained text-to-speech feature model, and the speech duration is estimated based on the phonetic features of each text segment. The text-to-speech feature model (such as GlowTTS or GradTTS) generates speech features for each segment based on the input phonetic representation. In this process, the model analyzes the pronunciation duration of each segment to ensure that the generated speech sequence is natural and fluent.

[0089] Based on the phonetic representation and duration of the text segment, a speech feature sequence containing pronunciation information is generated to indicate how the text is presented through speech. The pronunciation information of the text segment is converted into a sequence of time frames, and each frame usually contains information such as phonemes, intonation, and prosody. These features are arranged according to the pronunciation time of the segment and a complete speech feature sequence is generated.

[0090] In order to ensure that the second speech feature can be integrated with the first speech feature of the target speaker, the generated speech feature sequence needs to be adjusted in time length to ensure that the two can correspond. Use a time alignment algorithm (such as dynamic time warping, DTW) or a simple interpolation method to adjust the speech feature sequence. Through this process, it is ensured that the speech feature sequence has the same time frame length as the speech feature of the target speaker, thereby ensuring the correctness of subsequent processing.

[0091] After adjustment, the generated speech feature sequence is the second speech feature, which represents the pronunciation information corresponding to the text to be converted. The adjusted speech feature sequence is stored as the second speech feature, ready to be fused with the first speech feature of the target speaker. This feature vector represents the pronunciation style of the text to be converted, and already has a time frame structure that matches the characteristics of the target speaker.

[0092] This embodiment can efficiently generate voice features corresponding to the text content to be converted by converting text into voice features, thereby ensuring the naturalness and accuracy of the generated voice.

[0093] In one embodiment, the above S40 includes:

[0094] S401, performing frame-level processing on the second speech feature and the first speech feature to generate a plurality of second speech feature frames and a plurality of first speech feature frames;

[0095] S402, using a similarity analysis module to analyze the similarity between the second speech feature frame and the first speech feature frame frame by frame to generate a similarity matrix;

[0096] S403, for each second speech feature frame, selecting a preset number of first speech feature frames with top similarity rankings from the similarity matrix;

[0097] S404: Perform weighted averaging processing on the selected first speech feature frame to obtain target speaker features.

[0098] In this embodiment, in order to perform feature comparison, the system needs to decompose the second speech feature and the first speech feature into multiple time frames (feature frames) respectively. Each frame represents the specific information of the speech feature within a certain time period, such as pitch, speaking speed, rhythm, etc. Frame-level processing divides the entire speech feature sequence into multiple continuous small segments. The system performs frame-level segmentation on the input second speech feature and the first speech feature to ensure that each feature frame can independently participate in the similarity analysis. Each time frame represents a speech segment of a fixed length and contains the pronunciation features within the time period. The segmented frame features will be used for subsequent frame-by-frame similarity analysis.

[0099] The system compares each second speech feature frame with all first speech feature frames frame by frame, calculates the similarity between them, and records the results in a similarity matrix. Each element of the similarity matrix represents the similarity between a second speech feature frame and a first speech feature frame. The similarity analysis module uses an algorithm (such as cosine similarity, Euclidean distance) to compare two feature frames frame by frame. The system generates a similarity matrix, the rows of which represent the second speech feature frames, the columns represent the first speech feature frames, and each element of the matrix represents the similarity between the two feature frames.

[0100] The system will select several first speech feature frames with the highest similarity ranking for each second speech feature frame from the similarity matrix. These frames represent the speech features of the target speaker that are closest to the second speech feature frame. The system will search for the most similar first speech feature frame for each second speech feature frame, sort it by similarity, and select several frames with the highest ranking. The number of frames selected is a preset parameter. Usually, the system will select several first speech feature frames with the highest similarity for subsequent processing.

[0101] The first speech feature frame selected will be weighted averaged to generate the target speaker's features. The purpose of weighting is to highlight closer features through similarity weights to ensure that the generated target speaker features can accurately reflect the speech style. The system assigns weights based on the similarity of the selected frames. The higher the weight, the greater the impact on the final result. The final target speaker features are generated by weighted averaging these frames. This feature vector will represent the target speaker's speech style.

[0102] This embodiment analyzes the similarity between the second voice feature and the first voice feature frame by frame, and performs weighted average processing based on the similarity, so that the system can accurately determine the voice features of the target speaker. This improves the accuracy and naturalness of speech generation, especially in scenes with unseen speakers, and can ensure that the generated speech is highly matched with the style of the target speaker.

[0103] In one embodiment, the above S50 includes:

[0104] S501, inputting the second speech feature and the target speaker feature into a feature fusion module;

[0105] S502, performing time alignment processing on the second speech feature and the target speaker feature to make the time length of the second speech feature consistent with that of the target speaker feature;

[0106] S503, assigning weights to the second speech feature and the target speaker feature according to a preset fusion strategy;

[0107] S504: Based on the weight, fuse the second speech feature and the target speaker feature to generate a fused speech feature.

[0108] In this embodiment, in order to fuse the two speech features together, the second speech feature (from the text to be converted) and the target speaker feature need to be input into a specific feature fusion module. The task of the feature fusion module is to receive and process these two features and fuse them into a new speech feature. The second speech feature (the speech feature generated by the text) and the target speaker feature (representing the sound feature of the target speaker) are simultaneously input into the fusion module, which receives these two input features and fuses them according to the subsequent processing strategy. The feature fusion module can use a variety of fusion techniques, such as linear interpolation, weighted average, etc.

[0109] Time alignment means ensuring that the time lengths of the two feature sequences are the same to facilitate fusion. During speech generation, the second speech feature and the target speaker feature may have different lengths, so alignment is required to ensure that the two can correspond frame by frame. Use algorithms such as dynamic time warping (DTW) to time align the second speech feature and the target speaker feature. The system compares the time frames of the two feature sequences to ensure that the speech features of the two can correspond at the same time point.

[0110] In order to control which set of features influences the final result during the feature fusion process, weights need to be assigned to each feature sequence. The weights determine the proportion of the features in the final fused features. The fusion strategy can be set according to the needs of the speech style. The system assigns weights to the second speech features and the target speaker features according to the preset strategy. The weights may be set based on similarity, style requirements, or user-specified fusion strategies. Higher weights are assigned to features that need to be highlighted in the speech style.

[0111] The two feature sequences are synthesized into a new speech feature by weighted fusion. The fused feature contains both the speech information of the text to be converted and the speech style of the target speaker. The system fuses the two features on each time frame according to the assigned weights. The fusion method can be linear weighted average or other complex fusion techniques, and finally a fused speech feature sequence is generated, which can express the content of the text to be converted and has the speech characteristics of the target speaker.

[0112] In this embodiment, by time-aligning and weighted fusion of the second voice feature and the target speaker feature, the system can generate high-quality voice features that both retain the text content and match the voice style of the target speaker, thereby improving the flexibility of voice generation, especially in personalized voice services.

[0113] In one embodiment, the above S60 includes:

[0114] S601, inputting the fused speech features into a pre-trained vocoder;

[0115] S602, performing spectrum decoding on the fused speech feature by the vocoder to generate a speech spectrum;

[0116] S603, performing phase recovery processing on the speech spectrum to obtain a time domain signal;

[0117] S604, performing audio post-processing on the time domain signal, wherein the audio post-processing includes noise reduction processing, dynamic range compression, gain adjustment, equalization processing, dereverberation processing and / or high frequency enhancement processing to improve sound quality;

[0118] S605: Output the processed time domain signal as the speech signal of the target speaker.

[0119] In this embodiment, a pre-trained vocoder (such as HiFi-GAN or WaveGlow) is used to convert speech features into actual speech waveforms. The fused speech features contain the speech information of the text to be converted and the speech style of the target speaker. By inputting these features into the vocoder, the system can generate an audio signal that conforms to the characteristics of the target speaker. The system inputs the fused speech features into a pre-trained vocoder model. The vocoder has been trained on large-scale speech data and is capable of processing and generating high-quality speech output. The vocoder model decodes the input features and converts them into a spectral representation.

[0120] The first step of the vocoder is to decode the fused speech features into a spectrum. The spectrum is a visual representation of the frequency and time of speech, and contains the frequency component information of the audio. The vocoder maps the input speech features into a speech spectrum. The generated spectrum represents the frequency distribution of the audio at each time point. This spectrum contains the basic components of speech, and the system generates the basic form of speech through the spectrum.

[0121] The speech spectrum only contains the amplitude information of the audio signal and lacks phase information, so it needs to be converted into a complete audio waveform through phase recovery. Phase recovery can be performed by the Griffin-Lim algorithm, etc. The phase recovery process estimates the phase information from the speech spectrum and combines it with the spectrum to generate a complete time domain signal. The time domain signal is the final speech waveform. The system ensures that the audio waveform conforms to the actual speech performance through this recovery process.

[0122] After the time domain signal is generated, audio post-processing is required to improve its quality. Audio post-processing usually includes noise reduction, dynamic range compression, gain adjustment, equalization, dereverberation and high-frequency enhancement to improve audio quality. The system performs various post-processing operations on the generated time domain signal, such as noise reduction and equalization, to ensure that the generated speech signal is clear and high-quality. Different post-processing steps can be adjusted according to the application scenario of the audio.

[0123] After post-processing, the final time domain signal becomes a high-quality speech signal and is output as the speech of the target speaker. The speech signal can accurately reflect the content of the text to be converted and the speech style of the target speaker. The processed time domain signal will be output through the system for users to play or use for other speech processing tasks. The generated speech signal is not only clear and natural in sound quality, but also conforms to the characteristics of the target speaker in speech style.

[0124] This embodiment decodes the fused speech features through a pre-trained vocoder, which can efficiently generate a natural speech signal with clear sound quality and style matching the target speaker. Audio post-processing further improves the overall quality of the speech, making it suitable for a variety of application scenarios.

[0125] In one embodiment, the above S10 includes:

[0126] S101, receiving a text to be converted, where the text to be converted is a natural language sentence or paragraph;

[0127] S102, performing a preprocessing operation on the text to be converted, wherein the preprocessing operation includes removing invalid characters and formatting operations;

[0128] S103, obtaining a reference speech sample of the target speaker;

[0129] S104, performing audio processing on the reference speech sample, wherein the audio processing includes adjusting a sampling rate and performing denoising processing.

[0130] In this embodiment, the system receives text input by the user, which is usually a natural language sentence or paragraph. This is the starting step of speech generation, and the text is the basic content that needs to be converted into speech. The system receives text input through a user interface, and the text may be multiple sentences, a paragraph, or a complete conversation. The input text will be passed to subsequent modules for processing. The text can be obtained by keyboard input, automatic generation, etc.

[0131] The preprocessing step ensures that the text format is standardized for subsequent processing. Preprocessing includes removing invalid characters (such as extra spaces, punctuation marks) and formatting the text in a consistent manner. The system scans the input text, automatically removes invalid characters (such as line breaks, extra spaces, etc.), and formats the text according to predefined rules. For example, different styles of punctuation marks are uniformly replaced with a standard form. Preprocessed text is easier to be processed by the subsequent speech generation module.

[0132] The reference speech sample is used to extract the target speaker's voice features and ensure that the generated speech is consistent with the target speaker's voice style. The speech sample can be uploaded by the user or selected from a preset library. The system obtains the target speaker's speech sample based on the user's selection or automatically. The speech sample is a known audio file that contains the target speaker's pronunciation features, which will be used to imitate the target speaker's voice in the subsequent generation process.

[0133] The acquired speech samples need to be processed to ensure that the sample quality is suitable for the subsequent feature extraction step. Audio processing includes adjusting the sampling rate (making the sampling rates of different audios consistent) and denoising (removing environmental noise from the audio). The system processes the input speech samples, adjusts the sampling rate to ensure the consistency of all audio data, and performs noise removal operations to ensure that the speech samples are clean and clear. Common audio processing tools such as Librosa and FFmpeg can be used for such operations.

[0134] By receiving and processing the text to be converted and the reference speech sample, this embodiment can quickly and accurately obtain the speech features of the text and the target speaker, and provide high-quality input for subsequent speech generation. The standardized processing of the text and the audio processing ensure the naturalness and consistency of the generated speech, which is suitable for a variety of speech generation scenarios.

[0135] In one embodiment, a speech generation device is provided, and the speech generation device corresponds one-to-one to the speech generation method in the above embodiment. Figure 3 , Figure 3This is a functional module diagram of a preferred embodiment of the speech generation device of the present invention. Data acquisition module 10, self-supervised learning feature extraction module 20, text-to-speech feature conversion module 30, feature similarity analysis module 40, feature fusion module 50 and vocoder decoding module 60. The functional modules are described in detail as follows:

[0136] A data acquisition module 10 is used to acquire the text to be converted and the reference speech sample;

[0137] A self-supervised learning feature extraction module 20, configured to extract features from the reference speech sample using a pre-trained self-supervised learning encoder to obtain a first speech feature of a target speaker;

[0138] A text-to-speech feature conversion module 30, configured to convert the text to be converted into a second speech feature using a text-to-speech feature model;

[0139] a feature similarity analysis module 40, configured to analyze the similarity between the second speech feature and the first speech feature, and determine a target speaker feature from the first speech feature based on the similarity;

[0140] A feature fusion module 50, configured to fuse the second speech feature with the target speaker feature to generate a fused speech feature;

[0141] The vocoder decoding module 60 is used to decode the fused speech features using a pre-trained vocoder to generate a speech signal of the target speaker based on the text to be converted.

[0142] In one embodiment, the self-supervised learning feature extraction module 20 is specifically used to:

[0143] Segmenting the reference speech sample into a plurality of audio segments in time series, and performing frame-level processing on each audio segment to obtain a plurality of frame data;

[0144] Input multiple frames of data into the pre-trained self-supervised learning encoder;

[0145] Extracting multi-dimensional speech features including language information and speaker identity information from a plurality of frame data using the self-supervised learning encoder, wherein the language information includes semantics, syntactic structure and / or pronunciation;

[0146] The multi-dimensional speech feature is used as a first speech feature of a target speaker, and a time alignment process is performed on the first speech feature.

[0147] In one embodiment, the text-to-speech feature conversion module 30 is specifically configured to:

[0148] Segmenting and annotating the text to be converted to obtain text segments and phonetic representations of the text segments, wherein the phonetic representations include phoneme information, intonation information, and prosody information;

[0149] Inputting the phonetic representation into a pre-trained text-to-speech feature model, and analyzing the speech duration of each text segment based on the phonetic representation;

[0150] generating a speech feature sequence based on the phonetic representation of the text segment and the speech duration, wherein the speech feature sequence contains pronunciation information of the text;

[0151] Adjusting the length of the speech feature sequence so that the speech feature sequence matches the first speech feature of the target speaker;

[0152] The adjusted speech feature sequence is used as the second speech feature.

[0153] In one embodiment, the feature similarity analysis module 40 is specifically used to:

[0154] Performing frame-level processing on the second speech feature and the first speech feature to generate a plurality of second speech feature frames and a plurality of first speech feature frames;

[0155] Using a similarity analysis module to analyze the similarity between the second speech feature frame and the first speech feature frame frame by frame to generate a similarity matrix;

[0156] For each second speech feature frame, selecting a preset number of first speech feature frames with top similarity rankings from the similarity matrix;

[0157] The selected first speech feature frame is processed by weighted averaging to obtain the target speaker feature.

[0158] In one embodiment, the feature fusion module 50 is specifically used for:

[0159] Inputting the second speech feature and the target speaker feature into a feature fusion module;

[0160] Performing time alignment processing on the second speech feature and the target speaker feature so that the time lengths of the second speech feature and the target speaker feature are consistent;

[0161] assigning weights to the second speech feature and the target speaker feature according to a preset fusion strategy;

[0162] Based on the weight, the second speech feature and the target speaker feature are fused to generate a fused speech feature.

[0163] In one embodiment, the vocoder decoding module 60 is specifically configured to:

[0164] Inputting the fused speech features into a pre-trained vocoder;

[0165] Performing spectrum decoding on the fused speech feature by the vocoder to generate a speech spectrum;

[0166] Performing phase recovery processing on the speech spectrum to obtain a time domain signal;

[0167] Performing audio post-processing on the time domain signal, wherein the audio post-processing includes noise reduction processing, dynamic range compression, gain adjustment, equalization processing, dereverberation processing and / or high-frequency enhancement processing to improve sound quality;

[0168] The processed time domain signal is output as the speech signal of the target speaker.

[0169] In one embodiment, the data acquisition module 10 is specifically used for:

[0170] Receiving a text to be converted, wherein the text to be converted is a natural language sentence or paragraph;

[0171] Preprocessing the text to be converted, wherein the preprocessing includes removing invalid characters and formatting operations;

[0172] Obtain a reference speech sample of the target speaker;

[0173] The reference speech sample is subjected to audio processing, wherein the audio processing includes adjusting a sampling rate and performing a denoising process.

[0174] In one embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as follows: Figure 4 As shown. The computer device includes a processor, a memory, a network interface and a database connected via a system bus. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile and / or volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external user terminal via a network connection. When the computer program is executed by the processor, it implements a function or step on the server side of a speech generation method.

[0175] In one embodiment, a computer device is provided. The computer device may be a user terminal, and its internal structure diagram may be as follows: Figure 5As shown. The computer device includes a processor, a memory, a network interface, a display screen and an input device connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external server through a network connection. When the computer program is executed by the processor, it realizes the functions or steps of a user side of a speech generation method

[0176] In one embodiment, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the following steps when executing the computer program:

[0177] Obtain the text to be converted and the reference speech sample;

[0178] Using a pre-trained self-supervised learning encoder to extract features from the reference speech sample to obtain a first speech feature of the target speaker;

[0179] Converting the text to be converted into a second speech feature using a text-to-speech feature model;

[0180] analyzing a similarity between the second speech feature and the first speech feature, and determining a target speaker feature from the first speech feature based on the similarity;

[0181] Fusing the second speech feature with the target speaker feature to generate a fused speech feature;

[0182] The fused speech features are decoded using a pre-trained vocoder to generate a speech signal of a target speaker based on the text to be converted.

[0183] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored, and when the computer program is executed by a processor, the following steps are implemented:

[0184] Obtain the text to be converted and the reference speech sample;

[0185] Using a pre-trained self-supervised learning encoder to extract features from the reference speech sample to obtain a first speech feature of the target speaker;

[0186] Converting the text to be converted into a second speech feature using a text-to-speech feature model;

[0187] analyzing a similarity between the second speech feature and the first speech feature, and determining a target speaker feature from the first speech feature based on the similarity;

[0188] Fusing the second speech feature with the target speaker feature to generate a fused speech feature;

[0189] The fused speech features are decoded using a pre-trained vocoder to generate a speech signal of a target speaker based on the text to be converted.

[0190] It should be noted that the above functions or steps that can be implemented by the computer-readable storage medium or computer device can refer to the relevant descriptions on the server side and the user side in the aforementioned method embodiment. To avoid repetition, they will not be described one by one here.

[0191] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0192] Those skilled in the art can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example. In actual applications, the above-mentioned functions can be distributed and completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above.

[0193] It should be noted that if software tools or components other than those of the Company appear in the embodiments of the present application, they are only used for illustration and do not represent actual use. The above-described embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit them; although the present invention has been described in detail with reference to the above-mentioned embodiments, a person of ordinary skill in the art should understand that the technical solutions described in the above-mentioned embodiments can still be modified, or some of the technical features thereof can be replaced by equivalents; and these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be included in the protection scope of the present invention.

Claims

1. A speech generation method, characterized in that: The following steps are involved: Obtain the text to be converted and the reference speech sample; Using a pre-trained self-supervised learning encoder to extract features from the reference speech sample to obtain a first speech feature of the target speaker; Converting the text to be converted into a second speech feature using a text-to-speech feature model; analyzing a similarity between the second speech feature and the first speech feature, and determining a target speaker feature from the first speech feature based on the similarity; Fusing the second speech feature with the target speaker feature to generate a fused speech feature; The fused speech features are decoded using a pre-trained vocoder to generate a speech signal of a target speaker based on the text to be converted.

2. The speech generation method according to claim 1, characterized in that: The reference speech sample is subjected to feature extraction using a pre-trained self-supervised learning encoder to obtain a first speech feature of the target speaker, including: Segmenting the reference speech sample into a plurality of audio segments in time series, and performing frame-level processing on each audio segment to obtain a plurality of frame data; Input multiple frames of data into the pre-trained self-supervised learning encoder; Extracting multi-dimensional speech features including language information and speaker identity information from a plurality of frame data using the self-supervised learning encoder, wherein the language information includes semantics, syntactic structure and / or pronunciation; The multi-dimensional speech feature is used as a first speech feature of a target speaker, and a time alignment process is performed on the first speech feature.

3. The speech generation method according to claim 1, wherein: Converting the text to be converted into a second speech feature using a text-to-speech feature model includes: Segmenting and annotating the text to be converted to obtain text segments and phonetic representations of the text segments, wherein the phonetic representations include phoneme information, intonation information, and prosody information; Inputting the phonetic representation into a pre-trained text-to-speech feature model, and analyzing the speech duration of each text segment based on the phonetic representation; generating a speech feature sequence based on the phonetic representation of the text segment and the speech duration, wherein the speech feature sequence contains pronunciation information of the text; Adjusting the length of the speech feature sequence so that the speech feature sequence matches the first speech feature of the target speaker; The adjusted speech feature sequence is used as the second speech feature.

4. The speech generation method according to claim 1, wherein: Analyzing the similarity between the second voice feature and the first voice feature, and determining the target speaker feature from the first voice feature based on the similarity, comprises: Performing frame-level processing on the second speech feature and the first speech feature to generate a plurality of second speech feature frames and a plurality of first speech feature frames; Using a similarity analysis module to analyze the similarity between the second speech feature frame and the first speech feature frame frame by frame to generate a similarity matrix; For each second speech feature frame, selecting a preset number of first speech feature frames with top similarity rankings from the similarity matrix; The selected first speech feature frame is subjected to weighted averaging processing to obtain the target speaker feature.

5. The speech generation method as claimed in claim 1, characterized in that: The second speech feature and the target speaker feature are subjected to feature fusion. Generate fused speech features, including: Inputting the second speech feature and the target speaker feature into a feature fusion module; Performing time alignment processing on the second speech feature and the target speaker feature so that the time lengths of the second speech feature and the target speaker feature are consistent; assigning weights to the second speech feature and the target speaker feature according to a preset fusion strategy; Based on the weight, the second speech feature and the target speaker feature are fused to generate a fused speech feature.

6. The speech generation method according to claim 1, wherein: Decoding the fused speech features using a pre-trained vocoder to generate a speech signal of a target speaker based on the text to be converted, comprising: Inputting the fused speech features into a pre-trained vocoder; Performing spectrum decoding on the fused speech feature by the vocoder to generate a speech spectrum; Performing phase recovery processing on the speech spectrum to obtain a time domain signal; Performing audio post-processing on the time domain signal, wherein the audio post-processing includes noise reduction processing, dynamic range compression, gain adjustment, equalization processing, dereverberation processing and / or high-frequency enhancement processing to improve sound quality; The processed time domain signal is output as the speech signal of the target speaker.

7. The speech generation method according to claim 1, characterized in that: Get the text to be converted and reference speech samples, including: Receiving a text to be converted, wherein the text to be converted is a natural language sentence or paragraph; Performing a preprocessing operation on the text to be converted, wherein the preprocessing operation includes removing invalid characters and formatting operations; Obtain a reference speech sample of the target speaker; The reference speech sample is subjected to audio processing, wherein the audio processing includes adjusting a sampling rate and performing a denoising process.

8. A speech generating device, characterized in that: The speech generating device comprises: A data acquisition module, used to acquire the text to be converted and reference speech samples; A self-supervised learning feature extraction module, used to extract features from the reference speech sample using a pre-trained self-supervised learning encoder to obtain a first speech feature of a target speaker; A text-to-speech feature conversion module, used for converting the text to be converted into a second speech feature using a text-to-speech feature model; a feature similarity analysis module, configured to analyze the similarity between the second voice feature and the first voice feature, and determine a target speaker feature from the first voice feature based on the similarity; A feature fusion module, used for fusing the second speech feature with the target speaker feature to generate a fused speech feature; The vocoder decoding module is used to decode the fused speech features using a pre-trained vocoder to generate a speech signal of a target speaker based on the text to be converted.

9. A computer device, characterized in that: The computer device includes a memory, a processor, and a speech generation program stored in the memory and executable on the processor. When the speech generation program is executed by the processor, the steps of the speech generation method according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium, characterized in that: The storage medium stores a speech generation program, which, when executed by a processor, implements the steps of the speech generation method according to any one of claims 1 to 7.

Images