Training method, device, electronic device, storage medium and program product for speech conversion model

Abstract

This application provides a training method, device, electronic device, storage medium, and program product for a speech conversion model. The method includes: performing speech feature conversion on a text sample using a speech conversion model to obtain predicted speech features; splitting the mel speech with the same semantics as the text sample to obtain a first mel speech and a second mel speech, wherein the first mel speech is used to reflect the acoustic features of the mel speech, and the second mel speech is used to reflect the semantics of the mel speech; performing feature extraction on the first mel speech to obtain a first label feature, and performing feature extraction on the second mel speech to obtain a second label feature; and training the speech conversion model based on the first label feature, the second label feature, and the predicted speech feature. Through this application, the speech feature generation performance of the speech conversion model can be effectively improved.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a training method, device, electronic device, storage medium, and program product for a speech conversion model. Background Art

[0002] Artificial Intelligence (AI) is a comprehensive discipline encompassing both hardware and software technologies. Foundational AI technologies generally include cloud computing, big data processing, and pre-trained models. Pre-trained models, also known as large models or basic models, can be widely applied to downstream tasks across various AI disciplines after fine-tuning. AI software technologies primarily encompass computer vision, speech processing, and natural language processing.

[0003] In related technologies, the training of speech conversion models is usually achieved directly through the predicted speech features and speech label features obtained by the speech conversion model. Since the feature elements of various dimensions in the speech label features have different training effects on the speech conversion model, the acoustic features and semantics in the speech label features are not prominently expressed, resulting in low quality of speech feature generation by the trained speech conversion model, resulting in poor speech feature generation performance of the speech conversion model. Summary of the Invention

[0004] The embodiments of the present application provide a training method, device, electronic device, computer-readable storage medium, and computer program product for a speech conversion model, which can effectively improve the speech feature generation performance of the speech conversion model.

[0005] The technical solution of the embodiment of the present application is implemented as follows:

[0006] The present invention provides a method for training a speech conversion model, including:

[0007] Performing speech feature conversion on the text sample using the speech conversion model to obtain predicted speech features;

[0008] Splitting the mel speech having the same semantics as the text sample to obtain a first mel speech and a second mel speech, where the first mel speech is used to reflect the acoustic features of the mel speech, and the second mel speech is used to reflect the semantics of the mel speech;

[0009] Performing feature extraction on the first mel-level speech to obtain a first label feature, and performing feature extraction on the second mel-level speech to obtain a second label feature;

[0010] The speech conversion model is trained based on the first label feature, the second label feature, and the predicted speech feature.

[0011] The present invention provides a training device for a speech conversion model, including:

[0012] A feature conversion module, configured to convert speech features of text samples using the speech conversion model to obtain predicted speech features;

[0013] A splitting module is used to split the mel speech with the same semantics as the text sample to obtain a first mel speech and a second mel speech, wherein the semantics of the mel speech are the same as those of the text sample, the first mel speech is used to reflect the acoustic features of the mel speech, and the second mel speech is used to reflect the semantics of the mel speech;

[0014] a feature extraction module, configured to perform feature extraction on the first mel-level speech to obtain a first label feature, and perform feature extraction on the second mel-level speech to obtain a second label feature;

[0015] A training module is used to train the speech conversion model based on the first label feature, the second label feature and the predicted speech feature.

[0016] An embodiment of the present application provides an electronic device, including:

[0017] a memory for storing computer-executable instructions or computer programs;

[0018] The processor is used to implement the training method of the speech conversion model provided in the embodiment of the present application when executing the computer-executable instructions or computer program stored in the memory.

[0019] An embodiment of the present application provides a computer-readable storage medium storing computer-executable instructions for causing a processor to execute and implement the training method of the speech conversion model provided in the embodiment of the present application.

[0020] The present invention provides a computer program product comprising a computer program or computer-executable instructions stored in a computer-readable storage medium. A processor of an electronic device reads the computer-executable instructions from the computer-readable storage medium and executes the instructions, causing the electronic device to perform the speech conversion model training method described in the present invention.

[0021] The embodiments of the present application have the following beneficial effects:

[0022] Through the speech conversion model, the text sample is converted into speech features to obtain predicted speech features; by splitting the Mel speech with the same semantics as the text sample, a first Mel speech and a second Mel speech are obtained, the first Mel speech is used to reflect the acoustic features of the sample object, and the second Mel speech is used to reflect the semantics; feature extraction is performed on the first Mel speech to obtain a first label feature, and feature extraction is performed on the second Mel speech to obtain a second label feature; based on the first label feature, the second label feature and the predicted speech feature, the speech conversion model is trained, so that the first label feature and the second label feature can respectively reflect the acoustic features of the sample object and the semantics of the text sample, thereby splitting the acoustic features and semantics contained in the Mel speech to refine the modeling, so that the speech conversion model can fully learn the acoustic features and semantics contained in the Mel speech, which can effectively improve the speech feature generation performance of the speech conversion model. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] Figure 1 Schematic diagram of the architecture of the training speech conversion model system provided in an embodiment of the present application;

[0024] Figure 2 Schematic diagram of the structure of an electronic device for training a speech conversion model provided in an embodiment of the present application;

[0025] Figure 3 1 is a flow chart of a method for training a speech conversion model according to an embodiment of the present application;

[0026] Figure 4 Schematic diagram of the principle of the training method of the speech conversion model provided in the embodiment of the present application;

[0027] Figure 5 This is a schematic diagram of the principle of mel speech splitting provided by an embodiment of the present application;

[0028] Figure 6 This is a schematic diagram of the principles of the encoding and decoding model provided in the embodiment of the present application;

[0029] Figure 7 This is a schematic diagram of the principle of voice conversion provided by an embodiment of the present application. DETAILED DESCRIPTION

[0030] In order to make the purpose, technical solutions and advantages of this application clearer, the application will be further described in detail below with reference to the accompanying drawings. The described embodiments should not be regarded as limiting this application. All other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of this application.

[0031] In the following description, reference is made to “some embodiments”, which describes a subset of all possible embodiments, but it will be understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0032] In the following description, the terms "first\second\third" involved are merely used to distinguish similar objects and do not represent a specific ordering of the objects. It can be understood that "first\second\third" can be interchanged with a specific order or sequence where permitted, so that the embodiments of the present application described herein can be implemented in an order other than that illustrated or described herein.

[0033] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by those skilled in the art to which this application pertains. The terms used herein are for the purpose of describing the embodiments of this application only and are not intended to limit this application.

[0034] Before further explaining the embodiments of the present application in detail, the nouns and terms involved in the embodiments of the present application are explained. The nouns and terms involved in the embodiments of the present application are subject to the following interpretations.

[0035] 1) Artificial Intelligence (AI): This is a comprehensive discipline encompassing a wide range of fields, encompassing both hardware and software technologies. Foundational AI technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing, pre-trained models, operating / interaction systems, and mechatronics. Pre-trained models, also known as large models or basic models, can be fine-tuned and widely applied to downstream tasks across various AI disciplines. AI software technologies primarily encompass computer vision, speech processing, natural language processing, and machine learning / deep learning.

[0036] 2) Machine Learning (ML): This is a multidisciplinary field that encompasses probability theory, statistics, approximation theory, convex analysis, and algorithmic complexity theory. It specifically studies how computers can simulate or implement human learning behaviors to acquire new knowledge or skills and reorganize existing knowledge structures to continuously improve their performance. Machine learning is at the core of artificial intelligence and the fundamental way to make computers intelligent. Its applications span all areas of AI. Machine learning and deep learning typically include techniques such as artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning, and model-based learning. Pretrained models are the latest development in deep learning, integrating these techniques.

[0037] 3) Mel-frequency speech: Also known as Mel-Frequency Cepstral Coefficients (MFCCs), they are a set of feature vectors commonly used in speech recognition and speech synthesis. Mel-frequency cepstral coefficients reflect the auditory characteristics of the human auditory system and are a method for encoding speech signals. Mel-frequency is a linear scale based on the human ear's perception of pitch rather than actual physical frequency. The conversion between Mel-frequency and actual frequency is nonlinear, which better simulates the human ear's ability to perceive different pitch frequencies. On the Mel-frequency scale, lower frequencies correspond to areas where the human ear is more sensitive, while higher frequencies become increasingly sparse as the frequency increases.

[0038] 4) Speech: It is a natural form of human communication. It involves the coordinated action of the human vocal organs (such as the vocal cords, tongue, and lips) and the respiratory system, generating sound waves through air vibrations, which in turn convey information and emotion. Speech can be transmitted through various media, such as air and water. In computer science and engineering, speech processing refers to the analysis and processing of speech signals, aiming to extract useful information, identify linguistic content, generate speech, or improve speech quality. Speech processing technologies include speech recognition (SR), speech synthesis (SS), speech enhancement (Speech Enhancement), and speech coding. Speech recognition converts speech signals into corresponding text or commands, and involves multiple disciplines such as signal processing, pattern recognition, and machine learning. Speech synthesis, on the other hand, converts text into speech signals. It is widely used in voice assistants, automated telephone systems, and other fields. Speech signal processing, a branch of phonetics, studies how to analyze and synthesize speech using mathematical models and computational techniques. In the field of artificial intelligence, speech processing technology is a key component for achieving natural language processing and human-computer interaction.

[0039] During the implementation of the embodiments of this application, the applicant discovered that the related technology has the following problems:

[0040] In related technologies, the training of speech conversion models is usually achieved directly through the predicted speech features and speech label features obtained by the speech conversion model. Since the feature elements of various dimensions in the speech label features have different training effects on the speech conversion model, the acoustic features and semantics in the speech label features are not prominently expressed, resulting in low quality of speech feature generation by the trained speech conversion model, resulting in poor speech feature generation performance of the speech conversion model.

[0041] The embodiments of the present application provide a method, device, electronic device, computer-readable storage medium, and computer program product for training a speech conversion model, which can effectively improve the speech feature generation performance of the speech conversion model. The following describes an exemplary application of the training system for the speech conversion model provided in the embodiments of the present application.

[0042] See also Figure 1 , Figure 1 1 is a schematic diagram of the architecture of the training speech conversion model system 100 provided in an embodiment of the present application. The terminal (terminal 400 is shown as an example) is connected to the server 200 via the network 300. The network 300 can be a wide area network or a local area network, or a combination of the two.

[0043] The terminal 400 is used for the user to use the client 410 to display text samples on a graphical interface 410 - 1 (graphic interface 410 - 1 is shown as an example). The terminal 400 and the server 200 are connected to each other via a wired or wireless network.

[0044] In some embodiments, the server 200 can be an independent physical server, or a server cluster or business system composed of multiple physical servers. It can also be a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms. The terminal 400 can be a smart phone, tablet computer, laptop computer, desktop computer, smart speaker, smart TV, smart watch, car terminal, etc., but is not limited to this. The electronic device provided in the embodiment of the present application can be implemented as a terminal or as a server. The terminal and the server can be directly or indirectly connected via wired or wireless communication, which is not limited in the embodiment of the present application.

[0045] In some embodiments, the server 200 performs speech feature conversion on the text sample through a speech conversion model to obtain predicted speech features, splits the mel speech to obtain a first mel speech and a second mel speech, extracts features from the first mel speech to obtain a first label feature, extracts features from the second mel speech to obtain a second label feature, trains the speech conversion model based on the first label feature, the second label feature and the predicted speech feature, and sends the trained speech conversion model to the terminal 400.

[0046] In other embodiments, the terminal 400 performs speech feature conversion on the text sample through a speech conversion model to obtain predicted speech features, splits the mel speech to obtain a first mel speech and a second mel speech, extracts features from the first mel speech to obtain a first label feature, extracts features from the second mel speech to obtain a second label feature, trains the speech conversion model based on the first label feature, the second label feature and the predicted speech feature, and sends the trained speech conversion model to the server 200.

[0047] In other embodiments, the embodiments of the present application can be implemented with the help of cloud technology. Cloud technology refers to a hosting technology that unifies a series of resources such as hardware, software, and network within a wide area network or local area network to realize data calculation, storage, processing, and sharing.

[0048] Cloud technology is a general term for network, information, integration, management platform, and application technologies used in the cloud computing business model. It can form a resource pool that can be used flexibly and conveniently on demand. Cloud computing technology will become a key support. The backend services of technical network systems require a large amount of computing and storage resources.

[0049] See also Figure 2 , Figure 2 is a structural diagram of an electronic device 500 for training a speech conversion model provided in an embodiment of the present application, wherein: Figure 2 The electronic device 500 shown may be Figure 1 The server 200 or the terminal 400 in Figure 2 The electronic device 500 shown includes: at least one processor 430, a memory 450, and at least one network interface 420. The various components in the electronic device 500 are coupled together via a bus system 440. It is understood that the bus system 440 is used to achieve connection and communication between these components. In addition to including a data bus, the bus system 440 also includes a power bus, a control bus, and a status signal bus. However, for the sake of clarity, the bus system 440 is not described in detail. Figure 2 Various buses are labeled as bus system 440 .

[0050] The processor 430 can be an integrated circuit chip with signal processing capabilities, such as a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc., where the general-purpose processor can be a microprocessor or any conventional processor, etc.

[0051] The memory 450 may be removable, non-removable, or a combination thereof. Exemplary hardware devices include solid-state memory, hard drives, optical drives, etc. The memory 450 may optionally include one or more storage devices that are physically remote from the processor 430.

[0052] Memory 450 includes volatile memory or nonvolatile memory, or may include both volatile and nonvolatile memory. Nonvolatile memory may be read-only memory (ROM), and volatile memory may be random access memory (RAM). The memory 450 described in the embodiments of the present application is intended to include any suitable type of memory.

[0053] In some embodiments, the memory 450 can store data to support various operations, examples of which include programs, modules, and data structures, or a subset or superset thereof, as exemplified below.

[0054] Operating system 451, including system programs for processing various basic system services and performing hardware-related tasks, such as the framework layer, core library layer, and driver layer, which are used to implement various basic services and process hardware-based tasks;

[0055] The network communication module 452 is used to reach other electronic devices via one or more (wired or wireless) network interfaces 420. Exemplary network interfaces 420 include Bluetooth, Wireless Fidelity (WiFi), and Universal Serial Bus (USB).

[0056] In some embodiments, the training device for the speech conversion model provided in the embodiments of the present application can be implemented in software. Figure 2 A training device 455 for a speech conversion model stored in memory 450 is shown. This device can be software in the form of a program or plug-in, and includes the following software modules: a feature conversion module 4551, a segmentation module 4552, a feature extraction module 4553, and a training module 4554. These modules are logical and can be arbitrarily combined or further separated based on the functions they implement. The functions of each module will be described below.

[0057] In other embodiments, the training device for the speech conversion model provided in the embodiments of the present application may be implemented in hardware. As an example, the training device for the speech conversion model provided in the embodiments of the present application may be a processor in the form of a hardware decoding processor, which is programmed to execute the training method for the speech conversion model provided in the embodiments of the present application. For example, the processor in the form of a hardware decoding processor may be one or more application-specific integrated circuits (ASICs), DSPs, programmable logic devices (PLDs), complex programmable logic devices (CPLDs), field-programmable gate arrays (FPGAs), or other electronic components.

[0058] In some embodiments, a terminal or server can implement the speech conversion model training method provided in the embodiments of the present application by running a computer program or computer-executable instructions. For example, the computer program can be a native program (e.g., a dedicated training program) or a software module in the operating system, such as a training module that can be embedded in any program (e.g., an instant messaging client, a photo album program, an electronic map client, or a navigation client); for example, it can be a native application (APP), that is, a program that needs to be installed in the operating system to run. In short, the above-mentioned computer program can be any form of application, module, or plug-in.

[0059] The training method of the speech conversion model provided in the embodiment of the present application will be explained in combination with the exemplary application and implementation of the server or terminal provided in the embodiment of the present application.

[0060] See also Figure 3 , Figure 3 This is a flow chart of the training method of the speech conversion model provided in the embodiment of the present application, which will be combined with Figure 3 Steps 101 to 105 are shown for illustration. The training method of the speech conversion model provided in the embodiment of the present application can be implemented by the server or the terminal alone, or by the server and the terminal in collaboration. The following will be described using the server alone as an example.

[0061] In step 101, the speech feature conversion is performed on the text sample using the speech conversion model to obtain predicted speech features.

[0062] In some embodiments, the speech conversion model may be a natural language processing model such as a large language model, which converts speech features of text samples to obtain predicted speech features.

[0063] In some embodiments, natural language processing (NLP) is a key area in computer science and artificial intelligence. It studies various theories and methods that enable effective communication between humans and computers using natural language. Natural language processing (NLP) is a discipline that integrates linguistics, computer science, and mathematics. Therefore, research in this field involves natural language, the language we use in everyday life, and is closely related to the study of linguistics, yet also has important differences. Natural language processing is not simply the study of natural language in general, but rather the development of computer systems, particularly software systems, that can effectively implement natural language communication. As such, it is a part of computer science, with applications primarily in machine translation, automatic summarization, opinion extraction, text classification, question answering, text semantic comparison, and speech recognition.

[0064] In some embodiments, large language models are speech conversion models designed to understand and generate human language. They are trained on large amounts of text data and can perform a wide range of tasks, including text summarization, translation, sentiment analysis, and more. Large language models are characterized by their massive scale, containing billions of parameters that help them learn complex patterns in language data. They are typically based on deep learning architectures. Large language models, also known as large language models or large models, refer to deep learning models trained using large amounts of text data. They contain billions or even more parameters and can be used to generate and understand the meaning of natural language text. Through training, the model can learn the statistical laws and semantic associations of language to build a massive language knowledge base, thereby simulating human language understanding and generation capabilities.

[0065] For example, see Figure 4 , Figure 4 This is a schematic diagram of the principle of the training method of the speech conversion model provided in the embodiment of the present application. Through the speech conversion model, the speech feature conversion is performed on the text sample to obtain the predicted speech feature.

[0066] For example, in a voice assistant application scenario, a user queries the voice assistant for weather information. The voice assistant uses a speech conversion model to convert text into predicted speech features to match the user's voice. It obtains a mel-level speech sample with the same semantic meaning as the text "What's the weather like in Beijing today?" and splits it into two parts: The first mel-level speech captures the acoustic characteristics of the voice, such as intonation and volume. The second mel-level speech specifically reflects the semantic content of the speech, that is, the meaning expressed. Acoustic features (first label features) are extracted from the first mel-level speech. Semantic features (second label features) are extracted from the second mel-level speech. Combining the first and second label features with the predicted speech features, the voice assistant continuously trains its speech conversion model to provide more natural and personalized voice feedback.

[0067] For example, in a real-time speech translation application scenario, a real-time speech translation system was used to instantly translate a speaker's speech at an international conference. The speaker delivered the speech in Chinese. The real-time speech translation system converted the Chinese text into predicted speech features while simultaneously acquiring the speaker's voice features in real time. Mel-level speech samples with the same semantic meaning were split into acoustic and semantic features. These features were then extracted and used to understand and generate a translation into the target language. Continuous training improved the accuracy and synchronization of the translation, ensuring that the audience could hear the accurate translation in real time.

[0068] As an example, in the application scenario of automatic subtitle generation, in video content creation, the automatic subtitle generation system provides real-time subtitles for the video. When a character in the video begins speaking, the system converts the speech content into predicted speech features, matches the speaker's voice, searches for Mel-level speech samples that are semantically consistent with the speech content in the video, and splits them into acoustic and semantic features. Features are extracted from the split Mel-level speech to generate subtitles. Through training, the system improves the ability to recognize the voices of various speakers, generating more accurate and synchronized subtitles.

[0069] In step 102, the mel speech with the same semantics as the text sample is split to obtain a first mel speech and a second mel speech.

[0070] In some embodiments, the semantics of the mel speech is the same as the semantics of the text sample, the first mel speech is used to reflect the acoustic features of the mel speech, and the second mel speech is used to reflect the semantics of the mel speech.

[0071] In some embodiments, the above-mentioned Mel speech may refer to Mel-frequency cepstral coefficients, which are a set of feature vectors commonly used in speech recognition and speech synthesis. Mel-frequency cepstral coefficients reflect the auditory characteristics of the human auditory system and are a method of encoding speech signals. Mel frequency is a linear scale based on the human ear's perception of pitch rather than the actual physical frequency. The conversion relationship between Mel frequency and actual frequency is nonlinear, and this conversion can better simulate the human ear's ability to perceive different pitch frequencies. In the Mel frequency scale, lower frequencies correspond to areas where the human ear is more sensitive, while higher frequencies gradually become sparser as the frequency increases.

[0072] For example, see Figure 5 , Figure 5 This is a schematic diagram of the principle of mel speech splitting provided in an embodiment of the present application. The mel speech is split to obtain a first mel speech and a second mel speech.

[0073] In some embodiments, the above-mentioned splitting of the mel-speech having the same semantics as the text sample to obtain the first mel-speech and the second mel-speech can be achieved as follows: according to the first frequency range corresponding to the acoustic feature, the sub-speech in the mel-speech that is within the first frequency range is determined as the first mel-speech; according to the second frequency range corresponding to the semantics, the sub-speech in the mel-speech that is within the second frequency range is determined as the second mel-speech.

[0074] As an example, consider a recorded speech sample containing someone saying the phrase "I love you." Assume the Mel-Frequency Cepstral Coefficients (MFCCs) for this speech have already been extracted. Now, the goal is to segment this speech sample based on specific frequency ranges. Acoustic Feature Extraction: First, extract the MFCCs for the speech sample, obtaining a coefficient matrix containing multiple frames. Each frame contains multiple MFCC coefficients, each associated with a specific frequency range. Defining Frequency Ranges: First Frequency Range: Based on acoustic characteristics, define the first frequency range as corresponding to a specific MFCC coefficient. For example, the fourth MFCC coefficient may be used, which is often associated with the resonant properties of the speech. Second Frequency Range: Based on semantics, define the second frequency range as corresponding to another MFCC coefficient. For example, the eighth MFCC coefficient may be selected, which may be associated with the rhythm or intonation of the speech. Mel-Speech Segmentation: First Mel-Speech: All sub-speech components within the Mel-Speech Speech within the first frequency range are identified as the first Mel-Speech. Assuming the value of the fourth MFCC coefficient is between a certain threshold (for example, between -20 and 20), select all frames that meet this threshold to form a subset. This portion of the Mel-speech reflects the acoustic features associated with that frequency range. Second Mel-speech: Similarly, all sub-speech within the Mel-speech within the second frequency range are identified as the second Mel-speech. Assuming the value of the eighth MFCC coefficient is between another specific threshold (for example, between 30 and 50), all frames that meet this condition are selected to form another subset. This portion of the Mel-speech reflects the semantic features associated with that frequency range. The specific operation is as follows: Assume there are 100 frames of MFCCs, each containing 13 coefficients. Frames with a fourth MFCC coefficient value between -20 and 20 are selected. These frames are assumed to be frames 1 through 40. These frames constitute the first Mel-speech. Frames with an eighth MFCC coefficient value between 30 and 50 are selected. These frames are assumed to be frames 41 through 100. These frames constitute the second Mel-speech. In this way, the original Mel-speech is split into two subsets based on acoustic and semantic features: the first Mel-speech and the second Mel-speech. Each subset reflects different aspects of speech. This method can be used in applications such as speech signal processing, analysis, or synthesis.

[0075] For example, see Figure 5 According to the first frequency range (0-60 bins) corresponding to the acoustic feature, the sub-speech in the mel-level speech within the first frequency range is determined as the first mel-level speech; according to the second frequency range (60-80 bins) corresponding to the semantic feature, the sub-speech in the mel-level speech within the second frequency range is determined as the second mel-level speech.

[0076] In this way, by precisely defining the first frequency range corresponding to the acoustic features, sub-speech within the mel-level speech that falls within this range is effectively screened and identified as the first mel-level speech, helping to improve the accuracy and efficiency of speech recognition. Simultaneously, based on the semantically corresponding second frequency range, sub-speech within the mel-level speech that falls within this range is further identified as the second mel-level speech. This enhances the understanding and analysis of the speech meaning, enabling more accurate speech recognition and processing, and improving overall system performance and user experience.

[0077] In step 103, feature extraction is performed on the first mel speech to obtain a first label feature.

[0078] In some embodiments, the above step 103 can be implemented as follows: query a first encoding parameter adapted to the first mel speech from a plurality of preset encoding parameters, and perform feature extraction on the first mel speech through the encoding model of the first encoding parameter to obtain the first label feature.

[0079] In some embodiments, feature extraction refers to the process of extracting information that can represent the essential characteristics of the data from raw data in signal processing or machine learning.

[0080] In some embodiments, the preset encoding parameters are predefined model parameters, such as neural network weights in deep learning, which are obtained by training on a large amount of data. Each preset encoding parameter may correspond to a different model architecture or training configuration, such as a different number of layers, number of neurons, activation function, regularization parameter, etc. Querying a first encoding parameter adapted to a first mel-level speech from a plurality of preset encoding parameters, and querying a second encoding parameter adapted to a second mel-level speech from a plurality of preset encoding parameters, involves selecting or training a model that can select or generate the most appropriate encoding parameters based on minimum distortion or maximum similarity of an input mel-level speech signal. Feature extraction is performed on the first mel-level speech and the second mel-level speech respectively through their adapted encoding models.

[0081] In some embodiments, the first and second mel-level speech are subjected to feature extraction using their respective adapted coding models. For the first mel-level speech, this process includes: using the coding model corresponding to the queried first coding parameters; inputting the first mel-level speech into the model; and utilizing the model's feature extraction capabilities to obtain a feature representation of the speech, namely, a first label feature.

[0082] In some embodiments, for feature extraction of the first Mel-level speech, in deep learning and machine learning, particularly in the field of speech processing, an Acoustic Encoder is a neural network structure specifically designed to process and encode audio features. It is commonly used in tasks such as speech recognition, speaker recognition, and sentiment analysis to extract acoustic features and convert them into vectors that can effectively represent the input audio data. A typical Acoustic Encoder network structure may include the following components: Preprocessing layer: Feature extraction: Extracts Mel-frequency cepstral coefficients (MFCCs), filter bank features (such as FBANKs), or other acoustic features from the raw audio signal. Frame-level processing: Segments the audio signal into frames and performs feature extraction on each frame. Convolutional Neural Network (CNN) layer: Uses CNN to process acoustic features, capturing time-frequency characteristics and local patterns. It typically includes multiple convolutional layers, each of which may be followed by a pooling layer to reduce the size of the feature map. Activation functions such as ReLU are used to increase the nonlinearity of the network. Recurrent Neural Network (RNN) Layer: To capture the sequential nature of audio signals, RNN layers, such as LSTM (Long Short-Term Memory) or GRU (Gated Recurrent Unit), are typically added. RNNs can process variable-length inputs and are therefore well-suited for time series data such as audio. Fully Connected Layer: At the final stage of the network, fully connected layers may be added for final feature combination and mapping. These fully connected layers can be used to extract higher-level feature representations or generate the final output (such as for classification or sequence labeling tasks). Output Layer: Depending on the specific task, the output layer may be a softmax layer to generate a probability distribution or a linear layer to directly map to the desired output space. When constructing the AcousticEncoder, the network size, number of layers, and number of neurons per layer are adjusted based on task requirements and data characteristics. Regularization techniques (such as Dropout and weight decay) and optimization algorithms (such as Adam and RMSprop) are also used to improve model generalization and training efficiency.

[0083] In some embodiments, the above-mentioned encoding model belongs to a codec model. Before the above-mentioned step 102, the training of the codec model can be achieved in the following manner: through the codec model, a complete speech prediction is performed on the speech sample carrying the speech label to obtain the predicted speech of the speech sample, and the speech label includes the speech sample; based on the predicted speech and the speech label, the codec model is trained.

[0084] As an example, the above-mentioned speech sample is at least part of the speech tag. Through the codec model, a complete speech prediction is performed on the speech sample (e.g., I am going out to play tonight) carrying the speech tag (e.g., I am going out to play tonight) to obtain the predicted speech of the speech sample (e.g., I am going out to play today), and the speech tag (e.g., I am going out to play tonight) includes the speech sample (e.g., I am going out to play today).

[0085] For example, in speech recognition, the input speech sample may sometimes be incomplete, but the model still needs to be able to predict a complete speech label. The following is a specific example of how the encoder-decoder model handles this situation: Imagine a speech recognition system that aims to convert a user's partial speech input into a complete sentence. Speech Sample and Speech Label: Speech Sample: The user says "Today I..." Speech Label: The complete sentence is "I went out to play tonight." Speech Sample Preprocessing: The user's speech sample "Today I..." is input into the model. The model first preprocesses the speech to extract audio features such as MFCCs or spectrograms. The encoder-decoder model predicts: The model uses the extracted audio features to predict the next utterance. In this example, the model needs to predict the missing portion after "Today I...". Complete Speech Prediction: Suppose the model predicts "I went out to play today." This means the model successfully recognized the user's intended speech and correctly completed the sentence. Comparison with Speech Label: The model's predicted speech "I went out to play today" is compared with the actual speech label "I went out to play tonight." In this case, the prediction doesn't exactly match the speech label, but it does include the complete content of the speech sample. Training the Codec Model: The model is trained using the predicted speech "I'm going out today" and the actual speech label "I'm going out tonight." Through this training process, the model learns how to better predict complete sentences when receiving partial speech input. Training Process Details: The model calculates the difference between the predicted speech and the speech label, for example, using edit distance or sequence alignment algorithms to determine the deviation of the predicted text from the actual label. Backpropagation is used to update the model weights to reduce prediction error and improve the model's ability to handle partial speech input.

[0086] Thus, by utilizing the aforementioned codec model, when performing complete speech prediction for speech samples with speech tags, the model can effectively infer the complete speech content, even if the speech sample only contains at least a portion of the speech tag. For example, given the speech tag "I'm going out tonight," when the input speech sample is only "I'm going out today...", the model can intelligently predict and complete the remaining speech, outputting "I'm going out today." This processing has significant benefits, greatly improving the robustness and practicality of speech recognition systems, allowing the system to handle potentially incomplete or ambiguous user speech input, thereby providing a more fluid and natural interactive experience. This is particularly important for applications such as voice assistants, real-time speech translation, and automatic subtitle generation, as it ensures that in these scenarios, the system can provide accurate and complete output even with imperfect input.

[0087] In some embodiments, the speech sample includes a plurality of different sub-speech samples, and the predicted speech includes a sub-predicted speech corresponding to each of the sub-speech samples.

[0088] In some embodiments, the above-mentioned training of the codec model based on the predicted speech and the speech label can be achieved in the following manner: determining the speech similarity between each of the sub-predicted speech and the speech label, performing a preset operation on the speech similarity corresponding to each of the sub-predicted speech to obtain the loss value of the codec model; based on the loss value of the codec model, updating the model parameters of the codec model.

[0089] In some embodiments, the above-mentioned preset operation of the speech similarity corresponding to each sub-predicted speech to obtain the loss value of the codec model can be achieved by adding the speech similarity corresponding to each sub-predicted speech to obtain the loss value of the codec model.

[0090] As an example, the loss value of the above encoding and decoding model can be expressed as:

[0091] (1)

[0092] in, Used to indicate the loss value of the encoding and decoding model, It is used to indicate the speech similarity corresponding to the sub-predicted speech, and N is used to indicate the number of sub-predicted speech.

[0093] This approach, which determines the speech similarity between each sub-predicted speech and its corresponding speech label and applies a predefined calculation to these similarities to calculate the loss value for the codec model, yields significant benefits. It not only accurately quantifies the difference between the model's predictions and the actual speech labels, but also provides direct guidance for updating model parameters. Updating model parameters based on the loss value effectively optimizes model performance, making it more accurate and reliable in recognizing speech samples. This refined training strategy not only accelerates model convergence but also improves the model's generalization across diverse language environments and speaker variations, resulting in higher recognition accuracy and lower error rates in speech recognition applications.

[0094] In some embodiments, the above-mentioned complete speech prediction of the speech sample carrying the speech label through the codec model to obtain the predicted speech of the speech sample can be achieved as follows: splitting the speech sample to obtain a first speech sample and a second speech sample; performing feature extraction on the first speech sample through the first coding model to obtain a first speech feature, and performing feature extraction on the second speech sample through the second coding model to obtain a second speech feature; performing feature splicing on the first speech feature and the second speech feature to obtain a third speech feature; and performing complete speech prediction on the speech sample based on the third speech feature through the codec model to obtain the predicted speech.

[0095] As an example, consider the following example: Suppose a complete speech sample is the user saying, "I'm going out tonight." This speech sample is split into two parts: the first half, "I'm going out tonight," and the second half, "going out tonight." Feature extraction for the first speech sample: Use the first encoding model to extract features from the first speech sample "I'm going out tonight." The encoding model might analyze the speech's MFCC coefficients to generate a sequence of feature vectors, which constitute the first speech feature. Feature extraction for the second speech sample: Use the second encoding model to extract features from the second speech sample "going out tonight." Similarly, the model analyzes the speech's MFCC coefficients to generate another sequence of feature vectors, the second speech feature. Feature concatenation: Concatenate the first and second speech features to generate a longer sequence of feature vectors, which is the third speech feature. This concatenation can be accomplished through a simple concatenation operation or a more complex feature fusion strategy, such as weighted feature summation or feature transformation. Complete speech prediction: Use the resulting third speech feature as input into the encoding / decoding model. The encoder-decoder model, which can be a deep learning model such as a recurrent neural network (RNN) or a Transformer model, predicts the text of the entire sentence based on these features. Prediction results: Assume that the encoder-decoder model outputs the text: "I'm going out tonight," which is consistent with the complete sentence of the original speech sample. By splitting the speech sample, extracting features separately, concatenating the features, and making predictions, the model can more flexibly handle speech inputs of varying lengths and can better understand and predict the entire speech sample based on the extracted local features. This is particularly useful when processing long speech or complex speech structures, as it reduces model processing complexity while maintaining prediction accuracy.

[0096] For example, see Figure 6 , Figure 6 This is a schematic diagram of the principles of the coding and decoding model provided in an embodiment of the present application. Through the first coding model, feature extraction is performed on the first speech sample to obtain a first speech feature. Through the second coding model, feature extraction is performed on the second speech sample to obtain a second speech feature. Feature splicing is performed on the first speech feature and the second speech feature to obtain a third speech feature. Through the coding and decoding model, based on the third speech feature, a complete speech prediction is performed on the speech sample to obtain the predicted speech.

[0097] In this way, the method of splitting the speech sample, extracting features through specific encoding models, and then splicing the features together has achieved multiple beneficial effects. It allows the model to analyze different parts of the speech sample more carefully, capturing richer local feature information, thereby improving the accuracy of feature representation. Through splitting and feature extraction, the model can more effectively process long speech samples, reducing computational complexity and memory requirements, and improving the efficiency of training and inference. In addition, feature splicing not only integrates speech information from different parts, but also enhances the model's understanding of the overall structure of the speech, allowing the codec model to more accurately predict the complete speech content. Ultimately, this processing strategy significantly improves the performance of the speech recognition system, achieving higher recognition accuracy and lower error rates, providing users with a more precise and smooth voice interaction experience.

[0098] In step 104, feature extraction is performed on the second mel-level speech to obtain a second label feature.

[0099] In some embodiments, the above step 104 can be implemented as follows: query the second encoding parameter adapted to the second mel speech from multiple preset encoding parameters, and extract features of the second mel speech through the encoding model of the second encoding parameter to obtain the second label feature.

[0100] In some embodiments, the encoding model for the second encoding parameter may be a semantic encoder network structure. A semantic encoder generally refers to a network structure used to extract semantic information from text in natural language processing (NLP) tasks. This network structure is commonly used in deep learning models such as BERT (Bidirectional Encoder Representations from Transformers), RoBERTa (A Robustly Optimized BERT Pre-training Approach), and XLNet (XLNet: Generalized Autoregressive Pretraining for Language Modeling), all of which are variants of the Transformer architecture. The core purpose of a semantic encoder is to convert an input text sequence into a set of vectors that represent the text's semantic information. These vectors can be used in various downstream tasks such as text classification, sentiment analysis, and question answering. A typical semantic encoder network structure includes the following main components: An embedding layer: This layer maps words or characters in the text into fixed-dimensional vectors. In a Transformer model, this step is typically accomplished using an embedding matrix. Positional Encoding: Because the Transformer model itself does not have the ability to process sequence order, positional encoding is added to provide the model with positional information of elements in the sequence. Multi-Head Self-Attention: This is a core component of the Transformer model, allowing the model to simultaneously consider different parts of the sequence when generating representations. Feed-Forward Neural Networks: After the self-attention layer, two fully connected layers are applied to the output of each position to introduce nonlinear transformations. Residual Connections: Adding residual connections between the output of each sublayer and the input of the next sublayer helps the model learn more complex patterns. Layer Normalization: Applying layer normalization after residual connections helps stabilize gradients during learning. Classification Head / Output Layer: After pre-training, the corresponding output layer is added based on the specific downstream task. For example, in a text classification task, this may be a softmax layer used to generate class probabilities.The specific details of the Semantics Encoder may be adjusted based on different pre-training objectives and downstream tasks. For example, BERT adds two pre-training tasks, next sentence prediction and masked language modeling, to the Transformer to improve its semantic understanding capabilities. RoBERTa, on the other hand, removes the next sentence prediction task from BERT and instead adopts other training strategies, such as dynamic masking, to improve model performance.

[0101] For example, see Figure 4 , performing feature extraction on the second mel speech through the encoding model of the second encoding parameter to obtain the second label feature.

[0102] In some embodiments, the preset encoding parameters are predefined model parameters, such as neural network weights in deep learning, which are obtained by training on a large amount of data. Each preset encoding parameter may correspond to a different model architecture or training configuration, such as a different number of layers, number of neurons, activation function, regularization parameter, etc. Querying a first encoding parameter adapted to a first mel-level speech from a plurality of preset encoding parameters, and querying a second encoding parameter adapted to a second mel-level speech from a plurality of preset encoding parameters, involves selecting or training a model that can select or generate the most appropriate encoding parameters based on minimum distortion or maximum similarity of an input mel-level speech signal. Feature extraction is performed on the first mel-level speech and the second mel-level speech respectively through their adapted encoding models.

[0103] In some embodiments, the second mel-level speech and the second mel-level speech are each subjected to feature extraction using their respective adapted coding models. For the second mel-level speech, this process includes: using the coding model corresponding to the queried second coding parameters; inputting the second mel-level speech into the model; and utilizing the model's feature extraction capabilities to obtain a feature representation of the speech, i.e., a second label feature.

[0104] In step 105, the speech conversion model is trained based on the first label feature, the second label feature and the predicted speech feature.

[0105] In some embodiments, the above step 105 can be implemented as follows: perform feature splicing on the first label feature and the second label feature to obtain a third label feature; determine the feature similarity between the third label feature and the predicted speech feature, determine the feature similarity as a loss value, and update the model parameters of the speech conversion model based on the loss value.

[0106] For example, see Figure 4, perform feature splicing on the first label feature and the second label feature to obtain a third label feature; determine the feature similarity between the third label feature and the predicted speech feature, determine the feature similarity as a loss value, and update the model parameters of the speech conversion model based on the loss value.

[0107] For example, in speech recognition tasks, label features can be categorized into different types. For example, the first label feature can be obtained by extracting the Mel-Frequency Cepstral Coefficients (MFCCs) corresponding to the first frequency range of the first Mel-level speech. The second label feature can be obtained by extracting the Mel-Frequency Cepstral Coefficients (MFCCs) corresponding to the second frequency range of the second Mel-level speech. The third label feature can be a concatenation of the first and second Mel-level speech. The following are the specific steps for feature concatenation and model updating: Feature extraction: First, a Mel-Frequency Cepstral Coefficient (MFCC) extractor is used to extract the first label features, i.e., MFCCs, from the input speech signal. Next, the second label features are extracted using some method (such as a sequence labeling model) using the transcribed text information of phonemes or words. Finally, an acoustic model (such as one based on a deep neural network) is used to concatenate the first and second Mel-level speech features to form the third label feature. Feature concatenation: The first and second label features are concatenated to generate a combined feature vector, i.e., the third label feature. Concatenation can be a simple concatenation or a more complex feature fusion method, such as using an attention mechanism to dynamically weight different features. Feature Similarity Calculation: Next, the feature similarity between the third label features and the predicted speech features is calculated. This is typically achieved by calculating the Euclidean distance, cosine similarity, or other metrics between the features. The calculated feature similarity is used as a loss value. A larger loss value indicates a greater difference between the predicted speech features and the third label features. Model Update: The loss value is used to update the model parameters of the speech conversion model. This is typically accomplished using a gradient descent algorithm, where the gradient of the loss value with respect to the model parameters is calculated. These gradients are then used to adjust the model parameters so that the model more accurately captures the characteristics of speech in future predictions. Iterative Optimization: Over multiple iterations, the model continuously learns and improves its ability to predict speech features, thereby improving speech recognition accuracy. This process is typical of the training process of deep learning-based speech recognition systems. During this process, the model's goal is to reduce the difference between the predicted speech features and the actual third label features, that is, to minimize the loss value. In this way, the model learns how to better convert the input speech signal into corresponding text information.

[0108] In this way, the first label feature and the second label feature are spliced ​​to obtain the third label feature. The beneficial effect is that different types of label feature information can be comprehensively utilized to improve the model's understanding and representation capabilities of the speech signal. By determining the feature similarity between the third label feature and the predicted speech feature, and using the similarity as the loss value, it helps to accurately evaluate the gap between the model prediction and the actual speech feature. The model parameters of the speech conversion model are updated based on the loss value, so that the model can continuously learn and optimize, thereby improving the accuracy and robustness of speech recognition. In addition, this method also helps to improve the generalization ability of the model, so that it has better performance when dealing with different scenarios and speech signals. In short, through feature splicing and model updating, the overall performance of the speech recognition system can be effectively improved, providing users with a more intelligent and accurate voice interaction experience.

[0109] In some embodiments, after step 105 above, speech conversion can also be performed in the following manner: using a trained speech conversion model, performing speech feature conversion on the text to be converted to obtain predicted speech features of the text to be converted; performing speech conversion on the predicted speech features of the text to be converted to obtain converted speech of the text to be converted; the converted speech has the same semantics as the text to be converted, and the converted speech satisfies the acoustic features of the Mel speech.

[0110] For example, see Figure 7 , Figure 7 This is a schematic diagram of the principle of speech conversion provided by an embodiment of the present application. Through the trained speech conversion model, the speech feature conversion is performed on the text to be converted to obtain the predicted speech features of the text to be converted; the predicted speech features of the text to be converted are subjected to speech conversion to obtain the converted speech of the text to be converted; the converted speech has the same semantics as the text to be converted, and the converted speech meets the acoustic characteristics of the Mel speech.

[0111] For example, consider training a speech conversion model: First, a speech conversion model has been trained using a large amount of data. This model learns how to convert text into speech features. During this process, the model may use structures such as recurrent neural networks (RNNs) or transformers to learn the text-to-speech mapping. Conversion of speech features from the text to be converted: When a piece of text to be converted is received, it is fed into the trained speech conversion model. Based on its previously learned knowledge, the model converts the text into corresponding predicted speech features. These predicted speech features serve as an intermediate representation, representing the acoustic characteristics of the text as the model believes it should correspond to. Speech conversion: Next, these predicted speech features are used to generate the actual speech signal. This typically involves another model, such as an acoustic model, which uses these features to synthesize a speech waveform. During this process, the model takes into account the acoustic characteristics of the target sample, ensuring that the generated speech not only sounds like the original text but also meets certain acoustic properties, such as timbre, intonation, or speaking speed. Ensuring semantic identity and acoustic characteristics: Ultimately, the generated converted speech should have the same semantic content as the original text, meaning that the converted speech should convey the same meaning and information. At the same time, the converted speech must also meet the acoustic characteristics of the sample subject, meaning it should have the sound quality and style of the target speech. This process results in a speech output that is both faithful to the original text content and consistent with the style of the target speech. Such a system can be applied in a variety of scenarios, such as voice assistants, automated phone systems, and text-to-speech conversion. By continuously training and optimizing the model, the quality of the converted speech can be improved, making it more natural and fluent, while also better adapting to different acoustic characteristics requirements.

[0112] In this way, the trained speech conversion model is used to convert the speech features of the text to be converted. The beneficial effect is that the text information can be accurately mapped to the predicted speech features, thereby generating converted speech that meets specific acoustic characteristics. This processing process not only ensures that the converted speech is semantically identical to the original text and accurately conveys the information, but also meets the acoustic feature requirements of the sample object, so that the speech output is consistent with the target speech style in terms of sound quality, intonation, and speaking speed. This helps to improve the naturalness and fluency of speech synthesis and enhance the user experience. At the same time, this method also helps to expand the application areas of speech technology, such as voice assistants, automatic speech recognition, and text-to-speech conversion, providing users with more intelligent and personalized voice interaction services.

[0113] In some embodiments, the above-mentioned voice conversion of the predicted voice features of the text to be converted to obtain the converted voice of the text to be converted can be achieved as follows: decoding the predicted voice features of the text to be converted through a decoding model to obtain the Mel voice of the text to be converted; and voice conversion of the Mel voice of the text to be converted through a vocoder to obtain the converted voice of the text to be converted.

[0114] For example, see Figure 7 , through the decoding model, the predicted speech features of the text to be converted are decoded to obtain the Mel speech of the text to be converted; through the vocoder, the Mel speech of the text to be converted is voice-converted to obtain the converted speech of the text to be converted.

[0115] For example, in a speech synthesis system, the decoding model and vocoder are two key components that work together to convert text into speech. Decoding model: The decoding model (such as a neural network decoder) decodes the predicted speech features of the text to be converted. During this process, the model leverages its learned linguistic and acoustic knowledge to convert these features into a higher-level speech representation. For example, suppose we have a text sequence "Hello, how are you?" and the speech conversion model has already obtained the corresponding predicted speech features. The decoding model further processes these features to generate a corresponding mel-spectrogram, a graph of the acoustic characteristics of the speech signal. Vocoder: The vocoder is a model that converts the mel-spectrogram into the actual speech signal. It is typically a neural network-based model that converts the mel-spectrogram into a time-domain waveform. Continuing with the above example, the mel-spectrogram generated by the decoding model is input into the vocoder. The vocoder uses information in the mel-spectrogram, such as frequency and amplitude, to synthesize the corresponding speech waveform. This process is equivalent to converting acoustic features into audible speech. Generating the converted speech: Ultimately, the speech waveform generated by the vocoder is the converted speech of the text to be converted. This speech not only sounds consistent with the text content but also possesses the acoustic characteristics determined by the Mel-level speech. This process allows text content to be converted into natural-sounding speech that meets specific acoustic requirements. This technology is useful in many applications, such as voice assistants, automated phone systems, and text-to-speech conversion. By continuously training and optimizing the decoding model and vocoder, we can improve the quality of speech synthesis, making it more natural and fluent while meeting different acoustic requirements.

[0116] In this way, the predicted speech features of the text to be converted are decoded by the decoding model. The beneficial effect is that the text information can be converted into mel-level speech. This process helps preserve the acoustic characteristics of the speech and improve the quality of speech synthesis. The vocoder then performs speech conversion on the mel-level speech to generate converted speech that meets the target acoustic characteristics, making the output speech more natural and fluent in terms of sound quality, intonation, and speaking speed. This processing method not only ensures that the converted speech is semantically identical to the original text, but also meets the acoustic feature requirements of the sample object, thereby improving the overall performance of speech synthesis. At the same time, this method also helps expand the application areas of speech technology and provide users with a more intelligent and personalized voice interaction experience.

[0117] In some embodiments, the above-mentioned trained speech conversion model performs speech feature conversion on the text to be converted, and before obtaining the predicted speech features of the text to be converted, the following processing can be performed: in response to the question statement sent by the question consulting party, the answer statement is predicted for the question statement, and the answer statement is determined as the text to be converted.

[0118] For example, in an intelligent voice interaction system, when a user asks a question, the system first processes the question and predicts an answer. This process can be implemented using a trained answer prediction model. Question reception and processing: The user asks a question via voice or text input. Answer prediction: The system uses the trained answer prediction model to process the text to be converted. This model analyzes the semantics of the question and predicts a possible answer. For example, if a user asks, "What's the weather like today?" the answer prediction model might predict, "It's sunny today." Determining the text to be converted: The predicted answer sentence is determined as the text to be converted, i.e., "It's sunny today." Speech feature conversion: Next, the system uses the speech conversion model to convert the text to "It's sunny today" into speech features, generating the corresponding predicted speech features. Speech synthesis: Finally, the system uses a vocoder to convert these predicted speech features into actual speech output, answering the user's question. Through this process, the system can first predict and answer the user's question, and then output the answer in speech form, providing the user with a smooth voice interaction experience. This technology is very useful in applications such as intelligent assistants and customer service robots, significantly improving user satisfaction and interaction experience.

[0119] In some embodiments, after performing voice conversion on the predicted voice features of the text to be converted to obtain the converted voice of the text to be converted, the following processing may be performed: sending the converted voice of the text to be converted to the party asking the question.

[0120] In this way, by introducing the answer prediction step before the voice conversion model, the beneficial effect is that it can improve the intelligence and response speed of the voice interaction system. First, the system can answer users' questions instantly and improve the user experience. For example, when a user asks "What's the weather like today?", the system can quickly predict and answer "It's sunny today." Secondly, using the predicted answer sentence as the text to be converted helps improve the accuracy of subsequent speech synthesis and ensure the consistency between the speech output and the answer content. During the voice feature conversion and voice conversion process, the system will process the answer sentence based on the semantic and acoustic features to generate a high-quality voice response. Finally, sending the converted voice of the text to be converted to the user can achieve effective information transmission and meet the user's need to obtain answers. This makes the voice interaction system more intelligent and efficient, and can better meet the user's voice query needs.

[0121] In this way, the text sample is converted into speech features through the speech conversion model to obtain predicted speech features; the first mel speech and the second mel speech are obtained by splitting the mel speech, the semantics of the mel speech are the same as the semantics of the text sample, the first mel speech is used to reflect the acoustic features of the sample object, and the second mel speech is used to reflect the semantics of the mel speech; the first mel speech is feature extracted to obtain the first label feature, and the second mel speech is feature extracted to obtain the second label feature; based on the first label feature, the second label feature and the predicted speech feature, the speech conversion model is trained, so that the first label feature and the second label feature can respectively reflect the acoustic features of the sample object and the semantics of the text sample, thereby splitting the acoustic features and semantics contained in the mel speech to refine the modeling, so that the speech conversion model fully learns the acoustic features and semantics contained in the mel speech, which can effectively improve the speech feature generation performance of the speech conversion model.

[0122] Below, an exemplary application of the embodiment of the present application in an actual image editing application scenario will be described.

[0123] Large-model TTS speech synthesis The TTS model has attracted increasing attention from industry and academia. Compared with traditional speech synthesis models, such as FastSpeech and Vits, the naturalness, emotion, and rhythm of synthesized speech by the large-model TTS speech synthesis model are close to those of real people.

[0124] There are two forms of speech synthesis: end-to-end, where text is directly converted into a speech waveform; and text-to-mel, which is then converted back into a speech waveform through a vocoder. Popular large speech models such as Chattts and Tortoise-TTS all convert mel into vq code, and then use a large GPT-like NLP model to train the integration of text, vq code, and speaker embedding. However, because mel information is particularly rich in information, the vq encoder is very critical. As long as the encoder has a strong feature learning ability, the codebook performance will be very good, and the decoder will be under much less pressure.

[0125] In some embodiments, the encoding / decoding model provided in the embodiments of the present application (i.e., the encoding / decoding model described above) comprises three components: an encoder, a VQ, and a decoder. The encoder learns useful information from the target's true mel, obtaining an intermediate feature (a continuous vector feature). The VQ converts this intermediate feature into discretized features and latent intermediate features (continuous vector features). The latent intermediate features and the discretized features can be converted back to each other without the need for a separate network module. The decoder converts the latent intermediate features into mel, or predicted mel. The smaller the difference between the predicted mel and the target mel, the better the performance of the VQ network architecture. The encoder's learning ability determines the performance of subsequent modules, making it crucial in the VQ network architecture. Large-scale TTS models also utilize VQ discretized features. Mel features (i.e., the mel speech features described above) are rich in various speech information, such as acoustic and semantic information. Acoustic information primarily includes the speaker's timbre and prosodic characteristics, while semantic information primarily includes the textual meaning of the speech, specifically, the meaning of the speech. These two types of information together make up the mel feature, demonstrating its complexity and rich information content. VQ is the foundation and most critical component of current large-scale speech synthesis models, with the VQ Encoder playing a crucial role. Its effectiveness directly determines the capabilities of large-scale speech synthesis models, impacting the ability to perform sentence cloning and universal timbre. This typically manifests as unreliable sentence clones, poor sound quality, and poor prosody. Given the complexity of mel features, we separate the acoustic and semantic information in the mel features into separate models, leveraging a divide-and-conquer approach to improve the VQ Encoder's ability to learn features.

[0126] In some embodiments, see Figure 5The dimension of mel feature is [length, 80]. In the field of speech synthesis, audio is generally processed into mel feature with dimension [length, 80]. Of course, some models process mel feature into [length, 100] (in very rare cases) (Why 20-80 is semantic information and 0-20 is acoustic information? This is the most basic knowledge in the field of speech. The dimension 20-80 contains a small amount of acoustic information, but at most it is semantic information, that is, the content of the audio. The acoustic features of the dimension 0-20 are mainly rhythm and timbre information. That is, when listening to the audio of this dimension, you can tell whether it is spoken by a man or a woman, and the degree of rhythmic fluctuations. The dimension 20-80 is input into the ASR information, and its content can be accurately identified). The first 20 dimensions of information are mainly acoustic information, and the 20 to 80-dimensional information are mainly semantic information. The acoustic information [batch_size, length, 20] is passed through the acoustics encoder module to learn richer acoustic features (rhythm, timbre, etc.), and the output dimension is [batch_size, length, dim1]. The semantic information [batch_size, length, 60] is passed through the semantics encoder module to learn richer semantic features (emotion, sentence meaning), and the dimension is [batch_size, length, dim2]. Finally, the features learned by the two are concatenated, and the dimension is [batch_size, length, dim1+dim2]. This operation is a mathematical process and is input to VQ and Decoder for training and learning.

[0127] In some embodiments, the semantics encoder network structure (VQ code 2) includes an N-layer transformer encoder structure. The powerful semantic learning capabilities of transformer encoders have been widely used in speech and text fields, such as the BERT pre-trained model and FastSpeech2 model in the NLP field.

[0128] In some embodiments, the acoustics encoder network structure is mainly composed of Conv1D (one-dimensional convolution, extracting local features). The parameter kernel size value of Conv1D can focus on the information within a certain range of the current position, and enhance the acquisition of local information (that is, the associated information of adjacent frames). Looking from top to bottom, the kernel size of the first layer Conv1D is 5, which can obtain information far away from the current position, and also has the function of dimensionality change. The kernel sizes of subsequent Conv1D are different, and they all obtain information within different ranges of the current position, enhancing the ability to obtain local information. The main function of Linear is dimensionality transformation, Layer Norm normalizes the data to improve stability during training, ReLU is an activation function, and enhances the ability of the network module. The last layer ConvlD transforms the dimension into the dimension desired by the training model, and also has the ability to obtain local information. The acoustics encoder module focuses more on rhythmic features. The rhythmic expression of the current word in a sentence is often closely related to the words around it, that is, it is related to the words within a certain range (kernel_size setting). Therefore, the design of the acoustics encoder network structure is mainly based on Conv1D.

[0129] In some embodiments, the embodiments of the present application are mainly to improve the ability of the vq encoder to learn features, thereby improving the sound quality and rhythmic naturalness of the generated audio. The mel feature contains a lot of information, and it is difficult to learn it by a single model. Splitting the modeling is conducive to improving the ability to learn features, which is the principle of divide and conquer.

[0130] In some embodiments, VQ training is a self-supervised method that inputs target mel and outputs predicted mel, and the two perform loss calculation iteratively. The smaller the loss, the better the VQ performance. The input of the model only requires mel features, and the mel feature dimension [batch_size, length, 80] is split into (code splitting, because the dimension is fixed, [batch_size, length, :20] and [batch_size, length, 20:] can be separated) [batch_size, length, 20]a1 and [batch_size, length, 60]a2, a1 is input to the acoustic encoder module to learn acoustic features, and the output dimension is [batch_size, length, dim1]b1, a2 is input to the semantics encoder to learn semantic features, and the output dimension is [batch_size, length, dim2]b2, then b1 and b2 (the splicing dimension is on the third dimension) are spliced ​​together to get the final feature dimension of [batch_size, length, dim1+dim2], input the subsequent modules to participate in training and learning, output predict mel, and calculate the loss with the target mel, multiple rounds of iteration, wait for the loss to converge, and save the model.

[0131] In some embodiments, the saved model is loaded, the input is the real mel, and the vq code discretized feature is obtained (mel feature is a continuous feature, including semantic information and acoustic information. The mel feature is compressed into a discrete feature by VQ. For example, a discrete feature is a word list. The corresponding word can be found by the id in the word list. Conversely, the corresponding id can be found by the word. This id is a discrete feature, which is spliced ​​with the text and speaker embedding in a certain form (the network structure proposed in this patent is to make the extracted vq code richer in information. This is the training GPT module. This is not the focus of this patent. It is just a brief explanation of the subsequent application of this patent. Semantic information and acoustic information will be used in training VQ, but GPT training does not require it. It only needs to extract the vq code, and what is obtained after the semantic information and acoustic information are processed has been explained above, and on the VQ Encoder side. Training the GPT model, the input feature is a spliced ​​feature, such as [speaker embedding, text code, vq code], and the dimension is the second dimension as the splicing dimension), input to a large text model such as GPT to participate in training. See Figure 7 The trained model can be applied to general timbre synthesis and sentence cloning synthesis.

[0132] In some embodiments, when the algorithm is applied, the VQ model with good results can be directly applied to the training of the subsequent large speech synthesis model, and the trained model can be applied to outbound telemarketing calls and customer service collection. By improving the VQ encoder network model, the ability of the large speech synthesis model can be effectively improved, that is, the sound quality and rhythmic naturalness of the generated audio. When applied to outbound telemarketing calls, the speaker embedding should be a speaker vector feature that is friendly, has high affinity, or has excellent performance. This speaker may be in the training data set or not (sentence cloning). If the speaker takes leave or resigns (the right to use his or her own voice is required), audio can be generated. The VQ Encoder network structure proposed in this application has a voice similarity and rhythmic naturalness that is very similar to that of the speaker, achieving a fake-real effect.

[0133] This embodiment of the present application improves the feature learning capability of the VQ encoder (also known as the encoding model described above), contributing to improved performance (sound quality and prosodic information) of the large TTS model, such as sentence cloning and universal timbre audio generation. Because MEL features contain multiple information, a single network structure cannot fully learn them. By separating the acoustic and semantic information contained in them and modeling them separately, this approach achieves a divide-and-conquer approach. This improves the VQ encoder's feature learning capability, indirectly improving the sound quality and prosodic naturalness of the generated audio, achieving a realistic effect. In practical applications, a VQ model is first trained. The VQ encoder and VQ module are then used to extract discretized VQ code features. This is then combined with text and speaker embeddings to train the large TTS model for speech synthesis. In use, inputting text and speaker embeddings allows for the synthesis of speech with high similarity to the speaker embedding, enabling sentence cloning and universal timbre audio, suitable for outbound telemarketing or debt collection scenarios. First, the real-time voice stream signal from the telephone user end is received through MRCP, and then the voice stream signal is input into the ASR speech recognition model to obtain the customer's speech content. The customer's intention is obtained through the NLP model, and the content to be fed back is obtained. The feedback content is synthesized into speech through TTS and fed back to the customer through the phone for response, completing a round of interaction, and this cycle repeats.

[0134] This effectively improves the sound quality and rhythmic naturalness of voice cloning and general voice synthesis for a single sentence. Only a few seconds of a speaker's speech are needed to synthesize that speaker's voice, allowing for customized speaker voices in outbound call scenarios such as telesales and debt collection. The key lies in acquiring the VQ code, which directly impacts the performance of the large-scale speech synthesis model. The performance of the VQ code depends primarily on the VQ network structure, with the VQ encoder module's ability to learn features being a key factor. This patent proposes a VQ encoder network model based on Mel features. Because Mel features are rich in information and complex, the acoustic and semantic information in the Mel features are separated and refined for modeling, effectively improving the entire VQ encoder module's ability to acquire features. This, in turn, improves VQ performance, and ultimately the sound quality and rhythmic naturalness of the generated audio of the large-scale speech synthesis TTS model. These processes are interconnected.

[0135] It is understandable that in the embodiments of the present application, when text samples and other related data are involved, when the embodiments of the present application are applied to specific products or technologies, user permission or consent is required, and the collection, use and processing of relevant data must comply with relevant laws, regulations and standards of relevant countries and regions.

[0136] The following continues to describe the exemplary structure of the speech conversion model training device 455 provided in the embodiment of the present application implemented as a software module. In some embodiments, such as Figure 2 As shown, the software modules in the training device 455 of the speech conversion model stored in the memory 450 may include: a feature conversion module 4551, used to perform speech feature conversion on the text sample through the speech conversion model to obtain predicted speech features; a splitting module 4552, used to split the Mel speech with the same semantics as the text sample to obtain a first Mel speech and a second Mel speech, the first Mel speech is used to reflect the acoustic features of the Mel speech, and the second Mel speech is used to reflect the semantics of the Mel speech; a feature extraction module 4553, used to perform feature extraction on the first Mel speech to obtain a first label feature, and perform feature extraction on the second Mel speech to obtain a second label feature; a training module 4554, used to train the speech conversion model based on the first label feature, the second label feature and the predicted speech feature.

[0137] In some embodiments, the feature extraction module is further used to query a first encoding parameter adapted to the first mel speech from a plurality of preset encoding parameters, and perform feature extraction on the first mel speech through the encoding model of the first encoding parameter to obtain the first label feature; the feature extraction module is further used to query a second encoding parameter adapted to the second mel speech from a plurality of preset encoding parameters, and perform feature extraction on the second mel speech through the encoding model of the second encoding parameter to obtain the second label feature.

[0138] In some embodiments, the above-mentioned encoding model belongs to a codec model, and the above-mentioned training module is also used to perform complete speech prediction on a speech sample carrying a speech label through the codec model to obtain the predicted speech of the speech sample, and the speech label includes the speech sample; based on the predicted speech and the speech label, the codec model is trained.

[0139] In some embodiments, the above-mentioned speech sample includes multiple different sub-speech samples, and the predicted speech includes sub-predicted speech corresponding to each of the sub-speech samples; the above-mentioned training module is also used to determine the speech similarity between each of the sub-predicted speech and the speech label, and perform a preset operation on the speech similarity corresponding to each of the sub-predicted speech to obtain the loss value of the codec model; based on the loss value of the codec model, the model parameters of the codec model are updated.

[0140] In some embodiments, the above-mentioned training module is also used to split the speech sample to obtain a first speech sample and a second speech sample; perform feature extraction on the first speech sample through the first coding model to obtain a first speech feature, and perform feature extraction on the second speech sample through the second coding model to obtain a second speech feature; perform feature splicing on the first speech feature and the second speech feature to obtain a third speech feature; and perform complete speech prediction on the speech sample based on the third speech feature through the coding and decoding model to obtain the predicted speech.

[0141] In some embodiments, the above-mentioned training module is also used to perform feature splicing on the first label feature and the second label feature to obtain a third label feature; determine the feature similarity between the third label feature and the predicted speech feature, determine the feature similarity as a loss value, and update the model parameters of the speech conversion model based on the loss value.

[0142] In some embodiments, the training device for the above-mentioned speech conversion model further includes: a conversion module, which is used to perform speech feature conversion on the text to be converted through the trained speech conversion model to obtain the predicted speech features of the text to be converted; perform speech conversion on the predicted speech features of the text to be converted to obtain the converted speech of the text to be converted; the converted speech has the same semantics as the text to be converted, and the converted speech meets the acoustic characteristics of the Mel speech.

[0143] In some embodiments, the above-mentioned conversion module is further used to decode the predicted speech features of the text to be converted through a decoding model to obtain the Mel speech of the text to be converted; and perform speech conversion on the Mel speech of the text to be converted through a vocoder to obtain the converted speech of the text to be converted.

[0144] In some embodiments, the above-mentioned conversion module is also used to respond to the question statement sent by the question consultation party, predict the answer to the question statement, obtain the answer statement to the question statement, and determine the answer statement as the text to be converted; and send the converted voice of the text to be converted to the question consultation party.

[0145] In some embodiments, the splitting module is further configured to determine, according to the first frequency range corresponding to the acoustic feature, the sub-speech in the mel-speech that is within the first frequency range as the first mel-speech; and, according to the second frequency range corresponding to the semantic feature, to determine, according to the second frequency range corresponding to the semantic feature, the sub-speech in the mel-speech that is within the second frequency range as the second mel-speech.

[0146] The present invention provides a computer program product comprising a computer program or computer-executable instructions stored in a computer-readable storage medium. A processor of an electronic device reads the computer-executable instructions from the computer-readable storage medium and executes the instructions, causing the electronic device to perform the speech conversion model training method described in the present invention.

[0147] The embodiment of the present application provides a computer-readable storage medium storing computer-executable instructions, wherein the computer-executable instructions are stored. When the computer-executable instructions are executed by a processor, the processor will execute the training method of the speech conversion model provided in the embodiment of the present application, for example, Figure 3 The training method of the speech conversion model is shown.

[0148] In some embodiments, the computer-readable storage medium may be a memory such as FRAM, ROM, PROM, EPROM, EEPROM, flash memory, magnetic surface storage, optical disk, or CD-ROM; or various electronic devices including one or any combination of the above memories.

[0149] In some embodiments, computer-executable instructions may be in the form of a program, software, software module, script, or code, written in any form of programming language (including compiled or interpreted languages, or declarative or procedural languages), and may be deployed in any form, including as a stand-alone program or as a module, component, subroutine, or other unit suitable for use in a computing environment.

[0150] As an example, computer-executable instructions may, but need not, correspond to a file in a file system, may be stored as part of a file storing other programs or data, such as in one or more scripts in an HTML document, in a single file dedicated to the program in question, or in multiple coordinating files (e.g., files storing one or more modules, subroutines, or code portions).

[0151] By way of example, computer-executable instructions may be deployed to be executed on one electronic device, or on multiple electronic devices located at one site, or on multiple electronic devices distributed across multiple sites and interconnected by a communication network.

[0152] In summary, the embodiments of the present application have the following beneficial effects:

[0153] (1) The speech feature conversion of the text sample is performed through the speech conversion model to obtain the predicted speech feature; the first mel speech and the second mel speech are obtained by splitting the mel speech with the same semantics as the text sample, the first mel speech is used to reflect the acoustic features of the sample object, and the second mel speech is used to reflect the semantics of the mel speech; the feature extraction of the first mel speech is performed to obtain the first label feature, and the feature extraction of the second mel speech is performed to obtain the second label feature; based on the first label feature, the second label feature and the predicted speech feature, the speech conversion model is trained so that the first label feature and the second label feature can respectively reflect the acoustic features of the sample object and the semantics of the text sample, thereby splitting the acoustic features and semantics contained in the mel speech to refine the modeling, so that the speech conversion model fully learns the acoustic features and semantics contained in the mel speech, which can effectively improve the speech feature generation performance of the speech conversion model.

[0154] (2) By precisely defining the first frequency range corresponding to the acoustic features, the sub-speech within the mel-speech range is effectively screened out and identified as the first mel-speech, which helps improve the accuracy and efficiency of speech recognition. At the same time, based on the second frequency range corresponding to the semantics, the sub-speech within the mel-speech range is further identified as the second mel-speech, which can enhance the understanding and analysis of the speech meaning, thereby achieving more accurate speech recognition and processing, and improving the overall system performance and user experience.

[0155] (3) By utilizing the above-mentioned codec model, when predicting the complete speech of a speech sample with a speech label, the model can effectively infer the complete speech content even if the speech sample is only at least part of the speech label. For example, given the speech label "I went out to play tonight", when the input speech sample is only "I went out to play today...", the model can intelligently predict and complete the remaining speech and output "I went out to play today". Such processing has brought significant beneficial effects. It greatly improves the robustness and practicality of the speech recognition system, allowing the system to handle the user's possible incomplete or ambiguous speech input, thereby providing a more smooth and natural interactive experience. This is particularly important for applications such as voice assistants, real-time speech translation, and automatic subtitle generation, because it ensures that in these scenarios, the system can give accurate and complete output even in the face of imperfect input.

[0156] (4) By determining the speech similarity between each sub-predicted speech and the corresponding speech label, and performing preset operations on these similarities to calculate the loss value of the codec model, this method has brought significant benefits. It not only accurately quantifies the difference between the model prediction results and the actual speech label, but also provides direct guidance for updating the model parameters. Updating the model parameters based on the loss value can effectively optimize the model performance, making it more accurate and reliable when recognizing speech samples. This refined training strategy not only accelerates the convergence speed of the model, but also improves the model's generalization ability to different language environments and speaker variants, thereby achieving higher recognition accuracy and lower error rate in speech recognition applications.

[0157] (5) The method of splitting the speech sample, extracting features through specific encoding models, and then performing feature splicing has brought about many beneficial effects. It allows the model to analyze different parts of the speech sample more carefully and capture richer local feature information, thereby improving the accuracy of feature representation. Through splitting and feature extraction, the model can process long speech samples more effectively, reducing computational complexity and memory requirements, and improving the efficiency of training and inference. In addition, feature splicing not only integrates the speech information of different parts, but also enhances the model's understanding of the overall structure of the speech, allowing the encoding and decoding model to more accurately predict the complete speech content. Ultimately, this processing strategy significantly improves the performance of the speech recognition system, achieving higher recognition accuracy and lower error rate, and providing users with a more accurate and smooth voice interaction experience.

[0158] (6) The first label feature and the second label feature are concatenated to obtain the third label feature. The beneficial effect is that different types of label feature information can be comprehensively utilized to improve the model's understanding and representation ability of the speech signal. By determining the feature similarity between the third label feature and the predicted speech feature and using the similarity as the loss value, it helps to accurately evaluate the gap between the model prediction and the actual speech feature. The model parameters of the speech conversion model are updated based on the loss value, so that the model can continuously learn and optimize, thereby improving the accuracy and robustness of speech recognition. In addition, this method also helps to improve the generalization ability of the model, so that it has better performance when dealing with different scenarios and speech signals. In short, through feature concatenation and model updating, the overall performance of the speech recognition system can be effectively improved, providing users with a more intelligent and accurate voice interaction experience.

[0159] (7) The trained speech conversion model is used to convert the text to speech features. The beneficial effect is that the text information can be accurately mapped to the predicted speech features, and then the converted speech is generated in accordance with the specific acoustic features. This processing process not only ensures that the converted speech is semantically identical to the original text and accurately conveys the information, but also meets the acoustic feature requirements of the sample object, so that the speech output is consistent with the target speech style in terms of sound quality, intonation and speaking speed. This helps to improve the naturalness and fluency of speech synthesis and enhance the user experience. At the same time, this method also helps to expand the application areas of speech technology, such as voice assistants, automatic speech recognition and text-to-speech conversion, providing users with more intelligent and personalized voice interaction services.

[0160] (8) The predicted speech features of the text to be converted are decoded by the decoding model. The beneficial effect is that the text information can be converted into Mel speech. This process helps to preserve the acoustic characteristics of the speech and improve the quality of speech synthesis. The speech conversion of Mel speech by the vocoder can generate a converted speech that meets the target acoustic characteristics, making the output speech more natural and fluent in terms of sound quality, intonation and speaking speed. This processing method not only ensures that the converted speech is semantically identical to the original text, but also meets the acoustic feature requirements of the sample object, thereby improving the overall performance of speech synthesis. At the same time, this method also helps to expand the application field of speech technology and provide users with a more intelligent and personalized voice interaction experience.

[0161] (9) By introducing the answer prediction step before the speech conversion model, the beneficial effect is that it can improve the intelligence and response speed of the speech interaction system. First, the system can answer the user's questions immediately and improve the user experience. For example, when the user asks "What's the weather like today?", the system can quickly predict and answer "It's sunny today." Secondly, using the predicted answer sentence as the text to be converted helps to improve the accuracy of subsequent speech synthesis and ensure the consistency between the speech output and the answer content. During the speech feature conversion and speech conversion process, the system will process the semantic and acoustic features of the answer sentence to generate a high-quality speech response. Finally, the converted speech of the text to be converted is sent to the user, which can achieve effective information transmission and meet the user's need to obtain answers. This makes the speech interaction system more intelligent and efficient, and can better meet the user's speech query needs.

[0162] (10) It can effectively improve the sound quality and rhythmic naturalness of voice cloning and general voice synthesis of a sentence. Only a few seconds of a speaker's speech is needed to synthesize the speaker's voice. The speaker's voice can be customized for outbound call scenarios such as telesales and debt collection. The key point is the acquisition of VQ code, which directly affects the performance of the speech synthesis model. The performance of VQ code mainly depends on the VQ network structure. The most important thing is the ability of the VQ encoder module to learn features. This patent proposes a VQ encoder network model based on MEL features. Since MEL features are rich in information and complex, the acoustic information and semantic information in MEL features are separated and refined into models, which effectively improves the ability of the entire VQ encoder module to obtain features, thereby improving the performance of VQ and the sound quality and rhythmic naturalness of the generated audio of the speech synthesis TTS model.

[0163] (11) The embodiment of the present application improves the learning ability of the vq encoder feature, which helps to improve the effect of the TTS large model (sound quality and rhythm information), such as the effect of cloning a sentence and the effect of generating audio with a universal timbre. Since the mel feature is rich in multiple information, it is difficult for a single network structure to fully learn all of its information. The acoustic information and semantic information contained therein are separated and modeled separately to achieve the purpose of divide and conquer, thereby improving the ability of the VQ Encoder to learn features, indirectly improving the sound quality and rhythm naturalness of the generated audio, and achieving an effect that is indistinguishable from the real thing. In actual application, a VQ model is first trained, and then the VQ Encoder and vq modules are used to extract the discretized features of the vq code, and then the text and speaker embedding are combined to train the speech synthesis TTS large model. When in use, the text and speaker embedding are input to synthesize speech with high similarity to the speaker embedding, which has the ability of cloning a sentence and generating a universal timbre, and can be used in telemarketing outbound calls or debt collection scenarios. First, the real-time voice stream signal from the telephone user end is received through MRCP, and then the voice stream signal is input into the ASR speech recognition model to obtain the customer's speech content. The customer's intention is obtained through the NLP model, and the content to be fed back is obtained. The feedback content is synthesized into speech through TTS and fed back to the customer through the phone for response, completing a round of interaction, and this cycle repeats.

[0164] The above description is merely an embodiment of the present application and is not intended to limit the scope of protection of the present application. Any modifications, equivalent replacements, and improvements made within the spirit and scope of the present application are included in the scope of protection of the present application.

Claims

1. A method for training a speech conversion model, characterized in that: The method comprises: Performing speech feature conversion on the text sample using the speech conversion model to obtain predicted speech features; Splitting the mel speech with the same semantics as the text sample based on the frequency range to obtain a first mel speech and a second mel speech, where the first mel speech is used to reflect the acoustic features of the mel speech, and the second mel speech is used to reflect the semantics of the mel speech; Performing feature extraction on the first mel-level speech to obtain a first label feature, and performing feature extraction on the second mel-level speech to obtain a second label feature; The speech conversion model is trained based on the first label feature, the second label feature, and the predicted speech feature.

2. The method according to claim 1, characterized in that The extracting features of the first mel-level speech to obtain a first label feature includes: Querying a first encoding parameter adapted to the first mel-level speech from a plurality of preset encoding parameters, and performing feature extraction on the first mel-level speech using an encoding model of the first encoding parameter to obtain a first label feature; The performing feature extraction on the second mel speech to obtain a second label feature includes: A second encoding parameter adapted to the second mel-speech is searched from the plurality of preset encoding parameters, and a feature extraction is performed on the second mel-speech through an encoding model of the second encoding parameter to obtain the second label feature.

3. The method according to claim 2, characterized in that The encoding model belongs to a codec model. Before extracting features from the first mel speech to obtain a first label feature, the method further includes: Performing complete speech prediction on a speech sample carrying a speech label using the codec model to obtain a predicted speech of the speech sample, wherein the speech label includes the speech sample; The codec model is trained based on the predicted speech and the speech label.

4. The method according to claim 3, characterized in that The speech sample includes a plurality of different sub-speech samples, and the predicted speech includes the sub-predicted speech corresponding to each of the sub-speech samples; The training of the codec model based on the predicted speech and the speech label includes: Determining the speech similarity between each of the sub-predicted speech and the speech label, performing a preset operation on the speech similarity corresponding to each of the sub-predicted speech to obtain a loss value of the codec model; Based on the loss value of the codec model, the model parameters of the codec model are updated.

5. The method according to claim 3, characterized in that The method of performing complete speech prediction on the speech sample carrying the speech label by using the codec model to obtain the predicted speech of the speech sample includes: Splitting the voice sample to obtain a first voice sample and a second voice sample; Extracting features from the first speech sample using the coding model to obtain first speech features, and extracting features from the second speech sample using the coding model to obtain second speech features; performing feature concatenation on the first speech feature and the second speech feature to obtain a third speech feature; The coding and decoding model is used to perform complete speech prediction on the speech sample based on the third speech feature to obtain the predicted speech.

6. The method according to claim 1, characterized in that The training of the speech conversion model based on the first label feature, the second label feature, and the predicted speech feature includes: Performing feature splicing on the first label feature and the second label feature to obtain a third label feature; Determine a feature similarity between the third tag feature and the predicted speech feature, determine the feature similarity as a loss value, and update model parameters of the speech conversion model based on the loss value.

7. The method according to claim 1, characterized in that After training the speech conversion model based on the first label feature, the second label feature, and the predicted speech feature, the method further includes: Performing speech feature conversion on the text to be converted using the trained speech conversion model to obtain predicted speech features of the text to be converted; Performing voice conversion on the predicted voice features of the text to be converted to obtain a converted voice of the text to be converted; The converted speech has the same semantics as the text to be converted, and the converted speech meets the acoustic characteristics of the mel speech.

8. The method according to claim 7, characterized in that The performing speech conversion on the predicted speech features of the text to be converted to obtain the converted speech of the text to be converted includes: Decoding the predicted speech features of the text to be converted using a decoding model to obtain the mel speech of the text to be converted; The mel speech of the text to be converted is converted into speech by a vocoder to obtain the converted speech of the text to be converted.

9. The method according to claim 7, characterized in that Before performing speech feature conversion on the text to be converted using the trained speech conversion model to obtain predicted speech features of the text to be converted, the method further includes: In response to a question sentence sent by a question consulting party, predicting an answer to the question sentence to obtain an answer sentence to the question sentence, and determining the answer sentence as the text to be converted; After performing voice conversion on the predicted voice features of the text to be converted to obtain the converted voice of the text to be converted, the method further includes: The converted speech of the text to be converted is sent to the party asking the question.

10. The method according to claim 1, characterized in that The method of splitting the mel speech having the same semantics as the text sample based on the frequency range to obtain the first mel speech and the second mel speech includes: According to a first frequency range corresponding to the acoustic feature, determining a sub-speech within the first frequency range in the mel-level speech as the first mel-level speech; According to the second frequency range corresponding to the semantics, the sub-speech in the mel-level speech that is within the second frequency range is determined as the second mel-level speech.

11. A training device for a speech conversion model, characterized in that: The device comprises: A feature conversion module, configured to convert speech features of text samples using the speech conversion model to obtain predicted speech features; A splitting module is used to split the mel speech with the same semantics as the text sample based on the frequency range to obtain a first mel speech and a second mel speech, wherein the first mel speech is used to reflect the acoustic characteristics of the mel speech, and the second mel speech is used to reflect the semantics of the mel speech; a feature extraction module, configured to perform feature extraction on the first mel-level speech to obtain a first label feature, and perform feature extraction on the second mel-level speech to obtain a second label feature; A training module is used to train the speech conversion model based on the first label feature, the second label feature and the predicted speech feature.

12. An electronic device, characterized in that: The electronic device comprises: a memory for storing computer executable instructions or computer programs; The processor is configured to implement the method for training a speech conversion model according to any one of claims 1 to 10 when executing the computer-executable instructions or computer program stored in the memory.

13. A computer-readable storage medium storing computer-executable instructions or a computer program, characterized in that: When the computer executable instructions or computer program are executed by a processor, the training method of the speech conversion model according to any one of claims 1 to 10 is implemented.

14. A computer program product comprising computer executable instructions or a computer program, characterized in that: When the computer program or computer executable instructions are executed by a processor, the training method of the speech conversion model according to any one of claims 1 to 10 is implemented.

Images