Voice conversion model training method and device, electronic equipment, storage medium and program product

Abstract

The invention provides a voice conversion model training method and device, electronic equipment, a storage medium and a program product. The method comprises the following steps: performing voice feature conversion on a text sample through a voice conversion model to obtain predicted voice features; splitting the Mel speech having the same semantics as the text sample to obtain a first Mel speech and a second Mel speech, the first Mel speech being used for reflecting acoustic features of the Mel speech, and the second Mel speech being used for reflecting semantics of the Mel speech; performing feature extraction on the first Mel voice to obtain a first tag feature, and performing feature extraction on the second Mel voice to obtain a second tag feature; and training a voice conversion model based on the first tag feature, the second tag feature and the predicted voice feature. According to the invention, the voice feature generation performance of the voice conversion model can be effectively improved.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a training method, device, electronic device, storage medium and program product for a speech conversion model. Background Art

[0002] Artificial Intelligence (AI) is a comprehensive discipline that includes both hardware-level and software-level technologies. AI basic technologies generally include cloud computing, big data processing technology, and pre-trained model technology. Pre-trained models are also called large models or basic models. After fine-tuning, they can be widely used in downstream tasks in various directions of AI. AI software technologies mainly include computer vision technology, speech processing technology, and natural language processing technology.

[0003] In the related art, the training of the speech conversion model is usually achieved directly through the predicted speech features and speech label features obtained by the speech conversion model. Since the feature elements of various dimensions in the speech label features have different training effects on the speech conversion model, the acoustic features and semantics in the speech label features are not prominently expressed, the speech feature generation quality of the speech conversion model obtained by training is not high, resulting in poor speech feature generation performance of the speech conversion model. Summary of the invention

[0004] The embodiments of the present application provide a training method, device, electronic device, computer-readable storage medium and computer program product for a speech conversion model, which can effectively improve the speech feature generation performance of the speech conversion model.

[0005] The technical solution of the embodiment of the present application is implemented as follows:

[0006] The present application provides a method for training a speech conversion model, including:

[0007] The speech conversion model is used to convert the text sample into speech features to obtain predicted speech features;

[0008] Splitting the Mel voice with the same semantics as the text sample to obtain a first Mel voice and a second Mel voice, wherein the first Mel voice is used to reflect the acoustic features of the Mel voice, and the second Mel voice is used to reflect the semantics of the Mel voice;

[0009] Performing feature extraction on the first mel speech to obtain a first label feature, and performing feature extraction on the second mel speech to obtain a second label feature;

[0010] The speech conversion model is trained based on the first label feature, the second label feature and the predicted speech feature.

[0011] The present application provides a training device for a speech conversion model, including:

[0012] A feature conversion module, used to perform speech feature conversion on the text sample through the speech conversion model to obtain predicted speech features;

[0013] A splitting module, used for splitting the Mel speech having the same semantics as the text sample to obtain a first Mel speech and a second Mel speech, wherein the semantics of the Mel speech are the same as the semantics of the text sample, the first Mel speech is used to reflect the acoustic features of the Mel speech, and the second Mel speech is used to reflect the semantics of the Mel speech;

[0014] A feature extraction module, configured to perform feature extraction on the first mel speech to obtain a first label feature, and perform feature extraction on the second mel speech to obtain a second label feature;

[0015] A training module is used to train the speech conversion model based on the first label feature, the second label feature and the predicted speech feature.

[0016] An embodiment of the present application provides an electronic device, including:

[0017] A memory for storing computer executable instructions or computer programs;

[0018] The processor is used to implement the training method of the speech conversion model provided in the embodiment of the present application when executing the computer executable instructions or computer program stored in the memory.

[0019] An embodiment of the present application provides a computer-readable storage medium storing computer-executable instructions for causing a processor to execute and implement the training method of the speech conversion model provided in the embodiment of the present application.

[0020] The embodiment of the present application provides a computer program product, which includes a computer program or a computer executable instruction, and the computer program or the computer executable instruction is stored in a computer-readable storage medium. The processor of the electronic device reads the computer executable instruction from the computer-readable storage medium, and the processor executes the computer executable instruction, so that the electronic device executes the training method of the speech conversion model described in the embodiment of the present application.

[0021] The embodiments of the present application have the following beneficial effects:

[0022] Through the speech conversion model, the text sample is converted into speech features to obtain predicted speech features; by splitting the Mel speech with the same semantics as the text sample, a first Mel speech and a second Mel speech are obtained, the first Mel speech is used to reflect the acoustic features of the sample object, and the second Mel speech is used to reflect the semantics; the first Mel speech is feature extracted to obtain a first label feature, and the second Mel speech is feature extracted to obtain a second label feature; based on the first label feature, the second label feature and the predicted speech feature, the speech conversion model is trained, so that the first label feature and the second label feature can respectively reflect the acoustic features of the sample object and the semantics of the text sample, so that the acoustic features and semantics contained in the Mel speech are split to refine the modeling, so that the speech conversion model fully learns the acoustic features and semantics contained in the Mel speech, which can effectively improve the speech feature generation performance of the speech conversion model. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] Figure 1 It is a schematic diagram of the architecture of the speech conversion model system for training provided in an embodiment of the present application;

[0024] Figure 2 is a schematic diagram of the structure of an electronic device for training a speech conversion model provided in an embodiment of the present application;

[0025] Figure 3 It is a flowchart of a method for training a speech conversion model provided in an embodiment of the present application;

[0026] Figure 4 It is a schematic diagram of the principle of the training method of the speech conversion model provided in the embodiment of the present application;

[0027] Figure 5 It is a schematic diagram of the principle of mel speech splitting provided in an embodiment of the present application;

[0028] Figure 6 It is a schematic diagram of the principle of the encoding and decoding model provided in the embodiment of the present application;

[0029] Figure 7 It is a schematic diagram of the principle of voice conversion provided in an embodiment of the present application. DETAILED DESCRIPTION

[0030] In order to make the purpose, technical solutions and advantages of the present application clearer, the present application will be further described in detail below in conjunction with the accompanying drawings. The described embodiments should not be regarded as limiting the present application. All other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of this application.

[0031] In the following description, reference is made to “some embodiments”, which describe a subset of all possible embodiments, but it will be understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0032] In the following description, the terms "first\second\third" involved are merely used to distinguish similar objects and do not represent a specific ordering of the objects. It can be understood that "first\second\third" can be interchanged with a specific order or sequence where permitted, so that the embodiments of the present application described here can be implemented in an order other than that illustrated or described here.

[0033] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.

[0034] Before further describing the embodiments of the present application in detail, the nouns and terms involved in the embodiments of the present application are explained. The nouns and terms involved in the embodiments of the present application are subject to the following interpretations.

[0035] 1) Artificial Intelligence (AI): It is a comprehensive discipline that covers a wide range of fields, including both hardware-level and software-level technologies. The basic technologies of artificial intelligence generally include sensors, dedicated artificial intelligence chips, cloud computing, distributed storage, big data processing technology, pre-trained model technology, operation / interaction systems, mechatronics, etc. Among them, the pre-trained model is also called a large model or a basic model. After fine-tuning, it can be widely used in downstream tasks in various major directions of artificial intelligence. Artificial intelligence software technology mainly includes computer vision technology, speech processing technology, natural language processing technology, and machine learning / deep learning.

[0036] 2) Machine Learning (ML): It is a multi-disciplinary interdisciplinary subject involving probability theory, statistics, approximation theory, convex analysis, algorithm complexity theory and other disciplines. It specializes in studying how computers simulate or implement human learning behavior to acquire new knowledge or skills and reorganize existing knowledge structures to continuously improve their performance. Machine learning is the core of artificial intelligence and the fundamental way to make computers intelligent. Its applications are spread across all areas of artificial intelligence. Machine learning and deep learning usually include artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning, and learning by induction. The pre-trained model is the latest development in deep learning, which integrates the above technologies.

[0037] 3) Mel-Frequency Cepstral Coefficients (MFCC): is a set of feature vectors commonly used in speech recognition and speech synthesis. Mel-Frequency Cepstral Coefficients reflect the auditory characteristics of the human auditory system and are a method of encoding speech signals. Mel frequency is a linear scale based on the human ear's perception of pitch rather than the actual physical frequency. The conversion relationship between Mel frequency and actual frequency is nonlinear, and this conversion can better simulate the human ear's ability to perceive different pitch frequencies. In the Mel frequency scale, lower frequencies correspond to areas where the human ear is more sensitive, while higher frequencies gradually become sparser as the frequency increases.

[0038] 4) Speech: It is a natural way for humans to communicate. It involves the coordinated action of human vocal organs (such as vocal cords, tongue, lips, etc.) and respiratory system, and produces sound waves through air vibration, thereby conveying information and emotions. Speech can be transmitted through different media, such as air, water and other media. In computer science and engineering, speech processing refers to the technology of analyzing and processing speech signals, aiming to extract useful information, identify language content, generate speech or improve speech quality. Speech processing technology includes speech recognition (SR), speech synthesis (SS), speech enhancement (Speech Enhancement), speech coding (Speech Coding), etc. Speech recognition refers to the process of converting speech signals into corresponding text or commands, which involves multiple disciplines such as signal processing, pattern recognition, and machine learning. Speech synthesis, on the contrary, is the process of converting text information into speech signals. It is widely used in voice assistants, automatic telephone systems and other fields. Speech signal processing is one of the branches of phonetics, which studies how to analyze and synthesize speech through mathematical models and computing techniques. In the field of artificial intelligence, speech processing technology is a key component for achieving natural language processing and human-computer interaction.

[0039] During the implementation of the embodiments of the present application, the applicant discovered that the related technology has the following problems:

[0040] In the related art, the training of the speech conversion model is usually achieved directly through the predicted speech features and speech label features obtained by the speech conversion model. Since the feature elements of various dimensions in the speech label features have different training effects on the speech conversion model, the acoustic features and semantics in the speech label features are not prominently expressed, the speech feature generation quality of the speech conversion model obtained by training is not high, resulting in poor speech feature generation performance of the speech conversion model.

[0041] The embodiments of the present application provide a training method, device, electronic device, computer-readable storage medium and computer program product for a speech conversion model, which can effectively improve the speech feature generation performance of the speech conversion model. The following describes an exemplary application of the training system for the speech conversion model provided in the embodiments of the present application.

[0042] See also Figure 1 , Figure 1 It is a schematic diagram of the architecture of the trained speech conversion model system 100 provided in an embodiment of the present application. The terminal (terminal 400 is shown as an example) is connected to the server 200 via the network 300. The network 300 can be a wide area network or a local area network, or a combination of the two.

[0043] The terminal 400 is used for the user to use the client 410 to display the text sample on the graphical interface 410-1 (the graphical interface 410-1 is shown as an example). The terminal 400 and the server 200 are connected to each other via a wired or wireless network.

[0044] In some embodiments, the server 200 may be an independent physical server, or a server cluster or business system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (CDN), and big data and artificial intelligence platforms. The terminal 400 may be a smart phone, a tablet computer, a laptop computer, a desktop computer, a smart speaker, a smart TV, a smart watch, a car terminal, etc., but is not limited thereto. The electronic device provided in the embodiment of the present application may be implemented as a terminal or as a server. The terminal and the server may be directly or indirectly connected via wired or wireless communication, which is not limited in the embodiment of the present application.

[0045] In some embodiments, the server 200 performs speech feature conversion on the text sample through a speech conversion model to obtain predicted speech features, splits the Mel speech to obtain a first Mel speech and a second Mel speech, extracts features from the first Mel speech to obtain a first label feature, extracts features from the second Mel speech to obtain a second label feature, trains the speech conversion model based on the first label feature, the second label feature and the predicted speech feature, and sends the trained speech conversion model to the terminal 400.

[0046] In other embodiments, the terminal 400 performs speech feature conversion on the text sample through a speech conversion model to obtain predicted speech features, splits the Mel speech to obtain a first Mel speech and a second Mel speech, extracts features from the first Mel speech to obtain a first label feature, extracts features from the second Mel speech to obtain a second label feature, trains the speech conversion model based on the first label feature, the second label feature and the predicted speech feature, and sends the trained speech conversion model to the server 200.

[0047] In other embodiments, the embodiments of the present application can be implemented with the aid of cloud technology. Cloud technology refers to a hosting technology that unifies a series of resources such as hardware, software, and networks within a wide area network or a local area network to achieve data computing, storage, processing, and sharing.

[0048] Cloud technology is a general term for network technology, information technology, integration technology, management platform technology, and application technology based on the cloud computing business model. It can form a resource pool that can be used on demand and is flexible and convenient. Cloud computing technology will become an important support. The background services of the technical network system require a large amount of computing and storage resources.

[0049] See also Figure 2 , Figure 2 is a structural diagram of an electronic device 500 for training a speech conversion model provided in an embodiment of the present application, wherein: Figure 2 The electronic device 500 shown may be Figure 1 The server 200 or the terminal 400 in Figure 2 The electronic device 500 shown includes: at least one processor 430, a memory 450, and at least one network interface 420. The various components in the electronic device 500 are coupled together via a bus system 440. It is understood that the bus system 440 is used to achieve connection and communication between these components. In addition to the data bus, the bus system 440 also includes a power bus, a control bus, and a status signal bus. However, for the sake of clarity, the bus system 440 is not described in detail. Figure 2 Various buses are labeled as bus system 440 .

[0050] The processor 430 can be an integrated circuit chip with signal processing capabilities, such as a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc., where the general-purpose processor can be a microprocessor or any conventional processor, etc.

[0051] The memory 450 may be removable, non-removable, or a combination thereof. Exemplary hardware devices include solid-state memory, hard drives, optical drives, etc. The memory 450 may optionally include one or more storage devices that are physically remote from the processor 430.

[0052] The memory 450 includes a volatile memory or a non-volatile memory, and may also include both volatile and non-volatile memories. The non-volatile memory may be a read-only memory (ROM), and the volatile memory may be a random access memory (RAM). The memory 450 described in the embodiments of the present application is intended to include any suitable type of memory.

[0053] In some embodiments, memory 450 can store data to support various operations, examples of which include programs, modules, and data structures, or a subset or superset thereof, as exemplarily described below.

[0054] Operating system 451, including system programs for processing various basic system services and performing hardware-related tasks, such as a framework layer, a core library layer, a driver layer, etc., for implementing various basic services and processing hardware-based tasks;

[0055] The network communication module 452 is used to reach other electronic devices via one or more (wired or wireless) network interfaces 420. Exemplary network interfaces 420 include: Bluetooth, Wireless Fidelity (WiFi), and Universal Serial Bus (USB).

[0056] In some embodiments, the training device for the speech conversion model provided in the embodiments of the present application can be implemented in software. Figure 2 A training device 455 of a speech conversion model stored in a memory 450 is shown, which may be software in the form of a program or a plug-in, and includes the following software modules: a feature conversion module 4551, a splitting module 4552, a feature extraction module 4553, and a training module 4554. These modules are logical, and thus may be arbitrarily combined or further split according to the functions implemented. The functions of each module will be described below.

[0057] In other embodiments, the training device of the speech conversion model provided in the embodiments of the present application can be implemented in hardware. As an example, the training device of the speech conversion model provided in the embodiments of the present application can be a processor in the form of a hardware decoding processor, which is programmed to execute the training method of the speech conversion model provided in the embodiments of the present application. For example, the processor in the form of a hardware decoding processor can adopt one or more application-specific integrated circuits (ASICs), DSPs, programmable logic devices (PLDs), complex programmable logic devices (CPLDs), field-programmable gate arrays (FPGAs) or other electronic components.

[0058] In some embodiments, the terminal or server can implement the training method of the speech conversion model provided in the embodiment of the present application by running a computer program or a computer executable instruction. For example, the computer program can be a native program (e.g., a dedicated training program) or a software module in an operating system, for example, a training module that can be embedded in any program (such as an instant messaging client, an album program, an electronic map client, a navigation client); for example, it can be a native application (APP, Application), that is, a program that needs to be installed in the operating system to run. In short, the above-mentioned computer program can be an application, module or plug-in in any form.

[0059] The training method of the speech conversion model provided in the embodiment of the present application will be explained in combination with the exemplary application and implementation of the server or terminal provided in the embodiment of the present application.

[0060] See also Figure 3 , Figure 3 is a flow chart of the training method of the speech conversion model provided in the embodiment of the present application, which will be combined with Figure 3 Steps 101 to 105 are shown for illustration. The training method of the speech conversion model provided in the embodiment of the present application can be implemented by a server or a terminal alone, or by a server and a terminal in collaboration. The following will be described using the server alone as an example.

[0061] In step 101, the speech feature conversion is performed on the text sample through the speech conversion model to obtain the predicted speech feature.

[0062] In some embodiments, the speech conversion model may be a natural language processing model such as a large language model, which converts speech features of text samples to obtain predicted speech features.

[0063] In some embodiments, natural language processing (NLP) is an important direction in the fields of computer science and artificial intelligence. It studies various theories and methods that can achieve effective communication between people and computers using natural language. Natural language processing is a science that integrates linguistics, computer science, and mathematics. Therefore, research in this field will involve natural language, that is, the language used by people in daily life, so it is closely related to the study of linguistics, but there are important differences. Natural language processing is not a general study of natural language, but the development of computer systems that can effectively realize natural language communication, especially the software systems therein. Therefore, it is a part of computer science, and natural language processing is mainly used in machine translation, automatic summarization, opinion extraction, text classification, question answering, text semantic comparison, speech recognition, etc.

[0064] In some embodiments, a large language model is a speech conversion model designed to understand and generate human language. They are trained on large amounts of text data and can perform a wide range of tasks, including text summarization, translation, sentiment analysis, etc. Large language models are characterized by their large scale and billions of parameters that help them learn complex patterns in language data, usually based on deep learning architectures. Large language models are also called large language models or large models. Large language models refer to deep learning models trained with large amounts of text data, containing billions or even larger parameters, which can be used to generate natural language text and understand the meaning of natural language text. Through training, the model can learn the statistical laws and semantic associations of language to build a huge language knowledge base, thereby simulating human language understanding and generation capabilities.

[0065] As an example, see Figure 4 , Figure 4 It is a schematic diagram of the principle of the training method of the speech conversion model provided in the embodiment of the present application. Through the speech conversion model, the speech feature conversion is performed on the text sample to obtain the predicted speech feature.

[0066] As an example, in the application scenario of a voice assistant, the user queries the weather information through the voice assistant. The voice assistant uses a speech conversion model to convert text into predicted speech features to match the user's voice, obtain a Mel speech sample with the same semantics as the text "What's the weather like in Beijing today?" and split it into two parts: First Mel speech: captures the acoustic features of the voice, such as intonation and volume. Second Mel speech: specifically reflects the semantic content of the speech, that is, the meaning of the expression. Extract acoustic features (first label features) from the first Mel speech. Extract semantic features (second label features) from the second Mel speech. Combining the first label features, the second label features, and the predicted speech features, the voice assistant continuously trains its speech conversion model to provide more natural and personalized voice feedback.

[0067] As an example, in the application scenario of real-time speech translation, at an international conference, the real-time speech translation system was used to instantly translate the speaker's speech. The speaker gave a speech in Chinese. The real-time speech translation system converted the Chinese text into predicted speech features and simultaneously acquired the speaker's voice features in real time. Mel speech samples with the same semantics were split into acoustic features and semantic features, and acoustic and semantic features were extracted respectively to understand and generate translations in the target language. Continuous training was performed to improve the accuracy and synchronization of the translation, ensuring that the audience could hear accurate translations in real time.

[0068] As an example, in the application scenario of automatic subtitle generation, in video content creation, the automatic subtitle generation system provides real-time subtitles for the video. The character in the video starts speaking, and the speech content in the video is converted into predicted speech features, matching the speaker's voice, looking for Mel speech samples that are semantically consistent with the speech content in the video, and splitting them into acoustic features and semantic features. Features are extracted from the split Mel speech for generating subtitles. Through training, the recognition ability of various speakers' voices is improved, and more accurate and synchronized subtitles are generated.

[0069] In step 102, the mel speech having the same semantics as the text sample is split to obtain a first mel speech and a second mel speech.

[0070] In some embodiments, the semantics of the Mel voice is the same as the semantics of the text sample, the first Mel voice is used to reflect the acoustic features of the Mel voice, and the second Mel voice is used to reflect the semantics of the Mel voice.

[0071] In some embodiments, the above-mentioned Mel speech may refer to Mel frequency cepstral coefficients, which is a set of feature vectors commonly used in speech recognition and speech synthesis. Mel frequency cepstral coefficients reflect the auditory characteristics of the human auditory system and are a method of encoding speech signals. Mel frequency is a linear scale based on the human ear's perception of pitch rather than the actual physical frequency. The conversion relationship between Mel frequency and actual frequency is nonlinear, and this conversion can better simulate the human ear's ability to perceive different pitch frequencies. In the Mel frequency scale, lower frequencies correspond to areas where the human ear is more sensitive, while higher frequencies gradually become sparser as the frequency increases.

[0072] As an example, see Figure 5 , Figure 5 This is a schematic diagram of the principle of mel speech splitting provided in an embodiment of the present application, where the mel speech is split to obtain a first mel speech and a second mel speech.

[0073] In some embodiments, the above-mentioned splitting of the Mel-speech having the same semantics as the text sample to obtain the first Mel-speech and the second Mel-speech can be achieved as follows: according to the first frequency range corresponding to the acoustic feature, the sub-speech in the Mel-speech within the first frequency range is determined as the first Mel-speech; according to the second frequency range corresponding to the semantics, the sub-speech in the Mel-speech within the second frequency range is determined as the second Mel-speech.

[0074] As an example, suppose there is a recorded speech sample of someone saying the phrase "I love you." Assume that the Mel-Frequency Cepstral Coefficients (MFCCs) of the speech have been extracted, and now the speech sample needs to be split according to a specific frequency range. Acoustic Feature Extraction: First, extract the MFCCs of the speech sample to obtain a coefficient matrix containing multiple frames. Each frame contains multiple MFCC coefficients, each coefficient is associated with a specific frequency range. Define frequency ranges: First frequency range: Based on the acoustic features, define the first frequency range corresponding to a specific MFCC coefficient, for example, decide to use the 4th MFCC coefficient, which is usually related to the resonance characteristics of the speech. Second frequency range: Based on the semantics, define the second frequency range corresponding to another MFCC coefficient, for example, select the 8th MFCC coefficient, which may be related to the rhythm or intonation of the speech. Split Mel speech: First Mel speech: All sub-speech in the Mel speech within the first frequency range are determined as the first Mel speech. Assuming that the value of the 4th MFCC coefficient is between a certain threshold (for example, between -20 and 20), select all frames that meet this condition to form a subset. This part of the Mel speech reflects the acoustic features associated with the frequency range. Second Mel speech: Similarly, all sub-speech in the Mel speech within the second frequency range are determined as the second Mel speech. Assuming that the value of the 8th MFCC coefficient is between another specific threshold (for example, between 30 and 50), all frames that meet this condition are selected to form another subset. This part of the Mel speech reflects the semantic features associated with the frequency range. The specific operation is as follows: Assume that there are 100 frames of MF CCs, each containing 13 coefficients. Filter out frames with values ​​of the 4th MFCC coefficient between -20 and 20, assuming that these frames are frames 1 to 40. These frames constitute the first Mel speech. Filter out frames with values ​​of the 8th MFCC coefficient between 30 and 50, assuming that these frames are frames 41 to 100. These frames constitute the second Mel speech. In this way, the original Mel speech is split into two subsets based on acoustic features and semantics: the first Mel speech and the second Mel speech, each of which reflects different aspects of speech. This method may be used in applications such as speech signal processing, analysis or synthesis.

[0075] As an example, see Figure 5 According to the first frequency range (0-60bins) corresponding to the acoustic feature, the sub-speech in the Mel-speech within the first frequency range is determined as the first Mel-speech; according to the second frequency range (60-80bins) corresponding to the semantics, the sub-speech in the Mel-speech within the second frequency range is determined as the second Mel-speech.

[0076] In this way, by accurately defining the first frequency range corresponding to the acoustic features, the sub-speech in the mel-speech within this range is effectively screened out and determined as the first mel-speech, which helps to improve the accuracy and efficiency of speech recognition. At the same time, according to the second frequency range corresponding to the semantics, the sub-speech in the mel-speech within this range is further determined as the second mel-speech, which can enhance the understanding and analysis of the meaning of the speech, thereby achieving more accurate speech recognition and processing, and improving the performance of the overall system and user experience.

[0077] In step 103, feature extraction is performed on the first mel speech to obtain a first label feature.

[0078] In some embodiments, the above step 103 can be implemented as follows: query a first encoding parameter adapted to the first Mel-level speech from multiple preset encoding parameters, and extract features of the first Mel-level speech through the encoding model of the first encoding parameter to obtain the first label feature.

[0079] In some embodiments, feature extraction refers to the process of extracting information that can represent the essential characteristics of the data from raw data in signal processing or machine learning.

[0080] In some embodiments, the preset coding parameters are predefined model parameters, such as neural network weights in deep learning, which are obtained by training on a large amount of data, and each preset coding parameter may correspond to a different model architecture or training configuration, such as a different number of layers, number of neurons, activation function, regularization parameter, etc. Querying a first coding parameter adapted to the first Mel voice from a plurality of preset coding parameters, and querying a second coding parameter adapted to the second Mel voice from a plurality of preset coding parameters, involves selecting or training a model that can select or generate the most appropriate coding parameter according to the minimum distortion or maximum similarity of the input Mel voice signal, and the first Mel voice and the second Mel voice are respectively feature extracted through their adapted coding models.

[0081] In some embodiments, the first mel speech and the second mel speech are respectively subjected to feature extraction through their adapted coding models. For the first mel speech, this process includes: using the coding model corresponding to the queried first coding parameter. The first mel speech is input into the model, and the feature extraction capability of the model is used to obtain the feature representation of the speech, that is, the first label feature.

[0082] In some embodiments, the encoding model of the above-mentioned first encoding parameter can be a semantic encoder (semanticsencoder) network structure. Semantics Encoder generally refers to a network structure used to extract text semantic information in natural language processing (NLP) tasks. This network structure is usually used in deep learning models, such as BERT (Bidirectional Encoder Representations from Transformer s), RoBERTa (A Robustly Optimized BERT Pre-training Approach), XLNet (XLNet: Generalized Autor egressive Pretraining for Language Modeling), etc., which are all variants of the Transformer architecture. The core purpose of SemanticsEncoder is to convert the input text sequence into a set of vectors that can represent text semantic information. These vectors can be used for various downstream tasks, such as text classification, sentiment analysis, question answering, etc. A typical SemanticsEncoder network structure includes the following main parts: Embedding Layer: Map words or characters in the text to vectors of fixed dimensions. In the Transformer model, this step is usually completed through the Embedding matrix. Positional Encoding: Since the Transformer model itself does not have the ability to handle sequence order, positional encoding needs to be added to provide the model with the position information of the elements in the sequence. Multi-Head Self-Attention: This is the core component of the Transformer model, which allows the model to consider different parts of the sequence simultaneously when generating representations. Feed-Forward Neural Networks: After the self-attention layer, two fully connected layers are applied to the output of each position to introduce nonlinear transformations. Residual Connections: Adding residual connections between the output of each sub-layer and the input of the next sub-layer helps the model learn more complex patterns. Layer Normalization: Applying layer normalization after residual connections helps stabilize the gradient during learning. Classification Head / Output Layer: After pre-training is completed, the corresponding output layer is added according to the specific downstream task. For example, in a text classification task, it may be a softmax layer for generating category probabilities.The specific details of Semantics Encoder may be adjusted according to different pre-training goals and downstream tasks. For example, BERT adds two pre-training tasks, Next Sentence Prediction and Masked Language Modeling, based on Transformer to improve its semantic understanding ability. RoBE RTa removes the next sentence prediction task in BERT and instead adopts other training strategies, such as dynamic masking, to improve the performance of the model.

[0083] In some embodiments, the above-mentioned coding model belongs to a codec model. Before the above-mentioned step 102, the training of the codec model can be achieved in the following manner: through the codec model, a complete speech prediction is performed on a speech sample carrying a speech label to obtain a predicted speech of the speech sample, and the speech label includes the speech sample; based on the predicted speech and the speech label, the codec model is trained.

[0084] As an example, the above-mentioned speech sample is at least part of a speech tag. Through the codec model, a complete speech prediction is performed on a speech sample (e.g., I am going out to play tonight) carrying a speech tag (e.g., I am going out to play tonight) to obtain the predicted speech of the speech sample (e.g., I am going out to play today), and the voice tag (e.g., I am going out to play tonight) includes the speech sample (e.g., I am going out to play today).

[0085] As an example, in speech recognition, sometimes the input speech sample may be incomplete, but the model still needs to be able to predict the complete speech label. The following is a specific example of how to handle this situation through the encoder-decoder model: Suppose there is a speech recognition system that aims to convert the user's partial speech input into a complete sentence. Speech sample and speech label: Speech sample: The user said "I am..." today, and the speech label: The complete sentence is "I am going out to play tonight." Preprocessing of speech sample: Input the user's speech sample "I am..." to the model. The model first preprocesses the speech and extracts audio features such as MFCC or spectrogram. Prediction of the encoder-decoder model: The model uses the extracted audio features to predict the next utterance. In this example, the model needs to predict the missing part after "I am...". Complete speech prediction: Suppose the model predicts "I am going out to play today." This means that the model successfully recognizes the part of the speech that the user wants to express and correctly completes the sentence. Comparison with speech label: The predicted speech "I am going out to play today" obtained by the model is compared with the actual speech label "I am going out to play tonight." In this case, the prediction does not exactly match the speech label, but the prediction contains the complete content of the speech sample. Train the encoder-decoder model: Use the predicted speech "I went out to play today" and the actual speech label "I went out to play tonight" to train the model. Through this training process, the model learns how to better predict complete sentences when receiving partial speech input. Details of the training process: The model calculates the difference between the predicted speech and the speech label, such as using edit distance or sequence alignment algorithms, to determine the deviation of the predicted text from the actual label. Use the backpropagation algorithm to update the model weights to reduce the prediction error and improve the model's ability to handle partial speech input.

[0086] In this way, by using the above-mentioned codec model, when performing complete speech prediction for speech samples carrying speech tags, even if the speech sample is only at least part of the speech tag, the model can effectively infer the complete speech content. For example, given the speech tag "I went out to play tonight", when the input speech sample is only "I went out to play today...", the model can intelligently predict and complete the remaining speech and output "I went out to play today." Such processing brings significant beneficial effects. It greatly improves the robustness and practicality of the speech recognition system, allowing the system to handle possible incomplete or ambiguous speech input from users, thereby providing a smoother and more natural interactive experience. This is especially important for applications such as voice assistants, real-time speech translation, and automatic subtitle generation, because it ensures that in these scenarios, the system can give accurate and complete output even in the face of imperfect input.

[0087] In some embodiments, the speech sample includes a plurality of different sub-speech samples, and the predicted speech includes the sub-predicted speech corresponding to each of the sub-speech samples.

[0088] In some embodiments, the above-mentioned training of the codec model based on the predicted speech and the speech label can be achieved in the following manner: determining the speech similarity between each of the sub-predicted speech and the speech label, performing a preset operation on the speech similarity corresponding to each of the sub-predicted speech to obtain a loss value of the codec model; based on the loss value of the codec model, updating the model parameters of the codec model.

[0089] In some embodiments, the above-mentioned preset operation of the speech similarity corresponding to each sub-predicted speech to obtain the loss value of the codec model can be achieved by adding the speech similarity corresponding to each sub-predicted speech to obtain the loss value of the codec model.

[0090] As an example, the loss value of the above encoding and decoding model can be expressed as:

[0091]

[0092] Among them, L is used to indicate the loss value of the codec model, Li is used to indicate the speech similarity corresponding to the sub-predicted speech, and N is used to indicate the number of sub-predicted speech.

[0093] In this way, by determining the speech similarity between each sub-predicted speech and the corresponding speech label, and performing preset operations on these similarities to calculate the loss value of the codec model, this method brings significant beneficial effects. It can not only accurately quantify the difference between the model prediction results and the actual speech labels, but also provide direct guidance for the update of model parameters. Updating model parameters based on loss values ​​can effectively optimize model performance, making it more accurate and reliable when recognizing speech samples. This refined training strategy not only speeds up the convergence of the model, but also improves the model's generalization ability to different language environments and speaker variants, thereby achieving higher recognition accuracy and lower error rates in speech recognition applications.

[0094] In some embodiments, the above-mentioned complete speech prediction of the speech sample carrying the speech tag through the codec model to obtain the predicted speech of the speech sample can be achieved in the following way: splitting the speech sample to obtain a first speech sample and a second speech sample; performing feature extraction on the first speech sample through the first coding model to obtain a first speech feature, and performing feature extraction on the second speech sample through the second coding model to obtain a second speech feature; performing feature concatenation on the first speech feature and the second speech feature to obtain a third speech feature; performing complete speech prediction on the speech sample based on the third speech feature through the codec model to obtain the predicted speech.

[0095] As an example, the splitting of a speech sample: suppose there is a complete speech sample which is a sentence said by the user: "I'm going out to play tonight". Split this speech sample into two parts, the first speech sample may be the first half: "I'm going out to play tonight," and the second speech sample is the second half: "Going out to play." Feature extraction of the first speech sample: Use the first coding model to extract features from the first speech sample "I'm going out to play tonight". The coding model may analyze the MFCC coefficients of the speech to obtain a feature vector sequence, which constitutes the first speech feature. Feature extraction of the second speech sample: Use the second coding model to extract features from the second speech sample "going out to play". Similarly, the model analyzes the MF CC coefficients of the speech to obtain another feature vector sequence, namely the second speech feature. Feature concatenation: Concatenate the first speech feature and the second speech feature to obtain a longer feature vector sequence, which is the third speech feature. Concatenation can be done by a simple cascade operation, or it can be a more complex feature fusion strategy, such as feature weighted summation or feature transformation. Complete speech prediction: Use the obtained third speech feature to input it into the encoding and decoding model. The encoder-decoder model may be a deep learning model, such as a recurrent neural network (RNN) or a Transformer model, which predicts the text of the entire sentence based on these features. Prediction results: Assume that the encoder-decoder model outputs the text: "I'm going out tonight", which is consistent with the complete sentence of the original voice sample. Through this method of splitting the voice sample, extracting features separately, concatenating features and making predictions, the model can handle voice inputs of different lengths more flexibly, and can better understand and predict the entire voice sample based on the extraction of local features. It is particularly useful when processing long speech or complex speech structures because it can reduce the complexity of model processing while maintaining the accuracy of prediction.

[0096] As an example, see Figure 6 , Figure 6This is a schematic diagram of the principle of the coding and decoding model provided in an embodiment of the present application. The first coding model is used to extract features of the first speech sample to obtain first speech features. The second coding model is used to extract features of the second speech sample to obtain second speech features. The first speech features and the second speech features are concatenated to obtain third speech features. The coding and decoding model is used to predict the complete speech of the speech sample based on the third speech features to obtain the predicted speech.

[0097] In this way, the method of splitting the speech sample, extracting features through specific encoding models, and then splicing the features brings many beneficial effects. It allows the model to analyze different parts of the speech sample more carefully and capture richer local feature information, thereby improving the accuracy of feature representation. Through splitting and feature extraction, the model can process long speech samples more effectively, reduce computational complexity and memory requirements, and improve the efficiency of training and reasoning. In addition, feature splicing not only integrates the speech information of different parts, but also enhances the model's understanding of the overall structure of the speech, so that the encoding and decoding model can more accurately predict the complete speech content. Ultimately, this processing strategy significantly improves the performance of the speech recognition system, achieves higher recognition accuracy and lower error rate, and provides users with a more accurate and smooth voice interaction experience.

[0098] In step 104, feature extraction is performed on the second mel-level speech to obtain a second label feature.

[0099] In some embodiments, the above step 104 can be implemented as follows: query a second encoding parameter adapted to the second Mel-speech from a plurality of preset encoding parameters, and extract features of the second Mel-speech through an encoding model of the second encoding parameter to obtain the second label feature.

[0100] In some embodiments, in deep learning and machine learning, especially in the field of speech processing, AcousticEncoder is a neural network structure specially designed to process and encode audio features. It is commonly used in tasks such as speech recognition, speaker recognition, and sentiment analysis to extract acoustic features and convert them into vectors that can effectively represent input audio data. A typical Acoustic Encoder network structure may include the following parts: Preprocessing layer: Feature extraction: Extract Mel frequency cepstral coefficients (MFCCs), filter bank features (such as FBANKs), or other acoustic features from the original audio signal. Frame-level processing: Segment the audio signal into frames and perform feature extraction on each frame. Convolutional neural network (CNN) layer: Use CNN to process acoustic features, which can capture time-frequency features and local patterns. It usually contains multiple convolutional layers, each of which may be followed by a pooling layer to reduce the size of the feature map. Activation functions such as ReLU are used to increase the nonlinearity of the network. Recurrent neural network (RNN) layer: In order to capture the sequence characteristics of the audio signal, RNN layers such as LSTM (Long Short-Term Memory Network) or GRU (Gated Recurrent Unit) are usually added. RNN can handle variable-length inputs and is suitable for time series data such as audio. Fully connected layer: At the final stage of the network, a fully connected layer may be added for the final feature combination and mapping. These fully connected layers can be used to extract higher-level feature representations or to generate the final output (such as classification or sequence labeling tasks). Output layer: Depending on the specific task, the output layer may be a softmax layer for generating probability distributions, or a linear layer that maps directly to the required output space. When building the Acoustic Encoder, the size of the network structure, the number of layers, and the number of neurons in each layer are adjusted according to the task requirements and data characteristics. In addition, regularization techniques (such as Dropout, weight decay, etc.) and optimization algorithms (such as Adam, RMSprop, etc.) are also used to improve the generalization ability and training efficiency of the model.

[0101] As an example, see Figure 4 , performing feature extraction on the second Mel-level speech through the encoding model of the second encoding parameter to obtain the second label feature.

[0102] In some embodiments, the preset coding parameters are predefined model parameters, such as neural network weights in deep learning, which are obtained by training on a large amount of data, and each preset coding parameter may correspond to a different model architecture or training configuration, such as a different number of layers, number of neurons, activation function, regularization parameter, etc. Querying a first coding parameter adapted to the first Mel voice from a plurality of preset coding parameters, and querying a second coding parameter adapted to the second Mel voice from a plurality of preset coding parameters, involves selecting or training a model that can select or generate the most appropriate coding parameter according to the minimum distortion or maximum similarity of the input Mel voice signal, and the first Mel voice and the second Mel voice are respectively feature extracted through their adapted coding models.

[0103] In some embodiments, the second mel-voice and the second mel-voice are respectively subjected to feature extraction through their adapted coding models. For the second mel-voice, this process includes: using the coding model corresponding to the queried second coding parameter. The second mel-voice is input into the model, and the feature extraction capability of the model is used to obtain the feature representation of the speech, that is, the second label feature.

[0104] In step 105, the speech conversion model is trained based on the first label feature, the second label feature and the predicted speech feature.

[0105] In some embodiments, the above step 105 can be implemented as follows: perform feature concatenation on the first label feature and the second label feature to obtain a third label feature; determine the feature similarity between the third label feature and the predicted speech feature, determine the feature similarity as a loss value, and update the model parameters of the speech conversion model based on the loss value.

[0106] As an example, see Figure 4 , concatenate the first label feature and the second label feature to obtain a third label feature; determine the feature similarity between the third label feature and the predicted speech feature, determine the feature similarity as a loss value, and update the model parameters of the speech conversion model based on the loss value.

[0107] As an example, in a speech recognition task, the label features can be divided into different categories. For example, the first label feature can be the Mel-frequency cepstral coefficients (MFCCs) of the audio signal, the second label feature can be the transcribed text of the phoneme or word, and the third label feature can be the acoustic feature of the speech. The following are the specific steps of feature concatenation and model update: Feature extraction: For the input speech signal, first use the Mel-frequency cepstral coefficient (MFCC) extractor to extract the first label feature, i.e., MFCCs. Then, use the transcribed text information of the phoneme or word to extract the second label feature in some way (such as a sequence labeling model). Then, use an acoustic model (such as an acoustic model based on a deep neural network) to predict the acoustic features of the speech as the third label feature. Feature concatenation: These three label features are concatenated to obtain a comprehensive feature vector, i.e., the third label feature. Concatenation can be a simple concatenation or a more complex feature fusion method, such as using an attention mechanism to dynamically weight different features. Feature similarity calculation: Next, use the third label feature and the predicted speech feature to calculate the feature similarity between them. This is usually done by calculating the Euclidean distance, cosine similarity, or other metrics between the features. The calculated feature similarity is used as the loss value, and the larger the loss value, the greater the difference between the predicted speech features and the third label features. Model update: Use the loss value to update the model parameters of the speech conversion model. This is usually done through a gradient descent algorithm, where the gradient of the loss value with respect to the model parameters is calculated, and then these gradients are used to adjust the model parameters so that the model can more accurately capture the characteristics of the speech in future predictions. Iterative optimization: After multiple iterations, the model will continue to learn and improve its ability to predict speech features, thereby improving the accuracy of speech recognition. This process is a typical training process for a speech recognition system based on deep learning. In this process, the goal of the model is to reduce the difference between the predicted speech features and the actual third label features, that is, to minimize the loss value. In this way, the model can learn how to better convert the input speech signal into the corresponding text information.

[0108] In this way, the first label feature and the second label feature are feature spliced ​​to obtain the third label feature. The beneficial effect is that different types of label feature information can be comprehensively utilized to improve the model's understanding and characterization capabilities of speech signals. By determining the feature similarity between the third label feature and the predicted speech feature, and using the similarity as the loss value, it helps to accurately evaluate the gap between the model prediction and the actual speech feature. The model parameters of the speech conversion model are updated based on the loss value, so that the model can continuously learn and optimize, thereby improving the accuracy and robustness of speech recognition. In addition, this method also helps to improve the generalization ability of the model, so that it has better performance when dealing with different scenarios and speech signals. In short, through feature splicing and model updating, the overall performance of the speech recognition system can be effectively improved, providing users with a more intelligent and accurate voice interaction experience.

[0109] In some embodiments, after the above step 105, speech conversion can also be performed in the following manner: using a trained speech conversion model, performing speech feature conversion on the text to be converted to obtain predicted speech features of the text to be converted; performing speech conversion on the predicted speech features of the text to be converted to obtain converted speech of the text to be converted; the converted speech has the same semantics as the text to be converted, and the converted speech satisfies the acoustic features of the Mel speech.

[0110] As an example, see Figure 7 , Figure 7 This is a schematic diagram of the principle of speech conversion provided in an embodiment of the present application. Through the trained speech conversion model, speech feature conversion is performed on the text to be converted to obtain the predicted speech features of the text to be converted; speech conversion is performed on the predicted speech features of the text to be converted to obtain the converted speech of the text to be converted; the converted speech has the same semantics as the text to be converted, and the converted speech satisfies the acoustic features of the Mel speech.

[0111] As an example, training a speech conversion model: First, there is a speech conversion model that has been trained with a large amount of data. This model learns how to convert text information into speech features. In this process, the model may use structures such as recurrent neural networks (RNN) or transformers to learn the mapping from text to speech. Conversion of speech features of the text to be converted: When a piece of text to be converted is received, it is input into the trained speech conversion model. The model will convert the text into corresponding predicted speech features based on what it has learned before. These predicted speech features are an intermediate representation that represents the acoustic features that the model believes the text corresponds to. Speech conversion: Next, these predicted speech features are used to generate the actual speech signal. This usually involves another model, such as an acoustic model, which uses these features to synthesize a speech waveform. In this process, the model will take into account the acoustic features of the target sample object to ensure that the generated speech not only sounds like the corresponding text, but also meets certain acoustic properties, such as a certain timbre, intonation, or speaking speed. Ensure semantic identity and meet acoustic features: Finally, the generated converted speech should have the same semantic content as the original text, that is, the converted speech should be able to convey the same meaning and information. At the same time, the converted speech should also meet the acoustic characteristics of the sample object, which means that it should have the sound quality and style of the target speech. The result of this process is a speech output that is both faithful to the original text content and in line with the style of the target speech. Such a system can be applied to a variety of scenarios, such as voice assistants, automatic telephone systems, or text-to-speech conversion. By continuously training and optimizing the model, the quality of the converted speech can be improved, making it more natural and fluent, while better adapting to different acoustic feature requirements.

[0112] In this way, the trained speech conversion model is used to convert the speech features of the text to be converted. The beneficial effect is that the text information can be accurately mapped to the predicted speech features, and then the converted speech that meets the specific acoustic features can be generated. This processing process not only ensures that the semantics of the converted speech is the same as the original text, ensuring the accurate transmission of information, but also meets the acoustic feature requirements of the sample object, so that the speech output is consistent with the target speech style in terms of sound quality, intonation and speaking speed. This helps to improve the naturalness and fluency of speech synthesis and enhance the user experience. At the same time, this method also helps to expand the application areas of speech technology, such as voice assistants, automatic speech recognition and text-to-speech conversion, providing users with more intelligent and personalized voice interaction services.

[0113] In some embodiments, the above-mentioned voice conversion of the predicted voice features of the text to be converted to obtain the converted voice of the text to be converted can be achieved in the following ways: decoding the predicted voice features of the text to be converted through a decoding model to obtain the Mel voice of the text to be converted; and voice conversion of the Mel voice of the text to be converted through a vocoder to obtain the converted voice of the text to be converted.

[0114] As an example, see Figure 7 , through the decoding model, the predicted speech features of the text to be converted are decoded to obtain the Mel speech of the text to be converted; through the vocoder, the Mel speech of the text to be converted is voice-converted to obtain the converted speech of the text to be converted.

[0115] As an example, in a speech synthesis system, the decoding model and the vocoder are two key components that work together to convert text into speech. Decoding model: The role of the decoding model (such as a neural network decoder) is to decode the predicted speech features of the text to be converted. In this process, the model will use the learned linguistic and acoustic knowledge to convert these features into a higher-level speech representation. For example, suppose we have a text sequence "Hello, how are you?", and we have obtained the corresponding predicted speech features through the speech conversion model. The decoding model will further process these features to generate the corresponding Mel-spectrogram, which is an acoustic feature map representing the speech signal. Vocoder: The vocoder is a model that converts the Mel-spectrogram into the actual speech signal. It is usually a neural network-based model that can convert the Mel-spectrogram into a waveform in the time domain. Continuing with the above example, the Mel-spectrogram generated by the decoding model will be input into the vocoder. The vocoder will use the information in the Mel-spectrogram, such as frequency, amplitude, etc., to synthesize the corresponding speech waveform. This process is equivalent to converting acoustic features into audible speech. Generation of converted speech: Finally, the speech waveform generated by the vocoder is the converted speech of the text to be converted. This speech not only sounds consistent with the text content, but also has the acoustic characteristics determined by the Mel voice. Through this process, the text content can be converted into a speech that sounds natural and meets specific acoustic requirements. This technology is very useful in many applications, such as voice assistants, automatic telephone systems, text-to-speech conversion, etc. By continuously training and optimizing the decoding model and vocoder, we can improve the quality of speech synthesis, making it more natural and fluent, while meeting different acoustic feature requirements.

[0116] In this way, the predicted speech features of the text to be converted are decoded by the decoding model, which has the beneficial effect of converting text information into Mel speech. This process helps to preserve the acoustic characteristics of speech and improve the quality of speech synthesis. By converting Mel speech into speech through the vocoder, a converted speech that meets the target acoustic characteristics can be generated, making the output speech more natural and fluent in terms of sound quality, intonation and speaking speed. This processing method not only ensures that the converted speech is semantically the same as the original text, but also meets the acoustic feature requirements of the sample object, thereby improving the overall performance of speech synthesis. At the same time, this method also helps to expand the application field of speech technology and provide users with a more intelligent and personalized voice interaction experience.

[0117] In some embodiments, the above-mentioned trained speech conversion model performs speech feature conversion on the text to be converted, and before obtaining the predicted speech features of the text to be converted, the following processing can be performed: in response to a question sentence sent by the question consulting party, a response prediction is performed on the question sentence to obtain an answer sentence to the question sentence, and the answer sentence is determined as the text to be converted.

[0118] As an example, in an intelligent voice interaction system, when receiving a question from a user, the system will first process the question and predict an answer. This process can be achieved through a trained answer prediction model. Question reception and processing: The user asks a question through voice input or text input. Answer prediction: The system uses the trained answer prediction model to process the text to be converted. This model analyzes the semantics of the question and predicts a possible answer. For example, if the user asks "What's the weather like today?", the answer prediction model may predict the answer "It's sunny today." Determine the text to be converted: The predicted answer sentence is determined as the text to be converted, that is, "It's sunny today." Speech feature conversion: Then, the system uses the speech conversion model to convert the speech features of the text to be converted "It's sunny today" to obtain the corresponding predicted speech features. Speech synthesis: Finally, the system uses the vocoder to convert these predicted speech features into actual speech output to answer the user's question. Through this processing flow, the system can first predict and answer the user's question, and then output the answer in the form of speech, providing the user with a smooth voice interaction experience. This technology is very useful in applications such as intelligent assistants and customer service robots, and can greatly improve user satisfaction and interaction experience.

[0119] In some embodiments, after performing speech conversion on the predicted speech features of the text to be converted to obtain the converted speech of the text to be converted, the following processing may be performed: sending the converted speech of the text to be converted to the question consulting party.

[0120] In this way, by introducing the answer prediction step before the voice conversion model, the beneficial effect is that the intelligence and response speed of the voice interaction system can be improved. First, the system can answer users' questions instantly and improve the user experience. For example, when a user asks "How is the weather today?", the system can quickly predict and answer "It's sunny today." Secondly, using the predicted answer sentence as the text to be converted helps to improve the accuracy of subsequent speech synthesis and ensure the consistency between the speech output and the answer content. During the voice feature conversion and voice conversion process, the system will process the answer sentence according to the semantic and acoustic features to generate high-quality voice responses. Finally, sending the converted voice of the text to be converted to the user can achieve effective information transmission and meet the user's needs for obtaining answers. This makes the voice interaction system more intelligent and efficient, and can better meet the user's voice query needs.

[0121] In this way, the speech feature conversion is performed on the text sample through the speech conversion model to obtain the predicted speech feature; the first mel speech and the second mel speech are obtained by splitting the mel speech, the semantics of the mel speech is the same as the semantics of the text sample, the first mel speech is used to reflect the acoustic features of the sample object, and the second mel speech is used to reflect the semantics of the mel speech; the first mel speech is feature extracted to obtain the first label feature, and the second mel speech is feature extracted to obtain the second label feature; based on the first label feature, the second label feature and the predicted speech feature, the speech conversion model is trained, so that the first label feature and the second label feature can respectively reflect the acoustic features of the sample object and the semantics of the text sample, so that the acoustic features and semantics contained in the mel speech are split to refine the modeling, so that the speech conversion model fully learns the acoustic features and semantics contained in the mel speech, which can effectively improve the speech feature generation performance of the speech conversion model.

[0122] The following is an explanation of an exemplary application of the embodiment of the present application in an actual image editing application scenario.

[0123] Large-model TTS speech synthesis The TTS model has attracted more and more attention from industry and academia. Compared with traditional speech synthesis models, such as FastSpeech, Vits and other popular speech synthesis models in the industry, the naturalness, emotion and rhythm of the synthesized speech of the large-model TTS speech synthesis model are close to those of real people.

[0124] There are two forms of speech synthesis: one is end-to-end, that is, text is directly converted into speech waveforms, and the other is text to mel, and then converted into a speech waveform through a vocoder. The popular large speech models such as chattts and tortoise-tts all convert mel into vq code, and then use a GPT-like NLP large model to train the integration of text, vq code, and speaker embedding. However, since mel information is rich in information, vq encoder is very critical. As long as the encoder has a strong ability to learn features, the performance of the codebook will be very good, and the pressure on the decoder will be much less.

[0125] In some embodiments, the encoding and decoding model provided in the embodiment of the present application (that is, the encoding and decoding model described above) includes three parts: Encoder, VQ, and Decoder. The role of the encoder is to learn the useful information in the target real mel and obtain an intermediate feature (continuous vector feature). The role of the VQ is to convert this intermediate feature into a discretized feature and a hidden intermediate feature (continuous vector feature). The hidden intermediate feature and the discretized feature can be converted to each other without the need for another network module. The role of the decoder is to restore the hidden intermediate feature to mel, that is, predict mel. The smaller the gap between it and the target mel, the better the performance of the vq network structure. The learning ability of the encoder determines the performance of the subsequent modules, so in the vq network structure, the learning ability of the encoder is very critical. The vq discretized feature is also used in the speech synthesis TTS large model. The Mel feature (that is, the Mel speech feature described above) is rich in various information of speech, such as acoustic information and semantic information. The acoustic information mainly includes the speaker's timbre and rhythmic features, and the semantic information mainly includes the text semantics in the speech, that is, what the speech said. These two types of information together make up the mel feature, which is enough to show the complexity of the mel feature and that it contains a lot of information. VQ is the foundation of the current large-scale speech synthesis model and is also the most critical link. Among them, VQEncoder is the top priority. Its effect directly determines the ability of the large-scale speech synthesis model, that is, it will affect the performance of sentence cloning and universal timbre, which is usually manifested as a sentence clone that is not similar enough, with poor sound quality and rhythm. Considering the complexity of the mel feature information, we separate the acoustic information and semantic information in the mel feature and model them separately, divide and conquer, and thus improve the ability of VQ Encoder to learn features.

[0126] In some embodiments, see Figure 5The dimension of mel feature is [length, 80]. In the field of speech synthesis, audio is generally processed into mel feature with dimension [length, 80]. Of course, some models process mel feature into [length, 100] (in very rare cases) (Why 20-80 is semantic information and 0-20 is acoustic information? This is the most basic knowledge in the field of speech. The dimension 20-80 contains a small amount of acoustic information, but at most it is semantic information, that is, the content of the audio. The acoustic features of the dimension 0-20 are mainly rhythm and timbre information. That is, the audio of this dimension can be heard as soon as it is heard by a man or a woman, and the degree of rhythm fluctuation. The dimension 20-80 is input into the ASR information, and its content can be accurately identified). The first 20 dimensions of information are mainly acoustic information, and the 20 to 80 dimensions of information are mainly semantic information. The acoustic information [batch_size, length, 20] is passed through the acoustics encoder module to learn richer acoustic features (rhythm, timbre, etc.), and the output dimension is [batch_size, length, dim1]. The semantic information [batch_size, length, 60] is passed through the semantics encoder module to learn richer semantic features (emotion, sentence meaning), and the dimension is [batch_size, length, dim2]. Finally, the features learned by the two are concatenated, and the dimension is [batch_size, length, dim1+dim2]. This operation is a mathematical process and is input to VQ and Decoder to participate in training and learning.

[0127] In some embodiments, the semantics encoder network structure (vq code 2) includes an N-layer transformer encoder structure. The powerful semantic learning ability of transformer encoder has been widely used in speech and text fields, such as the Bert pre-training model and FastSpeech2 model in the NLP field.

[0128] In some embodiments, the acoustics encoder network structure is mainly composed of Conv1D (one-dimensional convolution, extracting local features). The parameter kernel size value setting of Conv1D can focus on the information within a certain range of the current position, and enhance the acquisition of local information (that is, the associated information of adjacent frames). From top to bottom, the kernel size of the first layer Conv1D is 5, which can obtain information far away from the current position, and another function is dimensionality change. The kernel sizes of subsequent Conv1D are different, and they all obtain information within different ranges of the current position to enhance the ability to obtain local information. The main function of Linear is dimensionality transformation, Layer Norm normalizes the data to improve stability during training, ReLU is an activation function, and enhances the ability of the network module. The last layer of ConvlD transforms the dimension into the dimension desired by the training model, and also has the ability to obtain local information. The acoustics encoder module focuses more on rhythmic features. The rhythmic expression of the current word in a sentence is often closely related to the words around it, that is, it is related to the words within a certain range (kernel_size setting), so the design of the acoustics encoder network structure is mainly based on Conv1D.

[0129] In some embodiments, the embodiments of the present application are mainly to improve the ability of vq encoder to learn features, thereby improving the sound quality and rhythmic naturalness of the generated audio. The mel feature contains a lot of information, and it is difficult to learn with a single model. Separating the modeling is conducive to improving the ability to learn features, which is the principle of divide and conquer.

[0130] In some embodiments, VQ training is a self-supervised method, which inputs target mel and outputs predicted mel, and performs loss calculation iteratively. The smaller the loss, the better the performance of VQ. The input of the model only needs mel features, and the mel feature dimension [batch_size, length, 80] is split into (code splitting, because the dimension is fixed, [batch_size, length,: 20] and [batch_size, length, 20:] can be separated) [batch_size, length, 20]a1 and [batch_size, length, 60]a2, a1 is input to the acoustic encoder module to learn acoustic features, and the output dimension is [batch_size, length, dim1]b1, a2 is input to the semantics encoder to learn semantic features, and the output dimension is [batch_size, length, dim2]b2, then b1 and b2 (the splicing dimension is on the third dimension) are spliced ​​together to obtain the final feature dimension of [batch_size, length, dim1+dim2], which is input into subsequent modules to participate in training and learning, output predict mel, and calculate the loss with target mel, multiple rounds of iterations, wait for the loss to converge, and save the model.

[0131] In some embodiments, the saved model is loaded, the input is the real mel, and the vq code discretization feature is obtained (mel feature is a continuous feature, including semantic information and acoustic information. The mel feature is compressed into a discrete feature by VQ. For example, a discrete feature is a word list. The corresponding word can be found through the id in the word list. Conversely, the corresponding id can be found through the word. This id is a discrete feature, which is spliced ​​with the text and speaker embedding in a certain form (the network structure proposed in this patent is to make the extracted vq code richer in information. This is the training GPT module. This is not the focus of this patent. It is just a brief explanation of the subsequent application of this patent. Semantic information and acoustic information will be used in training VQ, but GPT training does not require it. It only needs to extract vq code, and what is obtained after the semantic information and acoustic information are processed has been explained above, and it is at the VQ Encoder end. Train the GPT model, the input feature is a splicing feature, such as [speaker embedding, text code, vq code], and the dimension uses the second dimension as the splicing dimension), and input it to a large text model such as GPT to participate in training. See Figure 7 The trained model can be applied to general timbre synthesis and sentence cloning synthesis.

[0132] In some embodiments, when the algorithm is applied, the vq model with good effect can be directly applied to the training of the subsequent large speech synthesis model, and the trained model can be applied to telemarketing outbound calls and customer service collection. Through the improvement of the vq encoder network model, the ability of the large speech synthesis model can be effectively improved, that is, the sound quality and rhythmic naturalness of the generated audio. When applying telemarketing outbound calls, the speaker embedding should be a speaker vector feature that is friendly, has high affinity, or has excellent performance. This speaker may be in the training data set or not (sentence cloning). If the speaker takes leave or resigns (the right to use his own voice must be sought), audio can be generated. The VQ Encoder network structure proposed in this application has a voice similarity and rhythmic naturalness that is very similar to that of the speaker, achieving a fake-real effect.

[0133] The embodiment of the present application improves the learning ability of the vq encoder (that is, the encoding model described above) features, which helps to improve the effects of the TTS large model (sound quality and rhythmic information), such as the effect of cloning a sentence and the effect of generating audio with a universal timbre. Since the mel feature is rich in multiple information, it is difficult for a single network structure to fully learn all of its information. The acoustic information and semantic information contained therein are separated and modeled separately to achieve the purpose of divide and conquer, thereby improving the ability of the VQ Encoder to learn features, and indirectly improving the sound quality and rhythmic naturalness of the generated audio, achieving a fake-real effect. In practical applications, a VQ model is first trained, and then the VQ Encoder and vq modules are used to extract the vq code discrete features, and then the text and speaker embedding are combined to train the speech synthesis TTS large model. When in use, inputting text and speaker embedding can synthesize speech with high similarity to the speaker embedding, with the ability of cloning a sentence timbre and universal timbre, which can be used in telemarketing outbound calls or collection scenarios. First, the voice stream signal transmitted in real time from the telephone user end is received through MRCP, and then the voice stream signal is input into the ASR speech recognition model to obtain the customer's speaking content. The customer's intention is obtained through the NLP model, and the content to be fed back is obtained. The feedback content is synthesized into speech through TTS and fed back to the customer through the phone. The customer responds and a round of interaction is completed, and this cycle repeats.

[0134] In this way, the sound quality and rhythmic naturalness of voice cloning and general voice synthesis of a sentence can be effectively improved. Only a few seconds of speech of a certain speaker are needed to synthesize the voice of the speaker, and the speaker's voice can be customized for outbound call scenarios such as telemarketing and debt collection. The key point is the acquisition of vq code, which directly affects the performance of the large speech synthesis model. The performance of vqcode mainly depends on the vq network structure, among which the most important thing is the ability of the vq encoder module to learn features. This patent proposes a vq encoder network model based on mel features. Since mel features are rich in information and complex, the acoustic information and semantic information in the mel features are separated and refined for modeling, which effectively improves the ability of the entire vq encoder module to obtain features, which in turn improves the performance of vq, and in turn improves the sound quality and rhythmic naturalness of the generated audio of the speech synthesis TTS large model, one ring after another.

[0135] It is understandable that in the embodiments of the present application, related data such as text samples are involved. When the embodiments of the present application are applied to specific products or technologies, user permission or consent is required, and the collection, use and processing of relevant data need to comply with relevant laws, regulations and standards of relevant countries and regions.

[0136] The following is a description of an exemplary structure of a speech conversion model training device 455 provided in an embodiment of the present application implemented as a software module. In some embodiments, for example, Figure 2 As shown, the software modules in the training device 455 of the speech conversion model stored in the memory 450 may include: a feature conversion module 4551, which is used to perform speech feature conversion on the text sample through the speech conversion model to obtain predicted speech features; a splitting module 4552, which is used to split the Mel speech with the same semantics as the text sample to obtain a first Mel speech and a second Mel speech, wherein the first Mel speech is used to reflect the acoustic features of the Mel speech, and the second Mel speech is used to reflect the semantics of the Mel speech; a feature extraction module 4553, which is used to perform feature extraction on the first Mel speech to obtain a first label feature, and perform feature extraction on the second Mel speech to obtain a second label feature; a training module 4554, which is used to train the speech conversion model based on the first label feature, the second label feature and the predicted speech feature.

[0137] In some embodiments, the feature extraction module is further used to query a first encoding parameter adapted to the first Mel-speech from a plurality of preset encoding parameters, and perform feature extraction on the first Mel-speech through the encoding model of the first encoding parameter to obtain the first label feature; the feature extraction module is further used to query a second encoding parameter adapted to the second Mel-speech from a plurality of preset encoding parameters, and perform feature extraction on the second Mel-speech through the encoding model of the second encoding parameter to obtain the second label feature.

[0138] In some embodiments, the above-mentioned coding model belongs to a codec model, and the above-mentioned training module is also used to perform complete speech prediction on a speech sample carrying a speech label through the codec model to obtain the predicted speech of the speech sample, and the speech label includes the speech sample; based on the predicted speech and the speech label, the codec model is trained.

[0139] In some embodiments, the above-mentioned speech sample includes multiple different sub-speech samples, and the predicted speech includes sub-predicted speech corresponding to each of the sub-speech samples; the above-mentioned training module is also used to determine the speech similarity between each of the sub-predicted speech and the speech label, and perform preset operations on the speech similarities corresponding to each of the sub-predicted speech to obtain the loss value of the codec model; based on the loss value of the codec model, the model parameters of the codec model are updated.

[0140] In some embodiments, the above-mentioned training module is also used to split the speech sample to obtain a first speech sample and a second speech sample; perform feature extraction on the first speech sample through the first coding model to obtain a first speech feature, and perform feature extraction on the second speech sample through the second coding model to obtain a second speech feature; perform feature splicing on the first speech feature and the second speech feature to obtain a third speech feature; and perform complete speech prediction on the speech sample based on the third speech feature through the coding and decoding model to obtain the predicted speech.

[0141] In some embodiments, the above-mentioned training module is also used to perform feature splicing on the first label feature and the second label feature to obtain a third label feature; determine the feature similarity between the third label feature and the predicted speech feature, determine the feature similarity as a loss value, and update the model parameters of the speech conversion model based on the loss value.

[0142] In some embodiments, the training device for the above-mentioned speech conversion model also includes: a conversion module, which is used to perform speech feature conversion on the text to be converted through the trained speech conversion model to obtain the predicted speech features of the text to be converted; perform speech conversion on the predicted speech features of the text to be converted to obtain the converted speech of the text to be converted; the converted speech has the same semantics as the text to be converted, and the converted speech satisfies the acoustic characteristics of the Mel speech.

[0143] In some embodiments, the above-mentioned conversion module is also used to decode the predicted speech features of the text to be converted through a decoding model to obtain the Mel speech of the text to be converted; and perform speech conversion on the Mel speech of the text to be converted through a vocoder to obtain the converted speech of the text to be converted.

[0144] In some embodiments, the above-mentioned conversion module is also used to respond to a question sentence sent by the question consulting party, predict an answer to the question sentence, obtain an answer sentence to the question sentence, and determine the answer sentence as the text to be converted; and send the converted speech of the text to be converted to the question consulting party.

[0145] In some embodiments, the above-mentioned splitting module is also used to determine the sub-speech in the Mel-speech within the first frequency range corresponding to the acoustic feature as the first Mel-speech; and to determine the sub-speech in the Mel-speech within the second frequency range corresponding to the semantics as the second Mel-speech.

[0146] The embodiment of the present application provides a computer program product, which includes a computer program or a computer executable instruction, and the computer program or the computer executable instruction is stored in a computer-readable storage medium. The processor of the electronic device reads the computer executable instruction from the computer-readable storage medium, and the processor executes the computer executable instruction, so that the electronic device executes the training method of the speech conversion model described in the embodiment of the present application.

[0147] The present application embodiment provides a computer-readable storage medium storing computer-executable instructions, wherein the computer-executable instructions are stored. When the computer-executable instructions are executed by a processor, the processor will execute the training method of the speech conversion model provided in the present application embodiment, for example, Figure 3 The training method of the speech conversion model is shown.

[0148] In some embodiments, the computer-readable storage medium may be a memory such as FRAM, ROM, PROM, EPROM, EEPROM, flash memory, magnetic surface storage, optical disk, or CD-ROM; or it may be various electronic devices including one or any combination of the above memories.

[0149] In some embodiments, computer executable instructions may be in the form of a program, software, software module, script or code, written in any form of programming language (including compiled or interpreted languages, or declarative or procedural languages), and may be deployed in any form, including as a stand-alone program or as a module, component, subroutine or other unit suitable for use in a computing environment.

[0150] As an example, computer-executable instructions may, but need not, correspond to a file in a file system, may be stored as part of a file storing other programs or data, such as in one or more scripts in a HyperText Markup Language (HTML) document, in a single file dedicated to the program in question, or in multiple coordinated files (e.g., files storing one or more modules, subroutines, or code portions).

[0151] As an example, computer executable instructions may be deployed to be executed on one electronic device, or on multiple electronic devices located at one site, or on multiple electronic devices distributed at multiple sites and interconnected by a communication network.

[0152] In summary, the embodiments of the present application have the following beneficial effects:

[0153] (1) The speech feature conversion of the text sample is performed through the speech conversion model to obtain the predicted speech feature; the first mel speech and the second mel speech are obtained by splitting the mel speech with the same semantics as the text sample, the first mel speech is used to reflect the acoustic features of the sample object, and the second mel speech is used to reflect the semantics of the mel speech; the first mel speech is feature extracted to obtain the first label feature, and the second mel speech is feature extracted to obtain the second label feature; based on the first label feature, the second label feature and the predicted speech feature, the speech conversion model is trained so that the first label feature and the second label feature can respectively reflect the acoustic features of the sample object and the semantics of the text sample, so that the acoustic features and semantics contained in the mel speech are split to refine the modeling, so that the speech conversion model fully learns the acoustic features and semantics contained in the mel speech, which can effectively improve the speech feature generation performance of the speech conversion model.

[0154] (2) By accurately defining the first frequency range corresponding to the acoustic features, the sub-speech in the mel-speech within the range is effectively screened out and determined as the first mel-speech, which helps to improve the accuracy and efficiency of speech recognition. At the same time, according to the second frequency range corresponding to the semantics, the sub-speech in the mel-speech within the range is further determined as the second mel-speech, which can enhance the understanding and analysis of the meaning of the speech, thereby achieving more accurate speech recognition and processing, and improving the performance of the overall system and user experience.

[0155] (3) By utilizing the above-mentioned codec model, when predicting the complete speech of a speech sample with a speech tag, the model can effectively infer the complete speech content even if the speech sample is only at least part of the speech tag. For example, given the speech tag "I went out to play tonight," when the input speech sample is only "I went out to play today...", the model can intelligently predict and complete the remaining speech and output "I went out to play today." Such processing brings significant beneficial effects. It greatly improves the robustness and practicality of the speech recognition system, allowing the system to handle the user's possible incomplete or ambiguous speech input, thereby providing a smoother and more natural interactive experience. This is especially important for applications such as voice assistants, real-time speech translation, and automatic subtitle generation, because it ensures that in these scenarios, the system can give accurate and complete output even in the face of imperfect input.

[0156] (4) This method brings significant beneficial effects by determining the speech similarity between each sub-predicted speech and the corresponding speech label, and performing preset operations on these similarities to calculate the loss value of the codec model. It not only accurately quantifies the difference between the model prediction results and the actual speech labels, but also provides direct guidance for the updating of model parameters. Updating model parameters based on loss values ​​can effectively optimize model performance, making it more accurate and reliable when recognizing speech samples. This refined training strategy not only speeds up the convergence of the model, but also improves the model's generalization ability to different language environments and speaker variants, thereby achieving higher recognition accuracy and lower error rates in speech recognition applications.

[0157] (5) The method of splitting the speech sample, extracting features through specific encoding models, and then splicing the features has brought many beneficial effects. It allows the model to analyze different parts of the speech sample more carefully and capture richer local feature information, thereby improving the accuracy of feature representation. Through splitting and feature extraction, the model can process long speech samples more effectively, reduce computational complexity and memory requirements, and improve the efficiency of training and reasoning. In addition, feature splicing not only integrates the speech information of different parts, but also enhances the model's understanding of the overall structure of the speech, allowing the encoding and decoding model to more accurately predict the complete speech content. Ultimately, this processing strategy significantly improves the performance of the speech recognition system, achieves higher recognition accuracy and lower error rate, and provides users with a more accurate and smooth voice interaction experience.

[0158] (6) The first label feature and the second label feature are concatenated to obtain a third label feature. The beneficial effect is that different types of label feature information can be comprehensively utilized to improve the model's ability to understand and characterize speech signals. By determining the feature similarity between the third label feature and the predicted speech feature, and using the similarity as the loss value, it helps to accurately evaluate the gap between the model prediction and the actual speech feature. The model parameters of the speech conversion model are updated based on the loss value, so that the model can continuously learn and optimize, thereby improving the accuracy and robustness of speech recognition. In addition, this method also helps to improve the generalization ability of the model, so that it has better performance when dealing with different scenarios and speech signals. In short, through feature concatenation and model updating, the overall performance of the speech recognition system can be effectively improved, providing users with a more intelligent and accurate voice interaction experience.

[0159] (7) The trained speech conversion model is used to convert the speech features of the text to be converted. The beneficial effect is that the text information can be accurately mapped to the predicted speech features, and then the converted speech with specific acoustic features can be generated. This processing process not only ensures that the semantics of the converted speech is the same as the original text, ensuring the accurate transmission of information, but also meets the acoustic feature requirements of the sample object, so that the speech output is consistent with the target speech style in terms of sound quality, intonation and speaking speed. This helps to improve the naturalness and fluency of speech synthesis and enhance the user experience. At the same time, this method also helps to expand the application areas of speech technology, such as voice assistants, automatic speech recognition and text-to-speech conversion, providing users with more intelligent and personalized voice interaction services.

[0160] (8) The predicted speech features of the text to be converted are decoded by the decoding model. The beneficial effect is that the text information can be converted into Mel speech. This process helps to preserve the acoustic characteristics of speech and improve the quality of speech synthesis. By converting the Mel speech into speech through the vocoder, a converted speech that meets the target acoustic characteristics can be generated, making the output speech more natural and fluent in terms of sound quality, intonation and speaking speed. This processing method not only ensures that the converted speech is semantically the same as the original text, but also meets the acoustic feature requirements of the sample object, thereby improving the overall performance of speech synthesis. At the same time, this method also helps to expand the application field of speech technology and provide users with a more intelligent and personalized voice interaction experience.

[0161] (9) By introducing the answer prediction step before the speech conversion model, the beneficial effect is that it can improve the intelligence and response speed of the speech interaction system. First, the system can answer users' questions instantly and improve the user experience. For example, when a user asks "How is the weather today?", the system can quickly predict and answer "It's sunny today." Secondly, using the predicted answer sentence as the text to be converted helps to improve the accuracy of subsequent speech synthesis and ensure the consistency between the speech output and the answer content. During the speech feature conversion and speech conversion process, the system will process the answer sentence according to the semantic and acoustic features to generate high-quality speech responses. Finally, sending the converted speech of the text to be converted to the user can achieve effective information transmission and meet the user's need to obtain answers. This makes the speech interaction system more intelligent and efficient, and can better meet the user's voice query needs.

[0162] (10) It can effectively improve the sound quality and rhythmic naturalness of voice cloning and general voice synthesis of a sentence. Only a few seconds of speech of a certain speaker is needed to synthesize the voice of the speaker, and the speaker's voice can be customized for outbound call scenarios such as telemarketing and debt collection. The key point is the acquisition of vq code, which directly affects the performance of the large speech synthesis model. The performance of vqcode mainly depends on the vq network structure, among which the most important is the ability of the vq encoder module to learn features. This patent proposes a vq encoder network model based on mel features. Since mel features are rich in information and complex, the acoustic information and semantic information in mel features are separated and refined for modeling, which effectively improves the ability of the entire vq encoder module to obtain features, which in turn improves the performance of vq, and in turn improves the sound quality and rhythmic naturalness of the generated audio of the speech synthesis TTS large model.

[0163] (11) The embodiment of the present application improves the learning ability of the vq encoder feature, which helps to improve the effect of the TTS large model (sound quality and rhythmic information), such as the effect of cloning a sentence and generating audio with a universal timbre. Since the mel feature is rich in multiple information, it is difficult for a single network structure to fully learn all of its information. The acoustic information and semantic information contained therein are separated and modeled separately to achieve the purpose of divide and conquer, thereby improving the ability of the VQ Encoder to learn features, and indirectly improving the sound quality and rhythmic naturalness of the generated audio, achieving a fake-real effect. In practical applications, a VQ model is first trained, and then the VQ Encoder and vq modules are used to extract the vq code discrete features, and then the text and speaker embedding are combined to train the speech synthesis TTS large model. When in use, the text and speaker embedding can be input to synthesize speech with a high similarity to the speaker embedding, which has the ability of cloning a sentence timbre and universal timbre, and can be used in telemarketing outbound calls or debt collection scenarios. First, the voice stream signal transmitted in real time from the telephone user end is received through MRCP, and then the voice stream signal is input into the ASR speech recognition model to obtain the customer's speaking content. The customer's intention is obtained through the NLP model, and the content to be fed back is obtained. The feedback content is synthesized into speech through TTS and fed back to the customer through the phone. The customer responds and a round of interaction is completed, and this cycle repeats.

[0164] The above is only an embodiment of the present application and is not intended to limit the protection scope of the present application. Any modifications, equivalent substitutions and improvements made within the spirit and scope of the present application are included in the protection scope of the present application.

Claims

1. A method for training a speech conversion model, characterized in that: The method comprises: The speech conversion model is used to convert the text sample into speech features to obtain predicted speech features; Splitting the Mel voice with the same semantics as the text sample to obtain a first Mel voice and a second Mel voice, wherein the first Mel voice is used to reflect the acoustic features of the Mel voice, and the second Mel voice is used to reflect the semantics of the Mel voice; Performing feature extraction on the first mel speech to obtain a first label feature, and performing feature extraction on the second mel speech to obtain a second label feature; The speech conversion model is trained based on the first label feature, the second label feature and the predicted speech feature.

2. The method according to claim 1, characterized in that The step of extracting features from the first mel speech to obtain a first label feature includes: Querying a first encoding parameter adapted to the first mel-speech from a plurality of preset encoding parameters, and performing feature extraction on the first mel-speech through an encoding model of the first encoding parameter to obtain the first label feature; The extracting features of the second mel speech to obtain a second label feature includes: A second encoding parameter adapted to the second mel-speech is searched from the plurality of preset encoding parameters, and a feature of the second mel-speech is extracted through an encoding model of the second encoding parameter to obtain the second label feature.

3. The method according to claim 2, characterized in that The coding model belongs to a coding and decoding model. Before extracting features from the first mel speech to obtain a first label feature, the method further includes: By using the coding and decoding model, a complete speech prediction is performed on a speech sample carrying a speech label to obtain a predicted speech of the speech sample, wherein the speech label includes the speech sample; The codec model is trained based on the predicted speech and the speech label.

4. The method according to claim 3, characterized in that The speech sample includes a plurality of different sub-speech samples, and the predicted speech includes sub-predicted speech corresponding to each of the sub-speech samples; The training of the codec model based on the predicted speech and the speech tag includes: Determine the speech similarity between each of the sub-predicted speech and the speech label, perform a preset operation on the speech similarity corresponding to each of the sub-predicted speech, and obtain a loss value of the encoding and decoding model; Based on the loss value of the codec model, model parameters of the codec model are updated.

5. The method according to claim 3, characterized in that: The method of performing complete speech prediction on the speech sample carrying the speech tag through the encoding and decoding model to obtain the predicted speech of the speech sample includes: Splitting the voice sample to obtain a first voice sample and a second voice sample; Extracting features from the first speech sample using the coding model to obtain a first speech feature, and extracting features from the second speech sample using the coding model to obtain a second speech feature; Performing feature concatenation on the first speech feature and the second speech feature to obtain a third speech feature; The complete speech prediction is performed on the speech sample based on the third speech feature by using the coding and decoding model to obtain the predicted speech.

6. The method according to claim 1, characterized in that The step of training the speech conversion model based on the first label feature, the second label feature and the predicted speech feature includes: Performing feature splicing on the first label feature and the second label feature to obtain a third label feature; Determine a feature similarity between the third label feature and the predicted speech feature, determine the feature similarity as a loss value, and update a model parameter of the speech conversion model based on the loss value.

7. The method according to claim 1, characterized in that After training the speech conversion model based on the first label feature, the second label feature and the predicted speech feature, the method further includes: Performing speech feature conversion on the text to be converted by using the trained speech conversion model to obtain predicted speech features of the text to be converted; Performing voice conversion on the predicted voice features of the text to be converted to obtain the converted voice of the text to be converted; The converted speech has the same semantics as the text to be converted, and the converted speech satisfies the acoustic features of the Mel speech.

8. The method according to claim 7, characterized in that The performing speech conversion on the predicted speech features of the text to be converted to obtain the converted speech of the text to be converted includes: Decoding the predicted speech features of the text to be converted by a decoding model to obtain the Mel speech of the text to be converted; The mel speech of the text to be converted is converted by a vocoder to obtain the converted speech of the text to be converted.

9. The method according to claim 7, characterized in that: Before performing speech feature conversion on the text to be converted by using the trained speech conversion model to obtain predicted speech features of the text to be converted, the method further includes: In response to a question sentence sent by the question consulting party, predicting an answer to the question sentence, obtaining an answer sentence to the question sentence, and determining the answer sentence as the text to be converted; After performing voice conversion on the predicted voice features of the text to be converted to obtain the converted voice of the text to be converted, the method further includes: The converted speech of the text to be converted is sent to the question-asking party.

10. The method according to claim 1, characterized in that The step of splitting the Mel speech having the same semantics as the text sample to obtain the first Mel speech and the second Mel speech comprises: According to the first frequency range corresponding to the acoustic feature, determining the sub-speech in the mel-level speech that is within the first frequency range as the first mel-level speech; According to the second frequency range corresponding to the semantics, the sub-speech in the mel-level speech that is within the second frequency range is determined as the second mel-level speech.

11. A training device for a speech conversion model, characterized in that: The device comprises: A feature conversion module, used to perform speech feature conversion on the text sample through the speech conversion model to obtain predicted speech features; A splitting module, used for splitting the Mel speech with the same semantics as the text sample to obtain a first Mel speech and a second Mel speech, wherein the first Mel speech is used to reflect the acoustic features of the Mel speech, and the second Mel speech is used to reflect the semantics of the Mel speech; A feature extraction module, configured to perform feature extraction on the first mel speech to obtain a first label feature, and perform feature extraction on the second mel speech to obtain a second label feature; A training module is used to train the speech conversion model based on the first label feature, the second label feature and the predicted speech feature.

12. An electronic device, characterized in that: The electronic device comprises: A memory for storing computer executable instructions or computer programs; A processor, configured to implement the method for training a speech conversion model according to any one of claims 1 to 10 when executing computer executable instructions or computer programs stored in the memory.

13. A computer-readable storage medium storing computer-executable instructions or a computer program, characterized in that: When the computer executable instructions or computer program are executed by a processor, the training method of the speech conversion model described in any one of claims 1 to 10 is implemented.

14. A computer program product comprising computer executable instructions or a computer program, characterized in that: When the computer program or computer executable instructions are executed by a processor, the training method of the speech conversion model according to any one of claims 1 to 10 is implemented.

Images