Speech generation method and apparatus, computer device, and storage medium

Abstract

Embodiments of the present application provide a speech generation method and device, computer equipment and a storage medium, belonging to the technical field of artificial intelligence. The method comprises: extracting speech features from original speech data through an original speech conversion model to obtain reference speech features and selected speech features, the reference speech features having a target object voice style; performing similarity measurement on the reference speech features and the selected speech features to obtain feature similarity data; screening the selected speech features according to the feature similarity data to obtain target speech features; wherein the target speech features do not have the target object voice style; performing model training on an original speech generation model according to the target speech features, the reference speech features and training text to obtain a target speech generation model; and generating target speech data through the target speech generation model and target text. The embodiments of the present application can improve the stability of speech generation, reduce the dependence on reference audio, and improve the flexibility of speech generation.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of artificial intelligence, and in particular to a speech generation method and device, a computer device, and a storage medium. BACKGROUND

[0002] Speech generation technology, also known as text-to-speech technology, is a technology for converting text information into spoken language. In related technologies, a large speech model is used to generate text-to-speech. However, generating speech only through a large language model can easily mix multiple speaker voices, resulting in unstable generated speech. In order to control the speaker identity and style in the generated speech, a reference audio of a speaker needs to be provided for each speech generation, which not only increases the dependence on reference audio, but also limits the flexibility of speech generation. Therefore, how to improve the stability of generated speech, reduce the dependence on reference audio, and make speech generation more flexible has become a technical problem to be solved. SUMMARY

[0003] The main purpose of the embodiments of the present application is to provide a speech generation method and device, a computer device, and a storage medium, which aims to improve the stability of speech generation, reduce the dependence on reference audio, and improve the flexibility of speech generation.

[0004] To achieve the above-mentioned purpose, a first aspect of the embodiments of the present application provides a speech generation method, which comprises:

[0005] obtaining original speech data and training text;

[0006] performing speech feature extraction on the original speech data through a preset original speech conversion model to obtain a speech feature set; wherein the speech feature set comprises reference speech features and selected speech features, and the reference speech features have a target object voice style;

[0007] performing similarity measurement on the reference speech features and the selected speech features to obtain feature similarity data;

[0008] performing feature screening on the selected speech features according to the feature similarity data to obtain target speech features; wherein the target speech features do not have the target object voice style;

[0009] performing model training on a preset original speech generation model according to the target speech features, the reference speech features, and the training text to obtain a target speech generation model;

[0010] performing speech generation through the target speech generation model and target text to obtain target speech data.

[0011] In some embodiments, the original speech conversion model comprises a speech encoder, a first feature extractor and a second feature extractor; the speech feature extraction on the original speech data through the preset original speech conversion model comprises:

[0012] The speech encoder is used for feature extraction on the original speech data to obtain candidate speech features;

[0013] The first feature extractor is used for speech feature extraction on the candidate speech features to obtain the reference speech features;

[0014] The second feature extractor and the reference speech features are used for speech feature extraction on the candidate speech features to obtain the selected speech features; wherein the weights and values of the reference speech features and the selected speech features are 1.

[0015] In some embodiments, the feature screening on the selected speech features according to the feature similarity data comprises:

[0016] The selected speech features are subjected to feature quantization processing according to preset quantization features to obtain preliminary speech features;

[0017] The preliminary speech features are subjected to feature screening according to the feature similarity data to obtain the target speech features.

[0018] In some embodiments, the feature quantization processing on the selected speech features according to preset quantization features to obtain preliminary speech features comprises:

[0019] The preset quantization features and the selected speech features are subjected to feature distance calculation to obtain speech feature distances;

[0020] The selected speech features are screened out from the preset quantization features according to the speech feature distances;

[0021] The selected speech features are subjected to splicing processing to obtain the preliminary speech features.

[0022] In some embodiments, after the feature screening on the selected speech features according to the feature similarity data to obtain the target speech features, the method further comprises:

[0023] The target speech features and the reference speech features are subjected to splicing processing to obtain spliced speech features;

[0024] The spliced speech features are used for speech reconstruction to obtain reconstructed speech data;

[0025] perform loss calculation on the reconstructed speech data and the original speech data to obtain reconstructed loss data;

[0026] perform loss calculation on the preset similarity threshold and the feature similarity data to obtain similarity loss data;

[0027] perform object recognition according to the reference speech feature to obtain object recognition information;

[0028] perform loss calculation on the preset target object information and the object recognition information to obtain recognition loss data;

[0029] perform model parameter adjustment on the original speech conversion model according to the reconstructed loss data, the similarity loss data, and the recognition loss data to obtain a target speech conversion model.

[0030] In some embodiments, the original speech generation model comprises an original text encoder and an original decoder; and the model training on the preset original speech generation model according to the target speech feature, the reference speech feature, and the training text to obtain a target speech generation model comprises:

[0031] perform semantic extraction on the training text through the original text encoder to obtain text semantic information;

[0032] perform speech feature conversion on the text semantic information through the original decoder and the reference speech feature to obtain predicted speech feature; wherein the predicted speech feature does not have a target object sound style;

[0033] perform model parameter adjustment on the original speech generation model according to the predicted speech feature and the target speech feature to obtain the target speech generation model.

[0034] In some embodiments, the original text encoder comprises a word segmentation layer, a semantic parsing layer, and an encoding layer; and the semantic extraction on the training text through the original text encoder to obtain text semantic information comprises:

[0035] perform word segmentation processing on the training text through the word segmentation layer to obtain training word groups;

[0036] perform semantic parsing on the training word groups through the semantic parsing layer to obtain semantic parsing information;

[0037] perform semantic extraction on the training text through the encoding layer and the semantic parsing information to obtain the text semantic information.

[0038] To achieve the above object, a second aspect of the embodiment of the present application proposes a speech generation device, which comprises:

[0039] an acquisition module configured to acquire original speech data and training text;

[0040] a speech feature extraction module configured to perform speech feature extraction on the original speech data by using a preset original speech conversion model to obtain a speech feature set, wherein the speech feature set includes reference speech features and selected speech features, and the reference speech features have a target object voice style;

[0041] a similarity measurement module configured to perform similarity measurement on the reference speech features and the selected speech features to obtain feature similarity data;

[0042] a feature screening module configured to perform feature screening on the selected speech features according to the feature similarity data to obtain target speech features, wherein the target speech features do not have the target object voice style;

[0043] a model training module configured to perform model training on a preset original speech generation model according to the target speech features, the reference speech features, and the training text to obtain a target speech generation model;

[0044] a speech generation module configured to perform speech generation by using the target speech generation model and the target text to obtain target speech data.

[0045] To achieve the above object, a third aspect of embodiments of the present application provides a computer device, which comprises a memory and a processor, the memory stores a computer program, and the processor implements the method of the first aspect when executing the computer program.

[0046] To achieve the above object, a fourth aspect of embodiments of the present application provides a computer readable storage medium, which stores a computer program, and the computer program is executed by a processor to implement the method of the first aspect.

[0047] The speech generation method, apparatus, computer device, and storage medium proposed in this application extract reference speech features and selected speech features from the original speech data using an original speech conversion model. The reference speech features are speech features with the voice style of the target object. In this embodiment, to reduce the influence of non-target object voice styles in speech generation, speech features without the target object voice style are extracted. Therefore, by calculating the feature similarity data between the reference speech features and the selected speech features, target speech features without the target object voice style are extracted from the selected speech features based on the feature similarity data. Then, the original speech generation model is trained by combining the target speech features, the reference speech features, and the training text to obtain the target speech generation model. The speech data output by the target speech generation model only contains the voice style of the target object, and each speech generation does not require the use of reference audio, making speech generation more stable and flexible. Attached Figure Description

[0048] Figure 1 This is a flowchart of the speech generation method provided in the embodiments of this application;

[0049] Figure 2 yes Figure 1 The flowchart of step S102 in the document;

[0050] Figure 3 yes Figure 1 The flowchart of step S104 in the process;

[0051] Figure 4 yes Figure 3 The flowchart of step S301 in the process;

[0052] Figure 5 This is a flowchart of a speech generation method provided in another embodiment of this application;

[0053] Figure 6 yes Figure 1 The flowchart of step S105 in the process;

[0054] Figure 7 yes Figure 6 The flowchart of step S601 in the process;

[0055] Figure 8 This is a system framework diagram of the speech generation method provided in the embodiments of this application;

[0056] Figure 9 This is a schematic diagram of the structure of the speech generation device provided in the embodiments of this application;

[0057] Figure 10 This is a schematic diagram of the hardware structure of the computer device provided in the embodiments of this application. Detailed Implementation

[0058] In order to make the purposes, technical solutions and advantages of the present application clearer, further detailed description will be made to the present application in combination with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application, and are not used to limit the present application.

[0059] It should be noted that although the functional modules are divided in the device schematic diagram, and the logical sequence is shown in the flowchart, in some cases, the steps shown or described can be performed in a manner different from the module division in the device or the sequence in the flowchart. The terms "first", "second", etc. in the specification and claims and the above-described drawings are used to distinguish similar objects, and do not necessarily describe a specific order or sequence.

[0060] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which the present application belongs. The terms used herein are only for the purpose of describing the embodiments of the present application, and are not intended to limit the present application.

[0061] First, the meanings of several terms involved in the present application are analyzed:

[0062] Artificial intelligence (AI): is a new technical science that studies, develops theories, methods, technologies and application systems for simulating, extending and expanding human intelligence; artificial intelligence is a branch of computer science, and artificial intelligence aims to understand the essence of intelligence and produce a new intelligent machine that can react in a similar way to human intelligence. The research in this field includes robots, language recognition, image recognition, natural language processing and expert systems, etc. Artificial intelligence can simulate the information process of human consciousness and thinking. Artificial intelligence is also the theory, method, technology and application system of using digital computers or digital computer controlled machines to simulate, extend and expand human intelligence, to perceive the environment, acquire knowledge and use knowledge to obtain the best results.

[0063] Self-supervised speech conversion model (SSVC): is a model that uses self-supervised learning techniques to process speech data. Self-supervised learning is a powerful unsupervised learning method that does not rely on labeled data, but learns the representation of data by predicting some attributes of the data itself. In the field of speech processing, this method is particularly useful because it can learn from unlabeled speech data, thereby reducing the dependence on large amounts of labeled data.

[0064] Large Language Model (LLM): A significant branch of artificial intelligence that uses deep learning and extensive data training to understand and generate natural language text. LLMs have a wide range of applications, including but not limited to chatbots, text generation, language translation, content creation, information retrieval, document classification, and sentiment analysis.

[0065] Waveform Language Model (WavLM): A general-purpose speech pre-training model that uses the Denoising Masked Speech Modeling framework for self-supervised pre-training. This framework combines masked speech prediction and denoising, allowing the model to maintain its ability to model speech content while improving the potential for non-automatic speech recognition (ASR) tasks through speech denoising.

[0066] Softmax Function: A normalization function commonly used in machine learning, especially when dealing with multi-class classification problems. The Softmax function converts a real-valued vector into a probability distribution, so that each element in the vector is between 0 and 1, and the sum of all elements is 1.

[0067] Transformer Model: A deep learning model architecture widely used in natural language processing (NLP) and other sequence-to-sequence tasks. The Transformer model introduces the self-attention mechanism, a key innovation that makes it excel at processing sequential data.

[0068] CLS (Classification) Embedding: A special token commonly used in text classification tasks, usually placed at the beginning of the input sequence. In the BERT model, the embedding vector of the CLS token, after being processed by multiple layers of self-attention mechanism, can integrate the information of the entire input sequence, forming a representation of the entire sentence. This representation vector can be used for sentence-level classification tasks.

[0069] VQ (Vector Quantization) Layer: A technique used in deep learning for data compression and dimensionality reduction, which maps input data to a smaller fixed codebook, thereby achieving discrete representation of data. This method is particularly suitable for handling high-dimensional data such as images or audio signals, and can significantly reduce storage requirements and computational complexity.

[0070] BigVGAN Decoder: A general-purpose neural vocoder designed to synthesize audio waveforms from Mel spectrograms, using a fully convolutional architecture that includes multiple up-sampling blocks and residual dilated convolution layers.

[0071] The speech generation technology is a technology for converting text information into speech data, and there are multiple speech generation systems based on large language models in the field of artificial intelligence, such as TorToiSe, VALL-E, VALL-EX, and the like. It should be noted that these speech generation systems realize text-to-speech generation by predicting the quantized tokens of the EnCodec, HiFi-Codec or the discrete representation of the VQ-VAE. However, in the related art, when the speech generation system generates speech data, it will incorporate multiple speaker voices, resulting in unstable generated speech, affecting the listening experience and intelligibility. To overcome the problem of unstable speech generation, in the related art, a reference audio needs to be input every time speech generation is performed, but the setting of the reference audio affects the flexibility of the speech generation system, and some speech generation systems also need to convert the reference audio into transcribed text, affecting the effect of speech generation.

[0072] Therefore, the speech generation method and device, the computer device and the storage medium provided in the embodiments of the present application can extract target speech features without the sound style of the target object and reference speech features with the sound style of the target object from the original speech data through an original speech conversion model before speech generation, train an original speech generation model according to the target speech features, the reference speech features and training text, and train a target speech generation model that can accurately and stably output speech data with the sound style of the target object. Therefore, the target speech data corresponding to the target text is generated by the target speech generation model, which not only generates stable speech data, but also no longer depends on the reference audio for each speech generation, reduces the dependence on the reference audio and transcription, and improves the flexibility of speech generation.

[0073] The speech generation method and device, the computer device and the storage medium provided in the embodiments of the present application are specifically described by the following embodiments. First, the speech generation method in the embodiments of the present application is described.

[0074] The embodiments of the present application can acquire and process related data based on artificial intelligence technology. Artificial intelligence (AI) is a theory, method, technology and application system for simulating, extending and expanding human intelligence using a digital computer or a machine controlled by a digital computer, perceiving the environment, acquiring knowledge and using the knowledge to obtain the best results.

[0075] The artificial intelligence basic technology generally includes technologies such as sensors, special artificial intelligence chips, cloud computing, distributed storage, big data processing technology, operation / interaction system, mechatronics, etc. The artificial intelligence software technology mainly includes computer vision technology, robot technology, biometric identification technology, speech processing technology, natural language processing technology, and machine learning / deep learning, etc.

[0076] The voice generation method provided by the embodiments of the present application relates to the technical field of artificial intelligence. The voice generation method provided by the embodiments of the present application can be applied to a terminal, can also be applied to a server end, and can also be software running in the terminal or the server end. In some embodiments, the terminal can be a smart phone, a tablet computer, a notebook computer, a desktop computer, etc.; the server end can be configured as an independent physical server, can also be configured as a server cluster or a distributed system composed of multiple physical servers, can also be configured as a cloud server providing basic cloud computing services such as cloud service, cloud database, cloud computing, cloud function, cloud storage, network service, cloud communication, middleware service, domain name service, security service, CDN, and big data and artificial intelligence platform; and the software can be an application for implementing the voice generation method, but is not limited to the above forms.

[0077] The present application can be used in many general or special computer system environments or configurations. For example: personal computers, server computers, handheld devices or portable devices, tablet devices, multi-processor systems, microprocessor-based systems, set-top boxes, programmable consumer computer devices, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, etc. The present application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The present application can also be practiced in a distributed computing environment in which tasks are performed by remote processing devices connected by a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media, including storage devices.

[0078] It should be noted that in various specific embodiments of the present application, when relevant processing needs to be performed on data related to the identity or characteristics of the user, such as user information, user behavior data, user history data, and user location information, the user's permission or consent will be obtained first, and the collection, use, and processing of such data will comply with relevant laws, regulations, and standards. In addition, when the embodiments of the present application need to obtain sensitive personal information of the user, the separate permission or separate consent of the user will be obtained through a pop-up window or a jump to a confirmation page, and after obtaining the separate permission or separate consent of the user, the necessary user-related data for enabling the embodiments of the present application to function normally will be obtained.

[0079] Figure 1 is an optional flowchart of the voice generation method provided by the embodiments of the present application, Figure 1 The method in the method can include but is not limited to steps S101-S106.

[0080] Step S101, obtaining original voice data and training text;

[0081] Step S102, performing voice feature extraction on the original voice data through a preset original voice conversion model to obtain a voice feature set; wherein the voice feature set includes reference voice features and selected voice features, and the reference voice features have a target object voice style;

[0082] Step S103, performing similarity measurement on the reference voice features and the selected voice features to obtain feature similarity data;

[0083] Step S104, performing feature screening on the selected voice features according to the feature similarity data to obtain target voice features; wherein the target voice features do not have the target object voice style;

[0084] Step S105, performing model training on a preset original voice generation model according to the target voice features, the reference voice features, and the training text to obtain a target voice generation model;

[0085] Step S106, performing voice generation through the target voice generation model and target text to obtain target voice data.

[0086] The steps S101 to S106 shown in the embodiments of the present application extract reference speech features and selected speech features from the original speech data through the original speech conversion model, and the reference speech features are speech features with the speaking style of the target object. Then, the selected speech features are screened to obtain target speech features based on the feature similarity between the reference speech features and the selected speech features, so as to obtain sound features without the sound style of the target object. Then, the original speech generation model is trained based on the reference speech features, the target speech features and the training text, so as to construct a target speech generation model that can stably and purely generate speech with sound style. Therefore, the speech data generated by the target speech generation model only has the sound style of the target object, improves the stability of the speech data generation, and reduces the dependence on the reference audio, so that the speech generation is more flexible.

[0087] In step S101 of some embodiments, the original speech data is speech data containing the sound style of the target object, and the original speech data can be obtained in real time or extracted from the training speech database constructed in advance. The present embodiment does not make specific limitations on the acquisition method of the original speech data. The training text is a text used to train the original speech generation model, so as to determine whether the semantic content of the speech data generated by the original speech generation model conforms to the training text. The training text can be generated in real time or extracted from the pre-constructed text database, and the present embodiment also does not make specific limitations on the acquisition method of the training text.

[0088] For example, if the application scenario is an insurance business scenario in the financial field, speech data with multiple sound styles of insurance salespersons is constructed in advance, and the training text is also a text related to insurance business knowledge, so as to train the original speech data and the training text to accurately and stably output insurance-related speech data conforming to the sound style of the insurance salesperson. Specifically, it can be applied to intelligent customers in insurance business question and answer to provide customers with more stable question and answer speech data in sound style.

[0089] As disclosed above, after obtaining the original speech data, the original speech data is input into the original speech conversion model. The original speech conversion model includes a speech encoder, a first feature extractor and a second feature extractor. The speech encoder is used for encoding processing of the original speech data, the first feature extractor is used for extracting speech features with the sound style of the target object from the encoded speech features, and the second feature extractor is used for extracting speech features other than the sound style of the target object from the encoded speech features.

[0090] Specifically, the embodiment adopts a pre-trained WavLM model as a speech encoder, the WavLM model is trained by predicting the masked speech part, and further includes a denoising process, which can extend the effectiveness of the WavLM model from a speech recognition task to a non-content recognition speech task. Therefore, by encoding the original speech data through the WavLM model, speech features of different object sound styles can be extracted.

[0091] Please refer to Figure 2 In some embodiments, step S102 can include but is not limited to steps S201 to S203:

[0092] Step S201, feature extraction of original speech data through a speech encoder to obtain candidate speech features;

[0093] Step S202, speech feature extraction of the candidate speech features through a first feature extractor to obtain reference speech features;

[0094] Step S203, speech feature extraction of the candidate speech features through a second feature extractor and the reference speech features to obtain selected speech features; wherein the weights and values of the reference speech features and the selected speech features are 1.

[0095] In step S201 of some embodiments, as previously disclosed, the speech encoder of the embodiment is a pre-trained WavLM model, which extracts the hidden state of the original speech data through the WavLM model. The WavLM model includes multiple WavLM layers, and each WavLM layer outputs a hidden state representing different feature representations.

[0096] In step S202 of some embodiments, the first feature extractor is also called a speaker extractor, and the candidate speech features include speech features with a target object sound style. The first feature extractor is used to extract speech features with a target object sound style. Specifically, the first feature extractor weights the candidate speech features output by each WavLM layer, then inputs the weighted speech features into a normalization layer, and maps the weighted speech features through the normalization layer to the target object sound style to obtain the reference speech features. It should be noted that the normalization layer of the embodiment selects a softmax function, and other embodiments can select a decimal scaling normalization function, a logarithmic normalization function, and a power law normalization function, etc. The function of the normalization layer is not specifically limited in the embodiment.

[0097] In step S203 of some embodiments, as previously disclosed, the reference speech features are weighted to the candidate speech features and input to the normalization layer, so when the selected speech features are extracted, the feature extraction needs to be combined with the reference speech features to extract the speech features without the target object sound style. Specifically, the embodiments determine the target object speech weight according to the reference speech features, and then constrain 1 minus the target object speech weight to obtain the non-object speech weight. The second feature extractor weights the candidate speech features output by each WavLM layer according to the non-object speech weight, and inputs them to the normalization layer for normalization processing to obtain the selected speech features.

[0098] In steps S201 to S203 shown in the embodiments, the candidate speech features are extracted from the original speech data by the speech encoder, and the reference speech features with the target object sound style are extracted from the candidate speech features by the first feature extractor in a weighted manner. Finally, the non-object weighting weight is determined based on the reference speech features by the second feature extractor, and the selected speech features are obtained by weighting the candidate speech features with the non-object weighting weight, so that the reference speech features and the selected speech feature extraction operation is simple.

[0099] In step S103 of some embodiments, as previously disclosed, although the selected speech features are screened from the candidate speech features according to the reference speech features, in order to ensure that the selected speech features do not have the target object sound style, the selected speech features need to be screened to screen the target speech features without the target object sound style. Specifically, the feature similarity data is obtained by measuring the similarity between the reference speech features and the selected speech features, and the feature similarity data can be cosine similarity. Other embodiments of the feature similarity data are Pearson correlation coefficient, Euclidean distance, and log-likelihood similarity, etc. The embodiments do not make specific restrictions on the feature similarity data.

[0100] Please refer to Figure 3 In some embodiments, step S104 can include but is not limited to steps S301 to S302:

[0101] Step S301, performing feature quantization processing on the selected speech features according to the preset quantization features to obtain preliminary speech features;

[0102] Step S302, performing feature screening on the preliminary speech features according to the feature similarity data to obtain the target speech features.

[0103] In step S301 of some embodiments, the selected speech feature is first quantized, and the quantization is a process of converting a continuous numerical signal into a discrete numerical signal. It should be noted that the preset quantization feature is a vector representation of a specified numerical value, and the selected speech feature is quantized by the preset quantization feature, that is, the selected speech feature is converted into a vector representation of a fixed numerical value as the preset quantization feature, to obtain a target speech feature.

[0104] In step S302 of some embodiments, the preliminary speech feature is selected according to the feature similarity data, that is, the preliminary speech feature with the smallest feature similarity data is selected as the target speech feature. Alternatively, in this embodiment, the preliminary speech feature is adjusted according to the feature similarity data to eliminate the speech feature with the target object sound style and leave the speech feature without the target object sound style as the target speech feature.

[0105] In steps S301 to S302 of this embodiment, the selected speech feature is first converted into a preliminary speech feature with a fixed numerical value according to the preset quantization feature, and then the preliminary speech feature is adjusted to a target speech feature without the target object sound style according to the feature similarity data, so that the speech feature with the target object sound style and the speech feature without the target object sound style can be known by the target speech feature and the reference speech feature respectively, and the target speech generation model containing only the target object sound style can be trained conveniently.

[0106] Please refer to Figure 4 In some embodiments, step S301 can include but is not limited to steps S401 to S403:

[0107] Step S401, the feature distance between the preset quantization feature and the selected speech feature is calculated to obtain a speech feature distance;

[0108] Step S402, the selected speech feature is selected from the preset quantization feature according to the speech feature distance;

[0109] Step S403, the selected speech feature is spliced to obtain a preliminary speech feature.

[0110] It should be noted that the VQ layer is used in this embodiment to quantize the selected speech feature, and the input selected speech feature is mapped to a fixed codebook by the VQ layer to realize the discrete representation of data. Among them, the VQ layer mainly performs the following steps: encoding, quantization and decoding, wherein the quantization operation is to find the closest vector to the selected speech feature in the codebook, and use the closest vector to replace the original selected speech feature.

[0111] In step S401 of some embodiments, as disclosed previously, the quantization operation needs to find the closest vector in the codebook, so the vectors stored in the codebook in this embodiment are defined as preset quantization features, and the speech feature distance is obtained by calculating the feature distance between each selected speech feature and each preset quantization feature. It should be noted that the distance calculation between the selected speech feature and the preset quantization feature can use the Euclidean distance calculation or the cosine distance calculation, and the embodiment does not make specific restrictions on the distance calculation between the selected speech feature and the preset quantization feature.

[0112] In steps S402 to S403 of some embodiments, the minimum feature distance is selected from the speech feature distance, and the preset quantization feature corresponding to the minimum feature distance is selected as the selected speech feature. The selected speech feature replaces the selected speech feature, and then the selected speech feature is spliced as the preliminary speech feature.

[0113] For example, the preset codebook is 【1, 2, 3, 4, 5, 6, 7, 8, 9】, if the selected speech feature is 1.1, then the closest preset quantization feature of the selected speech feature is 1, and 1 is taken as the preliminary speech feature; if the selected speech feature is 8.7, then the closest preset quantization feature of the selected speech feature is 9, and 9 is taken as the preliminary speech feature. It should be noted that the role of quantizing the selected speech feature in this embodiment is to remove the local fluctuation noise in the selected speech feature. For example, if the representation of the speech "ah" should be stable (1, 1, 1, 1), rather than with timbre fluctuation noise (1.1, 0.9, 1.2, 0.8), so the selected speech feature is quantized to obtain the preliminary speech feature with more stable timbre.

[0114] In steps S401 to S403 of this embodiment, the closest preset quantization feature is taken as the selected speech feature of the selected speech feature, so that the selected speech feature is replaced by the selected speech feature, the quantization operation of the selected speech feature is realized, the local fluctuation noise in the selected speech feature is eliminated, and the preliminary speech feature with more stable timbre is obtained.

[0115] It should be noted that when the target speech feature is selected, in order to improve the accuracy of subsequent speech feature extraction, the original speech conversion model needs to be optimized to build a target speech conversion model that can accurately extract speech features with a target object sound style.

[0116] In some embodiments, after step S104, the speech generation method can further include adjusting the parameters of the original speech conversion model to obtain a target speech conversion model, optimizing the original speech conversion model, and building a target speech conversion model that can accurately extract speech features with a target object sound style.

[0117] It should be noted that the original speech conversion model is a self-supervised speech conversion model, specifically an SSVC model, which learns the high-dimensional features of the input speech by predicting the masked or altered parts, can capture the acoustic features of the speech, and map them to a compact vector space. Therefore, the context information can be learned through the SSVC model without the need for reference audio.

[0118] Please refer to Figure 5 In some embodiments, the parameter adjustment of the original speech conversion model to obtain the target speech conversion model can include but is not limited to steps S501 to S507:

[0119] Step S501, the target speech features and the reference speech features are spliced to obtain spliced speech features;

[0120] Step S502, speech reconstruction is performed according to the spliced speech features to obtain reconstructed speech data;

[0121] Step S503, loss calculation is performed on the reconstructed speech data and the original speech data to obtain reconstructed loss data;

[0122] Step S504, loss calculation is performed on the pre-set similarity threshold and the feature similarity data to obtain similarity loss data;

[0123] Step S505, object recognition is performed according to the reference speech features to obtain object recognition information;

[0124] Step S506, loss calculation is performed according to the pre-set target object information and the object recognition information to obtain recognition loss data;

[0125] Step S507, the original speech conversion model is adjusted according to the reconstructed loss data, the similarity loss data and the recognition loss data to obtain the target speech conversion model.

[0126] It should be noted that in order to judge the feature extraction effect of the original speech conversion model, it can be judged from multiple aspects, mainly from whether the speech features input by the original speech conversion model can accurately realize speaker recognition, speech reconstruction and feature similarity calculation, etc.

[0127] In steps S501-S502 of some embodiments, in order to judge the feature extraction effect of the original speech conversion model, the reconstruction effect of the speech features output by the original speech conversion model can be analyzed. The target speech features and the reference speech features are spliced into spliced speech features, and then the spliced speech features are reconstructed to obtain reconstructed speech data. Specifically, the spliced speech features are input into the BigVGAN vocoder, and the BigVGAN vocoder is used to reconstruct the spliced speech features to obtain the reconstructed speech data. The BigVGAN vocoder is used to synthesize audio waveforms from Mel spectrograms, and can output reconstructed speech data with better audio quality.

[0128] In step S503 of some embodiments, the loss between the reconstructed speech data and the original speech data is calculated to obtain reconstruction loss data, and the reconstruction loss data indicates whether the features extracted by the original speech conversion model can accurately represent the audio features of the original speech data. Specifically, the loss function is used to calculate the loss data between the original speech data and the reconstructed speech data, and the loss function can be a cross-entropy loss function, a mean absolute error loss function, a mean square error loss function, a log-likelihood loss function, etc. The loss function is not limited in the present embodiment.

[0129] In step S504 of some embodiments, the feature similarity data represents the similarity between the reference speech features and the selected speech features. In order to judge the accuracy of the similarity calculation, it is judged whether the original speech conversion model can accurately extract speech features with the target object voice style and without the target object voice style. The loss data between the similarity threshold and the feature similarity data is calculated as similarity loss data, and the accuracy of the feature similarity calculation is judged by the similarity loss data. Specifically, the loss function used to calculate the similarity loss data can be any one of a cross-entropy loss function, a mean absolute error loss function, a mean square error loss function, and a log-likelihood loss function. The loss function used to calculate the similarity loss data is not limited in the present embodiment.

[0130] In steps S505-S506 of some embodiments, in order to judge the speech feature extraction effect of the original speech conversion model, whether the reference speech features output by the original speech conversion model can accurately identify the target object can be judged. Specifically, the original speech conversion model extracts reference speech features without the sound style of the target object from the original speech data, and the original speech data can be set to two, taking two original speech data from different speaking objects as negative sample data, and taking two original speech data from the same speaking object as positive sample data. The reference speech features of the positive sample data are extracted by the original speech conversion model, and the object recognition information of the reference speech features is identified, and the two object recognition information is respectively loss calculated with the two target object information to obtain first loss data; at the same time, the original speech conversion model extracts the reference speech features of the negative sample data, and loss calculates the object recognition information of the reference speech features and the target object information to obtain second loss data, and splices the first loss data and the second loss data into identification loss data. It should be noted that the identification loss data represents the object recognition accuracy of the reference speech features, and also reflects the effect of the original speech conversion model on speech feature extraction. Specifically, the loss function for calculating the identification loss data can be any one of a cross-entropy loss function, a mean absolute error loss function, a mean square error loss function, and a log-likelihood loss function, and the present embodiment does not make specific restrictions on the loss function used for calculating the identification loss data.

[0131] In step S507 of some embodiments, the target loss data is obtained by splicing the reconstruction loss data, the similarity loss data, and the identification loss data, and the splicing method can be weighted summation, summation, and average calculation, etc. The present embodiment does not make specific restrictions on the splicing method of the target loss data. It should be noted that the original speech conversion model is adjusted according to the target loss data, and the target loss data is continuously updated until the target loss data converges, and the original speech conversion model after parameter adjustment is used as the target speech conversion model.

[0132] In steps S501-S507 shown in the present embodiment, the reference speech features output by the original speech conversion model are analyzed for reconstruction effect, evaluated for feature similarity, and judged for object recognition accuracy, so as to adjust the parameters of the original speech conversion model from three aspects, and construct a target speech conversion model with better speech feature extraction effect.

[0133] In some embodiments, the original speech generation model is a large language model, and the large language model is an LLM model, which can realize speech generation by learning the grammar, semantics and context information of the language through training a large amount of text. The original speech generation model includes an original text encoder and an original decoder. The original text encoder is a BERT model, and the original decoder is a BigVGAN decoder.

[0134] Referring to Figure 6 In some embodiments, step S105 includes but is not limited to steps S601 to S603.

[0135] In step S601, the original text encoder is used to extract the semantic information of the training text.

[0136] In step S602, the original decoder and the reference speech feature are used to convert the text semantic information into predicted speech features. The predicted speech features do not have the target object sound style.

[0137] In step S603, the original speech generation model is adjusted according to the predicted speech features and the target speech features to obtain a target speech generation model.

[0138] In step S601 of some embodiments, as previously disclosed, the original text encoder is a BERT model, which has a bidirectional encoding feature. When processing each word, it will consider the context information before and after it to obtain text semantic information containing the context information of the training text.

[0139] In step S602 of some embodiments, in order to verify whether the speech data generated by the original speech generation model is stable and has the target object sound style, the original speech generation model is first used to generate predicted speech features that do not have the target object sound style, and then the original speech generation model is trained based on the predicted speech features and the target speech features. Specifically, the reference speech features and the text semantic information are input into the original decoder together, and the original decoder is used to convert the reference speech features and the text semantic information into reference speech data with the target object sound style. The reference speech data is feature extracted according to the reference speech features to extract predicted speech features that do not have the target object sound style, so that the predicted speech feature acquisition operation is simple.

[0140] In step S603 of some embodiments, speech loss data is obtained by loss calculation of the predicted speech features and the target speech features, and the original speech generation model is adjusted according to the speech loss data. The original speech generation model is adjusted in a loop until the speech loss data converges, so as to output the target speech generation model.

[0141] In steps S601-S603 shown in the embodiment, the text semantic information in the training text is extracted by the original text encoder, the text semantic information is converted into speech features by the original decoder and the reference speech features to obtain predicted speech features without the sound style of the target object, the speech loss data between the predicted speech features and the target speech features is calculated, and the model parameters of the original speech generation model are adjusted according to the speech loss data to construct a target speech generation model that can generate stable speech data with the sound style of the target object.

[0142] In some embodiments, the original text encoder includes a word segmentation layer, a semantic analysis layer, and an encoding layer. It should be noted that the word segmentation layer is used for word segmentation processing of the text, the semantic analyzer is used for text analysis, and the encoding layer is used to extract text semantic information.

[0143] Please refer to Figure 7 In some embodiments, step S601 can include but is not limited to steps S701-S703:

[0144] Step S701, performing word segmentation processing on the training text by the word segmentation layer to obtain training word groups;

[0145] Step S702, performing semantic analysis on the training word groups by the semantic analysis layer to obtain semantic analysis information;

[0146] Step S703, performing semantic extraction on the training text by the encoding layer and the semantic analysis information to obtain text semantic information.

[0147] In step S701 of some embodiments, the training text is input to the word segmentation layer for word segmentation processing, and the specific word segmentation is BPE word segmentation, which is a word segmentation technique used in natural language processing for processing rare words and unknown words. Therefore, by performing BPE word segmentation on the training text, accurate word segmentation of the training text is achieved to obtain training word groups.

[0148] In step S702 of some embodiments, the training word groups are input to the semantic analysis layer, and the semantic analysis layer performs semantic analysis on the training word groups to obtain semantic analysis information. It should be noted that semantic analysis is to convert the training word groups into structured data or representations that can be understood and processed, so the semantic analysis information represents the structured data or representations of the training word groups.

[0149] In step S703 of some embodiments, the encoder extracts text semantic information from the training text in combination with the semantic parsing information to obtain text semantic information accurately representing context information of the training text. Specifically, the embodiment extracts high-level embedding information from the training text through an encoding layer, and uses a learnable embedding layer to obtain low-level embedding information in the training text, and concatenates the high-level embedding information and the low-level embedding information into the text semantic information.

[0150] In steps S701 to S703 shown in the embodiment, the text semantic information of the training text is extracted based on the parsed semantic parsing information after the training text is segmented and parsed, so that the text semantic information extraction operation is simple.

[0151] In step S106 of some embodiments, after the target speech generation model is constructed, the target text is segmented into target word groups by the target speech generation model, the target word groups are parsed to obtain word group parsing information, and the target semantic information is extracted from the target text by the target speech generation model and the word group parsing information, so that the target speech data is generated according to the target semantic information. Therefore, by training the target speech generation model capable of generating target speech with the sound style of the target object, the target speech data generated by the target speech generation model can have the sound style of the target object without being input to the reference audio every time, which not only improves the stability of the speech data generation, but also improves the flexibility of the speech data generation.

[0152] Please refer to Figure 8The embodiment of the present application obtains original speech data and training text, inputs the original speech data into a WavLM model, extracts candidate speech features from the original speech data through the WavLM model, and then inputs the candidate speech features into a first feature extractor and a second feature extractor. The WavLM model includes a plurality of WavLM layers, the first feature extractor weights the candidate speech features output by each WavLM layer to obtain reference speech features, the second feature extractor determines a non-object speech weight based on the reference speech features, and weights the candidate speech features output by each WavLM layer according to the non-object speech weight to obtain selected speech features. Although the selected speech features are output by weighting the non-object speech weight, it does not exclude that the selected speech features still have speech features of the target object sound style. Therefore, it is necessary to further screen the selected speech features to screen out target speech features without the target object sound style. Specifically, the speech feature distance between the preset quantization features and the selected speech features is calculated, the preset quantization feature closest to the speech feature distance is selected as the selected speech feature, and the selected speech feature is spliced into a preliminary speech feature. The similarity between the selected speech feature and the reference speech feature is calculated to obtain feature similarity data, and then the preliminary speech feature is adjusted according to the feature similarity data to reduce the speech feature of the target object sound style in the preliminary speech feature, and the target speech feature root is obtained.

[0153] When the target speech feature is generated, the target speech feature, the reference speech feature and the training text are input into an original speech generation model. The original speech generation model is an LLM model, and the LLM model includes an original text encoder and an original decoder, and the original text encoder includes a word segmentation layer, a semantic analysis layer and an encoding layer. The training text is segmented into training word groups through the word segmentation layer, the semantic analysis layer analyzes the semantic analysis information of the training word groups, the text semantic information is extracted from the training text through the encoding layer and the semantic analysis information, and the text semantic information is converted into a predicted speech feature through the original decoder and the reference speech feature. The predicted speech feature is a speech feature without the target object sound style. The loss data between the predicted speech feature and the target speech feature is calculated to obtain speech loss data, the model parameter of the original speech generation model is adjusted according to the speech loss data to obtain a target speech generation model, so as to construct a target speech generation model that can stably generate speech data with a target object sound style.

[0154] When the speech data generation is performed, the target text is segmented by the target speech generation model to obtain a target word group, and the target word group is parsed to obtain word group parsing information. The target semantic information is extracted from the target text by the target speech generation model and the word group parsing information, so as to generate the target speech data according to the target semantic information. Therefore, the target speech data with the target object voice style is generated by the target speech generation model trained in advance, and the speech data with the non-object voice style is reduced, so that the speech generation is more clean and stable.

[0155] Please refer to Figure 9 The embodiment of the present application also provides a speech generation device, which can implement the speech generation method described above, and the device comprises:

[0156] The acquisition module 901 is configured to acquire original speech data and training text.

[0157] The speech feature extraction module 902 is configured to perform speech feature extraction on the original speech data by using a preset original speech conversion model to obtain a speech feature set, wherein the speech feature set comprises reference speech features and selected speech features, and the reference speech features have a target object voice style.

[0158] The similarity measurement module 903 is configured to measure the similarity between the reference speech features and the selected speech features to obtain feature similarity data.

[0159] The feature screening module 904 is configured to screen the selected speech features according to the feature similarity data to obtain target speech features, wherein the target speech features do not have the target object voice style.

[0160] The model training module 905 is configured to perform model training on a preset original speech generation model according to the target speech features, the reference speech features and the training text to obtain a target speech generation model.

[0161] The speech generation module 906 is configured to perform speech generation by using the target speech generation model and target text to obtain target speech data.

[0162] The speech generation specific embodiment is basically the same as the specific embodiment of the speech generation method described above, and will not be described here again.

[0163] The embodiment of the present application also provides a computer device, which comprises a memory and a processor, the memory stores a computer program, and the processor implements the speech generation method described above when executing the computer program. The computer device can be any intelligent terminal, such as a tablet computer, a vehicle-mounted computer, etc.

[0164] Please refer to Figure 10 , Figure 10The hardware structure of a computer device of another embodiment is illustrated, and the computer device comprises:

[0165] The processor 1001 can be implemented by a general-purpose CPU (Central Processing Unit), a microprocessor, an ASIC (Application Specific Integrated Circuit), or one or more integrated circuits, and is configured to execute related programs to implement the technical solutions provided by the embodiments of the present application.

[0166] The memory 1002 can be implemented by a ROM (ReadOnly Memory), a static storage device, a dynamic storage device, or a RAM (Random Access Memory), and the like. The memory 1002 can store an operating system and other application programs, and when the technical solutions provided by the embodiments of the present application are implemented by software or firmware, the related program codes are stored in the memory 1002 and are called and executed by the processor 1001 to implement the voice generation method of the embodiments of the present application.

[0167] The input / output interface 1003 is configured to realize information input and output.

[0168] The communication interface 1004 is configured to realize the communication interaction between the device and other devices, and can realize the communication through a wired manner (for example, a USB, a network cable, or the like) or a wireless manner (for example, a mobile network, WIFI, Bluetooth, or the like).

[0169] The bus 1005 is configured to transmit information between various components (for example, the processor 1001, the memory 1002, the input / output interface 1003, and the communication interface 1004) of the device.

[0170] The processor 1001, the memory 1002, the input / output interface 1003, and the communication interface 1004 are connected to each other through the bus 1005 to realize the communication connection between the device.

[0171] The embodiments of the present application also provide a computer readable storage medium, which stores a computer program, and the computer program is executed by a processor to implement the voice generation method.

[0172] The memory, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs and non-transitory computer-executable programs. In addition, the memory can include a high-speed random access memory, and can also include a non-transitory memory, such as at least one magnetic disk storage device, a flash memory device, or other non-transitory solid-state memory device. In some embodiments, the memory can optionally include a memory disposed remotely with respect to the processor, which can be connected to the processor through a network. Examples of the above network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0173] The voice generation method and device, computer device and storage medium provided by the embodiments of the present application extract reference voice features with a target object sound style from original voice data through an original voice conversion model, and the original voice conversion model also extracts selected voice features from the original voice data. In order to obtain voice features without the target object sound style, the feature similarity data between the reference voice features and the selected voice features is calculated, and the target voice features without the target object sound style are selected from the selected voice features according to the feature similarity data. Therefore, the original voice generation model is trained by combining the target voice features, the reference voice features and the training text, so as to construct a target voice generation model which can generate voice data with only the target object sound style and reduce the interference of other sound styles. Therefore, the target voice generation model can generate clean and stable voice data, reduce the dependence on reference audio, and improve the flexibility of voice data generation.

[0174] The embodiments described in the embodiments of the present application are used to more clearly illustrate the technical solutions of the embodiments of the present application, and do not constitute a limitation on the technical solutions provided by the embodiments of the present application. Those skilled in the art can know that, with the evolution of technology and the appearance of new application scenarios, the technical solutions provided by the embodiments of the present application are also applicable to similar technical problems.

[0175] Those skilled in the art can understand that the technical solutions shown in the figures do not constitute a limitation on the embodiments of the present application, and can include more or fewer steps than the figures shown, or combine certain steps, or different steps.

[0176] The device embodiments described above are only schematic, and the units illustrated as separate components can or can not be physically separated, that is, can be located in one place, or can be distributed on multiple network units. Part or all of the modules can be selected according to actual needs to achieve the purpose of the embodiments of the present application.

[0177] Those skilled in the art can understand that all or some of the steps in the method disclosed above, the function modules / units in the system and the device can be implemented as software, firmware, hardware or appropriate combination thereof.

[0178] The terms "first", "second", "third", "fourth" and the like in the description of the application and in the claims, if any, are used for distinguishing between similar elements and not necessarily for describing a particular sequential or chronological order. It is to be understood that the use of these terms herein is to be construed to cover a general order and / or structure unless otherwise indicated. Furthermore, the terms "comprise", "comprising", "has", "having", "includes", "including", "contain", "containing" or any other similar forms are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises, has, includes, contains items or components does not include items or components not explicitly recited. The terms "a" or "an", as used herein in the detailed description and in the claims, mean "one or more" or "at least one", unless otherwise indicated.

[0179] It should be understood that, in the present application, "at least one" means one or more, and "multiple" means two or more. "And / or" is used to describe the relationship between associated objects, which means that there can be three relationships, for example, "A and / or B" can represent three cases: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally represents an "or" relationship between the associated objects. "At least one of the following" or similar expressions means any combination of these items, including any combination of single or multiple items. For example, at least one of a, b or c can mean a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be singular or plural.

[0180] In several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic, for example, the division of the above units is only a logical function division, and actual implementation can have another division manner, for example, a plurality of units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between the displayed or discussed objects can be indirect coupling or communication connection through some interfaces, devices or units, and can be electrical, mechanical or other forms.

[0181] The units described as separate components above can or can not be physically separate, and the components shown as units can or can not be physical units, that is, can be located in one place, or can be distributed to multiple network units. Part or all of the units can be selected according to actual needs to achieve the purpose of the embodiment scheme.

[0182] In addition, each functional unit in each embodiment of the present application can be integrated in one processing unit, or each unit can exist physically, or two or more units can be integrated in one unit. The integrated unit can be realized in the form of hardware or in the form of a software functional unit.

[0183] If the integrated unit is realized in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer readable storage medium. Based on this understanding, the technical scheme of the present application or the part that contributes to the prior art or the whole or part of the technical scheme can be embodied in the form of a software product. The computer software product is stored in a storage medium, including multiple instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the method of each embodiment of the present application. The foregoing storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, and various program storage media.

[0184] The preferred embodiments of the embodiments of the present application are described above with reference to the accompanying drawings, but this does not limit the scope of the embodiments of the present application. Any modifications, equivalent replacements and improvements made by those skilled in the art without departing from the scope and essence of the embodiments of the present application shall be within the scope of the embodiments of the present application.

Claims

1. A voice generation method characterized by, The method comprises: obtaining original speech data and training text; extracting speech features from the original speech data through a preset original speech conversion model to obtain a set of speech features; wherein the set of speech features comprises reference speech features and selected speech features, and the reference speech features have a target object voice style; measuring the similarity of the reference speech features and the selected speech features to obtain feature similarity data; selecting target speech features from the selected speech features according to the feature similarity data; wherein the target speech features do not have the target object voice style; training a preset original speech generation model according to the target speech features, the reference speech features and the training text to obtain a target speech generation model; generating speech through the target speech generation model and target text to obtain target speech data; the original speech conversion model comprises a speech encoder, a first feature extractor and a second feature extractor; the extraction of the set of speech features from the original speech data through the preset original speech conversion model comprises: extracting candidate speech features from the original speech data through the speech encoder; extracting the reference speech features from the candidate speech features through the first feature extractor; extracting the selected speech features from the candidate speech features through the second feature extractor and the reference speech features; wherein the weights and values of the reference speech features and the selected speech features are 1.

2. The method of claim 1, wherein, The selection of the target speech features from the selected speech features according to the feature similarity data comprises: quantizing the selected speech features according to a preset quantization feature to obtain preliminary speech features; selecting the target speech features from the preliminary speech features according to the feature similarity data.

3. The method of claim 2, wherein, The quantization of the selected speech features according to the preset quantization feature to obtain the preliminary speech features comprises: calculating the feature distance between the preset quantization feature and the selected speech feature to obtain a speech feature distance; selecting the selected speech feature from the preset quantization feature according to the speech feature distance; splicing the selected speech feature to obtain the preliminary speech feature.

4. The method according to any one of claims 1 to 3, characterized in that, After the selection of the target speech features from the selected speech features according to the feature similarity data, the method further comprises: splicing the target speech features and the reference speech features to obtain spliced speech features; reconstructing speech from the spliced speech features to obtain reconstructed speech data; calculating the loss of the reconstructed speech data and the original speech data to obtain reconstructed loss data; calculating the loss of a preset similarity threshold and the feature similarity data to obtain similarity loss data; identifying the object according to the reference speech features to obtain object identification information; According to the preset target object information and the object recognition information, loss calculation is performed to obtain recognition loss data; According to the reconstruction loss data, the similarity loss data and the recognition loss data, model parameter adjustment is performed on the original speech conversion model to obtain a target speech conversion model.

5. The method according to any one of claims 1 to 3, characterized in that, The original speech generation model includes an original text encoder and an original decoder; the model training of the preset original speech generation model according to the target speech feature, the reference speech feature and the training text to obtain a target speech generation model includes: The original text encoder is used to perform semantic extraction on the training text to obtain text semantic information; The original decoder and the reference speech feature are used to perform speech feature conversion on the text semantic information to obtain predicted speech features; wherein the predicted speech features do not have a target object sound style; The original speech generation model is adjusted according to the predicted speech features and the target speech features to obtain the target speech generation model.

6. The method of claim 5, wherein, The original text encoder includes a word segmentation layer, a semantic analysis layer and an encoding layer; the original text encoder is used to perform semantic extraction on the training text to obtain text semantic information, including: The word segmentation layer is used to perform word segmentation processing on the training text to obtain training word groups; The semantic analysis layer is used to perform semantic analysis on the training word groups to obtain semantic analysis information; The encoding layer and the semantic analysis information are used to perform semantic extraction on the training text to obtain the text semantic information.

7. A speech generating device, comprising: The device includes: An acquisition module is configured to acquire original speech data and training text; A speech feature extraction module is configured to extract speech features from the original speech data by using a preset original speech conversion model to obtain a set of speech features; wherein the set of speech features includes reference speech features and selected speech features, and the reference speech features have a target object sound style; A similarity measurement module is configured to measure the similarity between the reference speech features and the selected speech features to obtain feature similarity data; A feature screening module is configured to screen the selected speech features according to the feature similarity data to obtain target speech features; wherein the target speech features do not have the target object sound style; A model training module is configured to train a preset original speech generation model according to the target speech features, the reference speech features and the training text to obtain a target speech generation model; A speech generation module is configured to generate target speech data by using the target speech generation model and target text; The original speech conversion model includes a speech encoder, a first feature extractor and a second feature extractor; the original speech conversion model is used to extract speech features from the original speech data to obtain a set of speech features, including: The speech encoder is used to extract candidate speech features from the original speech data; extracting speech features from the candidate speech features through the first feature extractor to obtain the reference speech features; extracting speech features from the candidate speech features through the second feature extractor and the reference speech features to obtain the selected speech features; wherein the weights of the reference speech features and the selected speech features are both 1.

8. A computer device, comprising: The computer device comprises a memory and a processor, the memory stores a computer program, and the processor implements the speech generation method in any one of claims 1 to 6 when executing the computer program.

9. A computer-readable storage medium storing a computer program, the computer program comprising instructions that, when executed by a computer, cause the computer to perform the method of any one of claims 1 to 8. The computer program is executed by the processor to implement the speech generation method in any one of claims 1 to 6.

Images