Sound duplicating method, device and equipment and storage medium

By breaking down the sound replication model into feature extraction and synthesis modules, and pre-extracting and saving reference audio features, the problem of long audio synthesis time and high resource consumption in existing technologies is solved, achieving more efficient audio synthesis and lower privacy risks.

CN121214904APending Publication Date: 2025-12-26TENCENT TECHNOLOGY (SHENZHEN) CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202410824066.6
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-06-24
Publication Date
2025-12-26

AI Technical Summary

Technical Problem

In practical applications, existing sound replication models require reading reference audio and performing calculations each time audio is synthesized, resulting in long waiting times, high resource consumption, and the risk of privacy leaks.

Method used

The sound replication model is broken down into a sound feature extraction module and a sound synthesis module. Features of the reference audio are extracted in advance and saved as feature files. When synthesizing audio, only the feature files are read for synthesis, avoiding repeated reading of the reference audio.

Benefits of technology

It reduces the time and resource consumption of audio synthesis, improves the user experience, reduces the risk of privacy leaks, and increases the ease of model deployment.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN121214904A_ABST
    Figure CN121214904A_ABST
Patent Text Reader

Abstract

The embodiment of the invention provides a sound copying method and device, equipment and a storage medium, which can be applied to scenes such as cloud technology, artificial intelligence, intelligent traffic, auxiliary driving, audio and video, in the method, feature extraction is performed on reference audio in advance, and reference timbre features and reference rhythm features are obtained. And obtaining and storing a sound feature file of the reference audio based on the reference timbre feature and the reference rhythm feature, so that when the reference audio is used as input for multiple times of audio synthesis, only the sound feature file of the reference audio needs to be read in each time of audio synthesis, and based on the first text feature of the text to be synthesized and the sound feature file, the sound feature file of the reference audio is read. According to the method and the device, the synthetic audio corresponding to the to-be-synthesized text is generated without repeatedly reading the reference audio and repeatedly calculating the reference audio, so that the time consumption and resource consumption of sound copying are effectively reduced, and the waiting time of synthesizing the audio by using a sound copying model is also effectively reduced, thereby improving the use experience and enhancing the controllability of the system.
Need to check novelty before this filing date? Find Prior Art

Description

TECHNICAL FIELD

[0001] Embodiments of the present application relate to the field of artificial intelligence technology, and in particular to a sound replication method and device, equipment and a storage medium. BACKGROUND

[0002] With the development of artificial intelligence technology, the scheme of using artificial intelligence technology for sound replication is becoming mature. Sound replication refers to taking a recorded reference audio as input, using a sound replication model to learn and simulate the sound features of the reference audio, so as to synthesize audio based on the sound features.

[0003] For example, in a map navigation scene, a sound replication model can be used to synthesize map navigation voice of any announcer, and the synthesized map navigation voice can be used for map navigation broadcasting.

[0004] Under the related technology, after the training of the sound replication model is completed, the trained sound replication model is deployed to the actual application environment. Since the input of the sound replication model includes the reference audio, and the calculation related to the reference audio is coupled in the structure of the sound replication model, in the actual application environment, when the same segment of reference audio is used as input for multiple audio synthesis, the segment of reference audio needs to be read each time for audio synthesis, and the segment of reference audio is calculated through the sound replication model.

[0005] However, the process of reading the reference audio and calculating the reference audio each time is time-consuming, which leads to a long waiting time for sound replication. Moreover, repeated calculation of the same segment of reference audio leads to additional resource consumption, resulting in large resource consumption. SUMMARY

[0006] Embodiments of the present application provide a sound replication method, device, equipment and storage medium, which reduce the waiting time for synthesizing audio using a sound replication model, and reduce the time consumption and resource consumption of sound replication.

[0007] In one aspect, the present application provides a sound replication method, which comprises:

[0008] Obtaining a text to be synthesized, and performing feature extraction on the text to be synthesized to obtain first text features;

[0009] Reading a sound feature file of a pre-stored reference audio, the sound feature file containing reference timbre features and reference prosody features obtained by pre-extracting features of the reference audio;

[0010] Based on the first text features, the reference timbre features and the reference prosody features, predicting target prosody features corresponding to the text to be synthesized;

[0011] generate, based on the first text feature, the reference timbre feature, and the target prosody feature, synthesized audio corresponding to the text to be synthesized.

[0012] In one aspect, an embodiment of the present application provides a sound replication device, which comprises:

[0013] An acquisition unit is configured to acquire text to be synthesized and perform feature extraction on the text to be synthesized to obtain a first text feature.

[0014] A reading unit is configured to read a sound feature file of pre-stored reference audio, wherein the sound feature file comprises a reference timbre feature and a reference prosody feature obtained by performing feature extraction on the reference audio in advance.

[0015] A prediction unit is configured to predict, based on the first text feature, the reference timbre feature, and the reference prosody feature, a target prosody feature corresponding to the text to be synthesized.

[0016] A synthesis unit is configured to generate, based on the first text feature, the reference timbre feature, and the target prosody feature, synthesized audio corresponding to the text to be synthesized.

[0017] Optionally, the target prosody feature comprises sub-prosody features of N text units in the text to be synthesized, wherein N is greater than 1.

[0018] The prediction unit is specifically configured to:

[0019] predict, based on the first text feature, the reference timbre feature, and the reference prosody feature, sub-prosody features of N text units in the text to be synthesized in a frame-by-frame autoregressive manner.

[0020] Optionally, the prediction unit is specifically configured to:

[0021] predict, based on the first text feature, the reference timbre feature, and the reference prosody feature, a sub-prosody feature of a first text unit in the text to be synthesized.

[0022] For other N-1 text units in the text to be synthesized, the following operations are respectively performed:

[0023] predict, based on the first text feature, the reference timbre feature, and a sub-prosody feature of an i-th text unit, a sub-prosody feature of an i+1-th text unit, wherein i is greater than 0 and less than N.

[0024] Optionally, the sound feature file further comprises a history key-value pair of an attention mechanism network in a prosody large language model.

[0025] Optionally, the acquisition unit is specifically configured to:

[0026] inputting the first text feature, the reference timbre feature and the reference prosody feature into the prosody large language model, and predicting a target prosody feature associated with the text to be synthesized by reusing the history key-value pairs.

[0027] Optionally, the obtaining unit is specifically configured to:

[0028] obtain a prosody audio from the reference audio, and extract a mel-spectrogram feature of the prosody audio;

[0029] perform prosody coding on the mel-spectrogram feature of the prosody audio to obtain a preliminary prosody feature;

[0030] predict the reference prosody feature based on the reference timbre feature, the preliminary prosody feature and a second text feature of a reference text corresponding to the reference audio by using the prosody large language model.

[0031] Optionally, the obtaining unit is further configured to:

[0032] after the reference prosody feature is predicted based on the reference timbre feature, the preliminary prosody feature and the second text feature of the reference text corresponding to the reference audio by using the prosody large language model, cache the history key-value pairs of an attention mechanism network in the prosody large language model.

[0033] Optionally, the obtaining unit is specifically configured to:

[0034] obtain a timbre audio from the reference audio, and extract a mel-spectrogram feature of the timbre audio;

[0035] perform timbre coding on the mel-spectrogram feature of the timbre audio to obtain the reference timbre feature.

[0036] Optionally, the obtaining unit is specifically configured to:

[0037] perform an in-memory serialization operation on the reference timbre feature and the reference prosody feature to obtain a binary sound feature file;

[0038] save the sound feature file in a storage device.

[0039] Optionally, the reading unit is further configured to:

[0040] after the sound feature file corresponding to the reference audio is read, perform an in-memory deserialization operation on the sound feature file to obtain the reference timbre feature and the reference prosody feature.

[0041] Optionally, the sound feature file is generated by a sound feature extraction module in a sound replication model; the sound replication model further comprises a sound synthesis module for generating the synthesized audio.

[0042] The prediction unit is further configured to:

[0043] Before predicting the target prosody feature corresponding to the text to be synthesized based on the first text feature, the reference timbre feature and the reference prosody feature, the model version information corresponding to the sound feature extraction module is obtained from the sound feature file, and it is determined that the model version information corresponding to the sound feature extraction module is consistent with the model version information corresponding to the sound synthesis module.

[0044] In an aspect, an embodiment of the present application provides a computer device, comprising a memory, a processor and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the sound replication method when executing the computer program.

[0045] In an aspect, an embodiment of the present application provides a computer readable storage medium, which stores a computer program executable by a computer device, and when the computer program is executed on the computer device, the computer device executes the steps of the sound replication method.

[0046] In an aspect, an embodiment of the present application provides a computer program product, comprising a computer program stored in a computer readable storage medium, wherein the computer program comprises program instructions, and when the program instructions are executed by a computer device, the computer device executes the steps of the sound replication method.

[0047] In an embodiment of the present application, the reference audio is pre-processed for feature extraction to obtain reference timbre features and reference prosody features, and then based on the reference timbre features and the reference prosody features, a sound feature file of the reference audio is obtained and saved. In this way, when the reference audio is input for multiple audio synthesis, each audio synthesis only needs to read the sound feature file of the reference audio, and based on the first text feature of the text to be synthesized and the sound feature file, the synthesized audio corresponding to the text to be synthesized is generated, without the need to repeatedly read the reference audio and repeatedly calculate the reference audio, which effectively reduces the time consumption and resource consumption of sound replication, and also effectively reduces the waiting time of using the sound replication model to synthesize audio, thereby improving the use experience and enhancing the controllability of the system.

[0048] Secondly, the application extracts features of the reference audio in advance, obtains a sound feature file of the reference audio and saves the sound feature file. Compared with directly saving the reference audio, the sound features are saved in the form of a feature file, which not only reduces the consumption of storage resources and computing resources, but also avoids the risk of privacy leakage.

[0049] In addition, the application separates the process of extracting the sound feature file of the reference audio and the process of generating the synthesized audio corresponding to the to-be-synthesized text into two independent processes, decouples the sound feature file extraction and the audio synthesis, so that each audio synthesis no longer depends on reading the reference audio and related calculation on the reference audio, but only needs to directly read the sound feature file, which effectively improves the convenience of model deployment and reduces the time consumption of the model in the sound synthesis stage. BRIEF DESCRIPTION OF DRAWINGS

[0050] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings needed in the embodiment description will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0051] Figure 1 A structural schematic diagram of a system architecture provided by the embodiment of the present application;

[0052] Figure 2 A schematic diagram of an application scenario provided by the embodiment of the present application;

[0053] Figure 3 A schematic diagram of an application scenario provided by the embodiment of the present application;

[0054] Figure 4 A structural schematic diagram of a sound replication model provided by the embodiment of the present application;

[0055] Figure 5 A flowchart of a sound replication method provided by the embodiment of the present application;

[0056] Figure 6 A schematic diagram of the first stage of a prosodic large language model provided by the embodiment of the present application;

[0057] Figure 7 A structural schematic diagram of a sound feature extraction module provided by the embodiment of the present application;

[0058] Figure 8 A schematic diagram of the second stage of a prosodic large language model provided by the embodiment of the present application;

[0059] Figure 9 A structural schematic diagram of a sound synthesis module provided by an embodiment of the present application is provided.

[0060] Figure 10A A flowchart of a sound replication method provided by an embodiment of the present application is provided.

[0061] Figure 10B A flowchart of a sound replication method provided by an embodiment of the present application is provided.

[0062] Figure 11 A structural schematic diagram of a sound replication device provided by an embodiment of the present application is provided.

[0063] Figure 12 A structural schematic diagram of a computer device provided by an embodiment of the present application is provided. DETAILED DESCRIPTION

[0064] In order to make the objectives, technical solutions and beneficial effects of the present application clearer, the present application will be further described in detail below in combination with the drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and do not limit the present application.

[0065] In order to facilitate understanding of the technical solutions provided by the embodiments of the present application, some key terms used by the embodiments of the present application are first explained here:

[0066] Sound replication: taking a recorded reference audio as input, using a sound replication model to learn and simulate the sound features of the reference audio, so as to synthesize audio based on the sound features of the text.

[0067] Timbre: refers to the pronunciation characteristics of the voice itself when the speaker speaks, such as fine and thick.

[0068] Prosody: refers to the ups and downs of the tone and the emphasis of the pronunciation when the speaker speaks.

[0069] Model deployment: also known as model landing deployment, refers to deploying a trained machine learning model to an actual application environment for use in actual application. Before the model is released, the model needs to be exported from the training environment and then deployed to the actual application environment, usually in the form of a service or a library deployed in the actual application environment.

[0070] Mel-spectrogram features: A feature extraction method commonly used in speech signal processing and audio signal processing. It is designed based on the perceptual characteristics of human ears. The perception of audio signals by human ears is not linear, but very complex. Mel frequency is a frequency scale designed according to the hearing characteristics of human ears, which is different from the ordinary linear frequency scale. By filtering the signal in the Mel frequency, the perceptual characteristics of the human ear can be better simulated, and more distinctive signal features can be extracted. Mel-spectrogram features are widely used in speech recognition, speaker recognition, emotion recognition and other fields. It has good distinguishability and robustness, can extract the key features of speech signals, and improve the accuracy and stability of the model. At the same time, the extraction process of Mel-spectrogram features is simple and has small computational complexity, which is suitable for real-time application in embedded systems and mobile devices.

[0071] Word2Vec (word to vector) model: A classic word vector model that learns distributed representations of words by training a shallow neural network.

[0072] GloVe (Global Vectors for Word Representation) model: A word vector model based on global statistical information. GloVe model uses global co-occurrence matrix to capture the relevance between words, and obtains the final word vector representation by solving linear equations.

[0073] The embodiments of the present application are related to artificial intelligence and machine learning (Machine Learning, ML) technology, and are mainly designed based on machine learning and speech technology in artificial intelligence.

[0074] Artificial intelligence (Artificial Intelligence, AI) is to use digital computers or digital computer controlled machines to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results. In other words, artificial intelligence is a comprehensive technology of computer science, which tries to understand the essence of intelligence and produce a new intelligent machine that can react in a similar way to human intelligence. Artificial intelligence is to study the design principles and implementation methods of various intelligent machines, so that machines have the functions of perception, reasoning and decision-making.

[0075] Artificial intelligence technology is a comprehensive discipline, involving a wide range of fields, both hardware and software technologies. Artificial intelligence basic technologies generally include, such as sensors, special artificial intelligence chips, cloud computing, distributed storage, big data processing technology, pre-training model technology, operation / interaction system, mechatronics, etc. Among them, the pre-training model is also called large model, basic model, which can be widely used in artificial intelligence after fine-tuning. Artificial intelligence software technology mainly includes computer vision technology, speech processing technology, natural language processing technology and machine learning / deep learning, etc.

[0076] Machine Learning (ML) is a multi-disciplinary subject that involves probability theory, statistics, approximation theory, convex analysis, algorithm complexity theory, and other disciplines. It is a specialized study of how computers simulate or implement human learning behavior to acquire new knowledge or skills, reorganize existing knowledge structure to continuously improve their performance. Machine learning is the core of artificial intelligence and the fundamental approach to making computers intelligent, and its applications span all areas of artificial intelligence. Machine learning and deep learning generally include artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning, and rule-based learning. Pre-trained models are the latest development in deep learning, integrating the above technologies.

[0077] The key technologies of speech technology include automatic speech recognition technology (ASR) and speech synthesis technology (TTS) as well as voiceprint recognition technology. Enabling computers to hear, see, speak, and feel is the future direction of human-computer interaction, and voice is one of the promising ways of human-computer interaction in the future. Large model technology brings about a revolution in speech technology, and pre-training models such as WavLM (general speech pre-training model) and UniSpeech (general speech pre-training model) have strong generalization and universality, and can perform various speech processing tasks. The Transformer architecture is a deep learning model architecture based on self-attention mechanism, which can be used for natural language processing and sequence-to-sequence tasks. The Transformer architecture mainly includes encoding components and decoding components.

[0078] Large Language Model (LLM): refers to a deep learning model trained using a large amount of text data, which can generate natural language text or understand the meaning of language text. Large language models can handle a variety of natural language tasks, such as text classification, question answering, and dialogue, and are an important way to artificial intelligence.

[0079] Zero-shot learning: A learning method in the field of machine learning that aims to enable models to recognize, classify, or understand classes or concepts that have not been seen during the training process. Typical algorithms for zero-shot learning focus on establishing a knowledge transfer mechanism from known classes to unknown classes. These algorithms usually utilize shared attributes or semantic relationships between classes to bridge the gap between known classes and unknown classes.

[0080] With the research and progress of artificial intelligence technology, artificial intelligence technology has been researched and applied in many fields, such as common smart home, smart wearable devices, virtual assistants, smart speakers, smart marketing, unmanned vehicles, autonomous driving, drones, digital twins, virtual humans, robots, artificial intelligence generated content (AIGC), conversational interaction, smart medical care, smart customer service, game AI, etc. It is believed that with the development of technology, artificial intelligence technology will be applied in more fields and play an increasingly important role.

[0081] The scheme provided by the embodiments of the present application mainly relates to the application of artificial intelligence in sound replication, specifically relates to sound replication in map navigation scenarios, short video production scenarios, media broadcasting scenarios, voice social scenarios, etc. Before sound replication, a sound replication model for sound replication needs to be trained by combining machine learning and voice technology in artificial intelligence.

[0082] Specifically, the sound replication process in the embodiments of the present application can be divided into two parts, including a training part and an application part. The training part relates to the technical field of machine learning.

[0083] In the training part, original voice data is collected and preprocessed to obtain a voice dataset. The preprocessing process specifically includes: cleaning the voice data to ensure data quality; and annotating the voice data.

[0084] The dataset is divided into a training set and a test set. The sound replication model is trained using the training set. In the actual training process, when the sound replication model is a large model, the sound replication model can be trained in a distributed manner. Distributed training refers to splitting and sharing the workload of training the model to multiple microprocessors. The large model has a large number of parameters and a large amount of training data, which exceeds the capacity of a single machine, so distributed parallel speedup is needed. Parallel mechanisms include data parallelism (DP), model parallelism (MP), pipeline parallelism (PP), and hybrid parallelism (HP). Structural design includes parameter server-based, reduce-based, MPI-based, and other structures.

[0085] The test set is used to evaluate the training effect of the sound replication model. Evaluation indicators can include accuracy, recall rate, precision, and F1 score. Based on the evaluation results, the hyperparameters of the sound replication model are adjusted to obtain the best performance.

[0086] In the application part, the trained sound replication model is deployed to the production environment (such as various applications) to realize sound replication in various applications.

[0087] In actual application, in the training part or the application part of the sound replication model, model parallel computing can be used, that is, the calculation task of the model is distributed to multiple computing devices (such as central processing units (CPUs), graphics processing units (GPUs), etc.) to perform calculation at the same time, thereby accelerating the training and inference of the model. Model parallel computing can effectively utilize computing resources and improve the computing efficiency and training speed of the model.

[0088] It should be noted that the sound replication model in the embodiments of the present application can be online training or offline training, which is not limited here.

[0089] Next, the technical idea of the embodiments of the present application is briefly described.

[0090] Sound replication is an input of sound features based on recorded reference audio, and the sound replication model is used to learn and simulate the sound features of the reference audio to synthesize audio based on the sound features. In related technologies, the following two schemes are widely used in the deployment of the sound replication model:

[0091] Scheme one: base model fine-tuning scheme. This scheme refers to first training a base model of speech synthesis using a large amount of high-quality speech synthesis data. Then, the base model is fine-tuned using the speech recorded by the specific user to obtain the sound replication model. The base model fine-tuning scheme performs the following operations in the actual landing deployment process:

[0092] Step 1: Collect several speeches of a specific user and perform text annotation on them.

[0093] Step 2: Fine-tune the base model using the annotated data.

[0094] Step 3: Replace the sound replication model obtained after fine-tuning with the online model, and provide audio synthesis services for the specific user.

[0095] The above scheme one needs to record a long user speech and fine-tune the base model with the user speech to obtain the sound replication model for the user to provide audio synthesis services, which is low in efficiency.

[0096] Scheme two: Zero-shot scheme. This scheme uses the Zero-shot learning method to obtain the sound replication model, and then deploys the sound replication model in the actual application. Compared with scheme one, after the model is deployed in the actual application, only a few seconds to tens of seconds of reference audio needs to be recorded by the user, and the sound replication model can provide audio synthesis services for the user based on the reference audio, without the need for fine-tuning of the model, so the efficiency is higher.

[0097] However, due to the limitation of the structure of the sound replication model in scheme two, there are the following problems after actual landing deployment:

[0098] First, since the input of the sound replication model includes the reference audio, and the calculation related to the reference audio is coupled in the structure of the sound replication model, in the actual application environment, when the same reference audio is input for multiple audio synthesis, the reference audio needs to be read each time and the related calculation of the reference audio is performed through the sound replication model.

[0099] However, the process of reading the reference audio and calculating the reference audio each time is time-consuming, which leads to a long waiting time for sound replication; moreover, repeated calculation of the same reference audio causes additional resource consumption, resulting in large resource consumption.

[0100] Second, when using the sound replication model for audio synthesis, the reference audio needs to be read, so after collecting and uploading the reference audio, the reference audio needs to be saved in the server, which not only occupies a certain amount of storage resources, but also may cause privacy risks of data leakage.

[0101] To solve the above problems, the embodiment of the present application provides a sound replication method. In the method, when the sound replication model is deployed, the sound replication model is disassembled into a sound feature extraction module and a sound synthesis module, that is, the sound feature extraction module and the sound synthesis module are in a decoupled state.

[0102] Through the sound feature extraction module, the reference audio is pre-processed for feature extraction to obtain reference timbre features and reference prosody features. Based on the reference timbre features and the reference prosody features, a sound feature file of the reference audio is obtained, and the sound feature file of the reference audio is saved in a storage device.

[0103] Each time the sound synthesis module synthesizes audio, the text to be synthesized is obtained first, and the text to be synthesized is processed for feature extraction to obtain first text features. The pre-stored sound feature file of the reference audio is read, and then based on the first text features, the reference timbre features and the reference prosody features, the target prosody features corresponding to the text to be synthesized are predicted. Then, based on the first text features, the reference timbre features and the target prosody features, the synthesized audio corresponding to the text to be synthesized is generated.

[0104] In the embodiment of the present application, the reference audio is pre-processed for feature extraction to obtain reference timbre features and reference prosody features, and then based on the reference timbre features and the reference prosody features, a sound feature file of the reference audio is obtained and saved. In this way, when the reference audio is input for multiple audio synthesis, each audio synthesis only needs to read the sound feature file of the reference audio, and based on the first text features of the text to be synthesized and the sound feature file, the synthesized audio corresponding to the text to be synthesized is generated, without the need to repeatedly read the reference audio and repeatedly calculate the reference audio, which effectively reduces the time consumption and resource consumption of sound replication, and also effectively reduces the waiting time of using the sound replication model to synthesize audio, thereby improving the use experience and enhancing the controllability of the system.

[0105] Secondly, the present application pre-processes the reference audio for feature extraction to obtain a sound feature file of the reference audio and save the sound feature file. Compared with directly saving the reference audio, this way of saving the sound features in the form of a feature file not only reduces the consumption of storage resources and computing resources, but also avoids the risk of privacy leakage.

[0106] In addition, the present application disassembles the process of extracting the sound feature file of the reference audio and the process of generating the synthesized audio corresponding to the text to be synthesized into two independent processes, realizing the decoupling between the sound feature file extraction and the audio synthesis. In this way, each audio synthesis no longer depends on reading the reference audio and related calculations on the reference audio, but only needs to directly read the sound feature file, which effectively improves the convenience of model deployment and reduces the time consumption of the model in the sound synthesis stage.

[0107] The following is a brief introduction to the system architecture diagram applicable to the technical solutions of the embodiments of the present application. It should be noted that the system architecture diagram described below is only used to illustrate the embodiments of the present application and is not limiting.

[0108] Reference Figure 1 which is a system architecture diagram applicable to the embodiments of the present application. The system architecture includes at least a terminal device 101 and a server 102. The number of terminal devices 101 can be one or more, and the number of servers 102 can also be one or more. The number of terminal devices 101 and servers 102 is not specifically limited by the present application.

[0109] The terminal device 101 pre-installs an application with sound replication function. The application can be a client application, a web application, a small program application, etc. The terminal device 101 can be a smart phone, a tablet computer, a notebook computer, a desktop computer, a smart home appliance, a smart voice interaction device, a smart vehicle device, etc., but is not limited thereto.

[0110] The server 102 is the background server of the application. The server 102 can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content distribution networks (Content Delivery Network, CDN), and big data and artificial intelligence platforms, etc. Basic cloud computing services.

[0111] In the embodiments of the present application, the terminal device 101 and the server 102 can be directly or indirectly connected through one or more networks. The network can be a wired network or a wireless network, for example, the wireless network can be a mobile cellular network or a wireless fidelity (Wireless-Fidelity, WIFI) network, and of course it can also be other possible networks. The embodiments of the present application do not limit this.

[0112] It should be noted that the sound replication model in the embodiments of the present application can be deployed on the terminal device 101 or the server 102. That is, the sound replication method in the embodiments of the present application can be executed by the terminal device 101 or the server 102, or jointly executed by the terminal device 101 and the server 102. The present application does not specifically limit this.

[0113] The following will briefly introduce the application scenarios to which the technical solutions of the embodiments of the present application can be applied. It should be noted that the following introduction of the application scenarios is only for the purpose of illustration and not limitation. In the actual implementation process, the technical solutions provided by the embodiments of the present application can be flexibly applied according to actual needs.

[0114] The scheme provided by the embodiments of the present application can be applied to sound replication in various application scenarios, such as map navigation scenarios, short video production scenarios, media reporting scenarios, voice social scenarios, etc. The scheme can also be applied as a basic technology in various scenarios, including but not limited to cloud technology, artificial intelligence, intelligent transportation, assisted driving, audio and video scenarios, etc.

[0115] The application scenarios to which the scheme provided by the present application is applicable will be described by way of example.

[0116] Application scenario one, map navigation scenario.

[0117] Referring to Figure 2 , first, the 15-second voice of the reporter is collected as the reference audio. The reference audio is input into the sound feature extraction module in the sound replication model for sound feature extraction, and the sound feature file of the reporter is obtained and saved in the hard disk.

[0118] The navigation text associated with the map application is obtained, and the sound feature file is read from the hard disk. The navigation text and the sound feature file are input into the sound synthesis module in the sound replication model for audio synthesis, and the map navigation voice of the navigation text is output.

[0119] When a new route is added in the map application, the navigation text of the new route is obtained; the sound feature file is read from the hard disk. The navigation text of the new route and the sound feature file are input into the sound synthesis module in the sound replication model for audio synthesis, and the map navigation voice of the navigation text of the new route is output.

[0120] By replicating the sound features of the reporter through the sound replication model, the map navigation voice is generated, which not only reduces the recording work of the reporter and improves the production efficiency of the reporting content, but also increases the map navigation experience through the tone familiar and loved by the map users.

[0121] Application scenario two, short video production scenario.

[0122] Referring to Figure 3 , first, the 20-second voice of the video producer is collected as the reference audio. The reference audio is input into the sound feature extraction module in the sound replication model for sound feature extraction, and the sound feature file of the video producer is obtained and saved in the hard disk.

[0123] When a short video about cooking needs to be made, first, the commentary text of the cooking video is obtained. Then, the voice feature file of the video producer is obtained. The commentary text of the cooking video and the voice feature file are input into the voice synthesis module of the voice replication model for audio synthesis, and the commentary voice corresponding to the commentary text is output.

[0124] When a short video of other content needs to be made, the corresponding commentary voice can also be generated in the same way, which will not be described here. By replicating the voice features of the video producer through the voice replication model, the commentary voice of the short video can be generated, which can effectively improve the efficiency of short video production.

[0125] In addition, it should be noted that in the specific embodiments of the present application, data related to sound, voice, audio, etc. are involved. When the related embodiments in the present application are applied to specific products or technologies, the permission or consent of the user needs to be obtained, and the collection, use and processing of the related data need to comply with the relevant laws, regulations and standards of the country and region.

[0126] The method provided by the example embodiments of the present application will be described below in combination with the application scenarios and system architecture described above and in reference to the accompanying drawings. It should be noted that the above application scenarios are only shown to facilitate understanding of the spirit and principles of the present application, and the embodiments of the present application are not limited in this respect.

[0127] Since the voice replication method provided by the present application is based on the voice replication model after training, before introducing the actual voice replication process, the structure of the voice replication model will be introduced first.

[0128] Referring to Figure 4 A structural diagram of a voice replication model is provided for the embodiments of the present application, and the voice replication model includes:

[0129] The voice feature extraction module and the voice synthesis module are in a decoupled state. In actual application, the voice feature extraction module and the voice synthesis module can be deployed on the same device or on different devices.

[0130] The voice feature extraction module takes the reference audio as input, generates a corresponding voice feature file, and saves the voice feature file in a storage device.

[0131] The voice feature file is loaded from the storage device, and the text to be synthesized is received. The voice feature file and the text to be synthesized are input into the voice synthesis module, and the synthesized audio corresponding to the text to be synthesized can be obtained.

[0132] The structure of the voice replication model is shown in Figure 4 The voice replication process will be introduced in combination with the structural diagram of the voice replication model shown inFigure 5 A flowchart of a sound replication method provided by an embodiment of the present application is shown. The flow of the method is executed by a computer device, which can be Figure 1 The terminal device 101 and / or the server 102 shown in the figure include the following steps:

[0133] Step 501: Obtain the text to be synthesized and perform feature extraction on the text to be synthesized to obtain first text features.

[0134] Specifically, the text to be synthesized can be downloaded from the network, loaded from the local, or obtained in other ways, which is not limited in the present application.

[0135] The text encoder in the sound synthesis module is used to perform feature extraction on the text to be synthesized to obtain the first text features. In a specific implementation, the text encoder converts the input text to be synthesized into a fixed-dimensional vector representation. This vector representation captures the semantic features and syntactic features of the text to be synthesized, so as to facilitate subsequent processing and analysis. The types of text encoders include but are not limited to: Word2Vec model, GloVe model.

[0136] Step 502: Read the pre-stored sound feature file of the reference audio, which contains the reference timbre features and reference prosody features obtained by pre-feature extraction on the reference audio.

[0137] Specifically, the sound feature extraction module in the sound replication model is used to perform feature extraction on the reference audio to obtain the sound feature file, which at least includes the reference timbre features and the reference prosody features of the reference audio.

[0138] In some embodiments, the sound feature extraction module includes a mel-spectrum extractor and a timbre encoder. The reference timbre features of the reference audio are obtained in the following way:

[0139] The timbre audio is obtained from the reference audio, and the mel-spectrum features of the timbre audio are extracted. Then the mel-spectrum features of the timbre audio are encoded to obtain the reference timbre features.

[0140] Specifically, the reference audio can be directly used as the timbre audio (also referred to as timbre reference audio), or a preset time length of audio segment can be extracted from the reference audio as the timbre audio.

[0141] The timbre audio is input into a mel-spectrum extractor. The mel-spectrum extractor frames the timbre audio to obtain a plurality of timbre reference frames. For each timbre reference frame, the timbre reference frame is subjected to Fourier transform to obtain a timbre reference spectrum, and then the timbre reference spectrum is converted to a mel frequency to obtain a mel-spectrum feature of the timbre reference frame. Finally, the mel-spectrum extractor outputs the mel-spectrum features of the plurality of timbre reference frames as the mel-spectrum features of the timbre audio. Then, a timbre encoder is used to perform timbre encoding on the mel-spectrum features of the timbre audio to obtain the reference timbre features.

[0142] In the embodiments of the present application, since the mel-spectrum features have good distinguishability and robustness, the key features of the speech signal can be extracted. Therefore, when the mel-spectrum features of the timbre audio are extracted and the mel-spectrum features of the timbre audio are subjected to feature encoding to obtain the reference timbre features, the accuracy of extracting the reference timbre features can be improved, and thus the accuracy and stability of timbre replication can be improved.

[0143] In some embodiments, the sound feature extraction module comprises a mel-spectrum extractor, a prosody encoder, and a prosody large language model. The reference prosody features of the reference audio are obtained in the following manner:

[0144] The prosody audio is obtained from the reference audio, and the mel-spectrum features of the prosody audio are extracted. The mel-spectrum features of the prosody audio are subjected to prosody encoding to obtain preliminary prosody features. The reference prosody features are predicted based on the reference timbre features, the preliminary prosody features, and the second text features of the reference text corresponding to the reference audio by the prosody large language model.

[0145] Specifically, the reference audio can be directly used as the prosody audio (also referred to as prosody reference audio), or an audio segment with a preset time length can be intercepted from the reference audio as the prosody audio. In addition, the prosody audio can be the same audio as the timbre audio described in the foregoing, or can be different audio.

[0146] The prosody audio is input into a mel-spectrum extractor. The mel-spectrum extractor frames the prosody audio to obtain a plurality of prosody reference frames. For each prosody reference frame, the prosody reference frame is subjected to Fourier transform to obtain a prosody reference spectrum, and then the prosody reference spectrum is converted to a mel frequency to obtain a mel-spectrum feature of the prosody reference frame. Finally, the mel-spectrum extractor outputs the mel-spectrum features of the plurality of prosody reference frames as the mel-spectrum features of the prosody audio. Then, a prosody encoder is used to perform prosody encoding on the mel-spectrum features of the prosody audio to obtain preliminary prosody features.

[0147] The reference text corresponding to the reference audio can be obtained by performing speech recognition on the reference audio using a speech recognition model, can be obtained by performing speech recognition on the prosody audio using a speech recognition model, or can be obtained by performing speech recognition on the timbre audio using a speech recognition model. The types of the speech recognition model include, but are not limited to, Gaussian mixture model-hidden Markov model, deep neural network-hidden Markov model, and attention-based recurrent neural network.

[0148] In addition, the reference text corresponding to the reference audio can also be a text document associated with the reference audio directly downloaded from the network, and the present application does not make specific limitations on this.

[0149] The prosody large language model is a large language model based on a Transformer architecture. See Figure 6 The reference timbre feature, the preliminary prosody feature, and the second text feature of the reference text are input into the prosody large language model. In the sound feature extraction module, the prosody large language model only performs to the first stage, that is, the reference timbre feature, the preliminary prosody feature, and the second text feature of the reference text are spliced to obtain a first spliced feature, and then the first spliced feature is input into the Transformer architecture. Based on the first spliced feature, autoregressive decoding is performed in the Transformer architecture to obtain the reference prosody feature p0.

[0150] In the embodiments of the present application, the mel-spectrum feature of the prosody audio is extracted, and the mel-spectrum feature of the timbre audio is feature-encoded to obtain the preliminary prosody feature containing the key features of the prosody audio. Then, the reference prosody feature is predicted by using the prosody large language model in combination with the reference timbre feature, the preliminary prosody feature, and the second text feature of the reference text, which can improve the accuracy of the reference prosody feature extraction and further improve the accuracy and stability of the timbre replication.

[0151] In some embodiments, the reference timbre feature and the reference prosody feature are subjected to a memory serialization operation to obtain a binary sound feature file, and then the sound feature file is saved in a storage device.

[0152] Specifically, the memory serialization operation refers to converting the reference timbre feature and the reference prosody feature into a binary file in the form of a byte stream to achieve a long-term storage process. After obtaining the binary sound feature file, the sound feature file is saved in a storage device such as a hard disk, a database system, a cloud server, etc.

[0153] In actual applications, the storage device can save sound feature files of multiple speakers. In order to distinguish the sound feature files of different speakers, the identification information of the speaker can be saved in the storage device in correspondence with the sound feature file of the speaker.

[0154] In the embodiments of the present application, the reference audio is pre-processed to extract features, obtain reference timbre features and reference prosody features, and then the reference timbre features and the reference prosody features are converted into binary files for storage. Compared with directly saving the reference audio, saving the sound features of the reference audio in the form of binary files not only reduces the consumption of storage resources and computing resources, but also avoids the risk of privacy leakage.

[0155] In some embodiments, the Transformer architecture is a deep learning architecture based on self-attention mechanism, which significantly improves the performance of natural language processing tasks by introducing self-attention mechanism and multi-head attention mechanism.

[0156] In the attention mechanism network, the calculation of the jth output is only related to the jth query vector (query), and is not related to the historical query vector, so the historical query vector does not need to be cached. However, the calculation of the jth output is related to the complete key-value pair, which includes the key vector (key) and the value vector (Value). That is, the calculation of the jth output is not only related to the jth key vector and the jth value vector, but also related to the historical key-value pairs calculated before, wherein the historical key-value refers to all key vectors (key) and value vectors (Value) calculated when calculating the 0th output to the (j-1)th output, and j is greater than or equal to 0.

[0157] If the historical key-value pair is repeatedly calculated when calculating each output, it will result in a large resource consumption. In view of this, in the embodiments of the present application, after the prosody large language model outputs the reference prosody feature p0 in the first stage, the historical key-value pair of the attention mechanism network in the prosody large language model is also output. Then the historical key-value pair of the attention mechanism network in the prosody large language model is cached.

[0158] Specifically, the Transformer architecture includes multiple layers of attention mechanism networks, so for each layer of attention mechanism network, the historical key-value pair (i.e. key-value cache) of the attention mechanism network is stored in the server memory. The historical key-value pair includes the key vector (key) and the value vector (Value) calculated by the attention mechanism network when calculating the output of the reference prosody feature p0.

[0159] In addition, after obtaining the reference timbre features, the reference prosody features, and the historical key-value pair, the reference timbre features, the reference prosody features, and the historical key-value pair can be subjected to an in-memory serialization operation to obtain a binary sound feature file, and then the sound feature file is saved in a storage device.

[0160] In the embodiments of the present application, after the reference prosody feature p0 is output in the first stage of the prosody large language model, the historical key-value pairs output by the prosody large language model are cached in a key-value cache manner, so that the historical key-value pairs in the key-value cache can be directly reused in the subsequent stages of the prosody large language model without the need for repeated calculation of the historical key-value pairs, thereby reducing the calculation amount and improving the inference efficiency of the prosody large language model.

[0161] In some embodiments, when the sound replication model is deployed, the sound replication model is disassembled into a sound feature extraction module and a sound synthesis module. However, in actual application, it is still necessary to combine the sound feature extraction module and the sound synthesis module to replicate synthesized audio of any text, and therefore, it is necessary to ensure that the sound feature extraction module and the sound synthesis module correspond to the same version of the sound replication model.

[0162] Therefore, in addition to the reference timbre feature, the reference prosody feature, and the historical key-value pair, the model version information of the sound replication model corresponding to the sound feature extraction module is also added in the sound feature file of the present application. In specific implementation, after the reference timbre feature, the reference prosody feature, the historical key-value pair, and the model version information are obtained, the reference timbre feature, the reference prosody feature, the historical key-value pair, and the model version information are subjected to a memory serialization operation to obtain a binary sound feature file, and then the sound feature file is saved in a storage device.

[0163] For example, referring to Figure 7 , the sound feature extraction module includes a first mel-frequency spectrum extractor, a timbre encoder, a second mel-frequency spectrum extractor, a prosody encoder, a text encoder, a first stage of a prosody large language model, a speech recognition model, and a memory serialization module.

[0164] First, a reference audio is obtained, and the reference audio is input into the first mel-frequency spectrum extractor as a timbre reference audio. The first mel-frequency spectrum extractor extracts the mel-frequency spectrum feature of the timbre reference audio, and inputs the mel-frequency spectrum feature of the timbre reference audio into the timbre encoder. The timbre encoder encodes the timbre of the mel-frequency spectrum feature of the timbre reference audio to obtain a reference timbre feature, and inputs the reference timbre feature into the prosody large language model and saves the reference timbre feature in the server memory.

[0165] The reference audio is input into the speech recognition model as a prosody reference audio, and the speech recognition model performs speech recognition on the prosody reference audio to obtain a reference text, and inputs the reference text into the text encoder. The text encoder encodes the reference text to obtain a text feature of the reference text, and inputs the text feature of the reference text into the prosody large language model.

[0166] The reference audio is input into a second mel-spectrogram extractor as prosody reference audio. The second mel-spectrogram extractor extracts mel-spectrogram features of the prosody reference audio, and inputs the mel-spectrogram features of the prosody reference audio into a prosody encoder. The prosody encoder encodes the mel-spectrogram features of the prosody reference audio to obtain preliminary prosody features, and inputs the preliminary prosody features into a prosody large language model.

[0167] In the first stage of the prosody large language model, the reference timbre features, the preliminary prosody features, and text features of the reference text are spliced to obtain first spliced features. Then, the reference prosody features p0 are predicted based on the first spliced features using a Transformer architecture. The reference prosody features p0 and the key-value cache of the attention mechanism network (i.e., the key-value pairs calculated when the reference prosody features p0 are predicted) are output and saved in the server memory.

[0168] The model version information corresponding to the sound feature extraction module is obtained, and then the reference timbre features, the reference prosody features p0, the key-value cache, and the model version information are input into a memory serialization module to perform a memory serialization operation to obtain a binary sound feature file, and the sound feature file is saved in the hard disk.

[0169] In the embodiments of the present application, the reference audio is feature-extracted by the sound feature extraction module to obtain reference timbre features and reference prosody features, and the key-value cache of the attention mechanism network is output. Then, the reference timbre features, the reference prosody features, the key-value cache, and the model version information are converted into a binary file for saving. Compared with directly saving the reference audio, this not only reduces the consumption of storage resources and computing resources, but also avoids the risk of privacy leakage.

[0170] It should be noted that the content in the sound feature file can be other implementations in addition to the several implementations described above, such as the sound feature file including: reference timbre features, reference prosody features, and model version information; the sound feature file including: reference timbre features, reference prosody features, historical key-value pairs, model version information, and other information, which are not limited by the present application.

[0171] In step 503, based on the first text features, the reference timbre features, and the reference prosody features, the target prosody features corresponding to the text to be synthesized are predicted.

[0172] In some embodiments, the memory deserialization operation is performed on the sound feature file to obtain the reference timbre features and the reference prosody features.

[0173] Specifically, in the sound feature extraction module, the reference timbre features and the reference prosody features in vector representation are converted into binary sound feature files by using a memory serialization operation, so as to facilitate storage or transmission. Correspondingly, in the sound synthesis module, the binary sound feature files are restored into the reference timbre features and the reference prosody features in vector representation by using a memory deserialization operation, and the vector representation contains semantic features and syntactic features of the text to be synthesized, so as to facilitate subsequent processing and analysis to obtain accurate synthesized audio of the text to be synthesized.

[0174] In some embodiments, when the memory deserialization operation is performed on the sound feature file, not only the reference timbre features and the reference prosody features are obtained, but also model version information corresponding to the sound feature extraction module. In order to ensure that the sound feature extraction module and the sound synthesis module correspond to the same version of the sound replication model, after obtaining the model version information corresponding to the sound feature extraction module, it is verified whether the model version information corresponding to the sound feature extraction module is consistent with the model version information corresponding to the sound synthesis module.

[0175] When the model version information corresponding to the sound feature extraction module is consistent with the model version information of the sound synthesis module, the subsequent audio synthesis process is performed.

[0176] When the model version information corresponding to the sound feature extraction module is inconsistent with the model version information of the sound synthesis module, the subsequent audio synthesis process is stopped.

[0177] In the embodiments of the present application, by verifying the consistency of the model version information corresponding to the sound feature extraction module and the model version information corresponding to the sound synthesis module, the problem of misalignment of the sound replication model versions corresponding to the two modules is prevented, thereby improving the accuracy of sound replication.

[0178] In some embodiments, when the memory deserialization operation is performed on the sound feature file, not only the reference timbre features, the reference prosody features and the model version information corresponding to the sound feature extraction module are obtained, but also the historical key-value pairs of the attention mechanism network in the prosody large language model.

[0179] Therefore, when predicting the target prosody features corresponding to the text to be synthesized by using the prosody large language model, the first text features, the reference timbre features and the reference prosody features are used as inputs of the prosody large language model, and the historical key-value pairs are reused to predict the target prosody features associated with the text to be synthesized.

[0180] Specifically, the first text feature, the reference timbre feature, and the reference prosody feature are input into the prosody large language model. On the basis of the first stage of the prosody large language model being executed as described above, the second stage of the prosody large language model is started to be executed. That is, the first text feature, the reference timbre feature, and the reference prosody feature p0 are decoded to obtain the target prosody feature associated with the text to be synthesized, and in the process of predicting the target prosody feature associated with the text to be synthesized, the historical key-value pair of the cached attention mechanism network is reused.

[0181] The historical key-value pair output by the first stage of the prosody large language model is cached by the key-value caching mode, so that in the second stage of the prosody large language model, the historical key-value pair can be directly reused to predict the target prosody feature associated with the text to be synthesized, without the need to repeatedly calculate the historical key-value pair, thereby reducing the amount of calculation and improving the inference efficiency of the prosody large language model.

[0182] In some embodiments, the target prosody feature includes sub-prosody features of N text units in the text to be synthesized, where N is greater than 1. Based on the first text feature, the reference timbre feature, and the reference prosody feature, the sub-prosody features of the N text units in the text to be synthesized are predicted in a self-recurrent manner frame by frame.

[0183] Specifically, the text unit of the text to be synthesized can be a word, a phoneme, etc. The number of text units contained in the text to be synthesized is the same as the number of sub-prosody features contained in the target prosody feature.

[0184] For example, if one word is set as one text unit, and the text to be synthesized is "Today the weather is really good", then the text to be synthesized includes 6 text units, and therefore, the target prosody feature associated with the text to be synthesized obtained by prediction includes 6 sub-prosody features, each of which corresponds to one text unit in the text to be synthesized.

[0185] The self-recurrent manner means that each sub-prosody feature depends on the previously generated sub-prosody feature when generating the sequence of sub-prosody features, which ensures the coherence of the generated sequence of sub-prosody features.

[0186] In some embodiments, the process of predicting the sub-prosody features of the N text units in the text to be synthesized in a self-recurrent manner frame by frame includes the following steps:

[0187] Based on the first text feature, the reference timbre feature, and the reference prosody feature, the sub-prosody feature of the first text unit in the text to be synthesized is predicted.

[0188] For the other N-1 text units in the text to be synthesized, the following operations are performed respectively: based on the first text feature, the reference timbre feature and the sub-prosody feature of the i-th text unit, the sub-prosody feature of the i+1-th text unit is predicted, where i is greater than 0 and less than N.

[0189] Specifically, referring to Figure 8 , the first text unit in the text to be synthesized is predicted based on the first text feature, the reference timbre feature and the reference prosody feature p0 using the Transformer architecture. Then, the second text unit in the text to be synthesized is predicted based on the first text feature, the reference timbre feature and the sub-prosody feature p1 using the Transformer architecture, and so on, until the N-th text unit in the text to be synthesized is predicted based on the first text feature, the reference timbre feature and the sub-prosody feature p(N-1) using the Transformer architecture.

[0190] The sequence of sub-prosody features p1, p2, …, pN is the target prosody feature. Then, the first text feature, the reference timbre feature and the target prosody feature are concatenated to obtain the second concatenated feature.

[0191] In addition, in the process of predicting the sub-prosody feature p1 based on the first text feature, the reference timbre feature and the reference prosody feature p0 using the Transformer architecture, the first text feature, the reference timbre feature and the reference prosody feature p0 are input to calculate the key-value pair KV1 of the attention mechanism network in the Transformer architecture, and the historical key-value pair in the sound feature file is obtained (i.e. the key-value pair KV0 of the attention mechanism network calculated and obtained when predicting the reference prosody feature p0 in the first stage of the prosody large language model). Then, the sub-prosody feature p1 is predicted by combining the key-value pair KV0 and the key-value pair KV1, and the key-value pair KV1 is cached.

[0192] Similarly, in the process of predicting the sub-prosody feature p2 based on the first text feature, the reference timbre feature and the sub-prosody feature p1 using the Transformer architecture, the first text feature, the reference timbre feature and the sub-prosody feature p1 are input to calculate the key-value pair KV2 of the attention mechanism network in the Transformer architecture, and the cached key-value pair KV0 and the key-value pair KV1 are obtained. Then, the sub-prosody feature p2 is predicted by combining the key-value pair KV0, the key-value pair KV1 and the key-value pair KV2, and the key-value pair KV2 is cached. In turn, until the sub-prosody feature pN is obtained.

[0193] It should be noted that in addition to the key-value pair KV0 generated when predicting the reference prosody feature p0, other cached key-value pairs (such as the key-value pair KV1, the key-value pair KV2, …) are associated with the text to be synthesized, and the sound feature file in the present application is applicable to audio synthesis of any text, therefore, the other cached key-value pairs are not saved in the sound feature file, but are cached separately. And after generating the synthesized audio of the text to be synthesized, the other cached key-value pairs can be deleted to save storage resources. Of course, when the present application adopts the autoregressive method to predict the sub-prosody feature sequence frame by frame, only the key-value pair KV0 in the sound feature file can be reused during each frame prediction, and the other key-value pairs are obtained by calculation, which is not limited in the present application.

[0194] In the embodiments of the present application, the autoregressive method is adopted to predict the sub-prosody features of the N text units in the text to be synthesized frame by frame to obtain the target prosody feature, which can capture the dependency between any positions in the input sequence, thereby having better performance when processing long sequences and improving the accuracy and controllability of prosody feature extraction.

[0195] Secondly, in the process of predicting the sub-prosody features of the N text units in the text to be synthesized frame by frame in the autoregressive manner, the key-value pair output by each frame prediction is cached, which maximally reduces the calculation amount caused by repeated calculation of historical key-value pairs, and improves the inference efficiency of the prosody large language model.

[0196] Step 504, based on the first text feature, the reference timbre feature and the target prosody feature, generating the synthesized audio corresponding to the text to be synthesized.

[0197] Specifically, the sound synthesis module further comprises: a duration model, a mel-spectrum decoder and a vocoder.

[0198] In the second stage of the prosody large language model, the first text feature, the reference timbre feature and the target prosody feature are spliced to obtain a second spliced feature. Then the second spliced feature is input into the duration model to obtain the predicted duration of the synthesized audio. Based on the obtained predicted duration, the second spliced feature is expanded to obtain a target expanded feature. Then the target expanded feature is input into the mel-spectrum decoder, and the mel-spectrum decoder decodes the target expanded feature into the mel-spectrum feature of the synthesized audio. Then the mel-spectrum feature of the synthesized audio is input into the vocoder, which is an autoregressive-based neural network vocoder. The vocoder converts the mel-spectrum feature of the synthesized audio into an audio signal (i.e. obtains the synthesized audio).

[0199] For example, referring to Figure 9 , the sound synthesis module comprises: a duration model, a mel-spectrum decoder, a vocoder, a text encoder, a second stage of the prosody large language model, and a memory deserialization module.

[0200] For the speaker voice to be synthesized, the corresponding binary voice feature file is read from the hard disk based on the identification information of the speaker. The voice feature file is subjected to memory deserialization operation by the memory deserialization module to obtain the reference timbre feature, the reference prosody feature p0, the key value cache (i.e. the historical key value pair of the attention mechanism network calculated and obtained when the prosody large language model predicts the reference prosody feature p0 in the first stage), and the model version information of the voice feature extraction module.

[0201] First, check whether the model version information of the voice feature extraction module and the model version information of the voice synthesis module are consistent; if consistent, perform text encoding on the text to be synthesized by the text encoder to obtain the text feature of the text to be synthesized.

[0202] The reference timbre feature, the reference prosody feature p0, the key value cache, and the text feature of the text to be synthesized are input into the prosody large language model.

[0203] In the second stage of the prosody large language model, based on the reference timbre feature, the reference prosody feature p0, and the text feature of the text to be synthesized, the sub-prosody features p1, p2, …, pN are predicted frame by frame using the autoregressive method with the key value cache of the attention mechanism network to obtain the target prosody feature. Then, the text feature of the text to be synthesized, the reference timbre feature, and the target prosody feature are spliced to obtain the second spliced feature.

[0204] The duration model and the mel-spectrum decoder are used to expand the second spliced feature, and the target expanded feature obtained by expansion is decoded into the mel-spectrum feature of the synthesized audio. Then, the mel-spectrum feature of the synthesized audio is input into the vocoder, and the vocoder converts the mel-spectrum feature of the synthesized audio into the synthesized audio.

[0205] In the embodiments of the present application, the reference audio is pre-processed to obtain the reference timbre feature and the reference prosody feature, and then based on the reference timbre feature and the reference prosody feature, the voice feature file of the reference audio is obtained and saved. In this way, when the reference audio is input for multiple audio synthesis, each audio synthesis only needs to read the voice feature file of the reference audio, and based on the first text feature of the text to be synthesized and the voice feature file, the synthesized audio corresponding to the text to be synthesized is generated, without the need to repeatedly read and calculate the reference audio, which effectively reduces the time consumption and resource consumption of voice replication, and also effectively reduces the waiting time for synthesizing audio using the voice replication model, thereby improving the use experience and enhancing the controllability of the system.

[0206] Secondly, the application extracts features of the reference audio in advance, obtains a sound feature file of the reference audio, and saves the sound feature file. Compared with directly saving the reference audio, the sound features are saved in the form of a feature file, which not only reduces the consumption of storage resources and computing resources, but also avoids the risk of privacy leakage.

[0207] In addition, the application separates the process of extracting the sound feature file of the reference audio and the process of generating the synthesized audio corresponding to the to-be-synthesized text into two independent processes, thereby decoupling the sound feature file extraction and the audio synthesis. In this way, each audio synthesis no longer depends on reading the reference audio and performing related calculations on the reference audio, but only needs to directly read the sound feature file, thereby effectively improving the convenience of model deployment and reducing the time consumption of the model in the sound synthesis stage.

[0208] To better explain the embodiments of the application, a sound replication method provided by the embodiments of the application is introduced below in a map navigation scenario. The flow of the method can be executed by a server. After obtaining a sound replication model through training, the sound replication model is separated into a sound feature extraction module and a sound synthesis module and deployed in the server.

[0209] When synthesizing the audio for the first time, referring to Figure 10A , the terminal device collects 15 seconds of voice of the announcer as reference audio and sends the reference audio to the server. The server inputs the reference audio into the sound feature extraction module. The sound feature extraction module extracts features of the reference audio by using the flow shown in Figure 7 , and obtains a sound feature file of the announcer. The sound feature file of the announcer is saved in a hard disk.

[0210] The server obtains a red light reminder text in a map application, and then reads the sound feature file of the announcer from the hard disk. The red light reminder text and the sound feature file of the announcer are input into the sound synthesis module. The sound synthesis module generates a map navigation voice corresponding to the red light reminder text by using the flow shown in Figure 9 .

[0211] When synthesizing the audio for the second time, referring to Figure 10B , the server obtains a travel reminder text in a map application, and then reads the sound feature file of the announcer from the hard disk. The travel reminder text and the sound feature file of the announcer are input into the sound synthesis module. The sound synthesis module generates a map navigation voice corresponding to the travel reminder text by using the flow shown in Figure 9 .

[0212] Subsequently, the third audio synthesis, the fourth audio synthesis, and the like can be performed in the manner of the second audio synthesis.

[0213] In the embodiments of the present application, the reference audio of the announcer is subjected to feature extraction, and the voice feature file of the announcer is obtained and saved. In this way, when it is necessary to reproduce the map navigation voice of different navigation texts based on the voice features of the announcer, the voice feature file of the announcer can be directly read to synthesize the audio, without the need to repeatedly read the reference audio and repeatedly calculate the reference audio, thus effectively reducing the waiting time for synthesizing the audio using the voice reproduction model, and also being capable of reducing the recording work of the announcer, improving the production efficiency of the announcement content, and increasing the map navigation experience through the tone familiar and loved by the map user.

[0214] In addition, relevant practices show that after the voice reproduction model is disassembled into a voice feature extraction module and a voice synthesis module for on-site deployment, the voice feature extraction module can extract and save the voice features of the speaker within 5 seconds, and the voice synthesis module can synthesize at a speed of outputting 400 words of synthesized audio per second after inputting the to-be-synthesized text, greatly improving the user experience.

[0215] Based on the same technical concept, the embodiments of the present application provide a structural diagram of a voice reproduction device, as shown in Figure 11 The voice reproduction device 1100 includes:

[0216] The acquisition unit 1101 is configured to acquire a to-be-synthesized text, and perform feature extraction on the to-be-synthesized text to obtain first text features;

[0217] The reading unit 1102 is configured to read a voice feature file of a pre-stored reference audio, wherein the voice feature file includes reference tone features and reference prosody features obtained by performing feature extraction on the reference audio in advance;

[0218] The prediction unit 1103 is configured to predict target prosody features corresponding to the to-be-synthesized text based on the first text features, the reference tone features, and the reference prosody features;

[0219] The synthesis unit 1104 is configured to generate synthesized audio corresponding to the to-be-synthesized text based on the first text features, the reference tone features, and the target prosody features.

[0220] Optionally, the target prosody features include sub-prosody features of N text units in the to-be-synthesized text, wherein N is greater than 1.

[0221] The prediction unit 1103 is specifically configured to:

[0222] Based on the first text features, the reference tone features, and the reference prosody features, the sub-prosody features of the N text units in the to-be-synthesized text are predicted frame by frame in an autoregressive manner.

[0223] Optionally, the prediction unit 1103 is specifically configured to:

[0224] predict a sub-prosody feature of a first text unit in the text to be synthesized based on the first text feature, the reference timbre feature and the reference prosody feature;

[0225] for other N-1 text units in the text to be synthesized, the following operations are respectively performed:

[0226] predict a sub-prosody feature of an i+1th text unit based on the first text feature, the reference timbre feature and the sub-prosody feature of the i th text unit, where i is greater than 0 and less than N.

[0227] Optionally, the sound feature file further comprises: a history key-value pair of an attention mechanism network in a prosody large language model;

[0228] Optionally, the acquisition unit 1101 is specifically configured to:

[0229] take the first text feature, the reference timbre feature and the reference prosody feature as inputs of the prosody large language model, and predict a target prosody feature associated with the text to be synthesized by reusing the history key-value pair.

[0230] Optionally, the acquisition unit 1101 is specifically configured to:

[0231] acquire a prosody audio from the reference audio, and extract a mel-spectrum feature of the prosody audio;

[0232] perform prosody coding on the mel-spectrum feature of the prosody audio to obtain a preliminary prosody feature;

[0233] predict the reference prosody feature based on the reference timbre feature, the preliminary prosody feature and a second text feature of a reference text corresponding to the reference audio by the prosody large language model.

[0234] Optionally, the acquisition unit 1101 is further configured to:

[0235] after the prediction of the reference prosody feature based on the reference timbre feature, the preliminary prosody feature and the second text feature of the reference text corresponding to the reference audio by the prosody large language model, cache the history key-value pair of the attention mechanism network in the prosody large language model.

[0236] Optionally, the acquisition unit 1101 is specifically configured to:

[0237] acquire a timbre audio from the reference audio, and extract a mel-spectrum feature of the timbre audio;

[0238] The timbre audio is timbre coded to obtain the reference timbre feature.

[0239] Optionally, the obtaining unit 1101 is specifically configured to:

[0240] The reference timbre feature and the reference prosody feature are subjected to a memory serialization operation to obtain the binary sound feature file.

[0241] The sound feature file is saved in a storage device.

[0242] Optionally, the reading unit 1102 is further configured to:

[0243] After reading the sound feature file corresponding to the reference audio, the sound feature file is subjected to a memory deserialization operation to obtain the reference timbre feature and the reference prosody feature.

[0244] Optionally, the sound feature file is generated by a sound feature extraction module in a sound replication model; the sound replication model further comprises a sound synthesis module for generating the synthesized audio.

[0245] The prediction unit 1103 is further configured to:

[0246] Before predicting the target prosody feature corresponding to the text to be synthesized based on the first text feature, the reference timbre feature and the reference prosody feature, the model version information corresponding to the sound feature extraction module is obtained from the sound feature file, and it is determined that the model version information corresponding to the sound feature extraction module is consistent with the model version information corresponding to the sound synthesis module.

[0247] In the embodiments of the present application, the reference audio is pre-processed for feature extraction to obtain the reference timbre feature and the reference prosody feature. Based on the reference timbre feature and the reference prosody feature, the sound feature file of the reference audio is obtained and saved. Thus, when the reference audio is used as input for multiple audio synthesis, the sound feature file of the reference audio only needs to be read each time the audio is synthesized, and the synthesized audio corresponding to the text to be synthesized is generated based on the first text feature of the text to be synthesized and the sound feature file. It is not necessary to repeatedly read the reference audio and repeatedly calculate the reference audio, which effectively reduces the time consumption and resource consumption of sound replication, and also effectively reduces the waiting time for synthesizing audio using the sound replication model, thereby improving the use experience and enhancing the controllability of the system.

[0248] Secondly, the present application pre-processes the reference audio for feature extraction to obtain the sound feature file of the reference audio and saves it. This way of saving the sound feature in the form of a feature file, compared with directly saving the reference audio, not only reduces the consumption of storage resources and computing resources, but also avoids the risk of privacy leakage.

[0249] In this application embodiment, the terms "module" or "unit" refer to a computer program or part of a computer program that has a predetermined function and works with other related parts to achieve a predetermined goal, and can be implemented wholly or partially using software, hardware (such as processing circuitry or memory), or a combination thereof. Similarly, a processor (or multiple processors or memory) can be used to implement one or more modules or units. Furthermore, each module or unit can be part of an overall module or unit that includes the functionality of that module or unit.

[0250] Based on the same technical concept, embodiments of this application provide a computer device, which can be... Figure 1 The terminal devices and / or servers shown, such as Figure 12 As shown, it includes at least one processor 1201 and a memory 1202 connected to at least one processor. In this embodiment, the specific connection medium between the processor 1201 and the memory 1202 is not limited. Figure 12 Taking the connection between processor 1201 and memory 1202 via a bus as an example, the bus can be divided into address bus, data bus, control bus, etc.

[0251] In this embodiment of the application, the memory 1202 stores instructions that can be executed by at least one processor 1201. By executing the instructions stored in the memory 1202, at least one processor 1201 can perform the steps of the above-described sound replication method.

[0252] The processor 1201 is the control center of the computer device, capable of connecting various parts of the computer device via various interfaces and lines. It achieves sound replication by running or executing instructions stored in the memory 1202 and accessing data stored in the memory 1202. Optionally, the processor 1201 may include one or more processing units. The processor 1201 may integrate an application processor and a modem processor. The application processor primarily handles the operating system, user interface, and applications, while the modem processor primarily handles wireless communication. It is understood that the modem processor may not be integrated into the processor 1201. In some embodiments, the processor 1201 and the memory 1202 may be implemented on the same chip; in other embodiments, they may be implemented on separate chips.

[0253] The processor 1201 can be a general processor, such as a central processing unit (CPU), a digital signal processor, an application specific integrated circuit (ASIC), a field programmable gate array or other programmable logic device, a discrete gate or transistor logic component, a discrete hardware component, and can implement or execute the methods, steps and logical block diagrams disclosed in the embodiments of the present application. The general processor can be a microprocessor or any conventional processor. The steps of the method disclosed in the embodiments of the present application can be directly embodied as hardware processor execution, or executed by a combination of hardware and software modules in the processor.

[0254] The memory 1202 is a non-volatile computer readable storage medium, which can be used to store non-volatile software programs, non-volatile computer executable programs and modules. The memory 1202 can include at least one type of storage medium, such as flash memory, hard disk, multimedia card, card type memory, random access memory (RAM), static random access memory (SRAM), programmable read only memory (PROM), read only memory (ROM), electrically erasable programmable read only memory (EEPROM), magnetic storage, magnetic disk, optical disk, etc. The memory 1202 is any other medium capable of carrying or storing desired program code in the form of instructions or data structures and capable of being accessed by a computer device, but is not limited thereto. The memory 1202 in the embodiments of the present application can also be a circuit or any other device capable of realizing a storage function, used for storing program instructions and / or data.

[0255] Based on the same inventive concept, the embodiments of the present application provide a computer readable storage medium storing a computer program executable by a computer device, which, when executed on the computer device, causes the computer device to perform the steps of the sound replication method.

[0256] Based on the same inventive concept, the embodiments of the present application provide a computer program product, which includes a computer program stored on a computer readable storage medium, the computer program including program instructions, which, when executed by a computer device, cause the computer device to perform the steps of the sound replication method.

[0257] Those skilled in the art will understand that embodiments of the present invention can be provided as methods or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0258] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer apparatus or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0259] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer device or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0260] These computer program instructions may also be loaded onto a computer device or other programmable data processing equipment to cause a series of operational steps to be performed on the computer device or other programmable equipment to produce a process implemented by the computer device, thereby providing instructions that execute on the computer device or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0261] Although preferred embodiments of the invention have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including both the preferred embodiments and all changes and modifications falling within the scope of the invention.

[0262] It will be apparent to those skilled in the art that various modifications and variations can be made to the present application without departing from the spirit or scope of the application. Thus, it is intended that the present application cover modifications and variations of this application provided they come within the scope of the appended claims and their equivalents.

Claims

1. A sound reproduction method, characterized by, The method comprises: obtaining a text to be synthesized, and performing feature extraction on the text to be synthesized to obtain first text features; reading a pre-stored sound feature file of a reference audio, the sound feature file containing reference timbre features and reference prosody features obtained by performing feature extraction on the reference audio in advance; based on the first text features, the reference timbre features and the reference prosody features, predicting target prosody features corresponding to the text to be synthesized; based on the first text features, the reference timbre features and the target prosody features, generating synthesized audio corresponding to the text to be synthesized.

2. The method of claim 1, wherein, The target prosody features include sub-prosody features of N text units in the text to be synthesized, where N is greater than 1; The method of predicting target prosody features corresponding to the text to be synthesized based on the first text features, the reference timbre features and the reference prosody features comprises: based on the first text features, the reference timbre features and the reference prosody features, predicting sub-prosody features of N text units in the text to be synthesized in an autoregressive manner frame by frame.

3. The method of claim 2, wherein, The method of predicting sub-prosody features of N text units in the text to be synthesized in an autoregressive manner frame by frame based on the first text features, the reference timbre features and the reference prosody features comprises: based on the first text features, the reference timbre features and the reference prosody features, predicting sub-prosody features of a first text unit in the text to be synthesized; for other N-1 text units in the text to be synthesized, the following operations are performed respectively: based on the first text features, the reference timbre features and the sub-prosody features of the i-th text unit, predicting sub-prosody features of the i+1-th text unit, where i is greater than 0 and less than N.

4. The method of claim 1, wherein, The sound feature file further comprises a history key-value pair of an attention mechanism network in a prosody large language model; The method of predicting target prosody features corresponding to the text to be synthesized based on the first text features, the reference timbre features and the reference prosody features comprises: taking the first text features, the reference timbre features and the reference prosody features as inputs of the prosody large language model, and predicting target prosody features associated with the text to be synthesized by reusing the history key-value pair.

5. The method of claim 4, wherein, The reference prosody features of the reference audio are obtained in the following manner: obtaining prosody audio from the reference audio, and extracting mel-spectrogram features of the prosody audio; performing prosody coding on the mel-spectrogram features of the prosody audio to obtain preliminary prosody features; based on the reference timbre features, the preliminary prosody features and second text features of a reference text corresponding to the reference audio, predicting the reference prosody features through the prosody large language model.

6. The method of claim 5, wherein, After predicting the reference prosody features through the prosody large language model based on the reference timbre features, the preliminary prosody features and the second text features of a reference text corresponding to the reference audio, the method further comprises: caching the history key-value pair of the attention mechanism network in the prosody large language model.

7. The method of claim 5, wherein, The reference timbre feature of the reference audio is obtained in the following manner: Obtaining a timbre audio from the reference audio, and extracting a mel-spectrum feature of the timbre audio; Timbre encoding the mel-spectrum feature of the timbre audio to obtain the reference timbre feature.

8. The method of claim 7, wherein, The sound feature file is obtained in the following manner: Performing an in-memory serialization operation on the reference timbre feature and the reference prosody feature to obtain a binary sound feature file; Saving the sound feature file in a storage device.

9. The method of claim 1, wherein, After reading the sound feature file corresponding to the reference audio, the following steps are further included: Performing an in-memory deserialization operation on the sound feature file to obtain the reference timbre feature and the reference prosody feature.

10. The method of any one of claims 1 to 9, wherein, The sound feature file is generated by a sound feature extraction module in a sound replication model; the sound replication model further includes a sound synthesis module for generating the synthesized audio; Before predicting the target prosody feature corresponding to the text to be synthesized based on the first text feature, the reference timbre feature, and the reference prosody feature, the following steps are further included: Obtaining model version information corresponding to the sound feature extraction module from the sound feature file, and determining that the model version information corresponding to the sound feature extraction module is consistent with the model version information corresponding to the sound synthesis module.

11. A sound reproduction apparatus, characterized by comprising: It includes: An acquisition unit is configured to acquire text to be synthesized and perform feature extraction on the text to be synthesized to obtain a first text feature; A reading unit is configured to read a sound feature file of a pre-stored reference audio, wherein the sound feature file contains a reference timbre feature and a reference prosody feature obtained by pre-performing feature extraction on the reference audio; A prediction unit is configured to predict a target prosody feature corresponding to the text to be synthesized based on the first text feature, the reference timbre feature, and the reference prosody feature; A synthesis unit is configured to generate synthesized audio corresponding to the text to be synthesized based on the first text feature, the reference timbre feature, and the target prosody feature.

12. A computer device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, The processor executes the computer program to implement the steps of the method of any one of claims 1-10.

13. A computer-readable storage medium, characterized in that, The computer program product includes a computer program stored on a computer-readable storage medium, and the computer program includes program instructions, which, when executed by a computer device, cause the computer device to perform the steps of the method of any one of claims 1-10.

14. A computer program product, characterised in that, The computer program product includes a computer program stored on a computer-readable storage medium, and the computer program includes program instructions, which, when executed by a computer device, cause the computer device to perform the steps of the method of any one of claims 1-10.