Speech synthesis method and device, model training method and device, equipment, medium and program product

Abstract

The invention provides a speech synthesis method and device, a model training method and device, equipment, a medium and a program product, and relates to the technical field of artificial intelligence such as generative large language models, knowledge distillation, model distillation and speech synthesis. The method comprises the following steps: acquiring an output vector generated by a pre-trained teacher speech synthesis model for a first training sample; taking the output vector as supervision information, taking the first training sample as a training set, and training the multi-level features of the initial student model through the multi-level features in the teacher speech synthesis model to obtain a student speech synthesis model; wherein each level feature in the teacher speech synthesis model is used for training at least one level feature of which the level is not higher than the level feature in the initial student model. According to the embodiment of the invention, the storage requirement of the model can be obviously reduced, the computing resources required by the student model are correspondingly reduced, the student model is easier to deploy in a resource-limited environment, and the response speed of the application and the user experience are improved.

Description

Technical Field

[0001] The present disclosure relates to the field of computer technology, specifically to the field of artificial intelligence technology such as generative large language models, knowledge distillation, model distillation, speech synthesis, and more particularly to a speech synthesis, model training method, device, equipment, medium and program product. Background Art

[0002] The voice question-answering system is an important branch of the field of artificial intelligence. It understands and analyzes user questions, extracts relevant information from a large amount of data, and answers the user's questions. The emergence of large language models provides a new solution for voice question-answering systems. By learning from a large amount of text data, large language models can capture the deep structure and semantic information of the language, so as to understand the user's questions more accurately. However, the existing large language model solution for voice question-answering still has problems such as large resource usage and slow processing response. Summary of the invention

[0003] The embodiments of the present disclosure provide a speech synthesis, model training method, device, equipment, medium and program product.

[0004] In the first aspect, the embodiments of the present disclosure propose a method for training a speech synthesis model, including: obtaining an output vector generated by a pre-trained teacher speech synthesis model for a first training sample; using the output vector as supervision information and the first training sample as a training set, training the multi-level features of an initial student model through the multi-level features in the teacher speech synthesis model to obtain a student speech synthesis model; wherein each level feature in the teacher speech synthesis model is used to train at least one level feature in the initial student model whose level is not higher than that level feature.

[0005] In a second aspect, an embodiment of the present disclosure proposes a speech synthesis method, comprising: obtaining a reference speech for indicating the sound characteristics of a target speech; generating a target speech matching the sound characteristics based on the reference speech through a pre-trained student speech synthesis model; wherein the student speech synthesis model is trained by the training method of the speech synthesis model described in any implementation method of the first aspect.

[0006] In the third aspect, the embodiments of the present disclosure propose a training device for a speech synthesis model, comprising: an output vector acquisition module, configured to obtain the output vector generated by a pre-trained teacher speech synthesis model for a first training sample; a model training module, configured to use the output vector as supervision information and the first training sample as a training set, and train the multi-level features of the initial student model through the multi-level features in the teacher speech synthesis model to obtain a student speech synthesis model; wherein each level feature in the teacher speech synthesis model is used to train at least one level feature in the initial student model whose level is not higher than that level feature.

[0007] In a fourth aspect, an embodiment of the present disclosure proposes a speech synthesis device, comprising: a reference speech acquisition module, configured to acquire a reference speech indicating the sound characteristics of a target speech; a speech synthesis module, configured to generate a target speech matching the sound characteristics based on the reference speech through a pre-trained student speech synthesis model; wherein the student speech synthesis model is trained by the training method of the speech synthesis model described in any implementation method of the first aspect.

[0008] In a fifth aspect, an embodiment of the present disclosure provides an electronic device, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that when the at least one processor executes, it is able to implement the training method of the speech synthesis model described in any implementation manner in the first aspect, or implement the speech synthesis method described in any implementation manner in the second aspect.

[0009] In a sixth aspect, an embodiment of the present disclosure provides a non-transitory computer-readable storage medium storing computer instructions, which are used to enable a computer to implement a training method for a speech synthesis model as described in any implementation method in the first aspect, or to implement a speech synthesis method as described in any implementation method in the second aspect when executed.

[0010] In the seventh aspect, an embodiment of the present disclosure provides a computer program product comprising a computer program, which, when executed by a processor, can implement the training method of a speech synthesis model as described in any implementation manner in the first aspect, or implement the speech synthesis method as described in any implementation manner in the second aspect.

[0011] The disclosed embodiment trains the student model by using the trained teacher speech synthesis model, compresses the complex large model into a smaller model, and significantly reduces the storage requirements of the model. Since the student model is smaller in size, the required computing resources (such as CPU, memory, and power consumption) are also reduced accordingly, making the model easier to deploy in a resource-constrained environment. Since the student model has a simple structure, its reasoning speed is usually faster, which helps to improve the response speed of the application and user experience. In addition, by training the multi-level features of the student model through the multi-level features of the teacher speech synthesis model, the student model can learn the multi-level features of the teacher speech synthesis model more comprehensively, which can accelerate convergence and further improve training efficiency.

[0012] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] Other features, objects and advantages of the present disclosure will become more apparent from the detailed description of non-limiting embodiments made with reference to the following drawings:

[0014] Figure 1 is an exemplary system architecture in which the present disclosure may be applied;

[0015] Figure 2 A flowchart of a method for training a speech synthesis model provided in an embodiment of the present disclosure;

[0016] Figure 3 A schematic diagram of the training process of the teacher speech synthesis model and the initial student model provided in the embodiment of the present disclosure;

[0017] Figure 4 A flowchart of a speech synthesis method provided by an embodiment of the present disclosure;

[0018] Figure 5 A flowchart of intelligent question-answering of user-asked questions provided by an embodiment of the present disclosure;

[0019] Figure 6 A flowchart of searching a pre-stored knowledge base for knowledge data matching an intention recognition result based on an intention recognition result provided in an embodiment of the present disclosure;

[0020] Figure 7 A flowchart of a method for training a speech synthesis model in an application scenario provided by an embodiment of the present disclosure;

[0021] Figure 8 A structural block diagram of a training device for a speech synthesis model provided in an embodiment of the present disclosure;

[0022] Fig. 9 A structural block diagram of a speech synthesis device provided by an embodiment of the present disclosure;

[0023] Fig.10 A schematic diagram of the structure of an electronic device suitable for executing a training method for a speech synthesis model provided in an embodiment of the present disclosure. DETAILED DESCRIPTION

[0024] The following is a description of exemplary embodiments of the present disclosure in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Similarly, for clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description. It should be noted that the embodiments in the present disclosure and the features in the embodiments may be combined with each other without conflict.

[0025] In the technical solution of the present disclosure, the collection, storage, use, processing, transmission, provision and disclosure of user personal information involved are in compliance with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0026] Figure 1 An exemplary system architecture 100 is shown to which embodiments of the method, apparatus, electronic device, and computer-readable storage medium for training a speech synthesis model of the present disclosure can be applied.

[0027] like Figure 1 As shown, the system architecture 100 may include terminal devices 101, 102, 103, a network 104 and a server 105. The network 104 is used to provide a medium for communication links between the terminal devices 101, 102, 103 and the server 105. The network 104 may include various connection types, such as wired, wireless communication links or optical fiber cables, etc.

[0028] Users can use terminal devices 101, 102, 103 to interact with server 105 through network 104 to receive or send messages, etc. Terminal devices 101, 102, 103 and server 105 may be installed with various applications for realizing information communication between the two, such as instant messaging applications.

[0029] Terminal devices 101, 102, 103 and server 105 can be hardware or software. When terminal devices 101, 102, 103 are hardware, they can be various electronic devices with display screens, including but not limited to smart phones, tablet computers, laptop computers, desktop computers, etc.; when terminal devices 101, 102, 103 are software, they can be installed in the electronic devices listed above, which can be implemented as multiple software or software modules, or as a single software or software module, and no specific limitation is made here. When server 105 is hardware, it can be implemented as a distributed server cluster composed of multiple servers, or as a single server; when the server is software, it can be implemented as multiple software or software modules, or as a single software or software module, and no specific limitation is made here.

[0030] The server 105 can provide various services through various built-in applications. It should be noted that the data or information obtained by the server 105 can be obtained from the terminal devices 101, 102, 103 through the network 104, or can be pre-stored in the server 105 in various ways. Therefore, when the server 105 detects that the data has been stored locally, it can choose to directly obtain the data locally. In this case, the exemplary system architecture 100 may also not include the terminal devices 101, 102, 103 and the network 104.

[0031] Since the process of model training may require more computing resources and stronger computing power, the training method of the speech synthesis model provided in the subsequent embodiments of the present disclosure is generally performed by the server 105 with stronger computing power and more computing resources. Accordingly, the training device of the speech synthesis model is generally also set in the server 105. However, it should also be pointed out that when the terminal devices 101, 102, and 103 also have computing power and computing resources that meet the requirements, the terminal devices 101, 102, and 103 can also complete the above-mentioned various operations that are handed over to the server 105 through the communication interaction application installed thereon, and then output the same result as the server 105. Especially in the case where there are multiple terminal devices with different computing powers at the same time, but if the terminal device has stronger computing power and more remaining computing resources, the terminal device can be allowed to perform the above-mentioned operations, thereby appropriately reducing the computing pressure of the server 105. Accordingly, the training device of the speech synthesis model can also be set in the terminal devices 101, 102, and 103. In this case, the exemplary system architecture 100 may also not include the server 105 and the network 104.

[0032] It should be understood that Figure 1The number of terminal devices, networks and servers in the embodiment is only for illustration. Any number of terminal devices, networks and servers may be provided according to implementation requirements.

[0033] Please refer to Figure 2 , Figure 2 A flowchart of a method for training a speech synthesis model provided in an embodiment of the present disclosure, wherein process 200 includes the following steps:

[0034] Step 201: Obtain an output vector generated by a pre-trained teacher speech synthesis model for a first training sample.

[0035] This step is intended to be performed by the execution subject (e.g. Figure 1 The server 105 shown in the figure obtains the output vector generated by the pre-trained teacher speech synthesis model for the first training sample. In this embodiment, the initial student model is trained mainly by training the student model through the teacher model. This training method can also be called knowledge distillation. Knowledge distillation refers to using the teacher model to guide the student model training, and allowing the student model to learn the knowledge of the teacher model through distillation. Among them, the teacher model is a complex model trained in advance, and the student model is a smaller model. Under the same data, the trained teacher model guides the student model to learn by taking the predicted value of the teacher model for the first training sample as the prediction target of the student model. This predicted value generally refers to the class probability output by the teacher network. The parameter scale of the teacher model is large and can express better generalization ability. The parameter scale of the student model is small. Through the guidance of the teacher model, the student model learns the generalization ability of the teacher model to reach or match the accuracy of the teacher model. The output vector of the teacher model is the final representation after the teacher model processes the input data, which contains the model's understanding and prediction of the input data. The teacher model usually maps the input data to a high-dimensional space through a series of calculations and transformations to form an output vector. In the teacher model, the output vector specifically refers to the prediction result or feature representation obtained by the model after a series of calculations and transformations on the input data (such as images, text, etc.). These results are usually presented in the form of a vector, in which each element represents the model's prediction or representation of the input data in a specific aspect (such as category probability, feature importance, etc.). In knowledge distillation technology, the output vector of the teacher model is particularly important because it not only contains the model's deep understanding of the input data, but also serves as a supervisory signal to guide the training of the student model. By imitating the output vector of the teacher model, the student model learns its knowledge and behavior, thereby improving its own performance and generalization ability.

[0036] In the present embodiment, the first training sample includes a speech segment of any length, and the speech segment has the sound characteristics of the speech (such as timbre, pitch, loudness, etc.). The pre-trained teacher speech synthesis model is a trained model for speech timbre synthesis, which can generate a new speech content based on the sound characteristics of the first training sample. In some optional implementations, the speech content can be a speech of random content, but the sound characteristics are consistent with the sound characteristics of the first training sample. In other optional implementations, the speech content can be a speech generated based on some text information content, and the sound characteristics are consistent with the sound characteristics of the first training sample. The output vector of the teacher speech synthesis model is the output result generated based on the first training sample.

[0037] Step 202: Using the output vector as supervision information and the first training sample as a training set, the multi-level features of the initial student model are trained through the multi-level features in the teacher speech synthesis model to obtain a student speech synthesis model.

[0038] This step is intended to train the initial student model by the above-mentioned execution subject using the output vector as the supervision information and the first training sample as the training set to obtain a trained student speech synthesis model. Among them, the supervision information refers to the "knowledge" learned by the teacher model, which helps the student model to imitate the behavior of the teacher model. These supervision information usually include the output probability distribution of the teacher model, the intermediate layer features, etc. In this embodiment, even if the output vector of the teacher speech synthesis model is used as the supervision information, the initial student model is trained with the first training sample as the training set. In some optional implementations of the disclosed embodiments, the teacher speech synthesis model and the initial student model include multi-level features, and the multi-level features of the initial student model are trained by the multi-level features in the teacher speech synthesis model, and each level feature in the teacher speech synthesis model is used to train at least one level feature in the initial student model whose level is not higher than that of the level feature. Exemplarily, both the teacher speech synthesis model and the initial student model contain deep features and shallow features. The shallow features refer to the features extracted by the lower-level network in the speech synthesis model, which are usually closely related to the original attributes of the input speech signal and mainly capture the basic elements in the speech signal, such as frequency, amplitude, waveform, etc. Deep features refer to the features extracted by the higher-level network in the speech model, representing a high-level abstract representation of the speech signal. Deep features are highly abstract representations of speech signals, usually associated with specific speech content or intent. Deep features are rich in semantic information and can capture semantic structures and meanings in speech signals. During the training process, at least one of the deep features and shallow features of the initial student model can be trained by the deep features of the teacher speech synthesis model, and the shallow features of the initial student model can also be trained by the shallow features of the teacher speech synthesis model.

[0039] The training method of the speech synthesis model provided in the embodiment of the present disclosure trains the student model by using the trained teacher speech synthesis model, compresses the complex large model into a smaller model, and significantly reduces the storage requirements of the model. Since the student model is smaller in size, the required computing resources (such as CPU, memory, and power consumption) are also reduced accordingly, making the model easier to deploy in a resource-constrained environment. Since the student model has a simple structure, its reasoning speed is usually faster, which helps to improve the response speed of the application and user experience. In addition, by training the multi-level features of the student model through the multi-level features of the teacher speech synthesis model, the student model can learn the multi-level features of the teacher speech synthesis model more comprehensively, which can accelerate convergence and further improve training efficiency.

[0040] In some optional implementations of the disclosed embodiments, the teacher speech synthesis model and the initial student model both contain three levels of features: deep features, middle features, and shallow features. The shallow features refer to the features extracted by the lower-level networks in the speech synthesis model, which are usually closely related to the original attributes of the input speech signal and mainly capture the basic elements in the speech signal, such as frequency, amplitude, waveform, etc. The middle-level features refer to the features extracted by the networks at the middle level, which are between the shallow features and the deep features. Compared with the shallow features, the middle-level features begin to show a certain degree of abstraction and can capture more complex and abstract patterns in the speech signal. The middle-level features contain some semantic information, such as the representation of basic speech units such as phonemes and syllables. The middle-level features play a connecting role, containing the basic information of the shallow features and providing necessary support for the extraction of the deep features. The deep features refer to the features extracted by the higher-level networks in the speech model, representing the high-level abstract representation of the speech signal. The deep features are highly abstract representations of the speech signal, which are usually associated with specific speech content or intent. Deep features are rich in semantic information and can capture the semantic structure and meaning of speech signals. In the process of training the multi-level features of the teacher speech synthesis model with the multi-level features of the initial student model, the multi-level features of the teacher speech synthesis model are mainly used to train one or more layers of features in the initial student model whose level is not higher than that of the features, such as Figure 3 As shown, the training process may include the following:

[0041] As one implementation method, the shallow features of the teacher speech synthesis model can be used to train the shallow features of the initial student model so that the shallow features of the initial student model can learn the basic elements extracted by the teacher speech synthesis model, such as frequency, amplitude, waveform and other features.

[0042] As another implementation method, the middle-level features of the teacher speech synthesis model can be used to train the shallow features of the initial student model, so that the shallow features of the initial student model can learn the semantic information extracted by the teacher speech synthesis model, such as the representation of basic speech units such as phonemes and syllables, so that the initial student model can integrate the information of the middle-level features while learning the shallow features, thereby improving the learning ability of the initial student model and improving the student model's understanding and processing capabilities of the input vector, thereby improving its performance.

[0043] As another implementation method, the mid-level features of the teacher speech synthesis model can be used to train the mid-level features of the initial student model, so that the mid-level features of the initial student model can learn the semantic information extracted by the teacher speech synthesis model, such as the representation of basic speech units such as phonemes and syllables, thereby improving the learning ability of the initial student model, enabling the student model to learn the corresponding feature information more quickly, and improving the student model's understanding and processing capabilities of the input vector, thereby improving its performance.

[0044] As another implementation, the middle-level features of the teacher speech synthesis model can be used to train the shallow features and middle-level features of the initial student model, so that the shallow features of the initial student model learn the basic elements extracted by the teacher speech synthesis model, such as shallow features such as frequency, amplitude, waveform, etc., and the middle-level features of the initial student model learn the semantic information extracted by the teacher speech synthesis model, such as the representation of basic speech units such as phonemes and syllables, so that the initial student model integrates the information of the middle-level features while learning the shallow features, improves the learning ability of the initial student model, and improves the student model's understanding and processing ability of the input vector, thereby improving its performance. Exemplarily, in the training process, the middle-level features of the teacher speech synthesis model can be used to allocate the features used to train the shallow features of the initial student model and the middle-level features of the initial student model according to the weight distribution, that is, the first part of the middle-level features are used to train the shallow features of the initial student model, and the second part of the middle-level features of the teacher speech synthesis model are used to train the middle-level features of the initial student model, and the first part of the middle-level features are more than the second part of the middle-level features.

[0045] As another implementation, the deep features of the teacher speech synthesis model can be used to train the deep features of the initial student model so that the deep features of the initial student model can learn features such as the semantic structure and meaning of the speech signal extracted by the teacher speech synthesis model.

[0046] As another implementation method, the deep features of the teacher speech synthesis model can be used to train the middle-level features of the initial student model, so that the middle-level features of the initial student model can learn features such as the semantic structure and meaning of the speech signal extracted by the teacher speech synthesis model, so that the initial student model can integrate the information of the deep features while learning the middle-level features, thereby improving the learning ability of the initial student model and improving the student model's understanding and processing capabilities of the input vector, thereby improving its performance.

[0047] As another implementation, the deep features of the teacher speech synthesis model can be used to train the middle-level features and deep features of the initial student model, so that the middle-level features of the initial student model learn the representation of basic speech units such as phonemes and syllables extracted by the teacher speech synthesis model, and the deep features of the initial student model learn the features such as semantic structure and meaning extracted by the teacher speech synthesis model, so that the initial student model integrates the information of the deep features while learning the middle-level features, improves the learning ability of the initial student model, and improves the student model's understanding and processing ability of the input vector, thereby improving its performance. Exemplarily, in the training process, the deep features of the teacher speech synthesis model can be used to allocate the features used to train the middle-level features of the initial student model and the deep features of the initial student model according to the weight distribution, that is, the first part of the deep features are used to train the middle-level features of the initial student model, and the second part of the deep features of the teacher speech synthesis model are used to train the deep features of the initial student model, and the first part of the deep features are more than the second part of the deep features.

[0048] As another implementation, the deep features of the teacher speech synthesis model can be used to train the middle-level features and shallow features of the initial student model, so that the middle-level features and shallow features of the initial student model can learn the features such as the semantic structure and meaning extracted by the teacher speech synthesis model, so that the initial student model integrates the information of the deep features while learning the middle-level features and shallow features, thereby improving the learning ability of the initial student model and improving the student model's understanding and processing capabilities of the input vector, thereby improving its performance. Exemplarily, in the training process, the deep features of the teacher speech synthesis model can be used to allocate the features used to train the middle-level features of the initial student model and the shallow features of the initial student model according to the weight distribution, that is, the first part of the deep features are used to train the middle-level features of the initial student model, and the second part of the deep features of the teacher speech synthesis model are used to train the shallow features of the initial student model, and the first part of the deep features are more than the second part of the deep features.

[0049] As another implementation, the shallow features and deep features of the initial student model can be trained using the deep features of the teacher's speech synthesis model, so that the shallow features of the initial student model learn the features such as the semantic structure and meaning extracted by the teacher's speech synthesis model, and the deep features of the initial student model learn the features such as the semantic structure and meaning extracted by the teacher's speech synthesis model, so that the initial student model integrates the information of the deep features while learning the shallow features, thereby improving the learning ability of the initial student model, and improving the student model's understanding and processing ability of the input vector, thereby improving its performance. Exemplarily, in the training process, the deep features of the teacher's speech synthesis model can be used to allocate the features used to train the shallow features of the initial student model and the deep features of the initial student model according to the weight distribution, that is, the shallow features of the initial student model are trained using the first part of the deep features, and the deep features of the initial student model are trained using the second part of the deep features of the teacher's speech synthesis model, and the second part of the deep features are more than the first part of the deep features.

[0050] As another implementation, the shallow features, middle features and deep features of the initial student model can be trained using the deep features of the teacher's speech synthesis model, so that the shallow features, middle features and deep features of the initial student model correspond to the shallow features, middle features and deep features of the teacher's speech synthesis model, and learn the deep features such as semantic structure and meaning extracted by the teacher's speech synthesis model, so that the initial student model can integrate the information of the deep features of the teacher's speech synthesis model during the learning process, improve the learning ability of the initial student model, and improve the student model's understanding and processing ability of the input vector, thereby improving its performance. Exemplarily, in this training process, the deep features of the teacher's speech synthesis model can be used to allocate the shallow features, middle features and deep features used to train the initial student model according to the weight distribution, that is, the shallow features of the initial student model are trained using the first part of the deep features, the middle features of the initial student model are trained using the second part of the deep features, and the deep features of the initial student model are trained using the third part of the deep features of the teacher's speech synthesis model.

[0051] Please refer to Figure 4 , Figure 4 This is a flow chart of a speech synthesis method provided by an embodiment of the present disclosure, wherein process 400 includes the following steps:

[0052] Step 401: Acquire a reference speech for indicating the sound characteristics of a target speech.

[0053] The reference voice refers to a voice that contains the sound features of the target voice. For example, it can be the voice of a middle-aged man, the voice of a little girl, etc., or the voice of some specific characters or roles, such as the voices of Donald Duck, Crayon Shin-chan, etc. It should be noted that the voices of specific characters or roles used are used with the copyright and consent of the corresponding characters.

[0054] Step 402: Generate a target speech matching the sound features based on the reference speech using a pre-trained student speech synthesis model.

[0055] Wherein, the student speech synthesis model is obtained by training the training method of the speech synthesis model described in any of the above embodiments. Through the student speech synthesis model, a target speech matching the sound feature can be generated based on the sound feature of the target speech. In practical applications, the student speech synthesis model can have many application scenarios, for example, a speech content matching the sound feature of the target speech can be generated for a text content. For example, user A records a section of his own voice audio. After obtaining the authorization of user A, the execution subject can generate a section of voice content matching the voice feature of the user A based on the voice audio provided by user A through the student speech synthesis model. For entertainment purposes, the content information of the voice content can be randomly generated. In some optional embodiments, user A can also input another section of voice content using other voices, and the execution subject can generate the voice content replacing the other voices with the voice of user A through the student speech synthesis model, and the present embodiment is not limited thereto.

[0056] The speech synthesis method of the embodiment of the present disclosure can use a speech synthesis model to perform speech synthesis. The speech synthesis model can continuously improve the efficiency and accuracy of speech synthesis through further model training and optimization, reduce labor costs, and improve work efficiency.

[0057] In some optional implementations of the disclosed embodiments, the execution subject may also obtain target text information, and generate a second target voice matching the sound characteristics of the reference voice based on the target text information and the reference voice through the student voice synthesis model. The target text information may be a text content input by the user. For example, user A inputs the text content of a poem, and the execution subject may generate the voice content of the poem read aloud using the voice of user A through the student voice synthesis model.

[0058] In practical applications, the target text information can also be used to obtain answer information by executing the intelligent question-answering of the user's question information. Figure 5 As shown, the process 500 mainly includes:

[0059] Step 501: Perform intent recognition on the question information input by the user to obtain an intent recognition result. The execution subject can perform intent recognition on the question information raised by the user to obtain an intent recognition result. Intent recognition is a key technology in natural language processing (NLP), which involves identifying and interpreting the purpose or goal of the user in a given statement or question. Through intent recognition, the execution subject can understand and respond to human language in a meaningful and contextual way, which is the basis for achieving effective human-computer interaction in the fields of intelligent dialogue systems, personalized recommendations, smart homes, etc. For example, in an intelligent dialogue system, when a user enters "I want to book a flight from Beijing to Shanghai", the execution subject can classify it into the intent category of "booking a flight" through intent recognition. In this way, the execution subject can provide the user with corresponding flight booking services based on this intent.

[0060] In some optional implementations of the embodiments of the present disclosure, the question information input by the user can be input in a variety of ways, and can be input through input devices such as a keyboard and a handwriting board of a terminal device, or can be input through a voice input device such as a microphone of a terminal device. For information input using a voice input device, the execution subject first obtains the voice information input by the user, performs voice recognition on the voice information, and thus obtains the question information. In practical applications, the execution subject can be a method of efficiently converting sound into accurate text information through Silero Voice Activity Detector (silero vad) pickup technology, a speech-to-text algorithm based on a transformer structure (a relatively mature deep learning model in the relevant field), and sentence segmentation through punctuation marks, laying a solid foundation for subsequent natural language processing links. Among them, the speech-to-text algorithm based on the transformer structure uses 1,000 hours of Mandarin and dialect data training to meet Mandarin and eight common dialect recognition.

[0061] Step 502: Based on the intention recognition result, search the pre-stored knowledge base for knowledge data that matches the intention recognition result.

[0062] The execution subject can query the knowledge data matching the intention recognition result in the pre-stored knowledge base based on the intention recognition result. For example, the execution subject can search the knowledge base through the Retrieval-Augmented Generation (RAG) technology, which can further improve the accuracy and reliability of answering natural language questions.

[0063] Step 503: Generate answer information based on the knowledge data and determine it as target text information.

[0064] A knowledge base is a resource in NLP that is used to store and represent structured or semi-structured knowledge information. The knowledge base provides rich background knowledge for NLP tasks, which helps the system understand the meaning and context of the input text more accurately. By querying and utilizing the information in the knowledge base, the NLP system can perform complex reasoning and decision-making to generate more reasonable and accurate outputs. The information in the knowledge base can help the NLP model better adapt to new fields and new tasks and improve the generalization ability of the NLP model. In some optional implementations of the disclosed embodiments, the pre-stored knowledge base can be constructed by the following steps:

[0065] Step 1: Extract sentences from the historical text data based on the symbol features in the historical text data to obtain multiple sentences. The historical text data is historical information used to construct a knowledge base. For the historical text data, the sentences in the historical text data can be extracted based on the symbol features (such as a period, exclamation mark, line break, tab, page break, etc.) therein to obtain multiple sentences. Furthermore, the extraction can be limited to only based on the period or exclamation mark that is used to represent the end of the sentence, which can improve the integrity of sentence separation and reduce the occurrence of a complete sentence being separated.

[0066] Step 2: Vectorize multiple statements to obtain multiple statement vectors. Specifically, the execution subject can vectorize the statements through Embedding to obtain multiple statement vectors.

[0067] Step 3: Calculate the similarity between adjacent sentence vectors. In this embodiment, the execution subject further integrates the sentence vectors based on the similarity between the sentence vectors. Specifically, the similarity between the sentence vectors can be obtained by calculating cosine similarity, Euclidean distance, etc.

[0068] Step 4: Integrate the sentence vectors whose similarity is not less than the preset threshold to generate an integrated sentence vector, and determine the sentence vectors whose similarity is less than the preset threshold as independent sentence vectors. After calculating the similarity between the sentence vectors, if the similarity between two adjacent sentence vectors is not less than the preset threshold, it indicates that the semantics between the two sentence vectors are relatively similar, and the execution subject can integrate the two sentence vectors to obtain an integrated sentence; if the similarity between two adjacent sentence vectors is less than the preset threshold, it indicates that the semantics between the two sentence vectors are quite different, and the execution subject can split the two sentence vectors as independent sentences.

[0069] Step 5: Constructing a pre-stored knowledge base based on the integrated sentence vector and the independent sentence vector. After the sentence vectors are integrated and separated by similarity, the execution subject can construct the knowledge base based on the integrated sentence and the independent sentence.

[0070] Through the above process, the knowledge base constructed by further integrating the semantically similar sentences can improve the accuracy and efficiency of the query based on the user's intention recognition results when the execution subject queries the knowledge base.

[0071] In the intelligent question-and-answer scenario described in any of the above embodiments, the execution subject can also implement the question-and-answer interaction process with the user by calling the corresponding intelligent agent. Figure 6 , Figure 6 A flowchart of a speech synthesis method provided in an embodiment of the present disclosure, namely for Figure 5 Step 502 in the process 500 shown provides a specific implementation method, and the other steps in the process 500 are not adjusted. The specific implementation method provided in this embodiment is replaced by step 502 to obtain a new complete embodiment. The above step 502, based on the intention recognition result, searches for knowledge data matching the intention recognition result in the pre-stored knowledge base, mainly including:

[0072] Step 601: Obtain preset role information and task information.

[0073] The execution subject can first obtain the preset role information and task information. The role information and task information are used to indicate the role of the agent in the question-and-answer process. The role determines the field and scope of knowledge acquired by the agent, and also determines the characteristics of the agent in the question-and-answer process. For example, the role information and task information configured for the agent obtained by the execution subject are "As an assistant to accompany children, your main task is to meet the various needs of children, including chatting, listening to stories, understanding the weather and playing music. You need to judge the intentions of children based on their questions and give corresponding answers", which indicates that the agent thinks about and answers questions in the role of talking to children.

[0074] Step 602: Based on the role information, task information and intention recognition result, determine an agent that matches the type of the intention recognition result.

[0075] After the execution subject determines the role information and task information of the agent, it can determine the agent that matches the type of intention recognition result. For example, in the above example, the role of the agent is for children, and the tasks include chatting, listening to stories, understanding the weather, and playing music.

[0076] When the intention recognition result for the user contains multiple different types of intention content, the execution subject can also obtain the different types of intention content respectively, and determine the corresponding agents according to the different types of intention content. For example, in the above example, for different types of tasks such as chatting, listening to stories, understanding the weather, and playing music, the chatting agent, storytelling agent, weather analysis agent, and music playing agent can be determined respectively.

[0077] Step 603: Use the intelligent agent to query the pre-stored knowledge base for knowledge data that matches the intention recognition result.

[0078] The execution subject can query the knowledge data matching the intention recognition result in the pre-stored knowledge base through the intelligent agent determined by the above process.

[0079] When the intention recognition result for the user contains multiple different types of intention content, it is necessary to query the knowledge data matching the intention recognition result in the pre-stored knowledge base through the corresponding type of agent. For example, the chat agent is used to query the knowledge base for knowledge data related to the chat context, the storytelling agent is used to query the knowledge base for knowledge data of the story type selected by the user, the weather analysis agent is used to query the knowledge base for weather-related data or information, and the music player agent is used to query the knowledge base for information on related song libraries or singers.

[0080] For the knowledge data obtained by different types of intelligent agents, the execution entity can further integrate the knowledge data, generate a complete answer information, and feed it back to the user.

[0081] To deepen understanding, the present disclosure also provides a specific implementation solution in combination with a specific application scenario, see Figure 7 Flow 700 is shown.

[0082] In this implementation scheme, the intelligent question-and-answer process between the executing subject and the user is used as an example for explanation.

[0083] Step 1: User 71 performs voice input via terminal device 72, and question content 73 is "I want to know what the weather will be like tomorrow?".

[0084] Step 2: The execution body 74 obtains the user's question voice, performs voice recognition on it, and obtains the text information "I want to know what the weather will be like tomorrow."

[0085] Step 3: Based on the text information, perform intent recognition and obtain an intent recognition result, indicating that the user wishes to query weather information 75.

[0086] Step 4: Based on the intention recognition result, the weather agent searches for the corresponding knowledge data in the knowledge base and generates answer information 76. For example, "tomorrow's weather will be cloudy and then sunny, with a northerly wind of 2-3 levels".

[0087] Step 5: The answer information formed can be output through voice. Among them, the sound feature of the acquired target voice is a male middle and high voice. Therefore, through the pre-trained student speech synthesis model, based on the answer information "tomorrow's weather is cloudy and sunny, with a northerly wind of 2-3 levels" and the sound feature "male middle and high voice", speech synthesis is performed, and the voice "tomorrow's weather is cloudy and sunny, with a northerly wind of 2-3 levels" is output in a male middle and high voice, so that the user 71 can hear the answer directly without having to stare at the screen, thereby improving the freedom of content display.

[0088] Further references Figure 8 As an implementation of the methods shown in the above figures, the present disclosure provides an embodiment of a training device for a speech synthesis model. Figure 2 Corresponding to the method embodiment shown, the device can be specifically applied to various electronic devices.

[0089] like Figure 8 As shown, the speech synthesis model training device 800 of this embodiment may include: an output vector acquisition module 801 and a model training module 802. The output vector acquisition module 801 is configured to acquire the output vector generated by the pre-trained teacher speech synthesis model for the first training sample; the model training module 802 is configured to use the output vector as supervision information and the first training sample as a training set, and train the multi-level features of the initial student model through the multi-level features in the teacher speech synthesis model to obtain the student speech synthesis model; each level feature in the teacher speech synthesis model is used to train at least one level feature in the initial student model whose level is not higher than that level feature.

[0090] In the present embodiment, in the training device 800 for the speech synthesis model, the specific processing of the output vector acquisition module 801 and the model training module 802 and the technical effects thereof can be referred to in Figure 2 The relevant descriptions of steps 201 - 202 in the corresponding embodiment are not repeated here.

[0091] In some optional implementations of this embodiment, the model training module 802 is further configured to: in response to the multi-level features including three levels of shallow features, middle features and deep features, use the shallow features of the teacher speech synthesis model to train the shallow features of the initial student model.

[0092] In some optional implementations of this embodiment, the model training module 802 is further configured to: in response to the multi-level features including three levels of shallow features, middle features and deep features, use the middle features of the teacher speech synthesis model to train the shallow features and / or middle features of the initial student model.

[0093] In some optional implementations of this embodiment, the model training module 802 is further configured to: use the first part of the middle-level features of the teacher's speech synthesis model to train the shallow features of the initial student model, and use the second part of the middle-level features of the teacher's speech synthesis model to train the middle-level features of the initial student model, and the first part of the middle-level features are more than the second part of the middle-level features.

[0094] In some optional implementations of this embodiment, the model training module is further configured to: in response to the multi-level features including three levels of shallow features, middle features and deep features, use the deep features of the teacher speech synthesis model to train the middle features and / or deep features of the initial student model.

[0095] In some optional implementations of this embodiment, the model training module is further configured to: use the first part of deep features of the teacher speech synthesis model to train the middle-level features of the initial student model, and use the second part of deep features of the teacher speech synthesis model to train the deep features of the initial student model, and the second part of deep features has more features than the first part of deep features.

[0096] In some optional implementations of this embodiment, the model training module is further configured to: in response to the multi-level features including three levels of shallow features, middle features and deep features, use the deep features of the teacher speech synthesis model to train the middle features and / or deep features, as well as the shallow features of the initial student model.

[0097] This embodiment exists as an apparatus embodiment corresponding to the above-mentioned method embodiment. The training device of the speech synthesis model provided in this embodiment trains the student model through the trained teacher speech synthesis model, compresses the complex large model into a smaller model, and significantly reduces the storage requirements of the model. Due to the small scale of the student model, the required computing resources (such as CPU, memory and power consumption) are also reduced accordingly, making the model easier to deploy in a resource-constrained environment. Since the student model has a simple structure, its reasoning speed is usually faster, which helps to improve the response speed of the application and user experience. In addition, by training the multi-level features of the student model through the multi-level features of the teacher speech synthesis model, the student model can learn the multi-level features of the teacher speech synthesis model more comprehensively, which can accelerate convergence and further improve training efficiency.

[0098] Further references Fig. 9 As an implementation of the methods shown in the above figures, the present disclosure provides an embodiment of a speech synthesis device, which is similar to Figure 4 Corresponding to the method embodiment shown, the device can be specifically applied to various electronic devices.

[0099] like Fig. 9 As shown, the speech synthesis device 900 of this embodiment may include: a reference speech acquisition module 901 and a speech synthesis module 902. The reference speech acquisition module 901 is configured to acquire a reference speech indicating the sound characteristics of the target speech; the speech synthesis module 902 is configured to generate the target speech matching the sound characteristics based on the reference speech through a pre-trained student speech synthesis model; wherein the student speech synthesis model is trained by the speech synthesis model training method described in any of the above embodiments.

[0100] In the present embodiment, in the training device 900 for the speech synthesis model, the specific processing of the reference speech acquisition module 901 and the speech synthesis module 902 and the technical effects thereof can be referred to respectively. Figure 4 The relevant descriptions of steps 401 - 402 in the corresponding embodiment are not repeated here.

[0101] In some optional implementations of this embodiment, the speech synthesis device 900 also includes: a text acquisition module, configured to acquire target text information; the speech synthesis module is further configured to generate a second target speech matching the sound feature based on the target text information and the reference speech through the speech synthesis model.

[0102] In some optional implementations of the present embodiment, the speech synthesis device 900 also includes: an intention recognition module, configured to perform intention recognition on the question information input by the user to obtain an intention recognition result; a knowledge data query module, configured to query the knowledge data matching the intention recognition result in a pre-stored knowledge base based on the intention recognition result; and a text generation module, configured to generate answer information based on the knowledge data and determine it as the target text information.

[0103] In some optional implementations of the present embodiment, the pre-stored knowledge base is constructed by the following steps: extracting sentences in the historical text data based on symbolic features in the historical text data to obtain multiple sentences; vectorizing the multiple sentences to obtain multiple sentence vectors; calculating the similarity between adjacent sentence vectors; integrating the sentence vectors whose similarity is not less than a preset threshold to generate an integrated sentence vector, and determining the sentence vectors whose similarity is less than the preset threshold as independent sentence vectors; and constructing the pre-stored knowledge base based on the integrated sentence vector and the independent sentence vector.

[0104] In some optional implementations of the present embodiment, the knowledge data query module includes: a first information acquisition submodule, configured to acquire preset role information and task information; a first agent determination submodule, configured to determine an agent that matches the type of the intent recognition result based on the role information, task information and intent recognition result; and a first knowledge data query submodule, configured to query knowledge data that matches the intent recognition result in a pre-stored knowledge base through the agent.

[0105] In some optional implementations of the present embodiment, the knowledge data query module includes: a second information acquisition submodule, configured to acquire preset role information and task information; an intention content extraction submodule, configured to extract different types of intention content from the intention recognition result; a second agent determination submodule, configured to determine agents that match the type of each intention content based on the role information, task information and different types of intention content; and a second knowledge data query submodule, configured to acquire knowledge data that matches the intention recognition result by querying a pre-stored knowledge base through each of the agents.

[0106] In some optional implementations of this embodiment, the text generation module is further configured to: integrate the knowledge data obtained by the queries of each of the intelligent agents to obtain the answer information.

[0107] In some optional implementations of this embodiment, the speech synthesis device 900 further includes: a speech information acquisition module configured to acquire speech information input by a user; and a speech recognition module configured to perform speech recognition on the speech information to obtain the question information.

[0108] This embodiment exists as an apparatus embodiment corresponding to the above-mentioned method embodiment. The speech synthesis device provided by this embodiment can use a speech synthesis model for speech synthesis. The speech synthesis model can continuously improve the efficiency and accuracy of speech synthesis through further model training and optimization, reduce labor costs, and improve work efficiency.

[0109] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device, which includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can implement the training method of the speech synthesis model or the speech synthesis method described in any of the above embodiments when executing.

[0110] According to an embodiment of the present disclosure, the present disclosure also provides a readable storage medium, which stores computer instructions, and the computer instructions are used to enable a computer to implement the training method of the speech synthesis model or the speech synthesis method described in any of the above embodiments when executed.

[0111] According to an embodiment of the present disclosure, the present disclosure also provides a computer program product, which, when executed by a processor, can implement the training method of the speech synthesis model or the speech synthesis method described in any of the above embodiments.

[0112] Fig.10 A schematic block diagram of an example electronic device 1000 that can be used to implement an embodiment of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or required herein.

[0113] like Fig.10 As shown, the device 1000 includes a computing unit 1001, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 1002 or a computer program loaded from a storage unit 1008 into a random access memory (RAM) 1003. In the RAM 1003, various programs and data required for the operation of the device 1000 can also be stored. The computing unit 1001, the ROM 1002, and the RAM 1003 are connected to each other via a bus 1004. An input / output (I / O) interface 1005 is also connected to the bus 1004.

[0114] A number of components in the device 1000 are connected to the I / O interface 1005, including: an input unit 1006, such as a keyboard, a mouse, etc.; an output unit 1007, such as various types of displays, speakers, etc.; a storage unit 1008, such as a disk, an optical disk, etc.; and a communication unit 1009, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 1009 allows the device 1000 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0115] The computing unit 1001 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 1001 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 1001 performs the various methods and processes described above, such as a training method or a speech synthesis method for a speech synthesis model. For example, in some embodiments, the training method or the speech synthesis method for a speech synthesis model may be implemented as a computer software program, which is tangibly included in a machine-readable medium, such as a storage unit 1008. In some embodiments, part or all of the computer program may be loaded and / or installed on the device 1000 via the ROM 1002 and / or the communication unit 1009. When the computer program is loaded into the RAM 1003 and executed by the computing unit 1001, one or more steps of the training method or the speech synthesis method for the speech synthesis model described above may be executed. Alternatively, in other embodiments, the computing unit 1001 may be configured to execute the training method of the speech synthesis model or the speech synthesis method in any other appropriate manner (for example, by means of firmware).

[0116] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0117] The program code for implementing the method of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0118] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0119] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0120] The systems and techniques described herein may be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), and the Internet.

[0121] A computer system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The relationship between the client and the server is generated by computer programs running on the corresponding computers and having a client-server relationship with each other. The server may be a cloud server, also known as a cloud computing server or cloud host, which is a host product in the cloud computing service system to solve the defects of difficult management and weak business scalability in traditional physical hosts and virtual private servers (VPS) services.

[0122] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this disclosure can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved, and this document does not limit this.

[0123] The above specific implementations do not constitute a limitation on the protection scope of the present disclosure. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present disclosure shall be included in the protection scope of the present disclosure.

Claims

1. A method for training a speech synthesis model, comprising: Obtain an output vector generated by a pre-trained teacher speech synthesis model for the first training sample; Using the output vector as supervision information and the first training sample as a training set, the multi-level features of the initial student model are trained through the multi-level features in the teacher speech synthesis model to obtain a student speech synthesis model; wherein each level feature in the teacher speech synthesis model is used to train at least one level feature in the initial student model whose level is not higher than that level feature.

2. The training method according to claim 1, wherein: The step of training the multi-level features of the initial student model by using the multi-level features in the teacher speech synthesis model comprises: In response to the multi-level features including shallow features, middle features and deep features, the shallow features of the teacher speech synthesis model are used to train the shallow features of the initial student model; and / or, In response to the multi-level features including shallow features, middle features and deep features, the shallow features and / or middle features of the initial student model are trained using the middle features of the teacher speech synthesis model.

3. The training method according to claim 2, wherein: Using the middle-level features of the teacher speech synthesis model to train the shallow-level features and the middle-level features of the initial student model includes: The shallow features of the initial student model are trained using the first part of the middle-level features of the teacher speech synthesis model, and the middle-level features of the initial student model are trained using the second part of the middle-level features of the teacher speech synthesis model, and the first part of the middle-level features are more than the second part of the middle-level features.

4. The training method according to claim 1, wherein: The step of training the multi-level features of the initial student model by using the multi-level features in the teacher speech synthesis model comprises: In response to the multi-level features including shallow features, middle features and deep features, the middle features and / or deep features of the initial student model are trained using the deep features of the teacher speech synthesis model.

5. The training method according to claim 4, wherein: Using the deep features of the teacher speech synthesis model to train the middle features and deep features of the initial student model includes: The first part of deep features of the teacher speech synthesis model is used to train the middle-level features of the initial student model, and the second part of deep features of the teacher speech synthesis model is used to train the deep features of the initial student model, and the second part of deep features has more features than the first part of deep features.

6. The training method according to any one of claims 1 to 5, wherein: The step of training the multi-level features of the initial student model by using the multi-level features in the teacher speech synthesis model comprises: In response to the multi-level features including shallow features, middle features and deep features, the deep features of the teacher speech synthesis model are used to train the middle features and shallow features of the initial student model; or, In response to the multi-level features including shallow features, middle features and deep features, the deep features and shallow features of the initial student model are trained using the deep features of the teacher speech synthesis model; or, In response to the multi-level features including shallow features, middle features and deep features, the deep features of the teacher speech synthesis model are used to train the middle features, deep features and shallow features of the initial student model.

7. A speech synthesis method, comprising: Acquiring a reference speech for indicating the sound characteristics of the target speech; The target speech matching the sound feature is generated based on the reference speech by a pre-trained student speech synthesis model; wherein the student speech synthesis model is trained by the training method of the speech synthesis model as described in any one of claims 1-6.

8. The method according to claim 7, further comprising: Get target text information; A second target speech matching the sound feature is generated by the speech synthesis model based on the target text information and the reference speech.

9. The method according to claim 8, further comprising: Perform intent recognition on the question information input by the user to obtain the intent recognition result; Based on the intention recognition result, searching a pre-stored knowledge base for knowledge data matching the intention recognition result; Answer information is generated based on the knowledge data and determined as the target text information.

10. The method according to claim 9, wherein: The pre-stored knowledge base is constructed by the following steps: Extracting sentences in the historical text data based on symbol features in the historical text data to obtain multiple sentences; Performing vector representation on the multiple sentences to obtain multiple sentence vectors; Calculating the similarity between adjacent sentence vectors; Integrate the sentence vectors whose similarity is not less than a preset threshold to generate an integrated sentence vector, and determine the sentence vectors whose similarity is less than the preset threshold as independent sentence vectors; The pre-stored knowledge base is constructed based on the integrated sentence vector and the independent sentence vector.

11. The method according to claim 9 or 10, wherein: The step of searching a pre-stored knowledge base for knowledge data matching the intention recognition result based on the intention recognition result includes: Get preset role information and task information; Based on the role information, task information and intention recognition result, determining an agent that matches the type of the intention recognition result; The agent searches a pre-stored knowledge base for knowledge data that matches the intention recognition result.

12. The method according to claim 9 or 10, wherein: The step of searching a pre-stored knowledge base for knowledge data matching the intention recognition result based on the intention recognition result includes: Get preset role information and task information; Extracting different types of intent content from the intent recognition result; Based on the role information, task information and different types of intent content, respectively determine an agent that matches the type of each of the intent contents; Acquire knowledge data matching the intention recognition result by querying the pre-stored knowledge base by each of the intelligent agents.

13. The method according to claim 12, wherein: The generating answer information based on the knowledge data comprises: The knowledge data obtained by the queries of the intelligent agents are integrated to obtain the answer information.

14. The method according to any one of claims 9 to 13, further comprising: Get the voice information input by the user; Perform voice recognition on the voice information to obtain the question information.

15. A training device for a speech synthesis model, comprising: An output vector acquisition module is configured to acquire an output vector generated by a pre-trained teacher speech synthesis model for a first training sample; The model training module is configured to use the output vector as supervision information and the first training sample as a training set, and train the multi-level features of the initial student model through the multi-level features in the teacher speech synthesis model to obtain a student speech synthesis model; wherein each level feature in the teacher speech synthesis model is used to train at least one level feature in the initial student model whose level is not higher than that level feature.

16. The training device according to claim 15, wherein: The model training module is further configured to: In response to the multi-level features including shallow features, middle features and deep features, the shallow features of the teacher speech synthesis model are used to train the shallow features of the initial student model; and / or, In response to the multi-level features including shallow features, middle features and deep features, the shallow features and / or middle features of the initial student model are trained using the middle features of the teacher speech synthesis model.

17. The training device according to claim 16, wherein: The model training module is further configured to: The shallow features of the initial student model are trained using the first part of the middle-level features of the teacher speech synthesis model, and the middle-level features of the initial student model are trained using the second part of the middle-level features of the teacher speech synthesis model, and the first part of the middle-level features are more than the second part of the middle-level features.

18. The training device according to claim 15, wherein: The model training module is further configured to: In response to the multi-level features including shallow features, middle features and deep features, the middle features and / or deep features of the initial student model are trained using the deep features of the teacher speech synthesis model.

19. The training device according to claim 18, wherein: The model training module is further configured to: The first part of deep features of the teacher speech synthesis model is used to train the middle-level features of the initial student model, and the second part of deep features of the teacher speech synthesis model is used to train the deep features of the initial student model, and the second part of deep features has more features than the first part of deep features.

20. The training device according to any one of claims 15 to 19, wherein: The model training module is further configured to: In response to the multi-level features including shallow features, middle features and deep features, the deep features of the teacher speech synthesis model are used to train the middle features and shallow features of the initial student model; or, In response to the multi-level features including shallow features, middle features and deep features, the deep features and shallow features of the initial student model are trained using the deep features of the teacher speech synthesis model; or, In response to the multi-level features including shallow features, middle features and deep features, the deep features of the teacher speech synthesis model are used to train the middle features, deep features and shallow features of the initial student model.

21. A speech synthesis device, comprising: A reference speech acquisition module, configured to acquire a reference speech for indicating the sound characteristics of the target speech; A speech synthesis module is configured to generate the target speech matching the sound feature based on the reference speech through a pre-trained student speech synthesis model; wherein the student speech synthesis model is trained by the training method of the speech synthesis model as described in any one of claims 1 to 6.

22. The apparatus according to claim 21, further comprising: A text acquisition module is configured to acquire target text information; The speech synthesis module is further configured to generate a second target speech matching the sound feature based on the target text information and the reference speech through the speech synthesis model.

23. The apparatus according to claim 22, further comprising: The intention recognition module is configured to perform intention recognition on the question information input by the user and obtain an intention recognition result; a knowledge data query module, configured to query a pre-stored knowledge base for knowledge data matching the intention recognition result based on the intention recognition result; The text generation module is configured to generate answer information based on the knowledge data and determine it as the target text information.

24. The device according to claim 23, wherein: The pre-stored knowledge base is constructed by the following steps: Extracting sentences in the historical text data based on symbol features in the historical text data to obtain multiple sentences; Performing vector representation on the multiple sentences to obtain multiple sentence vectors; Calculating the similarity between adjacent sentence vectors; Integrate the sentence vectors whose similarity is not less than a preset threshold to generate an integrated sentence vector, and determine the sentence vectors whose similarity is less than the preset threshold as independent sentence vectors; The pre-stored knowledge base is constructed based on the integrated sentence vector and the independent sentence vector.

25. The device according to claim 23 or 24, wherein: The knowledge data query module includes: A first information acquisition submodule is configured to acquire preset role information and task information; A first agent determination submodule is configured to determine an agent matching the type of the intention recognition result based on the role information, task information and intention recognition result; The first knowledge data query submodule is configured to query the knowledge data matching the intention recognition result in the pre-stored knowledge base through the intelligent agent.

26. The device according to claim 23 or 24, wherein: The knowledge data query module includes: The second information acquisition submodule is configured to acquire preset role information and task information; an intention content extraction submodule, configured to extract different types of intention contents from the intention recognition result; A second agent determination submodule is configured to determine agents matching the types of the respective intention contents based on the role information, the task information and the different types of intention contents; The second knowledge data query submodule is configured to obtain knowledge data matching the intention recognition result by querying the pre-stored knowledge base through each of the intelligent agents.

27. The device according to claim 26, wherein: The text generation module is further configured to: The knowledge data obtained by the queries of the intelligent agents are integrated to obtain the answer information.

28. The apparatus according to any one of claims 23 to 27, further comprising: A voice information acquisition module is configured to acquire voice information input by a user; The speech recognition module is configured to perform speech recognition on the speech information to obtain the question information.

29. An electronic device comprising: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the training method of the speech synthesis model described in any one of claims 1-6, or so that the at least one processor can execute the speech synthesis method described in any one of claims 7-14.

30. A non-transitory computer-readable storage medium storing computer instructions, wherein the computer instructions are used to cause the computer to execute the training method for a speech synthesis model described in any one of claims 1 to 6, or to cause the computer to execute the speech synthesis method described in any one of claims 7 to 14.

31. A computer program product, comprising a computer program, which, when executed by a processor, implements the steps of the training method of the speech synthesis model according to any one of claims 1 to 6, or implements the steps of the speech synthesis method according to any one of claims 7 to 14.

Images