Display apparatus, server, and speech synthesis method

Abstract

The invention provides a display device, a server and a speech synthesis method. When receiving a voice interaction instruction input by a user, the display device calls a language extraction model to extract word-by-word vectors and whole sentence vectors in a sample text. The language extraction model is a model obtained by training according to a sample text, and a feature tag is arranged in the sample text. Through the feature tag, the model can have the ability of identifying the spoken features in the model training process, so that the spoken features are added in the synthesized speech, and the humanoid degree of the synthesized speech is improved. A speech speed tag vector and a phoneme vector can also be extracted from the sample text and the audio of the sample text. And inputting the word-by-word vector, the whole sentence vector, the speech speed label vector and the phoneme vector into a decoder together to obtain a target synthetic speech, and playing the target synthetic speech by an audio output device.

Description

Technical Field

[0001] The present application relates to the technical field of display devices, and in particular to a display device, a server and a speech synthesis method. Background Art

[0002] Speech synthesis refers to converting text into natural speech and playing it through an audio playback device to achieve voice interaction between humans and machines. Speech synthesis can convert text through a trained model, but due to the limitations of model accuracy and the development of model training methods, the speech synthesized by the model from text has a low similarity to human voice, such as lack of emotion.

[0003] To address the problem of lack of emotion in model-synthesized speech, the quality of synthesized speech can be improved by using models such as Generative Adversarial Networks (GANs). Such models can learn the speech features of human voices from speech data, thereby making the synthesized speech rich in emotion.

[0004] However, in the process of human-computer interaction, human voices also include some colloquial expression habits, such as slips of the tongue, changes in speech speed, long pauses for thinking, and erhua sounds. The model lacks training for colloquial expressions during training, resulting in insufficient anthropomorphism in the synthesized voice. Summary of the invention

[0005] The present application provides a display device, a server and a speech synthesis method to solve the problem that the model lacks training on spoken expression during the training process, resulting in insufficient anthropomorphism of the synthesized speech.

[0006] In a first aspect, the present application provides a display device, comprising: a display; a detector configured to collect voice data input by a user; an audio output device configured to play voice; and a controller, wherein the controller is configured to:

[0007] In response to a voice interaction instruction input by a user, a language extraction model is called to extract word-by-word vectors and whole sentence vectors in a sample text; the language extraction model is a model trained based on the sample text, and the sample text is provided with a feature label;

[0008] Extracting speech rate information from the sample audio according to the speech rate extraction model, and obtaining a speech rate label vector based on the speech rate information; the speech rate label vector is used to characterize the speech rate of the sample audio; the sample audio is audio data generated based on the sample text;

[0009] Converting the sample text into a phoneme sequence, and generating a phoneme vector of the sample text based on the phoneme sequence;

[0010] Inputting the word-by-word vector, the whole sentence vector, the speech rate label vector and the phoneme vector into a decoder to obtain a target synthesized speech;

[0011] The target synthesized speech is played through the audio output device.

[0012] In some feasible embodiments, the feature tags include prosody tags, stress tags, and speech rate tags; the controller is further configured to:

[0013] Setting a rhythm tag, a stress tag and a speech rate tag in the sample text;

[0014] Inputting the sample text into a language extraction model to be trained to obtain a classified word-by-word vector and a classified sentence vector; the classified word-by-word vector and the classified sentence vector include classification labels; the classification labels include classification rhythm labels, classification stress labels, and classification speech rate labels;

[0015] Calculating a first loss value between the prosody label and the classified prosody label, and, calculating a second loss value between the stress label and the classified stress label, and, calculating a third loss value between the speech rate label and the classified speech rate label;

[0016] The model parameters of the language extraction model to be trained are adjusted according to the first loss value, the second loss value, and the third loss value.

[0017] In some feasible embodiments, the language extraction model includes a feature extraction layer and a classification layer; the controller executes inputting the sample text into the language extraction model to be trained to obtain the classified word-by-word vector and the classified sentence vector, and is specifically configured as follows:

[0018] Inputting the sample text into a feature extraction layer of a language extraction model to obtain semantic features, contextual relationship features, vocabulary features, and sentiment features of the sample text;

[0019] The semantic features, contextual relationship features, vocabulary features and sentiment features are input into the classification layer to obtain classified word-by-word vectors and classified whole sentence vectors.

[0020] In some feasible embodiments, the controller extracts speech rate information from the sample audio according to the speech rate extraction model, and obtains a speech rate label vector based on the speech rate information, and is specifically configured as follows:

[0021] Calculate the speech rate of the sample audio using the speech rate extraction model;

[0022] Constructing a normal distribution curve of speech rate according to the speech rate of the sample audio;

[0023] Determine a first speech rate threshold and a second speech rate threshold based on the speech rate normal distribution curve;

[0024] A first speech rate label is set for a keyword whose speech rate is less than or equal to the first speech rate threshold, and a second speech rate label is set for a keyword whose speech rate is greater than the first speech rate threshold and less than a second speech rate threshold, and a third speech rate label is set for a keyword whose speech rate is greater than the second speech rate threshold.

[0025] In some feasible embodiments, the controller executes converting the sample text into a phoneme sequence, and obtaining a phoneme vector of the sample text based on the phoneme sequence, and is specifically configured as follows:

[0026] Using a phoneme model to convert the sample text into a phoneme sequence; the keyword in the sample text corresponds to at least one phoneme;

[0027] Encoding is performed on the phonemes in the phoneme sequence to obtain a phoneme vector.

[0028] In some feasible embodiments, after the controller converts the sample text into a phoneme sequence and obtains a phoneme vector of the sample text based on the phoneme sequence, the controller is further configured to:

[0029] Based on the phoneme sequence, obtaining the number of phonemes corresponding to the keyword;

[0030] Based on the number of phonemes, the word-by-word vector and the speech rate label corresponding to the word are copied so that the word-by-word vector of the keyword is aligned with the phoneme vector of the word.

[0031] In some feasible embodiments, the controller executes inputting the word-by-word vector, the whole sentence vector, the speech rate label vector, and the phoneme vector into the decoder to obtain the target synthesized speech, which is specifically configured as follows:

[0032] Set the historical information extraction threshold;

[0033] The whole sentence vector is divided into a first whole sentence sequence and a second whole sentence sequence according to the historical information extraction threshold; the sequence number of the whole sentence vector in the first whole sentence sequence is greater than the historical information extraction threshold; the sequence number of the whole sentence vector in the second whole sentence sequence is less than or equal to the historical information extraction threshold; the sequence number is used to represent the order of the text corresponding to the whole sentence vector in the sample text;

[0034] Inputting the first entire sentence sequence into a historical information extraction model to obtain a historical information vector;

[0035] The word-by-word vector, the history information vector, the speech rate label vector and the phoneme vector are input into a decoder to obtain a target synthesized speech.

[0036] In some feasible embodiments, the controller executes inputting the word-by-word vector, the history information vector, the speech rate label vector, and the phoneme vector into the decoder to obtain the target synthesized speech, which is specifically configured as follows:

[0037] Accumulating the word-by-word vector, the history information vector, the speech rate label vector, and the phoneme vector to obtain an integrated vector;

[0038] The integrated vector is input into the decoder to obtain the target synthesized speech.

[0039] In a second aspect, the present application provides a server, comprising: a communication unit configured to establish a communication connection with a display device; and a control unit configured to:

[0040] In response to the speech synthesis instruction sent by the display device, a language extraction model is called to extract word-by-word vectors and whole sentence vectors in the sample text; the language extraction model is a model trained based on the sample text; the sample text is provided with a feature label;

[0041] Extracting speech rate information from the sample audio according to the speech rate extraction model, and obtaining a speech rate label vector based on the speech rate information; the speech rate label vector is used to characterize the speech rate in the sample audio; the sample audio is audio data generated based on the sample text;

[0042] Converting the sample text into a phoneme sequence, and obtaining a phoneme vector of the sample text based on the phoneme sequence;

[0043] Inputting the word-by-word vector, the whole sentence vector, the speech rate label vector and the phoneme vector into a decoder to obtain a target synthesized speech;

[0044] Feedback receipt information is fed back to the display device, so that the display device plays the target synthesized speech according to the receipt information.

[0045] In a third aspect, the present application provides a speech synthesis method, applied to the display device described in the first aspect, comprising:

[0046] In response to a voice interaction instruction input by a user, a language extraction model is called to extract word-by-word vectors and whole sentence vectors in a sample text; the language extraction model is a model trained based on the sample text; the sample text is provided with a feature label;

[0047] Extracting speech rate information from the sample audio according to the speech rate extraction model, and obtaining a speech rate label vector based on the speech rate information; the speech rate label vector is used to characterize the speech rate of the sample audio; the sample audio is audio data generated based on the sample text;

[0048] Converting the sample text into a phoneme sequence, and obtaining a phoneme vector of the sample text based on the phoneme sequence;

[0049] Inputting the word-by-word vector, the whole sentence vector, the speech rate label vector and the phoneme vector into a decoder to obtain a target synthesized speech;

[0050] The target synthesized speech is played through the audio output interface.

[0051] In a fourth aspect, the present application provides a speech synthesis method, applied to the server described in the second aspect, comprising:

[0052] In response to the speech synthesis instruction sent by the display device, a language extraction model is called to extract word-by-word vectors and whole sentence vectors in the sample text; the language extraction model is a model trained according to the sample text; the sample text is provided with a feature label;

[0053] Extracting speech rate information from the sample audio according to the speech rate extraction model, and obtaining a speech rate label vector based on the speech rate information; the speech rate label vector is used to characterize the speech rate in the sample audio; the sample audio is audio data generated based on the sample text;

[0054] Converting the sample text into a phoneme sequence, and obtaining a phoneme vector of the sample text based on the phoneme sequence;

[0055] Inputting the word-by-word vector, the whole sentence vector, the speech rate label vector and the phoneme vector into a decoder to obtain a target synthesized speech;

[0056] Feedback receipt information is fed back to the display device, so that the display device plays the target synthesized speech according to the receipt information.

[0057] It can be seen from the above technical content that the present application provides a display device, a server and a speech synthesis method. When the display device receives the voice interaction instruction input by the user, it calls the language extraction model to extract the word-by-word vector and the whole sentence vector in the sample text. The language extraction model is a model trained according to the sample text, and the sample text is provided with a feature label. Through the feature label, the model can have the ability to recognize colloquial features in the process of training the model, thereby adding colloquial features to the synthesized speech and improving the anthropomorphism of the synthesized speech. According to the sample text and the audio of the sample text, the speech rate label vector and the phoneme vector can also be extracted. The word-by-word vector, the whole sentence vector, the speech rate label vector and the phoneme vector are input into the decoder together to obtain the target synthesized speech, and the target synthesized speech is played by the audio output device. BRIEF DESCRIPTION OF THE DRAWINGS

[0058] In order to more clearly illustrate the technical solution of the present application, the drawings required for use in the embodiments are briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0059] Figure 1 A schematic diagram of a display device operation scenario provided in some embodiments of the present application;

[0060] Figure 2 A schematic diagram of the hardware configuration of a display device provided in some embodiments of the present application;

[0061] Figure 3 A schematic diagram of software configuration of a display device provided in some embodiments of the present application;

[0062] Figure 4 A schematic diagram of a synthetic speech interaction scenario provided for some embodiments of the present application;

[0063] Figure 5 A flowchart of generating a target synthesized speech for a display device provided in some embodiments of the present application;

[0064] Figure 6 A vector extraction timing diagram provided for some embodiments of the present application;

[0065] Figure 7 A structural diagram of a language extraction model provided for some embodiments of the present application;

[0066] Figure 8 A schematic diagram of a normal distribution curve of speech rate provided in some embodiments of the present application;

[0067] Fig. 9 A schematic diagram of a decoder provided in some embodiments of the present application generating a target synthesized speech according to a vector;

[0068] Fig.10 A schematic diagram of extracting historical information vectors provided in some embodiments of the present application;

[0069] Fig.11 A schematic diagram of multi-language interaction of a display device provided in some embodiments of the present application;

[0070] Fig.12 A flowchart of a display device generating a target synthesized speech through a server provided in some embodiments of the present application;

[0071] Fig.13 A timing interaction diagram between a display device and a server provided for some embodiments of the present application. DETAILED DESCRIPTION

[0072] The following embodiments are described in detail, and examples thereof are shown in the accompanying drawings. When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The implementations described in the following embodiments do not represent all implementations consistent with the present application. They are only examples of systems and methods consistent with some aspects of the present application as detailed in the claims.

[0073] The following embodiments are described in detail, and examples thereof are shown in the accompanying drawings. When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The implementations described in the following embodiments do not represent all implementations consistent with the present application. They are only examples of systems and methods consistent with some aspects of the present application as detailed in the claims.

[0074] It should be noted that the brief description of terms in this application is only for the convenience of understanding the embodiments described below, and is not intended to limit the embodiments of this application. Unless otherwise specified, these terms should be understood according to their common and usual meanings.

[0075] The terms "first", "second", "third", etc. in the specification and claims of this application and the above drawings are used to distinguish similar or similar objects or entities, and do not necessarily mean to limit a specific order or sequence, unless otherwise noted. It should be understood that the terms used in this way can be interchangeable under appropriate circumstances.

[0076] The terms "comprises," "comprising," and "having," and any variations thereof, are intended to cover but not exclude inclusion, for example, a product or device comprising a list of components is not necessarily limited to all the components expressly listed but may include other components not expressly listed or inherent to such product or device.

[0077] The term "module" refers to any known or later developed hardware, software, firmware, artificial intelligence, fuzzy logic, or combination of hardware and / or software code that is capable of performing the functions associated with that element.

[0078] In the embodiment of the present application, the display device 200 generally refers to a device with image display and data processing capabilities. For example, the display device 200 includes but is not limited to a smart TV, a mobile terminal, a computer, a monitor, an advertising screen, a wearable device, a virtual reality device, an augmented reality device, etc.

[0079] Figure 1 This is a schematic diagram of an operation scenario between a display device and a control device provided in some embodiments of the present application. Figure 1As shown in FIG. 1 , the user can operate the display device 200 through touch operation, the mobile terminal 300 and the control device 100. For example, the control device 100 can be a remote controller, a stylus pen, a handle, etc.

[0080] The mobile terminal 300 can be used as a control device for performing human-computer interaction between the user and the display device 200. The mobile terminal 300 can also be used as a communication device for establishing a communication connection with the display device 200 and performing data interaction. In some embodiments, the mobile terminal 300 can install software applications with the display device 200, and achieve connection and communication through a network communication protocol to achieve the purpose of one-to-one control operation and data communication. The audio and video content displayed on the mobile terminal 300 can also be transmitted to the display device 200 to achieve a synchronous display function.

[0081] like Figure 1 As also shown in FIG. 4 , the display device 200 also communicates data with the server 400 through various communication methods. The display device 200 may be allowed to communicate and connect through a local area network (LAN), a wireless local area network (WLAN), and other networks.

[0082] The display device 200 may provide a broadcast receiving television function, and may also additionally provide an intelligent network television function with a computer support function, including but not limited to network television, smart television, Internet Protocol television (IPTV), and the like.

[0083] Figure 2 Some embodiments of the present application provide Figure 1 2 is a block diagram of the hardware configuration of the display device 200.

[0084] In some embodiments, the display device 200 may include at least one of a tuner 210, a communication device 220, a detector 230, a device interface 240, a controller 250, a display 260, an audio output device 270, a memory, a power supply, and a user input interface.

[0085] In some embodiments, the detector 230 is used to collect signals of the external environment or external interaction. For example, the detector 230 includes a light receiver, a sensor for collecting the intensity of ambient light; or, the detector 230 includes an image collector, such as a camera, which can be used to collect external environment scenes, user attributes or user interaction gestures; or, the detector 230 includes a sound collector, such as a microphone, etc., for receiving external sounds.

[0086] In some embodiments, the display 260 includes a display function component for presenting a picture, and a driving component for driving an image display. The display 260 is used to receive an image signal output from the controller 250 for display. For example, the display 260 can be used to display video content, image content, and components of a menu control interface and a user control UI interface.

[0087] In some embodiments, the communication device 220 is a component for communicating with an external device or server 400 according to various communication protocol types. The display device 200 may be provided with a plurality of communication devices 220 according to different supported communication modes. For example, when the display device 200 supports wireless network communication, the display device 200 may be provided with a communication device 220 including a WiFi function. When the display device 200 supports Bluetooth connection communication, the display device 200 needs to be provided with a communication device 220 including a Bluetooth function.

[0088] The communication device 220 can enable the display device 200 to communicate with the external device or server 400 by wireless or wired connection. Among them, the wired connection can connect the display device 200 with the external device through components such as data cables and interfaces. The wireless connection can connect the display device 200 with the external device through wireless signals or wireless networks. The display device 200 can establish a connection relationship with the external device directly, or indirectly establish a connection relationship through a gateway, a router, a connection device, etc.

[0089] In some embodiments, the controller 250 may include at least one of a central processing unit, a video processor, an audio processor, a graphics processor, and a power processor, and a first interface to an nth interface for input / output. The controller 250 controls the operation of the display device and responds to the user's operation through various software control programs stored in the memory. The controller 250 controls the overall operation of the display device 200.

[0090] In some embodiments, the controller 250 and the tuner-demodulator 210 may be located in different separate devices, that is, the tuner-demodulator 210 may also be located in an external device of the main device where the controller 250 is located, such as an external set-top box.

[0091] In some embodiments, the user may input a user command through a graphical user interface (GUI) displayed on the display 260 , and the user input interface receives the user input command through the graphical user interface (GUI).

[0092] In some embodiments, the audio output device 270 may be a local speaker of the display device 200, or may be an external audio output device of the display device 200. In particular, for the external audio output device of the display device 200, the display device 200 may also be provided with an external audio output terminal, and the audio output device may be connected to the display device 200 through the external audio output terminal to output the sound of the display device 200.

[0093] In some embodiments, the user input interface 280 may be used to receive instructions from a user.

[0094] In order to perform user interaction, in some embodiments, the display device 200 may run an operating system. The operating system is a computer program for managing and controlling hardware resources and software resources in the display device 200. The operating system can control the display device to provide a user interface, for example, the operating system can directly control the display device to provide a user interface, or can provide a user interface by running an application program. The operating system also allows the user to interact with the display device 200.

[0095] It should be noted that the operating system may be a native operating system based on a specific operating platform, or a third-party operating system deeply customized based on a specific operating platform, or an independent operating system specially developed for the display device.

[0096] The operating system can be divided into different modules or layers according to the functions implemented, such as Figure 3 As shown, in some embodiments, the system is divided into four layers, from top to bottom, namely, the application layer (Applications) layer (referred to as "application layer"), the application framework layer (Application Framework) layer (referred to as "framework layer"), the system library layer and the kernel layer.

[0097] In some embodiments, the application layer is used to provide services and interfaces for applications so that the display device 200 can run applications and interact with users based on the applications. At least one application can be run in the application layer, and these applications can be window programs, system settings programs, clock programs, etc. that come with the operating system; they can also be applications developed by third-party developers. In specific implementations, the application packages in the application layer are not limited to the above examples.

[0098] The framework layer provides application programming interfaces (APIs) and programming frameworks for applications. The application framework layer includes some predefined functions. The application framework layer is equivalent to a processing center that determines the actions that applications in the application layer take. Through the API interface, applications can access system resources and obtain system services during execution.

[0099] like Figure 3 As shown, in the embodiment of the present application, the application framework layer includes a view system, managers, content providers, etc., wherein the view system can design and implement the interface and interaction of the application, and the view system includes lists, grids, text boxes, buttons, etc. The manager includes at least one of the following modules: an activity manager for interacting with all activities running in the system; a location manager for providing system services or applications with access to system location services; a package manager for retrieving various information related to the application package currently installed on the device; a notification manager for controlling the display and clearing of notification messages; and a window manager for managing icons, windows, toolbars, wallpapers, and desktop widgets on the user interface.

[0100] In some embodiments, the activity manager is used to manage the life cycle of each application and the usual navigation back function, such as controlling the exit, opening, and back of the application. The window manager is used to manage all window programs, such as obtaining the display screen size, determining whether there is a status bar, locking the screen, capturing the screen, and controlling the display window changes, for example, reducing the display window, shaking the display, distorting the display, etc.

[0101] In some embodiments, the system runtime layer can provide support for the framework layer. When the framework layer is used, the operating system will run the instruction library contained in the system runtime layer, such as the C / C++ instruction library, to implement the functions to be implemented by the framework layer.

[0102] In some embodiments, the kernel layer is a functional layer between the hardware and software of the display device 200. The kernel layer can implement functions such as hardware abstraction, multitasking, and memory management. Figure 3As shown, the kernel layer may be configured with hardware drivers, and the drivers included in the kernel layer may be at least one of the following drivers: audio driver, display driver, Bluetooth driver, camera driver, WIFI driver, USB driver, HDMI driver, sensor driver (such as fingerprint sensor, temperature sensor, pressure sensor, etc.), and power driver, etc.

[0103] It should be noted that the above example is only a simple division of the operating system functions and does not constitute a limitation on the specific operating system form of the display device 200 in the embodiment of the present application. Depending on factors such as the function of the display device and the type of operating system, the number of levels and specific level types contained in the operating system may be expressed in other forms.

[0104] based on Figure 3 The provided viewing system allows the display device 200 to provide a user interface for the user to select the desired function when the user is using the display device 200. When the user selects the desired function based on the user interface, the display device 200 can be controlled to start the target function by inputting a voice command to the display device 200. In some embodiments, when the display device 200 receives the voice command input by the user, it can feedback the voice to the user through the audio output device 270 to achieve voice interaction with the user. Such an anthropomorphic interaction method is conducive to improving the user's experience, and through voice feedback, it is conducive to informing the user of information such as the execution progress related to the voice command.

[0105] It is understandable that if Figure 4 As shown in the figure, similar voice interaction scenarios also exist in the fields of smart cars, smart phones, computers, etc. Speech synthesis models can be deployed in these smart devices, and then the speech synthesis model generates the target synthesized speech based on the sample text to perform voice interaction with the user. By improving the accuracy of the speech synthesis model and adjusting the training method, the target synthesized speech can have human voice characteristics such as emotions.

[0106] However, in the process of training the speech synthesis model, the speech is synthesized only based on the text itself, lacking the semantic features of the context and the colloquial features in the text. It is difficult for the speech synthesis model to learn the colloquial features of human voice communication, such as rhythm, stress, and intonation of speech speed. Therefore, the target synthesized speech generated by the speech synthesis model lacks colloquial features, resulting in insufficient anthropomorphism.

[0107] like Figure 5 and Figure 6As shown, the present application provides a display device, which improves the anthropomorphism of the target synthesized speech by adding training of spoken features in the process of synthesizing speech. The display device includes: a display 260, a detector 230, an audio output device 270 and a controller 250. Among them, the display 260 user displays an interface that meets the user's needs according to the instructions input by the user. The detector 230 is used to collect the voice data input by the user. The audio output device 270 is used to play the target synthesized speech and various prompt sound effects. The controller 250 is configured as follows:

[0108] S100: In response to a voice interaction instruction input by a user, a language extraction model is called to extract word-by-word vectors and whole sentence vectors from a sample text.

[0109] In some embodiments, the voice interaction instruction may be an instruction input by the user to the display device 200 for starting the control. When the display device 200 receives the voice instruction for starting the control, it may output a prompt for prompting the user of the progress of the instruction execution through the audio output device 270 during the process of starting the control, such as "the control is starting". In this way, the user can obtain the execution progress of the voice interaction instruction in real time, which is conducive to improving the user experience. The prompt played by the audio output device 270 may be a pre-set corpus, or may be a target synthesized speech generated in real time according to the current scene.

[0110] In other embodiments, the user may input an instruction to the display device to start a human-computer dialogue. The display device 200 may communicate with the user when the user asks a question. In such a real-time scenario, the display device 200 needs to generate a target synthesized speech based on the sample text.

[0111] It should be noted that when the display device 200 generates the target synthesized speech using the model, it can extract the feature vector required to generate the target synthesized speech with the help of multiple sub-models, for example, by extracting the word-by-word vector E i,word (i refers to the sequence number of the vector, and the i appearing in the subsequent embodiments is also the sequence number of the vector) and the whole sentence vector E i,sentence The language extraction model is a model trained based on sample texts, and the sample texts are provided with feature labels. In the process of training the language extraction model, the feature labels can be used as learning targets of the training process. Under the supervision of the feature labels, the language extraction model can identify language features related to the feature labels during the training process.

[0112] The extraction of word-by-word vectors enables the model to process sample texts based on vector operations, and the extraction of word-by-word vectors enables the model to obtain the semantic relationship between keywords and adjacent words during training to solve the problem of polysemy. The extraction of whole sentence vectors further increases the feature of the correlation between contexts, so that the model can fully combine the correlation of contexts to generate target synthetic speech during training or reasoning to improve the degree of anthropomorphism.

[0113] like Figure 7 As shown, in some embodiments, word-by-word vectors and whole sentence vectors can be extracted through a trained language extraction model. The language extraction model can be composed of an open source BERT model, a Block neural network format, a linear layer, and a fully connected layer. Among them, the BERT model is a language model suitable for multiple tasks, which can be used to extract features in sample texts and classify sample texts by cooperating with linear layers and fully connected layers. The Block neural network format refers to a neural network format in which multiple layers are arranged in a preset order to achieve a target function. By adjusting the preset order, the model can have functions such as text classification.

[0114] In some embodiments, when the controller 250 inputs the sample text into the language extraction model, it is specifically configured to: input the sample text into the feature extraction layer of the language extraction model to obtain the semantic features, contextual features, lexical features, and sentiment features of the sample text. Input the semantic features, contextual features, lexical features, and sentiment features into the classification layer to obtain the classified word-by-word vector and the classified whole sentence vector.

[0115] In some embodiments, the feature extraction layer can use an open source BERT model to extract feature information such as semantic features, contextual relationship features, vocabulary features, and sentiment features in the sample text through the BERT model. Based on the BERT model, keywords, key words, and key sentences can also be converted into high-dimensional vector expressions. In this way, the BERT model can output representation vectors to represent the keywords, key words, or key sentences in the sample text. These vectors contain the above-mentioned feature information and are therefore suitable for subsequent classification.

[0116] After the BERT model outputs the representation vector, the classification layer in the language extraction model sets a classification label for the representation vector. In the training phase, the loss value can be calculated based on the classification label and the pre-set feature label to adjust the parameters of the classification layer. In the inference phase, the target synthesized speech can be generated based on the classification result.

[0117] In some embodiments, the feature labels include rhythm labels, stress labels, and speech rate labels. The feature labels can also be subdivided according to actual training needs, for example, the speech rate labels can be subdivided into slow speech rate labels, medium speech rate labels, and fast speech rate labels. The subdivision of feature labels can further improve the accuracy of model training, but it will also affect the training speed and inference speed of the model, so the feature labels can be subdivided according to actual needs.

[0118] In some embodiments, text classification refers to dividing sample text into different categories of labels through a language extraction model, for example, dividing extracted keywords into rereading labels. Based on this, the language extraction model can be supervised for training by setting feature labels for the sample text and taking the feature labels as learning targets. That is, when training the language extraction model, the controller is also configured to:

[0119] Set a prosody label, a stress label and a speech rate label in the sample text. Input the sample text into the language extraction model to be trained to obtain a classified word-by-word vector and a classified sentence vector. Calculate a first loss value between the prosody label and the classified prosody label, and, calculate a second loss value between the stress label and the classified stress label, and, calculate a third loss value between the speech rate label and the classified speech rate label. Adjust the model parameters of the language extraction model to be trained according to the first loss value, the second loss value and the third loss value.

[0120] In some embodiments, the language extraction model can extract keywords and key sentences from the sample text through the linear layer and the fully connected layer, and set labels for the keywords and key sentences by classification, thereby obtaining word-by-word vectors and whole sentence vectors containing classification labels. During the training process, classified word-by-word vectors and classified whole sentence vectors containing classification labels are obtained. Furthermore, during the training process of the language extraction model, the parameters of the linear layer and the fully connected layer can be adjusted by calculating the loss value between the feature label and the classification label.

[0121] It should be noted that the model parameters of the BERT model part of the language extraction model are not adjusted as the loss value is calculated during the training process. The BERT model is only used to extract keywords, key words, and key sentences based on the features in the sample text, and does not participate in the classification process, so it is not adjusted.

[0122] Therefore, by setting feature labels to supervise the training of the language extraction model, the word-by-word vectors and sentence vectors extracted by the language extraction model will contain features such as rhythm, speaking speed, and stress, which will help the decoder to have richer spoken features when generating the target synthesized speech, and generate a target synthesized speech with a higher degree of humanization.

[0123] S200: Extracting speech rate information from the sample audio according to a speech rate extraction model, and obtaining a speech rate label vector based on the speech rate information.

[0124] By extracting the speech rate information, a speech rate label X can be set for each keyword i,speed_tag , so that the model can learn the association between the speech and speech rate corresponding to the text according to the speech rate label, and then the model can adjust the speech rate of the target synthesized speech according to the corresponding scene in the text, thereby enhancing the spoken features of the target synthesized speech.

[0125] It can be understood that the sample audio is audio data generated based on the sample text, and the speech rate label can be encoded to obtain the speech rate label vector E i,speed_tag , so that it can be operated with vectors such as word-by-word vectors and whole sentence vectors.

[0126] The speech rate label in the feature label can calculate the speech rate of the sample audio through the speech rate extraction model to add the speech rate feature to the target synthesized speech. The controller 250 performs the extraction of speech rate information in the sample audio according to the speech rate extraction model, and obtains the speech rate label vector based on the speech rate information, and is specifically configured as: using the speech rate extraction model to calculate the speech rate of the sample audio. Construct a speech rate normal distribution curve according to the speech rate of the sample audio. Determine the first speech rate threshold and the second speech rate threshold based on the speech rate normal distribution curve. Set a first speech rate label for a keyword whose speech rate is less than or equal to the first speech rate threshold, and set a second speech rate label for a keyword whose speech rate is greater than the first speech rate threshold and less than the second speech rate threshold, and set a third speech rate label for a keyword whose speech rate is greater than the second speech rate threshold.

[0127] In some embodiments, the speech extraction model is a speech rate alignment tool (Kaldi), and the speech rate X of each keyword in the sample text can be obtained by using the speech rate alignment tool and the sample audio corresponding to the sample text. i,speed , and then set a speaking speed label for each word.

[0128] like Figure 8 As shown, when obtaining the speech rate label of each keyword, a normal distribution can be constructed based on the pronunciation duration of a single keyword, and a speech rate threshold can be set. In this way, by setting the first speech rate threshold and the second speech rate threshold, the speech rate of the keyword can be divided into three types of speech rate labels. Among them, the first speech rate threshold can be a slow speech rate threshold, and the second speech rate threshold can be a fast speech rate threshold. The speech rate threshold can be set according to actual needs, and "slow" and "fast" can also be adjusted according to the statistical results of the normal distribution.

[0129] It can be understood that based on the statistical results of the normal distribution, the speaking speed label of each keyword can be obtained, and then when generating the target synthetic speech, the speaking speed characteristics of each keyword will be fully combined and combined with rhythm, stress and other characteristics to generate a synthetic speech suitable for the application scenario.

[0130] S300: Convert the sample text into a phoneme sequence, and generate a phoneme vector of the sample text based on the phoneme sequence.

[0131] Phonemes refer to the smallest unit or smallest speech segment that constitutes a syllable. Each keyword can include 1 / 2 / 3 / 4 phonemes. By obtaining the phoneme sequence corresponding to the sample text, the model can learn the pronunciation duration, intensity, pitch and other characteristics of the text based on phonemes during the speech synthesis process, which is conducive to enhancing the spoken features in the target synthesized speech and improving the anthropomorphism of the target synthesized speech.

[0132] In order to improve the computational efficiency, the phoneme sequence can also be encoded to form a phoneme vector E i,phone , so that it can be input into the decoder together with the word-by-word vector, the whole sentence vector, and the speech rate label vector, and the decoder generates the target synthetic speech based on these vectors.

[0133] In some embodiments, building a model based on phonemes can improve the model's ability to learn features such as speech rate, rhythm, and stress. Therefore, in some embodiments of the present application, the sample text is converted into a phoneme sequence to obtain a phoneme vector of the sample text, and the phoneme vector is used to generate a target synthesized speech. When converting the sample text into a phoneme sequence, the controller 250 is configured to: use a phoneme model to convert the sample text into a phoneme sequence. The keywords in the sample text correspond to at least one phoneme. Encoding is performed on the phonemes in the phoneme sequence to obtain a phoneme vector.

[0134] It is understandable that the sample text can be converted into a phoneme sequence according to the phoneme extraction model, and a phoneme vector can be obtained based on the phoneme sequence encoding. The factor vector can quantify the acoustic features of the phoneme, and then can be input into the decoder for target speech generation. Moreover, based on the acquisition of the phoneme vector, the rhythm, pause and other rhythmic sense of the synthesized speech can be improved in the speech synthesis stage, thereby enhancing the spoken features of the target synthesized speech and improving the anthropomorphism of the target synthesized speech. In addition, the phoneme vector can include contextual information, such as the influence between the previous and next phonemes, so that when generating the target synthesized speech, the phenomenon of co-articulation can be fully simulated to improve the fluency of the target synthesized speech.

[0135] It should be noted that the number of phonemes corresponding to each keyword in the sample text is 1 or more, so it is necessary to align the phonemes of each keyword with the keyword to facilitate the calculation of the integrated vector. That is, after the controller 250 converts the sample text into a phoneme sequence and obtains the phoneme vector of the sample text based on the phoneme sequence, it is also configured to: obtain the number of phonemes of the phonemes corresponding to the keyword based on the phoneme sequence. Based on the number of phonemes, copy the word-by-word vector and speech rate label corresponding to the keyword to align the word-by-word vector of the keyword with the phoneme vector of the word.

[0136] In some embodiments, the number of phonemes refers to the number of phonemes corresponding to each keyword. In the process of integrating the phoneme vector, the word-by-word vector, the whole sentence vector, and the speech rate label vector, it is necessary to align the phoneme vector, the word-by-word vector, and the speech rate label vector, so that the feature vector used to describe the keyword can be obtained, and then the decoder can generate the target synthesized speech based on the accurate feature vector. By aligning, the problem of inaccurate description of the keyword by the feature vector due to misalignment between vectors, and then inaccurate target synthesized speech can be avoided.

[0137] In some embodiments, the word-by-word vector and speech rate label corresponding to the keyword are copied according to the number of phonemes of the keyword, so that the word-by-word vector, speech rate label vector, and phoneme vector are aligned. For example, when the number of phonemes is 2, it is necessary to copy the word-by-word vector once more to align the two word-by-word vectors with the phoneme vector. The speech rate label vector can also be aligned with the word-by-word vector and the phoneme vector by copying.

[0138] In this way, the phoneme vector, word-by-word vector, and speech rate label vector can be aligned by copying to improve the efficiency of vector integration and the accuracy of speech synthesis.

[0139] S400: Input the word-by-word vector, the whole sentence vector, the speech rate label vector and the phoneme vector into a decoder to obtain a target synthesized speech.

[0140] like Fig. 9 As shown, in some embodiments, the word-by-word vector, the whole sentence vector, the speech rate label vector and the phoneme vector are first integrated to obtain an integrated vector and then input into the decoder for decoding. Among them, the historical information vector can be further obtained according to the whole sentence vector, so as to increase the characteristics of the contextual relationship in the training data through the historical information vector, so that the decoder can fully combine the connection between the contexts in the sample text when generating the target synthesized speech, so as to enhance the spoken characteristics of the target synthesized speech. That is, the controller 250 executes the input of the word-by-word vector, the whole sentence vector, the speech rate label vector and the phoneme vector into the decoder to obtain the target synthesized speech, and is specifically configured as follows:

[0141] A historical information extraction threshold is set. The whole sentence vector is divided into a first whole sentence sequence and a second whole sentence sequence according to the historical information extraction threshold. The first whole sentence sequence is input into a historical information extraction model to obtain a historical information vector. The word-by-word vector, the historical information vector, the speech rate label vector and the phoneme vector are input into a decoder to obtain a target synthesized speech.

[0142] In some embodiments, the whole sentence vectors extracted from the sample text can be distinguished by adding serial numbers to the whole sentence vectors. For example, when 10 whole sentence vectors are extracted from the sample text, 10 serial numbers 1-10 can be assigned to the whole sentence vectors according to the order of the whole sentence vectors in the sample text. That is, the serial number of the whole sentence vector is used to represent the order of the text corresponding to the whole sentence vector in the sample text.

[0143] In some embodiments, in order to integrate the whole sentence vector into the historical information vector, so as to provide accurate and rich text semantics for the decoder to generate the target synthesized speech based on the historical information vector, a historical information extraction threshold can be set to divide the whole sentence vector. The historical information extraction threshold can be matched with the sequence number of the whole sentence vector to divide the whole sentence vector into a first whole sentence sequence and a second whole sentence sequence.

[0144] Taking the maximum number of the whole sentence vector as i as an example, the whole sentence vector with the maximum number can be expressed as E i,sentence .

[0145] Taking the historical information extraction threshold as c, and the sequence number of the whole sentence vector in the first whole sentence sequence is greater than the historical information extraction threshold, and the sequence number of the whole sentence vector in the second whole sentence sequence is less than or equal to the historical information extraction threshold as an example, the whole sentence vector included in the first whole sentence sequence is E i-c,sentence , E i-c+1,sentence ,…,E i,sentence The second whole sentence sequence includes the whole sentence vector E 1,sentence ,…,E c,sentence .

[0146] like Fig.10 As shown, the first whole sentence sequence can be input into a historical information extraction network composed of a linear layer-long short-term memory network (LSTM)-linear layer, and then the historical information extraction network generates a historical information vector E according to the first whole sentence sequence. i,history

[0147] It is understandable that the more whole sentence vectors are input to the decoder, the more semantic information and types the decoder can obtain about the sample text through the whole sentence vectors. However, during training or reasoning, too many whole sentence vectors input to the decoder will reduce the accuracy of the decoder's understanding of the text semantics, which is not conducive to training and reasoning. Therefore, by setting the historical information extraction threshold, the part input to the decoder can have sufficient context association, so that the decoder can fully obtain the semantic features between the texts, thereby improving the accuracy of the generated acoustic features.

[0148] In some embodiments, the word-by-word vector, the history information vector, the speech rate label vector and the phoneme vector are integrated and then input into the decoder, and the decoder generates the target synthesized speech according to these vectors. That is, the controller 250 is configured to: accumulate the word-by-word vector, the history information vector, the speech rate label vector and the phoneme vector to obtain the integrated vector. The integrated vector is input into the decoder to obtain the target synthesized speech.

[0149] In some embodiments, the decoder can be a sequence generation model, such as a recurrent neural network (RNN) or a Transformer model. Acoustic features, such as fundamental frequency and spectrum features, are generated based on these input vectors to generate target synthesized speech. During the decoding process, the vector sequence inputted by the decoder input can be dynamically adjusted with the help of an attention mechanism, thereby improving the generation effect of each speech frame.

[0150] S500: Play the target synthesized speech through the audio output device.

[0151] After the decoder generates the target synthesized speech, the controller 250 can input the target synthesized speech to the audio output device 270 for playback by the audio output device 270. The audio output device 270 here includes but is not limited to a speaker provided on the display device 200 body and a headset connected to the display device 200 via a wireless connection such as Bluetooth.

[0152] It is understandable that languages ​​have regional characteristics, such as northeastern dialects, northwestern dialects, and Cantonese dialects. Therefore, on the basis of training a language extraction model based on sample texts, sample texts corresponding to dialects in different regions can be obtained by classifying the sample texts. In this way, the language extraction model can be trained based on sample texts containing dialects from different regions. Moreover, in the process of model reasoning, dialect-based synthetic speech that is adapted to the region can be generated, which is conducive to improving the user experience.

[0153] like Fig.11As shown, in some embodiments, the user can also pre-select the language type used by the display device 200 during voice interaction. For example, a prompt box is presented on the user interface, and "Please select the voice type you like" can be displayed in the prompt box. If hardware resources and software resources allow, an audition control can also be set in the prompt box, so that after the user triggers the audition control, the voice is played for the user in the dialect type corresponding to the audition control, so as to enhance the user's experience.

[0154] It is understandable that multiple language extraction models can be deployed to meet the needs of using multiple dialects, or a language extraction model that integrates multiple dialect extraction capabilities can be deployed to meet the needs of using multiple dialects. The specific model deployment can be selected and adjusted according to the actual situation.

[0155] like Fig.12 and Fig.13 As shown, some embodiments of the present application provide a server, including a communication unit and a control unit. The communication unit is configured to establish a communication connection with a display device. The control unit is configured to:

[0156] In response to the speech synthesis instruction sent by the display device, a language extraction model is called to extract word-by-word vectors and whole sentence vectors in the sample text; the language extraction model is a model trained based on the sample text; the sample text is provided with a feature label;

[0157] Extracting speech rate information from the sample audio according to the speech rate extraction model, and obtaining a speech rate label vector based on the speech rate information; the speech rate label vector is used to characterize the speech rate in the sample audio; the sample audio is audio data generated based on the sample text;

[0158] Converting the sample text into a phoneme sequence, and obtaining a phoneme vector of the sample text based on the phoneme sequence;

[0159] Inputting the word-by-word vector, the whole sentence vector, the speech rate label vector and the phoneme vector into a decoder to obtain a target synthesized speech;

[0160] Feedback receipt information is fed back to the display device, so that the display device plays the target synthesized speech according to the receipt information.

[0161] In some embodiments, the server can establish a communication connection with the display device 200 through a communication unit. Then the server can receive the speech synthesis instruction sent by the display device 200 and respond to the speech synthesis instruction. For example, the language extraction model is called to extract the word-by-word vector and the whole sentence vector in the sample text, the speech rate information in the sample audio is extracted to generate the speech rate label vector, and the phoneme vector of the sample text is extracted from the sample text. In this way, the server can generate the target synthesized speech through the language extraction model, the speech rate extraction model and the speech synthesis module to which the decoder belongs, which are pre-deployed on the server side. After synthesizing the target speech, the server can feed back the target synthesized speech to the display device 200 so that the display device 200 can play the target synthesized speech to the user.

[0162] It is understandable that, through the communication between the display device 200 and the server, even when the hardware resources of the display device 200 are insufficient, the computing power provided by the server can still be used to achieve voice interaction with the user, so as to enhance the user experience. In addition, based on the extraction of word-by-word vectors, whole sentence vectors, phoneme vectors, and speech rate labels, as well as the rhythmic labels, speech rate labels, and stress labels in the feature labels, the model can learn colloquial features from the sample text and sample audio, thereby enriching the target synthesized speech with colloquial features, which is conducive to improving the anthropomorphism of the target synthesized speech.

[0163] Some embodiments of the present application provide a speech synthesis method that can be applied to a display device, the method comprising:

[0164] In response to a voice interaction instruction input by a user, a language extraction model is called to extract word-by-word vectors and whole sentence vectors in a sample text; the language extraction model is a model trained based on the sample text; the sample text is provided with a feature label;

[0165] Extracting speech rate information from the sample audio according to the speech rate extraction model, and obtaining a speech rate label vector based on the speech rate information; the speech rate label vector is used to characterize the speech rate of the sample audio; the sample audio is audio data generated based on the sample text;

[0166] Converting the sample text into a phoneme sequence, and obtaining a phoneme vector of the sample text based on the phoneme sequence;

[0167] Inputting the word-by-word vector, the whole sentence vector, the speech rate label vector and the phoneme vector into a decoder to obtain a target synthesized speech;

[0168] The target synthesized voice is played through the audio output interface to perform voice interaction with the user.

[0169] It can be seen from the above technical content that the present application provides a display device, a server and a speech synthesis method. When the display device receives the voice interaction instruction input by the user, it calls the language extraction model to extract the word-by-word vector and the whole sentence vector in the sample text. The language extraction model is a model trained according to the sample text, and the sample text is provided with a feature label. Through the feature label, the model can have the ability to recognize colloquial features in the process of training the model, thereby adding colloquial features to the synthesized speech and improving the anthropomorphism of the synthesized speech. According to the sample text and the audio of the sample text, the speech rate label vector and the phoneme vector can also be extracted. The word-by-word vector, the whole sentence vector, the speech rate label vector and the phoneme vector are input into the decoder together to obtain the target synthesized speech, and the target synthesized speech is played by the audio output device.

[0170] Similar parts between the embodiments provided in this application can be referenced to each other. The specific implementation methods provided above are only a few examples under the general concept of this application and do not constitute a limitation on the protection scope of this application. For those skilled in the art, any other implementation methods expanded based on the scheme of this application without creative work belong to the protection scope of this application.

Claims

1. A display device, characterized in that: include: monitor; A detector configured to collect voice data input by a user; an audio output device configured to play speech; The controller is configured as: In response to a voice interaction instruction input by a user, a language extraction model is called to extract word-by-word vectors and sentence-wide vectors from a sample text; the language extraction model is a model trained based on the sample text, and the sample text is provided with a feature label; Extracting speech rate information from the sample audio according to the speech rate extraction model, and obtaining a speech rate label vector based on the speech rate information; the speech rate label vector is used to characterize the speech rate of the sample audio; the sample audio is audio data generated based on the sample text; Converting the sample text into a phoneme sequence, and generating a phoneme vector of the sample text based on the phoneme sequence; Inputting the word-by-word vector, the whole sentence vector, the speech rate label vector, and the phoneme vector into a decoder to obtain a target synthesized speech; The target synthesized speech is played through the audio output device.

2. The display device according to claim 1, wherein The feature tags include a prosody tag, a stress tag, and a speech rate tag; the controller is further configured to: Setting a prosody tag, a stress tag, and a speech rate tag in the sample text; Inputting the sample text into a language extraction model to be trained to obtain a classified word-by-word vector and a classified sentence vector; the classified word-by-word vector and the classified sentence vector include classification labels; the classification labels include a classification prosody label, a classification stress label, and a classification speech rate label; Calculating a first loss value between the prosody label and the classification prosody label, calculating a second loss value between the stress label and the classification stress label, and calculating a third loss value between the speech rate label and the classification speech rate label; Adjust model parameters of the language extraction model to be trained according to the first loss value, the second loss value, and the third loss value.

3. The display device according to claim 1, wherein The language extraction model includes a feature extraction layer and a classification layer; the controller inputs the sample text into the language extraction model to be trained to obtain a classified word-by-word vector and a classified whole sentence vector, and is specifically configured as follows: Inputting the sample text into a feature extraction layer of a language extraction model to obtain semantic features, contextual relationship features, lexical features, and sentiment features of the sample text; The semantic features, contextual relationship features, vocabulary features and sentiment features are input into the classification layer to obtain classified word-by-word vectors and classified whole-sentence vectors.

4. The display device according to claim 1, wherein The controller extracts speech rate information from the sample audio according to the speech rate extraction model, and obtains a speech rate label vector based on the speech rate information, and is specifically configured as follows: Calculate the speech rate of the sample audio using the speech rate extraction model; Constructing a speech rate normal distribution curve according to the speech rate of the sample audio; Determining a first speech rate threshold and a second speech rate threshold based on the speech rate normal distribution curve; A first speech rate label is set for a keyword whose speech rate is less than or equal to the first speech rate threshold, and a second speech rate label is set for a keyword whose speech rate is greater than the first speech rate threshold and less than a second speech rate threshold, and a third speech rate label is set for a keyword whose speech rate is greater than the second speech rate threshold.

5. The display device according to claim 1, wherein The controller converts the sample text into a phoneme sequence and obtains a phoneme vector of the sample text based on the phoneme sequence, and is specifically configured to: Converting the sample text into a phoneme sequence using a phoneme model; wherein the keyword in the sample text corresponds to at least one phoneme; Encoding is performed on the phonemes in the phoneme sequence to obtain a phoneme vector.

6. The display device according to claim 5, wherein: After converting the sample text into a phoneme sequence and obtaining a phoneme vector of the sample text based on the phoneme sequence, the controller is further configured to: Based on the phoneme sequence, obtaining the number of phonemes corresponding to the keyword; Based on the number of phonemes, the word-by-word vector and the speech rate label corresponding to the word are copied so that the word-by-word vector of the keyword is aligned with the phoneme vector of the word.

7. The display device according to claim 1, wherein The controller executes the input of the word-by-word vector, the whole sentence vector, the speech rate label vector and the phoneme vector to the decoder to obtain the target synthesized speech, and is specifically configured as follows: Set the historical information extraction threshold; Dividing the whole sentence vector into a first whole sentence sequence and a second whole sentence sequence according to the historical information extraction threshold; the sequence number of the whole sentence vector in the first whole sentence sequence is greater than the historical information extraction threshold; the sequence number of the whole sentence vector in the second whole sentence sequence is less than or equal to the historical information extraction threshold; the sequence number is used to represent the order of the text corresponding to the whole sentence vector in the sample text; Inputting the first entire sentence sequence into a historical information extraction model to obtain a historical information vector; The word-by-word vector, history information vector, speech rate label vector, and phoneme vector are input into a decoder to obtain a target synthesized speech.

8. The display device according to claim 7, wherein: The controller executes inputting the word-by-word vector, the history information vector, the speech rate label vector, and the phoneme vector into the decoder to obtain the target synthesized speech, and is specifically configured as follows: Accumulating the word-by-word vector, the history information vector, the speech rate label vector, and the phoneme vector to obtain an integrated vector; The integrated vector is input into the decoder to obtain the target synthesized speech.

9. A server, characterized in that: include: a communication unit configured to establish a communication connection with a display device; A control unit configured to: In response to the speech synthesis instruction sent by the display device, calling a language extraction model to extract word-by-word vectors and sentence-wide vectors from a sample text; the language extraction model is a model trained based on the sample text; the sample text is provided with a feature label; Extracting speech rate information from the sample audio according to the speech rate extraction model, and obtaining a speech rate label vector based on the speech rate information; the speech rate label vector is used to represent the speech rate of the words in the sample audio; The sample audio is audio data generated based on the sample text; Converting the sample text into a phoneme sequence, and obtaining a phoneme vector of the sample text based on the phoneme sequence; Inputting the word-by-word vector, the whole sentence vector, the speech rate label vector, and the phoneme vector into a decoder to obtain a target synthesized speech; Feedback receipt information is fed back to the display device, so that the display device plays the target synthesized speech according to the receipt information.

10. A speech synthesis method, characterized in that: The display device according to any one of claims 1 to 8, comprising: In response to a voice interaction instruction input by a user, a language extraction model is called to extract word-by-word vectors and sentence-wide vectors from a sample text; the language extraction model is a model trained based on feature labels; the feature labels include prosody labels, stress labels, and speech rate labels; Extracting speech rate information from the sample audio according to a speech rate extraction model, and obtaining a speech rate label vector based on the speech rate information; the speech rate label vector is used to characterize the speech rate of the words in the sample audio; the sample audio is audio data generated based on the sample text; Converting the sample text into a phoneme sequence, and obtaining a phoneme vector of the sample text based on the phoneme sequence; Inputting the word-by-word vector, the whole sentence vector, the speech rate label vector, and the phoneme vector into a decoder to obtain a target synthesized speech; Play the target synthesized speech.

Images