Voice generation method and device, electronic equipment and storage medium

Abstract

The embodiment of the invention provides a voice generation method and device, electronic equipment and a storage medium, and belongs to the technical field of artificial intelligence. The method comprises the following steps: acquiring a to-be-processed text subjected to spoken language processing; inputting the to-be-processed text into a trained word segmentation device, and outputting a text discrete representation of the to-be-processed text; inputting the text discrete representation into a trained multi-task model, and outputting a voice discrete representation having a mapping relationship with the text discrete representation; inputting the voice discrete representation into a trained decoder, and outputting frequency spectrum information; and converting the frequency spectrum information into voice audio through the trained vocoder. Therefore, the speech discrete representation containing more deep information about the speech is used as a basis for subsequently generating the speech, so that the naturalness of the generated speech is improved.

Description

Technical Field

[0001] The present application relates to the field of speech generation technology, and in particular to a speech generation method, device, electronic device and storage medium. Background Art

[0002] With the continuous development of speech generation technology, its application scope is becoming wider and wider. At present, various speech synthesis methods have been proposed, such as splicing generation, end-to-end generation, and rule-based generation. Users' requirements for the generated speech are also getting higher and higher.

[0003] In the related technologies, due to the shortcomings of splicing generation in terms of splicing coherence, the model may overfit or underfit during end-to-end generation, and the difficulty of rule-based generation in adapting to various complex language situations and voice changes, the generated speech appears stiff and mechanical. Therefore, the above-mentioned speech generation methods will result in the generated speech being less natural. Therefore, it is urgent to provide a speech generation method to solve the above-mentioned technical problems. Summary of the invention

[0004] The main purpose of the embodiments of the present application is to provide a speech generation method, device, electronic device and storage medium, which can improve the naturalness of the generated speech.

[0005] To achieve the above object, a first aspect of an embodiment of the present application provides a speech generation method, the method comprising:

[0006] Obtain the text to be processed after oral processing;

[0007] Inputting the text to be processed into a trained word segmenter, and outputting a text discrete representation of the text to be processed;

[0008] Inputting the text discrete representation into the trained multi-task model, and outputting a speech discrete representation having a mapping relationship with the text discrete representation;

[0009] Inputting the speech discrete representation into a trained decoder and outputting spectrum information;

[0010] The spectral information is converted into speech audio through a trained vocoder.

[0011] In some embodiments, the step of inputting the text to be processed into a trained word segmenter and outputting a text discrete representation of the text to be processed includes:

[0012] Performing word segmentation processing on the text to be processed by the trained word segmenter to obtain multiple target word segments;

[0013] Based on the mapping relationship between the segmentation and the identifier, determine the target identifier corresponding to each target segmentation;

[0014] According to the sorting order of the plurality of target participles in the text to be processed, the target identifiers corresponding to each target participle are sorted in turn to obtain a text discrete representation of the text to be processed.

[0015] In some embodiments, the step of inputting the text discrete representation into a trained multi-task model and outputting a speech discrete representation having a mapping relationship with the text discrete representation includes:

[0016] Performing a first preprocessing on the text discrete features by a first embedding module to obtain preprocessed text discrete features;

[0017] Inputting the preprocessed text discrete features into the trained multi-task model, and determining the weight of each partial feature corresponding to the target word in the preprocessed text discrete features through a self-attention module;

[0018] Extracting the preprocessed text discrete features according to the weight of each of the partial features to extract first semantic information;

[0019] The first semantic information is normalized, and the normalized first semantic information is converted into a speech discrete representation having a mapping relationship with the text discrete representation.

[0020] In some embodiments, obtaining the text to be processed after being processed in a spoken language includes:

[0021] Get the regularized text to be converted;

[0022] The text to be converted is input into the trained multi-task model, and the text to be processed after colloquial processing is output.

[0023] In some embodiments, the step of inputting the text to be converted into a trained multi-task model and outputting the text to be processed after being colloquially processed includes:

[0024] Determine the word vector of each word in the text to be converted by a second embedding module;

[0025] Inputting the word vector of each of the words into the trained multi-task model, and determining the weight of each of the word vectors through a self-attention module;

[0026] Performing feature extraction on the preprocessed text to be converted according to the weight of each of the word vectors to extract second semantic information;

[0027] The second semantic information is normalized, and the normalized second semantic information is converted into a colloquial text to be processed.

[0028] In some embodiments, inputting the discrete representation of speech into a trained decoder and outputting spectrum information comprises:

[0029] Extracting a three-dimensional feature map of the discrete representation of the speech through a trained decoder;

[0030] Extracting local frequency spectrum features of the three-dimensional feature map through a convolution kernel;

[0031] Inputting the local spectrum feature into a recursive layer to obtain a time series feature, wherein the time series feature is a sequence composed of spectrum features corresponding to different time points in time order;

[0032] The time series features are mapped to a specified dimensional space through a fully connected layer to obtain the frequency spectrum information of the time series features in the specified dimensional space.

[0033] In some embodiments, converting the spectrum information into speech audio by a trained vocoder includes:

[0034] Converting the spectrum information into a speech waveform through a trained vocoder;

[0035] The speech waveform is converted into an audio signal and output as speech audio in a preset playback format.

[0036] To achieve the above-mentioned purpose, a second aspect of an embodiment of the present application provides a speech generation device, comprising:

[0037] An acquisition unit, used for acquiring the text to be processed after the spoken language processing;

[0038] A first input unit, used for inputting the text to be processed into a trained word segmenter, and outputting a text discrete representation of the text to be processed;

[0039] A second input unit, used to input the text discrete representation into the trained multi-task model, and output a speech discrete representation having a mapping relationship with the text discrete representation;

[0040] A third input unit, used to input the discrete representation of speech into a trained decoder and output spectrum information;

[0041] The conversion unit is used to convert the spectrum information into speech audio through a trained vocoder.

[0042] In some embodiments, the first input unit is used to:

[0043] Performing word segmentation processing on the text to be processed by the trained word segmenter to obtain multiple target word segments;

[0044] Based on the mapping relationship between the segmentation and the identifier, determine the target identifier corresponding to each target segmentation;

[0045] According to the sorting order of the plurality of target participles in the text to be processed, the target identifiers corresponding to each target participle are sorted in turn to obtain a text discrete representation of the text to be processed.

[0046] In some embodiments, the second input unit is used to:

[0047] Performing a first preprocessing on the text discrete features by a first embedding module to obtain preprocessed text discrete features;

[0048] Inputting the preprocessed text discrete features into the trained multi-task model, and determining the weight of each partial feature corresponding to the target word in the preprocessed text discrete features through a self-attention module;

[0049] Extracting the preprocessed text discrete features according to the weight of each of the partial features to extract first semantic information;

[0050] The first semantic information is normalized, and the normalized first semantic information is converted into a speech discrete representation having a mapping relationship with the text discrete representation.

[0051] In some embodiments, the acquisition unit further includes:

[0052] Obtain a subunit for obtaining the regularized text to be converted;

[0053] The input subunit is used to input the text to be converted into the trained multi-task model and output the text to be processed after the colloquial processing.

[0054] In some embodiments, the input subunit is used to:

[0055] Determine the word vector of each word in the text to be converted by a second embedding module;

[0056] Inputting the word vector of each of the words into the trained multi-task model, and determining the weight of each of the word vectors through a self-attention module;

[0057] Performing feature extraction on the preprocessed text to be converted according to the weight of each of the word vectors to extract second semantic information;

[0058] The second semantic information is normalized, and the normalized second semantic information is converted into a colloquial text to be processed.

[0059] In some embodiments, the third input unit is used to:

[0060] Extracting a three-dimensional feature map of the discrete representation of the speech through a trained decoder;

[0061] Extracting local frequency spectrum features of the three-dimensional feature map through a convolution kernel;

[0062] Inputting the local spectrum feature into a recursive layer to obtain a time series feature, wherein the time series feature is a sequence composed of spectrum features corresponding to different time points in time order;

[0063] The time series features are mapped to a specified dimensional space through a fully connected layer to obtain the frequency spectrum information of the time series features in the specified dimensional space.

[0064] In some embodiments, the conversion unit is used to:

[0065] Converting the spectrum information into a speech waveform through a trained vocoder;

[0066] The speech waveform is converted into an audio signal and output as speech audio in a preset playback format.

[0067] To achieve the above objectives, a third aspect of an embodiment of the present application proposes an electronic device, which includes a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the method described in the first aspect is implemented.

[0068] To achieve the above-mentioned purpose, the fourth aspect of an embodiment of the present application proposes a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method described in the first aspect is implemented.

[0069] The speech generation method, device, electronic device and storage medium proposed in the present application obtain the text to be processed after the colloquial processing; input the text to be processed into a trained word segmenter, and output the text discrete representation of the text to be processed; input the text discrete representation into a trained multi-task model, and output the speech discrete representation with a mapping relationship with the text discrete representation; input the speech discrete representation into a trained decoder, and output spectrum information; and convert the spectrum information into speech audio through a trained vocoder. In this way, the speech discrete representation containing more deep information about the speech (semantics, emotions, etc.) is used as the basis for the subsequent generation of speech to improve the naturalness of the generated speech. BRIEF DESCRIPTION OF THE DRAWINGS

[0070] Figure 1 is a flow chart of the speech generation method provided in an embodiment of the present application;

[0071] Figure 2 yes Figure 1 Flow chart of step S102 in FIG.

[0072] Figure 3 yes Figure 1 Flow chart of step S103 in FIG.

[0073] Figure 4 Schematic diagram of the model architecture of the multi-task model provided in the embodiment of the present application;

[0074] Figure 5 yes Figure 1 Flow chart of step S101 in FIG.

[0075] Figure 6 yes Figure 5 Flow chart of step S402 in FIG.

[0076] Figure 7 yes Figure 1 Flow chart of step S104 in FIG.

[0077] Figure 8 yes Figure 1 Flowchart of step S105 in .

[0078] Fig. 9 is a structural schematic diagram of a speech generating device provided in an embodiment of the present application;

[0079] Fig.10 It is a schematic diagram of the hardware structure of the electronic device provided in the embodiment of the present application. DETAILED DESCRIPTION

[0080] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0081] It should be noted that, although the functional modules are divided in the device schematic diagram and the logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than the module division in the device or the order in the flowchart. The terms "first", "second", etc. in the specification, claims and the above drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence.

[0082] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.

[0083] First, some nouns involved in this application are analyzed:

[0084] Artificial intelligence (AI) is a new technical science that studies and develops theories, methods, technologies and application systems for simulating, extending and expanding human intelligence. AI is a branch of computer science that attempts to understand the essence of intelligence and produce a new type of intelligent machine that can respond in a similar way to human intelligence. Research in this field includes robotics, speech recognition, image recognition, natural language processing and expert systems. AI can simulate the information process of human consciousness and thinking. AI is also a theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results.

[0085] Network model: A network model is an abstract representation and mathematical description of various network systems in the real world. It usually consists of a set of nodes and edges connecting the nodes, and is used to simulate and analyze different types of network structures and their behaviors. The components include: 1. Nodes, which represent entities in the network. They can be physical entities such as computers, sensors, etc., or abstract concepts such as people, organizations, etc. Each node may have specific attributes such as name, status, function, etc. 2. Edges, which represent the relationship or connection between nodes. Edges can be directed or undirected, with or without weights. For example, in a social network, an edge can represent the relationship between people, and the weight can represent the strength of the relationship.

[0086] The functions of network models include: 1. Understanding and analyzing network structure: By analyzing the network model, we can understand the network's topological structure, connection patterns between nodes, centrality and other characteristics. This helps to discover key nodes, community structures and potential vulnerabilities in the network. 2. Predicting network behavior: Based on the network model, various predictions can be made, such as the speed of information propagation in the network, the reliability of the network, the behavioral changes of nodes, etc. This is very important for decision-making and risk management. 3. Optimizing network performance: Network models can be used to optimize the performance indicators of the network, such as improving the throughput of communication networks, reducing the congestion of traffic networks, and enhancing the information dissemination efficiency of social networks. By adjusting the network structure, node attributes or edge weights, the optimal network configuration can be found. 4. Designing and planning networks: Network models can provide guidance when designing new network systems or expanding existing networks. According to specific needs and goals, appropriate network models can be constructed, and simulations and analyses can be performed to determine the best design solution.

[0087] Mel Spectrogram: It is a method of representing the characteristics of sound signals. The human ear's perception of sound frequency is not linear. The Mel scale is a frequency scale based on the human ear's auditory characteristics. It transforms the frequency nonlinearly according to the human ear's perception. On an ordinary frequency scale, the distances corresponding to equidistant frequency intervals on the Mel scale are not equal. For example, the human ear is more sensitive to the frequency changes of low-frequency sounds, but relatively less sensitive to the frequency changes of high-frequency sounds.

[0088] The Mel spectrum converts the sound signal into a spectrum representation in the Mel scale by performing a series of processing on the sound signal. First, the input sound signal is framed and windowed to divide it into short time segments, each of which is called a frame. Then, a fast Fourier transform (FFT) is performed on each frame to obtain its spectrum in the normal frequency scale. Next, the spectrum is filtered through a set of Mel filter banks to convert it into a spectrum in the Mel scale. Finally, the filtered spectrum is processed by logarithmic operations and other processing to obtain the Mel spectrum.

[0089] Based on this, the embodiments of the present application provide a speech generation method, device, electronic device and storage medium, aiming to improve the naturalness of the generated speech.

[0090] The speech generation method, device, electronic device and storage medium provided in the embodiments of the present application are specifically illustrated through the following embodiments. First, the speech generation method in the embodiments of the present application is described.

[0091] The embodiments of the present application can acquire and process relevant data based on artificial intelligence technology. Among them, artificial intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results.

[0092] AI basic technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing technology, operation / interaction systems, mechatronics, etc. AI software technologies mainly include computer vision technology, robotics technology, biometrics technology, speech processing technology, natural language processing technology, and machine learning / deep learning.

[0093] The speech generation method provided in the embodiment of the present application relates to the field of artificial intelligence technology. The speech generation method provided in the embodiment of the present application can be applied to a terminal, can be applied to a server side, or can be software running in a terminal or a server side. In some embodiments, the terminal can be a smart phone, a tablet computer, a laptop computer, a desktop computer, etc.; the server side can be configured as an independent physical server, or a server cluster or a distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms; the software can be an application that implements the speech generation method, etc., but is not limited to the above forms.

[0094] The present application can be used in many general or special computer system environments or configurations. For example: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, etc. The present application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The present application can also be practiced in distributed computing environments, in which tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media including storage devices.

[0095] It should be noted that in each specific implementation of the present application, when it comes to the need to perform relevant processing based on data related to user identity or characteristics such as user information, user behavior data, user historical data, and user location information, the user's permission or consent will be obtained first, and the collection, use, and processing of these data will comply with relevant laws, regulations, and standards. In addition, when the embodiment of the present application needs to obtain the user's sensitive personal information, the user's separate permission or consent will be obtained through a pop-up window or by jumping to a confirmation page. After clearly obtaining the user's separate permission or consent, the necessary user-related data for the normal operation of the embodiment of the present application will be obtained.

[0096] Figure 1 is an optional flowchart of the speech generation method provided in an embodiment of the present application. Figure 1 The method may include but is not limited to steps S101 to S105.

[0097] Step S101, obtaining the text to be processed after being processed into spoken language.

[0098] Among them, with the development of artificial intelligence (AI) technology, more and more application scenarios involve generating voice through AI to reply to users. For example, online customer service systems: on websites, mobile applications and other platforms, intelligent customer service can understand users' questions through text input, and then use voice generation models to generate natural and smooth voice responses to provide users with a more convenient service experience. For example, when users ask for product information, encounter technical problems or need after-sales service, intelligent customer service can quickly give accurate voice answers, reduce the user's reading burden, and improve problem solving efficiency; or smart speakers as the control center of home smart devices can answer users' questions through voice generation models, play music, news, stories and other content; or in-vehicle navigation systems can provide drivers with route guidance, road conditions information, etc. through voice generation models. For example, when the driver enters the destination, the system can generate voice prompts to guide the driver to follow the best route. At the same time, the system can also broadcast road conditions information in real time, such as traffic congestion and accidents, to help drivers adjust their driving routes in time.

[0099] Based on this, the focus of the embodiments of the present application is that after the model determines the reply content based on the user's input message, a series of naturalization processes are performed on the reply content so that the speech generated by the final model is more natural.

[0100] Specifically, in the related art, the basis for generating speech determined by the model is regularized content, and the text content is relatively standardized and unified. However, precisely because the text content is too regularized, the naturalness of the speech generated based on the regularized text to be processed is not high. Therefore, the embodiment of the present application first obtains the text to be processed after colloquial processing, so that when generating speech subsequently, the colloquial text to be processed is used as the basis to improve the naturalness and humanization of the generated speech.

[0101] For example, the basic text decided by the model for generating speech is "The weather is very good today". After being colloquially processed, the text to be processed becomes "Hey, the weather is very good today."

[0102] Step S102: input the text to be processed into a trained word segmenter, and output a text discrete representation of the text to be processed.

[0103] The trained tokenizer is mainly responsible for converting the input text to be processed into a digital vector representation suitable for model processing, that is, text discrete representation. It maps each word or character in the text to a high-dimensional vector space so that the text can be numerically calculated and processed by the model.

[0104] For example, for the input text "The weather is very good today", the trained tokenizer will convert "today", "weather", "very", and "good" into different vectors, such as [0.1, 0.2, 0.3], [0.4, 0.5, 0.6], [0.7, 0.8, 0.9], [1.0, 1.1, 1.2]. These vectors can be used as input for subsequent model processing to help the model understand the meaning of the text.

[0105] Step S103: input the text discrete representation into the trained multi-task model, and output a speech discrete representation having a mapping relationship with the text discrete representation.

[0106] Among them, the trained multi-task model can learn the complex mapping relationship between text discrete representation and speech discrete representation after being trained with a large amount of data. After inputting the text discrete representation, the model will output the corresponding speech discrete representation, which can contain some potential feature information of speech.

[0107] For example, the representation of "weather" in the text discrete representation is [0.7, 0.8, 0.9]. After being processed by the multi-task model, the output speech discrete representation corresponding to "weather" may be [0.5, 0.6, 0.7, 0.8]. This new representation may contain potential feature information such as pitch and timbre related to the word "weather".

[0108] Compared with the method of converting text features into Mel spectrum to generate speech in related technologies, the embodiment of the present application generates speech through discrete representation of speech, which is usually an abstract representation obtained through deep model learning, which can contain more in-depth information about speech. In contrast, Mel spectrum mainly focuses on the representation of frequency features and is relatively simple. Discrete representation of speech may integrate information such as semantics, emotion, speaker characteristics, etc. of speech, and can more comprehensively characterize the essential characteristics of speech, thereby producing a more natural effect when synthesizing speech.

[0109] Moreover, for different speakers, contexts, and emotional states, discrete speech representations can better capture these differences and reflect them in synthesized speech. For example, discrete speech representations of different speakers can reflect their unique pronunciation habits, intonation changes, etc., making the synthesized speech more personalized and natural.

[0110] Step S104: input the discrete representation of speech into the trained decoder, and output spectrum information.

[0111] The role of the decoder is to convert the discrete representation of speech into spectral information, which describes the intensity distribution, pitch, timbre and other characteristics of the speech signal at different frequencies.

[0112] For example, if the input speech is discretely represented as [0.2, 0.3, 0.4, 0.5], the decoder may output a spectrum information after processing, such as the amplitude values ​​in a certain frequency range are [10, 20, 30, 40], etc. This spectrum information can be used to describe the intensity of a piece of speech at different frequencies.

[0113] Step S105: converting the spectrum information into speech audio through the trained vocoder.

[0114] After acquiring the frequency spectrum information including the intensity distribution, pitch, timbre and other features at different frequencies in step S104, the frequency spectrum information can be synthesized into an audible speech audio signal through a trained vocoder.

[0115] For example, the spectral information describes the intensity distribution, pitch, timbre and other characteristics of each unit in "today", "weather", "old", and "good" at different frequencies. The trained vocoder converts it into actual audible speech audio, and the sound played back is the speech content of "Hey, the weather is really good today."

[0116] Steps S101 to S105 shown in the embodiment of the present application are as follows: obtaining the text to be processed after the colloquial processing; inputting the text to be processed into the trained word segmenter, outputting the text discrete representation of the text to be processed; inputting the text discrete representation into the trained multi-task model, outputting the speech discrete representation having a mapping relationship with the text discrete representation; inputting the speech discrete representation into the trained decoder, outputting the spectrum information; and converting the spectrum information into speech audio through the trained vocoder. In this way, the speech discrete representation containing more deep information about the speech (semantics, emotions, speaker characteristics, etc.) is used as the basis for the subsequent generation of speech to improve the naturalness of the generated speech.

[0117] See also Figure 2 In some embodiments, step S102 may include but is not limited to steps S201 to S203:

[0118] Step S201, performing word segmentation processing on the text to be processed by a trained word segmenter to obtain a plurality of target word segments.

[0119] The trained word segmenter is designed based on certain algorithms and rules and has been trained with a large amount of corpus. Its function is to split continuous text strings into relatively independent word units according to semantic and grammatical rules. These split word units are the target word segmentations.

[0120] For example, if the text to be processed is "I love natural language processing technology", the trained tokenizer will split it into target tokens such as "I", "love", "natural language processing", and "technology". This is because it has learned that "natural language processing" is a professional term in a specific field and should be treated as a whole vocabulary unit.

[0121] Step S202: determining a target identifier corresponding to each target word segmentation based on a mapping relationship between word segments and identifiers.

[0122] When building speech synthesis systems and other related applications, a mapping table between word segments and identifiers is created to enable computers to process text information efficiently. Each word segment in this mapping table corresponds to a unique identifier, usually a number or a specific code. When the target word segment is obtained, the target identifier corresponding to each target word segment can be found by querying this mapping table.

[0123] For example, there is the following mapping relationship: "I" corresponds to 101; "love" corresponds to 102; "natural language processing" corresponds to 103; "technology" corresponds to 104. For the target word segmentations "I", "love", "natural language processing", and "technology", their corresponding target identifiers are 101, 102, 103, and 104 respectively.

[0124] Step S203 , sorting the target identifiers corresponding to each target segmentation in turn according to the sorting order of the multiple target segmentations in the text to be processed, to obtain a text discrete representation of the text to be processed.

[0125] The order of words in the text is crucial to express semantics. After obtaining the target identifier corresponding to each target word, it is necessary to sort the identifiers according to the order of these target words in the original text to be processed. The ordered identifier sequence obtained in this way is the text discrete representation, which represents the text information in a discrete form that is easy for computers to process and understand.

[0126] For example, the text to be processed is "I love natural language processing technology", and the target identifiers corresponding to the target words "I", "love", "natural language processing", and "technology" are 101, 102, 103, and 104 respectively. These identifiers are arranged in the order they appear in the text, and the resulting discrete representation of the text is [101, 102, 103, 104].

[0127] In this way, in the process of generating discrete representation of text, the identifiers are arranged strictly according to the sorting order of the target words in the original text. This ensures that the semantic information of the text is fully preserved during the discretization process, and the computer can accurately restore the semantic structure of the text according to the order of the identifiers in the subsequent processing process.

[0128] See also Figure 3 In some embodiments, step S103 may include but is not limited to steps S301 to S304:

[0129] Step S301: performing a first preprocessing on the text discrete features through a first embedding module to obtain preprocessed text discrete features.

[0130] Among them, see Figure 4 , Figure 4 Model architecture diagram of the multi-task model provided for the embodiment of the present application. The multi-task model can perform two tasks, the first task is to convert the text discrete representation into the corresponding speech discrete representation, and the second task is to convert the regularized text to be converted into the text to be processed after the spoken processing. And the multi-task model includes a 20-layer decoder layer (LlamaDecoderLayer) to implement, each layer includes self-attention (LlamaSdpaAttention), multi-layer perceptron (LlamaMLP), layer normalization (LlamaRMSNorm) and post-normalization (LlamaRMSNorm) four modules. Before inputting to the Decoder Layer, the text information and text discrete features are respectively embedded in the input information by the text embedding module (embed_text) and the discrete representation embedding module (embed_code), and the channel conversion, and the output information is converted into the number of channels required for the task by the 4-layer linear layer (Linear) and the 1-layer Linear in the output layer.

[0131] Specifically, for the embodiment of the present application, which is used to convert text discrete features into speech discrete representations having a mapping relationship with the text discrete representations, it is necessary to perform a first preprocessing on the text discrete features through a discrete representation embedding module (embed_code), that is, a first embedding module, to obtain text discrete features that are subsequently suitable for processing by a multi-task model.

[0132] The first preprocessing is to perform some form of transformation or encoding optimization on the input text discrete features. It can adjust the dimension of the features, fuse information, etc., so that the subsequent modules can better process these features.

[0133] For example, a text discrete feature is a matrix with a dimension of [100,5], which means there are 100 text discrete feature vectors, each with a dimension of 5. The first embedding module may convert these features into a matrix with a dimension of [100,10] through linear transformation, etc. The new matrix is ​​the preprocessed text discrete feature. For example, an original feature vector [1,2,3,4,5] may become [1.5,3,4.5,6,7.5,8,9,10,11,12] after being processed by the first embedding module.

[0134] Step S302: input the preprocessed text discrete features into the trained multi-task model, and determine the weight of each partial feature corresponding to the target word in the preprocessed text discrete features through a self-attention module.

[0135] Among them, the self-attention module allows the model to automatically learn the importance of different parts of the text. For each partial feature corresponding to the target word, the self-attention mechanism assigns a weight based on its association with other partial features and the overall context information. A high weight means that the partial feature is more important in the current context.

[0136] For example, the preprocessed text discrete features correspond to the text "I love natural language processing technology", where "I", "love", "natural language processing", and "technology" correspond to partial features. The self-attention module will assign a higher weight (such as 0.8) to the partial feature "natural language processing" based on the overall semantics and context, because it may be the core concept in this text; and assign a relatively low weight (such as 0.3) to "I".

[0137] Step S303: extracting the preprocessed text discrete features according to the weight of each of the partial features to extract the first semantic information.

[0138] The weights determined in step S302 are used to perform weighted summation and other operations on the partial features to extract key semantic information. The partial features with greater weights contribute more to constructing semantic information.

[0139] For example, if the partial features of "I" are [1,2,3], with a weight of 0.3; the partial features of "love" are [4,5,6], with a weight of 0.4; the partial features of "natural language processing" are [7,8,9], with a weight of 0.8; the partial features of "technology" are [10,11,12], with a weight of 0.5. Then when extracting the first semantic information, the calculation is: 0.3×[1,2,3]+0.4×[4,5,6]+0.8×[7,8,9]+0.5×[10,11,12]=[9.9,11.8,13.7].

[0140] Step S304: normalize the first semantic information, and convert the normalized first semantic information into a speech discrete representation having a mapping relationship with the text discrete representation.

[0141] The purpose of normalization is to make the data distribution more stable and standardized, usually by mapping the data to a specific range (such as 0 to 1 or -1 to 1). The normalized first semantic information is then converted into a discrete representation of speech, and a mapping from text features to speech features is established.

[0142] Assume that the first semantic information is [9.9, 11.8, 13.7], and it is mapped to the range of [0, 1] through normalization, which may become [0.3, 0.4, 0.5]. Then, through the specific conversion mechanism in the model, this normalized information is converted into a discrete speech representation, such as a new vector [0.2, 0.3, 0.4]. This discrete speech representation has a specific mapping relationship with the original text discrete representation, which is used for subsequent speech synthesis-related operations.

[0143] In this way, the self-attention module can automatically learn and assign weights based on the associations between different parts of the text and the overall context, allowing the model to focus on the more critical parts of the text, thereby better retaining the core semantic information in subsequent tasks such as speech synthesis. In addition, this weight-based feature extraction method makes the extracted semantic information more logically coherent. The weight reflects the semantic importance of the feature in the text. These features are organically combined through weighted summation to form a more coherent and holistic semantic information representation, which is conducive to generating natural and fluent speech.

[0144] See also Figure 5 In some embodiments, step S101 may include but is not limited to steps S401 to S402:

[0145] Step S401, obtaining the regularized text to be converted.

[0146] Among them, the text to be converted is the reply content in the form of text generated by AI, which is a regularized text that complies with the standards.

[0147] Step S402: input the text to be converted into the trained multi-task model, and output the text to be processed after being processed in colloquial language.

[0148] Among them, the second task of the multi-task model is to convert the regularized text to be converted into the text to be processed after colloquial processing. Therefore, the text to be converted is input into the trained multi-task model to process the text to be converted through the multi-task model and output the text to be processed after colloquial processing.

[0149] In this way, the regularized text to be converted is converted into the text to be processed after colloquial processing, making the text closer to people's daily oral expression habits. For example, some relatively stiff written expressions are converted into more natural and colloquial expressions, so that the voice played after speech synthesis sounds more friendly and natural, improving the naturalness of the subsequent generated speech.

[0150] See also Figure 6 In some embodiments, step S402 may include but is not limited to steps S501 to S504:

[0151] Step S501: determining the word vector of each word in the text to be converted by a second embedding module.

[0152] Among them, Figure 3 As shown in the figure, when a multi-task model is used to convert the regularized text to be converted into a colloquial text to be processed, each word in the text needs to be converted into a mathematical vector representation through a text embedding module, that is, the second embedding module. This vector representation can capture the semantics, grammar and other information of the word. It establishes a mapping relationship between words and vectors by learning a large amount of text data.

[0153] For example, the text to be converted is "Today is sunny." The second embedding module may map the word "today" to a vector, such as [0.1, 0.2, 0.3]; "sunshine" to [0.4, 0.5, 0.6]; "very good" to [0.7, 0.8, 0.9]. These vectors are the word vectors of each word.

[0154] Step S502, inputting the word vector of each of the words into the trained multi-task model, and determining the weight of each of the word vectors through the self-attention module.

[0155] The self-attention module determines the weight of each word vector based on the relationship between word vectors and the contextual information of the entire text. In a text, the importance of different words is different, and the self-attention mechanism can automatically learn and assign weights.

[0156] For example, for the sentence “The sun is shining brightly today”, the self-attention module may judge from the context that the word “sunshine” is more important in expressing the core meaning of the text, so it assigns a higher weight, such as 0.8, to the word vector corresponding to “sunshine”; while “today” is relatively less important and is assigned a weight of 0.3; and “very good” is assigned a weight of 0.5.

[0157] Step S503: performing feature extraction on the preprocessed text to be converted according to the weight of each word vector to extract second semantic information.

[0158] According to the weights determined in step S502, weighted summation and other operations are performed on each word vector to extract key information that can represent the meaning of the entire text, that is, the second semantic information.

[0159] For example, the word vector of "today" is [0.1, 0.2, 0.3], with a weight of 0.3; the word vector of "sunshine" is [0.4, 0.5, 0.6], with a weight of 0.8; and the word vector of "very good" is [0.7, 0.8, 0.9], with a weight of 0.5.

[0160] Then when extracting the second semantic information, the calculation is:

[0161] 0.3×[0.1,0.2,0.3]+0.8×[0.4,0.5,0.6]+0.5×[0.7,0.8,0.9]=[0.71,0.86,1.01].

[0162] Step S504: normalize the second semantic information and convert the normalized second semantic information into a colloquial text to be processed.

[0163] The purpose of normalization is to make the data distribution more stable and standardized, usually by mapping the data to a specific range (such as 0 to 1 or -1 to 1). The normalized second semantic information is then converted into colloquial text to be processed through a specific conversion mechanism.

[0164] For example, the second semantic information [0.71, 0.86, 1.01] becomes [0.4, 0.5, 0.6] after normalization, and then the processed information is converted into a colloquial text form through the linear layer in the model, converting it into a colloquial text to be processed, such as "It's a nice sunny day."

[0165] In this way, through the collaborative work of multiple modules, the regularized text is gradually converted into colloquial text, achieving the transformation of text style. The colloquial text is used as the basis for subsequent speech generation, thereby improving the adaptability and naturalness of the generated speech.

[0166] See also Figure 7 In some embodiments, step S104 may include but is not limited to steps S601 to S604:

[0167] Step S601, extracting a three-dimensional feature map of the discrete representation of the speech through a trained decoder.

[0168] The trained decoder can convert the input discrete representation of speech into a form that is easier to understand and process. Here, it extracts a three-dimensional feature map, which can contain different aspects of information, such as frequency, time, and a specific speech feature in three dimensions.

[0169] For example, the discrete representation of speech is a complex vector representation. After being processed by the decoder, a three-dimensional feature map is obtained. For example, the dimension is (10, 20, 30), which represents 10 time slices, 20 frequency intervals and 30 specific speech feature combinations.

[0170] Step S602: extracting local frequency spectrum features of the three-dimensional feature map through a convolution kernel.

[0171] Among them, the convolution kernel extracts specific local spectral features by sliding on the three-dimensional feature map and performing weighted summation and other operations on the local area.

[0172] Step S603: input the local spectrum feature into a recursive layer to obtain a time series feature, where the time series feature is a sequence of spectrum features corresponding to different time points in chronological order.

[0173] Among them, the recurrent layer (such as the long short-term memory network (LSTM) or the gated recurrent unit (GRU) in the recurrent neural network) can process time series data. After the local spectrum features are input into the recurrent layer, the recurrent layer can capture the dependencies between different time points, thereby obtaining the time series features.

[0174] For example, suppose that at different time points, the local spectrum features are [0.1, 0.2, 0.3], [0.4, 0.5, 0.6], [0.7, 0.8, 0.9], etc. The recursive layer will learn the changing rules in the time series based on the features of these different time points and the time sequence between them, and obtain a time series feature.

[0175] Step S604: Map the time series features to a specified dimensional space through a fully connected layer to obtain frequency spectrum information of the time series features in the specified dimensional space.

[0176] Among them, the fully connected layer can perform a linear transformation on the input features and map them to a specified dimensional space. Here, the fully connected layer maps the time series features to a dimensional space suitable for subsequent processing or output to obtain the final spectrum information.

[0177] For example, a time series feature is a vector with a dimension of 50. Through the mapping of the fully connected layer, it is converted into a vector with a dimension of 100. This new vector is the spectrum information in the specified dimensional space, which can be used for further speech synthesis or other processing.

[0178] In this way, the trained decoder converts the discrete representation of speech into a three-dimensional feature map, making the originally abstract and complex speech information easier to understand and process. For example, the conversion from a vector representation that is difficult to interpret directly to a three-dimensional feature map with a clear dimensional meaning enables researchers and algorithms to better analyze and grasp the different aspects of speech. And through linear transformation, the fully connected layer can enhance the expressive power of features and extract more representative and discriminative information. This helps to improve the quality and accuracy of speech synthesis, making the synthesized speech more natural and realistic.

[0179] See also Figure 8 In some embodiments, step S105 may include but is not limited to steps S701 to S702:

[0180] Step S701: convert the spectrum information into a speech waveform through a trained vocoder.

[0181] Among them, the vocoder is a tool that can convert spectrum information into speech waveforms. After training, the vocoder learns the mapping relationship between spectrum features and speech waveforms. Spectral information describes the characteristics of speech signals such as the intensity distribution at different frequencies, while the speech waveform is the actual change of the sound signal over time. The vocoder converts the input spectrum information into a continuous speech waveform through specific algorithms and models.

[0182] For example, the spectral information indicates the distribution of energy at different frequencies, such as higher energy at some frequencies and lower energy at other frequencies. Based on these spectral features, the vocoder generates a speech waveform that varies over time, where the shape and amplitude of the wave reflect the energy at different frequencies and how it varies over time. For example, a speech waveform is generated that resembles a combination of sine waves, where the amplitudes and phases of the sine waves at different frequencies are determined based on the spectral information.

[0183] Step S702, converting the speech waveform into an audio signal, and outputting the audio signal as a speech audio in a preset playback format.

[0184] The speech waveform itself is a digital signal representation, but in order to be played and heard, it needs to be converted into an audio signal. This process usually involves converting the digital waveform into an analog signal (if it needs to be played through a device such as a speaker). At the same time, in order to use it in different playback devices and environments, the audio signal needs to be output to a preset playback format, such as the common MP3, WAV and other formats.

[0185] For example, a speech waveform is a series of digital values ​​that represent the intensity of sound at different points in time. Through technologies such as digital-to-analog conversion (DAC), these digital values ​​are converted into analog electrical signals, which can drive the speaker to produce sound. Then, this audio signal is encoded and stored according to a preset playback format, such as saved as a WAV format file. In this way, this file can be played on various audio playback software and devices and output as audible speech audio.

[0186] Therefore, since the trained vocoder has learned the mapping relationship between spectral features and speech waveforms, it can accurately generate speech waveforms based on spectral information. This means that the characteristics of the original speech, including timbre, pitch, rhythm, etc., can be restored as much as possible. For example, for the speech of a specific speaker, the vocoder can generate a speech waveform with similar timbre and intonation based on its spectral features, thereby improving the quality of speech synthesis.

[0187] The embodiment of the present application obtains the text to be processed after the colloquial processing; inputs the text to be processed into the trained word segmenter, outputs the text discrete representation of the text to be processed; inputs the text discrete representation into the trained multi-task model, outputs the speech discrete representation with the text discrete representation; inputs the speech discrete representation into the trained decoder, outputs the spectrum information; and converts the spectrum information into speech audio through the trained vocoder. In this way, the spectrum information is output by the speech discrete representation containing more deep information about the speech (semantics, emotions, speaker characteristics, etc.) to improve the naturalness of the generated speech.

[0188] See also Fig. 9 The present application also provides a speech generation device that can implement the above-mentioned speech generation method. The device includes:

[0189] An acquisition unit, used for acquiring the text to be processed after the spoken language processing;

[0190] A first input unit, used for inputting the text to be processed into a trained word segmenter, and outputting a text discrete representation of the text to be processed;

[0191] A second input unit, used to input the text discrete representation into the trained multi-task model, and output a speech discrete representation having a mapping relationship with the text discrete representation;

[0192] A third input unit, used to input the discrete representation of speech into a trained decoder and output spectrum information;

[0193] The conversion unit is used to convert the spectrum information into speech audio through a trained vocoder.

[0194] In some embodiments, the first input unit is used to:

[0195] Performing word segmentation processing on the text to be processed by the trained word segmenter to obtain multiple target word segments;

[0196] Based on the mapping relationship between the segmentation and the identifier, determine the target identifier corresponding to each target segmentation;

[0197] According to the sorting order of the plurality of target participles in the text to be processed, the target identifiers corresponding to each target participle are sorted in turn to obtain a text discrete representation of the text to be processed.

[0198] In some embodiments, the second input unit is used to:

[0199] Performing a first preprocessing on the text discrete features by a first embedding module to obtain preprocessed text discrete features;

[0200] Inputting the preprocessed text discrete features into the trained multi-task model, and determining the weight of each partial feature corresponding to the target word in the preprocessed text discrete features through a self-attention module;

[0201] Extracting the preprocessed text discrete features according to the weight of each of the partial features to extract first semantic information;

[0202] The first semantic information is normalized, and the normalized first semantic information is converted into a speech discrete representation having a mapping relationship with the text discrete representation.

[0203] In some embodiments, the acquisition unit further includes:

[0204] Obtain a subunit for obtaining the regularized text to be converted;

[0205] The input subunit is used to input the text to be converted into the trained multi-task model and output the text to be processed after the colloquial processing.

[0206] In some embodiments, the input subunit is used to:

[0207] Determine the word vector of each word in the text to be converted by a second embedding module;

[0208] Inputting the word vector of each of the words into the trained multi-task model, and determining the weight of each of the word vectors through a self-attention module;

[0209] Performing feature extraction on the preprocessed text to be converted according to the weight of each of the word vectors to extract second semantic information;

[0210] The second semantic information is normalized, and the normalized second semantic information is converted into a colloquial text to be processed.

[0211] In some embodiments, the third input unit is used to:

[0212] Extracting a three-dimensional feature map of the discrete representation of the speech through a trained decoder;

[0213] Extracting local frequency spectrum features of the three-dimensional feature map through a convolution kernel;

[0214] Inputting the local spectrum feature into a recursive layer to obtain a time series feature, wherein the time series feature is a sequence composed of spectrum features corresponding to different time points in time order;

[0215] The time series features are mapped to a specified dimensional space through a fully connected layer to obtain the frequency spectrum information of the time series features in the specified dimensional space.

[0216] In some embodiments, the conversion unit is used to:

[0217] Converting the spectrum information into a speech waveform through a trained vocoder;

[0218] The speech waveform is converted into an audio signal and output as speech audio in a preset playback format.

[0219] The speech generation device provided in the embodiment of the present application obtains the text to be processed after the colloquial processing through the acquisition unit; the first input unit inputs the text to be processed into the trained word segmenter, and outputs the text discrete representation of the text to be processed; the second input unit inputs the text discrete representation into the trained multi-task model, and outputs the speech discrete representation having a mapping relationship with the text discrete representation; the third input unit inputs the speech discrete representation into the trained decoder, and outputs the spectrum information; the conversion unit converts the spectrum information into speech audio through the trained vocoder. In this way, the speech discrete representation containing more deep information about the speech (semantics, emotions, speaker characteristics, etc.) is used as the basis for the subsequent generation of speech to improve the naturalness of the generated speech. In the process of generating the text discrete representation, the identifiers are arranged strictly according to the sorting order of the target word segmentation in the original text. This ensures that the semantic information of the text is completely retained during the discretization process, and the computer can accurately restore the semantic structure of the text according to the order of the identifiers in the subsequent processing process. The self-attention module can automatically learn and assign weights according to the association between different parts of the text and the overall context, so that the model can focus on the more critical parts of the text, thereby better retaining the core semantic information in subsequent tasks such as speech synthesis. In addition, this weight-based feature extraction method makes the extracted semantic information more logically coherent. The weight reflects the semantic importance of the feature in the text. These features are organically combined through weighted summation to form a more coherent and holistic semantic information representation, which is conducive to generating natural and fluent speech. Convert the regularized text to be converted into the text to be processed after colloquial processing, so that the text is closer to people's daily oral expression habits. For example, some relatively stiff written expressions are converted into more natural and colloquial expressions, so that the voice played after speech synthesis sounds more intimate and natural, and the naturalness of the subsequent generated speech is improved. Through the collaborative work of multiple modules, the regularized text is gradually converted into colloquial text, realizing the conversion of text style, and taking the colloquial text as the basis for subsequent speech generation, improving the adaptability and naturalness of the generated speech. The trained decoder converts the discrete representation of speech into a three-dimensional feature map, making the originally abstract and complex speech information easier to understand and process. For example, the conversion from a vector representation that is difficult to interpret directly to a three-dimensional feature map with a clear dimensional meaning enables researchers and algorithms to better analyze and grasp the different aspects of speech. And through linear transformation, the fully connected layer can enhance the expressive power of features and extract more representative and discriminative information. This helps to improve the quality and accuracy of speech synthesis, making the synthesized speech more natural and realistic. Because the trained vocoder learns the mapping relationship between spectral features and speech waveforms, it can accurately generate speech waveforms based on spectral information.This means that the characteristics of the original speech can be restored as much as possible, including timbre, pitch, rhythm, etc. For example, for the speech of a specific speaker, the vocoder can generate a speech waveform with similar timbre and intonation based on its spectral characteristics, thereby improving the quality of speech synthesis.

[0220] The specific implementation of the speech generation device is basically the same as the specific implementation of the above-mentioned speech generation method, and will not be repeated here.

[0221] The embodiment of the present application also provides an electronic device, the electronic device includes a memory and a processor, the memory stores a computer program, and the processor implements the above-mentioned speech generation method when executing the computer program. The electronic device can be any intelligent terminal including a tablet computer, a car computer, etc.

[0222] See also Fig.10 , Fig.10 The hardware structure of an electronic device of another embodiment is illustrated, and the electronic device includes:

[0223] The processor 901 may be implemented by a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (Application Specific Integrated Circuit, ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of the present application;

[0224] The memory 902 can be implemented in the form of a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 902 can store an operating system and other application programs. When the technical solution provided in the embodiment of this specification is implemented by software or firmware, the relevant program code is stored in the memory 902, and the processor 901 calls and executes the speech generation method of the embodiment of this application;

[0225] Input / output interface 903, used to implement information input and output;

[0226] Communication interface 904, used to realize communication interaction between the device and other devices, which can be realized by wired mode (such as USB, network cable, etc.) or wireless mode (such as mobile network, WIFI, Bluetooth, etc.);

[0227] A bus 905 that transmits information between various components of the device (e.g., the processor 901, the memory 902, the input / output interface 903, and the communication interface 904);

[0228] The processor 901 , the memory 902 , the input / output interface 903 and the communication interface 904 are connected to each other in communication within the device via a bus 905 .

[0229] An embodiment of the present application also provides a computer-readable storage medium, which stores a computer program, and when the computer program is executed by a processor, the above-mentioned speech generation method is implemented.

[0230] The memory, as a non-transient computer-readable storage medium, can be used to store non-transient software programs and non-transient computer executable programs. In addition, the memory may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some embodiments, the memory may optionally include a memory remotely disposed relative to the processor, and these remote memories may be connected to the processor via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0231] The speech generation method, speech generation device, electronic device and storage medium provided in the embodiment of the present application obtain the text to be processed after the colloquial processing; input the text to be processed into a trained word segmenter, and output the text discrete representation of the text to be processed; input the text discrete representation into a trained multi-task model, and output the speech discrete representation with a mapping relationship with the text discrete representation; input the speech discrete representation into a trained decoder, and output spectrum information; and convert the spectrum information into speech audio through a trained vocoder. In this way, the spectrum information is output by the speech discrete representation containing more deep information about the speech (semantics, emotions, speaker characteristics, etc.) to improve the naturalness of the generated speech.

[0232] The embodiments described in the embodiments of the present application are intended to more clearly illustrate the technical solutions of the embodiments of the present application and do not constitute a limitation on the technical solutions provided in the embodiments of the present application. Those skilled in the art will appreciate that with the evolution of technology and the emergence of new application scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.

[0233] Those skilled in the art will appreciate that the technical solutions shown in the figures do not constitute a limitation on the embodiments of the present application, and may include more or fewer steps than shown in the figures, or a combination of certain steps, or different steps.

[0234] The device embodiments described above are merely illustrative, and the units described as separate components may or may not be physically separated, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0235] Those skilled in the art will appreciate that all or some of the steps in the methods disclosed above, and the functional modules / units in the systems and devices may be implemented as software, firmware, hardware, or a suitable combination thereof.

[0236] The terms "first", "second", "third", "fourth", etc. (if any) in the specification of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0237] It should be understood that in the present application, "at least one (item)" means one or more, and "plurality" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the objects associated before and after are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0238] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the above units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0239] The units described above as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0240] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0241] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including multiple instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of various embodiments of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory, referred to as ROM), random access memory (Random Access Memory, referred to as RAM), disk or optical disk and other media that can store programs.

[0242] The preferred embodiments of the present invention are described above with reference to the accompanying drawings, but the scope of the rights of the present invention is not limited thereto. Any modification, equivalent substitution and improvement made by a person skilled in the art without departing from the scope and essence of the present invention should be within the scope of the rights of the present invention.

Claims

1. A speech generation method, characterized in that: include: Obtain the text to be processed after oral processing; Inputting the text to be processed into a trained word segmenter, and outputting a text discrete representation of the text to be processed; Inputting the text discrete representation into the trained multi-task model, and outputting a speech discrete representation having a mapping relationship with the text discrete representation; Inputting the speech discrete representation into a trained decoder and outputting spectrum information; The spectral information is converted into speech audio through a trained vocoder.

2. The speech generation method according to claim 1, characterized in that: The step of inputting the text to be processed into a trained word segmenter and outputting a text discrete representation of the text to be processed includes: Performing word segmentation processing on the text to be processed by the trained word segmenter to obtain multiple target word segments; Based on the mapping relationship between the segmentation and the identifier, determine the target identifier corresponding to each target segmentation; According to the sorting order of the plurality of target participles in the text to be processed, the target identifiers corresponding to each target participle are sorted in turn to obtain a text discrete representation of the text to be processed.

3. The speech generation method according to claim 2, characterized in that: The step of inputting the text discrete representation into the trained multi-task model and outputting a speech discrete representation having a mapping relationship with the text discrete representation includes: Performing a first preprocessing on the text discrete features by a first embedding module to obtain preprocessed text discrete features; Inputting the preprocessed text discrete features into the trained multi-task model, and determining the weight of each partial feature corresponding to the target word in the preprocessed text discrete features through a self-attention module; Extracting the preprocessed text discrete features according to the weight of each of the partial features to extract first semantic information; The first semantic information is normalized, and the normalized first semantic information is converted into a speech discrete representation having a mapping relationship with the text discrete representation.

4. The speech generation method according to claim 1, characterized in that: The step of obtaining the text to be processed after the spoken language processing includes: Get the regularized text to be converted; The text to be converted is input into the trained multi-task model, and the text to be processed after colloquial processing is output.

5. The speech generation method according to claim 4, characterized in that: The step of inputting the text to be converted into the trained multi-task model and outputting the text to be processed after being processed in a colloquial way comprises: Determine the word vector of each word in the text to be converted by a second embedding module; Inputting the word vector of each of the words into the trained multi-task model, and determining the weight of each of the word vectors through a self-attention module; Performing feature extraction on the preprocessed text to be converted according to the weight of each of the word vectors to extract second semantic information; The second semantic information is normalized, and the normalized second semantic information is converted into a colloquial text to be processed.

6. The speech generation method according to any one of claims 1 to 5, characterized in that: The step of inputting the discrete representation of speech into a trained decoder and outputting spectrum information comprises: Extracting a three-dimensional feature map of the discrete representation of the speech through a trained decoder; Extracting local frequency spectrum features of the three-dimensional feature map through a convolution kernel; Inputting the local spectrum feature into a recursive layer to obtain a time series feature, wherein the time series feature is a sequence composed of spectrum features corresponding to different time points in time order; The time series features are mapped to a specified dimensional space through a fully connected layer to obtain the frequency spectrum information of the time series features in the specified dimensional space.

7. The speech generation method according to any one of claims 1 to 5, characterized in that: The converting the spectrum information into speech audio by the trained vocoder comprises: Converting the spectrum information into a speech waveform through a trained vocoder; The speech waveform is converted into an audio signal and output as speech audio in a preset playback format.

8. A speech generating device, characterized in that: include: An acquisition unit, used for acquiring the text to be processed after the spoken language processing; A first input unit, used for inputting the text to be processed into a trained word segmenter, and outputting a text discrete representation of the text to be processed; A second input unit, used to input the text discrete representation into the trained multi-task model, and output a speech discrete representation having a mapping relationship with the text discrete representation; A third input unit, used to input the discrete representation of speech into a trained decoder and output spectrum information; The conversion unit is used to convert the spectrum information into speech audio through a trained vocoder.

9. An electronic device, characterized in that: The electronic device comprises a memory and a processor, the memory stores a computer program, and the processor implements the speech generation method according to any one of claims 1 to 7 when executing the computer program.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the speech generation method according to any one of claims 1 to 7 is implemented.

Images