Cross-language speech generation method based on artificial intelligence and related equipment thereof

Abstract

The invention belongs to the field of artificial intelligence, is applied to the field of intelligent medical treatment, and relates to a cross-language speech generation method based on artificial intelligence and related equipment thereof.The cross-language speech generation method comprises the steps that received text information and language information are input into a text coding model, and semantic features are obtained; inputting the received audio information into an emotion capture model and a tone extraction model to obtain an emotion feature and a tone feature, and generating an emotion feature according to the emotion feature and the tone feature; inputting the semantic features into a pronunciation duration prediction model to obtain the duration of each pronunciation unit, and marking the corresponding semantic features through the durations of the pronunciation units to obtain marked features; inputting the mark features into a stream model to obtain voice data features; and inputting the voice data features and the emotion features into an audio decoding model to obtain a voice signal. The invention also relates to a block chain technology, and the voice signal can be stored in a block chain. According to the method and the device, the problem that synthesized cross-language voice is unnatural in expression and has a mechanical feeling is solved.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to a cross-language speech generation method based on artificial intelligence, and related devices. Background Art

[0002] In the rapid development of artificial intelligence technology, speech synthesis, as one of its core areas, has shown great application potential. Speech synthesis aims to generate audio close to natural human speech through computers. This ability is of great significance to many fields such as human-computer interaction, virtual character creation, and multilingual translation assistance. However, although speech synthesis technology has made significant progress, the field of cross-language speech synthesis still faces many technical challenges and difficulties.

[0003] The problem that cross-language speech synthesis systems need to solve is how to effectively preserve the speaker's identity characteristics and emotional style of expression in different languages. Identity characteristics, including timbre, intonation patterns, and pronunciation habits, are key factors in distinguishing different speakers. In a single language environment, these characteristics are relatively easy to capture and reproduce, but in a cross-language scenario, there are significant differences in language structure, pronunciation rules, and rhythmic characteristics. Currently, the synthesized speech has the problem of being unnatural and mechanical. Summary of the invention

[0004] The purpose of the embodiments of the present application is to propose a cross-language speech generation method, apparatus, computer device and storage medium based on artificial intelligence to solve the problem that the synthesized cross-language speech has unnatural expression and mechanical feeling.

[0005] In order to solve the above technical problems, the present application embodiment provides a cross-language speech generation method based on artificial intelligence, which adopts the following technical solutions:

[0006] Receiving cross-language text information and language information, inputting the cross-language text information and language information into a preset text encoding model, and obtaining output semantic features;

[0007] Receive audio information of a target user, input the audio information into a preset emotion capture model and a timbre extraction model, respectively, obtain emotion features and timbre features, and generate emotional features according to the emotion features and the timbre features;

[0008] Inputting the semantic features into a pre-trained pronunciation duration prediction model to obtain the duration of each pronunciation unit output, and marking the semantic features corresponding to the duration of the pronunciation unit to obtain a marking feature;

[0009] Inputting the marking feature into a preset stream model to obtain output voice data features;

[0010] The speech data features and the emotion features are input into a preset audio decoding model to obtain an output speech signal.

[0011] Furthermore, the flow model includes a multi-layer neural network, and the step of inputting the label feature into the preset flow model to obtain the output voice data feature includes:

[0012] The marking feature is input into the stream model, and the emotion feature is respectively input into each layer of the neural network of the stream model to obtain the output speech data feature.

[0013] Furthermore, before the step of inputting the semantic features into the pre-trained pronunciation duration prediction model to obtain the marker features, the method further includes:

[0014] Receiving a text-speech pair, and determining the number of pronunciation frames of each pronunciation unit in the text-speech pair by a monotone alignment search algorithm;

[0015] Marking the text in the text-speech pair by the number of pronunciation frames to obtain training data;

[0016] The preset initial pronunciation duration prediction model is trained and adjusted based on the training data to obtain the pronunciation duration prediction model.

[0017] Furthermore, before the step of inputting the marking feature into a preset stream model to obtain the output voice data feature, the method further includes:

[0018] Receiving a speech linear spectrum, and generating a speech data coding feature based on the speech linear spectrum;

[0019] Inputting the speech data encoding features into a preset initial flow model to obtain output conversion features;

[0020] calculating a Kulbak-Leibler divergence between the transformation feature and the semantic feature;

[0021] The initial flow model is adjusted based on the Kulbak-Leibler divergence to obtain the flow model.

[0022] Furthermore, the step of generating speech data coding features based on the speech linear spectrum includes:

[0023] The speech linear spectrum is input into a preset a posteriori encoder to obtain the output speech data coding features.

[0024] Furthermore, the step of generating an emotional feature according to the emotional feature and the timbre feature includes:

[0025] The emotional feature is obtained by adding the data of the corresponding dimensions of the emotional feature and the timbre feature.

[0026] Furthermore, the step of generating an emotional feature according to the emotional feature and the timbre feature includes:

[0027] The emotion feature and the timbre feature are concatenated to obtain the emotion feature.

[0028] In order to solve the above technical problems, the embodiment of the present application also provides a cross-language speech generation device based on artificial intelligence, which adopts the following technical solution:

[0029] A cross-language speech generation device based on artificial intelligence, comprising:

[0030] A receiving module, used for receiving cross-language text information and language information, inputting the cross-language text information and language information into a preset text encoding model, and obtaining output semantic features;

[0031] A generating module, for receiving audio information of a target user, inputting the audio information into a preset emotion capture model and a timbre extraction model, respectively, obtaining emotion features and timbre features, respectively, and generating emotional features according to the emotion features and the timbre features;

[0032] A marking module, used for inputting the semantic features into a pre-trained pronunciation duration prediction model to obtain the duration of each pronunciation unit output, and marking the semantic features corresponding to the duration of the pronunciation unit to obtain a marking feature;

[0033] A first input module, used for inputting the marking feature into a preset stream model to obtain output voice data features;

[0034] The second input module is used to input the voice data features and the emotion features into a preset audio decoding model to obtain an output voice signal.

[0035] In order to solve the above technical problems, the embodiment of the present application further provides a computer device, which adopts the following technical solution:

[0036] A computer device includes a memory and a processor, wherein the memory stores computer-readable instructions, and the processor implements the steps of the above-mentioned cross-language speech generation method based on artificial intelligence when executing the computer-readable instructions.

[0037] In order to solve the above technical problems, the embodiment of the present application further provides a computer-readable storage medium, which adopts the following technical solution:

[0038] A computer-readable storage medium having computer-readable instructions stored thereon, wherein the computer-readable instructions, when executed by a processor, implement the steps of the above-mentioned cross-language speech generation method based on artificial intelligence.

[0039] Compared with the prior art, the embodiments of the present application have the following beneficial effects:

[0040] This application improves the accuracy of the output semantic features by inputting language information into the text encoding model. The emotion capture model and the timbre extraction model are used to extract the speaker's emotion features and timbre features, respectively, to generate emotion features. By predicting the duration of each pronunciation unit and marking the corresponding semantic features, it is ensured that the pronunciation length of each pronunciation unit in the generated speech is appropriate and the speech is not stiff. The marked features are converted into speech data features through the trained stream model to achieve the first step of text-to-speech conversion, and the speech data features and emotion features are input into the preset audio decoding model to obtain the output speech signal to achieve the second step of text-to-speech conversion. The emotion features are integrated with the speaker's emotion features and timbre features, which can ensure that even when processing cross-language texts, the generated cross-language speech data can be expressed naturally and close to the speaker's real voice. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] In order to more clearly illustrate the scheme in the present application, a brief introduction is given below to the drawings required for use in the description of the embodiments of the present application. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0042] Figure 1 is an exemplary system architecture diagram to which the present application may be applied;

[0043] Figure 2 A flowchart of an embodiment of a cross-language speech generation method based on artificial intelligence according to the present application;

[0044] Figure 3 is a schematic structural diagram of an embodiment of an artificial intelligence-based cross-language speech generation device according to the present application;

[0045] Figure 4 It is a structural diagram of an embodiment of a computer device according to the present application. DETAILED DESCRIPTION

[0046] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by technicians in the technical field of the present application; the terms used in the specification of the application herein are only for the purpose of describing specific embodiments and are not intended to limit the present application; the terms "including" and "having" and any variations thereof in the specification and claims of the present application and the above-mentioned drawings are intended to cover non-exclusive inclusions. The terms "first", "second", etc. in the specification and claims of the present application or the above-mentioned drawings are used to distinguish different objects, not to describe a specific order.

[0047] Reference to "embodiments" herein means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0048] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the accompanying drawings.

[0049] like Figure 1 As shown, the system architecture 100 may include terminal devices 101, 102, 103, a network 104 and a server 105. The network 104 is used to provide a medium for communication links between the terminal devices 101, 102, 103 and the server 105. The network 104 may include various connection types, such as wired, wireless communication links or optical fiber cables, etc.

[0050] Users can use terminal devices 101, 102, 103 to interact with server 105 through network 104 to receive or send messages, etc. Various communication client applications can be installed on terminal devices 101, 102, 103, such as web browser applications, shopping applications, search applications, instant messaging tools, email clients, social platform software, etc.

[0051] The terminal devices 101, 102, and 103 can be various electronic devices with display screens and supporting web browsing, including but not limited to smart phones, tablet computers, e-book readers, MP3 players (Moving Picture Experts Group Audio Layer III), MP4 (Moving Picture Experts Group Audio Layer IV) players, laptop computers, desktop computers, etc.

[0052] The server 105 may be a server that provides various services, such as a background server that provides support for web pages displayed on the terminal devices 101 , 102 , and 103 .

[0053] It should be noted that the cross-language speech generation method based on artificial intelligence provided in the embodiments of the present application is generally executed by a server / terminal device, and accordingly, the cross-language speech generation device based on artificial intelligence is generally set in the server / terminal device.

[0054] It should be understood that Figure 1 The number of terminal devices, networks and servers in the embodiment is only for illustration. Any number of terminal devices, networks and servers may be provided according to implementation requirements.

[0055] Continue to refer Figure 2 , showing a flow chart of an embodiment of a method for cross-language speech generation based on artificial intelligence according to the present application. The cross-language speech generation method based on artificial intelligence comprises the following steps:

[0056] Step S201, receiving cross-language text information and language information, inputting the cross-language text information and language information into a preset text encoding model, and obtaining output semantic features.

[0057] In this embodiment, the electronic device (e.g. Figure 1 The server / terminal device shown in the figure can receive text information and voice information through a wired connection or a wireless connection. It should be noted that the above wireless connection method may include but is not limited to 3G / 4G / 5G connection, Wi-Fi connection, Bluetooth connection, Wi MAX connection, Zigbee connection, UWB (Ultra Wi-Fi Band) connection, and other wireless connection methods currently known or to be developed in the future.

[0058] In this embodiment, the present application provides new possibilities for cross-language speech generation. The cross-language text information of the present application contains texts of multiple different language types, for example: I am from China. Language information is a label for the language type to which each character or each part of the character in the cross-language text information belongs. For example, in "I am from China", the language type of "I am from" is Chinese, and the language type of "China" is English. Each character in the cross-language text information has a one-to-one correspondence with each dimension in the semantic feature. By inputting both the cross-language text information and the language information into the text encoding model, the accuracy of the output semantic features can be improved.

[0059] For example, in the medical field, a cross-language text may be: I am working in a hospital. The language information indicates that the language types of "I am" and "working" in this sentence are Chinese, and the language type of "hospital" is English.

[0060] Step S202, receiving audio information of the target user, inputting the audio information into a preset emotion capture model and a timbre extraction model respectively, obtaining emotion features and timbre features respectively, and generating emotional features according to the emotion features and the timbre features.

[0061] In this embodiment, the emotion capture model extracts and analyzes the acoustic features of the audio information to identify the emotional state (such as happiness, sadness, anger, etc.) expressed by the speaker (i.e., the target user, for example, a doctor), and encodes the emotional information into a specific format to obtain the emotional features of the doctor. The emotion capture model continuously learns the corresponding emotional features from the preset reference audio, without additional

[0062] The learned emotion capture model can capture the emotional features of audio information. The timbre extraction model focuses on the global features of audio information and extracts the timbre features in audio information.

[0063] By extracting the emotional features and timbre features of the target user's audio information, the doctor's identity features and emotional style are obtained, which are then used to generate emotional voice signals, making the voice expression richer and realizing the personalized customization of the doctor's voice. The output voice is no longer mechanical, but fits the doctor's real speaking voice. This technology is installed in intelligent medical devices for human-computer interaction, which can realize emotional natural language interaction between intelligent medical devices and patients, and improve patients' usage experience and satisfaction.

[0064] Step S203, inputting the semantic features into a pre-trained pronunciation duration prediction model to obtain the duration of each pronunciation unit output, and obtaining a marking feature by marking the semantic features corresponding to the duration of the pronunciation unit.

[0065] In this embodiment, each word in the text information contains a pronunciation unit. This embodiment predicts the duration of each pronunciation unit. The semantic feature is a high-dimensional feature of the text information.

[0066] Therefore, there is a corresponding relationship between the pronunciation duration, semantic features, and text information. Through accurate pronunciation duration prediction, the synthesized speech can be closer to natural speech in terms of rhythm and intonation, thereby improving the naturalness and expressiveness of speech synthesis. This helps to improve user experience and enhance the fluency of voice interaction between doctors and patients.

[0067] Step S204: input the marked features into a preset stream model to obtain output voice data features.

[0068] · In this embodiment, the flow model is a bidirectional flow model. During training, the speech data encoding features are forward received, and the Kulbak-Leibler divergence between the speech data encoding features and the output semantic features of the text encoder is calculated. During reasoning, the marked features are accepted to generate speech feature data. The marked features are input into the flow model, and the transformation of text information into acoustic features is completed through the flow model. The powerful learning and generalization capabilities of the flow model are used to perform a deeper analysis and processing of the input marked features. This method helps to improve the performance of the entire speech processing method, including accuracy, response speed, and robustness.

[0069] Step S205: input the speech data features and the emotion features into a preset audio decoding model to obtain an output speech signal.

[0070] In this embodiment, the audio decoding model upgrades acoustic features such as voice data features and emotional features to voice data. By inputting the doctor's emotional features into the audio decoding model, the generated voice signal can accurately replicate the speaker's identity features, achieving highly personalized voice generation. This personalized voice not only contains the basic voice features of the target user (ie, a doctor), but also incorporates the doctor's specific emotional expressions, so that the generated voice is more in line with the user's personalized needs. This application can adapt to medical terminology and pronunciation habits in different languages, providing convenience for international medical exchanges.

[0071] This application improves the accuracy of the output semantic features by inputting language information into the text encoding model. The emotion capture model and the timbre extraction model are used to extract the speaker's emotion features and timbre features, respectively, to generate emotion features. By predicting the duration of each pronunciation unit and marking the corresponding semantic features, it is ensured that the pronunciation length of each pronunciation unit in the generated speech is appropriate and the speech is not stiff. The marked features are converted into speech data features through the trained stream model to achieve the first step of text-to-speech conversion, and the speech data features and emotion features are input into the preset audio decoding model to obtain the output speech signal to achieve the second step of text-to-speech conversion. The emotion features are integrated with the speaker's emotion features and timbre features, which can ensure that even when processing cross-language texts, the generated cross-language speech data can be expressed naturally and close to the speaker's real voice.

[0072] In some optional embodiments, step S202, i.e., the step of generating the emotional feature according to the emotional feature and the timbre feature, includes:

[0073] The emotional feature is obtained by adding the data of the corresponding dimensions of the emotional feature and the timbre feature.

[0074] In this embodiment, the emotional feature reflects the emotional state of the speech, such as happiness, sadness, anger, etc., while the timbre feature describes the sound quality characteristics of the speech, such as the brightness and roughness of the sound. The emotional feature obtained by adding these two features can more comprehensively describe the emotional expression in the speech, including both the information of the emotional state and the information of the sound quality characteristics. The addition method may lose some information, but it can simplify the feature dimension and improve the processing efficiency of the model.

[0075] In some optional embodiments, step S202, i.e., the step of generating the emotional feature according to the emotional feature and the timbre feature, includes:

[0076] The emotion feature and the timbre feature are concatenated to obtain the emotion feature.

[0077] In this embodiment, the emotional feature and the timbre feature are combined by splicing to obtain the emotional feature. Although the splicing method increases the feature dimension and calculation complexity, it can retain all the original information and the obtained emotional feature is more accurate.

[0078] In some optional implementations of this embodiment, in step S202, that is, before the step of inputting the semantic features into the pre-trained pronunciation duration prediction model to obtain the marker features, the electronic device may further perform the following steps:

[0079] Receiving a text-speech pair, and determining the number of pronunciation frames of each pronunciation unit in the text-speech pair by a monotone alignment search algorithm;

[0080] Marking the text in the text-speech pair by the number of pronunciation frames to obtain training data;

[0081] The preset initial pronunciation duration prediction model is trained and adjusted based on the training data to obtain the pronunciation duration prediction model.

[0082] In this embodiment, the monotonic alignment search (MAS) algorithm is an algorithm for finding the most likely monotonic alignment relationship between text and speech. The present application searches for the number of pronunciation frames of each pronunciation unit according to the monotonic alignment criterion, and this number of frames represents the pronunciation length of each unit as a label of the training data. This improves the quality and precision of the training data.

[0083] In some optional implementations of this embodiment, the stream model includes a multi-layer neural network, and step S204, that is, the step of inputting the marked features into the preset stream model to obtain the output voice data features, includes:

[0084] The marking feature is input into the stream model, and the emotion feature is respectively input into each layer of the neural network of the stream model to obtain the output speech data feature.

[0085] In this embodiment, after the marker feature is input into the flow model, it passes through each layer of the neural network of the flow model in turn. And the emotional features are respectively input into each layer of the neural network of the flow model, which strengthens the deep fusion of the doctor's emotional features and the original voice data features at all levels of the model. For example: after the marker feature and the emotional feature are input into the first layer of the neural network of the flow model, the neural network performs a fusion operation and a sampling operation on the two features to obtain the first layer of features. The first layer of features automatically enters the second layer of the neural network, and the emotional features are also input into the second layer of the neural network, which performs a fusion operation and a sampling operation on the first layer of features and the emotional features. Each subsequent layer of the neural network is analogous to this. Compared with the ordinary technology of inputting emotional features only in the first layer, each layer of the neural network in this embodiment is input with emotional features to achieve deep fusion of emotional features. This deep fusion helps the model to more comprehensively understand the emotional information of the doctor in the voice data, thereby generating more expressive and emotional voice data features.

[0086] In some optional implementations of this embodiment, before step S204, that is, the step of inputting the marking feature into a preset stream model to obtain the output voice data feature, the electronic device may further perform the following steps:

[0087] Receiving a speech linear spectrum, and generating a speech data coding feature based on the speech linear spectrum;

[0088] Inputting the speech data encoding features into a preset initial flow model to obtain output conversion features;

[0089] calculating a Kulbak-Leibler divergence between the transformation feature and the semantic feature;

[0090] The initial flow model is adjusted based on the Kulbak-Leibler divergence to obtain the flow model.

[0091] In this embodiment, the speech linear spectrum, i.e., the amplitude spectrum of speech, is the representation of speech in the frequency domain, reflecting the energy distribution of the speech signal on different frequency components. In simple terms, the speech linear spectrum is a representation method that reflects the frequency characteristics of speech. By generating the speech data encoding features through the speech linear spectrum, the essential characteristics of speech can be captured more accurately. By calculating the KL distance (Kullback-Leibler Divergence) between the conversion feature and the semantic feature, the performance of the current stream model in feature conversion can be quantitatively evaluated. The calculation formula is: KL(P||Q)=Σ(P(x)×log(P(x) / Q(x))). Wherein, P represents the conversion feature, Q represents the semantic feature, and the calculation result of the KL distance is a non-negative value. The KL distance is equal to 0 when and only when P and Q are exactly the same. The greater the difference between P and Q, the greater the value of the KL distance. Therefore, the KL distance can be used to quantify the degree of difference between P and Q. Based on the feedback of the Kulbak-Leibler divergence, the parameters and structure of the initial flow model can be dynamically adjusted to improve the generalization ability of the flow model so that it can still maintain a high processing level when dealing with complex speech data.

[0092] Specifically, the step of generating speech data coding features based on the speech linear spectrum includes:

[0093] The speech linear spectrum is input into a preset a posteriori encoder to obtain the output speech data coding features.

[0094] In this embodiment, the main function of the posterior encoder is to generate a corresponding variable representation: speech data coding features under the condition of a given speech linear spectrum. The speech data coding features generated in this way contain important information in the speech linear spectrum and are easier to model.

[0095] In some optional embodiments of the present application, the audio decoding model includes multiple decoding layers, and step S205, that is, the step of inputting the voice data feature and the emotion feature into the preset audio decoding model to obtain the output voice signal includes:

[0096] The speech data features are input into the audio decoding model, and the emotion features are respectively input into each of the decoding layers to obtain the output speech signal.

[0097] In this embodiment, the emotional features act on each decoding layer of the audio decoding model respectively, which can significantly improve the quality and emotional expression of the synthesized speech and enhance the personalized characteristics of the speech.

[0098] This application improves the accuracy of the output semantic features by inputting language information into the text encoding model. The emotion capture model and the timbre extraction model are used to extract the speaker's emotion features and timbre features respectively to generate emotion features. By predicting the duration of each pronunciation unit and marking the corresponding semantic features, it is ensured that the pronunciation length of each pronunciation unit in the generated speech is appropriate and the speech is not stiff. The marked features are converted into speech data features through the trained stream model to achieve the first step of text-to-speech conversion, and the speech data features and emotion features are input into the preset audio decoding model to obtain the output speech signal to achieve the second step of text-to-speech conversion. The emotion features are integrated with the speaker's emotion features and timbre features, which can ensure that even when processing cross-language texts, the generated cross-language speech signals can express naturally and be close to the speaker's real speech.

[0099] It should be emphasized that in order to further ensure the privacy and security of the above-mentioned voice signal, the above-mentioned voice signal can also be stored in a node of a blockchain.

[0100] The blockchain referred to in this application is a new application model of computer technologies such as distributed data storage, peer-to-peer transmission, consensus mechanism, encryption algorithm, etc. Blockchain is essentially a decentralized database, a string of data blocks generated by cryptographic methods. Each data block contains a batch of network transaction information, which is used to verify the validity of its information (anti-counterfeiting) and generate the next block. Blockchain can include the underlying blockchain platform, platform product service layer, and application service layer.

[0101] The embodiments of the present application can acquire and process relevant data based on artificial intelligence technology. Artificial Intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results.

[0102] The basic technologies of artificial intelligence generally include sensors, dedicated artificial intelligence chips, cloud computing, distributed storage, big data processing technology, operation / interaction systems, mechatronics and other technologies. Artificial intelligence software technology mainly includes computer vision technology, robotics technology, biometrics technology, speech processing technology, natural language processing technology, and machine learning / deep learning. This application can be applied to the field of smart medical care, thereby promoting the construction of smart cities.

[0103] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through computer-readable instructions, and the computer-readable instructions can be stored in a computer-readable storage medium. When the program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, the aforementioned storage medium can be a non-volatile storage medium such as a disk, an optical disk, a read-only memory (ROM), or a random access memory (RAM).

[0104] It should be understood that, although the steps in the flowchart of the accompanying drawings are displayed in sequence as indicated by the arrows, these steps are not necessarily executed in sequence in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least a part of the steps in the flowchart of the accompanying drawings may include multiple sub-steps or multiple stages, and these sub-steps or stages are not necessarily executed at the same time, but can be executed at different times, and their execution order is not necessarily sequential, but can be executed in turn or alternately with other steps or at least a part of the sub-steps or stages of other steps.

[0105] Further references Figure 3 , as a response to the above Figure 2 The present application provides an embodiment of a cross-language speech generation device based on artificial intelligence, and the device embodiment is similar to Figure 2 Corresponding to the method embodiment shown, the device can be specifically applied to various electronic devices.

[0106] like Figure 3As shown, the cross-language speech generation device 300 based on artificial intelligence described in this embodiment includes: a receiving module 301, a generating module 302, a marking module 303, a first input module 304 and a second input module 305. Among them: a receiving module 301 is used to receive cross-language text information and language information, input the cross-language text information and language information into a preset text encoding model, and obtain output semantic features; a generating module 302 is used to receive audio information of a target user, input the audio information into a preset emotion capture model and a timbre extraction model respectively, obtain emotion features and timbre features respectively, and generate emotional features based on the emotion features and the timbre features; a marking module 303 is used to input the semantic features into a pre-trained pronunciation duration prediction model, obtain the duration of each output pronunciation unit, and obtain the marking features by marking the semantic features corresponding to the duration of the pronunciation unit; a first input module 304 is used to input the marking features into a preset stream model, and obtain output voice data features; a second input module 305 is used to input the voice data features and the emotional features into a preset audio decoding model, and obtain output voice signals.

[0107] In this embodiment, the present application improves the accuracy of the output semantic features by inputting language information into the text encoding model. The emotion capture model and the timbre extraction model are used to extract the emotion features and timbre features of the speaker respectively to generate emotion features. By predicting the duration of each pronunciation unit and marking the corresponding semantic features, it is ensured that the pronunciation length of each pronunciation unit in the generated speech is appropriate and the speech is not stiff. The marked features are converted into speech data features through the trained stream model to achieve the first step of text-to-speech conversion, and the speech data features and emotion features are input into the preset audio decoding model to obtain the output speech signal to achieve the second step of text-to-speech conversion. The emotion features are integrated with the emotion features and timbre features of the speaker, which can ensure that even when processing cross-language texts, the generated cross-language speech data can be expressed naturally and close to the speaker's real speech.

[0108] In some optional implementations of this embodiment, the generating module 302 is further configured to add data of corresponding dimensions of the emotion feature and the timbre feature to obtain the emotion feature.

[0109] In some optional implementations of this embodiment, the generating module 302 is further configured to concatenate the emotion feature and the timbre feature to obtain the emotion feature.

[0110] In some optional implementations of the present embodiment, the above-mentioned device 300 also includes: a first training module, used to receive a text-speech pair, and determine the number of pronunciation frames of each pronunciation unit in the text-speech pair through a monotonic alignment search algorithm; mark the text in the text-speech pair by the pronunciation frame number to obtain training data; and train and adjust a preset initial pronunciation duration prediction model based on the training data to obtain the pronunciation duration prediction model.

[0111] In some optional implementations of this embodiment, the flow model includes a multi-layer neural network, and the first input module 304 is also used to input the marking features into the flow model, and input the emotional features into each layer of the neural network of the flow model respectively, to obtain the output speech data features.

[0112] In some optional implementations of the present embodiment, the above-mentioned device 300 also includes: a second training module, the second training module includes a generation submodule, an input submodule, a calculation submodule and an adjustment submodule, wherein the generation submodule is used to receive a speech linear spectrum and generate speech data coding features based on the speech linear spectrum; the input submodule is used to input the speech data coding features into a preset initial flow model to obtain output conversion features; the calculation submodule is used to calculate the Kulbak-Leibler divergence between the conversion features and the semantic features; the adjustment submodule is used to adjust the initial flow model based on the Kulbak-Leibler divergence to obtain the flow model.

[0113] In some optional implementations of this embodiment, the generation submodule is further used to input the speech linear spectrum into a preset a posteriori encoder to obtain the output speech data encoding features.

[0114] In some optional embodiments of the present application, the audio decoding model includes multiple decoding layers, and the second input module 305 is also used to input the voice data features into the audio decoding model, and input the emotional features into each of the decoding layers respectively to obtain the output voice signal.

[0115] This application improves the accuracy of the output semantic features by inputting language information into the text encoding model. The emotion capture model and the timbre extraction model are used to extract the speaker's emotion features and timbre features, respectively, to generate emotion features. By predicting the duration of each pronunciation unit and marking the corresponding semantic features, it is ensured that the pronunciation length of each pronunciation unit in the generated speech is appropriate and the speech is not stiff. The marked features are converted into speech data features through the trained stream model to achieve the first step of text-to-speech conversion, and the speech data features and emotion features are input into the preset audio decoding model to obtain the output speech signal to achieve the second step of text-to-speech conversion. The emotion features are integrated with the speaker's emotion features and timbre features, which can ensure that even when processing cross-language texts, the generated cross-language speech data can be expressed naturally and close to the speaker's real voice.

[0116] To solve the above technical problems, the present application also provides a computer device. Figure 4 , Figure 4 This is a basic structural block diagram of the computer device in this embodiment.

[0117] The computer device 4 includes a memory 41, a processor 42, and a network interface 43 that are interconnected through a system bus. It should be noted that the figure only shows a computer device 4 with components 41-43, but it should be understood that it is not required to implement all the components shown, and more or fewer components can be implemented instead. Among them, those skilled in the art can understand that the computer device here is a device that can automatically perform numerical calculations and / or information processing according to pre-set or stored instructions, and its hardware includes but is not limited to microprocessors, application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), digital processors (DSPs), embedded devices, etc.

[0118] The computer device may be a computing device such as a desktop computer, a notebook, a PDA, a cloud server, etc. The computer device may interact with a user through a keyboard, a mouse, a remote controller, a touch pad, or a voice control device.

[0119] The memory 41 includes at least one type of readable storage medium, and the readable storage medium includes flash memory, hard disk, multimedia card, card-type memory (for example, SD or DX memory, etc.), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, disk, optical disk, etc. In some embodiments, the memory 41 can be an internal storage unit of the computer device 4, such as a hard disk or memory of the computer device 4. In other embodiments, the memory 41 can also be an external storage device of the computer device 4, such as a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (FlashCard), etc. equipped on the computer device 4. Of course, the memory 41 can also include both the internal storage unit of the computer device 4 and its external storage device. In this embodiment, the memory 41 is generally used to store the operating system and various application software installed on the computer device 4, such as computer-readable instructions of a cross-language speech generation method based on artificial intelligence, etc. In addition, the memory 41 can also be used to temporarily store various types of data that have been output or are to be output.

[0120] The processor 42 may be a central processing unit (CPU), a controller, a microcontroller, a microprocessor, or other data processing chips in some embodiments. The processor 42 is generally used to control the overall operation of the computer device 4. In this embodiment, the processor 42 is used to run the computer-readable instructions stored in the memory 41 or process data, such as computer-readable instructions for running the cross-language speech generation method based on artificial intelligence.

[0121] The network interface 43 may include a wireless network interface or a wired network interface. The network interface 43 is generally used to establish a communication connection between the computer device 4 and other electronic devices.

[0122] This application improves the accuracy of the output semantic features by inputting language information into the text encoding model. The emotion capture model and the timbre extraction model are used to extract the speaker's emotion features and timbre features, respectively, to generate emotion features. By predicting the duration of each pronunciation unit and marking the corresponding semantic features, it is ensured that the pronunciation length of each pronunciation unit in the generated speech is appropriate and the speech is not stiff. The marked features are converted into speech data features through the trained stream model to achieve the first step of text-to-speech conversion, and the speech data features and emotion features are input into the preset audio decoding model to obtain the output speech signal to achieve the second step of text-to-speech conversion. The emotion features are integrated with the speaker's emotion features and timbre features, which can ensure that even when processing cross-language texts, the generated cross-language speech data can be expressed naturally and close to the speaker's real voice.

[0123] The present application also provides another implementation, namely, providing a computer-readable storage medium, wherein the computer-readable storage medium stores computer-readable instructions, and the computer-readable instructions can be executed by at least one processor to enable the at least one processor to perform the steps of the above-mentioned artificial intelligence-based cross-language speech generation method.

[0124] This application improves the accuracy of the output semantic features by inputting language information into the text encoding model. The emotion capture model and the timbre extraction model are used to extract the speaker's emotion features and timbre features, respectively, to generate emotion features. By predicting the duration of each pronunciation unit and marking the corresponding semantic features, it is ensured that the pronunciation length of each pronunciation unit in the generated speech is appropriate and the speech is not stiff. The marked features are converted into speech data features through the trained stream model to achieve the first step of text-to-speech conversion, and the speech data features and emotion features are input into the preset audio decoding model to obtain the output speech signal to achieve the second step of text-to-speech conversion. The emotion features are integrated with the speaker's emotion features and timbre features, which can ensure that even when processing cross-language texts, the generated cross-language speech data can be expressed naturally and close to the speaker's real voice.

[0125] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes a number of instructions for a terminal device (which can be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in each embodiment of the present application.

[0126] Obviously, the embodiments described above are only some embodiments of the present application, rather than all embodiments. The preferred embodiments of the present application are given in the accompanying drawings, but they do not limit the patent scope of the present application. The present application can be implemented in many different forms. On the contrary, the purpose of providing these embodiments is to make the understanding of the disclosure of the present application more thorough and comprehensive. Although the present application is described in detail with reference to the aforementioned embodiments, for those skilled in the art, it is still possible to modify the technical solutions recorded in the aforementioned specific implementation methods, or to perform equivalent replacement of some of the technical features therein. Any equivalent structure made using the contents of the specification and drawings of this application, directly or indirectly used in other related technical fields, is similarly within the scope of patent protection of this application.

Claims

1. A cross-language speech generation method based on artificial intelligence, characterized in that: The steps include: Receiving cross-language text information and language information, inputting the cross-language text information and language information into a preset text encoding model, and obtaining output semantic features; Receive audio information of a target user, input the audio information into a preset emotion capture model and a timbre extraction model, respectively, obtain emotion features and timbre features, and generate emotional features according to the emotion features and the timbre features; Inputting the semantic features into a pre-trained pronunciation duration prediction model to obtain the duration of each pronunciation unit output, and marking the semantic features corresponding to the duration of the pronunciation unit to obtain a marking feature; Inputting the marking feature into a preset stream model to obtain output voice data features; The speech data features and the emotion features are input into a preset audio decoding model to obtain an output speech signal.

2. The cross-language speech generation method based on artificial intelligence according to claim 1, characterized in that: The stream model includes a multi-layer neural network, and the step of inputting the label feature into the preset stream model to obtain the output voice data feature includes: The marking feature is input into the stream model, and the emotion feature is respectively input into each layer of the neural network of the stream model to obtain the output speech data feature.

3. The cross-language speech generation method based on artificial intelligence according to claim 1, characterized in that: Before the step of inputting the semantic features into the pre-trained pronunciation duration prediction model to obtain the marker features, the method further includes: Receiving a text-speech pair, and determining the number of pronunciation frames of each pronunciation unit in the text-speech pair by a monotone alignment search algorithm; Marking the text in the text-speech pair by the number of pronunciation frames to obtain training data; The preset initial pronunciation duration prediction model is trained and adjusted based on the training data to obtain the pronunciation duration prediction model.

4. The cross-language speech generation method based on artificial intelligence according to claim 1, characterized in that: Before the step of inputting the marking feature into a preset stream model to obtain the output voice data feature, the method further includes: Receiving a speech linear spectrum, and generating a speech data coding feature based on the speech linear spectrum; Inputting the speech data encoding features into a preset initial flow model to obtain output conversion features; calculating a Kulbak-Leibler divergence between the transformation feature and the semantic feature; The initial flow model is adjusted based on the Kulbak-Leibler divergence to obtain the flow model.

5. The cross-language speech generation method based on artificial intelligence according to claim 4 is characterized in that: The step of generating speech data coding features based on the speech linear spectrum comprises: The speech linear spectrum is input into a preset a posteriori encoder to obtain the output speech data coding features.

6. The cross-language speech generation method based on artificial intelligence according to claim 1, characterized in that: The step of generating an emotional feature according to the emotional feature and the timbre feature comprises: The emotional feature is obtained by adding the data of the corresponding dimensions of the emotional feature and the timbre feature.

7. The cross-language speech generation method based on artificial intelligence according to claim 1, characterized in that: The step of generating an emotional feature according to the emotional feature and the timbre feature comprises: The emotion feature and the timbre feature are concatenated to obtain the emotion feature.

8. A cross-language speech generation device based on artificial intelligence, characterized in that: include: A receiving module, used for receiving cross-language text information and language information, inputting the cross-language text information and language information into a preset text encoding model, and obtaining output semantic features; A generating module, for receiving audio information of a target user, inputting the audio information into a preset emotion capture model and a timbre extraction model, respectively, obtaining emotion features and timbre features, respectively, and generating emotional features according to the emotion features and the timbre features; A marking module, used for inputting the semantic features into a pre-trained pronunciation duration prediction model to obtain the duration of each pronunciation unit output, and marking the semantic features corresponding to the duration of the pronunciation unit to obtain a marking feature; A first input module, used for inputting the marking feature into a preset stream model to obtain output voice data features; The second input module is used to input the voice data features and the emotion features into a preset audio decoding model to obtain an output voice signal.

9. A computer device, characterized in that: The method comprises a memory and a processor, wherein the memory stores computer-readable instructions, and when the processor executes the computer-readable instructions, the steps of the cross-language speech generation method based on artificial intelligence as described in any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-readable instructions, which, when executed by a processor, implement the steps of the cross-language speech generation method based on artificial intelligence as described in any one of claims 1 to 7.

Images