Voice generation method and device, equipment and storage medium thereof

Abstract

The invention belongs to the technical field of voice generation, and relates to a voice generation method and device, equipment and a storage medium thereof. Obtaining a visual feature vector sequence of the target video frame sequence; inputting the text feature vector sequence and the visual feature vector sequence into a cross-modal fusion layer to obtain a feature fusion representation output by the cross-modal fusion layer; obtaining acoustic features output by the variant acoustic model according to the feature fusion representation; and inputting the acoustic features into a preset vocoder, and generating a target voice waveform through multi-scale convolution and up-sampling processing. According to the method, not only text features but also video features are introduced during voice generation, and voice generation is realized through cross-modal fusion features. The method is applied to financial or medical service intelligent customer service answering or service product marketing introduction scenes, and more natural and real voice is generated in combination with visual context information.

Description

Technical Field

[0001] The present application relates to the technical field of speech generation, and is applied to speech generation scenarios in financial or medical services. It involves a speech generation method, device, equipment, and its storage medium. Background Art

[0002] Currently, text-to-speech (TTS) technology mainly relies on text information to generate speech. Most of these methods only use text information to predict speech prosody (including pitch, rhythm, energy, etc.), while ignoring the important emotions and context clues conveyed by visual information in actual communication.

[0003] Existing text-to-speech service products, although performing well in ordinary text speech synthesis, still have problems when dealing with speech scenarios involving visual information such as videos, animation commentaries, or intelligent conversations. They only use simple feature splicing or attention mechanisms, rely too much on text content, and fail to fully utilize the influence of visual context on speech prosody, resulting in the generated speech lacking appeal and realism. Summary of the Invention

[0004] The purpose of the embodiments of the present application is to propose a speech generation method, device, equipment, and its storage medium to solve the problem that in the prior art, when generating speech, it relies too much on text content, fails to fully utilize the influence of visual context on speech prosody, and the generated speech lacks appeal and realism.

[0005] In a first aspect, the embodiments of the present application provide a speech generation method, which adopts the following technical solutions: A speech generation method includes the following steps: Obtain a target text sequence; Input the target text sequence into a preset text encoder to obtain a text feature vector sequence output by the text encoder; Obtain a target video frame sequence; Input the target video frame sequence into a preset visual encoder to obtain a visual feature vector sequence output by the visual encoder; Input the text feature vector sequence and the visual feature vector sequence into a cross-modal fusion layer to obtain a feature fusion representation output by the cross-modal fusion layer; Input the feature fusion representation into a preset non-autoregressive variant acoustic model to obtain acoustic features output by the variant acoustic model; Input the acoustic features into a preset vocoder, and generate a target speech waveform through multi-scale convolution and upsampling processing, where the vocoder includes a lightweight HiFi-GAN or WaveGlow vocoder.

[0006] In a second aspect, an embodiment of the present application further provides a voice generation device, which adopts the following technical solutions: A voice generation device includes: A target text sequence acquisition module, configured to acquire a target text sequence; A text feature vector sequence encoding module, configured to input the target text sequence into a preset text encoder to obtain a text feature vector sequence output by the text encoder; A target video frame sequence acquisition module, configured to acquire a target video frame sequence; A visual feature vector sequence encoding module, configured to input the target video frame sequence into a preset visual encoder to obtain a visual feature vector sequence output by the visual encoder; A cross-modal fusion module, configured to input the text feature vector sequence and the visual feature vector sequence into a cross-modal fusion layer to obtain a feature fusion representation output by the cross-modal fusion layer; An acoustic feature acquisition module, configured to input the feature fusion representation into a preset non-autoregressive variant acoustic model to obtain an acoustic feature output by the variant acoustic model; A target voice waveform generation module, configured to input the acoustic feature into a preset vocoder, and generate a target voice waveform through multi-scale convolution and upsampling processing, where the vocoder includes a lightweight HiFi-GAN or WaveGlow vocoder.

[0007] In a third aspect, an embodiment of the present application further provides a computer device, which adopts the following technical solutions: A computer device includes a memory and a processor. A computer-readable instruction is stored in the memory, and when the processor executes the computer-readable instruction, the steps of the above-mentioned voice generation method are implemented.

[0008] In a fourth aspect, an embodiment of the present application further provides a computer-readable storage medium, which adopts the following technical solutions: A computer-readable storage medium has a computer-readable instruction stored thereon. When the computer-readable instruction is executed by a processor, the steps of the above-mentioned voice generation method are implemented.

[0009] Compared with the prior art, the embodiments of the present application mainly have the following beneficial effects: The speech generation method described in the embodiments of the present application includes: obtaining a sequence of text feature vectors of a target text sequence; obtaining a sequence of visual feature vectors of a target video frame sequence; inputting the sequence of text feature vectors and the sequence of visual feature vectors into a cross-modal fusion layer to obtain a feature fusion representation output by the cross-modal fusion layer; inputting the feature fusion representation into a preset non-autoregressive variant acoustic model to obtain acoustic features output by the variant acoustic model; and inputting the acoustic features into a preset vocoder to generate a target speech waveform through multi-scale convolution and upsampling processing. This realizes introducing not only text features but also video features during speech generation, and jointly realizing speech generation with cross-modal fusion features, avoiding the generated speech from being too mechanical and improving the realism of the generated speech. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] To more clearly illustrate the solutions in the present application, the following briefly introduces the drawings required for the description of the embodiments of the present application. Obviously, the drawings in the following description are some embodiments of the present application. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative efforts.

[0011] Figure 1 is an exemplary system architecture diagram to which the present application can be applied; Figure 2 is a flowchart of an embodiment of a speech generation method according to the present application; Figure 3 is Figure 2 a flowchart of a specific embodiment of step 202 shown; Figure 4 is Figure 2 a flowchart of a specific embodiment of step 204 shown; Figure 5 is Figure 2 a flowchart of a specific embodiment of step 205 shown; Figure 6 is Figure 2 a flowchart of a specific embodiment of step 206 shown; Figure 7 is a flowchart of a specific embodiment of training the variant acoustic model in the speech generation method described in this embodiment; Figure 8 is Figure 7 a flowchart of a specific embodiment of determining whether the variant acoustic model is pre-trained completed in step 704 shown; Figure 9 is a schematic structural diagram of an embodiment of a speech generation device according to the present application; Figure 10It is a schematic structural diagram of an embodiment of a computer device according to the present application. Detailed implementation manners

[0012] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by those of ordinary skill in the technical field to which this application belongs; the terms used in the specification of this application are only for the purpose of describing specific embodiments and are not intended to limit this application; the terms "including" and "having" and any variations thereof in the specification and claims of this application and the above drawings are intended to cover non-exclusive inclusion. The terms "first", "second", etc. in the specification and claims of this application or the above drawings are used to distinguish different objects and not to describe a specific order.

[0013] Reference to "embodiment" herein means that a particular feature, structure, or characteristic described in connection with the embodiment can be included in at least one embodiment of this application. The phrase appears in various places in the specification and does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment mutually exclusive with other embodiments. Those skilled in the art will explicitly and implicitly understand that the embodiments described herein can be combined with other embodiments.

[0014] In order to enable those skilled in the art of this technology to better understand the solution of this application, the technical solutions in the embodiments of this application will be clearly and completely described below with reference to the drawings.

[0015] As Figure 1 shown, the system architecture 100 may include a terminal device 101, a network 102, and a server 103. The terminal device 101 may be a laptop computer 1011, a tablet computer 1012, or a mobile phone 1013. The network 102 is a medium for providing a communication link between the terminal device 101 and the server 103. The network 102 may include various connection types, such as wired, wireless communication links, or fiber optic cables, etc.

[0016] The user can use the terminal device 101 to interact with the server 103 through the network 102 to receive or send messages, etc. Various communication client applications may be installed on the terminal device 101, such as a web browser application, a shopping application, a search application, an instant messaging tool, an email client, a social platform software, etc.

[0017] The terminal device 101 can be various electronic devices with a display screen and supporting web browsing. In addition to the laptop 1011, tablet computer 1012 or mobile phone 1013, the terminal device 101 can also be an e-book reader, an MP3 player (Moving Picture Experts Group Audio Layer III), an MP4 (Moving Picture Experts Group Audio Layer IV) player, a laptop computer, a desktop computer, etc.

[0018] The server 103 can be a server that provides various services, such as a background server that supports the pages displayed on the terminal device 101.

[0019] It should be noted that a voice generation method provided by an embodiment of the present application is generally executed by a server. Correspondingly, a voice generation device is generally arranged in the server.

[0020] It should be understood that Figure 1 the numbers of the terminal devices, networks, and servers in

[0021] Continuing to refer to Figure 2 , a flowchart of an embodiment of a voice generation method according to the present application is shown. The voice generation method includes the following steps: Step 201, obtain a target text sequence.

[0022] In this embodiment, the target text sequence includes a speech text sequence to be subjected to voice generation. For example, in the application scenario of intelligent customer service in the financial or medical business, the speech text sequence to be subjected to voice generation is specifically customer service script text data, etc.; for another example, in the marketing introduction scenario of financial or medical service products, the speech text sequence to be subjected to voice generation is specifically marketing text data for product or service introduction.

[0023] By obtaining the target text sequence, it is convenient to subsequently use the target text sequence for specific voice generation.

[0024] Step 202, input the target text sequence into a preset text encoder to obtain a text feature vector sequence output by the text encoder.

[0025] In this embodiment, the preset text encoder is a pre-trained text encoder, including a lightweight Transformer or CNN encoder, which can at least meet the requirements of encoding text feature vectors for the target text sequence and serializing the encoding results of the text feature vectors.

[0026] By inputting the target text sequence into the preset text encoder, a corresponding text feature vector sequence is obtained, which is convenient for subsequent voice generation using the text feature vector sequence.

[0027] Step 203: Obtain the target video frame sequence.

[0028] In this embodiment, the target video frame sequence includes the facial dynamic change video frame sequence of the target person object when voice generation is to be performed. For example, when it is required to generate and output the target text sequence A in the voice of the target person object B, at this time, by real-time obtaining the facial dynamic change video frame sequence of the target person object B, more video state representations can be introduced in voice generation according to the facial dynamic change situation of the target person object B in the subsequent process, avoiding the final generated voice from being too mechanical and improving the prosodic alternation and authenticity of the generated voice.

[0029] Step 204: Input the target video frame sequence into the preset visual encoder to obtain the visual feature vector sequence output by the visual encoder.

[0030] In this embodiment, the preset visual encoder includes a facial video acquisition encoder composed of a pre-trained ResNet series model or Vision Transformer model. By collecting the facial dynamic changes of the target person object when speaking and using the facial video acquisition encoder to extract video feature vectors from the collected facial dynamic change video frame sequence, it is convenient to fully combine the facial dynamic changes of the target person object when speaking in subsequent actual voice generation, so as to realize the prediction of voice prosody changes using video features.

[0031] It should be understood that when a human object is in different emotional states such as happy, sad, or glad, there will be certain changes in the oral description of the same problem or the same text. Specifically, there will be certain changes in the prosody and tone of the voice. In this application, by collecting the facial dynamic changes of the target human object, for example, the facial dynamic changes of the target human object B when in a glad state, voice generation is performed based on the facial dynamic change characteristics. Obviously, the generated voice is the voice emitted by the target human object B when in a glad state. Correspondingly, if the collected facial dynamic changes are the facial dynamic changes of the target human object B when speaking in an emotionally out-of-control state, voice generation is performed based on the facial dynamic change characteristics, and the generated voice is the voice emitted by the target human object B when emotionally out of control. Although the target text sequence is the same, if the auxiliary video state features participating in voice generation, that is, the facial dynamic change characteristics are different, there will be significant differences in the voice generation results. Therefore, by introducing video state features into voice generation, it is ensured that the generated voice fully combines the specific speaking context and is more realistic.

[0032] Specifically, in the pre-trained facial video acquisition encoder, there are facial expression dynamic change videos corresponding to human objects in different emotions pre-labeled according to expert experience, for example, facial expression dynamic change videos corresponding to glad, sad, and crying states respectively. To enable subsequent free selection of generated voices in different emotions for output according to the requirements of the output voice; it is also convenient to predict the emotion of the target human object B based on the target video frame sequence, and generate the generated voice of the target human object B in different emotional states according to the emotion prediction result.

[0033] Step 205: Input the text feature vector sequence and the visual feature vector sequence into the cross-modal fusion layer to obtain the feature fusion representation output by the cross-modal fusion layer.

[0034] By inputting the text feature vector sequence and the visual feature vector sequence into the cross-modal fusion layer, the feature fusion representation output by the cross-modal fusion layer is obtained, so as to be fully combined with the corresponding generation context when performing voice generation.

[0035] Step 206: Input the feature fusion representation into a preset non-autoregressive variant acoustic model to obtain the acoustic features output by the variant acoustic model.

[0036] In this embodiment, the variant acoustic model refers to introducing a facial video acquisition encoder into the original standard acoustic model, and introducing facial expression dynamic changes into the voice generation process as cross-modal information, so that the finally generated voice not only combines the corresponding text features but also combines the actual facial expression dynamic change information, and is more rhythmic and realistic.

[0037] It should be understood that through the preset non-autoregressive variant acoustic model, it is possible to achieve speech generation for the same target text sequence with a predicted emotional state.

[0038] Step 207: Input the acoustic features into a preset vocoder, and generate a target speech waveform through multi-scale convolution and upsampling processing, where the vocoder includes a lightweight HiFi-GAN or WaveGlow vocoder.

[0039] Specifically, the preset vocoder is to decode the acoustic features into a target speech waveform and connect to the corresponding output device to achieve speech generation and output.

[0040] HiFi-GAN( High-Fidelity Generative Adversarial Network ) is a new type of speech synthesis model. As an innovative application of the generative adversarial network (GAN) in the field of speech synthesis, HiFi-GAN greatly improves the efficiency of speech generation while ensuring high sound quality; WaveGlow is a flow-based speech synthesis model that combines the characteristics of Glow and WaveNet, solves the parallel computing problem of traditional neural network vocoders during inference, and at the same time maintains high-quality audio synthesis.

[0041] In this embodiment, by obtaining the text feature vector sequence of the target text sequence; obtaining the visual feature vector sequence of the target video frame sequence; inputting the text feature vector sequence and the visual feature vector sequence into the cross-modal fusion layer to obtain the feature fusion representation output by the cross-modal fusion layer; inputting the feature fusion representation into the preset non-autoregressive variant acoustic model to obtain the acoustic features output by the variant acoustic model; inputting the acoustic features into the preset vocoder, and generating a target speech waveform through multi-scale convolution and upsampling processing. It realizes that not only text features but also video features are introduced during speech generation, and cross-modal fusion features are used to jointly achieve speech generation, avoiding the final generated speech from being too mechanical and improving the realism of the generated speech.

[0042] Continue to refer to Figure 3 , Figure 3 is Figure 2 a flowchart of a specific embodiment of step 202 shown in Step 301: Use the embedding layer in the text encoder to perform a text vector representation conversion on the target text sequence to obtain a text vector representation result; Step 302: Use the self-attention layer in the text encoder to capture the global text dependency information of the target text sequence; Step 303: Using the position encoding mechanism in the text encoder, obtain the position vector information of all texts in the target text sequence; Step 304: Combine the text vector representation result, global text dependency information, and the position vector information of all texts to generate the text feature vector sequence.

[0043] Continue to refer to Figure 4 , Figure 4 is Figure 2 a flowchart of a specific embodiment of step 204 shown in the figure, including the following steps: Step 401: Use the feature extraction layer of the visual encoder to perform image feature extraction on the video frame sequence, and obtain the image features included in each video frame, where the image features include image local features and image global features; Step 402: Use the temporal modeling layer of the visual encoder to perform temporal modeling on consecutive video frames, and obtain the temporal modeling result; Step 403: Based on the temporal modeling result and the image features corresponding to each video frame, capture the dynamic change information between consecutive video frames; Step 404: Combine the image features included in each video frame, the temporal modeling result, and the dynamic change information between consecutive video frames to generate the visual feature vector sequence.

[0044] Specifically, when performing speech generation on the target text sequence, the dynamic change information of the facial expressions of the target person object is fully combined, so that the generated speech is more authentic and has a certain actual emotional value, avoiding the mechanization of the generated speech.

[0045] In this embodiment, after performing the step of capturing the dynamic change information between consecutive video frames based on the temporal modeling result and the image features corresponding to each video frame, that is, after step 403, the method further includes: according to the captured dynamic change information between consecutive video frames, combined with the pre-trained facial video acquisition encoder, predict the speech emotion change when performing speech generation for the target text sequence.

[0046] Specifically, according to the captured dynamic change information between consecutive video frames, combined with the pre-trained facial video acquisition encoder, predict the speech emotion change when performing speech generation for the target text sequence, so as to subsequently combine the language emotion change to generate more authentic speech.

[0047] In this embodiment, before the step of inputting the target video frame sequence into a preset visual encoder to obtain the visual feature vector sequence output by the visual encoder, that is, before step 204, the method further includes: pre-collecting the voices emitted by the target person object B in different emotional states, and the facial expression change videos corresponding to each voice segment; extracting prosodic features from each voice segment to obtain the prosodic feature extraction results corresponding to each voice segment; screening out the common prosodic features corresponding to all voice segments, and screening out the distinct prosodic features corresponding to each voice segment; establishing an association mapping relationship among the distinct prosodic features, emotional states, and facial expression change states.

[0048] By screening out the common prosodic features corresponding to all voice segments and the distinct prosodic features corresponding to each voice segment, when using the feature fusion mechanism for fusion feature representation later, on the one hand, the common prosodic features are added to the fusion feature representation, and on the other hand, according to the predicted emotion corresponding to the generated voice, the corresponding distinct prosodic features are added to the fusion feature representation, making the generated voice more realistic and having corresponding prosodic features.

[0049] In this embodiment, the cross-modal fusion layer includes a cross-modal attention mechanism and a feature fusion mechanism based on Transformer. By using the cross-modal attention mechanism, more emotional context representation information that needs to be focused on when generating speech for the target text sequence can be screened out from the visual feature vector sequence output by the visual encoder. Then, through the feature fusion mechanism, the distinct prosodic features corresponding to the more emotional context representation information that needs to be focused on are fused into speech generation to obtain context features with enhanced prosodic information, and the context features with enhanced prosodic information are used as the feature fusion representation output by the cross-modal fusion layer.

[0050] Continue to refer to Figure 5 , Figure 5 is Figure 2 a flowchart of a specific embodiment of step 205 shown below, including the following steps: Step 501, using the text feature vector sequence as the query sequence, the visual feature vector sequence as the key and value, and calculating the correlation between the text feature vector sequence and the visual feature vector sequence by using the multi-head attention calculation method in the cross-modal attention mechanism; Step 502, based on the correlation, fusing the text feature vector sequence and the visual feature vector sequence to obtain context features with enhanced prosodic information; Step 503, using the context features with enhanced prosodic information as the feature fusion representation output by the cross-modal fusion layer.

[0051] In this embodiment, the variant acoustic model includes a variant acoustic model jointly composed of the FastSpeech standard acoustic model, the text encoder, and the visual encoder. By constructing the variant acoustic model, when generating speech for the target text sequence processed by the text encoder, it is possible to incorporate the facial expression changes of the target person object collected by the visual encoder, predict the current emotion of the target person object, and generate speech in combination with the current emotion. By introducing the visual encoder, visual context and text information are fully integrated, improving problems such as single prosody modeling and insufficient information in traditional text-to-speech systems, significantly enhancing the naturalness and realism of speech generation, and avoiding the generated speech from being too mechanical. Applying the speech generation method to the intelligent customer service application scenarios of finance or medical services, or in the marketing and introduction scenarios of financial or medical service products, it is possible to generate more natural and realistic speech by combining visual context information.

[0052] Continue to refer to Figure 6 , Figure 6 is Figure 2 a flowchart of a specific embodiment of step 206 shown below, including the following steps: Step 601, in the prosody prediction layer of the variant acoustic model, use the feature fusion representation to predict prosody parameters for all pronounced words in the target text sequence, obtaining a prosody parameter prediction result, where the prosody parameters include pronunciation pitch, pronunciation duration, and pronunciation energy value; Step 602, in the feature generation layer of the variant acoustic model, use the prosody parameter prediction result to adjust the initial Mel spectrogram generated according to the target text sequence, obtaining the adjusted Mel spectrogram as the prosody reconstruction result; Step 603, perform parallel decoding processing on the feature fusion representation in a parallel decoding manner to obtain the decoded text sequence; Step 604, generate a corresponding acoustic frame sequence as the acoustic feature output according to the decoded text sequence and the prosody reconstruction result.

[0053] Continue to refer to Figure 7 , in some specific embodiments, before performing step 206, the method further includes a step of training the variant acoustic model. Figure 7 is a flowchart of a specific embodiment of training the variant acoustic model in the speech generation method described in this embodiment, including: Step 701, obtain text and visual paired data and construct a training data set; Specifically, the text and visual pairing data can be the speech uttered by the target person object in different emotional states collected in advance, and the video of the corresponding facial expression changes for each segment of speech; through speech-to-text processing, text data is obtained, and the pairing relationship between the text data and the video of facial expression changes is constructed, that is, the corresponding text and visual pairing data is obtained.

[0054] Step 702: Extract text features and visual features from the training data set respectively to obtain the text feature vector sequence and visual feature vector sequence corresponding to each piece of training data. Step 703: Input the text feature vector sequence and visual feature vector sequence corresponding to each piece of training data into the cross-modal fusion layer in groups to obtain the feature fusion representation of each piece of training data output by the cross-modal fusion layer. Step 704: Input the feature fusion representation of each piece of training data into the to-be-trained non-autoregressive variant acoustic model, and perform prosody reconstruction training, acoustic feature generation training, and speech waveform reconstruction training on the variant acoustic model in sequence to obtain the trained variant acoustic model. Step 705: Set the trained variant acoustic model as the preset non-autoregressive variant acoustic model.

[0055] Continue to refer to Figure 8 , in some specific embodiments, during the training of the variant acoustic model, the speech generation method further includes using a joint loss method to determine whether the variant acoustic model is pre-trained. Figure 8 Yes Figure 7 FIG. 8 is a flowchart of a specific embodiment for determining whether the variant acoustic model is pre-trained in step 704, including the following steps: Step 801: Use the joint loss method to determine whether the variant acoustic model is trained. Specifically, the joint loss includes prosody reconstruction loss, acoustic feature reconstruction loss, speech waveform reconstruction loss, and text-speech alignment loss. Loss weights are set for the prosody reconstruction loss, acoustic feature reconstruction loss, speech waveform reconstruction loss, and text-speech alignment loss respectively, and the sum of the loss weights corresponding to the prosody reconstruction loss, acoustic feature reconstruction loss, speech waveform reconstruction loss, and text-speech alignment loss is 1. Step 802: If the prosody reconstruction loss, acoustic feature reconstruction loss, speech waveform reconstruction loss, and text-speech alignment loss all meet the corresponding preset loss thresholds, and the joint loss also meets the preset joint loss threshold, then the variant acoustic model is trained. Step 803, if any of the prosody reconstruction loss, acoustic feature reconstruction loss, speech waveform reconstruction loss, and text-speech alignment loss does not meet the corresponding preset loss threshold, or the combined loss does not meet the preset combined loss threshold, then the variant acoustic model is not trained successfully.

[0056] In this embodiment, a text feature vector sequence of a target text sequence is obtained; a visual feature vector sequence of a target video frame sequence is obtained; the text feature vector sequence and the visual feature vector sequence are input into a cross-modal fusion layer to obtain a feature fusion representation output by the cross-modal fusion layer; the feature fusion representation is input into a preset non-autoregressive variant acoustic model to obtain an acoustic feature output by the variant acoustic model; the acoustic feature is input into a preset vocoder, and through multi-scale convolution and upsampling processing, a target speech waveform is generated. It realizes that not only text features but also video features are introduced during speech generation, and cross-modal fusion features are used to jointly realize speech generation, avoiding the final generated speech from being too mechanical and improving the realism of the generated speech. By fully fusing visual context and text information, it improves problems such as single prosody modeling and insufficient information in traditional text-to-speech systems, significantly improving the naturalness and realism of speech generation and avoiding the generated speech from being too mechanical. Applying the speech generation method to financial or medical business intelligent customer service application scenarios, or in financial or medical service product marketing introduction scenarios, realizes generating more natural and realistic speech by combining visual context information.

[0057] Embodiments of this application can acquire and process relevant data based on artificial intelligence technology. Among them, artificial intelligence (AI) is a theory, method, technology, and application system that uses digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use knowledge to obtain the best results.

[0058] Artificial intelligence basic technologies generally include technologies such as sensors, dedicated artificial intelligence chips, cloud computing, distributed storage, big data processing technologies, operation / interaction systems, and mechatronics. Artificial intelligence software technologies mainly include several major directions such as computer vision technology, robotics, biometric technology, speech processing technology, natural language processing technology, and machine learning / deep learning.

[0059] In this embodiment, a text feature vector sequence of a target text sequence is obtained; a visual feature vector sequence of a target video frame sequence is obtained; the text feature vector sequence and the visual feature vector sequence are input into a cross-modal fusion layer to obtain a feature fusion representation output by the cross-modal fusion layer; the feature fusion representation is input into a preset non-autoregressive variant acoustic model to obtain an acoustic feature output by the variant acoustic model; and the acoustic feature is input into a preset vocoder to generate a target speech waveform through multi-scale convolution and upsampling processing. When generating speech, not only text features but also video features are introduced, and speech generation is jointly realized with cross-modal fusion features, avoiding the final generated speech from being too mechanical and improving the realism of the generated speech. By fully fusing visual context and text information, problems such as single prosody modeling and insufficient information in traditional text-to-speech systems are improved, significantly enhancing the naturalness and realism of speech generation and avoiding the generated speech from being too mechanical. Applying the speech generation method to intelligent customer service application scenarios in the financial or medical business, or in marketing and introduction scenarios of financial or medical service products, realizes generating more natural and realistic speech by combining visual context information.

[0060] Further referring to Figure 9 , as an implementation of the method shown above Figure 2 , an embodiment of a speech generation device is provided in the present application. This device embodiment corresponds to the method embodiment shown in Figure 2 , and this device can be specifically applied to various electronic devices.

[0061] As Figure 9 shown, the speech generation device 900 described in this embodiment includes: a target text sequence acquisition module 901, a text feature vector sequence encoding module 902, a target video frame sequence acquisition module 903, a visual feature vector sequence encoding module 904, a cross-modal fusion module 905, an acoustic feature acquisition module 906, and a target speech waveform generation module 907. Among them: The target text sequence acquisition module 901 is used to acquire a target text sequence; The text feature vector sequence encoding module 902 is used to input the target text sequence into a preset text encoder to obtain a text feature vector sequence output by the text encoder; The target video frame sequence acquisition module 903 is used to acquire a target video frame sequence; The visual feature vector sequence encoding module 904 is used to input the target video frame sequence into a preset visual encoder to obtain a visual feature vector sequence output by the visual encoder; The cross-modal fusion module 905 is configured to input the text feature vector sequence and the visual feature vector sequence into a cross-modal fusion layer, and obtain a feature fusion representation output by the cross-modal fusion layer; The acoustic feature acquisition module 906 is configured to input the feature fusion representation into a preset non-autoregressive variant acoustic model, and obtain acoustic features output by the variant acoustic model; The target speech waveform generation module 907 is configured to input the acoustic features into a preset vocoder, and generate a target speech waveform through multi-scale convolution and upsampling processing, where the vocoder includes a lightweight HiFi-GAN or WaveGlow vocoder.

[0062] In this application, a text feature vector sequence of a target text sequence is obtained; a visual feature vector sequence of a target video frame sequence is obtained; the text feature vector sequence and the visual feature vector sequence are input into a cross-modal fusion layer to obtain a feature fusion representation output by the cross-modal fusion layer; the feature fusion representation is input into a preset non-autoregressive variant acoustic model to obtain acoustic features output by the variant acoustic model; the acoustic features are input into a preset vocoder, and a target speech waveform is generated through multi-scale convolution and upsampling processing. It realizes that not only text features but also video features are introduced during speech generation, and cross-modal fusion features are used to jointly realize speech generation, avoiding the final generated speech from being too mechanical and improving the realism of the generated speech.

[0063] In this embodiment, the text feature vector sequence encoding module 902 includes a text vector representation conversion unit, a global text dependency information capturing unit, a position vector information encoding unit, and a text feature vector sequence generating unit. Among them: The text vector representation conversion unit is configured to use the embedding layer in the text encoder to perform text vector representation conversion on the target text sequence to obtain a text vector representation result; The global text dependency information capturing unit is configured to use the self-attention layer in the text encoder to capture the global text dependency information of the target text sequence; The position vector information encoding unit is configured to use the position encoding mechanism in the text encoder to obtain the position vector information of all texts in the target text sequence; The text feature vector sequence generating unit is configured to generate the text feature vector sequence by combining the text vector representation result, the global text dependency information, and the position vector information of all texts.

[0064] In this embodiment, the visual feature vector sequence encoding module 904 includes an image feature extraction unit, a temporal modeling unit, a dynamic change information capturing unit, and a visual feature vector sequence generating unit. Among them: An image feature extraction unit, which is used to extract image features from the video frame sequence by using the feature extraction layer of the visual encoder, so as to obtain the image features included in each video frame respectively, where the image features include image local features and image global features; A temporal modeling unit, which is used to perform temporal modeling on consecutive video frames by using the temporal modeling layer of the visual encoder to obtain a temporal modeling result; A dynamic change information capture unit, which is used to capture the dynamic change information between consecutive video frames based on the temporal modeling result and the image features corresponding to each video frame; A visual feature vector sequence generation unit, which is used to generate the visual feature vector sequence by combining the image features included in each video frame respectively, the temporal modeling result, and the dynamic change information between consecutive video frames.

[0065] In this embodiment, the cross-modal fusion module 905 includes a cross-modal attention mechanism calculation unit, a cross-modal fusion unit, and a feature fusion representation output unit. Among them: The cross-modal attention mechanism calculation unit is used to use the text feature vector sequence as the query sequence, the visual feature vector sequence as the key and value, and calculate the correlation between the text feature vector sequence and the visual feature vector sequence by using the multi-head attention calculation method in the cross-modal attention mechanism; The cross-modal fusion unit is used to fuse the text feature vector sequence and the visual feature vector sequence based on the correlation to obtain context features with the function of enhancing prosodic information; The feature fusion representation output unit is used to use the context features with the function of enhancing prosodic information as the feature fusion representation output by the cross-modal fusion layer.

[0066] In this embodiment, the acoustic feature acquisition module 906 includes a prosody parameter prediction unit, a prosody reconstruction unit, a feature fusion representation decoding unit, and an acoustic feature output unit. Among them: The prosody parameter prediction unit is used to predict prosody parameters for all pronounced characters in the target text sequence by using the feature fusion representation in the prosody prediction layer of the variant acoustic model to obtain a prosody parameter prediction result, where the prosody parameters include pronunciation pitch, pronunciation duration, and pronunciation energy value; The prosody reconstruction unit is used to adjust the initial Mel spectrogram generated according to the target text sequence by using the prosody parameter prediction result in the feature generation layer of the variant acoustic model to obtain the adjusted Mel spectrogram as the prosody reconstruction result; The feature fusion representation decoding unit is used to perform parallel decoding processing on the feature fusion representation by using a parallel decoding method to obtain the decoded text sequence; An acoustic feature output unit, configured to generate a corresponding acoustic frame sequence as the acoustic feature output according to the text sequence obtained by the decoding and the prosody reconstruction result.

[0067] Those of ordinary skill in the art can understand that all or part of the processes in the methods of the above embodiments can be completed by instructing relevant hardware through computer-readable instructions. The computer-readable instructions can be stored in a computer-readable storage medium. When the program is executed, it can include the processes of the embodiments of the above methods. Among them, the aforementioned storage medium can be a non-volatile storage medium such as a magnetic disk, an optical disk, a read-only memory (ROM), or a random access memory (RAM), etc.

[0068] It should be understood that although the steps in the flowchart of the accompanying drawings are shown in sequence according to the indication of the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless there is a clear indication in this article, the execution of these steps has no strict order limit, and they can be executed in other orders. Moreover, at least some of the steps in the flowchart of the accompanying drawings may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily executed at the same moment, but can be executed at different moments. Their execution order is not necessarily sequential, but can be executed alternately or alternately with at least a part of other steps or sub-steps or stages of other steps.

[0069] To solve the above technical problems, an embodiment of the present application further provides a computer device. For details, please refer to Figure 10 , Figure 10 which is the basic structural block diagram of the computer device in this embodiment.

[0070] The computer device 10 includes a memory 10a, a processor 10b, and a network interface 10c that are communicatively connected to each other through a system bus. It should be noted that Figure 10Only a computer device 10 with a component memory 10a, a processor 10b, and a network interface 10c is shown, but it should be understood that it is not required to implement all the shown components, and more or fewer components can be implemented alternatively. Among them, those skilled in the art of the present technology can understand that the computer device here is a device capable of automatically performing numerical calculations and / or information processing according to pre-set or stored instructions, and its hardware includes but is not limited to microprocessors, application specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), digital signal processors (DSPs), embedded devices, etc.

[0071] The computer device may be a computing device such as a desktop computer, a notebook, a palm computer, and a cloud server. The computer device can perform human-computer interaction with the user through means such as a keyboard, a mouse, a remote control, a touchpad, or a voice control device.

[0072] The memory 10a includes at least one type of readable storage medium, and the readable storage medium includes flash memory, a hard disk, a multimedia card, a card-type memory (such as an SD or DX memory, etc.), a random access memory (RAM), a static random access memory (SRAM), a read-only memory (ROM), an electrically erasable programmable read-only memory (EEPROM), a programmable read-only memory (PROM), a magnetic memory, a magnetic disk, an optical disk, etc. In some embodiments, the memory 10a may be an internal storage unit of the computer device 10, such as the hard disk or memory of the computer device 10. In other embodiments, the memory 10a may also be an external storage device of the computer device 10, such as a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, etc., equipped on the computer device 10. Of course, the memory 10a may also include both the internal storage unit and the external storage device of the computer device 10. In this embodiment, the memory 10a is generally used to store the operating system and various application software installed on the computer device 10, such as computer-readable instructions of a voice generation method. In addition, the memory 10a may also be used to temporarily store various data that have been output or will be output.

[0073] In some embodiments, the processor 10b may be a Central Processing Unit (CPU), a controller, a microcontroller, a microprocessor, or other data processing chips. The processor 10b is generally used to control the overall operation of the computer device 10. In this embodiment, the processor 10b is used to run the computer-readable instructions stored in the memory 10a or process data, such as running the computer-readable instructions of the voice generation method described above.

[0074] The network interface 10c may include a wireless network interface or a wired network interface, and this network interface 10c is generally used to establish a communication connection between the computer device 10 and other electronic devices.

[0075] The computer device proposed in this embodiment belongs to the technical field of voice generation and is applied to the voice generation scenario in financial or medical services. This application obtains the text feature vector sequence of the target text sequence; obtains the visual feature vector sequence of the target video frame sequence; inputs the text feature vector sequence and the visual feature vector sequence into the cross-modal fusion layer to obtain the feature fusion representation output by the cross-modal fusion layer; inputs the feature fusion representation into a preset non-autoregressive variant acoustic model to obtain the acoustic features output by the variant acoustic model; inputs the acoustic features into a preset vocoder, and through multi-scale convolution and upsampling processing, generates the target speech waveform. It realizes that not only text features but also video features are introduced during voice generation, and the voice is generated jointly with cross-modal fusion features, avoiding the final generated voice from being too mechanical and improving the realism of the generated voice.

[0076] This application also provides another implementation manner, that is, to provide a computer-readable storage medium storing computer-readable instructions, and the computer-readable instructions can be executed by a processor to enable the processor to execute the steps of a voice generation method as described above.

[0077] The computer-readable storage medium proposed in this embodiment belongs to the technical field of speech generation and is applied to the speech generation scenario in financial or medical services. In this application, a text feature vector sequence of a target text sequence is obtained; a visual feature vector sequence of a target video frame sequence is obtained; the text feature vector sequence and the visual feature vector sequence are input into a cross-modal fusion layer to obtain a feature fusion representation output by the cross-modal fusion layer; the feature fusion representation is input into a preset non-autoregressive variant acoustic model to obtain an acoustic feature output by the variant acoustic model; the acoustic feature is input into a preset vocoder, and through multi-scale convolution and upsampling processing, a target speech waveform is generated. It realizes that not only text features but also video features are introduced during speech generation, and speech generation is jointly realized with cross-modal fusion features, avoiding the final generated speech from being too mechanical and improving the realism of the generated speech.

[0078] Through the description of the above embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus a necessary general hardware platform. Of course, they can also be implemented by hardware, but in many cases, the former is a better implementation method. Based on such an understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions for causing a terminal device (which can be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in various embodiments of this application.

[0079] Obviously, the above-described embodiments are only a part of the embodiments of this application, rather than all embodiments. The accompanying drawings show the preferred embodiments of this application, but do not limit the patent scope of this application. This application can be implemented in many different forms. On the contrary, the purpose of providing these embodiments is to make the understanding of the disclosed content of this application more thorough and comprehensive. Although this application has been described in detail with reference to the foregoing embodiments, for those skilled in the art, they can still modify the technical solutions described in the foregoing specific embodiments, or perform equivalent replacements on some of the technical features. Any equivalent structure directly or indirectly using the content of the specification and drawings of this application in other related technical fields is equally within the scope of the patent protection of this application. The non-company software tools or components that appear in the embodiments of this application are only for illustration and do not represent actual use.

Claims

1. A voice generation method, characterized in that, Including the following steps: Obtain a target text sequence; Input the target text sequence into a preset text encoder to obtain a text feature vector sequence output by the text encoder; Obtain a target video frame sequence; Input the target video frame sequence into a preset visual encoder to obtain a visual feature vector sequence output by the visual encoder; Input the text feature vector sequence and the visual feature vector sequence into a cross-modal fusion layer to obtain a feature fusion representation output by the cross-modal fusion layer; Input the feature fusion representation into a preset non-autoregressive variant acoustic model to obtain acoustic features output by the variant acoustic model; Input the acoustic features into a preset vocoder, and generate a target speech waveform through multi-scale convolution and upsampling processing, where the vocoder includes a lightweight HiFi-GAN or WaveGlow vocoder.

2. The voice generation method according to claim 1, wherein The text encoder includes a lightweight Transformer or CNN encoder. The step of inputting the target text sequence into a preset text encoder to obtain a text feature vector sequence output by the text encoder specifically includes: Use the embedding layer in the text encoder to perform text vector representation conversion on the target text sequence to obtain a text vector representation result; Use the self-attention layer in the text encoder to capture the global text dependency information of the target text sequence; Use the position encoding mechanism in the text encoder to obtain the position vector information of all texts in the target text sequence; Combine the text vector representation result, the global text dependency information, and the position vector information of all texts to generate the text feature vector sequence.

3. The voice generation method according to claim 1, wherein The visual encoder includes a facial video acquisition encoder composed of a pre-trained ResNet series model or Vision Transformer model. The step of inputting the target video frame sequence into a preset visual encoder to obtain a visual feature vector sequence output by the visual encoder specifically includes: Use the feature extraction layer of the visual encoder to extract image features from the video frame sequence to obtain the image features included in each video frame, where the image features include image local features and image global features; Use the temporal modeling layer of the visual encoder to perform temporal modeling on consecutive video frames to obtain a temporal modeling result; Based on the temporal modeling result and the image features corresponding to each video frame, capture the dynamic change information between consecutive video frames; Combine the image features included in each video frame, the temporal modeling result, and the dynamic change information between consecutive video frames to generate the visual feature vector sequence.

4. The voice generation method according to claim 1, characterized in that, The cross-modal fusion layer includes a cross-modal attention mechanism and a feature fusion mechanism based on Transformer. The step of inputting the text feature vector sequence and the visual feature vector sequence into the cross-modal fusion layer to obtain a feature fusion representation output by the cross-modal fusion layer specifically includes: Using the text feature vector sequence as the query sequence, the visual feature vector sequence as the key and value, calculate the correlation between the text feature vector sequence and the visual feature vector sequence by using the multi-head attention calculation method in the cross-modal attention mechanism; Based on the correlation, fuse the text feature vector sequence and the visual feature vector sequence to obtain context features with the function of enhancing prosodic information; Use the context features with the function of enhancing prosodic information as the feature fusion representation output by the cross-modal fusion layer.

5. The voice generation method according to claim 1, wherein The variant acoustic model includes a variant acoustic model jointly composed of the FastSpeech standard acoustic model, the text encoder, and the visual encoder. The step of inputting the feature fusion representation into a preset non-autoregressive variant acoustic model to obtain the acoustic features output by the variant acoustic model specifically includes: In the prosody prediction layer of the variant acoustic model, use the feature fusion representation to predict the prosody parameters of all pronounced words in the target text sequence to obtain a prosody parameter prediction result, where the prosody parameters include pronunciation pitch, pronunciation duration, and pronunciation energy value; In the feature generation layer of the variant acoustic model, use the prosody parameter prediction result to adjust the initial Mel spectrogram generated according to the target text sequence to obtain the adjusted Mel spectrogram as the prosody reconstruction result; Perform parallel decoding processing on the feature fusion representation by using a parallel decoding method to obtain the decoded text sequence; According to the decoded text sequence and the prosody reconstruction result, generate a corresponding acoustic frame sequence as the acoustic feature output.

6. The speech generation method according to any one of claims 1 or 5, characterized in that Before performing the step of inputting the feature fusion representation into a preset non-autoregressive variant acoustic model to obtain the acoustic features output by the variant acoustic model, the method further includes: Obtain text and visual paired data and construct a training data set; Extract text features and visual features from the training data set respectively to obtain a text feature vector sequence and a visual feature vector sequence corresponding to each piece of training data; Group and input the text feature vector sequence and the visual feature vector sequence corresponding to each piece of training data into the cross-modal fusion layer to obtain the feature fusion representation of each piece of training data output by the cross-modal fusion layer; Input the feature fusion representation of each piece of training data into a non-autoregressive variant acoustic model to be trained, and perform prosody reconstruction training, acoustic feature generation training, and speech waveform reconstruction training on the variant acoustic model in sequence to obtain a trained variant acoustic model; Obtain the trained variant acoustic model and set it as the preset non-autoregressive variant acoustic model.

7. The voice generation method according to claim 6, wherein The step of inputting the feature fusion representation of each piece of training data into a non-autoregressive variant acoustic model to be trained, and performing prosody reconstruction training, acoustic feature generation training, and speech waveform reconstruction training on the variant acoustic model in sequence to obtain a trained variant acoustic model further includes: Adopt a combined loss method to determine whether the variant acoustic model is trained. Specifically, the combined loss includes prosody reconstruction loss, acoustic feature reconstruction loss, speech waveform reconstruction loss, and text-speech alignment loss. Loss weights are set for the prosody reconstruction loss, acoustic feature reconstruction loss, speech waveform reconstruction loss, and text-speech alignment loss respectively, and the sum of the loss weights corresponding to the prosody reconstruction loss, acoustic feature reconstruction loss, speech waveform reconstruction loss, and text-speech alignment loss is 1; If the prosody reconstruction loss, acoustic feature reconstruction loss, speech waveform reconstruction loss, and text-speech alignment loss all meet their respective preset loss thresholds, and the combined loss also meets the preset combined loss threshold, then the variant acoustic model is trained; If any of the prosody reconstruction loss, acoustic feature reconstruction loss, speech waveform reconstruction loss, and text-speech alignment loss does not meet its respective preset loss threshold, or the combined loss does not meet the preset combined loss threshold, then the variant acoustic model is not trained.

8. A voice generation device, characterized in that, Comprising: A target text sequence acquisition module for acquiring a target text sequence; A text feature vector sequence encoding module for inputting the target text sequence into a preset text encoder to obtain a text feature vector sequence output by the text encoder; A target video frame sequence acquisition module for acquiring a target video frame sequence; A visual feature vector sequence encoding module for inputting the target video frame sequence into a preset visual encoder to obtain a visual feature vector sequence output by the visual encoder; A cross-modal fusion module for inputting the text feature vector sequence and the visual feature vector sequence into a cross-modal fusion layer to obtain a feature fusion representation output by the cross-modal fusion layer; An acoustic feature acquisition module for inputting the feature fusion representation into a preset non-autoregressive variant acoustic model to obtain acoustic features output by the variant acoustic model; A target speech waveform generation module for inputting the acoustic features into a preset vocoder and generating a target speech waveform through multi-scale convolution and upsampling processing, where the vocoder includes a lightweight HiFi-GAN or WaveGlow vocoder.

9. A computer device, characterized in that, Comprising a memory and a processor, wherein computer-readable instructions are stored in the memory, and when the processor executes the computer-readable instructions, the steps of the speech generation method according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium, characterized in that, Computer-readable instructions are stored on the computer-readable storage medium, and when the computer-readable instructions are executed by the processor, the steps of the speech generation method according to any one of claims 1 to 7 are implemented.

Images