Voice generation method and device, equipment, storage medium and product

Abstract

The embodiment of the invention provides a voice generation method and device, equipment, a storage medium and a product, and the method comprises the steps: extracting the text features of a to-be-processed text, and extracting the scene image features of a to-be-processed scene image; and based on a mapping relationship between the image features and acoustic features, performing mapping processing on the scene image features to obtain acoustic features corresponding to the scene image features. And based on the text feature and the acoustic feature corresponding to the scene image feature, generating a target voice which corresponds to a to-be-processed text and is matched with the environment sound effect of a scene indicated by the to-be-processed scene image. According to the technical scheme, the environment factors indicated by the scene image are considered in the process of generating the voice corresponding to the text, the scene corresponding to the scene image can be accurately reflected while expected information can be conveyed, the generated voice is more immersive and realistic, and the voice generation effect is improved to a certain extent.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a speech generation method, device, equipment, storage medium and product. Background Art

[0002] With the rapid development of artificial intelligence technology, Text-to-Speech (TTS) technology, which converts written text into user-audible speech, has become one of the most important branches of artificial intelligence technology. Through complex algorithms and models, TTS technology can efficiently parse input text information and generate speech output close to natural language, enriching the way of information transmission and enhancing the convenience of users to obtain information.

[0003] At present, although TTS technology can imitate the way humans speak to a certain extent, such as simulating the speaker's emotional state and generating speech that can reflect specific emotions, and automatically adjusting the speaking speed, tone and stress according to different situations, the speech generation algorithms or models in TTS technology are usually designed based on a neutral or reverberation-free environment, which makes the speech generated by these algorithms or models lack realism and naturalness, and the generation effect is poor. Summary of the invention

[0004] The embodiments of the present application provide a speech generation method, apparatus, device, storage medium and product, which can convey the expected information while accurately reflecting the scenario given by the user, making the generated speech more immersive and realistic, and improving the effect of speech generation to a certain extent.

[0005] In a first aspect, an embodiment of the present application provides a speech generation method, comprising:

[0006] Extracting text features of the text to be processed, and extracting scene image features of the scene image to be processed;

[0007] Based on the mapping relationship between the image feature and the acoustic feature, mapping processing is performed on the scene image feature to obtain the acoustic feature corresponding to the scene image feature;

[0008] Based on the acoustic features corresponding to the text features and the scene image features, a target speech is generated that corresponds to the text to be processed and matches the ambient sound effects of the scene indicated by the scene image to be processed.

[0009] In a second aspect, an embodiment of the present application provides a speech generation method, comprising:

[0010] Acquire a training text, a training scene image, and a training voice corresponding to the training text and matching the environmental sound effects of the scene indicated by the training scene image;

[0011] Performing feature extraction processing on the frequency domain representation information corresponding to the training speech to obtain posterior distribution parameters;

[0012] Perform speech generation processing based on the posterior distribution parameters to obtain generated speech;

[0013] Based on the training text, the training scene image, the training speech, the generated speech and the posterior distribution parameters, the machine learning model is trained to obtain a speech generation model; wherein the speech generation model is used to generate a target speech corresponding to the input text and matching the environmental sound effects indicated by the input image based on the input text and the input image.

[0014] In a third aspect, an embodiment of the present application provides a speech generation device, including:

[0015] An extraction unit, used to extract text features of the text to be processed and to extract scene image features of the scene image to be processed;

[0016] A processing unit, configured to perform mapping processing on the scene image features based on a mapping relationship between the image features and the acoustic features, so as to obtain acoustic features corresponding to the scene image features;

[0017] A generating unit is used to generate a target speech corresponding to the text to be processed and matching the ambient sound effect of the scene indicated by the scene image to be processed based on the acoustic features corresponding to the text features and the scene image features.

[0018] In a possible implementation, the processing unit is used to perform mapping processing on the scene image feature based on the mapping relationship between the image feature and the acoustic feature to obtain the acoustic feature corresponding to the scene image feature, specifically for:

[0019] Performing a first linear transformation process on the scene image feature to obtain a transformed scene image feature;

[0020] Performing nonlinear mapping processing on the transformed scene image features based on a set activation function;

[0021] A second linear transformation is performed on the image features obtained through the nonlinear mapping process to obtain acoustic features corresponding to the scene image features.

[0022] In a possible implementation, the text feature is obtained by extracting a feature of a symbol sequence of pronunciation elements corresponding to the text to be processed; the generating unit is used to generate a target speech corresponding to the text to be processed and matching the environmental sound effect of the scene indicated by the scene image to be processed based on the text feature and the acoustic features corresponding to the scene image feature, specifically for:

[0023] Based on the mapping relationship between the pronunciation elements and the duration, mapping processing is performed on the duration of the pronunciation elements indicated by the text features to obtain a duration prediction feature;

[0024] Performing a third linear transformation process on the text features to obtain transformed text features, and converting the transformed text features into prior distribution parameters according to a set conversion function;

[0025] The target speech is generated based on the duration prediction feature, the prior distribution parameter and the acoustic features corresponding to the scene image features.

[0026] In a possible implementation, the prior distribution parameters include a prior mean parameter and a prior variance parameter; the generating unit is used to generate the target speech based on the duration prediction feature, the prior distribution parameters and the acoustic features corresponding to the scene image features, specifically for:

[0027] Based on the duration prediction feature, sampling processing is performed in the prior distribution defined by the prior mean parameter and the prior variance parameter to obtain a first sampling feature;

[0028] The target speech is generated based on the first sampling feature and the acoustic features corresponding to the scene image features.

[0029] In a possible implementation, the dimension of the transformed text feature is the same as the dimension of the duration prediction feature; the generating unit is used to perform sampling processing in the prior distribution defined by the prior mean parameter and the prior variance parameter based on the duration prediction feature to obtain a first sampling feature, specifically for:

[0030] Based on each duration feature element in the duration prediction feature, repeating the feature element at the corresponding position in the transformed text feature for the number of times indicated by each duration feature element to obtain an extended text feature;

[0031] Based on the extended text feature, sampling processing is performed from the prior distribution to obtain the first sampling feature; wherein the dimension of the first sampling feature is the dimension of the extended text feature.

[0032] In a possible implementation manner, the generating unit is used to generate the target speech based on the acoustic features corresponding to the first sampling features and the scene image features, specifically to:

[0033] Based on the mapping relationship between the sampling feature and the speech signal feature, mapping processing is performed on the first sampling feature to obtain the speech signal feature corresponding to the first sampling feature;

[0034] The speech signal feature corresponding to the first sampling feature and the acoustic feature corresponding to the scene image feature are fused to obtain the target speech.

[0035] In a fourth aspect, an embodiment of the present application provides a speech generation device, including:

[0036] An acquisition unit, configured to acquire a training text, a training scene image, and a training voice corresponding to the training text and matching the environmental sound effects of the scene indicated by the training scene image;

[0037] The processing unit is further used to perform feature extraction processing on the frequency domain representation information corresponding to the training speech to obtain posterior distribution parameters;

[0038] The processing unit is further used to perform speech generation processing based on the posterior distribution parameters to obtain generated speech;

[0039] A training unit is used to train a machine learning model based on the training text, the training scene image, the training speech, the generated speech and the posterior distribution parameters to obtain a speech generation model; wherein the speech generation model is used to generate a target speech corresponding to the input text and matching the environmental sound effects indicated by the input image based on the input text and the input image.

[0040] In a possible implementation, the acquisition unit is used to acquire a training voice corresponding to the training text and matching the environmental sound effect of the scene indicated by the training scene image, specifically for:

[0041] Acquire the environmental sound effects corresponding to the training scene image and the speech corresponding to the training text;

[0042] The training speech is obtained by performing convolution processing based on the speech corresponding to the training text and the environmental sound effect.

[0043] In a possible implementation, the posterior distribution parameters include a posterior mean parameter and a posterior variance parameter; the processing unit is further used to perform speech generation processing based on the posterior distribution parameters to obtain generated speech, specifically for:

[0044] Perform sampling processing in the posterior distribution defined by the posterior mean parameter and the posterior variance parameter to obtain a second sampling feature;

[0045] Speech generation processing is performed based on the second sampling feature to obtain the generated speech.

[0046] In a possible implementation, the training unit is used to train the machine learning model based on the training text, the training scene image, the training speech, the generated speech and the posterior distribution parameter to obtain the speech generation model, specifically for:

[0047] Performing a fourth linear transformation process on the training text features extracted based on the training text to obtain training transformed text features, and converting the training transformed text features into training prior distribution parameters according to a set conversion function;

[0048] Based on the mapping relationship between the pronunciation elements and the duration, mapping processing is performed on the duration of the pronunciation elements indicated by the training text features to obtain training duration prediction features;

[0049] Based on the training duration prediction feature, sampling is performed in the training prior distribution defined by the training prior distribution parameter to obtain a third sampling feature,

[0050] Performing an alignment process on the second sampling feature and the third sampling feature to obtain an aligned estimated feature;

[0051] Based on the training duration prediction feature, the training prior distribution, the training scene image, the training speech, the generated speech, the posterior distribution and the alignment estimation feature, the machine learning model is trained to obtain the speech generation model.

[0052] In a possible implementation, the training unit is used to train the machine learning model based on the training duration prediction feature, the training prior distribution, the training scene image, the training speech, the generated speech, the posterior distribution and the alignment estimation feature to obtain the speech generation model, specifically for:

[0053] Based on the mapping relationship between the image features and the acoustic features, mapping processing is performed on the training scene image features extracted based on the training scene image to obtain the acoustic features corresponding to the training scene image features;

[0054] Based on the training duration prediction feature, the training prior distribution, the training speech, the generated speech, the posterior distribution, the alignment estimation feature and the acoustic features corresponding to the training scene image features, the machine learning model is trained to obtain the speech generation model.

[0055] In a possible implementation, the training unit is used to train the machine learning model based on the training duration prediction feature, the training prior distribution, the training speech, the generated speech, the posterior distribution, the alignment estimation feature and the acoustic feature corresponding to the training scene image feature to obtain the speech generation model, specifically for:

[0056] generating reconstruction loss data based on a difference between the frequency domain representation information corresponding to the generated speech and the frequency domain representation information corresponding to the training speech;

[0057] generating distribution loss data based on the difference between the training prior distribution and the posterior distribution;

[0058] Generating duration loss data based on the difference between the training duration prediction feature and the alignment estimation feature;

[0059] generating acoustic feature loss data based on a difference between the acoustic feature corresponding to the training scene image feature and the scene category label corresponding to the training scene image;

[0060] Based on the sum of the reconstruction loss data, the distribution loss data, the duration loss data and the acoustic feature loss data, the machine learning model is trained to obtain the speech generation model.

[0061] In a possible implementation, the processing unit is further configured to perform scene classification processing on the frequency domain representation information corresponding to the generated speech to obtain a scene category corresponding to the generated speech;

[0062] generating scene category loss data based on a difference between the scene category and the scene category label;

[0063] The training unit is used to train the machine learning model based on the sum of the reconstruction loss data, the distribution loss data, the duration loss data and the acoustic feature loss data to obtain the speech generation model, specifically for:

[0064] Based on the sum of the reconstruction loss data, the distribution loss data, the duration loss data, the acoustic feature loss data and the scene category loss data, the machine learning model is trained to obtain the speech generation model.

[0065] In a possible implementation, the processing unit is further configured to perform discrimination processing on the training speech and the generated speech to obtain a probability that each speech in the training speech and the generated speech belongs to the training speech category;

[0066] A determination unit, configured to determine the discrimination loss data based on the probability that each speech is a training speech category;

[0067] The training unit is used to train the machine learning model based on the sum of the reconstruction loss data, the distribution loss data, the duration loss data, the acoustic feature loss data and the scene category loss data to obtain the speech generation model, specifically for:

[0068] Based on the sum of the reconstruction loss data, the distribution loss data, the duration loss data, the acoustic feature loss data, the scene category loss data and the discrimination loss data, the machine learning model is trained to obtain the speech generation model.

[0069] In a fifth aspect, an embodiment of the present application provides an electronic device, which includes one or more processors; and a memory for storing one or more computer programs, so that when the one or more computer programs are executed by the one or more processors, the electronic device implements the speech generation method of the first aspect mentioned above.

[0070] In a sixth aspect, an embodiment of the present application provides a computer-readable storage medium, in which instructions are stored. When the computer-readable storage medium is run on a computer, the computer executes the speech generation method of the first aspect.

[0071] In a seventh aspect, an embodiment of the present application provides a computer program product, which includes a computer program or computer instructions. When the computer program or computer instructions are executed by a processor, the speech generation method of the first aspect is implemented.

[0072] In the technical solutions provided in some embodiments of the present application, in the process of generating speech corresponding to the text, the text features of the text to be processed are extracted, and the scene image features of the scene image to be processed are extracted. Then, based on the mapping relationship between the image features and the acoustic features, the scene image features are mapped to obtain the acoustic features corresponding to the scene image features. After that, based on the acoustic features corresponding to the text features and the scene image features, the target speech corresponding to the text to be processed is generated. Among them, the target speech matches the environmental sound effects of the scene indicated by the scene image to be processed. It can be seen that in the process of generating speech based on text, the reverberation effect brought by environmental factors is considered. Compared with the generated speech that only matches the input text, the generated speech can also match the acoustic characteristics of the visual scene, so that the generated speech can accurately reflect the given scene, improving the accuracy and effect of speech generation. While being able to convey the expected information, it can also include the environmental sound effects of the visual scene, making the generated speech more realistic, and to a certain extent, it can provide users with a more immersive and immersive auditory experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0073] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0074] Figure 1 It is a schematic diagram of the architecture of a speech generation system provided in an embodiment of the present application;

[0075] Figure 2 It is a flowchart of a speech generation method provided in an embodiment of the present application;

[0076] Figure 3 It is a structural schematic diagram of a speech generation method provided in an embodiment of the present application;

[0077] Figure 4 It is a timing diagram of a speech generation process provided by an embodiment of the present application;

[0078] Figure 5 is another flow chart of a speech generation method provided by an embodiment of the present application;

[0079] Figure 6 It is a timing diagram of a model training process provided by an embodiment of the present application;

[0080] Figure 7 It is a structural diagram of a scene classification model provided in an embodiment of the present application;

[0081] Figure 8 It is a schematic diagram of the change of spatial recognition error of a scene classification model during the training process provided by an embodiment of the present application;

[0082] Fig. 9 It is a heat map generated based on gradient weighted class activation mapping provided in an embodiment of the present application;

[0083] Fig.10 It is a structural schematic diagram of a speech generating device provided in an embodiment of the present application;

[0084] Fig.11 is a schematic diagram of the structure of another speech generating device provided in an embodiment of the present application;

[0085] Fig.12 It is a structural schematic diagram of a speech generating device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0086] It should be noted in advance that, in order to enable those skilled in the art to better understand the technical solutions proposed in the embodiments of the present application, the embodiments of the present application will be combined with one or more drawings to clearly and completely describe the implementation of the technical solutions proposed in the embodiments of the present application. In addition, the various drawings shown in the embodiments of the present application are only exemplary illustrations, and for example, the execution order of the various steps in the drawings can be adaptively adjusted according to the actual application scenario. In addition, in the embodiments of the present application, the block diagrams shown in the drawings are only functional entities and do not necessarily correspond to physically independent entities. That is, these functional entities can be implemented in software form, or these functional entities can be implemented in one or more hardware modules or integrated circuits, or these functional entities can be implemented in different networks and / or processor devices and / or microcontroller devices.

[0087] In the embodiments of the present application, the term "module" or "unit" refers to a computer program or a part of a computer program with a predetermined function, and works together with other related parts to achieve a predetermined goal, and can be implemented in whole or in part by using software, hardware (such as processing circuits or memories) or a combination thereof. Similarly, a processor (or multiple processors or memories) can be used to implement one or more modules or units. In addition, each module or unit can be part of an overall module or unit that includes the function of the module or unit.

[0088] It should be noted that the "multiple" mentioned in this article refers to two or more. "And / or" describes the association relationship of the associated objects, indicating that there can be three relationships. For example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. The character " / " generally indicates that the associated objects before and after are in an "or" relationship.

[0089] With the rapid development of artificial intelligence technology, driven by deep learning and neural network architecture, TTS technology has been widely used in various scenarios such as virtual assistants and barrier-free technology due to its ability to generate high-definition and high-naturalness voices, enriching the way of information transmission and improving the convenience of users to obtain information. At present, although TTS technology can imitate the way humans speak to a certain extent, such as imitating the speaker's emotional state, target audience, etc., to generate voices that can reflect specific emotions; for example, the rhythm, intonation and stress pattern of the generated voice are adjusted according to different situations. However, the algorithms or models used to generate voices in TTS technology are usually designed based on a neutral or non-reverberant environment, which makes the generated voice lack realism and spatial sense, and cannot generate voices that can accurately reflect a given scene, thus failing to meet the needs of various scenarios such as games and virtual reality (VR) that require a more immersive auditory experience.

[0090] Based on this, the embodiment of the present application provides a speech generation solution, which can generate speech in the scene specified by the scene image based on text and scene image, and consider the reverberation effect brought by the external environment in the process of generating speech. Compared with only matching the generated speech with the input text, the generated speech can also be matched with the acoustic characteristics of the visual scene, so as to accurately reflect the scene given by the user, and improve the accuracy and authenticity of speech generation. While ensuring the quality of speech, the speech is made more spatial, which is conducive to the audience to feel the sense of presence and immersion that matches the real world, and can adapt to complex and changeable application scenarios, and to a certain extent improve the effect of speech generation.

[0091] Based on the above description, please refer to Figure 1 , Figure 1 is a schematic diagram of the architecture of a speech generation system provided in an embodiment of the present application, such as Figure 1 As shown, the speech generation system includes a speech generation device 101, a data acquisition device 102, a model training device 103, and a downstream application device 104. The speech generation solution provided in the present application can be executed by the speech generation device 101 or the model training device 103. Among them, the speech generation device 101 can be directly or indirectly connected to the data acquisition device 102, the model training device 103, and the downstream application device 104 through wired or wireless means.

[0092] It should be noted that Figure 1The number and form of devices shown are for illustrative purposes only and do not constitute a limitation on the embodiments of the present application. In some embodiments, any two or more of the speech generation device 101, the data acquisition device 102, the model training device 103, and the downstream application device 104 may be the same electronic device. For example, the speech generation device 101, the data acquisition device 102, and the model training device 103 may be the same electronic device, and this application does not limit this. The following method embodiments of the present application are explained using the speech generation device 101, the data acquisition device 102, and the model training device 103 as the same electronic device as an example.

[0093] Among them, the speech generation device 101, the data acquisition device 102, the model training device 103 and the downstream application device 104 can specifically be terminal devices or servers. The terminal devices may include but are not limited to: smart phones (such as Android phones, IOS phones, etc.), tablet computers, portable personal computers, mobile Internet devices (Mobile Internet Devices, MID), intelligent voice interaction devices, smart home appliances, vehicle-mounted terminals, aircraft, wearable devices, etc., and the embodiments of the present application do not limit this; the server can be an independent physical server, or it can be a server cluster or distributed system composed of multiple physical servers, and it can also be a cloud server that provides cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (Content Delivery Network, CDN), and big data and artificial intelligence platforms and other basic cloud computing services, and the embodiments of the present application do not limit this.

[0094] In some embodiments, the speech generation device 101 may be deployed with an artificial intelligence driven software system, and may specifically deploy a speech generation model based on TTS technology to generate corresponding speech based on text and scene images.

[0095] In some embodiments, the data acquisition device 102 may be an electronic device for acquiring text and visual images, and the visual images may include images indicating scenes. The data acquisition device 102 may be an electronic device configured with a text input device and a capture device, and the text input device may be used to input text content that needs to be converted into speech, such as a keyboard, a touch screen, a microphone, etc. The capture device may be used to collect visual images of the real world, such as a camera, etc.

[0096] In some embodiments, the model training device 103 may be an electronic device for training a machine learning model, and the model training device 103 may obtain training sample data for training the machine learning model, such as the training text, training scene image, and training speech associated with the training text and training scene image in the embodiment of the present application. Furthermore, the model training device 103 may train the machine learning model based on the obtained training sample data.

[0097] In some embodiments, the downstream application device 104 can obtain the voice generated by the voice generating device 101, and perform subsequent applications based on the voice generated by the voice generating device 101, such as adding the voice generated by the voice generating device 101 to a game that includes an immersive scene, or adding the voice generated by the voice generating device 101 to a VR scene.

[0098] The general process of the speech generation method provided in this application is as follows:

[0099] The speech generation device 101 can obtain the text to be processed and the scene image to be processed from the data acquisition device 102, and then extract the text features of the text to be processed and extract the scene image features of the scene image to be processed. Furthermore, the speech generation device 101 can map the scene image features based on the mapping relationship between the image features and the acoustic features to obtain the acoustic features corresponding to the scene image features. Afterwards, the speech generation device 101 can generate a target speech corresponding to the text to be processed and matching the ambient sound effects of the scene indicated by the scene image to be processed based on the text features and the acoustic features corresponding to the scene image features.

[0100] Among them, the text to be processed and the scene image to be processed can be the text content and the visual image including the spatial scene obtained by the data acquisition device 102 from the Internet. The text to be processed and the scene image to be processed can also be generated by the data acquisition device 102 based on the text generation tool and the image generation tool respectively. The text generation tool can be a natural language processing model, etc., and the image generation tool can be a text-generated image model, etc., or it can be obtained by other means, and this application does not limit this. The target speech can be generated based on the text to be processed and the scene image to be processed by the speech generation model trained by the call model training device 103. After obtaining the target speech, the target speech can be used by the downstream application device 104 in a specific application scenario.

[0101] Among them, the model training device 103 can specifically obtain training text, training scene images, and training speech corresponding to the training text and matching the environmental sound effects of the scene indicated by the training scene image. Then, feature extraction processing is performed on the frequency domain representation information corresponding to the training speech to obtain posterior distribution parameters, and speech generation processing is performed based on the posterior distribution parameters to obtain generated speech. Afterwards, the model training device 103 can train the machine learning model based on the training text, training scene images, training speech, generated speech and posterior distribution parameters to obtain a speech generation model.

[0102] In one implementation, the above-mentioned text to be processed, scene image to be processed, target speech, visual training text, training scene image and training speech can all be stored in the blockchain, which can prevent such information from being tampered with. Blockchain is a new application mode of computer technologies such as distributed data storage, point-to-point transmission, consensus mechanism, encryption algorithm, etc. It is essentially a decentralized database, a string of data blocks generated by cryptographic methods, each of which contains a batch of network transaction information, which is used to verify the validity of its information (anti-counterfeiting) and generate the next block.

[0103] It can be understood that the speech generation system described in the embodiment of the present application is to more clearly illustrate the technical solution of the embodiment of the present application, and does not constitute a limitation on the technical solution provided in the embodiment of the present application. Ordinary technicians in this field can know that with the evolution of the system architecture and the emergence of new business scenarios, the technical solution provided in the embodiment of the present application is also applicable to similar technical problems.

[0104] Based on the above-mentioned speech generation system, the embodiment of the present application provides a speech generation method. The speech generation method described in the embodiment of the present application can be executed by an electronic device, which can be Figure 1 The speech generating device 101 in the speech generating system shown in FIG. Figure 2 , Figure 2 : is a flow chart of a speech generation method provided in an embodiment of the present application, the speech generation method comprises the following steps S201-S203:

[0105] S201: extracting text features of a text to be processed, and extracting scene image features of a scene image to be processed.

[0106] In the embodiment of the present application, the text to be processed may refer to the text content that needs to be converted into speech, and the text to be processed may be obtained from the Internet, or may be generated based on a text generation tool (such as a natural language processing model). Text features may also be referred to as text embeddings, which may refer to the representation obtained by converting the text to be processed into a series of vectors, which may capture the semantic and grammatical information in the text to be processed.

[0107] The scene image to be processed may refer to a visual image representing a specific environment, such as a bathroom environment, a tunnel environment, etc. The image to be processed may be obtained from the Internet, or may be generated based on an image generation tool (such as a text graph model). Scene image features may also be referred to as image embeddings, which refer to the representation obtained by converting the image to be processed into a series of vectors, which may include visual information in the scene image to be processed, such as color, texture, shape, etc. Since the scene image to be processed may represent a specific scene, the scene image features may be referred to as scene prompt features, etc., and this application does not limit this. It will be understood that text features and scene image features may be used to generate speech, i.e., to generate waveforms for representing speech.

[0108] In the embodiment of the present application, since the generation of speech requires text content and visual images, which belong to different modalities, the task of generating corresponding speech based on text content and visual images can be called a multimodal TTS task, that is, image-indicated immersive text-to-speech synthesis (Immersive Text-to-speech Synthesis, I 2 TTS) task.

[0109] Please also read Figure 3 , Figure 3 is a structural diagram of a speech generation method provided in an embodiment of the present application, such as Figure 3 As shown, a multi-modal text-to-speech system (Multi-modal TTS system) can be used to perform the I 2 TTS task. The input of the multimodal text-to-speech system consists of two parts: the text to be processed and the scene image to be processed. Figure 3 Four different scene images to be processed are used as examples for drawing and explanation, representing four different environments: bathroom, forest trail, tunnel, and auditorium. After being processed by the multimodal text-to-speech system, the multimodal text-to-speech system outputs a target speech, which can also be called immersive speech. The target speech can be a speech that can not only convey the text content of the text to be processed, but also a speech that matches the ambient sound effects of the environment represented by the scene image to be processed.

[0110] Among them, the text feature can be obtained by extracting the features of the symbol sequence based on the pronunciation elements corresponding to the text to be processed. The pronunciation element can also be referred to as phoneme, which refers to the smallest phonetic unit divided from the natural attributes of the voice, and is distinguished according to the pronunciation action. One pronunciation action forms a phoneme, such as vowel phonemes and consonant phonemes in English. The phoneme corresponding to the text to be processed refers to the pronunciation unit corresponding to each character in the text to be processed. For example, if the text to be processed is English, one letter in it can correspond to one phoneme, or two or more phonemes, or one phoneme is composed of multiple letters. The symbol sequence of phonemes can refer to a sequence arranged in the order of the phonemes corresponding to the text to be processed in the pronunciation, and represented by corresponding symbols. Among them, the symbol sequence can refer to an International Phonetic Alphabet (IPA) sequence, or a Pinyin character sequence, which is not limited in this application.

[0111] That is to say, after obtaining the text to be processed, the text to be processed can be preprocessed first, that is, the text to be processed is converted into a symbol sequence of corresponding phonemes, and then feature extraction processing is performed on the symbol sequence to convert it into a vector representation to obtain text features, which can be used to indicate at least one pronunciation element of the text to be processed.

[0112] In a possible implementation, the symbol sequence of the pronunciation elements corresponding to the text to be processed can be subjected to feature extraction processing by a machine learning model for processing text data or a component in the machine learning model. The model for processing text data or the component in the model can be, for example, a text encoder (Text Encoder). The text encoder can be composed of a multi-layer Transformer encoder or other types of neural networks, which is not limited in this application. The text encoder can learn the potential representation of the input text data and convert it into a feature representation form that is easier for subsequent model processing, such as a vector representation. In an embodiment of the present application, the text encoder can receive the text to be processed (the symbol sequence of the pronunciation elements corresponding to the text to be processed), and convert it into a vector representation to obtain text features.

[0113] In a possible implementation, feature extraction processing can be performed on the scene image to be processed by a machine learning model that processes image data or a component in the machine learning model. The model that processes image data or the component in the model can be, for example, an image encoder (Image Encoder). The image encoder can be, for example, a convolutional neural network (CNN), or a model based on contrastive language-image pre-training (CLIP), or it can be composed of other neural networks, which is not limited in this application. The image encoder can extract useful information from the input scene image to be processed, such as color, texture, shape, object position, etc., and encode this information into a feature vector or a feature map to obtain scene image features.

[0114] In some embodiments, since the target speech is generated based on text and images, in order to better combine the extracted text features and scene image features for processing, the image encoder in the embodiment of the present application can be an image encoder based on the CLIP model. The CLIP model can be composed of an image encoder and a text encoder, which can be used to convert images and texts into feature vectors respectively. The CLIP model is pre-trained based on a large amount of paired data of images and texts, and can learn the alignment relationship between images and texts. In an embodiment of the present application, the image encoder based on the CLIP model can be used to perform feature extraction processing on the image to obtain scene image features.

[0115] Furthermore, in order to better generate speech based on the environment in the scene image to be processed, the scene image features can be mapped based on the mapping relationship between image features and acoustic features to obtain acoustic features corresponding to the scene image features, and then the target speech can be generated based on the acoustic features.

[0116] S202: Based on the mapping relationship between the image features and the acoustic features, the scene image features are mapped to obtain acoustic features corresponding to the scene image features.

[0117] In the embodiments of the present application, image features refer to feature representation information extracted from an image, which can represent key information such as color, texture, shape, object position, etc. extracted from the image, and can be in the form of a vector. Acoustic features refer to feature representation information of a sound, which can represent parameters and attributes of the sound frequency characteristics, and can be in the form of a vector.

[0118] In an embodiment of the present application, acoustic features refer to parameters and properties of the sound frequency characteristics that can represent the ambient sound effects of a specific scene, and can be used to reflect the acoustic characteristics of this scene. For example, in a tunnel scene, because it is a long and narrow space, sound will produce obvious echoes when it propagates in the tunnel, and the reverberation time is long, tending to be low-frequency, making the sound in the tunnel sound dull and low. For another example, in a bathroom scene, because it is a small and closed space, sound will produce shorter echoes when it propagates in the bathroom, and the reverberation time is short, tending to be high-frequency, making the sound in the bathroom sound sharp and clear (compact). In other words, the characteristics of how sound sounds in a specific scene can refer to the acoustic characteristics of this scene, and the effect created by the sound of the environment in a specific scene can be called ambient sound.

[0119] Among them, the mapping relationship between image features and acoustic features can be understood as a cross-modal correspondence relationship. Certain features in the image (such as shape, color, texture, etc.) will be converted into acoustic features (such as pitch, timbre, etc.). Since image features and acoustic features have different properties, this mapping relationship is a nonlinear (complex) correspondence relationship. The mapping relationship between image features and acoustic features can be obtained by training a machine learning model or a component in a machine learning model for converting image features into acoustic features, or it can be a mathematical model established based on image features and acoustic features. This application does not limit this. The acoustic features corresponding to the scene image features may refer to features that can reflect the acoustic characteristics of the environment in the scene image to be processed, and may indicate the parameters and properties of the sound frequency characteristics of the ambient sound effects of the scene indicated by the processing scene image.

[0120] In one possible implementation, based on the mapping relationship between image features and acoustic features, the scene image features are mapped to obtain the acoustic features corresponding to the scene image features. Specifically, the scene image features may be firstly subjected to a first linear transformation to obtain transformed scene image features, and then the transformed scene image features are subjected to a nonlinear mapping based on a set activation function, and then the image features obtained by the nonlinear mapping are subjected to a second linear transformation to obtain the acoustic features corresponding to the scene image features.

[0121] Among them, the first linear transformation processing can refer to a linear combination of features (such as scene image features) to form a new feature representation, which can be specifically implemented through a linear function, and the adjusted features obtained are the transformed scene image features. The activation function is a nonlinear function, which is used to introduce nonlinearity and realize complex mapping relationships. The nonlinear mapping processing can be understood as treating the activation function as a complex function. The dependent variable can be calculated based on the independent variable, that is, the image feature obtained after the nonlinear mapping processing. The second linear transformation processing is similar to the first linear transformation processing, which is a linear combination of the image features obtained after the nonlinear mapping processing to form a new feature representation, which can also be implemented through a linear function to obtain the acoustic features corresponding to the scene image features.

[0122] It should be noted that the parameters of the first linear transformation process and the second linear transformation process (such as the parameters of the linear function) can be the same or different, and this application does not limit this. This application uses the example of performing two transformation processes on scene image features and one nonlinear mapping process as an example. The acoustic features corresponding to the scene image features can also be obtained by performing multiple transformation processes and multiple nonlinear mapping processes on the scene image features, and this application does not limit this either.

[0123] In one possible implementation, the mapping relationship between image features and acoustic features can be obtained by training a machine learning model, which can be called an adapter. The model can also be used as a component of other machine learning models to convert image features into acoustic features that can be used for subsequent speech generation. The adapter model can be composed of at least one Gaussian Error Linear Unit (GELU) and two or more fully connected layers (FC). GELU is an activation function, which can be understood as a mathematical function that represents a complex mapping relationship and is used to introduce nonlinear relationships to deal with complex problems. FC can integrate and transform the input features to facilitate subsequent nonlinear mapping processing, such as nonlinear mapping processing through GELU, or integrating and transforming the features obtained through nonlinear mapping processing.

[0124] For the convenience of description, the adapter model is taken as an example consisting of a GELU and two FCs, and the GELU can be located between the two FCs. The input of the adapter model can be an image feature, such as a scene image feature, and the output of the adapter model can be an acoustic feature, such as an acoustic feature corresponding to the scene image feature. After the scene image feature is input into the adapter model, the first FC in the adapter model can be used to perform a first linear transformation on the input vector (scene image feature), such as applying a linear transformation on the input vector, and the linear transformation can be implemented through a weight matrix and a bias term, thereby obtaining a transformed scene image feature. Then, the transformed scene image feature is passed to the GELU, and the GELU can perform a nonlinear mapping process on the transformed scene image feature. The parameters of the GELU can be obtained through training and can be used to perform the task of mapping the image feature to the acoustic feature, thereby obtaining the image feature obtained through the nonlinear mapping process. Afterwards, the image feature obtained through the nonlinear mapping process is passed to the second FC, and the second FC performs a second linear transformation on the image feature obtained through the nonlinear mapping process, which can also be implemented based on the weight matrix and the bias term, thereby obtaining the acoustic feature corresponding to the scene image feature.

[0125] It is understandable that the model parameters (such as weight matrix and bias term) of the first FC and the second FC can be the same or different, and this application does not limit this. Thus, through the adapter model, the visual features of the scene can be effectively integrated into the process of speech generation.

[0126] S203: Based on the acoustic features corresponding to the text features and the scene image features, generate a target speech corresponding to the text to be processed and matching the ambient sound effects of the scene indicated by the scene image to be processed.

[0127] In an embodiment of the present application, the target voice is a generated voice, which can be a waveform for representing a voice, and can be understood as a voice signal that can be output by a playback device such as a speaker. Corresponding to the text to be processed means that the target voice can convey the content of the text to be processed, and can also be understood as the sound of reading the text to be processed. The scene indicated by the scene image to be processed refers to a specific scene represented in the scene image to be processed, such as a tunnel scene and a bathroom scene. The ambient sound effect refers to the effect created by the sound in the environment. The ambient sound effects of different environments can reflect the acoustic characteristics of different scenes. Based on different ambient sound effects, the audience can feel a specific environmental atmosphere. Matching the ambient sound effects of the scene indicated by the scene image to be processed means that the target voice can reflect this specific environment, so that the audience can not only receive the information of the text to be processed, but also feel that the speaker is in the scene (environment) indicated by the scene image to be processed.

[0128] That is to say, the target voice can not only convey the expected information, but also include the ambient sound effects related to the visual scene, so that the target voice is more realistic and immersive than the voice corresponding to the text only, allowing the listener to feel the atmosphere of the scene indicated by the scene image to be processed.

[0129] Specifically, based on the acoustic features corresponding to the text features and the scene image features, the specific process of generating the target speech corresponding to the text to be processed and matching the ambient sound effects of the scene indicated by the scene image to be processed can be: first, based on the mapping relationship between the pronunciation elements and the duration, the duration of the pronunciation elements indicated by the text features is mapped to obtain the duration prediction feature. Then, the text features are subjected to a third linear transformation to obtain the transformed text features, and the transformed text features are converted into prior distribution parameters according to the set conversion function. After that, the target speech is generated based on the duration prediction feature, the prior distribution parameters and the acoustic features corresponding to the scene image features.

[0130] Among them, the text features are processed by two branches respectively, and in one branch, the text features are used to predict the duration. In this branch, the pronunciation element (phoneme) refers to the smallest speech unit in the speech corresponding to the text, and the duration refers to the duration corresponding to the phoneme in the speech. The mapping relationship between the phoneme and the duration can also be understood as a cross-modal correspondence. The phoneme (vector representation) indicated by the text feature, such as at least one phoneme of the text to be processed, will be converted into a vector representation of the duration. In the vector representation of the duration, each element can be used to indicate the duration of a phoneme in the generated speech, and the unit is the number of frames. This mapping relationship is also a nonlinear correspondence. The mapping relationship between the phoneme and the duration can be obtained by training a machine learning model or a component in a machine learning model for predicting the duration of the phoneme, or it can be obtained by a mathematical model established based on text features and duration prediction features, and this application does not limit this.

[0131] In a possible implementation, the mapping relationship between phonemes and durations can be obtained by training a machine learning model, which can be called a duration predictor. The duration predictor can predict the waveform duration of the corresponding speech based on the input text features. It is understandable that the duration of each word spoken by the same person in a sentence may be different, and the duration predictor can be used to handle such changes, making the generated speech more natural and expressive.

[0132] The input of the duration predictor may be the text feature and noise, and the noise input to the duration predictor may be, for example, a vector representation of Gaussian noise. The output of the duration predictor may be a duration prediction feature, which is a vector representation of duration and contains information about the duration of the phoneme indicated by the text feature. For example, if the duration prediction feature is [2,2,1], it may indicate that the duration of the first phoneme is 2 frames, the duration of the second phoneme is 2 frames, and the duration of the third phoneme is 1 frame. The model parameters of the neural network structure of the duration predictor may be obtained by modeling the speaker's speech features during training.

[0133] Among them, in another branch, text features are used to generate prior distribution parameters. In this branch, the third linear transformation process is similar to the first linear transformation process and the second linear transformation process, and may refer to a linear combination of text features to form a new feature representation, which may be specifically implemented through a linear function, and the adjusted features are the transformed text features, so as to facilitate subsequent conversion into prior distribution parameters. The set conversion function may refer to a function that converts a vector representation (transformed text features) into prior distribution parameters, such as an exponential function. The prior distribution parameters may include a prior mean parameter μ and a prior variance parameter σ2. The prior mean parameter and the prior variance parameter may define a prior distribution, such as a normal distribution (Gaussian distribution) N(μ,σ 2 ).

[0134] In some embodiments, the third linear transformation processing can be performed by a mapping network (Projection) in the model. The mapping network can be a linear projection layer (Linear Projection Layer) immediately after the text encoder. The linear projection layer can also be understood as FC, which is used for linear transformation processing.

[0135] In some embodiments, when converting the transformed text features into a priori mean parameters, the conversion function may be, for example, a function in which the independent variable is equal to the dependent variable, that is, the transformed text features are taken as the mean (mean vector). When converting the transformed text features into a priori variance parameters, the conversion function may be, for example, an exponential function, and then the variance (variance vector) may be calculated based on the transformed text features and the set exponential function. It should be noted that the above conversion function is only an example, and may also be other functions, or functions corresponding to other neural networks, and this application does not limit this.

[0136] It can be understood that the prior distribution can refer to the distribution followed by the features required for generating speech in the absence of additional information, that is, before generating speech, an assumption is made about what kind of features can generate speech, so that speech can be generated based on the prior distribution. Since the corresponding prior distribution can be generated based on the text encoder and the linear projection layer, the model composed of the text encoder and the linear projection layer can be called a prior encoder.

[0137] Furthermore, the target speech is generated based on the duration prediction feature and prior distribution parameters obtained by the two branch processes, as well as the acoustic features corresponding to the scene image features. Specifically, based on the duration prediction feature, sampling processing can be performed in the prior distribution defined by the prior mean parameter and the prior variance parameter to obtain the first sampling feature. Among them, the prior distribution can be a normal distribution defined by the prior mean parameter and the prior variance parameter, which can also be called a Gaussian distribution. Sampling processing refers to samples extracted from a normal distribution, and sampling processing based on the duration prediction feature refers to extracting a corresponding number of samples from a normal distribution based on the dimension of the duration prediction feature. The vector representation obtained by the sampling processing can be called a first sampling feature, and the first sampling feature can also be called a latent variable, a latent vector, etc. It can be understood that since the sampling processing is performed based on the dimension of the duration prediction feature, it can ensure to a certain extent that the generated speech matches the pronunciation duration and rhythm of the text to be processed.

[0138] Specifically, the specific process of sampling processing based on the duration prediction feature can be: first, based on each duration feature element in the duration prediction feature, repeat the feature element at the corresponding position in the transformed text feature for the number of times indicated by each duration feature element to obtain an extended text feature. Then, based on the extended text feature, sampling processing is performed from the prior distribution to obtain the first sampling feature. Among them, since the duration prediction feature indicates the duration corresponding to each phoneme, the text feature is also used to indicate each phoneme, and the third linear transformation processing performed on the text feature does not change the dimension of the text feature, the dimension of the duration prediction feature is the same as the dimension of the transformed text feature, which can be understood as the same dimension of the vector.

[0139] Among them, each duration feature element refers to the feature element in the duration prediction feature (vector representation), for example, the duration prediction feature is [t1, t2, t3], and the duration feature elements are t1, t2, t3. Similarly, the feature elements in the transformed text feature are also the feature elements in the pointing quantity representation, such as the transformed text feature is [h1, h2, h3], and the duration feature elements are h1, h2, h3. Due to the same dimension, the duration feature elements in the duration prediction feature can correspond to the feature elements in the transformed text feature in position. The extended text feature refers to the feature obtained by repeating the feature elements of the transformed text feature. Exemplarily, taking the duration prediction feature as [1,2,3] and the transformed text feature as [h1,h2,h3] as an example, based on each duration feature element in the duration prediction feature, the feature elements at the corresponding position in the transformed text feature are repeated the number of times indicated by the each duration feature element, which can be understood as repeating h1 once, h2 twice, and h3 three times, to obtain the extended text feature [h1,h2,h2,h3,h3,h3].

[0140] Thus, based on the extended text feature, the dimension of the first sampling feature obtained by sampling from the prior distribution is the same as the dimension of the extended text feature. That is to say, firstly based on the dimension of the duration prediction feature, the number of samples to be sampled is determined, and then samples are extracted from the prior distribution based on the number of samples to obtain the first sampling feature. For example, for the extended text feature [h1, h2, h2, h3, h3, h3], 6 samples can be extracted from the prior distribution to obtain the first sampling feature (feature sequence). It can be understood that the above sampling process is only a simple principle example, which can be performed by a related neural network, which is a more complex process, and the present application does not limit the structure of the neural network.

[0141] In a possible implementation, after obtaining the first sampling feature, the target speech can be generated based on the acoustic features corresponding to the first sampling feature and the scene image feature. Specifically, the speech signal feature corresponding to the first sampling feature and the acoustic features corresponding to the scene image feature can be fused to obtain the target speech. The fusion process can be performed by a machine learning model for generating a speech waveform. The machine learning model can be, for example, a decoder, also known as a vocoder. The decoder is a neural network model specially designed for generating speech waveforms. It generates the final speech waveform based on the input feature representation. For example, it can be a high-fidelity generative adversarial network (High Fidelity GAN).

[0142] Among them, the input of the decoder is the acoustic features corresponding to the first sampling features and the scene image features, and the output is a waveform used to represent the target speech. The decoder can perform decoding processing, audio feature generation, waveform generation and other processing on the acoustic features corresponding to the first sampling features and the scene image features, so as to obtain the target speech (waveform).

[0143] In some embodiments, the fusion processing performed by the decoder may include processing performed by convolutional layers, residual connections, jump connections, etc., which can be specifically determined based on the model structure of the decoder. This application does not limit the model structure of the decoder.

[0144] In another possible implementation, the first sampling feature may be mapped based on the mapping relationship between the sampling feature and the speech signal feature to obtain the speech signal feature corresponding to the first sampling feature, and then the speech signal feature corresponding to the first sampling feature and the acoustic feature corresponding to the scene image feature may be fused to obtain the target speech.

[0145] Among them, the mapping relationship between the sampling features and the speech signal features can be understood as a cross-modal correspondence, which can map the sampling features to feature controls that are closer to the target speech characteristics. This mapping relationship may involve a series of linear transformation processing and nonlinear transformation processing, which can be used to capture the complex relationship between the sampling features and the speech signal features, and can map out a more refined feature representation that is more in line with the characteristics of the speech signal. The mapping relationship between the sampling features and the speech signal features can be obtained by training a machine learning model or a component in a machine learning model, or it can be a mathematical model established based on text features and duration prediction features, and this application does not limit this.

[0146] In some embodiments, the mapping relationship between the sampling features and the speech signal features can be learned based on the normalized flow model of the transformer. The normalized flow model can be used to transform from one distribution to another, and its execution process is reversible. During the reasoning process, since the sampling features obtained based on the text features are mapped to the speech signal features, the sampling features can be optimized and transformed to retain the key information of the original sampling features while incorporating details about the characteristics of the speech signal, which can be understood as making simple features more complex and delicate.

[0147] Furthermore, the process of fusing the speech signal features corresponding to the first sampling features and the acoustic features corresponding to the scene image features to obtain the target speech can be similar to the process of directly fusing the acoustic features corresponding to the first sampling features and the scene image features to obtain the target speech, except that the features input to the decoder are changed from the first sampling features to the refined features, i.e., the speech signal features corresponding to the first sampling features, thereby obtaining the target speech (waveform) output by the decoder.

[0148] Below, the process of generating the target speech based on the text to be processed and the scene image to be processed by the machine learning model is described in detail.

[0149] Please also read Figure 4 , Figure 4 is a timing diagram of a speech generation process provided by an embodiment of the present application. Figure 4 As shown, the machine learning model can be called a speech generation model, and the speech generation model can include a scene prompt encoder (Scene Prompt Encoder, SPE) and a TTS backbone network (TTS backbone).

[0150] Among them, SPE may include an image encoder and an adapter, and the image encoder may be, for example, an image encoder based on the Clip model, and the adapter may be, for example, composed of a GELU and two FCs. SPE can be used to integrate visual scene information into the process of speech generation to control the generation of speech. The TTS backbone network may be an end-to-end speech synthesis (Variational Inference with Adversarial Learning for End-to-End Text-to-Speech, VITS) model based on conditional variational autoencoder and adversarial learning, which can be used to control the generation of speech based on text to output sounds containing text semantics. The VITS model may include a priori encoder, a duration predictor, a normalization stream, and a decoder. The priori encoder may be composed of a text encoder and a linear projection layer.

[0151] Specifically, the scene image to be processed I i and the text to be processed t i Input into the speech generation model, and extract the text to be processed through the text encoder i The text features are extracted by the image encoder of the scene prompt encoder in the speech generation model. i Then, the scene image feature is input into the adapter to obtain the acoustic feature I corresponding to the scene image feature output by the adapter. e, the acoustic feature can also be called acoustic space embedding, which can be close to the feature representation of a scene category. Thus, the SPE processing flow ends, and then the acoustic feature I corresponding to the scene image feature e It can be input into the TTS backbone network and can be used to integrate the scene perception information obtained based on the scene image to be processed into the TTS backbone network to generate the speech process.

[0152] Specifically, the text features can be processed in two different ways. One branch can be to input it into the linear projection layer for mapping to obtain the transformed text features, and then convert the transformed text features into prior distribution parameters based on the set conversion function, such as the mean μ θ and variance σ θ Another branch can be to input it into the duration predictor to obtain the duration prediction feature d predicted by the duration predictor. Then, based on the duration prediction feature d, the mean μ θ and variance σ θ The first sampling feature is obtained by sampling the prior distribution defined in the decoder. After that, the first sampling feature and the acoustic features corresponding to the scene image features can be input into the decoder to obtain the target speech. In this way, the environmental sound effects (acoustic features corresponding to the scene image features) are integrated into the process of speech generation, rather than being regarded as an optional post-processing effect.

[0153] In some embodiments, the first sampling feature may be first input into the normalized stream to obtain the optimized first sampling feature, that is, the speech signal feature, and then the acoustic features corresponding to the speech signal feature and the scene image feature are input into the decoder to obtain the target speech.

[0154] In the technical solutions provided in some embodiments of the present application, in the process of generating speech corresponding to the text, the text features of the text to be processed are extracted, and the scene image features of the scene image to be processed are extracted. Then, based on the mapping relationship between the image features and the acoustic features, the scene image features are mapped to obtain the acoustic features corresponding to the scene image features. After that, based on the acoustic features corresponding to the text features and the scene image features, the target speech corresponding to the text to be processed is generated. Among them, the target speech matches the environmental sound effects of the scene indicated by the scene image to be processed. It can be seen that in the process of generating speech based on text, the reverberation effect brought by environmental factors is considered. Compared with the generated speech that only matches the input text, the generated speech can also match the acoustic characteristics of the visual scene, so that the generated speech can accurately reflect the given scene, improving the accuracy and effect of speech generation. While being able to convey the expected information, it can also include the environmental sound effects of the visual scene, making the generated speech more realistic, and to a certain extent, it can provide users with a more immersive and immersive auditory experience.

[0155] The embodiment of the present application also provides a method for processing a speech generation model. The method for processing a speech generation model described in the embodiment of the present application can be executed by an electronic device, which can be Figure 1 The speech generation model processing system shown in FIG. 1 is a speech generation device 101 or a model training device 103. Figure 5 , Figure 5 4 is another flow chart of a speech generation method provided in an embodiment of the present application, the speech generation method comprising the following steps S501-S504:

[0156] S501, obtaining a training text, a training scene image, and a training voice corresponding to the training text and matching the environmental sound effects of the scene indicated by the training scene image.

[0157] In the embodiment of the present application, the training text, the training scene image and the training voice respectively refer to the text content, the scene image and the voice used to train the speech generation model (machine learning model) to be trained. The voice can refer to a waveform that can represent the voice, that is, a voice signal that can be output by a playback device such as a speaker. Among them, the training text corresponds to the training voice, which means that the training voice outputs the text content of the training text, which is the sound of the training text. The training scene image refers to a visual image representing a specific environment. The specific environment can be, for example, an auditorium environment, a forest environment, a tunnel environment, a bathroom environment, etc. The environmental sound effect of the scene indicated by the training scene image refers to the effect created by the sound in the specific environment represented by the training scene image. The environmental sound effects of different environments can reflect the acoustic characteristics of different scenes. The training voice matches the environmental sound effect of the scene indicated by the training scene image, which means that the training voice can reflect this specific environment. In other words, the training voice can be both the text content of the training text and the environmental sound effect of the visual scene.

[0158] Among them, the training text, training scene images and training speech can be obtained from the Internet, such as from an existing data set. The training text, training scene images and training speech can also be input by the user, such as the training text input by the user through an input device, the training scene images collected by an image acquisition device (such as a camera), and the training speech collected by a speech acquisition device (such as a recorder). The training text, training scene images and training speech can also be generated based on corresponding generation tools, such as the training text can be generated based on a text generation tool (such as a natural language processing model), the training scene image can be generated based on an image generation tool (such as a text image model), and the training speech can be based on a speech synthesis tool, synthesizing the speech corresponding to the training text and the environmental sound effects corresponding to the training scene image. It can also be obtained in other ways, and this application does not limit this.

[0159] In a possible implementation, in the process of obtaining a training voice corresponding to a training text and matching the environmental sound effects of the scene indicated by a training scene image, the environmental sound effects corresponding to the training scene image and the voice corresponding to the training text may be obtained first. The environmental sound effects corresponding to the training scene image may be understood as sound data used to synthesize the reverberant of the scene, and may refer to the room impulse response (RIR) corresponding to the training scene image. RIR is the sound wave response generated by the space when a very short sound pulse is played in this space. The voice corresponding to the training text may refer to the voice used to output the text content of the training text, without any environmental reverberation, and is clean speech data. The above-mentioned environmental sound effects and clean voice may be obtained from the Internet, such as the clean voice may be obtained from the LJSpeech data set, and the environmental sound effects may be obtained from the Image2Reverb data set.

[0160] Furthermore, convolution processing can be performed based on the speech and environmental sound effects corresponding to the training text to obtain the training speech. Convolution processing refers to a mathematical operation method that can describe the process of interaction between two functions. In the process of speech synthesis, convolution processing of two speech can refer to convolution operation of one speech (which can be regarded as a function) with another speech (which can also be regarded as another function) to obtain the training speech. Therefore, by convolving the pure speech with the RIR representing a specific environment, the reverberation effect that the pure speech should have in the specific environment can be simulated, so that the obtained training speech can sound more like it is recorded in a real environment, which increases the sense of reality, which is the training goal of the speech generation model.

[0161] In another possible implementation, the training speech can be obtained first, and then the training text and training scene image can be obtained based on the training speech. Specifically, multiple clean speech data can be obtained, and multiple environmental sound effects in different specific scenes can be obtained, such as obtaining multiple clean speech data from the LJSpeech data set, and obtaining multiple environmental sound effects (RIR) from the Image2Reverb data set. Then, random convolution processing is performed based on multiple clean speech data and multiple environmental sound effects, that is, any clean speech data can be convolved with any environmental sound effect to obtain multiple training speech. Then, based on the text content output in the clean speech data, the training text is obtained, and based on the specific scene corresponding to the environmental sound effect subjected to the convolution processing, the corresponding scene image can be generated in the image generation tool to obtain the training scene image.

[0162] Since the training speech is obtained by convolution processing, the training speech can also be called convolutional speech samples. It can be understood that the training speech corresponds to the visual scene depicted by the training scene image, that is, each training speech can correspond to a scene category, and the training scene image can also correspond to the scene category. The scene category can also refer to the reverberation category of the speech corresponding to the scene. The scene category (reverberation category) is the benchmark for training, that is, the scene category label for training.

[0163] S502: Perform feature extraction processing on the frequency domain representation information corresponding to the training speech to obtain posterior distribution parameters.

[0164] In the embodiment of the present application, the frequency domain representation information corresponding to the training speech may refer to the mel-spectrogram, linear spectrum, etc. of the training speech. Mel-spectrogram refers to a method of converting an audio signal into a visual representation, and linear spectrum refers to a spectrum composed of discrete frequency components. Both the mel-spectrogram and the linear spectrum may be obtained by performing Fourier transform and other related processing on the speech. For example, the mel-spectrogram may be obtained by converting the converted spectrum graph to the mel scale after performing short-time Fourier transform (STFT) on the speech. Since the mel-spectrogram is expressed in the form of an image, the mel-spectrogram may also be called a mel-spectrogram.

[0165] In an embodiment of the present application, feature extraction processing performed on frequency domain representation information may refer to processing for extracting features such as fundamental frequency, resonance peaks, etc. in the spectral representation information, which aims to capture local or global patterns in the training speech, and can be used to estimate the connection between actual speech samples and intermediate features to facilitate subsequent training.

[0166] In an embodiment of the present application, the posterior distribution parameters may include a posterior mean parameter and a posterior variance parameter, and the posterior mean parameter and the posterior variance parameter may define a posterior distribution, such as a normal distribution (Gaussian distribution). The posterior distribution refers to the probability distribution calculated after obtaining the observed data (training speech), and the posterior distribution may reflect the most likely value of a parameter or variable after the given observed data, and the parameter or variable may refer to a latent variable, that is, an intermediate representation of the model during the processing. The posterior distribution defined by the posterior mean parameter and the posterior variance parameter may describe the connection between the actual speech (training speech) and the latent variable. By estimating the posterior distribution, the model can better capture the detailed features in the training speech, so that when generating new speech, it can be closer to the characteristics of the actual speech (such as the training speech).

[0167] In a possible implementation, feature extraction processing is performed on the frequency domain representation information corresponding to the training speech, which can be performed by a spectral encoder (Spectrogram Encoder) in the speech generation model to be trained. Since the spectral encoder is used to generate a posterior distribution, the spectral encoder can also be called a posterior encoder. The input of the spectral encoder is an actual speech sample, that is, the frequency domain representation information corresponding to the training speech, such as the Mel spectrum of the training speech, which can better represent the spectral characteristics of the speech signal. Furthermore, the spectral encoder can extract the features of the frequency domain representation information corresponding to the training speech through a multi-layer CNN or other network structure to capture the local and global patterns in the training speech. After feature extraction processing, the spectral encoder can generate an intermediate representation, which can be further processed, such as through linear transformation processing, to estimate the posterior distribution parameters.

[0168] S503: Perform speech generation processing based on the posterior distribution parameters to obtain generated speech.

[0169] In an embodiment of the present application, the posterior distribution parameters may include a posterior mean parameter and a posterior variance parameter, and the posterior mean parameter and the posterior variance parameter may define a normal distribution (Gaussian distribution). Different from the inference stage, in the process of training the speech generation model to be trained, speech generation processing may be performed directly based on the posterior distribution parameters to obtain generated speech, which may be a speech corresponding to the training speech, and one of the goals of the training may be to make the generated speech closer to the training speech.

[0170] In a possible implementation, speech generation processing is performed based on the posterior distribution parameters. Specifically, sampling processing may be performed in the posterior distribution defined by the posterior mean parameter and the posterior variance parameter to obtain a second sampling feature, and then speech generation processing is performed based on the second sampling feature to obtain generated speech. The sampling process refers to extracting a certain number of samples from a normal distribution, that is, extracting a certain number of samples from a normal distribution (Gaussian distribution) defined by the posterior mean parameter and the posterior variance parameter. Specifically, the speech generation model to be trained may be obtained from the posterior distribution through a reparameterization trick. The reparameterization trick may decompose the sampling process into the sampling of differentiable parameters by introducing a differentiable function to the random variable. Gradient information may be retained during the sampling process so that the speech generation model to be trained may be trained using a gradient-based optimization algorithm.

[0171] Furthermore, the second sampling feature obtained by the sampling process can be input into a model for generating a speech waveform, which can be a decoder in the speech generation model to be processed, and can also be called a vocoder, for example Figure 4 The decoder shown in FIG. The input of the decoder is the second sampling feature, and the output is the generated speech (the waveform representing the generated speech), that is, the decoder converts the second sampling feature into a speech waveform.

[0172] In some embodiments, in order to reduce the amount of data calculation during the model training process, the second sampled feature can be sliced ​​and only a portion of the slices can be selected to generate the speech waveform. In other words, the input of the decoder can be only a portion of the slice features in the second sampled feature, and the decoder converts the portion of the slice features into a speech waveform to obtain the generated speech.

[0173] Therefore, the speech generation model to be trained (machine learning model) can be trained based on the training text, training scene images, training speech, generated speech, and posterior distribution parameters to obtain a speech generation model.

[0174] S504. Based on the training text, the training scene image, the training speech, the generated speech and the posterior distribution parameters, the machine learning model is trained to obtain a speech generation model.

[0175] In the embodiment of the present application, the machine learning model may refer to a speech generation model to be trained, and the model obtained by training is the speech generation model. In order to train the machine learning model by making the training speech correspond to the training text, the training text may be processed, and the processed features may be aligned with the processed features of the training speech, so that the model learns the correspondence between the text and the speech.

[0176] Specifically, the text features of the training text can be extracted to obtain the training text features. The training text features can then be processed in two branches, one branch can perform a fourth linear transformation on the training text features to obtain training transformation text features, and convert the training transformation text features into prior distribution parameters according to a set conversion function. The other branch can map the duration of the pronunciation element indicated by the training text feature based on the mapping relationship between the pronunciation element and the duration to obtain a training duration prediction feature. Then, based on the training duration prediction feature, sampling processing can be performed in the training prior distribution defined by the training prior distribution parameters to obtain a third sampling feature. Thus, the second sampling feature obtained by sampling based on the posterior distribution and the second sampling feature obtained by sampling based on the prior distribution can be aligned to obtain an alignment estimation feature.

[0177] Among them, the principle of performing two-branch processing on the training text features extracted based on the training text to obtain the third sampling features is the same as the principle of performing two-branch processing on the text features extracted based on the text to be processed in the reasoning stage to obtain the first sampling features. For details, please refer to the relevant description in the reasoning stage, which will not be repeated here.

[0178] After obtaining the third sampling feature, the second sampling feature and the third sampling feature are aligned, which means that a monotonic alignment relationship is established between the text feature and the speech feature. This alignment relationship can be used to guide the model to map the text feature to the corresponding speech feature when generating speech. The so-called monotonic alignment relationship means that a feature element in the third sampling feature can be aligned with one or more continuous feature elements in the second sampling feature. Therefore, it can be understood that the alignment process is used to determine the duration of each phoneme indicated by the training text feature in the training speech, and the obtained alignment estimation feature is a feature used to indicate the duration (one or more continuous frames) of each phoneme indicated by the training text in the training speech. The alignment estimation feature can be a two-dimensional matrix.

[0179] In some embodiments, during the training process, since the second sampling feature is relatively complex, the feature obtained by processing based on the training text feature, such as the third sampling feature, is relatively simple, then the second sampling feature can be mapped based on the mapping relationship between the speech signal feature and the sampling feature to obtain the sampling feature after mapping. Then, the sampling feature after mapping and the third sampling feature are aligned to obtain the alignment estimation feature. Among them, the mapping relationship between the speech signal feature and the sampling feature can be learned based on the normalized flow model of the transformer. During the reasoning process, the normalized flow model can be a reverse processing process to make simple features more complex and delicate. During the training process, the normalized flow model can be a forward processing process to make complex features simpler, so as to facilitate alignment processing.

[0180] Among them, the alignment process can be performed by a monotone alignment search (MAS) model in a machine learning model (a speech generation model to be trained), which can be located between a priori encoder and a posteriori encoder, and is used to align the sampling features generated by the two encoders. Specifically, the MAS model can initialize an alignment matrix, and then the MAS module can traverse all possible alignment methods, calculate the score of each alignment method based on a similarity metric, and update the alignment matrix based on the score. Afterwards, the MAS model can determine the optimal alignment method, which can be the alignment method with the highest score, which can represent the best correspondence between the two sampling features, thereby obtaining an alignment estimation feature.

[0181] Furthermore, the machine learning model can be trained based on the training duration prediction features, training prior distribution, training scene images, training speech, generated speech, posterior distribution and alignment estimation features to obtain a speech generation model. In order to make the trained speech generation model also match the environmental sound effects of a specific scene, the training images need to be further processed. Specifically, the training scene image features of the training scene image can be first extracted, and then based on the mapping relationship between the image features and the acoustic features, the training scene image features can be mapped to obtain the acoustic features corresponding to the training scene image features. Thus, the machine learning model can be trained based on the training duration prediction features, training prior distribution, training speech, generated speech, posterior distribution, alignment estimation features and acoustic features corresponding to the training scene image features to obtain a speech generation model.

[0182] Among them, based on the mapping relationship between image features and acoustic features, the training scene image features extracted based on the training scene images are mapped to obtain the acoustic features corresponding to the training scene image features. This principle is the same as the principle of mapping the scene image features extracted based on the scene image to be processed based on the mapping relationship between image features and acoustic features in the inference stage to obtain the acoustic features corresponding to the scene image features. For details, please refer to the relevant description in the inference stage, which will not be repeated here.

[0183] In a possible implementation, in the process of training a machine learning model to obtain a speech generation model based on acoustic features corresponding to training duration prediction features, training prior distribution, training speech, generated speech, posterior distribution, alignment estimation features, and training scene image features, the loss data may include the following parts:

[0184] The first part is based on the loss data generated between the generated speech and the training speech, so as to make the generated speech close to the training speech. Specifically, the reconstruction loss data can be generated based on the difference between the frequency domain representation information corresponding to the generated speech and the frequency domain representation information corresponding to the training speech. For example, the loss data can be generated based on the mean absolute error (MAE) between the Mel spectrum of the generated speech and the Mel spectrum of the training speech.

[0185] The second part is based on the loss data generated between the prior distribution and the posterior distribution output by the prior encoder and the posterior encoder respectively, so that the prior distribution output by the prior encoder is close to the posterior distribution. Specifically, the duration loss data can be generated based on the difference between the training duration prediction feature and the alignment estimation feature. It should be noted that since the TTS backbone network of the speech generation model to be trained is a VITS model, which is a model combined with variational reasoning, variational reasoning can be used to process approximate calculations involving hidden variable problems. Therefore, loss data can be generated based on the difference between the prior distribution and the posterior distribution. For example, loss data can be generated based on the relative entropy information divergence (Kullback-Leibler Divergence, KL divergence) between the training prior distribution and the posterior distribution.

[0186] The third part is based on the loss function generated between the duration prediction feature output by the duration predictor and the alignment estimation feature, so that the duration prediction feature obtained by the duration predictor mapping the duration of the phoneme indicated by the text feature is closer to the alignment estimation feature obtained by the alignment process, and specifically, the duration loss data can be generated based on the difference between the training duration prediction feature and the alignment estimation feature. For example, MAE loss data can be generated based on the training duration prediction feature and the alignment estimation feature.

[0187] The fourth part is based on the loss data generated between the acoustic features to which the adapter adapts the scene image features and the scene category labels, so that the acoustic features output by the adapter are closer to a specific scene category (reverberation category). Specifically, the acoustic feature loss data can be generated based on the difference between the acoustic features corresponding to the training scene image features and the scene category labels corresponding to the training scene images. For example, cross entropy (CE) loss data can be generated based on the acoustic features corresponding to the training scene image features and the scene category labels corresponding to the training scene images. Specifically, based on the difference between the acoustic features corresponding to the training scene image features and the scene category labels corresponding to the training scene images, the generated acoustic feature loss data can be shown in Formula 1:

[0188] L s =CrossEntropy(I e ,r c ) Formula 1

[0189] Among them, L s refers to the acoustic characteristic loss data, CrossEntropy refers to the cross entropy function, I e It refers to the acoustic features corresponding to the training scene image features obtained by mapping processing by the adapter model, r c It refers to the scene category label, which can also be understood as the environmental sound effect label (reverberation category label).

[0190] Therefore, the machine learning model (the speech generation model to be trained) can be trained based on the sum of the generated loss data to obtain a speech generation model.

[0191] In an embodiment of the present application, in order to ensure that the environmental sound effects of the generated speech accurately match the visual scene, after obtaining the generated speech, it can be further subjected to scene classification processing to obtain the scene category corresponding to the generated speech, so as to ensure that the generated speech matches the environmental sound effects represented by the scene. Among them, a scene classification model can be integrated into the speech generation model to be trained. The scene classification model can also be called a speech reverberation classification model (SRC), which can perform scene classification processing based on the input speech to obtain the scene category corresponding to the input speech, and then generate scene category loss data based on the difference between the scene category and the scene category label.

[0192] That is to say, during the training of the machine learning model, the loss data may include the fifth part, and specifically, the Mel spectrum corresponding to the generated speech may be input into SRC, and the Mel spectrum corresponding to the generated speech may be subjected to feature extraction processing through SRC, and then classification processing may be performed based on the extracted features to obtain the predicted scene category. Furthermore, scene category loss data may be generated based on the difference between the scene category obtained through the scene classification processing and the scene category label. For example, CE loss data may be generated based on the scene category obtained through the scene classification processing and the scene category label corresponding to the training scene image. Specifically, based on the difference between the scene category and the scene category label, the scene category loss data may be generated as shown in Formula 2:

[0193] L m =CrossEntropy(r p ,r c ) Formula 2

[0194] Among them, L m refers to the acoustic characteristic loss data, CrossEntropy refers to the cross entropy function, r p It refers to the scene category obtained by SRC scene classification processing, r c It refers to the scene category label, which can also be understood as the environmental sound effect label (reverberation category label).

[0195] Therefore, the machine learning model can be trained based on the sum of the above five parts of loss data, namely, the sum of reconstruction loss data, distribution loss data, duration loss data, acoustic feature loss data and scene category loss data, to obtain a speech generation model.

[0196] Since the TTS backbone network of the machine learning model (speech generation model to be trained) is a VITS model, which is a model combined with adversarial learning, the backbone network may include a generator and a discriminator. The goal of the generator is to generate false data as close to the real data as possible, while the goal of the discriminator is to distinguish between the generated data and the real data. Through this adversarial training, the generator gradually improves the authenticity of the generated data, and the discriminator improves the ability to distinguish, and finally reaches a dynamic balance. In other words, the training speech and the generated speech can be discriminated to obtain the probability that each speech in the training speech and the generated speech is a training speech category, and then, based on the probability that each speech is a training speech category, and each speech is a training speech category and a generated speech category, the discrimination loss data is determined, and the discrimination loss data can be used as part of the loss data to train the machine learning model to obtain a speech generation model.

[0197] That is to say, the sixth part of the loss data can be generated based on the probability obtained by the discriminator in the speech generation model to be trained to discriminate the training speech and the generated speech, and each speech belongs to the training speech category. After obtaining the probability that each speech belongs to the training speech category, the probability that each speech belongs to the training speech category can be used to determine the category to which each speech belongs, that is, the probability can refer to the probability distribution generated by the discriminator to indicate that each speech belongs to each category, and then the category with the largest probability is used as the discrimination result. Afterwards, the category to which each speech belongs (such as the training speech category, the generated speech category) can be used as a discriminant label, and the difference between the category to which each speech belongs can be discriminated to determine the discrimination loss data. For example, MAE loss data can be generated based on the category to which each speech belongs and the category to which each speech belongs.

[0198] It should be noted that during the adversarial training process, there is no need to freeze the model parameters of the generator. The loss data of the generator can be the sum of the losses of the above five parts. The machine learning model can be trained based on the loss data of the generator and the loss data of the discriminator, that is, the sum of the loss data of the six parts, to obtain a speech generation model.

[0199] Specifically, the machine learning model can be trained based on the sum of the six parts of loss data to obtain a speech generation model. The sum of the six parts of loss data can be shown as formula 3:

[0200] L total =λ1L VITS +λ2L s +λ3L m Formula 3

[0201] Among them, L total Represents the sum of loss data, L VITS It can represent the loss data of the TTS backbone network in the machine learning model, which can be understood as the loss data of the VITS model, which is the sum of the first part (reconstruction loss data), the second part (distribution loss data), the third part (duration loss data), and the sixth part (discrimination loss data). s For the fourth part (acoustic feature loss data) mentioned above, it can be obtained based on formula 1, L m The fifth part (scene category loss data) can be obtained based on Formula 2. λ1, λ2 and λ3 are hyperparameters used to adjust the contribution of each loss component, and can be set to 10, 2 and 1 respectively in the actual training process.

[0202] Please also read Figure 6 , Figure 6 is a timing diagram of a model training process provided in an embodiment of the present application, such as Figure 6As shown, the machine learning model may include SPE, TTS backbone network (TTS backbone) and scene classification model (speech reverberation classifier SRC). Among them, SPE may include an image encoder and an adapter, and the image encoder may be, for example, an image encoder based on the Clip model, and the adapter may be, for example, composed of GELU and two FCs. The TTS backbone network may be a VITS model, including a priori encoder, a duration predictor, a normalized stream, a spectral encoder (posteriori encoder), a MAS model and a decoder. The priori encoder may be composed of a text encoder and a linear projection layer.

[0203] Specifically, the training samples include training text t i , training scene image I i , and the training speech x obtained by convolution processing (conv.) based on pure speech data and environmental sound effects i . The training scene image I i , training text t i , and training speech x i Input into the machine learning model and extract the training text t through the text encoder i The text features are extracted through the SPE image encoder to extract the training scene image I i Then, the training scene image features are input into the adapter to obtain the acoustic features I corresponding to the training scene image features output by the adapter. e .

[0204] The spectrum encoder in the TTS backbone network can perform feature extraction processing on the frequency domain representation information corresponding to the training speech to obtain posterior distribution parameters, such as the posterior distribution mean parameter and the posterior distribution variance parameter. Then, sampling processing is performed in the posterior distribution defined by the posterior distribution parameters to obtain the second sampling feature Z i Afterwards, the second sampling feature Z i Cutting is performed, and some slices are selected and input into the decoder to obtain the speech waveform (generated speech) generated by the decoder.

[0205] The TTS backbone network can perform two different processing on the training text features. One branch can be to input it into the linear projection layer to obtain the training transformed text features, and convert the transformed text features into training prior distribution parameters such as mean μθ and variance σθ based on the set conversion function. The other branch can be to input it into the duration predictor to obtain the training duration prediction features predicted by the duration predictor. The duration predictor can receive the training text features as input and output the training duration prediction features and noise.

[0206] Among them, the second sampling feature Z i The mapped sampling features can be obtained by mapping through the normalized flow model in the TTS backbone network. The mapped sampling features can then be aligned with the third sampling features sampled from the training prior distribution parameters through the MAS model in the TTS backbone network to obtain the aligned estimated features. In addition, the generated speech output by the decoder is converted into the corresponding frequency domain representation information (Mel spectrum), and is input into the SRC to obtain the scene category r output by the SRC. p .

[0207] Among them, the TTS backbone network also includes a discriminator ( Figure 6 (not shown), the training speech and the generated speech can be discriminated, and the probability of each speech in the training speech and the generated speech being the training speech category can be obtained, and then the discrimination result of discriminating each speech as the training speech category or the generated speech category can be determined. Afterwards, the machine learning model can be trained based on the discrimination result.

[0208] Therefore, the acoustic features I corresponding to the training scene image features output by the adapter can be e and scene category label r c Generate CE loss data (acoustic characteristic loss data); Generate distribution loss data based on the KL divergence between the prior distribution defined by the training prior distribution parameters and the posterior distribution defined by the posterior distribution parameters; Generate reconstruction loss data based on the difference between the frequency domain representation information (Mel spectrum) corresponding to the generated speech and the frequency domain representation information (Mel spectrum) corresponding to the training speech; Generate duration loss data based on the difference between the training duration prediction features and the alignment estimation features; Generate scene category r based on the SRC output p and scene category label r c Generate CE loss data (scene category loss data); generate discrimination loss data based on the discrimination results output by the discriminator and the category to which each speech belongs.

[0209] Furthermore, the machine learning model can be trained based on the sum of these loss data (as shown in Formula 3), which can be used to adjust the model parameters in the TTS backbone network such as the spectrum encoder, text encoder, MAS model, duration predictor, and the model parameters of the adapter in SPE. Thus, a speech generation model can be obtained for generating speech corresponding to the input text and matching the ambient sound effects of the scene indicated by the input scene image.

[0210] In one possible implementation, the SRC may include a structure for capturing local time-frequency features and global dependencies across time frames. Figure 7, Figure 7 is a schematic diagram of the structure of a scene classification model provided in an embodiment of the present application, such as Figure 7 As shown, Figure 7 This is an example of a model structure of SRC, which is not limited in this application. Specifically, SRC may include 4 2-D convolution blocks and a transformer block. Each 2-D convolution block may include a 2-D convolution layer (Conv2d), a batch normalization layer (Batch Normalization, BN), a rectified linear unit (Rectified Linear Unit, RELU), and a 2-D maximum pooling layer (Maxpool2d) in sequence. The SRC also includes a normalization layer (softmax) and a classification layer (Classification Layer, CLS).

[0211] Among them, the two-dimensional convolution layer is used to extract the spatial features of the input data. BN is used to normalize each batch of data to accelerate the convergence speed and reduce the internal covariate shift. RELU is used to introduce nonlinearity so that the model can fit more complex functions. The two-dimensional maximum pooling layer is used to reduce the dimension of the data while retaining the most important features. The transformer block is used to capture long-distance dependencies and enhance the understanding ability of the model. Feature splicing (⊕) represents feature splicing, which can be used to form a more comprehensive feature representation (comprehensive feature representation). The normalization layer is used to convert the output of the last layer into a probability distribution, which is convenient for multi-classification tasks. CLS is used to make classification decisions for the category with the highest probability in the probability distribution.

[0212] Specifically, the frequency domain representation information (Mel spectrum) corresponding to the generated speech is input into the SRC. In one branch, it is processed through a series of convolutional layers (two-dimensional convolutional layers), BN, ReLU activation functions, and maximum pooling layers, which are repeated 4 times to extract the local time-frequency features of the generated speech. In another branch, the global dependencies (global time-frequency features) across time frames of the generated speech are extracted through a two-dimensional maximum pooling layer and a transformer block. The features of the two blocks are then concatenated to form a comprehensive feature representation. After the comprehensive feature representation is input into the normalization layer for further conversion, it is input into the CLS to output the predicted scene category r p .

[0213] In a possible implementation, in the process of training the machine learning model, the SRC model can be trained first, and specifically, the SRC model can be trained based on the training speech and scene category labels, and then the trained SRC model is integrated into the TTS backbone network, and its parameters are frozen for training the TTS backbone network and SPE. In the process of training the SRC model, after outputting the predicted scene category, the frequency domain representation information (such as Mel spectrum) corresponding to the generated speech can be fine-tuned to be closer to the frequency domain representation information corresponding to the speech under the scene category label. Furthermore, the frequency domain representation information after the SRC is fine-tuned can continue to be used for the training of the SRC to improve the effect of the machine learning model learning the acoustic characteristics indicated by the training scene image. In the process of training the SRC model, the SRC fine-tuning the frequency domain representation information can refer to the refinement of the features extracted from the frequency domain representation information to ensure that it is consistent with the acoustic characteristics indicated by the scene category label.

[0214] In some embodiments, the SRC model can be trained 300 times based on the training speech and scene category labels. During the training process, the Space Recognition Error (SRE) can be calculated to evaluate the accuracy of the SRC model classification. Among them, SRE measures the correctness of the room acoustics of the generated speech. A lower SRE indicates that the spatial characteristics of the training speech are very consistent with the spatial characteristics of the scene indicated by the scene category label, indicating that the scene-environment sound alignment is effective. Please also refer to Figure 8 , Figure 8 is a schematic diagram of the change of spatial recognition error of a scene classification model during the training process provided by an embodiment of the present application, such as Figure 8 As shown in the figure, with the increase of training rounds, SRE gradually decreases and tends to be stable, which can reflect that the SRC model becomes more accurate in classifying and fine-tuning the scene. The stabilization in the later rounds indicates that the SRC model has effectively converged. As the training progresses, the matching degree between the scene and the environmental sound effects continues to improve.

[0215] After the pre-trained SRC model is integrated into the TTS backbone network, the TTS backbone network (and the SPE model) can be trained for 200K iterations based on the training text, training scene images, and training speech. The initial learning rate can be set to 1e-4, and the batch size of the training is 24.

[0216] In order to understand which visual features in the scene image have the greatest impact on the image encoder (such as the Clip-based image encoder), we can use Gradient-weighted Class Activation Mapping (Grad-CAM) to intuitively display the image area that contributes most to scene understanding. Specifically, we can generate the corresponding heat map for the training scene image based on the SPE model. Please refer to Fig. 9 , Fig. 9 is a heat map generated based on gradient weighted class activation mapping provided by an embodiment of the present application. Fig. 9 As shown, the left side is a scene image (such as a training scene image), and the right side shows the heat map corresponding to the training scene image. These heat maps show the image areas that the SPE model considers to be more important for extracting image features related to acoustic characteristics. Fig. 9 The first row of training scene images in the figure are scene images representing a wide environment. The heat map shows that the contribution of wide structural elements is relatively large, such as the seat image area and the wall image area in the white dotted box. Fig. 9 The second row of training scene images in the figure are scene images representing narrow environments. The heat map shows that narrow linear elements have a greater contribution, such as the corridor image area in the white dotted box.

[0217] The speech generation model in the speech generation method provided in the embodiment of the present application is objectively and subjectively evaluated below.

[0218] In order to evaluate the effectiveness of the speech generation model, Mel Cepstral Distortion (MCD) and SRE can be used to evaluate the quality of the generated speech and the correspondence between the scene and the environmental sound effects. MCD can measure the spectral distance between the generated speech and the real speech. It is calculated by comparing the Mel-Frequency Cepstral Coefficients (MFCC) of the generated speech with the MFCC of the real speech. A lower MCD value indicates that the generated speech is closer to the real speech in terms of spectral characteristics, that is, the speech quality is higher.

[0219] Among them, the speech generation model proposed in the embodiment of the present application can be compared with other speech generation models, and the evaluation objects include: 1) training speech (ground-truth, GT), that is, the convolutional speech sample (the ground-truth convolutional speech); 2) the VITS model trained based on the convolutional speech sample, without any additional modules (SPE model and SRC model), that is, only including the TTS backbone network; 3) only the SPE model is added to the TTS backbone network; 4) the speech generation model proposed in this application, that is, the SPE model and the SRC model are added to the TTS backbone network. Specifically, the objective evaluation results can be seen in Table 1:

[0220] Evaluation object MCD↓ SRE↓ GT - 0.214 VITS model (without add-on modules) 4.01 0.782 Only SPE model is attached to the TTS backbone network (VITS model) 4.29 0.397 The speech generation model proposed in this application 4.22 0.272

[0221] Table 1

[0222] At the same time, in order to evaluate perceptual aspects that are difficult to capture quantitatively, the speech quality is also evaluated based on the Mean Opinion Score (MOS) with a 95% confidence interval. Listeners can rate the speech separately in the range of 1-5 points. And remove 5% of the extreme values ​​to retain 95% of the scores to calculate the final scores of each item. Specifically, the clarity of the speech can be evaluated by the Clarity-Mean Opinion Score (CMOS) to evaluate the quality of the pronunciation of the generated speech and the difficulty of the listener to recognize words. The naturalness of the generated speech is measured by the Naturalness-Mean Opinion Score (NMOS), and its overall fluency, expressiveness and similarity to human speech are evaluated. The Immersive-Mean Opinion Score (IMOS) is used to evaluate the spatial characteristics of the scene and the matching degree of the speech (that is, the matching degree of the speech and the environmental sound effects) to evaluate the effectiveness in keeping consistent with the audience's perception of the scene acoustics. Specifically, the subjective evaluation results can be found in

[0223] As shown in Table 2:

[0224] Evaluation object CMOS↑ NMOS↑ IMOS↑ GT 4.57 4.54 4.63 VITS model (without add-on modules) 3.75 3.92 3.28 Only SPE model is attached to the TTS backbone network (VITS model) 3.72 3.77 3.81 The speech generation model proposed in this application 3.87 3.98 4.06

[0225] Table 2

[0226] It can be seen from Tables 1 and 2 that the speech generation model proposed in the embodiment of the present application is conducive to enhancing the matching of speech with the ambient sound indicated by the scene image, and can ensure the clarity and high quality of the generated speech to a certain extent.

[0227] In the technical solutions provided in some embodiments of the present application, in the process of generating speech corresponding to the text, the text features of the text to be processed are extracted, and the scene image features of the scene image to be processed are extracted. Then, based on the mapping relationship between the image features and the acoustic features, the scene image features are mapped to obtain the acoustic features corresponding to the scene image features. After that, based on the acoustic features corresponding to the text features and the scene image features, the target speech corresponding to the text to be processed is generated. Among them, the target speech matches the environmental sound effects of the scene indicated by the scene image to be processed. It can be seen that in the process of generating speech based on text, the reverberation effect brought by environmental factors is considered. Compared with the generated speech that only matches the input text, the generated speech can also match the acoustic characteristics of the visual scene, so that the generated speech can accurately reflect the given scene, improving the accuracy and effect of speech generation. While being able to convey the expected information, it can also include the environmental sound effects of the visual scene, making the generated speech more realistic, and to a certain extent, it can provide users with a more immersive and immersive auditory experience.

[0228] The method of the embodiment of the present application is described in detail above. In order to facilitate better implementation of the above scheme of the embodiment of the present application, the device of the embodiment of the present application is provided below accordingly.

[0229] See also Fig.10 , Fig.10 is a schematic diagram of the structure of a speech generating device provided in an embodiment of the present application. The speech generating device 100 can be used to perform Figure 2 The corresponding steps in the speech generation method shown. The speech generation device 100 includes the following units:

[0230] An extraction unit 1001 is used to extract text features of a text to be processed and to extract scene image features of a scene image to be processed;

[0231] The processing unit 1002 is used to perform mapping processing on the scene image feature based on the mapping relationship between the image feature and the acoustic feature to obtain the acoustic feature corresponding to the scene image feature;

[0232] The generating unit 1003 is used to generate a target speech corresponding to the text to be processed and matching the ambient sound effect of the scene indicated by the scene image to be processed based on the acoustic features corresponding to the text features and the scene image features.

[0233] In a possible implementation, the processing unit 1002 is used to perform mapping processing on the scene image feature based on the mapping relationship between the image feature and the acoustic feature to obtain the acoustic feature corresponding to the scene image feature, specifically for:

[0234] Performing a first linear transformation process on the scene image feature to obtain a transformed scene image feature;

[0235] Performing nonlinear mapping processing on the transformed scene image features based on a set activation function;

[0236] A second linear transformation is performed on the image features obtained through the nonlinear mapping process to obtain acoustic features corresponding to the scene image features.

[0237] In a possible implementation, the text feature is obtained by extracting a feature of a symbol sequence of pronunciation elements corresponding to the text to be processed; the generating unit 1003 is used to generate a target speech corresponding to the text to be processed and matching the environmental sound effect of the scene indicated by the scene image to be processed based on the text feature and the acoustic features corresponding to the scene image feature, specifically for:

[0238] Based on the mapping relationship between the pronunciation elements and the duration, mapping processing is performed on the duration of the pronunciation elements indicated by the text features to obtain a duration prediction feature;

[0239] Performing a third linear transformation process on the text features to obtain transformed text features, and converting the transformed text features into prior distribution parameters according to a set conversion function;

[0240] The target speech is generated based on the duration prediction feature, the prior distribution parameter and the acoustic features corresponding to the scene image features.

[0241] In a possible implementation, the prior distribution parameters include a prior mean parameter and a prior variance parameter; the generating unit 1003 is used to generate the target speech based on the duration prediction feature, the prior distribution parameters and the acoustic features corresponding to the scene image features, specifically for:

[0242] Based on the duration prediction feature, sampling processing is performed in the prior distribution defined by the prior mean parameter and the prior variance parameter to obtain a first sampling feature;

[0243] The target speech is generated based on the first sampling feature and the acoustic features corresponding to the scene image features.

[0244] In a possible implementation, the dimension of the transformed text feature is the same as the dimension of the duration prediction feature; the generating unit 1003 is used to perform sampling processing in the prior distribution defined by the prior mean parameter and the prior variance parameter based on the duration prediction feature to obtain a first sampling feature, specifically for:

[0245] Based on each duration feature element in the duration prediction feature, repeating the feature element at the corresponding position in the transformed text feature for the number of times indicated by each duration feature element to obtain an extended text feature;

[0246] Based on the extended text feature, sampling processing is performed from the prior distribution to obtain the first sampling feature; wherein the dimension of the first sampling feature is the dimension of the extended text feature.

[0247] In a possible implementation, the generating unit 1003 is configured to generate the target speech based on the acoustic features corresponding to the first sampling features and the scene image features, specifically:

[0248] Based on the mapping relationship between the sampling feature and the speech signal feature, mapping processing is performed on the first sampling feature to obtain the speech signal feature corresponding to the first sampling feature;

[0249] The speech signal feature corresponding to the first sampling feature and the acoustic feature corresponding to the scene image feature are fused to obtain the target speech.

[0250] According to one embodiment of the present application, Figure 2 The steps involved in the method shown can all be performed by Fig.10 The various units in the speech generating device shown are executed. For example, Figure 2 Step S201 shown is composed of Fig.10 The extraction unit 1001 shown in FIG. 1 is used to perform step S202. Fig.10 The processing unit 1002 shown in FIG. 1 is used to perform step S203. Fig.10 The generation unit 1003 shown is executed.

[0251] According to one embodiment of the present application, Fig.10 The various units in the illustrated speech generating device 100 can be respectively or all merged into one or several other units to constitute, or some (some) units therein can also be split into multiple smaller units in function to constitute, which can achieve the same operation without affecting the realization of the technical effect of the embodiment of the present application. The above-mentioned units are divided based on logical functions. In practical applications, the function of a unit can also be realized by multiple units, or the function of multiple units is realized by one unit. In other embodiments of the present application, the speech generating device 100 can also include other units. In practical applications, these functions can also be assisted by other units to realize, and can be realized by the collaboration of multiple units.

[0252] According to another embodiment of the present application, the program can be executed by running a program on a general-purpose computing device including a general-purpose computer including processing elements and storage elements such as a central processing unit (CPU), a random access memory medium (RAM), and a read-only memory medium (ROM). Figure 2 A computer program (including program code) for each step involved in the corresponding method shown in Fig.10 The speech generating device 100 shown in the figure can be used to implement the speech generating method of the embodiment of the present application. The computer program can be recorded on a computer-readable storage medium, for example, and loaded into the computer-readable storage medium. Figure 1 The speech generation system shown is in a speech generation device and runs therein.

[0253] Based on the same inventive concept, the principles and beneficial effects of solving the problems by the speech generation device provided in the embodiment of the present application are similar to the principles and beneficial effects of solving the problems by the speech generation method in the method embodiment of the present application. Please refer to the principles and beneficial effects of the implementation of the method. For the sake of concise description, they will not be repeated here.

[0254] See also Fig.11 , Fig.11 is a schematic diagram of the structure of another speech generating device provided in an embodiment of the present application. The speech generating device 100 can be used to perform Figure 5 The corresponding steps in the speech generation method shown. The speech generation device 100 includes the following units:

[0255] The acquisition unit 1101 is used to acquire a training text, a training scene image, and a training voice corresponding to the training text and matching the environmental sound effect of the scene indicated by the training scene image;

[0256] The processing unit 1102 is further used to perform feature extraction processing on the frequency domain representation information corresponding to the training speech to obtain posterior distribution parameters;

[0257] The processing unit 1102 is further configured to perform speech generation processing based on the posterior distribution parameters to obtain generated speech;

[0258] The training unit 1103 is used to train the machine learning model based on the training text, the training scene image, the training speech, the generated speech and the posterior distribution parameters to obtain a speech generation model; wherein the speech generation model is used to generate a target speech corresponding to the input text and matching the environmental sound effects indicated by the input image based on the input text and the input image.

[0259] In a possible implementation, the acquisition unit 1101 is used to acquire a training speech corresponding to the training text and matching the environmental sound effect of the scene indicated by the training scene image, specifically for:

[0260] Acquire the environmental sound effects corresponding to the training scene image and the speech corresponding to the training text;

[0261] The training speech is obtained by performing convolution processing based on the speech corresponding to the training text and the environmental sound effect.

[0262] In a possible implementation, the posterior distribution parameters include a posterior mean parameter and a posterior variance parameter; the processing unit 1102 is further configured to perform speech generation processing based on the posterior distribution parameters to obtain generated speech, specifically for:

[0263] Perform sampling processing in the posterior distribution defined by the posterior mean parameter and the posterior variance parameter to obtain a second sampling feature;

[0264] Speech generation processing is performed based on the second sampling feature to obtain the generated speech.

[0265] In a possible implementation, the training unit 1103 is used to train the machine learning model based on the training text, the training scene image, the training speech, the generated speech and the posterior distribution parameter to obtain the speech generation model, specifically for:

[0266] Performing a fourth linear transformation process on the training text features extracted based on the training text to obtain training transformed text features, and converting the training transformed text features into training prior distribution parameters according to a set conversion function;

[0267] Based on the mapping relationship between the pronunciation elements and the duration, mapping processing is performed on the duration of the pronunciation elements indicated by the training text features to obtain training duration prediction features;

[0268] Based on the training duration prediction feature, sampling is performed in the training prior distribution defined by the training prior distribution parameter to obtain a third sampling feature,

[0269] Performing an alignment process on the second sampling feature and the third sampling feature to obtain an aligned estimated feature;

[0270] Based on the training duration prediction feature, the training prior distribution, the training scene image, the training speech, the generated speech, the posterior distribution and the alignment estimation feature, the machine learning model is trained to obtain the speech generation model.

[0271] In a possible implementation, the training unit 1103 is used to train the machine learning model based on the training duration prediction feature, the training prior distribution, the training scene image, the training speech, the generated speech, the posterior distribution and the alignment estimation feature to obtain the speech generation model, specifically for:

[0272] Based on the mapping relationship between the image features and the acoustic features, mapping processing is performed on the training scene image features extracted based on the training scene image to obtain the acoustic features corresponding to the training scene image features;

[0273] Based on the training duration prediction feature, the training prior distribution, the training speech, the generated speech, the posterior distribution, the alignment estimation feature and the acoustic features corresponding to the training scene image features, the machine learning model is trained to obtain the speech generation model.

[0274] In a possible implementation, the training unit 1103 is used to train the machine learning model based on the training duration prediction feature, the training prior distribution, the training speech, the generated speech, the posterior distribution, the alignment estimation feature and the acoustic feature corresponding to the training scene image feature to obtain the speech generation model, specifically for:

[0275] generating reconstruction loss data based on a difference between the frequency domain representation information corresponding to the generated speech and the frequency domain representation information corresponding to the training speech;

[0276] generating distribution loss data based on the difference between the training prior distribution and the posterior distribution;

[0277] Generating duration loss data based on the difference between the training duration prediction feature and the alignment estimation feature;

[0278] generating acoustic feature loss data based on a difference between the acoustic feature corresponding to the training scene image feature and the scene category label corresponding to the training scene image;

[0279] Based on the sum of the reconstruction loss data, the distribution loss data, the duration loss data and the acoustic feature loss data, the machine learning model is trained to obtain the speech generation model.

[0280] In a possible implementation, the processing unit 1102 is further configured to perform scene classification processing on the frequency domain representation information corresponding to the generated speech to obtain a scene category corresponding to the generated speech;

[0281] generating scene category loss data based on a difference between the scene category and the scene category label;

[0282] The training unit 1103 is used to train the machine learning model based on the sum of the reconstruction loss data, the distribution loss data, the duration loss data and the acoustic feature loss data to obtain the speech generation model, specifically for:

[0283] Based on the sum of the reconstruction loss data, the distribution loss data, the duration loss data, the acoustic feature loss data and the scene category loss data, the machine learning model is trained to obtain the speech generation model.

[0284] In a possible implementation, the processing unit is further configured to perform discrimination processing on the training speech and the generated speech to obtain a probability that each speech in the training speech and the generated speech belongs to the training speech category;

[0285] A determination unit 1104, configured to determine discrimination loss data based on the probability that each speech is a training speech category;

[0286] The training unit 1103 is used to train the machine learning model based on the sum of the reconstruction loss data, the distribution loss data, the duration loss data, the acoustic feature loss data and the scene category loss data to obtain the speech generation model, specifically for:

[0287] Based on the sum of the reconstruction loss data, the distribution loss data, the duration loss data, the acoustic feature loss data, the scene category loss data and the discrimination loss data, the machine learning model is trained to obtain the speech generation model.

[0288] According to one embodiment of the present application, Figure 5 The steps involved in the method shown can all be performed by Fig.11 The various units in the speech generating device shown are executed. For example, Figure 5 Step S501 shown is composed of Fig.11 The acquisition unit 1101 shown is executed, and steps 502 to S503 are performed by Fig.11 The processing unit 1102 shown in FIG. 1 is used to perform step S504. Fig.11 The training unit 1103 shown is used for execution.

[0289] According to one embodiment of the present application, Fig.11The various units in the illustrated speech generating device 110 can be respectively or all merged into one or several other units to constitute, or some of the units (some) can also be split into multiple smaller units in function to constitute, which can achieve the same operation without affecting the realization of the technical effect of the embodiment of the present application. The above-mentioned units are divided based on logical functions. In practical applications, the function of a unit can also be realized by multiple units, or the function of multiple units is realized by one unit. In other embodiments of the present application, the speech generating device 110 can also include other units. In practical applications, these functions can also be assisted by other units to be realized, and can be realized by the collaboration of multiple units.

[0290] According to another embodiment of the present application, the program can be executed by running a program on a general-purpose computing device including a general-purpose computer including processing elements and storage elements such as a central processing unit (CPU), a random access memory medium (RAM), and a read-only memory medium (ROM). Figure 5 A computer program (including program code) for each step involved in the corresponding method shown in Fig.11 The speech generating device 110 shown in the figure can be used to implement the speech generating method of the embodiment of the present application. The computer program can be recorded on a computer-readable storage medium, for example, and loaded into the computer-readable storage medium. Figure 1 The speech generation system shown is in a speech generation device and runs therein.

[0291] Based on the same inventive concept, the principles and beneficial effects of solving the problems by the speech generation device provided in the embodiment of the present application are similar to the principles and beneficial effects of solving the problems by the speech generation method in the method embodiment of the present application. Please refer to the principles and beneficial effects of the implementation of the method. For the sake of concise description, they will not be repeated here.

[0292] Based on the description of the above-mentioned speech generation method embodiment, the present application embodiment also discloses a speech generation device, see Fig.12 The speech generating device 120 may include at least a processor 1201, an input device 1202, an output device 1203, and a memory 1204. The processor 1201, the input device 1202, the output device 1203, and the memory 1204 in the speech generating device 120 may be connected via a bus or other means.

[0293] The above-mentioned memory 1204 is a memory device in the speech generating device 120, which is used to store programs and data. It can be understood that the memory 1204 here can include both the built-in storage medium of the speech generating device and the extended storage medium supported by the speech generating device 120. The memory 1204 provides a storage space, which stores the operating system of the speech generating device 120. In addition, computer programs (including program codes) are also stored in the storage space. It should be noted that the computer storage medium here can be a high-speed RAM memory; optionally, it can also be at least one computer storage medium away from the aforementioned processor. The aforementioned processor can be called a central processing unit (CPU), which is the core and control center of the speech generating device and is used to run the computer program stored in the above-mentioned memory 1204.

[0294] In one implementation, the processor 1201 may load and execute a computer program stored in the memory 1204 to implement the corresponding steps of the method in the above-mentioned speech generation method embodiment; specifically, the processor 1201 loads and executes the computer program stored in the memory 1204, which is used to:

[0295] Extracting text features of the text to be processed, and extracting scene image features of the scene image to be processed;

[0296] Based on the mapping relationship between the image feature and the acoustic feature, mapping processing is performed on the scene image feature to obtain the acoustic feature corresponding to the scene image feature;

[0297] Based on the acoustic features corresponding to the text features and the scene image features, a target speech is generated that corresponds to the text to be processed and matches the ambient sound effects of the scene indicated by the scene image to be processed.

[0298] In a possible implementation, the processor 1201 loads and executes a computer program stored in the memory 1204, which is used to perform mapping processing on the scene image features based on the mapping relationship between the image features and the acoustic features to obtain the acoustic features corresponding to the scene image features, specifically for:

[0299] Performing a first linear transformation process on the scene image feature to obtain a transformed scene image feature;

[0300] Performing nonlinear mapping processing on the transformed scene image features based on a set activation function;

[0301] A second linear transformation is performed on the image features obtained through the nonlinear mapping process to obtain acoustic features corresponding to the scene image features.

[0302] In a possible implementation, the text feature is obtained by extracting a feature of a symbol sequence of pronunciation elements corresponding to the text to be processed; the processor 1201 loads and executes a computer program stored in the memory 1204, which is used to generate a target speech corresponding to the text to be processed and matching the ambient sound effect of the scene indicated by the scene image to be processed based on the acoustic features corresponding to the text feature and the scene image feature, specifically for:

[0303] Based on the mapping relationship between the pronunciation elements and the duration, mapping processing is performed on the duration of the pronunciation elements indicated by the text features to obtain a duration prediction feature;

[0304] Performing a third linear transformation process on the text features to obtain transformed text features, and converting the transformed text features into prior distribution parameters according to a set conversion function;

[0305] The target speech is generated based on the duration prediction feature, the prior distribution parameter and the acoustic features corresponding to the scene image features.

[0306] In a possible implementation, the prior distribution parameters include a prior mean parameter and a prior variance parameter; the processor 1201 loads and executes a computer program stored in the memory 1204, which is used to generate the target speech based on the duration prediction feature, the prior distribution parameters and the acoustic features corresponding to the scene image features, specifically for:

[0307] Based on the duration prediction feature, sampling processing is performed in the prior distribution defined by the prior mean parameter and the prior variance parameter to obtain a first sampling feature;

[0308] The target speech is generated based on the first sampling feature and the acoustic features corresponding to the scene image features.

[0309] In a possible implementation, the dimension of the transformed text feature is the same as the dimension of the duration prediction feature; the processor 1201 loads and executes a computer program stored in the memory 1204, which is used to perform sampling processing in the prior distribution defined by the prior mean parameter and the prior variance parameter based on the duration prediction feature to obtain a first sampling feature, which is specifically used to:

[0310] Based on each duration feature element in the duration prediction feature, repeating the feature element at the corresponding position in the transformed text feature for the number of times indicated by each duration feature element to obtain an extended text feature;

[0311] Based on the extended text feature, sampling processing is performed from the prior distribution to obtain the first sampling feature; wherein the dimension of the first sampling feature is the dimension of the extended text feature.

[0312] In a possible implementation, the processor 1201 loads and executes a computer program stored in the memory 1204, which is used to generate the target speech based on the acoustic features corresponding to the first sampling features and the scene image features, specifically for:

[0313] Based on the mapping relationship between the sampling feature and the speech signal feature, mapping processing is performed on the first sampling feature to obtain the speech signal feature corresponding to the first sampling feature;

[0314] The speech signal feature corresponding to the first sampling feature and the acoustic feature corresponding to the scene image feature are fused to obtain the target speech.

[0315] In a possible implementation, the processor 1201 may load and execute a computer program stored in the memory 1204 to implement the corresponding steps in the above embodiment of another method for processing a recommendation of an order service provider; specifically, the processor 1201 loads and executes the computer program stored in the memory 1204 to:

[0316] Acquire a training text, a training scene image, and a training voice corresponding to the training text and matching the environmental sound effects of the scene indicated by the training scene image;

[0317] Performing feature extraction processing on the frequency domain representation information corresponding to the training speech to obtain posterior distribution parameters;

[0318] Perform speech generation processing based on the posterior distribution parameters to obtain generated speech;

[0319] Based on the training text, the training scene image, the training speech, the generated speech and the posterior distribution parameters, the machine learning model is trained to obtain a speech generation model; wherein the speech generation model is used to generate a target speech corresponding to the input text and matching the environmental sound effects indicated by the input image based on the input text and the input image.

[0320] In a possible implementation, the processor 1201 loads and executes a computer program stored in the memory 1204 to obtain a training voice corresponding to the training text and matching the environmental sound effect of the scene indicated by the training scene image, specifically for:

[0321] Acquire the environmental sound effects corresponding to the training scene image and the speech corresponding to the training text;

[0322] The training speech is obtained by performing convolution processing based on the speech corresponding to the training text and the environmental sound effect.

[0323] In a possible implementation, the posterior distribution parameters include a posterior mean parameter and a posterior variance parameter; the processor 1201 loads and executes a computer program stored in the memory 1204, which is used for performing speech generation processing based on the posterior distribution parameters to obtain generated speech, specifically for:

[0324] Perform sampling processing in the posterior distribution defined by the posterior mean parameter and the posterior variance parameter to obtain a second sampling feature;

[0325] Speech generation processing is performed based on the second sampling feature to obtain the generated speech.

[0326] In a possible implementation, the processor 1201 loads and executes a computer program stored in the memory 1204, and is used to train the machine learning model based on the training text, the training scene image, the training speech, the generated speech, and the posterior distribution parameter to obtain a speech generation model, specifically for:

[0327] Performing a fourth linear transformation process on the training text features extracted based on the training text to obtain training transformed text features, and converting the training transformed text features into training prior distribution parameters according to a set conversion function;

[0328] Based on the mapping relationship between the pronunciation elements and the duration, mapping processing is performed on the duration of the pronunciation elements indicated by the training text features to obtain training duration prediction features;

[0329] Based on the training duration prediction feature, sampling is performed in the training prior distribution defined by the training prior distribution parameter to obtain a third sampling feature,

[0330] Performing an alignment process on the second sampling feature and the third sampling feature to obtain an aligned estimated feature;

[0331] Based on the training duration prediction feature, the training prior distribution, the training scene image, the training speech, the generated speech, the posterior distribution and the alignment estimation feature, the machine learning model is trained to obtain the speech generation model.

[0332] In a possible implementation, the processor 1201 loads and executes a computer program stored in the memory 1204, and is used to train the machine learning model based on the training duration prediction feature, the training prior distribution, the training scene image, the training speech, the generated speech, the posterior distribution, and the alignment estimation feature to obtain the speech generation model, specifically for:

[0333] Based on the mapping relationship between the image features and the acoustic features, mapping processing is performed on the training scene image features extracted based on the training scene image to obtain the acoustic features corresponding to the training scene image features;

[0334] Based on the training duration prediction feature, the training prior distribution, the training speech, the generated speech, the posterior distribution, the alignment estimation feature and the acoustic features corresponding to the training scene image features, the machine learning model is trained to obtain the speech generation model.

[0335] In a possible implementation, the processor 1201 loads and executes a computer program stored in the memory 1204, and is used to train the machine learning model based on the training duration prediction feature, the training prior distribution, the training speech, the generated speech, the posterior distribution, the alignment estimation feature, and the acoustic feature corresponding to the training scene image feature to obtain the speech generation model, specifically for:

[0336] generating reconstruction loss data based on a difference between the frequency domain representation information corresponding to the generated speech and the frequency domain representation information corresponding to the training speech;

[0337] generating distribution loss data based on the difference between the training prior distribution and the posterior distribution;

[0338] Generating duration loss data based on the difference between the training duration prediction feature and the alignment estimation feature;

[0339] generating acoustic feature loss data based on a difference between the acoustic feature corresponding to the training scene image feature and the scene category label corresponding to the training scene image;

[0340] Based on the sum of the reconstruction loss data, the distribution loss data, the duration loss data and the acoustic feature loss data, the machine learning model is trained to obtain the speech generation model.

[0341] In a possible implementation, the processor 1201 loads and executes a computer program stored in the memory 1204, and is further configured to perform scene classification processing on the frequency domain representation information corresponding to the generated speech to obtain a scene category corresponding to the generated speech;

[0342] generating scene category loss data based on a difference between the scene category and the scene category label;

[0343] The step of training the machine learning model based on the sum of the reconstruction loss data, the distribution loss data, the duration loss data and the acoustic feature loss data to obtain the speech generation model comprises:

[0344] Based on the sum of the reconstruction loss data, the distribution loss data, the duration loss data, the acoustic feature loss data and the scene category loss data, the machine learning model is trained to obtain the speech generation model.

[0345] In a possible implementation, the processor 1201 loads and executes a computer program stored in the memory 1204, and is further configured to perform discrimination processing on the training speech and the generated speech to obtain a probability that each speech in the training speech and the generated speech is a training speech category;

[0346] Determining discrimination loss data based on the probability that each speech is a training speech category;

[0347] The step of training the machine learning model based on the sum of the reconstruction loss data, the distribution loss data, the duration loss data, the acoustic feature loss data, and the scene category loss data to obtain the speech generation model comprises:

[0348] Based on the sum of the reconstruction loss data, the distribution loss data, the duration loss data, the acoustic feature loss data, the scene category loss data and the discrimination loss data, the machine learning model is trained to obtain the speech generation model.

[0349] It should be understood that in the embodiment of the present application, the processor 1201 may be a central processing unit (CPU), and the processor 1201 may also be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.

[0350] In an embodiment of the present application, a computer-readable storage medium is provided. The computer-readable storage medium stores a computer program. The computer program includes program instructions. When the program instructions are executed by a processor, the steps performed in all the above embodiments can be executed.

[0351] An embodiment of the present application also provides a computer program product or a computer program, which includes computer instructions stored in a computer-readable storage medium. When the computer instructions are executed by a processor of a computer device, the methods in all the above embodiments are executed.

[0352] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through a computer program, and the above-mentioned program can be stored in a computer-readable storage medium, and when the program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, the above-mentioned storage medium can be a disk, an optical disk, a read-only memory (ROM) or a random access memory (RAM), etc.

[0353] What is disclosed above is only a preferred embodiment of the present invention, and it certainly cannot be used to limit the scope of rights of the present invention. Ordinary technicians in this field can understand that all or part of the processes of the above embodiment and equivalent changes made according to the claims of the present invention still fall within the scope of the invention.

[0354] It is also particularly important to note that when the above embodiments of the present application are applied to specific products or technologies, if it is necessary to obtain user data, the user's permission or consent must be obtained, and the collection, use and processing of relevant data must comply with relevant laws, regulations and standards of relevant countries and regions.

Claims

1. A speech generation method, characterized in that: include: Extracting text features of the text to be processed, and extracting scene image features of the scene image to be processed; Based on the mapping relationship between the image feature and the acoustic feature, mapping processing is performed on the scene image feature to obtain the acoustic feature corresponding to the scene image feature; Based on the acoustic features corresponding to the text features and the scene image features, a target speech is generated that corresponds to the text to be processed and matches the ambient sound effects of the scene indicated by the scene image to be processed.

2. The method according to claim 1, characterized in that The mapping process is performed on the scene image feature based on the mapping relationship between the image feature and the acoustic feature to obtain the acoustic feature corresponding to the scene image feature, including: Performing a first linear transformation process on the scene image feature to obtain a transformed scene image feature; Performing nonlinear mapping processing on the transformed scene image features based on a set activation function; A second linear transformation is performed on the image features obtained through the nonlinear mapping process to obtain acoustic features corresponding to the scene image features.

3. The method according to claim 1, characterized in that The text feature is obtained by extracting a symbol sequence of pronunciation elements corresponding to the text to be processed; and the target speech corresponding to the text to be processed and matching the environmental sound effect of the scene indicated by the scene image to be processed based on the text feature and the acoustic feature corresponding to the scene image feature to be processed is generated, including: Based on the mapping relationship between the pronunciation elements and the duration, mapping processing is performed on the duration of the pronunciation elements indicated by the text features to obtain a duration prediction feature; Performing a third linear transformation process on the text features to obtain transformed text features, and converting the transformed text features into prior distribution parameters according to a set conversion function; The target speech is generated based on the duration prediction feature, the prior distribution parameter and the acoustic features corresponding to the scene image features.

4. The method according to claim 3, characterized in that The prior distribution parameters include a prior mean parameter and a prior variance parameter; and the generating of the target speech based on the duration prediction feature, the prior distribution parameters and the acoustic features corresponding to the scene image features includes: Based on the duration prediction feature, sampling processing is performed in the prior distribution defined by the prior mean parameter and the prior variance parameter to obtain a first sampling feature; The target speech is generated based on the first sampling feature and the acoustic features corresponding to the scene image features.

5. The method according to claim 4, characterized in that The dimension of the transformed text feature is the same as the dimension of the duration prediction feature; the sampling process based on the duration prediction feature is performed in the prior distribution defined by the prior mean parameter and the prior variance parameter to obtain the first sampling feature, including: Based on each duration feature element in the duration prediction feature, repeating the feature element at the corresponding position in the transformed text feature for the number of times indicated by each duration feature element to obtain an extended text feature; Based on the extended text feature, sampling processing is performed from the prior distribution to obtain the first sampling feature; wherein the dimension of the first sampling feature is the dimension of the extended text feature.

6. The method according to claim 4, characterized in that The generating the target speech based on the acoustic features corresponding to the first sampling features and the scene image features includes: Based on the mapping relationship between the sampling feature and the speech signal feature, mapping processing is performed on the first sampling feature to obtain the speech signal feature corresponding to the first sampling feature; The speech signal feature corresponding to the first sampling feature and the acoustic feature corresponding to the scene image feature are fused to obtain the target speech.

7. A speech generation method, characterized in that: include: Acquire a training text, a training scene image, and a training voice corresponding to the training text and matching the environmental sound effects of the scene indicated by the training scene image; Performing feature extraction processing on the frequency domain representation information corresponding to the training speech to obtain posterior distribution parameters; Perform speech generation processing based on the posterior distribution parameters to obtain generated speech; Based on the training text, the training scene image, the training speech, the generated speech and the posterior distribution parameters, the machine learning model is trained to obtain a speech generation model; wherein the speech generation model is used to generate a target speech corresponding to the input text and matching the environmental sound effects indicated by the input image based on the input text and the input image.

8. The method according to claim 7, characterized in that The step of acquiring a training voice corresponding to the training text and matching the environmental sound effects of the scene indicated by the training scene image includes: Acquire the environmental sound effects corresponding to the training scene image and the speech corresponding to the training text; The training speech is obtained by performing convolution processing based on the speech corresponding to the training text and the environmental sound effect.

9. The method according to claim 7, characterized in that: The posterior distribution parameters include a posterior mean parameter and a posterior variance parameter; the speech generation process is performed based on the posterior distribution parameters to obtain the generated speech, including: Perform sampling processing in the posterior distribution defined by the posterior mean parameter and the posterior variance parameter to obtain a second sampling feature; Speech generation processing is performed based on the second sampling feature to obtain the generated speech.

10. The method according to claim 9, characterized in that The method of training a machine learning model based on the training text, the training scene image, the training speech, the generated speech and the posterior distribution parameter to obtain a speech generation model includes: Performing a fourth linear transformation process on the training text features extracted based on the training text to obtain training transformed text features, and converting the training transformed text features into training prior distribution parameters according to a set conversion function; Based on the mapping relationship between the pronunciation elements and the duration, mapping processing is performed on the duration of the pronunciation elements indicated by the training text features to obtain training duration prediction features; Based on the training duration prediction feature, sampling is performed in the training prior distribution defined by the training prior distribution parameter to obtain a third sampling feature, Performing an alignment process on the second sampling feature and the third sampling feature to obtain an aligned estimated feature; Based on the training duration prediction feature, the training prior distribution, the training scene image, the training speech, the generated speech, the posterior distribution and the alignment estimation feature, the machine learning model is trained to obtain the speech generation model.

11. The method according to claim 10, characterized in that The step of training the machine learning model based on the training duration prediction feature, the training prior distribution, the training scene image, the training speech, the generated speech, the posterior distribution and the alignment estimation feature to obtain the speech generation model includes: Based on the mapping relationship between the image features and the acoustic features, mapping processing is performed on the training scene image features extracted based on the training scene image to obtain the acoustic features corresponding to the training scene image features; Based on the training duration prediction feature, the training prior distribution, the training speech, the generated speech, the posterior distribution, the alignment estimation feature and the acoustic features corresponding to the training scene image features, the machine learning model is trained to obtain the speech generation model.

12. The method according to claim 11, characterized in that The method of training the machine learning model based on the training duration prediction feature, the training prior distribution, the training speech, the generated speech, the posterior distribution, the alignment estimation feature and the acoustic feature corresponding to the training scene image feature to obtain the speech generation model includes: generating reconstruction loss data based on a difference between the frequency domain representation information corresponding to the generated speech and the frequency domain representation information corresponding to the training speech; generating distribution loss data based on the difference between the training prior distribution and the posterior distribution; Generating duration loss data based on the difference between the training duration prediction feature and the alignment estimation feature; generating acoustic feature loss data based on a difference between the acoustic feature corresponding to the training scene image feature and the scene category label corresponding to the training scene image; Based on the sum of the reconstruction loss data, the distribution loss data, the duration loss data and the acoustic feature loss data, the machine learning model is trained to obtain the speech generation model.

13. The method according to claim 12, characterized in that The method further comprises: Performing scene classification processing on the frequency domain representation information corresponding to the generated speech to obtain the scene category corresponding to the generated speech; generating scene category loss data based on a difference between the scene category and the scene category label; The step of training the machine learning model based on the sum of the reconstruction loss data, the distribution loss data, the duration loss data and the acoustic feature loss data to obtain the speech generation model comprises: Based on the sum of the reconstruction loss data, the distribution loss data, the duration loss data, the acoustic feature loss data and the scene category loss data, the machine learning model is trained to obtain the speech generation model.

14. The method according to claim 13, characterized in that The method further comprises: Performing discrimination processing on the training speech and the generated speech to obtain the probability that each speech in the training speech and the generated speech belongs to the training speech category; Determining discrimination loss data based on the probability that each speech is a training speech category; The step of training the machine learning model based on the sum of the reconstruction loss data, the distribution loss data, the duration loss data, the acoustic feature loss data, and the scene category loss data to obtain the speech generation model comprises: Based on the sum of the reconstruction loss data, the distribution loss data, the duration loss data, the acoustic feature loss data, the scene category loss data and the discrimination loss data, the machine learning model is trained to obtain the speech generation model.

15. A speech generating device, characterized in that: include: An extraction unit, used to extract text features of the text to be processed and to extract scene image features of the scene image to be processed; A processing unit, configured to perform mapping processing on the scene image features based on a mapping relationship between the image features and the acoustic features, so as to obtain acoustic features corresponding to the scene image features; A generating unit is used to generate a target speech corresponding to the text to be processed and matching the ambient sound effect of the scene indicated by the scene image to be processed based on the acoustic features corresponding to the text features and the scene image features.

16. A speech generating device, characterized in that: include: An acquisition unit, configured to acquire a training text, a training scene image, and a training voice corresponding to the training text and matching the environmental sound effect of the scene indicated by the training scene image; A processing unit, configured to perform feature extraction processing on the frequency domain representation information corresponding to the training speech to obtain posterior distribution parameters; The processing unit is further used to perform speech generation processing based on the posterior distribution parameters to obtain generated speech; A training unit is used to train a machine learning model based on the training text, the training scene image, the training speech, the generated speech and the posterior distribution parameters to obtain a speech generation model; wherein the speech generation model is used to generate a target speech corresponding to the input text and matching the environmental sound effects indicated by the input image based on the input text and the input image.

17. An electronic device, characterized in that: include: one or more processors; A memory for storing one or more computer programs, which, when executed by the one or more processors, enables the electronic device to implement the speech generation method described in any one of claims 1-6 and 7-14.

18. A computer readable medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the speech generation method according to any one of claims 1 to 6 and 7 to 14 is implemented.

19. A computer program product, characterized in that The computer program product includes a computer program, which is stored in a computer-readable storage medium. A processor of an electronic device reads and executes the computer program from the computer-readable storage medium, so that the electronic device executes the speech generation method according to any one of claims 1-6 and 7-14.

Images