An audio synthesis method and related device

Abstract

The application provides an audio synthesis method and related equipment applied to a voice interaction scene. A text for generating a voice is obtained, text features (for example, semantic features and phoneme length features) are extracted from the text, and a sound scene matched with the text is determined, wherein background sound frequencies corresponding to the sound scene are included in a preset sound scene database; text features and indication information of the sound scene are input into a correction flow model to obtain a voice signal with background sound as a final synthesized audio signal. Compared with a scheme of simply playing a voice signal for a user, the increase of background sound can optimize the experience of the sense of hearing, optimize the effect of a terminal device sounding to the user, thereby improving the feeling and cognition of the user to the text content, optimizing the effect of the electronic device interacting with the user, and improving the user experience.

Description

Technical Field

[0001] This application relates to the field of terminal technology, and in particular to an audio synthesis method and related equipment. Background Technology

[0002] Speech synthesis technology is often used in scenarios where terminal devices interact with users. For example, in scenarios where users interact with terminal devices via voice, voice assistants can use speech synthesis technology to speak to users.

[0003] Currently, taking voice assistants as an example, the primary function is to generate human voices for conversation. Speech synthesis technology mainly focuses on optimizing the timbre, speech rate, and emotion of the human voice itself. However, as the intelligence of terminal devices increases, users' demands for the synthesized speech are also rising.

[0004] Therefore, optimizing the quality of voice messages played by terminal devices to users has become an urgent problem to be solved. Summary of the Invention

[0005] This application provides an audio synthesis method and related equipment, which can optimize the effect of terminal devices emitting sound to users, enhance users' perception and understanding of text content, thereby optimizing the interaction between electronic devices and users and improving user experience.

[0006] In a first aspect, an audio synthesis method is provided, applied to an electronic device. The method includes: acquiring a first text to be generated as speech; determining a first audio that matches the first text, wherein the sound scene in which the first audio is located matches the content of the first text; acquiring text features of the first text; and obtaining a second audio based on the first audio and the text features, wherein the second audio is associated with the first audio and the first speech, the first speech being speech obtained based on the text features and having the same content as the first text, wherein in the second audio, the first audio is used as background sound for the first speech.

[0007] The above solution applies the concept of sound scenes to voice interaction scenarios. For the text used to generate the voice signal, it determines the sound scene matching the text and identifies the corresponding first audio. This first audio is then used as the background noise for the voice signal, synthesizing a voice signal with background noise, i.e., the second audio. Playing this second audio to the user, compared to simply playing the voice signal, enhances the auditory experience, improves the effect of the terminal device's sound output, thereby increasing the user's perception and understanding of the text content, optimizing the interaction between the electronic device and the user, and ultimately improving the user experience.

[0008] In one possible embodiment, determining the first audio that matches the first text includes: based on the content of the first text, determining the category of the sound scene that matches the first text as the first sound scene in a preset sound scene database, wherein the preset sound scene database also includes the first audio corresponding to the first sound scene.

[0009] It is understandable that by pre-setting a sound scene database, corresponding audio can be configured for various sound scenes, so as to creatively apply the concept of sound scenes to voice interaction scenarios and conveniently obtain the first audio that matches the text.

[0010] In one possible embodiment, based on the content of the first text, determining the category of the sound scene matching the first text as the first sound scene in a preset sound scene database includes: obtaining short-time features of the first text, the short-time features being used to characterize the relevance of sub-texts corresponding to different time frames in the first text, the sub-texts including one or more characters; mapping the short-time features to a feature domain that is easy to classify, to obtain mapped features; classifying the mapped features to obtain indication information of the sound scene matching the first text, the indication information indicating the first sound scene.

[0011] The above solution provides a possible implementation method for determining the first sound scene of text matching, making it possible to apply the concept of sound scene to voice interaction scenarios.

[0012] In one possible embodiment, obtaining a second audio based on a first audio and text features includes: inputting indication information and text features into a modified flow model to obtain a first acoustic feature, the first acoustic feature being used to characterize a first speech using the first audio as background sound; and decoding the first acoustic feature to obtain the second audio.

[0013] The above scheme uses a correction flow module to process text features. Compared with the scheme that processes text features through a diffusion model, it requires significantly fewer iterations during the generation process, which can reduce computational costs and improve inference speed, thereby improving generation efficiency.

[0014] In one possible embodiment, the text features include semantic features and phoneme length features. Obtaining the text features of the first text includes: obtaining the semantic features of the first text, which are used to characterize the semantic information of the first text; and obtaining the phoneme length features of the first text based on the semantic features, which are used to indicate the phoneme length corresponding to each character in the first text, where the phoneme length is the duration of the phoneme.

[0015] The above scheme provides possible implementation methods for text features. Since the generated speech is consistent with the content of the text, semantic features are crucial for successful speech generation. Because a phoneme is the smallest unit of speech, and phoneme length features are phoneme features corresponding to the text, phoneme length features are also very important for successful speech generation.

[0016] In one possible embodiment, inputting indication information and text features into a modified flow model to obtain a first acoustic feature includes: inputting the indication information, text features, and randomly sampled first Gaussian noise into the modified flow model to obtain a first derivative of the first Gaussian noise; performing Euler sampling on the first derivative to obtain a first sampled signal; inputting the indication information, text features, and the first sampled signal into the modified flow model to obtain a second derivative of the first sampled signal; performing Euler sampling on the second derivative to obtain a second sampled signal; and obtaining the first acoustic feature based on the second sampled signal.

[0017] It is understandable that, theoretically, the generated result can be obtained through single-step sampling after using the modified flow network. Therefore, the sampling process is relatively simple. In this case, the effect achieved by using simple or complex sampling methods is almost the same. Therefore, this application prefers a simple sampling method, such as Eulerian sampling. Alternatively, other simple sampling methods can also be used, and this application is not limited to them. Furthermore, when K≥2, the final generation effect can be improved compared to sampling only once.

[0018] In one possible embodiment, the method further includes: acquiring a clean sample speech signal and a corresponding sample text, wherein the clean sample speech signal and the sample text have the same content; generating a sample background sound audio and a sample sound scene index corresponding to the sample text; fusing the clean sample speech signal and the sample background sound audio to obtain a sample speech signal with sample background sound; acquiring the sample acoustic features of the sample speech signal with sample background sound; generating a first sample signal based on randomly generated second Gaussian noise and the sample acoustic features; obtaining sample semantic features and sample phoneme length features based on the sample text; inputting the first sample signal, sample semantic features, and sample phoneme length features into a modified flow model to obtain a sample derivative; calculating a loss function based on the sample derivative and target data, wherein the target data is the difference obtained by subtracting the second Gaussian noise from the sample acoustic features; and updating the parameters of the modified flow model based on the loss function.

[0019] The above scheme uses the difference between the acoustic features of the sample and the second Gaussian noise as the training target to realize the training process of the modified flow model, so that the inference side of the modified flow model can improve the audio synthesis effect and the inference speed compared with the diffusion model.

[0020] In a second aspect, this application provides an electronic device including one or more processors and one or more memories; wherein the one or more memories are coupled to one or more processors, and the one or more memories are used to store computer program code, the computer program code including computer instructions, which, when executed by one or more processors, cause the electronic device to perform the method described in the first aspect and any possible implementation thereof.

[0021] Thirdly, embodiments of this application provide a chip system applied to an electronic device. The chip system includes one or more processors, which are used to invoke computer instructions to cause the electronic device to perform the methods described in the first aspect and any possible implementation thereof.

[0022] Fourthly, this application provides a computer-readable storage medium including instructions that, when executed on an electronic device, cause the electronic device to perform the method described in the first aspect and any possible implementation thereof.

[0023] Fifthly, this application provides a computer program product containing instructions that, when the computer program product is run on an electronic device, cause the electronic device to perform the method described in the first aspect and any possible implementation thereof.

[0024] Understandably, the electronic device provided in the second aspect, the chip system provided in the third aspect, the computer storage medium provided in the fourth aspect, and the computer program product provided in the fifth aspect are all used to execute the method provided in this application. Therefore, the beneficial effects they can achieve can be referred to the beneficial effects in the corresponding methods, and will not be repeated here. Attached Figure Description

[0025] Figure 1 A schematic diagram illustrating an example of a voice interaction scenario provided in an embodiment of this application;

[0026] Figure 2 A schematic diagram illustrating the sound output effect of the terminal device provided in the embodiments of this application;

[0027] Figure 3 An exemplary flowchart of an audio synthesis method provided in this application embodiment;

[0028] Figure 4 A schematic flowchart of the audio synthesis method 100 provided in the embodiments of this application;

[0029] Figure 5 A schematic flowchart illustrating a specific example of the sound scene classifier 204 provided in an embodiment of this application;

[0030] Figure 6 A schematic flowchart illustrating the specific inference process of the audio synthesis module 205 based on the modified flow model provided in this application embodiment;

[0031] Figure 7 A schematic diagram illustrating an example of the training process of a modified stream network provided in an embodiment of this application;

[0032] Figure 8 A schematic diagram of the hardware structure of an electronic device 1000 provided in an embodiment of this application;

[0033] Figure 9 A block diagram of a software system for an electronic device 1000 provided in an embodiment of this application. Detailed Implementation

[0034] To make the objectives, technical solutions, and advantages of this application clearer, the embodiments of this application will be described in further detail below with reference to the accompanying drawings.

[0035] Speech synthesis technology is often used in scenarios where terminal devices interact with users. For example, in scenarios where users interact with terminal devices via voice, voice assistants can use speech synthesis technology to speak to users.

[0036] Figure 1 This is a schematic diagram illustrating an example of a voice interaction scenario provided in an embodiment of this application.

[0037] like Figure 1 As shown, the electronic device detects the user saying, "Hello YOYO, can you tell me about Weihai?" The smart assistant panel on interface 1 displays the text matching the user's voice, "Can you tell me about Weihai?", followed by interface 2, which plays the audio, "Weihai is a beautiful coastal city...". The smart assistant panel on interface 2 displays the text matching the user's voice and the audio in a dialogue format: "Weihai is a beautiful coastal city with abundant natural and cultural landscapes. Weihai's seascape attracts visitors from all over with its fresh air, azure waters, and charming scenery. Weihai not only has beautiful seascapes but also a rich culinary culture. As a coastal city, Weihai's seafood is naturally a must-try. Tourists can buy fresh seafood at the local seafood market and then take it to a restaurant to have it cooked and enjoy authentic Weihai seafood."

[0038] Currently, taking voice assistants as an example, the primary function is to generate audio for conversations with users. The main generation logic involves first obtaining the text corresponding to the audio, and then generating human voice based on the text. Building upon this, different speech synthesis technologies focus on optimizing one or more aspects of the audio, such as timbre, speech rate, and emotion, to make it as close as possible to a realistic human voice. However, as the intelligence of terminal devices increases, users' demands for their voices are also rising.

[0039] Therefore, optimizing the effect of terminal devices communicating with users has become an urgent problem to be solved.

[0040] In view of this, this application provides an audio synthesis method in which an electronic device generates an audio signal with background noise based on text, wherein the audio signal includes human voice and background noise.

[0041] by Figure 1 Taking the text in this application as an example, the audio synthesis method can generate corresponding audio with background sounds based on the text related to the seascape ("Weihai is a beautiful coastal city with rich natural and cultural landscapes. Weihai's seascape attracts visitors from all over the world with its fresh air, azure waters, and charming scenery."). This audio includes human voices and the sound of waves crashing on the shore. The voice assistant then plays this audio to the user. Based on the background sounds used to enhance the seascape rendering, compared to simply using human voices to describe the seascape, this provides a more immersive experience for the user, optimizing the interaction between the electronic device and the user and improving the user experience.

[0042] return Figure 1 Taking the text in this application as an example, the audio synthesis method can generate corresponding audio with background sounds based on the text related to food ("As a coastal city, Weihai's seafood is naturally a must-try delicacy. Tourists can buy fresh seafood at the local seafood market and then take it to a restaurant to have it cooked and taste authentic Weihai seafood."). This audio includes human voices and background sounds of multiple people dining together. The voice assistant plays this audio to the user. The background sounds of multiple people dining together create an atmosphere that the food is very popular. Compared to simply introducing the food with human voices, this allows users to more clearly experience the popularity of the food, optimizes the interaction between electronic devices and users, and improves the user experience.

[0043] Figure 2 This is a schematic diagram illustrating the sound output effect of the terminal device provided in an embodiment of this application. Figure 2 The following explanation uses the time-domain graph and spectrogram of the audio signal corresponding to the text "Weihai is a beautiful coastal city" played by a terminal device to a user as an example.

[0044] In a time-domain plot, the horizontal axis represents time, and the vertical axis represents amplitude. The time-domain plot represents the amplitude of the audio signal in the time domain. A spectrogram, also known as a spectrum diagram, time-frequency diagram, or feature diagram, has the same horizontal axis as the time-domain plot, and the vertical axis represents frequency.

[0045] Spectrograms represent audio signals through the distribution of multiple time-frequency points; in other words, they represent the probability distribution of audio signals, specifically the distribution of each time-frequency point. The brightness or darkness of a time-frequency point indicates its energy level; for example, a brighter or lighter point indicates greater energy, and vice versa. In other words, a spectrogram can represent the energy level of an audio signal at different times and frequencies. For instance, a region in a spectrogram that appears black indicates no signal in that region; a region that is close to bright yellow (in a color spectrogram) or close to white (in a grayscale spectrogram) indicates that the signal energy in that region is very high.

[0046] Figure 2 (a) is a schematic diagram of the time-domain graph #1 and spectrogram #1 of the audio signal obtained based on traditional speech synthesis technology.

[0047] As can be seen from spectrogram #1, the distribution of time-frequency points clearly shows the characteristics of human voice, namely the fundamental frequency and harmonics. Furthermore, the formants in the harmonic components—a typical characteristic of human voice—are clearly visible. In regions where human voice characteristics are not displayed (e.g., regions A3 and A4), there is almost no distribution of time-frequency points.

[0048] As can be seen from time-domain plot #1, the magnitude of the audio signal's amplitude in the time domain corresponds to the brightness of the time-frequency points in spectrogram #1. In the audio signal excluding regions containing human voices (e.g., regions A1 and A2), the amplitude is almost zero.

[0049] Depend on Figure 2 As shown in (a), traditional speech synthesis technology mainly focuses on human voices and neglects background noise, and even less so on generating matching background noise based on text; thus ignoring the importance of background noise in voice interaction scenarios. Furthermore, synthesized speech without background noise lacks expressiveness and appeal in many scenarios.

[0050] Figure 2 (b) is a schematic diagram of the time-domain diagram #2 and spectrogram #2 of the audio signal obtained based on the audio synthesis method provided in this application.

[0051] As can be seen from spectrogram #2, the distribution of time-frequency points reveals vocal features similar to those in spectrogram #1, as well as non-vocal features almost entirely absent in spectrogram #1. The time-frequency points exhibiting non-vocal features not only overlap with those exhibiting vocal features but also lie in regions where vocal features are absent (e.g., regions B3 and B4, and the gaps between formants). In other words, in spectrogram #2, time-frequency points exhibiting vocal features are surrounded by time-frequency points exhibiting non-vocal features.

[0052] As can be seen from time-domain plot #2, the magnitude of the audio signal's amplitude in the time domain corresponds to the brightness of the time-frequency points in spectrogram #2. The amplitude of the regions in the audio signal that do not include human voices (such as regions B1 and B2) is still significantly smaller than the amplitude of the regions that include human voices.

[0053] Depend on Figure 2 As can be seen from (b) in this application, the audio signal generated by the audio synthesis method includes not only the human voice corresponding to the text, but also background sound. In the time domain diagram #2, the amplitude of the background sound is significantly lower than that of the human voice. While the background sound is helpful in creating an atmosphere or enhancing the atmosphere of the human voice, it does not interfere with the human voice and ensures the clarity and intelligibility of the human voice.

[0054] Figure 3 This is an exemplary flowchart of an audio synthesis method provided in an embodiment of this application.

[0055] like Figure 3 As shown, the audio synthesis method provided in this application involves obtaining text for generating speech, extracting text features (such as semantic features and phoneme length features) from the text, and performing sound scene classification processing on the text to obtain a sound scene matching the text. The preset sound scene database includes background audio corresponding to the sound scene. The text features and the indication information of the sound scene are input into a modified flow model to obtain a speech signal with background sound, which is used as the final synthesized audio signal.

[0056] In essence, a soundscape refers to the acoustic environment perceived, experienced, and understood by an individual or group within a specific context. Soundscapes can be broadly categorized into natural environmental soundscapes and human activity soundscapes. Natural environmental soundscapes include typical soundscapes corresponding to large-scale natural environments such as oceans and deserts, such as the babbling of flowing water and the calls of deer. Human activity soundscapes can be further divided into social life soundscapes and artistic creation soundscapes, such as the shouts used in traditional farming, boat pulling, and construction work in rural areas, the hawking of vendors and the honking of vehicles in cities, as well as music, opera, and folk arts from various regions. The study of soundscapes helps to describe and analyze "landscapes from an observer's perspective." From a biological perspective, both auditory and visual signals can enhance an individual's (including humans') perception and cognition of a landscape.

[0057] The above solution applies the concept of sound scenes to voice interaction scenarios. For the text used to generate the voice signal, it determines the sound scene matching the text and the corresponding audio for that sound scene. This audio is then used as the background sound of the voice signal, synthesizing a voice signal with background noise, which is then played back to the user. Compared to simply playing the voice signal to the user, adding background noise optimizes the auditory experience, improves the effect of the terminal device's sound output, enhances the user's perception and understanding of the text content, optimizes the interaction between the electronic device and the user, and improves the user experience.

[0058] It's also understandable that traditional diffusion models are a general generative modeling technique used to create high-dimensional, perceptual data, such as images and videos. By simulating the physical diffusion process, data is progressively transformed into noise, and then the inverse process is learned to gradually recover the original data from the noise, achieving high-quality generation results. The generation process requires multiple iterations, resulting in high computational costs and slow inference speeds. Significant computational resources are needed during training and generation, especially when generating high-resolution images or high-quality audio.

[0059] The Rectified Flow involved in this application is an emerging generative model that constructs a flow-based generative model through linear interpolation and then uses flow operations to draw the trajectory of a straight line. It connects data and noise with a straight line, offering better theoretical properties and conceptual simplicity compared to diffusion models. For example, it requires significantly fewer iterations during the generation process; theoretically, one iteration (differentiating the line) is sufficient to obtain the generated result, reducing computational costs and increasing inference speed, thereby improving generation efficiency.

[0060] The above scheme uses a correction flow module to process text features. Compared with the scheme that processes text features through a diffusion model, it requires significantly fewer iterations during the generation process, which can reduce computational costs and improve inference speed, thereby improving generation efficiency.

[0061] The following will combine Figures 4 to 7 This application provides a detailed description of the audio synthesis method provided.

[0062] Figure 4 This is a schematic flowchart of an audio synthesis method 100 provided in an embodiment of this application.

[0063] S101, Obtain the first text of the speech to be generated.

[0064] For example, the first text may be uploaded by a user, downloaded from the network, or generated by a neural network model configured on an electronic device. For instance, Figure 1In the scenario shown, the first text, "Weihai is a beautiful coastal city...", is generated by the neural network model upon which the voice assistant application relies based on the user's voice.

[0065] The following example uses text as the first text and the implementation of S101 by the text acquisition module 201 as an example for illustration.

[0066] Text acquisition module 201 acquires the text used for audio synthesis: text = [text0, text1, ..., text...]. p ,…text P-1 ], where P represents the length of the text. The length of the text can be understood as the number of characters included in the text. The text acquisition module 201 transmits the text to the text encoder 202 and the sound scene classifier 204.

[0067] S102, determine the first audio that matches the first text.

[0068] Among them, the sound scene in which the first audio is located matches the content of the first text.

[0069] For example, based on the content of the first text, the category of the sound scene that matches the first text is determined in the preset sound scene database as the first sound scene. The preset sound scene database also includes the first audio corresponding to the first sound scene.

[0070] The following uses index ID sense The sound scene is used as an example of the first sound scene, and the implementation of S102 by the sound scene classifier 204 is used as an example for illustration.

[0071] The sound scene classifier 204 analyzes the text (text) and classifies the sound scene information contained in the text based on its semantic information (or content), thus obtaining the index ID of the sound scene. sense For example, the text "Weihai is a beautiful coastal city" contains the sound scene information "coastal city". Assuming the sound scene database includes a sound scene with index 003 whose category is "sea", then the sound scene index is 003. The index ID of this sound scene is... sense Used to indicate the background audio corresponding to the sound scene.

[0072] For example, index ID sense An index is created for the sound scene within a pre-defined sound scene category database (also referred to as a preset sound scene database in this application). This database includes multiple categories of sound scenes, each category corresponding to one or more background audio files. Each type of sound scene is identified using an index. For example, ID... senseThe identified sound scenario is a rainy day. This scenario can correspond to multiple background audio files; for example, background audio file 1 could be the sound of light drizzle, and background audio file 2 could be the sound of heavy rain. Another example is the ID. sense The identified sound scene is birdsong. A sound scene can correspond to multiple background audio files. For example, background audio file 3 is the sound of a crow calling, and background audio file 2 is the sound of many birds chirping. The sound scene classifier 204 selects the sound scene that matches the text from the sound scene category database based on the text's feature information and outputs the index corresponding to that sound scene.

[0073] It is understandable that the sound scene category database is used during the training of the audio synthesis module 205 based on the modified flow model, but it may not be involved in the training of the sound scene classifier 204.

[0074] Specifically, the implementation of the sound scene classifier 204 will be discussed below. Figure 5 Let me introduce it.

[0075] The sound scene classifier 204 will use the index ID of the sound scene. sense The audio is transmitted to the audio synthesis module 205 based on the modified stream model.

[0076] S103, Obtain the text features of the first text.

[0077] For example, text features include semantic features and phoneme length features.

[0078] The hidden layer representation tensor will be discussed below. text As a possible example of semantic features, the length vector L pre This is an example of a possible phoneme length feature, which will be explained below.

[0079] Text encoder 202 encodes text using a pre-trained model to obtain the text's hidden layer representation tensor. text Among them, the hidden layer representation tensor is... text It can be used to represent the semantic information of text. The text encoder 202 represents the hidden layer tensor. text The data is transmitted to the length prediction module 203 and the audio synthesis module 205 based on the modified stream model.

[0080] Length prediction module 203 uses a pre-trained model based on the hidden layer representation tensor. text Predict the phoneme length of each character in the text and output a length vector L consisting of the phoneme lengths of all characters. pre .

[0081] It is understood that a character comprises one or more phonemes, and the phoneme length can be understood as the time frame or spectrum frame in which the phoneme lasts. The phoneme length of each character can be understood as the time frame or spectrum frame in which the phonemes included in each character last.

[0082] It can also be understood that the length vector L pre This includes P numbers arranged in character order, where each of the P numbers represents the phoneme length of one of the P characters in the text. In other words, the length vector L... pre Indicates the phoneme length of each character in the text.

[0083] Length prediction module 203 will predict the length vector L pre The audio is transmitted to the audio synthesis module 205 based on the modified stream model.

[0084] S104. Based on the first audio and text features, obtain the second audio.

[0085] The second audio is associated with the first audio and the first speech. The first speech is a speech obtained based on text features and is the same as the content of the first text. In the second audio, the first audio is used as the background sound of the first speech.

[0086] Using the first audio as background sound for the first speech can also be understood as having a significantly lower amplitude than the first speech. In this way, the first audio does not affect the clarity and intelligibility of the first speech, but can also play a role in creating atmosphere or enhancing mood, thereby optimizing the user's perception of the speech content and improving the user experience.

[0087] For example, the ratio of the amplitude of the first audio to the amplitude of the first speech is lower than a first threshold; or, the amplitude of the first audio falls within a first amplitude range, the amplitude of the first speech falls within a second amplitude range, the minimum amplitude of the second amplitude range is greater than the maximum amplitude of the first amplitude range, and the difference between the two is greater than the second threshold. Alternatively, other conditions may be met to avoid the first audio amplitude being too large and affecting the clarity and intelligibility of the first speech, which is not limited in this application.

[0088] One implementation of S104 is based on the indication information of the first sound scene where the first audio is located (e.g., index ID). sense The first acoustic feature is obtained by combining the semantic features and phoneme length features of the text with the first Gaussian noise input modified flow model obtained by random sampling. For example, S104 is implemented by the audio synthesis module 205 based on the modified flow model.

[0089] The audio synthesis module 205 based on the modified flow model will index ID. sense Hidden layer representation tensor text and length vector Lpre As conditional information, acoustic features AF of synthesized speech with background noise are generated from randomly sampled white noise samples (i.e., the first Gaussian noise mentioned above) using a modified flow model. For example, acoustic features AF can be represented as a Mel spectrum.

[0090] Specifically, the following will combine Figure 6 This paper details the possible implementations of the audio synthesis module 205 based on the modified flow model.

[0091] Understandably, using the correction flow module to process text features requires significantly fewer iterations during the generation process compared to using a diffusion model, which reduces computational costs and increases inference speed, thereby improving generation efficiency.

[0092] The audio synthesis module 205 based on the modified flow model transmits the acoustic features AF to the audio decoder 206.

[0093] The audio decoder 206 decodes the acoustic feature AF to obtain a playable time-domain audio signal y(t).

[0094] The audio decoder 206 transmits the audio signal y(t) to the output or playback module 207.

[0095] The output or playback module 207 outputs the playable time-domain audio signal y(t) to the downstream module or speaker for playback. This solution applies the concept of sound scenes to voice interaction scenarios. For the text used to generate the voice signal, it determines the sound scene matching the text and the corresponding audio for that sound scene. This audio is then used as the background sound of the voice signal, synthesizing a voice signal with background noise, which is then played to the user. Compared to simply playing the voice signal to the user, adding background noise optimizes the auditory experience, improves the effect of the terminal device's sound output, enhances the user's perception and understanding of the text content, optimizes the interaction between the electronic device and the user, and improves the user experience.

[0096] Figure 5 A schematic flowchart illustrating a specific example of the sound scene classifier 204 provided in an embodiment of this application.

[0097] The text feature extraction module 301 receives text from the text acquisition module 201 and extracts vector features of the time-series text based on the text. text .

[0098] Understandable, vector feature text It is used to establish the correlation between text and time frames, and is also related to the content or voice information of the text.

[0099] The text feature extraction module 301 extracts vector features. text Transmitted to the timing relationship construction module 302.

[0100] The temporal relationship construction module 302, through the temporal neural network module, based on vector features... text To model the temporal relationships within text and output short-term features with temporal correlation. tmp .

[0101] The temporal relationship within the text can be understood as the correlation between the sub-texts in the text and the sub-texts corresponding to different time frames. For example, a sub-text can be one or more characters.

[0102] For example, a temporal neural network module is a combination of one or more of a recurrent neural network, a gated neural network, and a Transformer.

[0103] The temporal relationship construction module 302 will construct short-term features with temporal correlation. tmp Transmitted to fully connected module 303.

[0104] The fully connected module 303 will store short-term features with temporal correlation. tmp Map the data to a feature domain that facilitates classification, and output the mapped features. map .

[0105] The fully connected module 303 will map the feature. map Transmitted to classification module 304.

[0106] Classification module 304, mapping features map Classify and output index IDs sense .

[0107] For example, the training data of the classification module 304 includes input data and label data. The input data is sample audio data from the aforementioned preset sound scene category database, and the label data is the sample sound scene index corresponding to the sample audio data. It can be understood that multiple sound scenes included in the sound scene category database can serve as sample sound scenes. Each sample sound scene index corresponds one-to-one with a sample sound scene, and the sample sound scene index is used to uniquely indicate a sample sound scene. One sample sound scene can correspond to one or more sample audio data.

[0108] For details, please refer to the beneficial effects related to the sound scene classifier 204 mentioned above.

[0109] Figure 6A schematic flowchart illustrating the specific inference process of the audio synthesis module 205 based on the modified flow model provided in the embodiments of this application.

[0110] It is understandable that a total of K sampling steps are performed, where K is a pre-set number of sampling steps and is an integer greater than 1, preferably K ≥ 2, for example, K = 3. Although theoretically the modified flow can obtain the generated result in one iteration, this application prefers K ≥ 2 to optimize the generation effect.

[0111] First execution Figure 6 In the process shown, k = K. Where 1 ≤ k ≤ K and both k and K are integers.

[0112] like Figure 6 As shown, index ID sense Length vector L pre and hidden layer representation tensor text Referring to the descriptions above, it is transmitted to the modified stream network inference module 403.

[0113] Gaussian noise sampling module 401 randomly samples Gaussian noise (i.e., the first Gaussian noise mentioned above) from a Gaussian distribution with zero mean and variance of 1. K Alternatively, the sampled Gaussian noise is x when k = K. k .

[0114] Gaussian noise sampling module 401 will sample Gaussian noise x K Transmitted to the corrected stream network inference module 403.

[0115] Sampling time update module 402, according to the formula Generate the sampling time for the k-th step, where t min It can be 0.003, t max The value can be 1, which indicates the initial execution. Figure 6 The process shown is such that k = K, therefore t k =0.

[0116] Sampling time update module 402, will t k Transmitted to the corrected stream network inference module 403.

[0117] The input data for the corrected streaming network inference module 403 includes x k t k Index ID sense Length vector L pre and hidden layer representation tensor text , where x k and t k As an observation signal or reference signal for the corrected flow network inference module 403, the index ID senseLength vector L pre and hidden layer representation tensor text As conditional information, the output data of the corrected stream network inference module 403 is x. k For t k The derivative v k .

[0118] For example, the modified flow network inference module 403 can be implemented based on one or a combination of network models such as Noise Conditional Scoring Network (NCSN)++, Convolutional Neural Network (CNN), Convolutional Recurrent Neural Network (CRNN), U-shaped Neural Network Pooled Subgraph Structure (U-Net).

[0119] Specifically, the training process of the modified flow network will be discussed below. Figure 7 A detailed introduction will be provided.

[0120] The modified stream network inference module 403 will convert the derivative v k Transmitted to Euler sampling module 404.

[0121] Euler sampling module 404, through formula Perform sampling and update the sampling count k = k-1.

[0122] It is understandable that, after using the modified flow, theoretically, single-step sampling can be used to obtain the generated result. Therefore, the sampling process is relatively simple. In this case, the effect achieved by using simple or complex sampling methods is almost the same. Therefore, this application prefers a simple sampling method, such as Eulerian sampling. Alternatively, other simple sampling methods can also be used, and this application is not limited to them. Furthermore, when K≥2, the final generation effect can be improved compared to sampling only once.

[0123] Module 405 determines: Has the preset number of sampling steps been reached?

[0124] For example, determining whether the preset number of sampling steps has been reached can be done by checking if the number of sampling steps, k, is equal to 0. If k = 0, it is determined that the preset number of sampling steps has been reached, and then x... k+1 The acoustic feature AF output is given as background noise; if k≠0, then for x k Sampling continues, meaning that the sampling time update module 402 continues to generate the updated t based on the updated k (updated by the Euler sampling module 404). k And will update x k and t kAs an observation signal or reference signal, and as an index ID sense Length vector L pre and hidden layer representation tensor text As conditional information input to the modified flow network inference module 403, execution continues. Figure 6 The process is shown below.

[0125] In other words, when K≥2 and k>1, starting from the second generation of the sampling time by the sampling time update module 402 (including the second generation), in each subsequent sampling, x will not be changed. K Instead of inputting the corrected flow network inference module 403, it uses the x obtained after the previous sampling. k Input correction flow network inference module 403.

[0126] The above scheme utilizes the index ID of the sound scene in the inference process of the audio synthesis module 205 based on the modified flow model. sense This is used to indicate the background audio corresponding to the sound scene, so as to finally generate acoustic features (AF) with background audio.

[0127] It is understandable that, when k = K, the derivative v k As an example of the first derivative mentioned above, x obtained by Euler sampling k-1 This is an example of the first sampled signal described above. When k < K, the derivative v k As an example of the second derivative mentioned above, x obtained by Euler sampling k-1 This is an example of the second sampled signal described above.

[0128] Figure 7 This is a schematic diagram illustrating an example of the training process of a modified stream network provided in an embodiment of this application.

[0129] The clean speech acquisition module 501 is used to acquire sample clean audio signals Yx(t) from a clean sample audio dataset.

[0130] The clean speech acquisition module 501 transmits the sample clean audio signal Yx(t) to the audio fusion module 504.

[0131] The text acquisition module 502 acquires the sample text Ytext corresponding to the clean audio signal of the sample.

[0132] For example, if the sample text dataset includes sample text that can form an audio-text data pair with the sample clean audio signal, then the sample text Ytext corresponding to the sample clean audio signal is obtained from the sample text dataset.

[0133] For example, if the sample text dataset does not include sample text corresponding to the clean audio signal, an automatic speech recognition system can be used to identify the text corresponding to the clean audio signal and use it as the sample text Ytext.

[0134] The text acquisition module 502 transmits the sample text Ytext to the text encoder 509.

[0135] The sound scene generation module 503 randomly generates a sample sound scene from the existing sound scene category database, transmits the corresponding sample background audio Yn(t) to the audio fusion module 504, and indexes the corresponding sample sound scene YID. sense Transmitted to the correction stream network 511.

[0136] The audio fusion module 504 fuses the sample background sound audio Yn(t) with the sample clean audio signal Yx(t) at a certain signal-to-noise ratio to obtain the sample speech signal Yy(t) with the sample background sound.

[0137] The audio fusion module 504 transmits the sample speech signal Yy(t) to the audio feature conversion module 505.

[0138] The audio feature conversion module 505 converts the sample speech signal Yy(t) into sample acoustic features YAF with sample background sound.

[0139] The audio feature conversion module 505 transmits the sample acoustic features YAF with sample background sound to the sample generation module 508.

[0140] Sampling time generation module 506, this module is used to randomly generate a Yt min To Yt max The decimal between these two values ​​is taken as the sample sampling time Yt, where Yt min For the minimum value, such as 0.003, Yt max It is the maximum value, such as 1.

[0141] The sampling time generation module 506 transmits the sample sampling time Yt to the Gaussian noise generation module 507 and the corrected flow network 511.

[0142] Gaussian noise generation module 507 generates sample Gaussian noise Ynoise with zero mean and standard deviation σ(Yt) based on time t. The standard deviation can be calculated in various ways, such as...

[0143] The Gaussian noise generation module 507 outputs the generated Gaussian noise Ynoise to the sample generation module 508.

[0144] The sample generation module 508 adds the sample Gaussian noise Ynoise and the sample acoustic feature YAF with sample background noise to obtain the noisy sample Ynoisy_AF.

[0145] The sample generation module 508 transmits the noisy sample Ynoisy_AF to the corrected stream network 511.

[0146] Text encoder 509 encodes sample text Ytext using a pre-trained model to obtain the sample hidden layer representation tensor Ylatent of sample text Ytext. text Among them, the sample hidden layer representation tensor Ylatent text It can be used to characterize the semantic information of sample text Ytext.

[0147] The text encoder 509 represents the sample hidden layer representation tensor Ylatent. text The data is transmitted to the length prediction module 510 and the corrected flow network 511.

[0148] The length prediction module 510 uses a pre-trained model to predict the Ylatent tensor based on the hidden layer representation of the samples. text Predict the phoneme length of each character in the sample text Ytext, and output the sample length vector YL composed of the phoneme lengths of all characters. pre .

[0149] For an understanding of phoneme length and sample length vector, please refer to the description of the length vector of phoneme length above.

[0150] The length prediction module 510 will generate the sample length vector YL pre Transmitted to the correction stream network 511.

[0151] Modify the flow network 511, and use the sample hidden layer representation tensor Ylatent text and sample length vector YL pre As conditional information, the modified flow network 511 generates the sample derivative Yv based on the noisy sample Ynoisy_AF and the sample time Yt. The sample derivative Yv is the modified flow.

[0152] For example, the modified flow network 511 can be implemented based on one or a combination of network models such as NCSN, CNN, CRNN, U-Net, etc.

[0153] The modified flow network 511 transmits the sample derivative Yv to the loss function calculation module 512.

[0154] The loss function calculation module 512 calculates the loss function corresponding to the sample derivative Yv: loss = mse(Yv, YAF - Ynoise), where the mean squared error (mse) is a commonly used loss function in regression tasks. It is defined as the average of the squared differences between the predicted and true values.

[0155] The loss function calculation module 512 transmits the loss function loss to the network parameter update module 513.

[0156] The network parameter update module 513 updates the parameters of the corrected flow network 511 according to the loss function loss.

[0157] The above scheme uses YAF-Ynoise as the training target to realize the training process of the modified flow model, so that the inference side of the modified flow model can improve the audio synthesis effect and inference speed compared with the diffusion model.

[0158] Figure 8 This is a schematic diagram of the hardware structure of an electronic device 1000 provided in an embodiment of this application. See also... Figure 8 The electronic device 1000 may include a processor 1010, an external memory interface 1020, an internal memory 1021, a universal serial bus (USB) interface 1030, a charging management module 1040, a power management module 1041, a battery 1042, an antenna 1, an antenna 2, a mobile communication module 1050, a wireless communication module 1060, an audio module 1070, a speaker 1070A, a receiver 1070B, a microphone 1070C, a headphone jack 1070D, a sensor module 1080, buttons 1090, a motor 1091, an indicator 1092, a camera 1093, a display screen 1094, and a subscriber identification module (SIM) card interface 1095, etc. The sensor module 1080 may include a pressure sensor 1080A, a gyroscope sensor 1080B, a barometric pressure sensor 1080C, a magnetic sensor 1080D, an accelerometer sensor 1080E, a distance sensor 1080F, a proximity sensor 1080G, a fingerprint sensor 1080H, a temperature sensor 1080J, a touch sensor 1080K, an ambient light sensor 1080L, a bone conduction sensor 1080M, etc.

[0159] It is understood that the structures illustrated in the embodiments of this application do not constitute a specific limitation on the electronic device 1000. In other embodiments of this application, the electronic device 1000 may include more or fewer components than illustrated, or combine some components, or split some components, or have different component arrangements. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

[0160] The processor 1010 may include one or more processing units, such as an application processor (AP), a modem processor, a graphics processing unit (GPU), an image signal processor (ISP), a controller, a memory, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural network processing unit (NPU).

[0161] The controller can be the nerve center and command center of the electronic device 1000. The controller can generate operation control signals according to the instruction opcode and timing signals to complete the control of instruction fetching and execution.

[0162] The processor 1010 may also include a memory for storing instructions and data. In some embodiments, the memory in the processor 1010 is a cache memory. This memory can store instructions or data that the processor 1010 has just used or that are used repeatedly. If the processor 1010 needs to use the instruction or data again, it can retrieve it directly from this memory. This avoids repeated accesses, reduces the waiting time of the processor 1010, and thus improves the efficiency of the system.

[0163] In some embodiments, the processor 1010 may include one or more interfaces, such as an inter-integrated circuit (I2C) interface, an inter-integrated circuit sound (I2S) interface, a pulse code modulation (PCM) interface, a universal asynchronous receiver / transmitter (UART) interface, a mobile industry processor interface (MIPI), a general-purpose input / output (GPIO) interface, a subscriber identity module (SIM) interface, and / or a universal serial bus (USB) interface, etc.

[0164] The MIPI interface can be used to connect the processor 1010 to peripheral devices such as the display screen 1094 and the camera 1093. The MIPI interface includes a camera serial interface (CSI) and a display serial interface (DSI). In some embodiments, the processor 1010 and the camera 1093 communicate via the CSI interface to enable the electronic device 1000 to perform its shooting function. The processor 1010 and the display screen 1094 communicate via the DSI interface to enable the electronic device 1000 to perform its display function.

[0165] It is understood that the interface connection relationships between the modules illustrated in the embodiments of this application are merely illustrative and do not constitute a structural limitation on the electronic device 1000. In other embodiments of this application, the electronic device 1000 may also employ different interface connection methods or combinations of multiple interface connection methods as described in the above embodiments.

[0166] Digital signal processors (DSPs) are used to process digital signals. Besides digital image signals, they can also process other digital signals. For example, when electronic device 1000 is selecting a frequency, the DSP is used to perform Fourier transforms on the frequency energy.

[0167] Video codecs are used to compress or decompress digital video. Electronic device 1000 can support one or more video codecs. Thus, electronic device 1000 can play or record videos in various encoding formats, such as Moving Picture Experts Group (MPEG) 1, MPEG2, MPEG3, MPEG4, etc.

[0168] NPU stands for Neural Network (NN) Computing Processor. By borrowing the structure of biological neural networks, such as the transmission patterns between neurons in the human brain, it can rapidly process input information and continuously learn on its own. NPUs can enable intelligent cognitive applications in electronic devices, such as image recognition, facial recognition, speech recognition, and text understanding.

[0169] Electronic device 1000 can implement audio functions, such as music playback and recording, through audio module 1070, speaker 1070A, receiver 1070B, microphone 1070C, headphone jack 1070D, and application processor. Buttons 1090 include a power button and volume buttons. Buttons 1090 can be mechanical buttons or touch buttons. Electronic device 1000 can receive button input and generate key signal inputs related to user settings and function control of electronic device 1000.

[0170] The software system of electronic device 1000 will be described next.

[0171] For example, electronic device 1000 may be a mobile phone. The software system of electronic device 1000 may adopt a layered architecture, event-driven architecture, microkernel architecture, microservice architecture, or cloud architecture. This application embodiment takes the layered architecture Android system as an example to illustrate the software system of electronic device 1000.

[0172] Figure 9 A block diagram of a software system for an electronic device 1000 provided in an embodiment of this application. See also... Figure 9 A layered architecture divides software into several layers, each with a clear role and function. Layers communicate with each other through software interfaces. In some embodiments, the Android system is divided into five layers, from top to bottom: the application layer, the application framework layer, the hardware abstraction layer (HAL), the driver layer, and the hardware layer.

[0173] The application layer may include a series of application packages, such as dialer applications, gallery applications, etc. (not shown in the figure). In this embodiment, the application packages may include applications such as video, music, and voice assistants, all of which play audio to the user. In particular, voice assistant applications allow the terminal device to synthesize audio based on text and play the generated audio signal to the user.

[0174] Alternatively, the application layer may also include other applications that need to play audio signals generated from the text to the user, which is not limited in this application.

[0175] The framework layer provides an application programming interface (API) and a programming framework for applications in the application layer. The application framework layer includes some predefined functions. In this embodiment, the framework layer includes an audio synthesis service interface. This audio synthesis service interface provides an API and a programming framework for applications that obtain audio synthesis services.

[0176] The Hardware Abstraction Layer (HAL) is an interface layer located between the operating system kernel and upper-layer software, providing a virtual hardware platform for the operating system. In this embodiment, the HAL may include an audio synthesis algorithm. The audio synthesis algorithm may include runtime code and data that implement the audio synthesis method provided in this embodiment.

[0177] The driver layer is the layer between hardware and software. It includes drivers for various hardware components. The driver layer can include drivers for audio digital signal processors (DSPs), such as those for audio DSPs. Audio DSP drivers are used to drive audio DSPs to process audio digital signals.

[0178] The hardware layer includes an audio digital signal processor. The audio digital signal processor can be used to process audio digital signals. It should be noted that the application provided... Figure 9 The software architecture diagram of the electronic device shown is only an example and does not limit the specific module division in different layers of the Android operating system. For details, please refer to the introduction of the Android operating system software architecture in conventional technology.

[0179] The method in the embodiments of this application will be described in detail below with reference to the above hardware structure and system structure:

[0180] In response to applications that enable video, music, or voice assistants, these applications can call the audio synthesis service interface to obtain the application programming interface and programming framework provided by the audio synthesis service.

[0181] The audio synthesis service initializes the audio synthesis algorithm. The audio synthesis algorithm takes the text used to generate the audio signal, generates acoustic features based on the stored audio synthesis method, and processes the acoustic features using an audio signal processor to obtain an output time-domain audio signal.

[0182] For details on how to process the audio signal, please refer to the previous section. Figures 4 to 7 The flowchart of the method is shown.

[0183] Finally, the audio synthesis algorithm can send the generated audio signal back to the audio synthesis service, and then back to the application layer.

[0184] This application provides a chip system including one or more processors for calling and executing instructions stored in memory, thereby performing the methods described in this application. The chip system may be composed of chips or may include chips and other discrete devices.

[0185] The chip system may include input circuits or interfaces for transmitting information or data, and output circuits or interfaces for receiving information or data.

[0186] This application also provides a computer program product that, when executed by a processor, implements the methods described in any of the method embodiments of this application.

[0187] The computer program product can be stored in memory and, after processes such as preprocessing, compilation, assembly, and linking, is finally converted into an executable object file that can be executed by a processor.

[0188] This application also provides a computer-readable storage medium storing a computer program thereon, which, when executed by a computer, implements the methods described in any of the method embodiments of this application. The computer program may be a high-level language program or an executable object program.

[0189] The computer-readable storage medium can be volatile memory or non-volatile memory, or it can include both volatile and non-volatile memory. The non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. The volatile memory can be random access memory (RAM), which is used as an external cache. By way of example, but not limitation, many forms of RAM are available, such as static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous linked dynamic random access memory (SLDRAM), and direct rambus RAM (DR RAM).

[0190] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working process and technical effects of the above-described apparatus and equipment can be referred to the corresponding processes and technical effects in the foregoing method embodiments, and will not be repeated here.

[0191] In the several embodiments provided in this application, the systems, apparatuses, and methods disclosed can be implemented in other ways. For example, some features of the method embodiments described above can be ignored or not performed. The apparatus embodiments described above are merely illustrative; the division of units is only a logical functional division, and in actual implementation, there may be other division methods. Multiple units or components can be combined or integrated into another system. Furthermore, the coupling between units or components can be direct coupling or indirect coupling, including electrical, mechanical, or other forms of connection.

[0192] It should be understood that in the various embodiments of this application, the sequence number of each process does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0193] It should be understood that "multiple" as used in this application refers to two or more. The term "and / or" in this document is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone. Furthermore, the character " / " in this document generally indicates that the preceding and following related objects have an "or" relationship.

[0194] The terms (or numbers) "first," "second," etc., appearing in the embodiments of this application are for descriptive purposes only, that is, only to distinguish different objects, such as different "coordinates," etc., and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Therefore, features defined with "first," "second," etc., may explicitly or implicitly include one or more features. In the description of the embodiments of this application, "at least one (item)" refers to one or more. "Multiple" means two or more. "At least one (item) below" or similar expressions refer to any combination of these items, including any combination of a single (item) or a plurality of (items).

[0195] In summary, the above description is merely a preferred embodiment of the technical solution of this application and is not intended to limit the scope of protection of this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of protection of this application.

Claims

1. An audio synthesis method, applied to electronic devices, characterized in that, The method includes: Obtain the first text of the speech to be generated; Determine a first audio that matches the first text, wherein the sound scene in which the first audio is located matches the content of the first text; Obtain the text features of the first text; A second audio is obtained based on the first audio and the text features, wherein the second audio is associated with the first audio and the first speech, the first speech is a speech obtained based on the text features and is the same as the content of the first text, and wherein the first audio is used as the background sound of the first speech in the second audio; The method further includes: Obtain a clean speech signal and a corresponding sample text, wherein the clean speech signal and the sample text have the same content; Generate sample background audio and sample sound scene indexes corresponding to the sample text; The clean speech signal of the sample is fused with the background noise audio of the sample to obtain a sample speech signal with the background noise of the sample; Acquire the sample acoustic features of the sample speech signal with sample background noise; A first sample signal is generated based on the randomly generated second Gaussian noise and the acoustic features of the sample; Based on the sample text, we obtain the sample semantic features and sample phoneme length features; The first sample signal, sample semantic features, and sample phoneme length features are input into the modified flow model to obtain the sample derivative; The loss function is calculated based on the sample derivative and the target data, where the target data is the difference between the sample acoustic features and the second Gaussian noise. The parameters of the corrected flow model are updated based on the loss function.

2. The method as described in claim 1, characterized in that, The determination of the first audio that matches the first text includes: Based on the content of the first text, the category of the sound scene that matches the first text is determined in the preset sound scene database as the first sound scene. The preset sound scene database also includes the first audio corresponding to the first sound scene.

3. The method as described in claim 2, characterized in that, The step of determining the category of the sound scene matching the first text in a preset sound scene database based on the content of the first text includes: Obtain short-time features of the first text, wherein the short-time features are used to characterize the correlation between the first text and sub-texts corresponding to different time frames, wherein the sub-texts include one or more characters; The short-term features are mapped to a feature domain that facilitates classification to obtain the mapped features; The mapping features are classified to obtain indication information of the sound scene that matches the first text, and the indication information indicates the first sound scene.

4. The method as described in claim 3, characterized in that, The step of obtaining the second audio based on the first audio and the text features includes: The instruction information and the text features are input into the modified flow model to obtain the first acoustic feature, which is used to characterize the first speech that uses the first audio as background sound. The first acoustic feature is decoded to obtain the second audio.

5. The method according to any one of claims 1 to 4, characterized in that, The text features include semantic features and phoneme length features. Obtaining the text features of the first text includes: Obtain the semantic features of the first text, wherein the semantic features are used to characterize the semantic information of the first text; The phoneme length feature of the first text is obtained based on the semantic features. The phoneme length feature is used to indicate the phoneme length corresponding to each character in the first text. The phoneme length is the duration of the phoneme.

6. The method as described in claim 4, characterized in that, The step of inputting the indication information and the text features into the modified flow model to obtain the first acoustic features includes: The indication information, the text features, and the first Gaussian noise obtained by random sampling are input into the modified flow model to obtain the first derivative of the first Gaussian noise; Euler sampling is performed on the first derivative to obtain the first sampled signal; The indication information, the text features, and the first sampled signal are input into the modified flow model to obtain the second derivative of the first sampled signal; Euler sampling is performed on the second derivative to obtain the second sampled signal; The first acoustic feature is obtained based on the second sampled signal.

7. An electronic device, characterized in that, The electronic device includes: one or more processors, and memory; The memory is coupled to the one or more processors, the memory being used to store computer program code, the computer program code including computer instructions, the one or more processors invoking the computer instructions to cause the electronic device to perform the method as described in any one of claims 1 to 6.

8. A chip system, characterized in that, The chip system is applied to an electronic device, the chip system including one or more processors, the one or more processors being used to invoke computer instructions to cause the electronic device to perform the method as described in any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium includes instructions that, when executed on an electronic device, cause the electronic device to perform the method as described in any one of claims 1 to 6.

Images