Audio synthesis method, training method of audio synthesis model and related device

Abstract

The invention provides an audio synthesis method, an audio synthesis model training method and a related device. The method comprises the steps of determining text features of a first text and audio data of a first object; performing first coding on the audio data to obtain voice features of the first object, and performing second coding on the audio data to obtain melody features of the audio data; performing attention processing on the melody feature and the text feature to obtain a first feature; performing fusion processing on the voice feature, the first feature and the text feature to obtain a second feature; and decoding the second feature to obtain a synthesized audio. According to the invention, the audio synthesis accuracy can be improved, and the synthesized audio consistent with the sound of the first object can be obtained.

Description

Technical Field

[0001] The present application relates to artificial intelligence technology, and in particular to an audio synthesis method, a training method for an audio synthesis model, and related devices. Background Art

[0002] Speech synthesis technology has made significant progress in recent years, especially driven by deep learning models, which can generate synthesized speech that is rich in expressiveness and close to natural human speech. Nevertheless, this technology still faces some challenges. For example, the synthesized speech may not conform to the unique voice characteristics of a specific speaker. At the same time, achieving high-quality speech synthesis requires a lot of computing resources and relies on complex optimization algorithms, which may be difficult to implement in resource-constrained environments. Therefore, although speech synthesis technology has great potential in providing fast and flexible speech output, these technical obstacles still need to be addressed for wider application. Summary of the invention

[0003] The embodiments of the present application provide an audio synthesis method, an audio synthesis model training method and related devices, which can improve the accuracy of audio synthesis.

[0004] The technical solution of the embodiment of the present application is implemented as follows:

[0005] The present invention provides an audio synthesis method, which comprises:

[0006] Determining text features of a first text and determining audio data of a first object;

[0007] Performing a first encoding on the audio data to obtain a speech feature of the first object, and performing a second encoding on the audio data to obtain a musical feature of the audio data;

[0008] Performing attention processing on the musical feature and the text feature to obtain a first feature;

[0009] fusing the speech feature, the first feature, and the text feature to obtain a second feature;

[0010] The second feature is decoded to obtain synthesized audio.

[0011] The present application embodiment provides a training method for an audio synthesis model, the training method comprising:

[0012] fusing the plurality of first audio samples of the second object to obtain a second audio sample;

[0013] Performing audio synthesis on the second audio sample and the text sample to obtain a synthesized audio sample of the second object;

[0014] A model is trained based on the synthesized audio samples of the second object.

[0016] The present application provides an audio synthesis device, including:

[0017] A data collection module, configured to determine text features of the first text and determine audio data of the first object;

[0018] A data processing module, configured to perform a first encoding on the audio data to obtain a speech feature of the first object, and perform a second encoding on the audio data to obtain a musical feature of the audio data; perform attention processing on the musical feature and the text feature to obtain a first feature; and perform fusion processing on the speech feature, the first feature and the text feature to obtain a second feature;

[0019] The audio synthesis module is used to decode the second feature to obtain synthesized audio.

[0020] An embodiment of the present application provides an electronic device, including:

[0021] A memory for storing computer executable instructions;

[0022] The processor is used to implement the audio synthesis method provided in the embodiment of the present application when executing the computer executable instructions stored in the memory.

[0023] An embodiment of the present application provides a computer-readable storage medium storing a computer program or computer-executable instructions for implementing the audio synthesis method provided in the embodiment of the present application when executed by a processor.

[0024] An embodiment of the present application provides a computer program product, including a computer program or computer executable instructions. When the computer program or computer executable instructions are executed by a processor, the audio synthesis method provided in the embodiment of the present application is implemented.

[0025] The embodiments of the present application have the following beneficial effects:

[0026] The audio data is respectively subjected to a first encoding and a second encoding to obtain speech features of a first object and musical features of the audio data, and then the musical features and the text features are subjected to attention processing to obtain a first feature, and the speech features, the first features and the text features are fused to obtain a second feature. In this way, the speech features provide sound information related to the first object, and the text features provide semantic content. By integrating multimodal information such as musical features, speech features and text features, the performance of audio synthesis is improved to obtain synthesized audio that conforms to the sound of the first object, and the text content of the synthesized audio is consistent with the text content of the first text, thereby improving the accuracy of audio synthesis. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] Figure 1 is a schematic diagram of the architecture of the audio synthesis system provided in an embodiment of the present application;

[0028] Figure 2A is a first structural schematic diagram of an electronic device provided in an embodiment of the present application;

[0029] Figure 2B is a second structural schematic diagram of an electronic device provided in an embodiment of the present application;

[0030] Figure 3A It is a schematic diagram of a first flow chart of the audio synthesis method provided in an embodiment of the present application;

[0031] Figure 3B is a second flow chart of the audio synthesis method provided in an embodiment of the present application;

[0032] Figure 3C is a third flow chart of the audio synthesis method provided in an embodiment of the present application;

[0033] Figure 3D is a fourth flow chart of the audio synthesis method provided in an embodiment of the present application;

[0034] Figure 3E is a fifth flow chart of the audio synthesis method provided in an embodiment of the present application;

[0035] Figure 4 It is a flowchart of a training method for an audio synthesis model provided in an embodiment of the present application;

[0036] Figure 5 is a first structural diagram of a speech synthesis model provided in an embodiment of the present application;

[0037] Figure 6 is a schematic diagram of the speaker recognition model structure provided in an embodiment of the present application;

[0038] Figure 7It is a schematic diagram of the structure of a neural network model provided in an embodiment of the present application;

[0039] Figure 8 It is a schematic diagram of the structure of the scaled dot product attention model provided in an embodiment of the present application;

[0040] Fig. 9 is a second structural diagram of the speech synthesis model provided in an embodiment of the present application;

[0041] Fig.10 is a third structural diagram of the speech synthesis model provided in an embodiment of the present application;

[0042] Fig.11 It is a schematic diagram of the structure of a vector quantization model provided in an embodiment of the present application;

[0043] Fig.12 It is a schematic diagram of audio synthesis provided in an embodiment of the present application.

[0044] It should be pointed out that the above-mentioned "first" and "second" are only used to distinguish different solutions, and do not represent the degree of superiority or inferiority of the solutions or the priority in the implementation process. DETAILED DESCRIPTION

[0045] In order to make the purpose, technical solutions and advantages of the present application clearer, the present application will be further described in detail below in conjunction with the accompanying drawings. The described embodiments should not be regarded as limiting the present application. All other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of this application.

[0046] In the following description, reference is made to “some embodiments”, which describe a subset of all possible embodiments, but it will be understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0047] In the following description, the terms "first\second\third" involved are merely used to distinguish similar objects and do not represent a specific ordering of the objects. It can be understood that "first\second\third" can be interchanged with a specific order or sequence where permitted, so that the embodiments of the present application described herein can be implemented in an order other than that illustrated or described herein.

[0048] In the embodiments of the present application, the term "module" or "unit" refers to a computer program or a part of a computer program with a predetermined function, and works together with other related parts to achieve a predetermined goal, and can be implemented in whole or in part by using software, hardware (such as processing circuits or memories) or a combination thereof. Similarly, a processor (or multiple processors or memories) can be used to implement one or more modules or units. In addition, each module or unit can be part of an overall module or unit that includes the function of the module or unit.

[0049] The relevant data collection and processing in the embodiments of this application should be strictly in accordance with the requirements of relevant national laws and regulations when applied in examples, obtain the informed consent or separate consent of the personal information subject, and carry out subsequent data use and processing within the scope of authorization of laws and regulations and the personal information subject.

[0050] Unless otherwise defined, all technical and scientific terms used in the embodiments of the present application have the same meanings as those commonly understood by those skilled in the art. The terms used in the embodiments of the present application are only for the purpose of describing the embodiments of the present application and are not intended to limit the present application.

[0051] Before further describing the embodiments of the present application in detail, the nouns and terms involved in the embodiments of the present application are explained. The nouns and terms involved in the embodiments of the present application are subject to the following interpretations.

[0052] 1) Audio data is sound stored in digital format. Audio data can be original waveform data, compressed or encoded data.

[0053] 2) The first object is a specific object that sends audio data.

[0054] In the related art, the audio of different texts of the same object has inconsistent timbre, and the timbre and sound style of the object are ignored when synthesizing the audio. To address the above problems, the embodiments of the present application provide an audio synthesis method, a training method, device, electronic device, computer-readable storage medium and computer program product for an audio synthesis model, which can improve the accuracy of audio synthesis and obtain synthesized audio consistent with the sound of the first object.

[0055] The audio synthesis method recorded in the embodiments of the present application can be applied to various fields, such as user interaction, digital human live broadcast (i.e., live broadcast through a digital human, the voice of the digital human live broadcast is generated by preset text and voice), etc., that is, the audio synthesis method in the embodiments of the present application is not limited to a certain field.

[0056] The following describes an exemplary application of the electronic device provided in the embodiment of the present application. The device provided in the embodiment of the present application can be implemented as a terminal or a server. The following describes an exemplary application when the device is implemented as a server.

[0057] See also Figure 1 , Figure 1 It is a schematic diagram of the architecture of the audio synthesis system 100 provided in an embodiment of the present application. To support an audio synthesis application, a terminal (terminal 400 is shown as an example) is connected to a server 200 via a network 300. The network 300 may be a wide area network or a local area network, or a combination of the two.

[0058] Terminal 400 is used to send the text features of the first text and the audio data of the first object to server 200 through network 300. Server 200 is used to perform a first encoding on the audio data through a trained audio synthesis model to obtain the voice features of the first object, and to perform a second encoding on the audio data to obtain the musical features of the audio data, to perform attention processing on the musical features and the text features to obtain the first feature, to fuse the voice features, the first features and the text features to obtain the second feature, to decode the second feature to obtain synthesized audio, and to return the synthesized audio to terminal 400. Terminal 400 displays the synthesized audio through a graphical interface 410.

[0059] Next, an example of audio synthesis performed by terminal 400 will be described.

[0060] In some embodiments, terminal 400 can independently complete an audio synthesis task. For example, terminal 400 is used to determine text features of a first text and determine audio data of a first object, perform a first encoding on the audio data through a trained audio synthesis model to obtain speech features of the first object, perform a second encoding on the audio data to obtain musical features of the audio data, perform attention processing on the musical features and text features to obtain a first feature, perform fusion processing on the speech features, the first features and the text features to obtain a second feature, perform decoding processing on the second feature to obtain synthesized audio, and display the synthesized audio through the graphical interface 410.

[0061] In one implementation scenario, the server or terminal can generate information push audio for interaction based on the interactive text with the user, determine the text features of the interactive text, and determine the historical interactive audio of the agent, perform a first encoding on the historical interactive audio through a trained push audio synthesis model to obtain the voice features of the agent, and perform a second encoding on the historical interactive audio to obtain the musical features of the historical interactive audio, perform attention processing on the musical features and text features to obtain a first feature, perform fusion processing on the voice features, the first features and the text features to obtain a second feature, perform decoding processing on the second feature to obtain information push audio, and push the information push audio to the user to facilitate interaction with the user.

[0062] In an implementation scenario, a server or terminal can generate speech audio of a digital person in a speech scenario, determine the text features of the speech, and determine the historical speech audio of the user corresponding to the digital person, perform a first encoding on the historical speech audio through a trained audio synthesis model to obtain the voice features of the user corresponding to the digital person, and perform a second encoding on the historical speech audio to obtain the musical features of the historical speech audio, perform attention processing on the musical features and text features to obtain the first feature, perform fusion processing on the voice features, the first feature and the text features to obtain the second feature, and perform decoding processing on the second feature to obtain the speech audio.

[0063] In some embodiments, server 200 can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers. It can also be a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (CDN), as well as big data and artificial intelligence platforms.

[0064] The terminal 400 may be a smart phone, a tablet computer, a laptop computer, a desktop computer, a smart speaker, a smart watch, an intelligent voice interaction device, a smart home appliance, a vehicle terminal, an aircraft, etc., but is not limited thereto. The terminal and the server may be directly or indirectly connected via wired or wireless communication, which is not limited in the embodiments of the present application.

[0065] See also Figure 2A , Figure 2A is a first structural diagram of an electronic device provided in an embodiment of the present application. The electronic device 500 shown in FIG. 2 may be Figure 1In the terminal 400 or server 200, the electronic device 500 includes: at least one processor 510, a memory 550, and at least one network interface 520. The various components in the server 200 are coupled together through a bus system 540. It can be understood that the bus system 540 is used to realize the connection and communication between these components. In addition to the data bus, the bus system 540 also includes a power bus, a control bus, and a status signal bus. However, for the sake of clarity, various buses are labeled as bus system 540 in FIG. 2.

[0066] The processor 510 can be an integrated circuit chip with signal processing capabilities, such as a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc., where the general-purpose processor can be a microprocessor or any conventional processor, etc.

[0067] The user interface 530 includes one or more output devices 531 that enable presentation of media content, including one or more speakers and / or one or more visual display screens. The user interface 530 also includes one or more input devices 532, including user interface components that facilitate user input, such as a keyboard, mouse, microphone, touch screen display, camera, other input buttons and controls;

[0068] In some embodiments, when the terminal 400 independently completes the audio synthesis task, the server 200 provided in the embodiment of the present application does not include the user interface 530 .

[0069] The memory 550 may be removable, non-removable, or a combination thereof. Exemplary hardware devices include solid-state memory, a hard disk drive, an optical disk drive, etc. The memory 550 may optionally include one or more storage devices that are physically located away from the processor 510.

[0070] The memory 550 includes a volatile memory or a non-volatile memory, and may also include both volatile and non-volatile memories. The non-volatile memory may be a read-only memory (ROM), and the volatile memory may be a random access memory (RAM). The memory 550 described in the embodiments of the present application is intended to include any suitable type of memory.

[0071] In some embodiments, the memory 550 can store data to support various operations, examples of which include programs, modules, and data structures, or a subset or superset thereof, as exemplarily described below.

[0072] Operating system 551, including system programs for processing various basic system services and performing hardware-related tasks, such as framework layer, core library layer, driver layer, etc., for implementing various basic services and processing hardware-based tasks;

[0073] A network communication module 552, for reaching other computing devices via one or more (wired or wireless) network interfaces 520, exemplary network interfaces 520 include: Bluetooth, Wireless Compatibility Certification (WiFi), and Universal Serial Bus (USB);

[0074] a presentation module 553 for enabling presentation of information via one or more output devices 531 (e.g., display screen, speaker, etc.) associated with the user interface 530 (e.g., a user interface for operating peripherals and displaying content and information);

[0075] In some embodiments, when the audio synthesis task is completed independently by the terminal 400, the server 200 provided in the embodiment of the present application may not include the presentation module 553.

[0076] The input processing module 554 is used to detect one or more user inputs or interactions from one of the one or more input devices 532 and translate the detected inputs or interactions; in some embodiments, when the embodiment independently completes the audio synthesis task by the terminal 400, the server 200 provided in the embodiment of the present application may not include the presentation module 553.

[0077] In some embodiments, the audio synthesis device provided in the embodiments of the present application can be implemented in software. Figure 2A The audio synthesis device 555 stored in the memory 550 is shown, which can be software in the form of a program and a plug-in, etc., including the following software modules: a data acquisition module 5551, a data processing module 5552 and an audio synthesis module 5553. These modules are logical, so they can be arbitrarily combined or further split according to the functions implemented. The functions of each module will be explained below.

[0078] In some embodiments, the training device for the audio synthesis model provided in the embodiments of the present application can also be implemented in software, see Figure 2B , Figure 2B is a second structural diagram of an electronic device provided in an embodiment of the present application, Figure 2B Except for the training device 556 based on the audio synthesis model shown, the rest can be the same as Figure 2AThe same is not repeated here. The training device 556 of the audio synthesis model stored in the memory 550 can be software in the form of programs and plug-ins, including the following software modules: a preprocessing module 5561 and a model training module 5562. These modules are logical, so they can be arbitrarily combined or further split according to the functions implemented. The functions of each module will be explained below.

[0079] It should be noted that in the audio synthesis example below, those skilled in the art may apply the audio synthesis method provided in the embodiment of the present application to synthesized audio based on their understanding of the following text.

[0080] See also Figure 3A , Figure 3A is a first flow chart of the audio synthesis method provided in the embodiment of the present application, which will be combined with Figure 3A The steps shown are used for illustration. The audio synthesis method provided in the embodiment of the present application can be implemented by a server or a terminal alone, or by a server and a terminal in collaboration. The following will take the collaborative implementation by a server and a terminal as an example for illustration.

[0081] In step 101 , text features of a first text and audio data of a first object are determined.

[0082] Here, the first text includes a series of symbols, such as letters, Chinese characters, numbers, punctuation marks, etc. These symbols are arranged according to specific grammatical rules to express specific meanings (such as knowledge, emotions, opinions, instructions, etc.). Grammatical rules are rules for constructing texts and are used to characterize the arrangement order of symbols; text features are deep information extracted from the original text, and the deep information can be used to perform tasks such as text classification and sentiment analysis. The embodiment of the present application does not limit the text features. The text features can be word frequency, word embedding vectors, etc.; the first object is a specific object that emits audio data, and different objects have different object information (such as object identifiers, where the object identifier is used to identify a unique object, and each object corresponds to its own object identifier); audio data is sound stored in a digital format. The embodiment of the present application does not limit the audio data. The audio data can be original waveform data, compressed or encoded data, etc. Audio data is used in multiple fields such as speech recognition, music analysis, and sound synthesis.

[0083] In some embodiments, "determining the text features of the first text" in step 101 can be achieved by: performing word segmentation processing on the first text to obtain text segmentations; performing stem extraction on each text segmentation to obtain target text segmentations; filtering the target text segmentations, and vectorizing the filtered target text segmentations to obtain text features of the first text.

[0084] For example, the first text is segmented to obtain text segmentations, and continuous character sequences in the first text are split into meaningful words or phrases; stem extraction is performed on each text segmentation to convert the word into its basic form to reduce the dimension of the data and eliminate the impact of word form changes, for example, "running", "runs" and "ran" can all be restored to "run"; stop words (such as "the") are removed; vectorization is performed on the target text segmentations obtained after stem extraction to convert the text into a numerical vector, and the vectorization can be achieved in the following ways: a vocabulary is constructed based on the first text, and the vocabulary contains all non-repeating words; the first text is segmented, and the text segmentations are matched with the vocabulary, and the frequency of each text segmentation in the first text is counted; the first text is converted into a vector, each element in the vector corresponds to a word in the vocabulary, and the value of the element is used to represent the probability of occurrence of the corresponding word in the first text.

[0085] Continuing with the above example, the above “building a vocabulary based on the first text” can be implemented in the following way: extracting an initial vocabulary from historical text data, the vocabulary contains all non-repeating first words, taking two consecutive characters in the first text as the second word, and updating the initial vocabulary according to the probability of occurrence of the second word in the first text and the probability of occurrence of the first word in the first text to obtain a vocabulary.

[0086] Continuing with the above example, given the text is "The quick brown fox", and the given vocabulary is ["the", "quick", "brown", "fox", "jumped", "over", "lazy", "dog", "outran"], the text features of the text are [0.25, 0.25, 0.25, 0.25, 0, 0, 0, 0], where the value of each element represents the probability of the corresponding word appearing in the text, and 0 means that the word does not appear in the text.

[0087] In some embodiments, "determining the audio data of the first object" in step 101 can be achieved by: determining the initial audio data of the first object; when the duration of the initial audio data is less than a duration threshold, determining a first quantity based on the duration of the initial audio data and the duration threshold; splicing the initial audio data that meets the first quantity to obtain the audio data of the first object.

[0088] Continuing from the above embodiment, the above “determining the first quantity based on the duration of the initial audio data and the duration threshold” can be implemented in the following way: the first quantity is determined as the sum of the ratio of the duration threshold to the duration of the initial audio data and a preset quantity (such as 1).

[0089] It should be noted that the initial audio data is audio data with a duration less than the duration threshold. By splicing the initial audio data, audio data with a duration greater than or equal to the duration threshold is obtained, so that the audio synthesis model can learn rich sound information from the audio data.

[0090] For example, initial audio data (such as [1, -1, 2, -2]) that meets a first quantity (such as 2) is concatenated to obtain audio data of a first object (such as [1, -1, 2, -2, 1, -1, 2, -2]).

[0091] In step 102, the audio data is first encoded to obtain the speech features of the first object, and the audio data is second encoded to obtain the musical features of the audio data.

[0092] Here, the first encoding and the second encoding are different encoding methods. The encoding is used to map audio data to a high-dimensional space and to convert audible audio data into feature vectors that cannot be perceived. The speech feature is manifested in the spectrum as a sound pattern unique to the object. The embodiments of the present application do not limit the speech feature. The speech feature can be used to characterize the timbre of the sound (such as the unique distribution of the timbre) and the style of the sound. The timbre of the sound distinguishes different objects through the specific frequency components in the spectrum and the relative intensity of the specific frequency components. The style of the sound is the unique characteristics and expression form of the sound, which is used to reflect the individual characteristics or emotional state of the sound source. The embodiments of the present application do not limit the speech feature. There is no restriction on the style of the sound, which can be the range, volume, etc., among which the range is the range from the lowest to the highest note that an object can make, and the volume is the loudness of the sound, which is used to affect the perceived intensity and emotional expression of the sound; the musical characteristics include the basic components of speech reflected in the spectrum, such as the distribution of resonance peaks of vowels and consonants, which are used to distinguish the spectral characteristics of different phonemes in the language. Taking the sound as an example, when a vowel (such as a) is uttered, a set of specific resonance peaks will appear in the spectrum of each object, and the positions of the resonance peaks are roughly the same. When a consonant (such as p) is uttered, the vocal cords do not vibrate, and the spectrum is mainly composed of high-frequency noise components, and the resonance peaks are not obvious or missing.

[0093] It should be noted that the musical characteristics also include the frequency distribution of different punctuation marks in the frequency spectrum. Since the duration of pauses at punctuation marks of different objects is not exactly the same, the musical characteristics of different objects are not exactly the same.

[0094] In some embodiments, see Figure 3B , Figure 3B is a second flow chart of the audio synthesis method provided in the embodiment of the present application, for Figure 3A Step 102 shown in the figure performs a first encoding on the audio data to obtain the speech features of the first object, which can be obtained by Figure 3BSteps 1021A to 1024A are implemented as described in detail below.

[0095] In step 1021A, the audio data is processed by cepstrum to obtain the eighth feature.

[0096] Here, cepstrum processing extracts features by analyzing the frequency spectrum of audio data, and the eighth feature is used to capture important properties of the audio data while reducing the impact of noise and instability.

[0097] In some embodiments, step 1021A and the following step 1021B can be implemented in the following manner: perform frequency domain transform processing on the audio data to obtain an audio spectrum; convert the audio spectrum into a first power spectrum, and filter the first power spectrum to obtain a second power spectrum; perform cosine transform on the logarithm of the second power spectrum to obtain the eighth feature.

[0098] It should be noted that the embodiment of the present application does not limit the frequency conversion method, and the frequency conversion method can be fast Fourier transform, two-dimensional discrete Fourier transform, etc. The filtering process is used to modify or extract specific frequency components of the signal (such as the first power spectrum); the embodiment of the present application does not limit the filtering process method, and the filtering process can be low-pass filtering, high-pass filtering, etc., wherein low-pass filtering allows low-frequency signals to pass through and blocks high-frequency signals, and is used to eliminate high-frequency noise and smooth the signal; high-pass filtering allows high-frequency signals to pass through and blocks low-frequency signals, and is used to highlight the rapidly changing parts of the signal.

[0099] In some embodiments, the above-mentioned "performing frequency domain transform processing on the audio data to obtain an audio spectrum" can be achieved in the following way: performing Fourier transform on the audio data, transforming the audio data from the spatial domain to the frequency domain, and obtaining an audio spectrum of the audio data, wherein each spectrum point of the audio spectrum is the amplitude at each frequency, and each spectrum point corresponds to a specific frequency value. The amplitude of the spectrum point reflects the intensity or energy of the frequency component in the signal. Since different sounds have unique frequency distributions, the amplitude of the spectrum point is used to identify the timbre of different sounds.

[0100] For example, Fourier transform is performed on audio data (such as [1, -1, 2, -2]), and the audio data is transformed from the spatial domain to the frequency domain to obtain an audio spectrum of the audio data (such as [0, 0.6, 0.8, ..., 0.5], where the spectrum point 0 with the subscript 0 is used to represent the amplitude of the audio corresponding to the frequency when the frequency is 0), wherein the length of the audio data is used to represent the length of time, the time interval between two consecutive data points of the audio data is the same, and the value of the data point in the audio data is the amplitude of the audio (such as 1 and -1 are the amplitudes of audio data collected at different time points).

[0101] Following the above embodiment, the above “converting the audio spectrum into a first power spectrum” may be implemented in the following manner: each spectrum point in the audio spectrum is replaced by the square of each spectrum point to obtain the first power spectrum.

[0102] It should be noted that the power spectrum is the distribution of signal power at each frequency, and the power is proportional to the square of the amplitude.

[0103] For example, given an audio spectrum (such as frequency 1 is 440 Hz, the value of spectrum point A1 of the audio spectrum at frequency 1 is 1, the frequency 2 is 880 Hz, and the value of spectrum point A2 of the audio spectrum at frequency 2 is 0.5), each spectrum point in the audio spectrum is replaced by the square of each spectrum point (the square of the value of spectrum point A1 of the audio spectrum at frequency 1 is 1, and the square of the value of spectrum point A2 of the audio spectrum at frequency 1 is 0.25) to obtain a first power spectrum.

[0104] Continuing with the above embodiment, taking high-pass filtering as an example, the above-mentioned "filtering the first power spectrum to obtain the second power spectrum" can be achieved in the following way: determine the power threshold, perform the following processing for each power in the first power spectrum, and when the power is less than the power threshold, replace the power in the first power spectrum with a preset power (such as 0) to obtain the second power spectrum.

[0105] For example, given a first power spectrum (such as [0.6, 0.8, 0.5], when the power in the first power spectrum is less than a power threshold (such as 0.6), the power in the first power spectrum (such as 0.5) is replaced with a preset power (such as 0) to obtain a second power spectrum (such as [0.6, 0.8, 0]).

[0106] For example, a cosine transform is performed on the logarithm (such as [-1, -2, 0]) of the second power spectrum (such as [0.5, 0.25, 1]), the logarithm is expressed as a set of coefficients of cosine functions, and each set of cosine functions is fused (cosine function A is "-1*cos(0*x)", cosine function B is "-2*cos(0.01*π*x)", and cosine function A is "0*cos(0.01*π*2*x)") to obtain the eighth feature, where the frequency of each set of cosine functions is the product of the index subscript of the logarithm and a preset frequency (such as the ratio of π to 100, 100 is used to represent the length of the second power spectrum).

[0107] Through the embodiments of the present application, filtering processing is performed to effectively reduce the impact of noise and improve the clarity of audio data. In addition, cepstrum processing can extract key frequency components in audio data, thereby improving the accuracy of audio synthesis. At the same time, it reduces the instability of audio data and improves the robustness of features.

[0108] In step 1022A, residual processing is performed on the eighth feature to obtain a ninth feature.

[0109] Here, residual processing solves the gradient vanishing problem in network training by adding skip connections, and the ninth feature is used to measure the difference between the outputs of different neural network layers.

[0110] In some embodiments, step 1022A may be implemented in the following manner: performing convolution processing on the eighth feature to obtain a spectrum convolution feature, and fusing the spectrum convolution feature and the eighth feature to obtain a ninth feature.

[0111] It should be noted that the above-mentioned step of "convolving the eighth feature to obtain a spectral convolution feature" is similar to the step of "convolving the eighth feature to obtain a convolution feature" in the following step 1022B, and the above-mentioned step of "fusing the spectral convolution feature and the eighth feature to obtain the ninth feature" is similar to the step of "fusing the first normalized feature and the multi-head attention feature to obtain a fused feature", which will not be repeated here.

[0112] In step 1023A, context mask processing is performed on the ninth feature to obtain the tenth feature.

[0113] Here, the context mask processing extracts context information of different scales through global and segment-level pooling operations, and the extracted context information is used to remove irrelevant noise in the features.

[0114] In some embodiments, step 1023A can be implemented in the following manner: globally pooling the ninth feature to obtain a first pooling feature; performing segment-level pooling on the ninth feature to obtain a second pooling feature, fusing the first pooling feature and the second pooling feature to obtain a third pooling feature; mapping the third pooling feature to obtain a residual mapping feature, and convolving the ninth feature to obtain a residual convolution feature; fusing the residual mapping feature and the residual convolution feature to obtain the tenth feature.

[0115] It should be noted that global pooling is used to compress the entire spatial dimension of each ninth feature into a single value and convert the ninth feature into a vector of fixed length. The embodiment of the present application does not limit the global pooling. Global pooling can be global average pooling, global maximum pooling, etc., wherein global average pooling is used to calculate the average value of all elements of the ninth feature, and global maximum pooling is used to select the maximum value from each element of the ninth feature; segment-level pooling is used to split the ninth feature into multiple windows and then apply the pooling operation on each window.

[0116] For example, the ninth feature (such as [0.28, 0.16, 0, 0.14, 0.13, -0.06, 0.15, 0.03]) is subjected to global maximum pooling to obtain the first pooling feature (such as 0.28); the ninth feature is subjected to segment-level pooling to obtain the second pooling feature (such as [0.28, 0, -0.06, 0.15]), and the first pooling feature and the second pooling feature are fused to obtain the third pooling feature (such as [0.28, 0.28, 0, -0 .06, 0.15]); the third pooling feature is mapped to obtain the residual mapping feature (such as [0.14, 0.14, 0, -0.03, 0.075]), and the ninth feature is convolved to obtain the residual convolution feature (such as [0.19, 0.10, 0, 0.28, 0.16]); the residual mapping feature and the residual convolution feature are fused to obtain the tenth feature (such as [0.33, 0.24, 0, 0.25, 0.235]).

[0117] In step 1024A, dense convolution processing is performed on the tenth feature to obtain the speech feature of the first object.

[0118] Here, dense convolution processing is implemented through multiple dense convolution layers. The input features of each layer are obtained by concatenating the output features of the previous layer with the input features of the previous layer. The network structure of each dense convolution layer is consistent.

[0119] For example, taking three layers of dense convolutional layers as an example, the tenth feature (such as [0.33, 0.24, 0, 0.25, 0.235]) is convolved through dense convolutional layer A to obtain the first dense convolutional feature (such as [0.34, 0.23, 0, 0.24, 0.22]), the first dense convolutional feature and the tenth feature are concatenated to obtain the first concatenated feature (such as [0.33, 0.24, 0, 0.25, 0.235, 0.34, 0.23, 0, 0.24, 0.22]), and the first concatenated feature is concatenated through dense convolutional layer B. The feature is convolved to obtain a second dense convolution feature such as [0.23, 0.3, 0, 0.2, 0.5]), the second dense convolution feature and the first splicing feature are concatenated to obtain a second splicing feature (such as [0.33, 0.24, 0, 0.25, 0.235, 0.34, 0.23, 0, 0.24, 0.22, 0.23, 0.3, 0, 0.2, 0.5]), and the second splicing feature is convolved through a dense convolution layer C to obtain the speech feature of the first object (such as [0.32, 0.3, 0, 0.6, 0.1]).

[0120] Through the embodiments of the present application, residual connections and context mask processing can improve the model's ability to learn complex speech features. The design of dense convolutional layers can extract deeper features to enhance the model's ability to capture sound details, thereby improving the accuracy of audio synthesis.

[0121] In some embodiments, see Figure 3C , Figure 3C is a third flow chart of the audio synthesis method provided in the embodiment of the present application, for Figure 3A In step 102, the audio data is secondly encoded to obtain the musical characteristics of the audio data, which can be obtained by Figure 3C Steps 1021B to 1023B are implemented as described in detail below.

[0122] In step 1021B, the audio data is subjected to cepstrum processing to obtain the eighth feature.

[0123] Here, step 1021B is similar to step 1021A and will not be repeated here.

[0124] In step 1022B, the eighth feature is convolved, and the convolved eighth feature is pooled to obtain the eleventh feature.

[0125] Here, convolution processing is used to extract local information of the eighth feature (such as edge information, texture information, etc.), and pooling processing is used to reduce the spatial dimension of the data (such as the eighth feature after convolution) while retaining the most important features. The embodiment of the present application does not limit the method of pooling processing, and the pooling processing can be maximum pooling, average pooling, etc.

[0126] In some embodiments, the above-mentioned "convolution processing of the eighth feature" can be implemented in the following way: determine the convolution kernel, perform sliding processing on the eighth feature based on the convolution kernel to obtain multiple coverage areas, calculate the dot product between the convolution kernel and the eighth feature in each coverage area, combine each dot product, and obtain the convolved eighth feature, wherein the size of the coverage area is consistent with the size of the convolution kernel.

[0127] It should be noted that the embodiment of the present application does not limit the convolution kernel, and the convolution kernel can be a Gaussian filter, a sharpening filter, etc.

[0128] For example, determine the convolution kernel (such as [1, 0, -1]), perform sliding processing on the eighth feature (such as [1, 2, 3, 4, 5, 4]) based on the convolution kernel, and obtain multiple coverage areas (such as coverage area A is [1, 2, 3], coverage area B is [2, 3, 4], coverage area C is [3, 4, 5], and coverage area D is [4, 5, 4]), calculate the dot product of the convolution kernel and the eighth feature in each coverage area (such as the dot product of coverage area A is -2, the dot product of coverage area B is -2, the dot product of coverage area C is -2, and the dot product of coverage area D is 0), combine each dot product, and obtain the eighth feature after convolution (such as [-2, -2, -2, 0]).

[0129] In some embodiments, the above-mentioned "pooling the eighth feature after convolution to obtain the eleventh feature" can be achieved in the following way: region-dividing the eighth feature after convolution to obtain multiple pooling windows, determining the maximum value of the elements in each pooling window, combining each maximum value, and obtaining the eleventh feature.

[0130] It should be noted that the pooling window is a window of a specific size.

[0131] For example, the eighth feature after convolution (such as [1, 2, 3, 4, 5, 4]) is divided into regions to obtain multiple pooling windows (such as pooling window A is [1, 2], pooling window B is [3, 4], and pooling window C is [5, 4]), and the maximum value of the elements in each pooling window is determined (such as the maximum value of the elements in pooling window A is 2, the maximum value of the elements in pooling window B is 4, and the maximum value of the elements in pooling window C is 5), and each maximum value is combined to obtain the eleventh feature (such as [2, 4, 5]).

[0132] In step 1023B, the eleventh feature is mapped and scaled to obtain the musical characteristics of the audio data.

[0133] Here, the mapping processing is used to map the fused features to a new feature space to obtain features for characterizing the potential structure of the eighth feature. The embodiment of the present application does not limit the mapping processing. The mapping processing can be linear mapping, nonlinear mapping, etc. The scaling processing is used to adjust the numerical range of the feature to a specific interval.

[0134] In some embodiments, taking the mapping process as a linear mapping process as an example, the above-mentioned "mapping the eleventh feature" can be implemented in the following way: determining a mapping parameter, and determining the product of the eleventh feature and the mapping parameter as the mapped eleventh feature.

[0135] For example, a mapping parameter (such as 2) is determined, and the product of the eleventh feature (such as [1, 0, -1]) and the mapping parameter (such as [2, 0, -2]) is determined as the eleventh feature after mapping.

[0136] In some embodiments, the above-mentioned "scaling the mapped eleventh feature to obtain the musical characteristics of the audio data" can be achieved in the following way: convolving the mapped eleventh feature through one-dimensional convolution to obtain the musical characteristics of the audio data, wherein the dimension of the one-dimensional convolution is a specific dimension (such as 1), and the step size of the one-dimensional convolution is a specific step size (such as 10).

[0137] For example, the convolution kernel of a one-dimensional convolution is determined (such as 0.5), the step size of the one-dimensional convolution is 3, and the eleventh feature after mapping based on the convolution kernel (such as [1, 2, 3, 4, 5, 4]) is slidingly processed to obtain multiple coverage areas (such as coverage area A is [1], coverage area B is [4]), and the dot product between the convolution kernel and the mapped eleventh feature in each coverage area is calculated (such as the dot product of coverage area A is 0.5, and the dot product of coverage area B is 2), and each dot product is combined to obtain the musical characteristics of the audio data (such as [0.5, 2]).

[0138] Through the embodiments of the present application, local details and global structures of audio signals can be extracted through convolution and pooling processing, and scaling processing helps to adjust the numerical range and time scale of features to make the features more stable and reduce the spatial dimension of the features, thereby reducing the number of parameters and computational burden of the model, thereby improving the efficiency of audio synthesis.

[0139] Continue to see Figure 3A In step 103, attention processing is performed on the musical features and the text features to obtain the first feature.

[0140] Here, step 103 can be implemented in the following manner: by using an attention layer, attention processing is performed on the musical features and the text features to obtain the first feature.

[0141] It should be noted that the attention layer includes an attention model that performs attention processing on the data. The embodiment of the present application does not limit the attention model. The attention model can be a multi-head self-attention network, or a single-head self-attention network, etc. The attention processing is used to learn important information in the musical features and text features from different angles at the same time through the attention model.

[0142] For example, attention processing is performed on the rhythm feature (such as [0.14, 0.13, -0.06]) and the text feature (such as [0.15, 0.03, 0.06]) to obtain the first feature (such as [0.28, 0.16, 0]).

[0143] In some embodiments, see Figure 3D , Figure 3D is a fourth flow chart of the audio synthesis method provided in the embodiment of the present application, for Figure 3A Step 103 shown can be performed by Figure 3D Steps 1031 to 1033 are implemented as described in detail below.

[0144] In step 1031, the musical feature is normalized to obtain the third feature, and the text feature is normalized to obtain the fourth feature.

[0145] Here, normalization is used to adjust the data to a uniform scale to avoid deviations caused by different features having different measurement units and numerical ranges. The embodiment of the present application does not limit the normalization method, and normalization can be batch normalization, layer normalization, etc.

[0146] In some embodiments, the above-mentioned "normalizing the musical feature to obtain a normalized feature" can be achieved in the following way: based on a preset norm parameter, performing a power operation on each element of the musical feature to obtain a power element; summing up each power element to obtain a sum element; performing a power operation on the sum element based on the inverse of the norm parameter to obtain a target element; determining the ratio of each element of the musical feature to the target element as a normalized feature, wherein the norm parameter (i.e., the exponent) is used to characterize the number of times the element itself is multiplied, and the norm parameter is a real number greater than 0.

[0147] For example, given a musical feature (such as [2, 4]), a norm parameter (such as 2), calculate the square of each element of the musical feature to obtain a power element (such as 2 squared is 4, 2 squared is 4), calculate the sum of each power element to obtain a sum element (such as 20), use the reciprocal of the norm parameter (such as 0.5, that is, the square root) to perform a power operation on the sum element to obtain a target element (such as 4.5), and determine the ratio of each element of the musical feature to the target element (such as the ratio of 2 to 4.5 is 0.44, and the ratio of 2 to 4.5 is 0.88) as a normalized feature (such as [0.44, 0.88]).

[0148] In some embodiments, the above step of "normalizing the text feature to obtain the fourth feature" is similar to the above step of "normalizing the musical feature to obtain the third feature", which will not be repeated here.

[0149] In step 1032, multi-head attention processing is performed on the third feature and the fourth feature to obtain the fifth feature.

[0150] Here, multi-head attention processing is the process of performing data processing on the third feature and the fourth feature through a multi-head attention network, which is used to simultaneously pay attention to information at different positions when processing data (such as the third feature and the fourth feature), thereby improving the ability to capture information.

[0151] In some embodiments, step 1032 can be implemented by: determining a key vector of the audio data and a value vector of the audio data based on the third feature, and determining a query vector of the first text based on the fourth feature; determining a similarity matrix of the first text based on the query vector and the key vector; and obtaining a fifth feature based on the similarity matrix and the value vector. In this embodiment, the product of the similarity matrix and the value vector can be determined as the fifth feature.

[0152] Here, the query vector is a vector representation used for comparison with the key vector, and the query vector is used to find the correlation between any element in the text and any element in the audio data; the key vector is a vector representation that matches the query vector, and the key vector of each element in the audio data determines the importance of the element in the attention weight. The higher the degree of match between the key vector and the query vector, the greater the attention weight of the corresponding element; after calculating the attention weight of each element, the value vector is used to perform weighted summation according to the attention weight to generate the fifth feature; the similarity matrix is ​​obtained by calculating the similarity between the query vector and the key vector, and each element of the similarity matrix represents the degree of match between the corresponding query vector and the key vector. The embodiment of the present application does not limit the similarity between features, and the similarity can be the dot product of features, the cosine similarity between features, etc.

[0153] In some embodiments, the above-mentioned "determining the key vector of the audio data and the value vector of the audio data based on the third feature, and determining the query vector of the first text based on the fourth feature" can be implemented in at least one of the following ways: determining the third feature as a key vector and a value vector, and determining the fourth feature as a query vector; or, mapping the third feature to obtain a key vector and a value vector, and mapping the fourth feature to obtain a query vector.

[0154] It should be noted that the above-mentioned "mapping the third feature to obtain a key vector and a value vector, and mapping the fourth feature to obtain a query vector" is similar to mapping the third feature to obtain a key vector and a value vector, and mapping the fourth feature to obtain a query vector in step 1023B, and will not be repeated here.

[0155] Following the above embodiment, the above “determining a similarity matrix based on the query vector and the key vector” can be implemented in the following way: determining the dot product of the query vector and the key vector as a similarity matrix.

[0156] For example, a dot product (such as [[1, 0.6, 1.2], [1.5, 0.9, 1.8], [-0.5, -0.3, -0.6]]) based on the query vector (such as [2, 3, -1]) and the key vector (such as [0.5, 0.3, 0.6]) is determined as a similarity matrix; the product (such as [1.4, 2.05, -0.7]) of the similarity matrix and the value vector (such as [0.5, 0.3, 0.6]) is determined as the fifth feature.

[0157] Through the embodiments of the present application, the model can pay attention to different parts of the input data at the same time, which helps to capture local and global contextual information. By allocating attention weights, the model can automatically focus on important features, thereby optimizing the feature selection process. The model learns different spatial representations, increases the expressive power of the model, and is able to understand complex data structures.

[0158] In step 1033, the first feature is determined based on the third feature and the fifth feature.

[0159] In some embodiments, step 1032 can be implemented by: fusing the third feature and the fifth feature to obtain the sixth feature; mapping the sixth feature to obtain the seventh feature; fusing the sixth feature and the seventh feature to obtain the first feature.

[0160] Continuing from the above embodiment, the above “fusing the third feature and the fifth feature to obtain the sixth feature” can be achieved in the following way: concatenating the third feature and the fifth feature to obtain the sixth feature, or performing weighted summation of the third feature and the fifth feature to obtain the sixth feature.

[0161] For example, the third feature (such as [0.14, 0.13, -0.06]) and the fifth feature (such as [0.15, 0.03, 0.06]) are concatenated to obtain the sixth feature (such as [0.14, 0.13, -0.06, 0.15, 0.03, 0.06]); or, the third feature and the fifth feature are weighted summed to obtain the sixth feature (taking the example that the weights of the third feature and the fifth feature are both 0.5, the sixth feature (such as [0.28, 0.16, 0]) is obtained.

[0162] Following the above embodiment, mapping is used to map the sixth feature to a new feature space to obtain a seventh feature for characterizing the potential structure of the sixth feature. The embodiment of the present application does not limit the mapping method, and the mapping method can be linear mapping, nonlinear mapping, etc. The above step of "fusing the seventh feature and the sixth feature to obtain the first feature" is similar to the above step of "fusing the third feature and the fifth feature to obtain the sixth feature", which will not be repeated here.

[0163] For example, the third feature and the fifth feature are fused to obtain the sixth feature, and the sixth feature (such as [0.28, 0.16, 0]) is mapped to obtain the seventh feature (such as [0.14, 0.08, 0]); the seventh feature and the sixth feature are fused to obtain the first feature (such as [0.42, 0.24, 0]).

[0164] Continue to see Figure 3A In step 104, the speech feature, the first feature and the text feature are fused to obtain the second feature.

[0165] Here, in some embodiments, the above-mentioned "step of fusing speech features, first features and text features to obtain the sixth feature" can be implemented in the following way: splicing the speech features, the first features and the text features to obtain the sixth feature, or performing weighted summation of the speech features, the first features and the text features to obtain the sixth feature.

[0166] For example, the speech feature (such as [0.28, 0.16, 0]), the first feature (such as [0.14, 0.13, -0.06]) and the text feature (such as [0.15, 0.03, 0.06]) are concatenated to obtain the sixth feature (such as [0.28, 0.16, 0, 0.14, 0.13, -0.06, 0.15, 0.03, 0.06]); or, the speech feature, the first feature and the text feature are weighted and summed to obtain the sixth feature (taking the example that the weights of the speech feature, the first feature and the text feature are all 0.33, the sixth feature is such as [0.19, 0.10, 0]).

[0167] Through the embodiments of the present application, the speech feature, the first feature and the text feature are integrated so that the synthesized audio contains not only the text feature but also the speech feature of the object, so as to ensure that the synthesized audio conforms to the sound of the object and improve the accuracy of audio synthesis.

[0168] In step 105, the second feature is decoded to obtain synthesized audio.

[0169] The synthesized audio matches the sound of the first object, and the text content of the synthesized audio is consistent with the text content of the first text.

[0170] Here, the decoding process is used to convert the feature vectors of the high-dimensional space back into the form of original data or a form that can be understood by the object (such as audio data). The synthesized audio is audio data obtained by performing audio synthesis processing on the audio data and the first text through an audio synthesis method. The sound of the first object includes the timbre and pronunciation style unique to the first object. Taking different ages as an example, the pronunciation style can be a juvenile voice, a young voice, an old voice, etc. The text content of the first text is each character in the first text. The text content of the synthesized audio can be the text content obtained by performing speech recognition processing on the synthesized audio.

[0171] It should be noted that the melody of the synthesized audio conforms to the melody of the audio data of the first object.

[0172] In some embodiments, see Figure 3E , Figure 3E is a fifth flow chart of the audio synthesis method provided in the embodiment of the present application, for Figure 3A Step 105 shown can be performed by Figure 3E Steps 1051 to 1053 are implemented as described in detail below.

[0173] In step 1051, the speech feature and the second feature are predicted to obtain the twelfth feature.

[0174] In some embodiments, step 1051 can be implemented by normalizing the speech feature to obtain a first normalized feature, normalizing the second feature to obtain a second normalized feature, performing multi-head attention processing on the first normalized feature and the second normalized feature to obtain a twelfth feature.

[0175] It should be noted that the above-mentioned step of "normalizing the speech features to obtain the first normalized features, and normalizing the second features to obtain the second normalized features" is similar to step 1031, and the above-mentioned step of "performing multi-head attention processing on the first normalized features and the second normalized features to obtain the twelfth feature" is similar to step 1032, which will not be repeated here.

[0176] In step 1052, the twelfth feature is mapped, and the mapped twelfth feature is upsampled to obtain the thirteenth feature.

[0177] Here, step 1052 is similar to the step of mapping the eleventh feature in step 1023B, and will not be repeated here. The embodiment of the present application does not limit the upsampling process, and the upsampling process can be a nearest neighbor interpolation method, a bilinear interpolation method, etc. The upsampling process is used to increase the spatial dimension of the feature map, and the nearest neighbor interpolation method is used as an example to illustrate that the nearest neighbor pixel point is selected to fill the new spatial position.

[0178] For example, the mapped twelfth feature (such as [[1, 2], [3, 4]]) is upsampled to obtain the thirteenth feature (such as [[1, 1, 2, 2], [1, 1, 2, 2], [3, 3, 4, 4], [3, 3, 4, 4]]).

[0179] In step 1053, the thirteenth feature is deconvolved to obtain synthesized audio.

[0180] In some embodiments, step 1053 can be implemented in the following manner: determine the transposed convolution kernel, slide the transposed convolution kernel on the thirteenth feature by means of a sliding window, calculate the dot product of each element of the thirteenth feature and the transposed convolution kernel, fuse each dot product to obtain a synthetic audio feature, activate the audio feature, and obtain a synthetic audio.

[0181] It should be noted that the deconvolution process is the reverse process of the convolution process, and the activation process introduces nonlinearity through the activation function to solve the nonlinear problem. The embodiment of the present application does not limit the activation process. The activation function can be a linear mapping unit (Rectified Linear Unit, ReLU), a leaky linear mapping unit (Leaky Rectified Linear Unit, Leaky-ReLU), etc., wherein the linear mapping unit is used to characterize that when the input is greater than 0, the input value is output, otherwise the output is 0. The linear mapping unit is simple to calculate and has a fast training speed.

[0182] For example, determine the transposed convolution kernel (such as [[1, 2], [3, 4]]), slide the transposed convolution kernel on the thirteenth feature (such as [[1, 2], [3, 4]]) through a sliding window, calculate the dot product of each element of the thirteenth feature and the transposed convolution kernel (such as the dot product corresponding to element 1 [[1, 2], [3, 4]], the dot product corresponding to element 2 [[2, 4], [6, 8]], the dot product corresponding to element 3 [[3, 6], [9, 12]], and the dot product corresponding to element 4 [[4, 8], [12, 16]]), fuse each dot product to obtain a synthetic audio feature (such as [[1, 4, 4], [6, 20, 16], [9, 24, 16]]), activate the audio feature, and obtain a synthetic audio (such as [1, 3, 4, -5, -2, 4]).

[0183] See also Figure 4 , Figure 4 is a flowchart of the training method of the audio synthesis model provided in the embodiment of the present application, which will be combined with Figure 4 The steps shown are used to illustrate that the training method of the model provided in the embodiment of the present application can be implemented by a server or a terminal alone, or by a server and a terminal in collaboration. The following will illustrate the collaborative implementation by a server and a terminal as an example.

[0184] In step 201, a plurality of first audio samples of a second object are fused to obtain a second audio sample.

[0185] Here, the embodiment of the present application does not impose any restrictions on the second object. The second object can be the first object or any object other than the first object. The first audio sample and the second object are audio samples emitted for different texts. The text content of each first audio sample is different. The embodiment of the present application does not impose any restrictions on the duration of the first audio samples. The duration of the first audio samples can be the same or different.

[0186] In some embodiments, step 201 may be implemented in the following manner: performing splicing processing on a plurality of first audio samples of the second object to obtain a second audio sample.

[0187] For example, multiple first audio samples of the second object (such as audio sample A of the second object for the text "I'm eating" such as [0.28, 0.16, 0, 0.14, 0.13, -0.06, 0.15, 0.03, 0.06], and audio sample A of the second object for the text "delicious" such as [0, 0.14, -0.06, 0.15]) are concatenated to obtain a second audio sample (such as [0.28, 0.16, 0, 0.14, 0.13, -0.06, 0.15, 0.03, 0.06, 0, 0.14, -0.06, 0.15]).

[0188] In step 202, audio synthesis is performed on the second audio sample and the text sample to obtain a synthesized audio sample of the second object.

[0189] Here, this application does not limit the model. The model is used to implement the audio synthesis method. The model can be an audio synthesis model to be trained. The audio synthesis model to be trained includes a first encoding layer, a second encoding layer, an attention layer, and a decoding layer.

[0190] In some embodiments, the above-mentioned "performing audio synthesis on the second audio sample and the text sample to obtain a synthesized audio sample of the second object" can be achieved in the following ways: performing a first encoding on the second audio sample through a first encoding layer to obtain a sound sample feature of the second object; performing a second encoding on the second audio sample through a second encoding layer to obtain a musicality sample feature of the second audio sample; performing attention processing on the musicality sample feature and the text sample feature through an attention layer to obtain a first sample feature; performing fusion processing on the sound sample feature, the first sample feature and the text sample feature to obtain a second sample feature; and performing decoding processing on the second sample feature through a decoding layer to obtain a synthesized audio sample.

[0191] In step 203, a model is trained based on the synthesized audio sample of the second object.

[0192] Here, taking the model as an audio synthesis model to be trained as an example, the process of training the model can be implemented in the following way: updating the parameters of the audio synthesis model to be trained, and using the updated parameters of the audio synthesis model to be trained as the parameters of the audio synthesis model.

[0193] In some embodiments, the above-mentioned "updating the parameters of the audio synthesis model to be trained based on the synthetic audio sample of the second object, and using the updated parameters of the audio synthesis model to be trained as the parameters of the audio synthesis model" can be achieved in the following way: constructing a loss function based on the synthetic audio sample and the audio sample label; updating the parameters of the audio synthesis model to be trained until the loss function converges, and using the parameters of the audio synthesis model to be trained when the loss function converges as the parameters of the audio synthesis model.

[0194] Continuing from the above embodiment, the above-mentioned “constructing a loss function based on synthetic audio samples and audio sample labels” can be implemented in the following way: encode the synthetic audio sample (such as [0.25, 0.25, 0.25, 0, 0, 0, 0, 1, -1, 2, -2]) to obtain a first encoding feature; encode the audio sample label (such as [1, -1, 2, -2, 1, -1, 2, -2]) to obtain a second encoding feature; construct a loss function based on the first encoding feature and the second encoding feature.

[0195] It should be noted that the audio sample label is the audio sample of the second object for the text sample. The embodiment of the present application does not limit the method for constructing the loss function. The loss function can be the mean absolute error, mean square error, etc. of two encoding vectors.

[0196] Continuing from the above embodiment, the above “updating the parameters of the audio synthesis model to be trained” can be implemented in the following manner: performing backpropagation in the audio synthesis model to be trained based on the loss function to obtain the gradient; and updating the parameters of the audio synthesis model to be trained based on the gradient.

[0197] It should be noted that back propagation is implemented through the back propagation algorithm, and the gradient of the loss function with respect to the parameters of the audio synthesis model to be trained is calculated by the derivative chain rule.

[0198] Through the embodiments of the present application, multiple first audio samples of the second object are trained to learn the speaking style of the first audio samples of the second object at different times, thereby ensuring that the synthesized audio samples have timbre consistency.

[0199] The following is an explanation of an exemplary application of the audio synthesis method provided in an embodiment of the present application in a practical application scenario.

[0200] In the related art, the audio of different texts of the same speaker has inconsistent timbre, and the speaker's speaking style is ignored when customizing the timbre of the target speaker.

[0201] In order to solve the above problems, an embodiment of the present application proposes an audio synthesis method, which deeply integrates the text and acoustic information of the target speaker, incorporates the speaker's speaking style into the synthesized audio, and makes full use of the data, filters out the data set with only one audio of the speaker and uses multiple audio data to encode to obtain a unified coding feature, ensuring the consistency of the timbre and enriching the speaker's timbre.

[0202] For example, see audio synthesis. Figure 5 , Figure 5 This is a first structural diagram of the speech synthesis model provided in an embodiment of the present application. The first process of training the speech synthesis model provided in an embodiment of the present application is described in detail below.

[0203] The model training method provided in the embodiment of the present application first filters the initial training data, only retains the data with the number of audios of the same speaker (i.e., the second object) greater than or equal to 2, and directly filters out the data with less than 2, and then processes the filtered training data audio (i.e., audio samples) into Mel features, and takes the dimension of the processed Mel feature as [batch_size, mel_length, 80] as an example for explanation, wherein batch_size is the number of batches, which is used to characterize the number of training data processed simultaneously, mel_length is the length of the Mel feature, and 80 is used to characterize the dimension (dim) of the dimensional space in which the Mel feature is located as 80. The Mel feature can be obtained by processing the audio data through Mel-frequency cepstral coefficients (MFCC). The Mel feature is a feature widely used in speaker segmentation, voiceprint recognition, speech recognition, and speech synthesis. The Mel frequency is extracted based on the auditory characteristics of the human ear, and the auditory characteristics of the human ear are in a nonlinear correspondence with the frequency of the audio. The spectral feature (i.e., the eighth feature) is calculated from the nonlinear correspondence according to the Mel-frequency cepstral coefficients. Multiple Mel features of the same speaker are concatenated to obtain a concatenated Mel concatenated feature 501. The dimension of the concatenated Mel concatenated feature 501 is [batch_size, 2000, 80]. The Mel features used for concatenation can be randomly selected from multiple Mel features. Because the speaker (i.e., the object, including the first object and the second object) is difficult to ensure the uniformity of the speaker's timbre due to changes in factors such as emotions and environment during recording or conversation, the use of the random Mel features of the speaker greatly maintains the uniformity of the speaker's timbre and maintains the stability of training. When concatenating 2000 frames of Mel concatenated features 501, the Mel concatenated features 501 have a specific length (such as 2000). The Mel concatenated features 501 with more than 2000 frames are directly cut to 2000 frames. For those with less than 2000 frames, the concatenated Mel concatenated features 501 are repeatedly concatenated multiple times until 2000 frames are reached and then stopped to obtain the final Mel concatenated features 501.

[0204] The training data provided in the embodiment of the present application is presented in the form of (<text, Mel feature label, Mel concatenated feature>), where the text corresponds to the Mel feature label one-to-one, and the Mel concatenated feature does not include the Mel feature label (that is, the Mel feature corresponding to the input audio whose text is consistent with the input text) to prevent information leakage. The Mel concatenated feature is input into the pre-trained speaker recognition model 502 for feature extraction to obtain the speaker feature, thereby ensuring the consistency of the timbre of the same speaker during the training process, wherein the dimension of the speaker feature is [batch_size, 1, dim], batch_size is the number of batches, which is used to characterize the number of speaker features processed simultaneously by the pre-trained speaker recognition model 502, and dim is used to characterize the dimension of the speaker feature in the high-dimensional space (such as 80), see Figure 6 , Figure 6 1 is a schematic diagram of the structure of a speaker recognition model provided in an embodiment of the present application. The reasoning process of the speaker recognition model is described in detail below.

[0205] First, the Mel-value concatenation feature is used as the input feature of the pre-trained speaker recognition model 502, and the input feature is subjected to residual processing through the residual module 5011 to obtain the residual feature, wherein the residual module 5011 includes multiple residual layers, and the number of residual layers is 4 as an example. The residual module 5011 includes residual layer 50111, residual layer 50112, residual layer 50113, residual layer 50114, etc. The feature input to the residual layer 50111 is the input feature, and the feature input to other residual layers is the output feature of the previous residual layer. The residual features are densely convolved through multiple dense convolution layers. The number of dense convolution layers is 3 for illustration. The multiple dense convolution layers include dense convolution layer 5012, dense convolution layer 5013 and dense convolution layer 5014. The dense convolution layer 5012 includes multiple neurons. Neuron 50120 is used as an example for illustration. The input feature input to neuron 50120 is encoded through a feedforward neural network 50121 to obtain a first feedforward encoded feature. The first feedforward encoded feature is encoded through a time delay neural network 50122 to obtain a second feedforward encoded feature. The first feedforward encoded feature is globally pooled and segment-level pooled respectively to obtain a global pooling feature and a segment-level pooling feature. The global pooling features and the segment-level pooling features are fused to obtain fused pooling features, the fused pooling features are activated through the activation network 50123 to obtain activation features, and the fused pooling features are mapped through the mapping network 50124 to obtain mapping features, and the mapping features and the second feedforward encoding features are fused to obtain the output features of the neuron 50120, wherein the activation network 50123 includes a first feedforward neural network and an activation layer (such as a linear mapping unit), the mapping network 50124 includes a second feedforward neural network and a mapping layer (such as a Sigmod function), and the model structures of the first feedforward neural network and the second feedforward neural network are the same as the model structure of the feedforward neural network 50121.

[0206] In the embodiment of the present application, a pre-trained speaker recognition model is used. Usually, the training data of the pre-trained speaker recognition model reaches hundreds of thousands of speaker audio clips, and the extracted speaker features are of high precision. Mel features are a common and frequently used speech feature in the field of speech synthesis. In the embodiment of the present application, multiple random Mel features of the same speaker are input and spliced ​​to extract speaker features to ensure timbre consistency. The pre-trained speaker recognition model 502 takes into account both recognition performance and reasoning efficiency. On public Chinese and English data sets, the recognition accuracy is improved. At the same time, it has a faster reasoning speed. The speaker recognition model is used to infer the input acoustic features to obtain speaker features (i.e., speech features, speaker embedding).

[0207] Continue to see Figure 5, the speaker features are linearly mapped by the linear mapping layer 503, and the speaker features are mapped to a specific dimension to obtain a linear mapping feature (ie, speech feature), and the dimension of the linear mapping feature is [batch_size, 1, dim].

[0208] Secondly, the Mel concatenation feature 501 is encoded through a neural network model 504 (such as a residual network) to obtain acoustic feature information (i.e., musical features). The dimension of the acoustic feature information is [batch_size, 2000, dim1], where dim1 is the number of convolution kernels in the last layer of the neural network in the speaker recognition model. Figure 7 , Figure 7 is a schematic diagram of the neural network model structure provided in the embodiment of the present application, such as Figure 7 As shown, the neural network model 504 includes a convolution layer, a maximum pooling layer and a fully connected layer, and performs multi-layer convolution and pooling operations on the Mel features, and the output is a rhythmic feature (i.e., a musical feature), wherein the convolution layer contains multiple convolution kernels, and each convolution layer corresponds to a convolution kernel of the same number of convolution kernels.

[0209] Continue to see Figure 5 , the speaker's acoustic features are convolved through one-dimensional convolution 505, and the convolution features after convolution (i.e., the third features) are used as key features (i.e., key vectors) and value features (i.e., value vectors) of the scaled dot product attention mechanism 508, wherein the step size of the convolution kernel in the one-dimensional convolution 505 is a specific step size (e.g., 10) to reduce the amount of calculation, the one-dimensional convolution learns the local feature information of the acoustic feature information and reduces the length by a specific ratio (e.g., one tenth) to obtain the convolution feature, and the one-dimensional convolution is used to scale the length of the concatenated Mel feature to 1 / 10. Since the computational complexity of the scaled dot product attention is the square of the length of the feature, the length is scaled to 1 / 10, which greatly reduces the amount of calculation, wherein the dimension of the convolution feature is [batch_size, 200, dim2], and dim2 is the number of convolution kernels in the last layer of the neural network in the one-dimensional convolution.

[0210] Then, the text features are obtained, wherein the text features are obtained by encoding the text (i.e., the text sample) through a Byte Pair Encoder (BPE) and an embedding layer, and the text features are encoded through the encoder module 506 to learn the emotional information of the text. In the embodiment of the present application, after BPE encoding is performed on the text (i.e., the first text), the text features obtained after BPE encoding are input into the embedding layer, and the semantic information, emotional information, etc. contained in the text are learned through the embedding layer to obtain the encoded text features (i.e., the text features of the first text) so as to guide the content and emotion of the synthesized audio. The text can be Chinese characters, words, etc., and the text is not a phoneme.

[0211] The encoded text features are convolved through the dense convolution layer 507, and the encoded convolution features are mapped to a specific dimension as the query features (i.e., query vectors) of the scaled dot product attention mechanism 508. The output dimension of the attention features output by the scaled dot product attention mechanism 508 is [batch_size, text_length, dim]. The scaled dot product attention mechanism 508 makes the acoustic information and text information pay attention to each other and influence each other, so that the text features pay attention to the associated acoustic features (such as rhythm), thereby maintaining the rhythmic style characteristics of the speaker when generating audio. It should be noted that rhythm is presented in the speech features, but it is also closely related to the text features. See Figure 8 , Figure 8 is a schematic diagram of the structure of the scaled dot product attention model provided in the embodiment of the present application, such as Figure 8 As shown, the scaled dot product attention model 508 includes N multi-head attention layers 801, a residual normalization module 802, a feedforward neural network 803, a residual normalization module 804, a linear layer 805 and a mapping layer 806. The N multi-head attention layers 801 perform multi-head attention processing on the key feature (i.e., key vector), the value feature (i.e., value vector) and the query feature (i.e., query vector) to obtain the multi-head attention feature. The residual normalization module 802 performs residual processing and normalization processing on the multi-head attention feature and the query feature to obtain the first residual feature. The first residual feature is encoded through the feedforward neural network 803 to obtain the feedforward encoded feature, the feedforward encoded feature and the first residual feature are residually processed and normalized through the residual normalization module 804 to obtain the second residual feature, and the second residual feature is data processed through the linear layer 805 and the mapping layer 806 to obtain the attention feature, wherein the residual normalization module 802 includes a residual layer and a normalization layer, the residual layer is used to prevent network degradation, and the normalization layer is used to normalize the activation value obtained after the activation processing of the feature output by the residual layer.

[0212] Finally, continue to see Figure 5 , the attention feature is added to the speaker feature to obtain the first fusion feature. The first fusion feature is related to the speaker's speaking style and rhythm. The first fusion feature is rich in feature information such as the speaker's speaking style and rhythm. The dimension of the first fusion feature is [batch_size, text_length, dim]. Then the first fusion feature is added to the text feature to obtain the second fusion feature. The dimension of the second fusion feature is [batch_size, text_length, dim]. Compared with the first fusion feature, the second fusion feature contains more text information.

[0213] See also Fig. 9 , Fig. 9This is a schematic diagram of the second structure of the speech synthesis model provided in the embodiment of the present application. The process of reasoning through the second structure of the speech synthesis model provided in the embodiment of the present application is described in detail below.

[0214] The obtained second fusion feature is concatenated with the text feature, and the concatenated feature is input into the autoregressive module 901 (such as the GPT model) to obtain the hidden feature. The hidden feature is decoded by the Mel decoder 902 to obtain the predicted Mel feature. The loss of the predicted Mel audio (i.e., the audio sample data) and the target Mel audio (the speech of the speaker corresponding to the text, i.e., the audio sample label) is calculated, and the parameters of the audio synthesis model are iteratively updated for multiple rounds based on the calculated loss until the loss converges. When the loss converges, the audio synthesis model is saved.

[0215] See also Fig.10 , Fig.10 This is a third structural diagram of the speech synthesis model provided in an embodiment of the present application. The process of training the autoregressive module 901 and the Mel decoder 902 provided in an embodiment of the present application is described in detail below.

[0216] The real Mel feature (i.e., the audio sample label) is encoded by the quantization encoder 9031 and the quantization module 9032 in the vector quantization model 903 to obtain the quantized feature. Fig.11 , Fig.11 It is a schematic diagram of the structure of a vector quantization model provided in an embodiment of the present application. The vector quantization model includes a quantization encoder 9031, a quantization module 9032 and a quantization decoder 9033. The quantization encoder 9031 is used to learn useful information in the real Mel feature to obtain an intermediate feature (continuous vector feature). The quantization module 9032 converts the intermediate feature into a discrete feature and a latent intermediate feature (continuous vector feature), wherein the latent intermediate feature and the discretized feature can be converted into each other without the need for an additional network module. The quantization decoder 9033 is used to restore the latent intermediate feature to a Mel feature.

[0217] The obtained second fusion feature, text feature and quantization feature are spliced, and the spliced ​​features are input into the autoregressive module 901 (such as the GPT model) to obtain hidden features. The hidden features are decoded by the Mel decoder 902 to obtain the first predicted Mel features, or the hidden features are decoded by the quantization decoder 9033 to obtain the second predicted Mel features. The predicted Mel features include the first predicted Mel features and the second predicted Mel features. The predicted Mel audio (i.e., audio sample data) obtained by processing the predicted Mel features through the reverberator is subjected to loss calculation with the target Mel audio (the speech of the speaker corresponding to the text, i.e., the audio sample label) to obtain the first loss, and the hidden features are subjected to cross entropy loss with the true Mel features to obtain the second loss. Based on the calculated first loss and second loss, the parameters of the autoregressive module 901 and the Mel decoder 902 are iteratively updated for multiple rounds until the loss converges. When the loss converges, the autoregressive module 901 and the Mel decoder 902 are saved.

[0218] See also Fig.12 , Fig.12 This is a schematic diagram of audio synthesis provided in an embodiment of the present application. The process of audio synthesis provided in an embodiment of the present application is explained below.

[0219] Load the saved model, input a 10-20s audio of the target speaker and any text, and you can synthesize audio that is consistent with the target speaker's timbre. During the inference process, if the audio is too short, you can repeatedly splice the audio to get an audio of moderate length. The text features are the features obtained by encoding any text.

[0220] First, the audio 601 of the target speaker is processed into Mel features, and the Mel features can be repeatedly spliced ​​multiple times to obtain Mel splicing features 602. The specific number of splicing times is determined according to the length of the Mel features. The pre-trained speaker recognition model 603 extracts features from the Mel splicing features 602 to obtain speaker features.

[0221] Secondly, the Mel-concatenated features 602 are encoded through a neural network model 604 (such as a residual network) to obtain the acoustic features of the speaker, and the acoustic features of the speaker are convolved through a one-dimensional convolution 605 to obtain the key features and value features of the scaled dot product attention mechanism 606, wherein the step size of the convolution kernel in the one-dimensional convolution 605 is a specific step size (such as 10).

[0222] Then, the text features are obtained, and the text features are encoded through the dense convolution layer 607. The encoded convolution text features are used as query features of the scaled dot product attention mechanism 606. The query features, key features and value features are paid attention to through the scaled dot product attention mechanism 606 to obtain attention example features, and the attention features are added to the speaker features to obtain the first fused features, and then the second fused features are added to the text features to obtain the enriched second fused features. The second fused features are fused with the text features to obtain the third fused features, which are input into the autoregressive module 608 (such as the GPT model, which can mask the multi-head attention module) to obtain hidden features. The hidden features are decoded by the Mel decoder 609 to obtain predicted Mel features, and the predicted Mel features are restored by the vocoder 610 to obtain predicted Mel audio.

[0223] When synthesizing audio, the autoregressive module 608 with good effect is applied to the field of user interaction. By optimizing the text features input to the autoregressive module 608, the synthesized speech not only maintains the speaking style of the target speaker such as rhythm, but also improves the speaker similarity, reaching a level that is indistinguishable from the real thing.

[0224] When applied in the field of user interaction, audio 601 may be a 10-20 s audio recorded by an agent whose timbre is suitable for the scene and who has strong interactive ability. By inputting audio 601 into the saved model, the agent's voice can be synthesized. After obtaining the right to use the agent's timbre, corresponding synthesized audio is generated for different interactive texts through the audio recorded by the agent, so as to facilitate interaction with the user based on the synthesized audio, thereby effectively improving the similarity between the timbre and speaking style of the target speaker and the synthesized audio.

[0225] In practical applications, a vector quantization model is first trained, wherein the encoder and quantization module of the vector quantization model are used to extract quantitative features, and then the fusion features obtained by fusing the text features and the Mel-concatenation features according to the solution proposed in the embodiment of the present application are concatenated with the quantization features, and the concatenated features obtained by concatenation are used to train the autoregressive module 608. When in use, the input text (i.e., the first text) and a 10-20s audio (i.e., audio data) of the target speaker can be used to synthesize a speech (i.e., synthesized audio data) with a high similarity to the timbre and speaking style of the target speaker, which can be used in user interaction scenarios. In the user interaction scenario, the voice stream signal transmitted in real time from the telephone user end is first received, and then the voice stream signal is input into the speech recognition model to obtain the user's speech content. After the natural language processing model performs speech recognition, the user's intention is obtained, and the text content to be fed back (i.e., the first text) is obtained. The fed back text content is subjected to audio synthesis by the audio synthesis method provided in the embodiment of the present application to obtain synthesized speech (i.e., synthesized audio data), and the synthesized speech is fed back to the customer via the phone for response, thereby completing the interaction with the user.

[0226] In summary, the similarity between the timbre and speaking style of the target speaker and the synthesized audio in the zero-sample voice cloning scenario can be effectively improved. The timbre of the agent can be customized in the outbound call scenario. With the consent of the agent, a 10-20s audio of the agent is recorded and input into the trained model to create a cloned timbre model of the agent. The agent's timbre can then be used for outbound calls. By deeply integrating the speaker's text information and acoustic information, the two interact and influence each other, and the acoustic information such as the speaker's speaking style is integrated into the speaker's text. Considering that the speaker's speaking style may be inconsistent at different times, multiple Mel features are processed during training to obtain speaker features to ensure the consistency of the timbre, so that when used, a fake effect can be achieved.

[0227] The following is a description of an exemplary structure of the audio synthesis device 555 provided in the embodiment of the present application implemented as a software module. In some embodiments, Figure 2A As shown, the software modules stored in the audio synthesis device 555 of the memory 550 may include:

[0228] The data collection module 5551 is used to determine the text features of the first text and the audio data of the first object.

[0229] The data processing module 5552 is used to perform a first encoding on the audio data to obtain the speech features of the first object, and to perform a second encoding on the audio data to obtain the musical features of the audio data; to perform attention processing on the musical features and the text features to obtain the first features; and to perform fusion processing on the speech features, the first features and the text features to obtain the second features.

[0230] The audio synthesis module 5553 is used to decode the second feature to obtain synthesized audio.

[0231] In some embodiments, the data processing module 5552 is also used to normalize the musical features to obtain the third feature, and to normalize the text features to obtain the fourth feature; perform multi-head attention processing on the third feature and the fourth feature to obtain the fifth feature; and determine the first feature based on the third feature and the fifth feature.

[0232] In some embodiments, the data processing module 5552 is also used to determine the key vector of the audio data and the value vector of the audio data based on the third feature, and determine the query vector of the first text based on the fourth feature; determine the similarity matrix based on the query vector and the key vector; and determine the product of the similarity matrix and the value vector as the fifth feature.

[0233] In some embodiments, the data processing module 5552 is also used to fuse the third feature and the fifth feature to obtain the sixth feature, and map the sixth feature to obtain the seventh feature; and fuse the sixth feature and the seventh feature to obtain the first feature.

[0234] In some embodiments, the data processing module 5552 is also used to perform cepstrum processing on the audio data to obtain an eighth feature; perform residual processing on the eighth feature to obtain a ninth feature; perform context mask processing on the ninth feature to obtain a tenth feature; and perform dense convolution processing on the tenth feature to obtain a speech feature of the first object.

[0235] In some embodiments, the data processing module 5552 is also used to perform cepstrum processing on the audio data to obtain an eighth feature; perform convolution processing on the eighth feature, perform pooling processing on the convolved eighth feature to obtain an eleventh feature; perform mapping processing on the eleventh feature, and perform scaling processing on the mapped eleventh feature to obtain the musical characteristics of the audio data.

[0236] In some embodiments, the data processing module 5552 is also used to perform frequency domain transform processing on the audio data to obtain an audio spectrum; convert the audio spectrum into a first power spectrum, and filter the first power spectrum to obtain a second power spectrum; perform cosine transform on the logarithm of the second power spectrum to obtain the eighth feature.

[0237] In some embodiments, the audio synthesis module 5553 is also used to perform prediction processing on the speech feature and the second feature to obtain a twelfth feature; perform mapping processing on the twelfth feature and up-sample the mapped twelfth feature to obtain a thirteenth feature; perform deconvolution processing on the thirteenth feature to obtain synthesized audio.

[0238] The following is a description of an exemplary structure of the audio synthesis model training device 556 provided in the embodiment of the present application implemented as a software module. In some embodiments, Figure 2B As shown, the software modules stored in the training device 556 of the audio synthesis model in the memory 550 may include:

[0239] The preprocessing module 5561 is used to fuse multiple first audio samples of the second object to obtain a second audio sample.

[0240] The model training module 5562 is used to perform audio synthesis on the second audio sample and the text sample based on the model to obtain a synthesized audio sample of the second object; and train the model based on the synthesized audio sample of the second object.

[0241] An embodiment of the present application provides a computer program product, which includes computer executable instructions. The computer executable instructions are stored in a computer-readable storage medium. A processor of an electronic device reads the computer executable instructions from the computer-readable storage medium, and the processor executes the computer executable instructions, so that the electronic device executes the audio synthesis method described above in the embodiment of the present application.

[0242] The present application embodiment provides a computer-readable storage medium storing computer-executable instructions, wherein the computer-executable instructions or computer programs are stored. When the computer-executable instructions or computer programs are executed by a processor, the processor will be caused to execute the audio synthesis method provided by the present application embodiment, for example, FIG. 3A to FIG. 3E The audio synthesis method shown or Figure 4 The training method of the audio synthesis model is shown.

[0243] In some embodiments, the computer-readable storage medium may be a memory such as RAM, ROM, flash memory, magnetic surface memory, optical disk, or CD-ROM; or may be various devices including one or any combination of the above memories.

[0244] In some embodiments, computer executable instructions may be in the form of a program, software, software module, script or code, written in any form of programming language (including compiled or interpreted languages, or declarative or procedural languages), and may be deployed in any form, including as a stand-alone program or as a module, component, subroutine or other unit suitable for use in a computing environment.

[0245] As an example, computer-executable instructions may, but need not, correspond to a file in a file system, may be stored as part of a file that stores other programs or data, such as in one or more scripts in a HyperText Markup Language (HTML) document, in a single file dedicated to the program in question, or in multiple coordinated files (e.g., files storing one or more modules, subroutines, or code portions).

[0246] As an example, computer executable instructions may be deployed to be executed on one electronic device, or on multiple electronic devices located at one site, or on multiple electronic devices distributed at multiple sites and interconnected by a communication network.

[0247] To summarize, the audio data is respectively subjected to the first encoding and the second encoding to obtain the speech features of the first object and the musical features of the audio data, and then the musical features and the text features are subjected to attention processing to obtain the first feature, and the speech features, the first features and the text features are fused to obtain the second feature. In this way, the speech features provide sound information related to the first object, and the text features provide semantic content. By integrating multimodal information such as musical features, speech features, and text features, the performance of audio synthesis is improved to obtain synthesized audio that conforms to the sound of the first object, and the text content of the synthesized audio is consistent with the text content of the first text, thereby improving the accuracy of audio synthesis.

[0248] The above is only an embodiment of the present application and is not intended to limit the protection scope of the present application. Any modifications, equivalent substitutions and improvements made within the spirit and scope of the present application are included in the protection scope of the present application.

Claims

1. An audio synthesis method, characterized in that: The method comprises: determining text features of a first text and audio data of a first object; Performing a first encoding on the audio data to obtain a speech feature of the first object, and performing a second encoding on the audio data to obtain a musical feature of the audio data; Performing attention processing on the musical feature and the text feature to obtain a first feature; fusing the speech feature, the first feature, and the text feature to obtain a second feature; The second feature is decoded to obtain synthesized audio.

2. The method according to claim 1, characterized in that The performing attention processing on the musical feature and the text feature to obtain a first feature includes: Normalizing the musical feature to obtain a third feature, and normalizing the text feature to obtain a fourth feature; Performing multi-head attention processing on the third feature and the fourth feature to obtain a fifth feature; The first feature is determined based on the third feature and the fifth feature.

3. The method according to claim 2, characterized in that The performing multi-head attention processing on the third feature and the fourth feature to obtain the fifth feature includes: Based on the third feature, determining a key vector of the audio data and a value vector of the audio data, and based on the fourth feature, determining a query vector of the first text; Determining a similarity matrix of the first text based on the query vector and the key vector; The fifth feature is obtained according to the similarity matrix and the value vector.

4. The method according to claim 2, characterized in that: The determining the first feature based on the third feature and the fifth feature includes: The third feature and the fifth feature are fused to obtain a sixth feature; Mapping the sixth feature to obtain a seventh feature; The sixth feature and the seventh feature are fused to obtain the first feature.

5. The method according to claim 1, characterized in that The first encoding of the audio data to obtain the speech feature of the first object includes: Performing cepstrum processing on the audio data to obtain an eighth feature; Performing residual processing on the eighth feature to obtain a ninth feature; Performing context mask processing on the ninth feature to obtain a tenth feature; Perform dense convolution processing on the tenth feature to obtain the speech feature of the first object.

6. The method according to claim 1, characterized in that The performing a second encoding on the audio data to obtain the musical characteristics of the audio data includes: Performing cepstrum processing on the audio data to obtain an eighth feature; Performing convolution processing on the eighth feature, and performing pooling processing on the convolved eighth feature to obtain an eleventh feature; The eleventh feature is mapped, and the mapped eleventh feature is scaled to obtain the musicality feature of the audio data.

7. The method according to claim 5 or 6, characterized in that: The audio data is subjected to cepstrum processing to obtain an eighth feature, including Performing frequency domain transformation processing on the audio data to obtain an audio spectrum; Converting the audio spectrum into a first power spectrum, and filtering the first power spectrum to obtain a second power spectrum; Perform a cosine transform on the logarithm of the second power spectrum to obtain the eighth feature.

8. The method according to claim 1, characterized in that: The decoding process of the second feature to obtain synthesized audio includes: Performing prediction processing on the speech feature and the second feature to obtain a twelfth feature; Performing mapping processing on the twelfth feature, and performing upsampling processing on the mapped twelfth feature to obtain a thirteenth feature; Deconvolution is performed on the thirteenth feature to obtain the synthesized audio.

9. A model training method, the training method comprising: fusing the plurality of first audio samples of the second object to obtain a second audio sample; Performing audio synthesis on the second audio sample and the text sample to obtain a synthesized audio sample of the second object; A model is trained based on the synthesized audio samples of the second object.

10. An electronic device, characterized in that: The electronic device comprises: A memory for storing computer executable instructions; The processor is used to implement the audio synthesis method described in any one of claims 1 to 8, or the model training method described in claim 9 when executing the computer executable instructions or computer program stored in the memory.

11. A computer-readable storage medium storing computer-executable instructions or a computer program, characterized in that: When the computer executable instructions or computer program are executed by a processor, the audio synthesis method described in any one of claims 1 to 8, or the model training method described in claim 9 is implemented.

12. A computer program product comprising computer executable instructions or a computer program, characterized in that: When the computer executable instructions or computer program are executed by a processor, the audio synthesis method described in any one of claims 1 to 8, or the model training method described in claim 9 is implemented.

Images