Audio file generation method, device, equipment and computer-readable storage medium

Abstract

The embodiments of the present application provide an audio file generation method, device, equipment and computer-readable storage medium, which relate to the field of artificial intelligence technology. The method includes: extracting features of the received text and the target object respectively, and obtaining a first feature vector of the text and a second feature vector of the target object; splicing the first feature vector and the second feature vector to form a spliced ​​feature vector; predicting the audio attribute features of the spliced ​​feature vector to obtain at least two audio attribute features of different types; and generating an audio file containing the text and having the sound characteristics of the target object based on at least two of the audio attribute features. Through the embodiments of the present application, the actual application scope of the audio file generation method can be improved, and the generalization of the method can be improved.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of Internet technology, and relate to but are not limited to a method, apparatus, device, and computer-readable storage medium for generating an audio file. Background Art

[0002] In recent years, with the development of deep learning technology and hardware computing capabilities such as graphics processing units (GPUs), the generation of digital content has suddenly become a hot research direction, such as audio, pictures, videos, virtual humans, etc. In the audio field, text to speech (TTS) technology can currently simulate human levels, such as the generation of music content.

[0003] In the related art, the generation of music content can be roughly divided into two categories: the first category is pure music, without lyrics, mainly generating a piece of music for a single or multiple instruments, such as piano music, guitar music, pure music, etc.; the second category is vocal music, in which the core is the vocal part, singing the lyrics, and the accompaniment part corresponds to the background music. For the second category of vocal music, the method in the related art is to add more music attribute features (such as pitch, energy, singer, rhythm, etc.) on the basis of the traditional TTS model to generate music.

[0004] The methods in the related art need to explicitly give multiple features of the generated audio. In this way, multiple features need to be explicitly extracted from the audio. The cost of manually annotating data is high, and it can only be applied to the training stage of the model. In the prediction stage during actual use, since the generated audio is unknown, it is impossible to directly give accurate music attribute features, resulting in a limited scope of actual use. The methods in the related art have poor generalization. Summary of the invention

[0005] The embodiment of the present application provides an audio file generation method, device, equipment and computer-readable storage medium, which relates to the field of artificial intelligence technology. By extracting features from text and target objects, and predicting the concatenated feature vectors corresponding to the text and target objects, at least two audio attribute features with different types are obtained, and an audio file is generated. In this way, an audio file can be generated by simply inputting text and target objects, thereby increasing the practical application scope of the audio file generation method and improving the generalization of the method.

[0006] The technical solution of the embodiment of the present application is implemented as follows:

[0007] An embodiment of the present application provides an audio file generation method, comprising: receiving an audio file generation request, the audio file generation request including text and a target object; performing feature extraction on the text and the target object respectively, and correspondingly obtaining a first feature vector of the text and a second feature vector of the target object; splicing the first feature vector and the second feature vector to form a spliced ​​feature vector; predicting audio attribute features of the spliced ​​feature vector to obtain at least two audio attribute features of different types; and generating an audio file containing the text and having the sound characteristics of the target object based on the at least two audio attribute features.

[0008] The embodiment of the present application provides an audio file generation device, including: a receiving module, used to receive an audio file generation request, the audio file generation request including text and a target object; a feature extraction module, used to perform feature extraction on the text and the target object respectively, and obtain a first feature vector of the text and a second feature vector of the target object accordingly; a splicing module, used to splice the first feature vector and the second feature vector to form a spliced ​​feature vector; a prediction module, used to predict audio attribute features of the spliced ​​feature vector, and obtain at least two audio attribute features of different types; and a generation module, used to generate an audio file containing the text and having the sound characteristics of the target object according to at least two of the audio attribute features.

[0009] An embodiment of the present application provides a computer program product or a computer program, wherein the computer program product or the computer program includes computer instructions, and the computer instructions are stored in a computer-readable storage medium; wherein a processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor is used to execute the computer instructions to implement the above-mentioned network structure search method.

[0010] An embodiment of the present application provides an audio file generation device, comprising: a memory for storing executable instructions; and a processor for implementing the above-mentioned audio file generation method when executing the executable instructions stored in the memory.

[0011] An embodiment of the present application provides a computer-readable storage medium storing executable instructions for causing a processor to execute the executable instructions to implement the above-mentioned audio file generation method.

[0012] The embodiments of the present application have the following beneficial effects: by performing feature extraction on the text and the target object respectively, a first feature vector of the text and a second feature vector of the target object are obtained correspondingly; and the first feature vector and the second feature vector are spliced ​​to form a spliced ​​feature vector; the spliced ​​feature vector is predicted to obtain at least two audio attribute features of different types, and an audio file is generated according to the different types of audio attribute features. In this way, an audio file can be generated by simply inputting the text and the target object, thereby improving the actual use scope of the audio file generation method and improving the generalization of the method. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] Figure 1 It is a schematic diagram of a process for generating vocal music audio in the related technology;

[0014] Figure 2 This is an optional architecture diagram of the audio file generation system provided in the embodiment of the present application;

[0015] Figure 3 It is a schematic diagram of the structure of the server provided in the embodiment of the present application;

[0016] Figure 4 This is an optional flowchart of the audio file generation method provided in the embodiment of the present application;

[0017] Figure 5 This is an optional flowchart of the audio file generation method provided in the embodiment of the present application;

[0018] Figure 6 This is an optional flowchart of the audio file generation method provided in the embodiment of the present application;

[0019] Figure 7 This is an optional flowchart of the audio file generation method provided in the embodiment of the present application;

[0020] Figure 8 This is an optional flowchart of the audio file generation method provided in the embodiment of the present application;

[0021] Fig. 9 This is an optional flowchart of the audio file generation method provided in the embodiment of the present application;

[0022] Fig.10 It is an optional flowchart of the training method of the speech synthesis model provided in the embodiment of the present application;

[0023] Fig.11 This is an interface diagram of an audio file generation product provided in an embodiment of the present application;

[0024] Fig.12It is a schematic diagram of the overall process of generating an audio file in an embodiment of the present application;

[0025] Fig.13A is a network structure diagram of an encoder module provided in an embodiment of the present application;

[0026] Fig. 13B is a schematic diagram of a transformer structure provided in an embodiment of the present application;

[0027] Fig.14 It is a network structure diagram of the overall sub-decoder module provided in an embodiment of the present application. DETAILED DESCRIPTION

[0028] In order to make the purpose, technical solutions and advantages of the present application clearer, the present application will be further described in detail below in conjunction with the accompanying drawings. The described embodiments should not be regarded as limiting the present application. All other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of this application.

[0029] In the following description, reference is made to "some embodiments", which describe a subset of all possible embodiments, but it is understood that "some embodiments" may be the same subset or different subsets of all possible embodiments, and may be combined with each other without conflict. Unless otherwise defined, all technical and scientific terms used in the embodiments of the present application have the same meaning as those commonly understood by those skilled in the art of the technical field of the embodiments of the present application. The terms used in the embodiments of the present application are only for the purpose of describing the embodiments of the present application and are not intended to limit the present application.

[0030] In order to better understand the audio file generation method provided in the embodiment of the present application, the audio file generation method in the related art is first described:

[0031] In the generation of vocal music audio, the methods in the related technologies mainly draw on the TTS method, adding more music attribute features (such as pitch, energy, singer and rhythm, etc.) to generate music on the basis of the traditional TTS model. Figure 1 It is a flowchart of generating vocal music audio in the related technology, such as Figure 1 As shown, the model input information includes text 101, pitch 102, energy 103, rhythm 104 and singer 105. This information is input into the encoder 106 for encoding and feature extraction, and then the extracted features are input into the decoder 107 for decoding to form a spectrogram 108, and then the spectrogram 108 is input into the vocoder 109 for spectrum sound conversion to obtain audio / sound wave 110.

[0032] For the text-to-music task, the biggest disadvantage of the above-mentioned related technologies is that they need to explicitly give multiple features of the generated audio, such as Figure 1 Pitch 102, energy 103, rhythm 104, and singer 105 in the audio. This will bring at least two problems: First, except for singer 105, the rest of the features need to be explicitly extracted from the audio; and pitch 102 and energy 103 can be calculated using existing tools, but rhythm 104 requires audio alignment at the text 101 or phoneme level. Obviously, the cost of manually annotating data is high, and training a specific alignment model also requires manually annotated data. Second, in the training phase, these features can be extracted based on the original audio, but in the prediction phase, the generated audio is unknown, and it is impossible to directly give accurate pitch, energy, rhythm and other information. The actual use is limited and the generalization is poor.

[0033] In order to solve at least one problem existing in the audio file generation method in the related art, the embodiment of the present application proposes an audio file generation method, which is an end-to-end text-to-music generation method, which can generate a piece of music by simply giving any piece of text and specifying a singer. The vocal part corresponds to the content of the text, the accompaniment part corresponds to the background music, and the generated music has the voice characteristics of the specified singer.

[0034] The audio file generation method provided by the embodiment of the present application first receives an audio file generation request, wherein the audio file generation request includes text and a target object; then, feature extraction is performed on the text and the target object respectively, and a first feature vector of the text and a second feature vector of the target object are obtained correspondingly; and the first feature vector and the second feature vector are spliced ​​to form a spliced ​​feature vector; then, audio attribute features are predicted for the spliced ​​feature vector respectively, and at least two audio attribute features of different types are obtained correspondingly; finally, based on the at least two audio attribute features, an audio file containing the text and having the sound characteristics of the target object is generated. In this way, an audio file can be generated by simply inputting text and a target object, thereby improving the actual application scope of the audio file generation method and improving the generalization of the method.

[0035] The following describes an exemplary application of the audio file generating device of the embodiment of the present application. In one implementation, the audio file generating device provided by the embodiment of the present application can be implemented as any terminal that can play audio, such as a laptop, a tablet computer, a desktop computer, a mobile device (e.g., a mobile phone, a portable music player, a personal digital assistant, a dedicated messaging device, a portable gaming device), an intelligent robot, etc. In another implementation, the audio file generating device provided by the embodiment of the present application can also be implemented as a server. The following describes an exemplary application when the audio file generating device is implemented as a server.

[0036] See also Figure 2 , Figure 2 It is an optional architecture diagram of the audio file generation system 10 provided by the embodiment of the present application. To realize the generation of an audio file containing text content and having the sound characteristics of a target object, the audio file generation system 10 provided by the embodiment of the present application includes a terminal 100, a network 200 and a server 300, wherein an audio file generation application is run on the terminal 100, such as the audio file generation application can be a music player that automatically arranges and generates music, and a user can request to generate an audio file (i.e., a piece of music) on the client of the music player. In the process of implementation, the terminal 100 sends an audio file generation request to the server 300 through the network 200, and the audio file generation request includes a text and a target object, the text is the lyrics of the music finally generated, the target object can be a certain singer that the user likes, and the music finally generated can be the music sung by the singer. After receiving the audio file generation request, the server 300 responds to the audio file generation request by performing feature extraction on the text and the target object respectively, and correspondingly obtaining a first feature vector of the text and a second feature vector of the target object; splicing the first feature vector and the second feature vector to form a spliced ​​feature vector; predicting audio attribute features of the spliced ​​feature vector respectively, and correspondingly obtaining at least two audio attribute features of different types; generating an audio file containing the text and having the sound characteristics of the target object based on the at least two audio attribute features; and determining the generated audio file as the request result of the audio file generation request, and sending the generated audio file to the terminal 100, which can play the audio file after receiving the audio file.

[0037] The audio file generation method provided in the embodiment of the present application also relates to the field of artificial intelligence technology, and can be implemented at least through speech technology and machine learning technology in artificial intelligence technology. Among them, the key technologies of speech technology (Speech Technology) include automatic speech recognition technology (ASR) and speech synthesis technology (TTS) and voiceprint recognition technology, which enable computers to listen, see, speak and feel, and are the development direction of human-computer interaction in the future, among which speech becomes one of the most promising human-computer interaction methods in the future. Machine learning (ML, Machine Learning) is a multi-field cross-disciplinary subject involving probability theory, statistics, approximation theory, convex analysis, algorithm complexity theory and other disciplines. It specializes in studying how computers simulate or realize human learning behavior to acquire new knowledge or skills, reorganize existing knowledge structures and continuously improve their own performance. Machine learning is the core of artificial intelligence and the fundamental way to make computers intelligent. Its applications are spread across all fields of artificial intelligence. Machine learning and deep learning usually include artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning, and teaching learning. In the embodiment of the present application, the response to the network structure search request is realized by machine learning technology to automatically search for the target network structure, and to realize the training and model optimization of the controller and the score model.

[0038] Figure 3 is a schematic diagram of the structure of the server 300 provided in the embodiment of the present application, Figure 3 The server 300 shown includes: at least one processor 310, a memory 350, at least one network interface 320 and a user interface 330. The various components in the server 300 are coupled together via a bus system 340. It is understood that the bus system 340 is used to achieve connection and communication between these components. In addition to the data bus, the bus system 340 also includes a power bus, a control bus and a status signal bus. However, for the sake of clarity, the bus system 340 is not described in detail. Figure 3 Various buses are labeled as bus system 340 .

[0039] The processor 310 can be an integrated circuit chip with signal processing capabilities, such as a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc., where the general-purpose processor can be a microprocessor or any conventional processor, etc.

[0040] The user interface 330 includes one or more output devices 331 that enable presentation of media content, including one or more speakers and / or one or more visual display screens. The user interface 330 also includes one or more input devices 332, including user interface components that facilitate user input, such as a keyboard, mouse, microphone, touch screen display, camera, other input buttons and controls.

[0041] The memory 350 may be removable, non-removable or a combination thereof. Exemplary hardware devices include solid-state memory, hard disk drive, optical disk drive, etc. The memory 350 may optionally include one or more storage devices physically away from the processor 310. The memory 350 includes a volatile memory or a non-volatile memory, and may also include both volatile and non-volatile memory. The non-volatile memory may be a read-only memory (ROM), and the volatile memory may be a random access memory (RAM). The memory 350 described in the embodiment of the present application is intended to include any suitable type of memory. In some embodiments, the memory 350 can store data to support various operations, and examples of these data include programs, modules, and data structures or subsets or supersets thereof, as exemplarily described below.

[0042] Operating system 351, including system programs for processing various basic system services and performing hardware-related tasks, such as framework layer, core library layer, driver layer, etc., for implementing various basic services and processing hardware-based tasks;

[0043] A network communication module 352, for reaching other computing devices via one or more (wired or wireless) network interfaces 320, exemplary network interfaces 320 include: Bluetooth, Wireless Compatibility Authentication (WiFi), and Universal Serial Bus (USB);

[0044] The input processing module 353 is used to detect one or more user inputs or interactions from one of the one or more input devices 332 and translate the detected inputs or interactions.

[0045] In some embodiments, the device provided in the embodiments of the present application can be implemented in software. Figure 3An audio file generating device 354 stored in the memory 350 is shown. The audio file generating device 354 may be an audio file generating device in the server 300. The audio file generating device 354 may be software in the form of a program or a plug-in, and may include the following software modules: a receiving module 3541, a feature extraction module 3542, a splicing module 3543, a prediction module 3544, and a generating module 3545. These modules are logical, and thus may be arbitrarily combined or further split according to the functions implemented. The functions of each module will be described below.

[0046] In other embodiments, the device provided in the embodiments of the present application can be implemented in hardware. As an example, the device provided in the embodiments of the present application can be a processor in the form of a hardware decoding processor, which is programmed to execute the audio file generation method provided in the embodiments of the present application. For example, the processor in the form of a hardware decoding processor can adopt one or more application-specific integrated circuits (ASICs), DSPs, programmable logic devices (PLDs), complex programmable logic devices (CPLDs), field programmable gate arrays (FPGAs) or other electronic components.

[0047] The following will describe the audio file generation method provided by the embodiment of the present application in conjunction with the exemplary application and implementation of the server 300 provided by the embodiment of the present application. Figure 4 , Figure 4 is an optional flow chart of the audio file generation method provided in the embodiment of the present application, which will be combined with Figure 4 The steps shown are explained.

[0048] Step S401: receiving an audio file generation request, where the audio file generation request includes text and a target object.

[0049] Here, the server may receive an audio file generation request sent by the user through the terminal, where the audio file generation request is used to request generation of an audio file, and the audio file finally generated contains the text and has the sound characteristics of the target object.

[0050] In the embodiment of the present application, the generated audio file may be a song, the text is the lyrics in the song, the target object may be a singer, and the audio file that is ultimately desired to be generated may be an audio file having given lyrics and having the voice characteristics of a given singer. Alternatively, the generated audio file may be a poetry recitation, the text is the poem to be recited, the target object may be any reader, and the audio file that is ultimately desired to be generated may be an audio file of the reader reciting the poem. Alternatively, the generated audio file may be a video soundtrack, the text is the text information corresponding to the video content, the target object may be a character in the video, and the audio file that is ultimately desired to be generated may be an audio file having the text information and having the voice characteristics of the character in the video.

[0051] Step S402 , extracting features from the text and the target object respectively, and obtaining a first feature vector of the text and a second feature vector of the target object accordingly.

[0052] Here, feature extraction can be performed on the text and the target object respectively. When feature extraction is performed on the text, the feature vector corresponding to each word in the text can be extracted, so as to obtain a first feature vector for representing the text information based on the feature vector of each word; when feature extraction is performed on the target object, at least one sound feature related to sound of the target object can be extracted, and a second feature vector can be obtained based on the extracted sound features.

[0053] Step S403: concatenate the first feature vector and the second feature vector to form a concatenated feature vector.

[0054] Here, after obtaining the first feature vector and the second feature vector, the first feature vector and the second feature vector are connected to form a concatenated feature vector with a higher dimension. The concatenated feature vector is a feature vector that takes into account the information of the text input by the user, and also takes into account the feature vector related to the sound features of the target object input by the user. In this way, when the concatenated feature vector is used for subsequent processing, subsequent processing is performed based on both the information of the text and the sound features of the target object.

[0055] Step S404: predicting audio attribute features of the concatenated feature vector to obtain at least two audio attribute features of different types.

[0056] In an embodiment of the present application, a decoder can be used to predict the audio attribute features of the spliced ​​feature vector, and the decoder is used to decode the spliced ​​feature vector obtained after feature extraction and feature splicing in different types of audio attribute feature dimensions to predict an audio attribute feature of this type. In an embodiment of the present application, multiple types of audio attribute features can be predicted. In some embodiments, at least two decoders can be used to predict the audio attribute features of the spliced ​​feature vector respectively, and correspondingly at least two audio attribute features of different types are obtained, wherein each decoder can predict one type of audio attribute feature, so multiple decoders can be used for prediction to obtain multiple types of audio attribute features.

[0057] In the embodiment of the present application, the audio attribute characteristics include, but are not limited to: timbre characteristics, pitch characteristics, energy characteristics, rhythm characteristics, audio characteristics, and any other attribute characteristics related to sound.

[0058] Step S405: Generate an audio file containing text and having the sound characteristics of the target object according to at least two audio attribute features.

[0059] Here, after the audio attribute features are determined, an audio file is generated based on these audio attribute features, wherein the audio file includes text and has the sound characteristics of the target object. For example, when the audio file is a song, the lyrics of the song are the text, and the person singing the song is the target object, that is, the singing audio of the song has the determined audio attribute features.

[0060] The audio file generation method provided in the embodiment of the present application performs feature extraction on the text and the target object respectively to obtain a first feature vector of the text and a second feature vector of the target object; and concatenates the first feature vector and the second feature vector to form a concatenated feature vector; uses at least two decoders to predict the concatenated feature vector to obtain at least two audio attribute features of different types, and generates an audio file according to the different types of audio attribute features. In this way, the audio file can be generated by simply inputting the text and the target object, thereby improving the actual application scope of the audio file generation method and improving the generalization of the method.

[0061] The method of the embodiment of the present application can be applied to an audio file generation system formed by at least a terminal and a server. An audio file generation application runs on the terminal, so the user can request to generate an audio file that meets the user's needs on the terminal. Figure 5 is an optional flow chart of the audio file generation method provided in the embodiment of the present application, such as Figure 5 As shown, the method comprises the following steps:

[0062] Step S501: The terminal receives a user operation to obtain text and a target object.

[0063] In the embodiment of the present application, the user operation may be an input operation and a selection operation.

[0064] In the first implementation, the user inputs text through an input operation, and the user selects the name of his or her favorite singer through a selection operation, and the selected singer is used as the target object. In the second implementation, the user can input text and the name of the singer through an input operation. In the third implementation, the user can input text through an input operation, and the terminal automatically determines and selects the singer, that is, the terminal determines the user's favorite singer as the target object based on the user's historical listening habits. In the fourth implementation, the user can input text through an input operation, and the terminal automatically matches multiple singers that the user likes, and displays the names of multiple singers on the current interface to provide the user with a choice. In the fifth implementation, the user can input text through an input operation, and the terminal matches multiple singers based on the text input by the user, and displays the names of multiple singers on the current interface to provide the user with a choice.

[0065] Step S502: The terminal sends an audio file generation request to the server, where the audio file generation request includes text and a target object.

[0066] Step S503: the server extracts features of the text and the target object respectively, and obtains a first feature vector of the text and a second feature vector of the target object accordingly.

[0067] Step S504: the server concatenates the first feature vector and the second feature vector to form a concatenated feature vector.

[0068] Step S505: The server uses at least two decoders to predict audio attribute features of the concatenated feature vectors respectively, and correspondingly obtains at least two audio attribute features of different types; wherein each decoder is used to predict one type of audio attribute feature.

[0069] Step S506: The server generates an audio file containing text and having the sound characteristics of the target object according to at least two audio attribute features.

[0070] It should be noted that the implementation process of step S502 and step S506 is the same as that of the above-mentioned steps S401 to S405, so they are not repeated here.

[0071] Step S507: the server sends the generated audio file to the terminal.

[0072] Step S508: the terminal plays the audio file.

[0073] The audio file generation method provided in the embodiment of the present application, through the interaction between the terminal and the server, enables the user to input text and information of a favorite singer on the terminal, and then generate an audio file whose lyrics are the text and has the voice characteristics of the singer, without the need for the user to arrange the music himself, thereby realizing intelligent arrangement and improving the user experience. In addition, since the audio file can be generated by only inputting text and a target object without the user providing more information, the user operation is simplified, so that general users (i.e. users who do not have the ability to create music) can also create new music through the terminal, thereby increasing the actual use scope of the audio file generation method and improving the generalization of the method.

[0074] based on Figure 4 , Figure 6 is an optional flow chart of the audio file generation method provided in the embodiment of the present application, such as Figure 6 As shown, step S402 can be implemented by the following steps:

[0075] Step S601, performing phoneme conversion processing and position information embedding processing on the text in sequence to obtain a first feature vector of the text.

[0076] Here, the phoneme conversion process refers to converting each word in the text into at least one phoneme represented by pinyin; the position information embedding process refers to embedding the position information of each word in the text into the phoneme sequence after the phoneme conversion process, and finally obtaining the first feature vector of the text. In this way, the first feature vector takes into account both the phoneme of each word in the text and the position of each word in the text, that is, the contextual semantics of each word in the text.

[0077] Step S602: perform feature embedding processing on the target object to obtain a second feature vector for characterizing attribute information of the target object.

[0078] Here, after determining the target object, the sound information of the target object can be obtained in the speech library, and the sound features in the sound information of the target object can be extracted, and the sound features of the target object can be feature embedded to use a second feature vector to represent the sound information of the target object.

[0079] based on Figure 6 , Figure 7 is an optional flow chart of the audio file generation method provided in the embodiment of the present application, such as Figure 7 As shown, step S601 can be implemented by the following steps:

[0080] Step S701, performing phoneme conversion processing on each word in the text in turn to obtain a phoneme sequence corresponding to the text.

[0081] For example, for the text "重新再来" (Repeat), performing phoneme conversion processing on it can convert the text into a phoneme sequence "chong2 xin1 zai4 lai2", where the numbers in the phoneme sequence represent the tones of each character.

[0082] Step S702: Obtain the position information of each phoneme in the phoneme sequence to form a position sequence.

[0083] The position information of each phoneme refers to the position of each phoneme in the phoneme sequence. In the embodiments of the present application, multiple phonemes corresponding to the same character may have the same position information. That is to say, the position information may be the position information in the character dimension. Taking the above phoneme sequence "chong2 xin1 zai4 lai2" as an example, the position information of the phonemes "c", "h", "o", "n", "g", "2" corresponding to the character "重" may all be "1", and the position information of the phonemes "x", "i", "n", "1" corresponding to the character "新" may all be "2". Thus, corresponding to the phoneme sequence "chong2 xin1 zai4 lai2", the position sequence is "111111222233334444".

[0084] Step S703: Use the position sequence to perform position information embedding processing on each phoneme in the phoneme sequence to obtain a first feature vector corresponding to the phoneme sequence and used to represent the content of the text.

[0085] In some embodiments, step S703 can be implemented through the following steps:

[0086] Step S7031: Look up the phoneme vector corresponding to each phoneme in the phoneme sequence and the position vector corresponding to each position information in the position sequence in a preset embedding dictionary.

[0087] Here, the preset embedding dictionary stores the phoneme vectors of each phoneme and the position vectors corresponding to each position information.

[0088] Step S7032: Sum the elements in the phoneme vector of each phoneme and the corresponding elements in the position vector of the position information corresponding to the phoneme to obtain an embedding vector after embedding the position information for each phoneme.

[0089] Here, the phoneme vector of each phoneme has the same dimension as the position vector of the position information corresponding to the phoneme. The elements at each element position in the phoneme vector and the position vector can be added correspondingly, so as to obtain an embedding vector after summing the phoneme vector and the position vector. This embedding vector is the embedding vector after embedding the position information for the phoneme.

[0090] Step S7033: determine a first feature vector according to the embedding vectors of all phonemes in the phoneme sequence.

[0091] Here, the embedding vectors of all phonemes can be concatenated to obtain the first feature vector.

[0092] In some embodiments, step S7033 may be implemented by the following steps:

[0093] Step S7033a, copy the embedding vector at least once to obtain multiple identical embedding vectors.

[0094] In an embodiment of the present application, the embedding vector may be copied to obtain three identical embedding vectors, and the copied three identical embedding vectors may be input into a transformer model for calculation.

[0095] Step S7033b, performing multi-dimensional attention calculation on multiple identical embedding vectors to obtain multiple attention vectors.

[0096] In an embodiment of the present application, a transformer model may be used to implement the process of determining the first eigenvector, that is, multi-dimensional attention calculation is performed on multiple identical embedding vectors through a multi-head attention module (Multi-headAttention) in the transformer structure.

[0097] Step S7033c, summing and normalizing multiple attention vectors in sequence to obtain a first eigenvector.

[0098] Here, during the summing and normalizing processes, the embedding vector and the multiple attention vectors may be summed and normalized together. That is, the inputs of the summing and normalizing processes include not only the multiple attention vectors but also the embedding vector.

[0099] In the embodiment of the present application, the purpose of normalization is that since vector multiplication and other calculations are performed in the multi-dimensional attention calculation in the previous step, the values ​​of the elements in the multiple attention vectors finally output are relatively large, so in order to reduce the difficulty of calculation and improve the calculation efficiency, a step of normalization can be performed. The normalization here can be to divide each element in the attention vector by the same number at the same time, for example, this same number can be 100 or 1000, so as to realize the normalization of the attention vector.

[0100] In some embodiments, the at least two decoders include at least a pitch decoder, an energy decoder, a rhythm decoder and a spectrum decoder; correspondingly, the audio attribute features include at least a pitch feature, an energy feature, a rhythm feature and a spectrum feature; based on Figure 4 , Figure 8 is an optional flow chart of the audio file generation method provided in the embodiment of the present application, such as Figure 8 As shown, step S404 can be implemented by at least the following steps:

[0101] Step S801, using a pitch decoder to predict the pitch feature of the concatenated feature vector to obtain a pitch feature. Step S802, using an energy decoder to predict the energy feature of the concatenated feature vector to obtain an energy feature. Step S803, using a rhythm decoder to predict the rhythm feature of the concatenated feature vector to obtain a rhythm feature. Step S804, using a spectrum decoder to predict the spectrum feature of the concatenated feature vector to obtain a spectrum feature.

[0102] Correspondingly, step S405 can be implemented by the following steps:

[0103] Step S805: Generate an audio file containing the text and having the sound characteristics of the target object according to the pitch features, energy features, rhythm features and spectrum features.

[0104] In the embodiment of the present application, the pitch decoder, energy decoder, rhythm decoder and spectrum decoder have the same network structure; and the parameters inside the network structure of the pitch decoder, energy decoder, rhythm decoder and spectrum decoder are different.

[0105] In some embodiments, the above text includes at least two paragraphs, and the acquisition of audio clips in sequence for the multiple paragraphs included in the text is described below.

[0106] Fig. 9 is an optional flow chart of the audio file generation method provided in the embodiment of the present application, such as Fig. 9 As shown, the method comprises the following steps:

[0107] Step S901: receiving an audio file generation request, where the audio file generation request includes text and a target object.

[0108] Step S902: Use a speech synthesis model to extract features from the first paragraph in the text to obtain a first paragraph feature vector of the first paragraph. Here, the speech synthesis model is a pre-trained model for generating audio files.

[0109] Step S903: Use the speech synthesis model to extract features of the target object to obtain a second feature vector of the target object.

[0110] Step S904: concatenate the first paragraph feature vector and the second feature vector to form a paragraph concatenation vector.

[0111] Step S905: using at least two decoders in the speech synthesis model, respectively predict the audio attribute features of the paragraph concatenation vector, and correspondingly obtain at least two paragraph audio attribute features with different types.

[0112] Step S906: generating a first audio segment including a first paragraph and having the sound characteristics of the target object according to at least two of the paragraph audio attribute features.

[0113] Step S907, determining whether the currently processed paragraph is the last paragraph in the text.

[0114] If the judgment result is yes, execute step S910; if the judgment result is no, execute step S908.

[0115] In the embodiment of the present application, each paragraph is input into the speech synthesis model in turn for prediction, and a first audio segment corresponding to each paragraph is obtained.

[0116] Step S908: after acquiring the first audio segment, determining the pitch feature, energy feature, rhythm feature and spectrum feature corresponding to the first audio segment.

[0117] Step S909, inputting the second paragraph following the first paragraph in the text and the pitch features, energy features, rhythm features and spectral features corresponding to the first audio segment into the speech synthesis model, so as to generate a second audio segment containing the second paragraph and having the sound characteristics of the target object through the speech synthesis model.

[0118] In an embodiment of the present application, after predicting the first paragraph, when predicting the next paragraph, not only the text information of the next paragraph and the information corresponding to the target object can be used as the basis for predicting the next paragraph, but also the pitch features, energy features, rhythm features, spectrum features and other features related to sound features of the first audio segment obtained by predicting the first paragraph can be used as the basis for predicting the next paragraph, that is, the second paragraph after the first paragraph in the text and the pitch features, energy features, rhythm features and spectrum features corresponding to the first audio segment are all input into the speech synthesis model for prediction.

[0119] After the second audio segment is obtained, the process returns to step S907 and continues to execute, and the process repeats in this way, inputting each paragraph in the text into the speech synthesis model in turn, so as to obtain the audio segment corresponding to each paragraph.

[0120] Step S910, according to the sequence of each paragraph in the text, the audio clips corresponding to each paragraph are spliced ​​to form an audio file.

[0121] Here, after obtaining the audio segments corresponding to each paragraph, the audio segments are spliced ​​together to obtain an audio file corresponding to the entire text.

[0122] The audio file generation method provided in the embodiment of the present application divides the text into multiple paragraphs, and then predicts each paragraph. After predicting the first paragraph, the prediction result of the first paragraph (i.e., the first audio segment) is combined to predict the audio segments corresponding to each subsequent paragraph. This can further improve the accuracy of the predicted audio segments and obtain an audio file with a more natural connection between the front and the back.

[0123] In the embodiment of the present application, the process of forming an audio file according to the input text and target object can be implemented by the above-mentioned speech synthesis model, that is, the text and target object can be input into the speech synthesis model, and processed by the speech synthesis model to obtain the final audio text. The embodiment of the present application provides a training method for a speech synthesis model, Fig.10 is an optional flow chart of the training method of the speech synthesis model provided in the embodiment of the present application, such as Fig.10 As shown, the training method includes the following steps:

[0124] Step S101: input sample text and sample objects into a speech synthesis model.

[0125] Step S102, through the encoder in the speech synthesis model, feature extraction is performed on the sample text and the sample object respectively, and a first sample feature vector of the sample text and a second sample feature vector of the sample object are correspondingly obtained, and the first sample feature vector and the second sample feature vector are concatenated to obtain a sample concatenation vector.

[0126] In the embodiment of the present application, the encoder is a feature extraction network in the speech synthesis model, that is, the encoder can be used to extract features from the input sample data.

[0127] Step S103, using at least two decoders in the speech synthesis model, respectively predicting the audio attribute features of the sample concatenated vectors, and correspondingly obtaining at least two sample audio attribute features of different types.

[0128] In an embodiment of the present application, a speech synthesis model may have multiple decoders, each decoder being used to predict a type of audio attribute feature.

[0129] Step S104: input each sample audio attribute feature into a preset loss model to obtain a loss result corresponding to each sample audio attribute feature.

[0130] Here, the preset loss model is used to compare the sample audio attribute features with the preset audio attribute features to obtain a loss result, wherein the preset audio attribute features may be audio attribute features pre-set by the user corresponding to the sample text and the sample object.

[0131] In the embodiment of the present application, the preset loss model includes a loss function, through which the similarity between the sample audio attribute feature and the preset audio attribute feature can be calculated, and the above loss result can be determined based on the similarity. When the similarity between the sample audio attribute feature and the preset audio attribute feature is smaller, it indicates that the gap between the training result of the model and the true value is large, and further training is required; when the similarity between the sample audio attribute feature and the preset audio attribute feature is greater, it indicates that the training result of the model is closer to the true value.

[0132] Step S105, according to the loss result corresponding to each sample audio attribute feature, the corresponding parameters in each decoder are modified until the sample audio attribute feature output by each decoder meets the preset conditions and the training is stopped to obtain a trained speech synthesis model.

[0133] Here, when the above similarity is less than the preset threshold, the loss result indicates that a decoder corresponding to the current loss result cannot accurately predict the audio attribute features of the sample splicing vector. For example, when the decoder is a pitch decoder, when the similarity between the sample pitch attribute features and the preset pitch attribute features is small, it indicates that the pitch decoder cannot accurately predict the pitch features of the sample splicing vector. Therefore, it is necessary to correct the parameters in the decoder until the sample pitch attribute features output by the pitch decoder meet the preset conditions (that is, the similarity between the output sample pitch attribute features and the preset pitch attribute features is greater than the threshold), and then stop training the sample pitch encoder.

[0134] It should be noted that the embodiment of the present application can use the same method to train each decoder separately until each decoder in the speech synthesis model is trained to meet preset conditions and then stop training, thereby obtaining a trained speech synthesis model.

[0135] The training method of the speech synthesis model provided in the embodiment of the present application, because the sample text and sample object are input into the speech synthesis model, feature extraction and audio attribute feature prediction are performed in sequence through an encoder and at least two decoders, at least two sample audio attribute features with different types are obtained, and each sample audio attribute feature is input into a preset loss model to obtain a loss result corresponding to each sample audio attribute feature. Therefore, the parameters in each decoder can be corrected according to the loss result, and the obtained trained speech synthesis model can accurately predict the audio attribute features, thereby obtaining an audio file that better meets the user's needs and improves the user's experience.

[0136] The following is an explanation of an exemplary application of an embodiment of the present application in a practical application scenario.

[0137] The embodiment of the present application provides an audio file generation method, which can generate a piece of music based on a given arbitrary text, and can be used as a platform, such as Fig.11 As shown, it is an interface diagram of the audio file generation product provided in the embodiment of the present application. The user can enter a text in the input box 1101 and click the submit button 1102, and the system will automatically generate an audio file 1103. Fig.11 The link in is only an example. Of course, in other embodiments, it can also be applied as a plug-in to multiple scenarios of other products, such as text soundtrack, video soundtrack and game music generation.

[0138] Compared with the related art which uses information such as pitch, energy, and rhythm as input features, the technical solution of the embodiment of the present application is to use these features as predicted output targets. Fig.12 FIG. 1 is a schematic diagram of the overall process of generating an audio file in an embodiment of the present application. Fig.12 As shown, the input information includes text 1201 and singer 1202. The input module only needs text and singer can specify the voice of a certain singer. Compared with the existing solution, no additional information such as pitch is required here.

[0139] The encoder module 1203 performs serialization modeling on the input text and singer.

[0140] The decoder module has multiple sub-encoder modules 1204 to predict multiple targets (e.g., pitch, energy, rhythm, and spectrogram, etc.) according to the output of the encoder module 1203. The multiple sub-decoder modules 1204 share the same network structure, but each has independent parameters. The targets of the multiple sub-decoder modules 1204 are to calculate the loss results (Loss) respectively, and then update the parameters inside the module.

[0141] The output obtained after the sub-decoder module 1204 includes pitch 1205, energy 1206, rhythm 1207, and spectrogram 1208 (which can be, for example, a mel-spectrogram), where pitch 1205, energy 1206, and rhythm 1207 are the results ensuring training. In the prediction stage, only the mel-spectrogram is needed to participate in subsequent predictions.

[0142] The vocoder module 1209 generates the original acoustic wave / audio data 1210 based on the obtained mel-spectrogram result. Since this step is relatively common, the waveglow model can be directly adopted here and obtained during Chinese data training.

[0143] In some embodiments, for the above-mentioned encoder module 1203, the encoder module 1203 performs sequential modeling on the input text and singer. Fig.13A It is the network structure diagram of the encoder module provided by the embodiments of the present application, as Fig.13A shown. The network operations of the encoder module include:

[0144] First, the pypinyin tool can be used to convert Chinese text into pinyin, which can better model the text at the pronunciation (phoneme-level) level. For example, for the input text "重新再来" (chong2 xin1 zai4 lai2), the numbers in the sequence represent tones. After obtaining the phoneme vocabulary, a phoneme embedding vector is constructed as the parameter representation of the text. In addition, considering the position information of each phoneme character (which can be a char-type character), each phoneme character has a position embedding vector to represent the position information of the phoneme character.

[0145] In the embodiments of the present application, for the converted pinyin sequence 1301 and position sequence 1302, look-up operations are performed through their respective embedding dictionaries, and addition processing is performed according to the position 1303. Then, the transformer structure 1304 is used for sequential modeling to obtain the hidden layer representation T1 of the text (i.e., the first feature vector).

[0146] Then, for the input singer information, a global singer embedding vector can be maintained as the parameter representation. After the look-up operation, the hidden layer representation S1 of the singer (i.e., the second feature vector) is obtained.

[0147] Then, the hidden layer representation T1 of the text and the hidden layer representation S1 of the singer are concatenated to obtain the output H of the final encoder module 1203 (i.e., the concatenated feature vector).

[0148] Fig. 13Bis a schematic diagram of the transformer structure provided in the embodiment of the present application, such as Fig. 13B As shown, the transformer structure includes a multi-head attention module 1321 (Multi-headAttention), a summation and normalization module 1322 (Add&Norm)

[0149] In some embodiments, for the above decoder module 1204, the decoder module has multiple sub-decoder modules 1204 to predict multiple targets (e.g., pitch, energy, rhythm, and spectrogram, etc.) according to the output H of the encoder module 1203. Fig.14 is a network structure diagram of the overall sub-decoder module provided in the embodiment of the present application, such as Fig.14 As shown, the network structure of the sub-decoder module 1204 includes: an input module 1401, wherein the information input by the input module includes the output H of the encoder module 1203 and the current generated part (including pitch, energy and rhythm), a transformer structure 1402 (the same as the transformer structure 1304 in the encoder module 1203) and an output module 1403.

[0150] Next, the target of each sub-decoder module 1204 is described:

[0151] For the pitch sub-decoder module, the pitch represents the frequency of a certain period of time. Here, the pitch can be mapped to [-256, 255], and the target pitch is calculated based on the target waveform.

[0152] For the energy sub-decoder module, energy represents the strength of the audio signal over a period of time. Generally, the louder the sound, the higher the energy. It is represented as the amplitude in the waveform. Here, the energy is also mapped to [-256, 255]. The target energy is calculated based on the target waveform.

[0153] For the rhythm sub-decoder module, the rhythm represents the audio duration corresponding to a phoneme token. The target duration here can be obtained through the automatic speech recognition (ASR, Automatic Speech Recognition) alignment model or manual annotation.

[0154] For the spectrum sub-decoder module, the spectrum represents the data of the audio in the frequency domain space for a certain period of time. Here, the python librosa tool can be used to calculate the mel-spectrogram as the spectrum target.

[0155] In the embodiment of the present application, the purpose of multiple sub-decoder modules is to better learn the representation of the encoder module and the decoder module through multi-objective training with a small number of input features, and then generate better spectrograms, which can be converted into better music.

[0156] Compared with the solutions in the related art, the method of the embodiment of the present application uses input features such as pitch as output targets for prediction, which has the benefit of solving the generalization problem in the prediction stage. The original technical solution requires these features to be explicitly provided, but in the absence of generated audio, it is difficult to directly provide features such as pitch. The solution of the embodiment of the present application only uses the original text and the information of a specified singer to generate the corresponding music.

[0157] It should be noted that the modules that can be replaced in the embodiments of the present application mainly include the following aspects: the model structure and parameter size of the encoder module, decoder module and vocoder module can be adjusted according to actual use requirements. In the decoding stage of the decoder module, more targets other than the above-listed ones such as pitch, energy, rhythm and spectrum can also be added to participate in the training. The vocoder module can also be removed, that is, the target of the waveform format can be directly predicted in the decoding stage of the decoder module.

[0158] The following is a description of an exemplary structure of the audio file generating device 354 provided in the embodiment of the present application implemented as a software module. In some embodiments, for example, Figure 3 As shown, the software module stored in the audio file generating device 354 of the memory 350 may be the audio file generating device in the server 300, including:

[0159] A receiving module 3541 is used to receive an audio file generation request, wherein the audio file generation request includes text and a target object; a feature extraction module 3542 is used to perform feature extraction on the text and the target object respectively, and obtain a first feature vector of the text and a second feature vector of the target object accordingly; a splicing module 3543 is used to splice the first feature vector and the second feature vector to form a spliced ​​feature vector; a prediction module 3544 is used to predict audio attribute features of the spliced ​​feature vector to obtain at least two audio attribute features of different types; a generation module 3545 is used to generate an audio file containing the text and having the sound characteristics of the target object based on at least two of the audio attribute features.

[0160] In some embodiments, the prediction module is further used to use at least two decoders to predict audio attribute features of the spliced ​​feature vector respectively, and correspondingly obtain at least two audio attribute features of different types; wherein each decoder is used to predict one type of audio attribute feature.

[0161] In some embodiments, the feature extraction module is also used to: perform phoneme conversion and position information embedding processing on the text in sequence to obtain a first feature vector of the text; and perform feature embedding processing on the target object to obtain the second feature vector used to characterize the attribute information of the target object.

[0162] In some embodiments, the feature extraction module is also used to: perform the phoneme conversion process on each word in the text in turn to obtain a phoneme sequence corresponding to the text; obtain the position information of each phoneme in the phoneme sequence to form a position sequence; use the position sequence to perform the position information embedding process on each phoneme in the phoneme sequence to obtain the first feature vector corresponding to the phoneme sequence and used to characterize the content of the text.

[0163] In some embodiments, the feature extraction module is also used to: search for the phoneme vector corresponding to each phoneme in the phoneme sequence and the position vector corresponding to each position information in the position sequence in a preset embedding dictionary; sum the elements in the phoneme vector of each phoneme and the corresponding elements in the position vector of the position information corresponding to the phoneme to obtain an embedding vector after each phoneme is embedded in the position information; determine the first feature vector based on the embedding vectors of all phonemes in the phoneme sequence.

[0164] In some embodiments, the feature extraction module is also used to: copy the embedding vector at least once to obtain multiple identical embedding vectors; perform multi-dimensional attention calculation on the multiple identical embedding vectors to obtain multiple attention vectors; and sum and normalize the multiple attention vectors in turn to obtain the first feature vector.

[0165] In some embodiments, the at least two decoders include at least a pitch decoder, an energy decoder, a rhythm decoder and a spectrum decoder; correspondingly, the audio attribute features include at least pitch features, energy features, rhythm features and spectrum features; the prediction module is also used to: use the pitch decoder to predict the pitch features of the spliced ​​feature vector to obtain pitch features; use the energy decoder to predict the energy features of the spliced ​​feature vector to obtain energy features; use the rhythm decoder to predict the rhythm features of the spliced ​​feature vector to obtain rhythm features; use the spectrum decoder to predict the spectrum features of the spliced ​​feature vector to obtain spectrum features.

[0166] In some embodiments, the pitch decoder, the energy decoder, the rhythm decoder and the spectrum decoder have the same network structure; and the parameters within the network structures of the pitch decoder, the energy decoder, the rhythm decoder and the spectrum decoder are different.

[0167] In some embodiments, the text includes at least two paragraphs; the device also includes: a paragraph input module, which is used to input each paragraph in the text into a pre-trained speech synthesis model in turn to obtain an audio segment corresponding to each paragraph; an audio segment splicing module, which is used to splice the audio segments corresponding to each paragraph according to the order of each paragraph in the text to form the audio file.

[0168] In some embodiments, the paragraph input module is also used to: use the speech synthesis model to extract features from the first paragraph in the text to obtain a first paragraph feature vector of the first paragraph; use the speech synthesis model to predict the audio attribute features of the first paragraph feature vector and the second feature vector, and obtain at least two paragraph audio attribute features of different types; and generate a first audio segment containing the first paragraph and having the sound characteristics of the target object based on the at least two paragraph audio attribute features; after obtaining the first audio segment, determine the pitch features, energy features, rhythm features and spectrum features corresponding to the first audio segment; input the second paragraph after the first paragraph in the text and the pitch features, energy features, rhythm features and spectrum features corresponding to the first audio segment into the speech synthesis model, so as to generate a second audio segment containing the second paragraph and having the sound characteristics of the target object through the speech synthesis model; and so on, input each paragraph in the text into the speech synthesis model in a loop to obtain the audio segment corresponding to each paragraph.

[0169] In some embodiments, the speech synthesis model is trained by the following steps: inputting sample text and sample objects into the speech synthesis model; performing feature extraction on the sample text and the sample object respectively through the encoder in the speech synthesis model, and obtaining a first sample feature vector of the sample text and a second sample feature vector of the sample object, and concatenating the first sample feature vector and the second sample feature vector to obtain a sample concatenation vector; performing audio attribute feature prediction on the sample concatenation vector respectively through at least two decoders in the speech synthesis model, and obtaining at least two sample audio attribute features of different types; inputting each of the sample audio attribute features into a preset loss model, and obtaining a loss result corresponding to each of the sample audio attribute features; and modifying the parameters in each corresponding decoder according to the loss result corresponding to each of the sample audio attribute features, and stopping the training until the sample audio attribute features output by each decoder meet the preset conditions to obtain the trained speech synthesis model.

[0170] It should be noted that the description of the device of the embodiment of the present application is similar to the description of the above method embodiment, and has similar beneficial effects as the method embodiment, so it is not repeated. For technical details not disclosed in the embodiment of the device, please refer to the description of the method embodiment of the present application for understanding.

[0171] The embodiment of the present application provides a computer program product or a computer program, which includes computer instructions stored in a computer-readable storage medium. The processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device executes the above-mentioned method of the embodiment of the present application.

[0172] The embodiment of the present application provides a storage medium storing executable instructions, wherein the executable instructions are stored. When the executable instructions are executed by a processor, the processor will execute the method provided by the embodiment of the present application, for example, Figure 4 The method shown.

[0173] In some embodiments, the storage medium can be a computer-readable storage medium, for example, a ferroelectric random access memory (FRAM), a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), a flash memory, a magnetic surface memory, an optical disk, or a compact disk-read-only memory (CD-ROM); it can also be various devices including one or any combination of the above memories.

[0174] In some embodiments, executable instructions may be in the form of a program, software, software module, script or code, written in any form of programming language (including compiled or interpreted languages, or declarative or procedural languages), and may be deployed in any form, including as a stand-alone program or as a module, component, subroutine or other unit suitable for use in a computing environment.

[0175] As an example, executable instructions may, but need not necessarily, correspond to a file in a file system, may be stored as part of a file storing other programs or data, such as one or more scripts in a HyperText Markup Language (HTML) document, in a single file dedicated to the program in question, or in multiple coordinated files (e.g., files storing one or more modules, subroutines, or code portions). As an example, executable instructions may be deployed to be executed on one computing device, or on multiple computing devices located at one location, or on multiple computing devices distributed at multiple locations and interconnected by a communication network.

[0176] The above is only an embodiment of the present application and is not intended to limit the protection scope of the present application. Any modifications, equivalent substitutions and improvements made within the spirit and scope of the present application are included in the protection scope of the present application.

Claims

1. A method for generating an audio file, characterized in that: include: receiving an audio file generation request, wherein the audio file generation request includes text and a target object; Performing phoneme conversion processing on each word in the text in turn to obtain a phoneme sequence corresponding to the text; Acquire position information of each phoneme in the phoneme sequence to form a position sequence; Searching a preset embedding dictionary for a phoneme vector corresponding to each phoneme in the phoneme sequence and a position vector corresponding to each position information in the position sequence; Summing the elements in the phoneme vector of each phoneme and the corresponding elements in the position vector of the position information corresponding to the phoneme to obtain an embedding vector after each phoneme is embedded in the position information; Determining a first feature vector of the text according to the embedding vectors of all phonemes in the phoneme sequence; Performing feature embedding processing on the target object to obtain a second feature vector for characterizing attribute information of the target object; concatenating the first feature vector and the second feature vector to form a concatenated feature vector; Predicting audio attribute features of the concatenated feature vector to obtain at least two audio attribute features of different types; An audio file containing the text and having the sound characteristics of the target object is generated according to at least two of the audio attribute features.

2. The method according to claim 1, characterized in that The predicting of the audio attribute feature of the concatenated feature vector to obtain at least two audio attribute features of different types includes: At least two decoders are used to predict the audio attribute features of the concatenated feature vector respectively, and correspondingly at least two audio attribute features of different types are obtained; wherein each decoder is used to predict one type of audio attribute feature.

3. The method according to claim 1, characterized in that The step of determining a first feature vector of the text according to the embedding vectors of all phonemes in the phoneme sequence comprises: Copying the embedding vector at least once to obtain multiple identical embedding vectors; Performing multi-dimensional attention calculation on the multiple identical embedding vectors to obtain multiple attention vectors; The multiple attention vectors are summed and normalized in sequence to obtain the first feature vector.

4. The method according to claim 2, characterized in that: The at least two decoders include at least a pitch decoder, an energy decoder, a rhythm decoder and a spectrum decoder; correspondingly, the audio attribute features include at least a pitch feature, an energy feature, a rhythm feature and a spectrum feature; The at least two decoders are used to predict the audio attribute features of the concatenated feature vector respectively, and at least two audio attribute features with different types are correspondingly obtained, including: Using the pitch decoder to predict the pitch feature of the concatenated feature vector to obtain the pitch feature; Using the energy decoder to predict the energy feature of the concatenated feature vector to obtain the energy feature; Using the rhythm decoder to predict the rhythm feature of the concatenated feature vector to obtain the rhythm feature; The spectrum decoder is used to predict the spectrum feature of the concatenated feature vector to obtain the spectrum feature.

5. The method according to claim 4, characterized in that The pitch decoder, the energy decoder, the rhythm decoder and the spectrum decoder have the same network structure; and The parameters inside the network structures of the pitch decoder, the energy decoder, the rhythm decoder and the spectrum decoder are different.

6. The method according to any one of claims 1 to 5, characterized in that: The text includes at least two paragraphs; the method further includes: Inputting each paragraph of the text into a pre-trained speech synthesis model in turn to obtain an audio segment corresponding to each paragraph; According to the sequence of each paragraph in the text, the audio clips corresponding to each paragraph are spliced ​​to form the audio file.

7. The method according to claim 6, characterized in that The step of sequentially inputting each paragraph in the text into the pre-trained speech synthesis model to obtain an audio segment corresponding to each paragraph includes: Using the speech synthesis model, extracting features from a first paragraph in the text to obtain a first paragraph feature vector of the first paragraph; The speech synthesis model is used to predict the audio attribute features of the first paragraph feature vector and the second feature vector, and correspondingly obtain at least two paragraph audio attribute features of different types; and generating a first audio segment including the first paragraph and having the sound characteristics of the target object according to at least two of the paragraph audio attribute features; After acquiring the first audio segment, determining a pitch feature, an energy feature, a rhythm feature, and a spectrum feature corresponding to the first audio segment; Inputting the pitch features, energy features, rhythm features, and spectrum features corresponding to a second paragraph following the first paragraph in the text and the first audio segment into the speech synthesis model, so as to generate a second audio segment containing the second paragraph and having the sound characteristics of the target object through the speech synthesis model; In this way, each paragraph in the text is cyclically input into the speech synthesis model in turn to obtain the audio segment corresponding to each paragraph.

8. The method according to claim 6, characterized in that The speech synthesis model is trained by the following steps: Inputting sample text and sample objects into the speech synthesis model; By using the encoder in the speech synthesis model, feature extraction is performed on the sample text and the sample object respectively, a first sample feature vector of the sample text and a second sample feature vector of the sample object are correspondingly obtained, and the first sample feature vector and the second sample feature vector are spliced ​​to obtain a sample splicing vector; Predicting audio attribute features of the sample concatenated vectors respectively through at least two decoders in the speech synthesis model, and correspondingly obtaining at least two sample audio attribute features of different types; Input each of the sample audio attribute features into a preset loss model to obtain a loss result corresponding to each of the sample audio attribute features; According to the loss result corresponding to each of the sample audio attribute features, the parameters in each corresponding decoder are modified until the training is stopped when the sample audio attribute features output by each decoder meet the preset conditions to obtain a trained speech synthesis model.

9. An audio file generating device, characterized in that: include: A receiving module, used for receiving an audio file generation request, wherein the audio file generation request includes text and a target object; A feature extraction module is used to perform phoneme conversion processing on each word in the text in turn to obtain a phoneme sequence corresponding to the text; obtain the position information of each phoneme in the phoneme sequence to form a position sequence; search a preset embedding dictionary for a phoneme vector corresponding to each phoneme in the phoneme sequence and a position vector corresponding to each position information in the position sequence; Summing the elements in the phoneme vector of each phoneme and the corresponding elements in the position vector of the position information corresponding to the phoneme to obtain an embedding vector after each phoneme is embedded in the position information; determining a first feature vector of the text according to the embedding vectors of all phonemes in the phoneme sequence; Performing feature embedding processing on the target object to obtain a second feature vector for characterizing attribute information of the target object; A concatenation module, used for concatenating the first feature vector and the second feature vector to form a concatenated feature vector; A prediction module, used for predicting audio attribute features of the concatenated feature vector to obtain at least two audio attribute features of different types; A generating module is used to generate an audio file containing the text and having the sound characteristics of the target object according to at least two of the audio attribute features.

10. An audio file generating device, characterized in that: include: A memory for storing executable instructions; The processor is used to implement the audio file generation method according to any one of claims 1 to 8 when executing the executable instructions stored in the memory.

11. A computer-readable storage medium, characterized in that: Executable instructions are stored, which are used to cause a processor to execute the executable instructions to implement the audio file generation method described in any one of claims 1 to 8.

12. A computer program product, the computer program product comprising computer instructions stored in a computer readable storage medium; in, The processor of the computer device reads the computer instructions from the computer-readable storage medium, and the processor is used to execute the computer instructions to implement the audio file generation method according to any one of claims 1 to 8.

Images