Audio processing method, audio processing apparatus, electronic device, computer-readable storage medium and computer program product

Abstract

An audio processing method and apparatus and a computer-readable storage medium. The method comprises: receiving an audio synthesis request, the audio synthesis request comprising audio data to be synthesized (310); on the basis of said audio data, generating chunk hidden vectors of a plurality of chunks, each chunk comprising a plurality of consecutive audio frames in a target audio, the target audio being an audio synthesized on the basis of the audio synthesis request, the plurality of chunks being sorted on the basis of the sequence of the audio frames contained in the chunks, and the audio frames of every two adjacent chunks being consecutive (320); and inputting the plurality of chunk hidden vectors into an audio generation network on the basis of the sequence of the chunks, the audio generation network being used for executing the following processing: with regard to each chunk in the target audio, while playing back a chunk previous to said chunk, generating the audio frames of said chunk on the basis of the chunk hidden vector of said chunk and the chunk hidden vector of the chunk previous to said chunk, and adding to a target playback queue the generated audio frames to be played back (330).

Description

Audio processing method, audio processing apparatus, electronic device, computer-readable storage medium, and computer program product

[0001] Cross-reference to Related Applications

[0002] This application is based on and claims priority to Chinese Patent Application No. 202410865331.5, filed on June 28, 2024, the entire contents of which are incorporated herein by reference. TECHNICAL FIELD

[0003] Embodiments of the present application relate to the field of artificial intelligence, and in particular to an audio processing method, an audio processing apparatus, an electronic device, a computer-readable storage medium, and a computer program product. BACKGROUND

[0004] In the current audio processing, when receiving an audio synthesis request of an object, an entire audio is usually synthesized based on audio data (such as a musical score) provided by the object, and then played to the object. Synthesizing the entire audio is time-consuming, and therefore, the response efficiency of the audio synthesis processing in the related art is not high. There is a lack of a technology that can improve the response efficiency of audio synthesis while ensuring the accuracy of synthesis. SUMMARY

[0005] In view of this, embodiments of the present application provide an audio processing method, an audio processing apparatus, an electronic device, a computer-readable storage medium, and a computer program product, which can improve the response efficiency of an audio synthesis request while maintaining the fluency of audio synthesis.

[0006] According to an aspect of embodiments of the present application, an audio processing method is provided, which is executed by an electronic device, and includes:

[0007] receiving an audio synthesis request, the audio synthesis request including to-be-synthesized audio data;

[0008] generating a plurality of block hidden vectors of a plurality of blocks based on the to-be-synthesized audio data, wherein each of the blocks includes a plurality of continuous audio frames in a target audio, the target audio is an audio synthesized based on the audio synthesis request, the plurality of blocks are sorted based on the order of the audio frames included in the blocks, and the audio frames of two adjacent blocks are continuous;

[0009] The second generation unit is configured to input the plurality of the chunk hidden vectors into an audio generation network in the order of the chunks, wherein the audio generation network is configured to perform the following processing: for each chunk in the target audio, generating the audio frame of the chunk based on the chunk hidden vector of the chunk and the chunk hidden vectors of the preceding chunks of the chunk while playing the preceding chunks of the chunk, and adding the generated audio frame into a target play queue for playing.

[0010] According to an aspect of some embodiments of the present application, an audio processing apparatus is provided, comprising:

[0011] The receiving unit is configured to receive an audio synthesis request, wherein the audio synthesis request comprises to-be-synthesized audio data.

[0012] The first generation unit is configured to generate a plurality of chunk hidden vectors of chunks based on the to-be-synthesized audio data, wherein each of the chunks comprises a plurality of continuous audio frames in a target audio, the target audio is an audio synthesized based on the audio synthesis request, the order of the plurality of chunks is sorted based on the order of the audio frames comprised by the chunks, and the audio frames of two adjacent chunks are continuous.

[0013] The second generation unit is configured to input the plurality of the chunk hidden vectors into an audio generation network in the order of the chunks, wherein the audio generation network is configured to perform the following processing: for each chunk in the target audio, generating the audio frame of the chunk based on the chunk hidden vector of the chunk and the chunk hidden vectors of the preceding chunks of the chunk while playing the preceding chunks of the chunk, and adding the generated audio frame into a target play queue for playing.

[0014] According to an aspect of some embodiments of the present application, an electronic device is provided, comprising a memory and a processor, wherein the memory stores a computer program, and the processor implements the audio processing method as described above when executing the computer program.

[0015] According to an aspect of some embodiments of the present application, a computer readable storage medium is provided, wherein the storage medium stores a computer program, and the computer program is executed by a processor to implement the audio processing method as described above.

[0016] According to an aspect of some embodiments of the present application, a computer program product is provided, comprising a computer program, wherein the computer program is read and executed by a processor of a computer device, so that the computer device performs the audio processing method as described above.

[0017] In the embodiments of this application, after receiving an audio synthesis request, instead of generating an audio frame hidden vector of the entire target audio based on the entire to-be-synthesized audio data, and then generating the target audio based on the audio frame hidden vector of the entire target audio by using an audio generation network, the target audio is considered as a plurality of blocks, and each block corresponds to one or more audio frames in the target audio. Then, based on the to-be-synthesized audio data, a block hidden vector of each block is generated. When the target audio is generated by using the audio generation network, for each block, an audio frame of the block is generated based on the block hidden vector of the block and the block hidden vectors of the previous blocks of the block, and the generated audio frame is added to a target playback queue for playback. The process of adding the audio frame of the block to the target playback queue is performed at the same time as the process of playing the audio frames of the previous blocks of the block. In this way, the user does not have to wait until the entire target audio is synthesized before hearing the played target audio, but can play the audio frame of a block as soon as the block is synthesized, while the audio frame of the next block is being generated at the same time. The manner of simultaneously generating the audio frame of a block and playing the previous audio frame improves the response efficiency of audio synthesis. At the same time, the audio frame of a block is generated based on the block hidden vector of the block and the block hidden vectors of the previous blocks of the block, which ensures that the features of each block are not considered in isolation when generating the target audio, but the connection and mutual influence among multiple blocks are considered, thereby improving the accuracy of audio synthesis.

[0018] Other features and advantages of the embodiments of the present application will be described in the following description, and part of them will become apparent from the description, or will be learned from the practice of the embodiments of the present application. The purposes and other advantages of the embodiments of the present application can be achieved and obtained by the structures specifically pointed out in the specification, claims and drawings. BRIEF DESCRIPTION OF DRAWINGS

[0019] The accompanying drawings are used to provide a further understanding of the technical solutions of the embodiments of the present application, and constitute a part of the specification, and are used to explain the technical solutions of the embodiments of the present application together with the embodiments of the embodiments of the present application, and do not constitute a limitation to the technical solutions of the embodiments of the present application.

[0020] FIG. 1 is a framework diagram of a system to which an audio processing method according to an embodiment of the present application is applied;

[0021] FIGS. 2A to 2D are interface schematic diagrams in a song generation scenario to which an embodiment of the present application is applied;

[0022] FIG. 3 is a flowchart of an audio processing method according to an embodiment of the present application;

[0023] FIG. 4 is a schematic diagram of block division of a plurality of audio frames in a target audio according to an embodiment of the present application;

[0024] FIG. 5 is a flowchart of step 320 according to an embodiment of the present application;

[0025] FIG. 6 is a schematic diagram of generating a chunk-wise latent vector of a chunk based on audio data to be synthesized according to an embodiment of the present application;

[0026] FIG. 7 is another flowchart of step 320 according to an embodiment of the present application;

[0027] FIG. 8 is a flowchart of step 730 according to an embodiment of the present application;

[0028] FIG. 9 is a flowchart of step 820 according to an embodiment of the present application;

[0029] FIG. 10 is a flowchart of step 930 according to an embodiment of the present application;

[0030] FIG. 11 is another flowchart of step 930 according to an embodiment of the present application;

[0031] FIG. 12 is a schematic diagram of obtaining a chunk-wise latent vector based on a previous chunk-wise latent vector and an intermediate latent vector according to an embodiment of the present application;

[0032] FIGS. 13A to 13C are schematic diagrams of an audio generation network generating audio frames included in a chunk one by one and adding the audio frames to a target play queue according to an embodiment of the present application;

[0033] FIGS. 14A to 14D are schematic diagrams of a second neural network and an audio generation network generating audio frames included in a chunk one by one and adding the audio frames to a target play queue according to an embodiment of the present application;

[0034] FIG. 15 is a flowchart of step 330 according to an embodiment of the present application;

[0035] FIG. 16 is a flowchart of step 1510 according to an embodiment of the present application;

[0036] FIG. 17 is a schematic diagram of padding a chunk-wise latent vector of a chunk with a chunk-wise latent vector of a previous chunk of the chunk according to an embodiment of the present application;

[0037] FIG. 18 is a flowchart of step 1510 according to an embodiment of the present application;

[0038] FIG. 19 is a flowchart of step 1830 according to an embodiment of the present application;

[0039] FIG. 20 is a schematic diagram of convolution in one layer in an audio generation network according to an embodiment of the present application;

[0040] FIG. 21 is a flowchart of training a first neural network, a second neural network, and an audio generation network according to an embodiment of the present application;

[0041] FIG. 22 is a flowchart of training a third neural network, a fourth neural network, and an audio generation network according to an embodiment of the present application;

[0042] FIG. 23 is a flowchart of an embodiment of an audio processing method according to an embodiment of the present application;

[0043] FIG. 24 is a block diagram of an audio processing apparatus according to an embodiment of the present application;

[0044] FIG. 25 is a block diagram of a terminal structure for performing the audio processing method shown in FIG. 3 according to an embodiment of the present application;

[0045] FIG. 26 is a block diagram of a server structure for performing the audio processing method shown in FIG. 3 according to an embodiment of the present application. DETAILED DESCRIPTION

[0046] In order to make the objectives, technical solutions, and superiorities of the embodiments of the present application clearer, the embodiments of the present application are further explained in detail below with reference to the drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the embodiments of the present application and should not be used to limit the embodiments of the present application.

[0047] Before the embodiments of the present application are further explained, terms and names involved in the embodiments of the present application are explained, and the terms and names involved in the embodiments of the present application are applicable to the following explanations:

[0048] Deep Neural Networks (DNN): a kind of multi-layer unsupervised neural network. Deep neural networks learn features from the output of the previous layer as the input of the next layer, and map the features of the existing space samples to another feature space through layer-by-layer feature mapping, so as to learn better feature expression for the existing input. Deep neural networks have multiple nonlinear mapping feature transformations, and can fit highly complex functions.

[0049] Diffusion model: a generative model based on latent diffusion, which can be applied to sound generation and other fields. The core idea of diffusion model is to regard the data generation process as a process of gradually transitioning from ordered state to disordered state, and then recover from disordered state to ordered state through inverse process, so as to generate new data samples. The forward diffusion process starts from a data point, and then gradually increases the noise through a series of operations, so that the data point gradually becomes disordered. The inverse diffusion process gradually removes the noise through a series of inverse operations, so that the disordered state is restored to the ordered state, thereby generating new data points.

[0050] Variational autoencoder: a deep generative model that maps input data to a latent space through an encoder, and outputs a mean vector and a variance vector, which together describe a Gaussian distribution. The role of the encoder is to learn the probability distribution of the data, so that the points sampled from the latent space can generate new data similar to the original data.

[0051] Self-attention: a mechanism widely used in sequence models, which allows the model to consider the information of other elements in the sequence when processing each element in the sequence. The core idea of self-attention mechanism is to calculate the attention weight of each element in the sequence to all other elements, and then weight sum the information of other elements according to these weights. For each element in the sequence, the model generates three vectors: query (Q), key (K) and value (V). These vectors are usually obtained through different linear transformations. A key feature of self-attention mechanism is that it can capture long-range dependencies within the sequence, and due to its parallelization characteristics, it has high computational efficiency.

[0052] Mel-frequency cepstral coefficients: extract the frequency domain features of the signal by performing Mel filter bank analysis and then calculating the cepstral coefficients. Mel filter bank analysis is a nonlinear transformation method that simulates human ear frequency perception. It converts linear frequency spectrum into nonlinear frequency spectrum based on Mel scale. Mel scale is a logarithmic frequency unit that is closer to the perception of the human auditory system. Cepstral analysis is a method of decomposing a signal into its slowly varying envelope and rapidly varying details (such as periodic oscillations).

[0053] System architecture and scenario description

[0054] Figure 1 is a system architecture diagram applied to the audio processing method according to an embodiment of the present application. It includes: terminal 110, Internet 120, gateway 130, and server 140.

[0055] The server 140 refers to a computer system capable of providing the terminal 110 with an audio synthesis service. In comparison with the terminal 110, the server 140 has higher requirements in stability, security, performance, and the like. The server 140 can be a high-performance computer in a network platform, a cluster of multiple high-performance computers, a portion of a high-performance computer (e.g., a virtual machine), a combination of portions of multiple high-performance computers (e.g., virtual machines), or the like. The server 140 can also communicate with the Internet 120 in a wired or wireless manner to exchange data.

[0056] The gateway 130, also referred to as an internetworking connector or a protocol converter, implements network interconnection at the transport layer and is a computer system or device that acts as a conversion function. The gateway 130 is a translator between two systems that use different communication protocols, data formats, or languages, or even have completely different architectures. Meanwhile, the gateway 130 can also provide filtering and security functions. Messages sent by the terminal 110 to the server 140 are sent to the corresponding server 140 through the gateway 130. Messages sent by the server 140 to the terminal 110 are also sent to the corresponding terminal 110 through the gateway 130.

[0057] The terminal 110 is a device used by a user to view synthesized audio. It includes various forms such as desktop computers, laptop computers, PDAs (personal digital assistants), mobile phones, vehicle-mounted terminals, home theater terminals, and dedicated terminals. In addition, it can be a single device or a collection of multiple devices. For example, multiple devices are connected through a local area network, share a display device, and work cooperatively to form a terminal. The terminal 110 can also communicate with the Internet 120 in a wired or wireless manner to exchange data.

[0058] The embodiments of the present application can be applied in various scenarios, such as the song synthesis scenario shown in FIGS. 2A-2D.

[0059] As shown in FIG. 2A, a user can upload a song score that the user wants to synthesize. After uploading the song score, the user can click “Start Synthesis” to submit a song synthesis request, so that the audio processing method of the embodiments of the present application performs song synthesis based on the song score.

[0060] The resulting song composition returned to the object is shown in Figure 2B. First, the object receives the first segment of the composed song audio, which is one minute long. The object can then start playing the audio. As shown in Figure 2C, after 30 seconds of playback, a second segment of the song audio, also one minute long, is synthesized and concatenated with the first segment to form a two-minute song audio. As shown in Figure 2D, after 30 seconds of playback, a third segment of the song audio, also one minute long, is synthesized and concatenated with the preceding segments to form a three-minute song audio. Throughout this process, the audio continues to play.

[0061] Assuming it takes 30 seconds to generate each one-minute segment of song audio, and the song audio duration is 3 minutes, then after generating the complete song audio and returning it to the object, the object needs to wait 1 minute and 30 seconds after submitting the song synthesis request. However, in Figures 2B to 2D, the song audio is synthesized in three segments. After submitting the song synthesis request, the object only needs to wait 30 seconds for the first one-minute segment of song audio to be synthesized before starting to play the audio. The subsequent two minutes of song audio can be synthesized during playback. Therefore, the audio processing method of this embodiment significantly reduces the waiting time for the object after submitting the song synthesis request, improving the response efficiency of the song synthesis request while maintaining the smoothness of the song.

[0062] General Description of Embodiments in this Application

[0063] According to one embodiment of the present application, an audio processing method is provided.

[0064] Audio processing methods can be processes that synthesize audio based on an object's audio synthesis request. In related technologies, to ensure the smoothness of the synthesized audio, the synthesized audio needs to be returned to the object after the complete audio synthesis. This results in low response efficiency to the object's audio synthesis request. Therefore, this application provides an audio processing method that aims to improve the response efficiency of audio synthesis requests without sacrificing the smoothness of the synthesized audio.

[0065] The audio processing method of this application embodiment can be executed by the terminal 110 alone, or by the server 140 alone, or it can be partially executed in the terminal 110 and partially executed in the server 140.

[0066] As shown in Figure 3, according to one embodiment of this application, the audio processing method includes:

[0067] Step 310: Receive audio synthesis request.

[0068] Step 320: Based on the audio data to be synthesized, generate block latent vectors for multiple blocks.

[0069] Step 330: Input multiple block latent vectors into the audio generation network in the order of the blocks. For each block in the target audio, while playing the preceding blocks, generate the block's audio frame based on the block's latent vector and the preceding blocks' latent vectors, and add the generated audio frame to the target playback queue for playback.

[0070] The following is a detailed description of steps 310-330 above.

[0071] Detailed description of step 310

[0072] In step 310, an audio synthesis request is received.

[0073] An audio synthesis request can be issued by the terminal device owned by the object, used to synthesize the desired audio. The audio synthesis request includes the audio data to be synthesized. The content of the audio data to be synthesized varies depending on the audio type. For example, when the object wants to synthesize a song audio, the audio data to be synthesized may include the sheet music data corresponding to the song; when the object wants to synthesize a speech audio, the audio data to be synthesized may include the text content of the speech and the speaker's voice data.

[0074] The audio data to be synthesized can be created by the object itself or obtained from an open-source audio database, such as obtaining sheet music data from a song audio database.

[0075] Detailed description of step 320

[0076] In step 320, based on the audio data to be synthesized, block latent vectors for multiple blocks are generated.

[0077] Each block comprises multiple consecutive audio frames from the target audio, which is synthesized based on an audio synthesis request. The blocks are ordered according to the order of the audio frames within them, with adjacent blocks containing consecutive audio frames. For example, as shown in Figure 4, the target audio contains 12 audio frames. When divided into blocks of 4 audio frames each, three blocks are obtained: block s1, block s2, and block s3. Block s1 includes consecutive audio frames f1-f4, block s2 includes consecutive audio frames f5-f6, and block s3 includes consecutive audio frames f9-f12. The blocks are ordered according to the order of the audio frames within them; therefore, the result of sorting the blocks according to the order of audio frames f1-f12 is: block s1, block s2, and block s3.

[0078] Based on the audio data to be synthesized, a block latent vector can be generated for each block. The block latent vector is a latent representation obtained by mapping the features of the audio frames in the block to a latent space. The latent space is a high-dimensional, abstract vector space, typically smaller than the original data space, used to extract the main features and results of the data while ignoring irrelevant details. A latent representation refers to a point or vector of data in the latent space; it represents the intrinsic features of the data while removing noise and other unnecessary information. Therefore, the block latent vector is the result of extracting important features of the audio frames in the block and removing noise and unnecessary features.

[0079] In one implementation, as shown in Figure 5, generating multiple block latent vectors based on the audio data to be synthesized can be achieved in the following way:

[0080] Step 510: Input the audio data to be synthesized into the third neural network for feature extraction to obtain the audio frame features of each audio frame in the target audio.

[0081] Step 520: Input the audio frame features of each audio file into the fourth neural network for convolution to obtain the audio frame latent vector corresponding to each audio frame.

[0082] Step 530: Refer to the correspondence between the reference blocks and audio frames, and generate block latent vectors based on the audio frame latent vectors.

[0083] The third neural network can extract audio frame features from each audio frame in the target audio based on the audio data to be synthesized. The third neural network may include a duration control module, configured to first determine the audio frame length of the target audio based on the audio data to be synthesized; that is, how many audio frames are in the target audio. After determining the audio frame length, audio frame features can then be extracted from each audio frame based on the audio data to be synthesized.

[0084] After obtaining the audio frame features of each audio frame using the third neural network, these features can be input into the fourth neural network for convolution to obtain the latent audio vector for each audio frame. The fourth neural network performs convolution based on the audio frame features corresponding to all audio frames in the target audio to obtain the audio frame features corresponding to each audio frame.

[0085] The fourth neural network can be a variational autoencoder, which can obtain the audio frame latent vectors corresponding to multiple audio frames based on the audio frame features corresponding to multiple audio frames.

[0086] The fourth neural network can also be a diffusion model. It can progressively denoise an initial latent feature filled with Gaussian noise based on the audio frame features corresponding to multiple audio frames, obtaining audio frame features corresponding to multiple audio frames. The initial latent feature is a pre-generated feature vector that has not yet observed any features or attributes. By using the fourth neural network to denoise the initial latent feature based on the audio frame features of multiple audio frames, the initial latent feature can be integrated into the audio frame features, resulting in audio frame latent vectors corresponding to multiple audio frames.

[0087] The correspondence between audio blocks and audio frames refers to which audio frames(s) are contained in each block. This correspondence can be predetermined; for example, block s1 contains audio frames f4 and f5, and block s2 contains audio frames f1, f2, and f3. Alternatively, the correspondence can be based on a predetermined block length. For instance, if the predetermined block length is 4, meaning each block contains 4 audio frames, then the block can be divided starting from the first audio frame, with each block consisting of 4 audio frames.

[0088] Based on the correspondence between reference blocks and audio frames, the latent vectors of the audio frames in a block are concatenated to form the latent vectors of the corresponding blocks.

[0089] The specific process can be represented as shown in Figure 6. First, the audio data to be synthesized is input into the third neural network to obtain the audio frame features of audio frames f1-f12. The audio frame features of audio frames f1-f12 are then input into the fourth neural network to obtain the audio frame latent vectors of audio frames f1-f12. The correspondence between blocks and audio frames is that every four consecutive audio frames constitute a block. Therefore, audio frames f1-f4 can form a block, audio frames f5-f8 can form a block, and audio frames f9-f12 can form a block. The audio frame latent vectors corresponding to the audio frames in each block are then used to form the block latent vector.

[0090] The implementation of steps 510-530 utilizes the fourth neural network to calculate latent vectors for the audio frame features of all audio frames, ensuring the accuracy of audio frame feature acquisition. In other words, by calculating latent vectors based on global audio frame features, the accuracy of audio synthesis based on these features is ensured. Therefore, the implementation of steps 510-530 is beneficial for improving both the response efficiency and accuracy of audio synthesis.

[0091] In another implementation, as shown in Figure 7, generating multiple block latent vectors based on the audio data to be synthesized can be achieved in the following way:

[0092] Step 710: Input the audio data to be synthesized into the first neural network for feature extraction to obtain the audio frame features of each audio frame in the target audio.

[0093] Step 720: Refer to the correspondence between the reference blocks and audio frames, and generate block feature vectors based on the audio frame features.

[0094] Step 730: Input the block feature vectors of each block into the second neural network for convolution to obtain the block latent vectors of each block.

[0095] The first neural network in step 710 is the same as the third neural network in step 730, and the process of generating audio frame features for each audio frame in the target audio is also the same. To save space, it will not be described again here.

[0096] In step 720, generating a block feature vector based on audio frame features, referring to the correspondence between blocks and audio frames, is similar to generating a block latent vector based on audio frame latent vectors in step 530, referring to the correspondence between blocks and audio frames. That is, the audio frame features corresponding to the audio frames in a block are used to construct the block feature vector. For example, if a block contains audio frames f1 to f4, then the audio frame features corresponding to audio frames f1 to f4 are used to construct the block feature vector.

[0097] In step 730, the block feature vectors of each block are input into the second neural network for convolution to obtain the block latent vectors of each block.

[0098] In one implementation, as shown in Figure 8, the block feature vectors of each block are input into a second neural network for convolution to obtain the block latent vectors of each block. This can be achieved in the following way:

[0099] Step 810: For the first block among multiple blocks, obtain the first intermediate parameters generated in the second neural network by the second block before the first block.

[0100] Step 820: Based on the first intermediate parameter, the first block and the block feature vector of the third block after the first block, the second neural network is used to perform convolution to obtain the block latent vector of the first block.

[0101] The first block can be any one of multiple blocks. When the second neural network generates the block latent vector corresponding to the first block, the first intermediate parameter generated by the second block before the first block in the second neural network is obtained. The block latent vector of the first block is obtained based on the first intermediate parameter, the first block, and the block feature vector of the third block after the first block.

[0102] If the first block is the first block among multiple blocks, the block latent vector of the first block can be obtained directly based on the block feature vectors of the first block and the third block after the first block.

[0103] Unlike the fourth neural network in the embodiments of steps 510-530, which performs convolution based on the audio frame features corresponding to all audio frames in the target audio, the second neural network performs convolution based on the audio frame features corresponding to the audio frames in each block.

[0104] Because there are connections between consecutive audio frames, if the block latent vector for the first block is generated solely based on the audio frames within the first block itself, the audio frames in the first block cannot be linked to those in other blocks, resulting in low accuracy. Therefore, when generating the block latent vector for the first block, the third block before and after the first block can be considered simultaneously. This allows the block latent vector for the first block to be generated based on the connections between the audio frames in the first block and other blocks, thus improving the accuracy of the block latent vector generation.

[0105] The first intermediate parameters are the parameters and values ​​generated by the second neural network when generating the block latent vectors for the second block before the first block. For example, they include latent features generated by the second neural network during the process of generating the block latent vectors for the second block before the first block, used to obtain key features in the block feature vectors; block semantic association information between the second blocks before the first block, etc. For example, if the first block includes block s1 and block s2, the first intermediate parameters may include key features of block s1, key features of block s2, and block semantic association information between block s1 and block s2, etc.

[0106] Since the first intermediate parameter is the parameter and value generated by the second neural network when generating the block latent vector corresponding to the second block before the first block, it can fully express the key information in the process of mapping the block feature vector of the second block before the first block to the latent space. Therefore, using the first intermediate parameter can improve the accuracy of generating the block latent vector of the first block. The block latent vector of the third block after the first block is generated after the block latent vector corresponding to the first block is generated. Therefore, when generating the block latent vector of the third block after the first block, it can refer to the first intermediate parameter corresponding to the first block. And when the first block considers the third block, it can directly use the block feature vector corresponding to the third block after the first block.

[0107] In one implementation, as shown in Figure 9, the block latent vector of the first block is obtained by convolution using a second neural network based on the block feature vectors of the first intermediate parameter, the first block, and the third block following the first block. This can be achieved in the following way:

[0108] Step 910: Obtain the key weight matrix, value weight matrix, and query weight matrix of the block feature vector of the first block.

[0109] Step 920: Based on the first intermediate parameter, the block feature vector of the first block and the third block after the first block, the key weight matrix and the value weight matrix, determine the first key and the first value corresponding to the first block.

[0110] Step 930: Based on the self-attention calculation results of the query weight matrix, the first key, the first value, and the block feature vector of the first block, determine the block latent vector of the first block.

[0111] The key weight matrix is ​​used to convert the block feature vectors into key vectors. The dot product of the key vectors and the query vector determines the attention score. The value weight matrix is ​​used to convert the block feature vectors into value vectors, which contain the actual information of the feature vectors. The query weight matrix is ​​used to convert the feature vectors into query vectors, which represent the query representation of the block feature vectors when calculating attention. All three matrices can be obtained through iterative training based on initial values.

[0112] In step 920, based on the first intermediate parameters, the block feature vectors of the first block and the third block following the first block, the key weight matrix, and the value weight matrix, the first key and the first value corresponding to the first block are determined. The first intermediate parameters may include, in addition to semantically relevant information such as the latent features of the second block preceding the first block and block semantic association information, the key weight matrix and the value weight matrix corresponding to the block feature vector of the block preceding the first block.

[0113] Since the second neural network requires multiple layers of computation, the first key and first value corresponding to the first block in each layer can be determined.

[0114] Based on the first intermediate parameter, the block feature vectors of the first block and the third block after the first block, and the key weight matrix, the process of determining the first key corresponding to the first block can be expressed as Equation 1:

[0115] In Formula 1, W represents the first key corresponding to the nth layer of the first block i in the second neural network; i represents the identifier of the first block among multiple blocks; n represents the number of neural network layers in the second neural network that calculate the block latent vector based on the block feature vector of the first block;k Represents the key weight matrix; It is the joint representation vector of the first intermediate parameter; This represents the key weight matrix corresponding to the block feature vector of the block preceding the first block; This represents the block feature vector of the first block; This represents the block feature vector of the third block after the first block.

[0116] Multiply the joint representation vector of the first intermediate parameter by the key weight matrix of the first block to obtain the first product; multiply the key weight matrix of the first block by the block feature vector to obtain the second product; multiply the key weight matrix of the first block by the block feature vector of the third block after the first block to obtain the third product; concatenate the first product, the key weight matrix representing the block feature vector of the block preceding the first block, the second product, and the third product to obtain the first key corresponding to the first block.

[0117] Based on the first intermediate parameter, the first block, and the block feature vector and value weight matrix of the third block after the first block, the process of determining the first value corresponding to the first block can be expressed as Formula 2:

[0118] In Formula 1, W represents the first value corresponding to the nth layer of the second neural network for the first block i; v Represents the value weight matrix; This represents the value weight matrix corresponding to the block feature vector of the previous block of the first block; other parameters are the same as in Formula 1, and will not be repeated here.

[0119] Multiply the joint representation vector of the first intermediate parameter by the value weight matrix of the first block to obtain the fourth product; multiply the value weight matrix of the first block by the block feature vector to obtain the fifth product; multiply the value weight matrix of the first block by the block feature vector of the third block after the first block to obtain the sixth product; concatenate the fourth product, the value weight matrix representing the block feature vector of the block preceding the first block, the fifth product, and the sixth product to obtain the first value corresponding to the first block.

[0120] In step 930, the block latent vector of the first block is determined based on the self-attention calculation results of the query weight matrix, the first key, the first value and the block feature vector of the first block.

[0121] The block latent vector of the first block can be determined using a feed-forward network (FFN), which is used to further process and refine the feature representation after the self-attention mechanism.

[0122] After obtaining the first key and first value in the nth layer of the second neural network, the self-attention result of the nth layer can be calculated, and the hidden vector of the nth layer can be determined using a feedforward network. The hidden vector of the nth layer is then passed to the next layer as the block feature vector of the next layer. This continues until the last layer of the second neural network outputs the block hidden vector of the first block.

[0123] Therefore, the process of determining the hidden vector of the nth layer, which is the block feature vector of the (n+1)th layer, using the feedforward network in the nth layer of the second neural network can be expressed as Equation 3:

[0124] In Formula 3, This indicates the result of the self-attention calculation of the weight matrix, the first key, the first value, and the block feature vector of the first block at the nth layer.

[0125] In the (n+1)th layer, not only is it necessary to use the latent vector of the first block in the nth layer as the block feature vector of the first block in the (n+1)th layer, but it is also necessary to use the latent vector of the first block in the nth layer as the block feature vector of the third block after the first block in the (n+1)th layer. Therefore, it is also necessary to iterate on the block feature vector of the third block after the first block. The process of iterating on the latent vector of the third block after the first block using a feedforward network can be expressed as Equation 4:

[0126] In Formula 4, This represents the self-attention calculation result at layer n, based on the query weight matrix, the first key, the first value, and the block feature vector of the third block after the first block.

[0127] In one implementation, as shown in Figure 10, the block latent vector of the first block is determined based on the self-attention calculation result of the query weight matrix, the first key, the first value, and the block feature vector of the first block. This can be achieved in the following way:

[0128] Step 1010: Normalize the block feature vector of the first block to obtain the normalized block feature vector.

[0129] Step 1020: Add the self-attention calculation result of the query weight matrix, the first key, the first value and the normalized block feature vector to the block feature vector of the first block, and determine the block latent vector of the first block based on the addition result.

[0130] Normalizing the block feature vectors before calculating self-attention to determine the block latent vectors helps reduce data perturbation during model training and improves the model's generalization ability. Since calculating the block latent vectors also requires the block feature vectors of the third block after the first block, the block feature vectors of the third block after the first block can also be normalized.

[0131] The process of dividing the feature vectors of the first block and the third block following the first block can be represented as the formula...

[0132] Formula 5:

[0133] In Formula 5, This can be represented as normalizing the block feature vector of the first block to obtain the normalized block feature vector; LayerNorm represents the normalization of the feature vectors of the third block after the first block to obtain the normalized feature vector; LayerNorm represents the normalization process.

[0134] The process of adding the self-attention calculation result of the query weight matrix, the first key, the first value, and the normalized block feature vector to the block feature vector of the first block can be expressed as Equation 6:

[0135] In Formula 6, W q Represents the query weight matrix; The process of self-attention processing involves first multiplying the query weight matrix with the normalized block feature vector corresponding to the first block, and then performing self-attention calculation based on the product, the first key, and the first value.

[0136] Similarly, the self-attention calculation result of the query weight matrix, the first key, the first value, and the block feature vector of the third block after the first block can be expressed as Equation 7:

[0137] The first intermediate parameter can include not only the parameters and values ​​generated by the second block before the first block when generating the block latent vector, but also the parameters and values ​​generated by the layers before the current layer when the first block is computed in the current layer of the second neural network.

[0138] Therefore, the first intermediate parameters can also be iterated. The parameters and values ​​generated in the nth layer of the second neural network can be expressed as Equation 8:

[0139] In Formula 8, This represents the parameters and values ​​generated in the nth layer of the second neural network. Represents the block feature vector of the first block. The average value.

[0140] The parameters and values ​​generated in the nth layer of the second neural network are added to the first intermediate parameter, which is used to iterate the first intermediate parameter for calculating the block hidden vector of the (n+1)th layer.

[0141] In another implementation, as shown in Figure 11, the block latent vector of the first block is determined based on the self-attention calculation result of the query weight matrix, the first key, the first value, and the block feature vector of the first block. This can be achieved in the following way:

[0142] Step 1110: Determine the intermediate latent vector based on the self-attention calculation results of the query weight matrix, the first key, the first value, and the block feature vector of the first block.

[0143] Step 1120: Obtain the last vector in the implicit vector of the preceding block corresponding to the second block before the first block.

[0144] Step 1130: Determine the block latent vector of the first block based on the last latent vector and the middle latent vector.

[0145] The process of determining the intermediate hidden vector is the same as the process of determining the block hidden vector in the aforementioned implementation. The second neural network obtains the intermediate hidden vector through multiple iterations, which will not be described in detail here.

[0146] The preceding block latent vector is an intermediate latent vector generated through multiple iterations of the second block preceding the first block. The last vector can be a vector of predetermined length from the last element of the preceding block latent vector. In one implementation, the block latent vector of the first block is determined based on the last vector and the intermediate latent vectors, which can be achieved in the following way:

[0147] The last vector is concatenated to the left of the middle latent vector to obtain the first concatenated vector; the subvector of a predetermined length starting from the last position of the first concatenated vector is deleted to obtain the second concatenated vector; the second concatenated vector is subjected to causal convolution to obtain the block latent vector of the first block.

[0148] For example, a vector consists of multiple elements, and the length of a vector can be represented by the number of elements in the vector. That is, a predetermined length can be represented by a predetermined number of elements. The predetermined length of a subvector of the first concatenated vector can be deleted starting from the last element. This can be achieved by deleting the predetermined number of elements of the first concatenated vector starting from the last element, thus forming the second concatenated vector.

[0149] As shown in Figure 12, the last vector in the preceding block latent vector is concatenated with the intermediate latent vector after removing the last vector to obtain the final vector (the second concatenated vector mentioned above). This final vector is then input into a causal convolutional layer for convolution to obtain the block latent vector. Causal convolution considers only the information at the current time step and before in the input sequence, ignoring future information. This characteristic makes causal convolution particularly useful for processing time series data, as it can capture the temporal dependencies of the sequence while avoiding the prediction of future data. After performing causal convolution, the convolution result can be normalized to obtain the block latent vector.

[0150] In the implementation of steps 1110-1130, fusing the intermediate latent vector of the first block with the latent vector of the preceding block can further reduce marginal effects and improve the naturalness and fluency of the synthesized audio. Marginal effects refer to the situation where, when each block is processed independently, the boundary parts of each block may have incomplete or inaccurate information. Fusing the intermediate latent vector of the first block with the latent vector of the preceding block, when processing the current block, not only considers the information of the current block but also utilizes the information of previous blocks, thereby supplementing the missing context at the block boundaries and reducing errors caused by a lack of preceding information. Fusing latent vectors allows the model to better acquire global information from the audio data. By fusing the latent vectors of different blocks, the model can consider the context of the entire audio sequence when processing each block, thereby improving the model's global perception ability, reducing marginal effects caused by local processing, resulting in natural transitions between audio blocks and improving the coherence of the synthesized audio.

[0151] In the implementation of steps 910-930, determining the block latent vectors based on the self-attention mechanism is beneficial for obtaining the dependencies between long-distance audio frames and improving the prediction accuracy of the block latent vectors. The self-attention mechanism can learn more abstract and higher-level feature representations in the audio. The self-attention mechanism can fuse global information into the block latent vectors, making the block latent vectors more accurately represent the features of the audio.

[0152] In the implementation of steps 710-730, the audio frame features of multiple audio frames are first divided into blocks, and then a second neural network is used to calculate the block latent vector based on each block. After calculating the block latent vector of one block, the second neural network can directly input the block latent vector into the audio generation network for audio generation, while simultaneously calculating the block latent vector of the next block. In this way, the system does not need to wait for the generation of the audio frame latent vectors corresponding to all audio frames, but only needs to wait for the generation of the block latent vector corresponding to the first block and the audio frame, further improving the response efficiency of audio synthesis processing.

[0153] Detailed description of step 330

[0154] In step 330, multiple block latent vectors are input into the audio generation network in the order of the blocks.

[0155] The audio generation network generates audio frames for each block in the target audio while playing the preceding blocks. These frames are based on the block's latent vector and the latent vectors of its preceding blocks. The generated audio frames are then added to the target playback queue for playback. The preceding blocks can be one or more blocks that precede the current block in the block order.

[0156] An audio generation network can be a type of generative network used for audio generation. A target playback queue is used to store playable audio. Once the audio frame corresponding to the first segment is generated, it is added to the target playback queue, and the object can then begin playing the audio through the queue. During audio playback, audio frames corresponding to subsequent segments are continuously generated and added to the target playback queue. When an audio frame corresponding to a subsequent segment is added to the target playback queue, the audio currently playing in the queue does not pause; the audio frame is automatically added to the end of the queue, awaiting playback.

[0157] Audio sequences are time-dependent sequences; therefore, preceding audio frames influence the generation of subsequent audio frames. Consequently, causal convolution can be used in audio generation networks for audio generation.

[0158] As shown in Figures 13A to 13C, block latent vectors M1, M2, and M3 can be obtained according to the block segmentation order. In Figure 13A, the block latent vector M1 is first input into the audio generation network to obtain audio frame N1, which includes multiple audio frames included in the block. Audio frame N1 is added to the target playback queue, and the target can start playing audio frame N1. At the same time as adding audio frame N1 to the target playback queue, the block latent vector M2 is input into the audio generation network for audio generation, as shown in Figure 13B, to obtain audio frame N2. At this time, the audio in the target playback queue plays to the position indicated by the triangle, and audio frame N2 is added to the target playback queue. Simultaneously, the block latent vector M3 is input into the audio generation network, as shown in Figure 13C, to obtain audio frame N3. At this time, the audio in the target playback queue plays to the position indicated by the triangle, and audio frame N3 is added to the target playback queue.

[0159] In the implementation of steps 710-730, the audio frame features are first divided into multiple block feature vectors based on the correspondence between blocks and audio frames. Then, the second neural network generates the block latent vectors of each block in the block order. Based on this, in one implementation, the multiple block latent vectors are input into the audio generation network in the block order, which can be achieved as follows: when the block latent vector of the first block is generated, the block latent vector is input into the audio generation network. The step of inputting the block latent vector into the audio generation network occurs before the generation of the third block after the first block, or after the generation of the third block after the first block.

[0160] In other words, when the second neural network generates a block latent vector for the first block, it directly feeds the block latent vector into the audio generation network to generate audio. At the same time, it starts generating block latent vectors for the next block, and the audio generation network generates audio frames based on the block latent vectors of the first block.

[0161] As shown in Figures 14A-14D, block feature vectors K1, K2, and K3 can be obtained according to the block segmentation order. As shown in Figure 14A, firstly, block feature vector K1 is input into the second neural network for convolution to obtain block latent vector M1. Block latent vector M1 is then input into the audio generation network, and block feature vector K2 is input into the second neural network. As shown in Figure 14B, after the audio generation network generates audio frame N1 based on block latent vector M1, audio frame N1 is added to the target playback queue. Simultaneously, the second neural network generates block latent vector M2 based on block feature vector K2. As shown in Figure 14C, block latent vector M2 is input into the audio generation network, and block feature vector K3 is input into the second neural network. The audio generation network generates audio frame N2 based on block latent vector M2 and adds audio frame N2 to the target playback queue. Simultaneously, the second neural network generates block latent vector M3 based on block feature vector K3. As shown in Figure 14D, the block latent vector M3 is input into the audio generation network to obtain audio frame N3. Audio frame N3 is added to the target playback queue, and the target audio is generated.

[0162] In this way, while waiting for the target audio to be generated, the object only needs to wait for the generation time of the block latent vector corresponding to the first block and the audio frame before it can start playing the audio, which further improves the response efficiency of audio synthesis.

[0163] Since the first segment in the target audio has no preceding segments, the value of the preceding segments is empty when the segment is the first segment in the target audio. Based on this, in one implementation, for each segment in the target audio, while playing the preceding segments, an audio frame for the segment is generated based on the segment's latent vector and the latent vectors of the preceding segments, and the generated audio frame is added to the target playback queue for playback. This can be achieved in the following way:

[0164] For the first segment in the target audio, the audio frame of the first segment is generated based on the latent vector of the first segment and added to the target playback queue for playback.

[0165] For subsequent blocks after the first block, audio frames of the subsequent blocks are generated based on the block latent vectors of the subsequent blocks and the block latent vectors of the preceding blocks, and added to the target playback queue for playback. The generation of audio frames of the subsequent blocks and their addition to the target playback queue are performed simultaneously with the playback of the preceding blocks.

[0166] Therefore, the first segment in the target audio generates the initial audio frame based on its own latent vector. Subsequent segments after the first segment generate their own audio frames based on their latent vectors and the latent vectors of the preceding segments. This results in a smoother splicing of the audio frames from subsequent segments with those from preceding segments in the target playback queue, improving the accuracy of the target audio generation.

[0167] In one implementation, as shown in Figure 15, based on the block latent vector and the block latent vector of the preceding block, audio frames of the blocks are generated, and the generated audio frames are added to the target playback queue for playback. This can be achieved in the following way:

[0168] Step 1510: Use the block latent vectors of the preceding blocks to fill the target block latent vector of the block, and obtain the extended audio frame latent vector.

[0169] Step 1520: Generate segmented audio frames based on the extended audio frame latent vector, and add the generated audio frames to the target playback queue for playback.

[0170] Since the vectors at both ends of the block latent vector are difficult to fully convolve with other features during the convolution process, the features at both ends of the block latent vector can be fully integrated by padding the ends of the block latent vector. The target block latent vector is the block latent vector corresponding to the block in which the audio frame is being generated. To consider the influence of the preceding blocks on the block audio synthesis, the target block latent vector of the block can be padded with the block latent vectors of the preceding blocks to ensure that the block latent vectors can be fully integrated.

[0171] In one implementation, as shown in Figure 16, the target block latent vector of the block is filled with the block latent vector of the block using the block latent vector of the preceding block to obtain the extended audio frame latent vector. This can be achieved in the following way:

[0172] Step 1610: Obtain the padding latent vector from the block latent vector of the preceding block according to the predetermined padding length.

[0173] Step 1620: Fill the left side of the target block latent vector with the padding latent vector to obtain the augmented audio frame latent vector.

[0174] The predetermined padding length is the length of the vector used to pad the target block implicit vector of the block. For example, if the predetermined padding length is 4, then 4 bits can be obtained from the end of the implicit vector of the preceding block as the padding implicit vector.

[0175] After obtaining the padding latent vector, it is padded to the left of the target block latent vector to obtain the augmented audio frame latent vector. As shown in Figure 17, z t With z t+1 It is the target block latent vector for the block; z t-1 To z t-4 The padding latent vector is obtained from the block latent vectors of the preceding blocks and filled into z. t To the left. Therefore, z t-4 To z t+1 Together they form the extended audio frame latent vector, which is used to generate the audio frames corresponding to the blocks.

[0176] Filling the latent vector with padding vectors to the left of the target block latent vectors facilitates the causal convolution learning of the continuity between audio frame sequences, thereby improving the accuracy of audio frame generation.

[0177] When a chunk has multiple preceding chunks, the degree of association between different preceding chunks and the chunk itself may vary. For example, in a speech audio clip, frames 1 to 10 are the speech audio, frames 11 to 20 are a pause, and frames 21 to 30 continue the narration. Therefore, the audio from frames 1 to 10 may have a greater influence on the generation of the audio from frames 21 to 30. Based on this, in one implementation, as shown in Figure 18, the target chunk latent vector of the chunk is filled with the chunk latent vector of the chunk's preceding chunks to obtain the extended audio frame latent vector. This can be achieved in the following way:

[0178] Step 1810: Input the block and its multiple preceding blocks into the semantic association prediction model to obtain the semantic association between the block and each preceding block.

[0179] Step 1820: Based on semantic relevance, determine the target block among multiple preceding blocks of the block.

[0180] Step 1830: Fill the target block latent vector of the block based on the block latent vector of the target block to obtain the extended audio frame latent vector.

[0181] A semantic relevance prediction model can be a neural network model used to predict the semantic relevance between a block and its preceding blocks. Preceding blocks with high semantic relevance to a block may have a significant impact on the generation of audio frames for that block. Therefore, the target block can be determined from among multiple preceding blocks of a block based on semantic relevance.

[0182] Based on semantic relevance, the target block is determined among multiple preceding blocks. The preceding block with a semantic relevance greater than a predetermined threshold can be determined as the target block. Alternatively, the semantic relevance of each preceding block can be sorted, and the preceding block ranked higher than the predetermined rank can be determined as the target block.

[0183] After determining the target blocks, the target block latent vectors of the blocks can be filled based on the block latent vectors of the target blocks to obtain the augmented audio frame latent vectors. During the filling process, multiple target blocks can be arranged according to their block order, and the filling latent vectors can be obtained from them to fill the target block latent vectors; alternatively, multiple target blocks can be arranged according to their semantic relevance, and the filling latent vectors can be obtained from them to fill the target block latent vectors.

[0184] In one implementation, as shown in Figure 19, the target block latent vector of the segment is filled based on the block latent vector of the target segment to obtain the extended audio frame latent vector. This can be achieved in the following way:

[0185] Step 1910: Determine the weight of the target block based on the semantic relevance between the target block and the blocks.

[0186] Step 1920: Based on the target block's latent vector and weight, fill in the target block's latent vector to obtain the extended audio frame latent vector.

[0187] Since target blocks with different semantic relevance have varying impacts on the generation of audio frames from blocks, in addition to adjusting the order of target blocks, different weights can be assigned to different target blocks. Target blocks with the highest semantic relevance have higher weights, so that the impact of target blocks with different semantic relevance on the process of generating audio frames from blocks is more significant.

[0188] For example, the target blocks include blocks S1, S2, and S3, with semantic relevance of 0.69, 0.82, and 0.73, respectively. The weights assigned to these three target blocks are 0.2, 0.5, and 0.3, respectively.

[0189] When filling the latent vector of the target block, the latent vector of the block corresponding to the target block can be multiplied with its respective weight, and the target block latent vector can be filled based on the multiplication result to obtain the extended audio frame latent vector.

[0190] Filling the latent vector of the target block with the weight of the target block can make the target block with different semantic relevance to the block more significant in the process of generating audio frames, thus improving the accuracy of audio frame generation.

[0191] In the implementation of steps 1810-1830, target blocks that significantly influence the generation of audio frames can be identified from multiple preceding blocks based on semantic relevance. Filling the latent vectors of these target blocks with their latent vectors improves the accuracy of generating audio frames based on the expanded audio frame latent vectors. Filling the latent vectors with target blocks of high semantic relevance makes the generated audio frames more semantically and acoustically compatible with the context. In music generation, the filled latent vectors allow the generated music segments to blend better with previous melodies and rhythms, making the music sound smoother. The latent vectors of the target blocks contain more accurate semantic and feature information, providing more reliable guidance for audio frame generation. For example, in speech synthesis tasks, this reduces errors in pronunciation and unnatural pauses in the generated speech, improving the accuracy and quality of speech synthesis.

[0192] After obtaining the extended audio frame latent vector, segmented audio frames can be generated based on the extended audio frame latent vector.

[0193] The audio generation network can be a multi-layer neural network that iteratively generates audio frames in blocks through causal convolution based on augmented audio frame latent vectors. In each layer of the multi-layer neural network, the augmented audio latent vector can be obtained by filling the input vector of the block in that layer with the latent vectors of the preceding blocks. Taking Figure 17 as an example, in the z-th layer of the multi-layer neural network, z... t With z t+1 It is the initial vector input to the z-th layer in blocks, z t-1 To z t-4 This is the vector input to the z-th layer from the preceding blocks during the audio frame generation process. When based on z... t-4 To z t+1After obtaining the output vector of layer z, which is also the input vector of layer y, we can obtain the vectors of the preceding blocks that were input to layer y during the audio frame generation process and fill them into the input vector of layer y corresponding to the block. This process is repeated iteratively to generate audio frames for the blocks. The filling process is the same as in the aforementioned implementation method and will not be described again here.

[0194] In each layer of a multi-layer neural network, causal convolution is first performed on the expanded audio latent vector, followed by transposed causal convolution on the result to obtain the output feature vector of that layer. After multiple iterations, the final layer of the audio generation network outputs segmented audio frames. The successive use of causal convolution and transposed causal convolution serves to obtain the long-term dependencies of the audio frame feature sequences and to restore the feature size at each layer.

[0195] As shown in Figure 20, in the z-th layer of a multi-layer neural network, z t With z t+1 It is the initial vector input to the z-th layer in blocks, z t-1 To z t-4 This is the vector input to the z-th layer from the preceding blocks during audio frame generation. The causal convolution has a kernel of 4 and a stride of 1, while the transposed causal convolution has a kernel of 4 and a stride of 2. In the output vector of the transposed causal convolution, the vectors at both ends are removed because they do not fully integrate the input features of the z-th layer. Therefore, after causal convolution and transposed causal convolution, the final input vector for the y-th layer is y. t to y t+3 .

[0196] In the implementation of steps 1510-1520, the target block latent vector of the block is filled by using the block latent vector of the preceding block. This realizes the fusion of the block information of the previous second block during the causal convolution of the block, thereby improving the accuracy of the audio frame generation of the block.

[0197] The model training process of the audio processing method in this application embodiment

[0198] The audio processing method of this application has two implementation methods: One method involves inputting the audio data to be synthesized into a first neural network for feature extraction to obtain audio frame features for each audio frame in the target audio; then dividing the multiple audio frame features into blocks to generate block feature vectors; using a second neural network to calculate the block latent vectors for each block based on the block feature vectors; and finally using an audio synthesis network to generate block audio frames based on the block latent vectors. The other method involves inputting the audio data to be synthesized into a third neural network for feature extraction to obtain audio frame features for each audio frame in the target audio; then inputting the audio frame features into a fourth neural network for convolution to obtain the audio frame latent vector corresponding to each audio frame; dividing the multiple audio frame latent vectors into blocks to generate block latent vectors for each block; and finally using an audio synthesis network to generate block audio frames based on the block latent vectors.

[0199] Regarding the first implementation method described above, in one embodiment, as shown in Figure 21, the first neural network, the second neural network, and the audio generation network are jointly trained through the following process:

[0200] Step 2110: Obtain the first sample audio data, the first sample target audio, the first sample audio features, and the first sample audio frame latent vector.

[0201] Step 2120: Input the first sample audio data into the first neural network for feature extraction to obtain the first predicted audio frame feature of each audio frame in the first sample target audio frame, and the first predicted audio feature of the first sample target audio.

[0202] Step 2130: Using the second neural network, obtain the first predicted block latent vector of each block based on the first predicted audio frame features corresponding to each block.

[0203] Step 2140: Divide the latent vector of the first sample audio frame into multiple blocks to obtain the first sample block latent vector of multiple blocks, and input the first sample block latent vector into the audio generation network in the order of the blocks to obtain the first predicted target audio.

[0204] Step 2150: Calculate the first loss function based on the first predicted audio features and the first sample audio features; calculate the second loss function based on the first predicted block latent vector and the first sample audio frame latent vector of each block; and calculate the third loss function based on the first predicted target audio and the first sample target audio.

[0205] Step 2160: Train the first neural network, the second neural network, and the audio generation network using the first loss function, the second loss function, and the third loss function.

[0206] The first sample audio data can be the source data used for audio synthesis. The first sample target audio can be the target audio corresponding to the first sample audio data. The first sample audio feature can be the overall audio features of the target audio, such as the overall audio fundamental frequency, audio duration, and Mel-spectral coefficients. The first sample audio frame latent vector can be the frame-level latent feature obtained by mapping the sample audio frame features to the latent space. The first sample audio data, the first sample target audio, and the first sample audio features can be obtained from a sample audio library. The first sample audio frame latent feature can be calculated by inputting the first sample audio data into a predetermined coding network.

[0207] The first sample audio data is input into the first neural network to obtain the first predicted audio frame feature for each audio frame in the first sample target audio. The prediction process has been described in detail in the aforementioned embodiments and will not be repeated here. The first neural network can predict not only frame-level features but also the overall audio features of the first sample target audio. For example, the first sample audio features include the overall audio fundamental frequency, audio duration, and Mel-frequency cepstral coefficients. The first neural network may include a fundamental frequency prediction network, a duration prediction network, and a Mel-frequency cepstral coefficient prediction network, which are used to predict the above three first predicted audio features, respectively.

[0208] The first predicted audio frame features of multiple audio frames are divided into multiple blocks, resulting in predicted block features for each block. These block-specific predicted features are then input into a second neural network in block order. The second neural network then uses these features to obtain the first predicted block latent vector for each block based on the first predicted audio frame features within each block. The prediction process has been described in detail in the preceding embodiments and will not be repeated here.

[0209] Multiple first-sample block latent vectors are input into the audio generation network in the order of the blocks to obtain the predicted audio frames for each block. The process of the audio generation network generating predicted audio frames has been described in detail in the previous implementation and will not be repeated here. The predicted audio frames of each block are concatenated in the order of the audio frames to obtain the target audio.

[0210] After obtaining the aforementioned parameters, a first loss function is first calculated based on the first predicted audio features and the first sample audio features. The first loss function can be the sum of the distances between the sample values ​​and predicted values ​​corresponding to different audio features. The first loss function evaluates the accuracy of the first neural network in extracting features from the first sample target audio.

[0211] A second loss function is calculated based on the first predicted block latent vector and the first sample audio frame latent vector for each block. The second loss function can be the KL loss between the first predicted block latent vector and the first sample audio frame latent vector. The second loss function evaluates the accuracy of feature acquisition by the second neural network model.

[0212] A third loss function is calculated based on the first predicted target audio and the first sample target audio. The third loss function can be the distance between the spectral vectors of the first predicted target audio and the first sample target audio. This third loss function evaluates the audio generation accuracy of the audio generation network.

[0213] The first neural network, the second neural network, and the audio generation network can be trained using the first loss function, the second loss function, and the third loss function. This can be achieved by summing the first loss function, the second loss function, and the third loss function, and then using the summation result to train the first neural network, the second neural network, and the audio generation network.

[0214] In addition to the first, second, and third loss functions, the model training loss can also include the generator loss and feature matching loss of the audio generation network. The generator loss is the loss between the first predicted target audio generated by the audio synthesis network and the first sample target audio. The feature matching loss measures the degree of difference between the features extracted by the discriminator from the first sample target audio and the first predicted target audio. The first, second, and third loss functions, the generator loss, and the feature matching loss are summed, and the summed loss result is used to train the first neural network, the second neural network, and the audio generation network.

[0215] The advantage of using the above method for model training is that the accuracy of the first neural network, the second neural network, and the audio generation network can be evaluated and trained separately, which helps to improve the accuracy of model training.

[0216] Regarding the second implementation method described above, in one embodiment, as shown in Figure 22, the third neural network, the fourth neural network, and the audio generation network are jointly trained through the following process:

[0217] Step 2210: Obtain the second sample audio data, the second sample target audio, the second sample audio features, and the second sample audio frame latent vector.

[0218] Step 2220: Input the second sample audio data into the third neural network for feature extraction to obtain the second predicted audio frame features of each audio frame in the second sample target audio frame, and the second predicted audio features of the second sample target audio.

[0219] Step 2230: Using the fourth neural network, convolution is performed based on the second predicted audio frame features of multiple audio frames to obtain the second predicted block latent vectors for each block.

[0220] Step 2240: Divide the latent vector of the second sample audio frame into multiple blocks to obtain the latent vector of the second sample block in multiple blocks, and input the latent vector of the second sample block into the audio generation network in the order of the blocks to obtain the second predicted target audio.

[0221] Step 2250: Calculate the fourth loss function based on the second predicted audio features and the second sample audio features; calculate the fifth loss function based on the second predicted block latent vector and the second sample audio frame latent vector of each block; and calculate the sixth loss function based on the second predicted target audio and the second sample target audio.

[0222] Step 2260: Train the third neural network, the fourth neural network, and the audio generation network using the fourth loss function, the fifth loss function, and the sixth loss function.

[0223] The difference between the training processes in steps 2210-2260 and steps 2110-2160 lies in the following: In steps 2210-2260, the fourth neural network predicts the latent vectors of audio frames based on the features of the second predicted audio frames corresponding to all audio frames of the second sample target audio, and then divides all audio frame vectors into blocks to obtain the second predicted block latent vectors for each block. In contrast, in steps 2110-2160, the second neural network directly predicts the first predicted block latent vectors corresponding to the blocks based on the features of the first predicted audio frames after block division. The remaining steps are the same and will not be described further.

[0224] Detailed implementation diagrams of the audio processing method in this application.

[0225] Referring now to Figure 23, the implementation details of one embodiment of the audio processing method of this application will be described in detail by way of example.

[0226] In step 2310, an audio synthesis request is received, which includes the audio data to be synthesized.

[0227] In step 2320, the audio data to be synthesized is input into the first neural network for feature extraction to obtain the audio frame features of each audio frame in the target audio.

[0228] In step 2330, the correspondence between the reference blocks and audio frames is used to generate block feature vectors based on the audio frame features.

[0229] In step 2340, for the first block among multiple blocks, the first intermediate parameters generated in the second neural network by the second block preceding the first block are obtained.

[0230] In step 2350, based on the first intermediate parameter, the block feature vectors of the first block and the third block after the first block, the key weight matrix and the value weight matrix, the first key and the first value corresponding to the first block are determined; based on the self-attention calculation results of the query weight matrix, the first key, the first value and the block feature vector of the first block, the block latent vector of the first block is determined.

[0231] In step 2360, when the block latent vector of the first block is generated, the block latent vector is input into the audio generation network. The block latent vector is input into the audio generation network before the generation of the third block after the first block in a plurality of blocks, or after the generation of the third block after the first block in a plurality of blocks.

[0232] In step 2370, for the first block in the target audio, an audio frame for the first block is generated based on the latent vector of the first block and added to the target playback queue for playback.

[0233] In step 2380, for subsequent blocks after the first block, the block latent vector of the subsequent block is filled with the block latent vector of the second block before the subsequent block to obtain the extended audio frame latent vector; the audio frame of the subsequent block is generated based on the extended audio frame latent vector and added to the target playback queue.

[0234] Description of apparatus and devices in embodiments of this application

[0235] It is understood that although the steps in the above flowcharts are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated in this embodiment, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the above flowcharts may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages in other steps.

[0236] It should be noted that in various specific embodiments of this application, when processing is required based on data related to the characteristics of the target content, such as target content attribute information or attribute information sets, permission or consent from the target content provider will be obtained first. Furthermore, the collection, use, and processing of this data will comply with relevant laws, regulations, and standards. In addition, when embodiments of this application require obtaining target content attribute information, separate permission or consent from the target content provider will be obtained through pop-ups or redirection to a confirmation page. Only after obtaining the separate permission or consent from the target content provider will the necessary target content-related data for the normal operation of the embodiments of this application be obtained.

[0237] Figure 24 is a schematic diagram of the structure of an audio processing device 2400 provided in an embodiment of this application. The audio processing device 2400 includes:

[0238] The receiving unit 2410 is configured to receive an audio synthesis request, wherein the audio synthesis request includes audio data to be synthesized.

[0239] The first generation unit 2420 is configured to generate multiple block latent vectors based on the audio data to be synthesized. Each block includes multiple consecutive audio frames in the target audio, which is audio synthesized based on an audio synthesis request. The order of the multiple blocks is based on the order of the audio frames included in the block, and the audio frames of two adjacent blocks are consecutive.

[0240] The second generation unit 2430 is configured to input multiple block latent vectors into an audio generation network in the order of the blocks. The audio generation network is used to perform the following processing: for each block in the target audio, while playing the preceding blocks of the block, the audio frame of the block is generated based on the block latent vector of the block and the block latent vector of the preceding blocks of the block, and the generated audio frame is added to the target playback queue for playback.

[0241] In some embodiments, the first generating unit 2420 is further configured to:

[0242] The audio data to be synthesized is input into the first neural network for feature extraction to obtain the audio frame features of each audio frame in the target audio.

[0243] Based on the correspondence between reference blocks and audio frames, block feature vectors are generated based on audio frame features;

[0244] The block feature vectors of each block are input into the second neural network for convolution to obtain the block latent vectors of each block.

[0245] In some embodiments, the first generating unit 2420 is further configured to:

[0246] For the first block in a plurality of blocks, obtain the first intermediate parameters generated in the second neural network by the second block preceding the first block;

[0247] Based on the block feature vectors of the first intermediate parameter, the first block, and the third block after the first block, the block latent vector of the first block is obtained by convolution using the second neural network.

[0248] In some embodiments, the first generating unit 2420 is further configured to:

[0249] When the hidden vector of the first block is generated, the hidden vector of the block is input into the audio generation network, wherein the step of inputting the hidden vector of the block into the audio generation network is before the generation of the third block after the first block in the plurality of blocks, or after the generation of the third block after the first block in the plurality of blocks.

[0250] In some embodiments, the first generating unit 2420 is further configured to:

[0251] Obtain the key weight matrix, value weight matrix, and query weight matrix of the block feature vector of the first block;

[0252] Based on the first intermediate parameter, the block feature vectors of the first block and the third block after the first block, the key weight matrix and the value weight matrix, determine the first key and the first value corresponding to the first block;

[0253] Based on the self-attention calculation results of the query weight matrix, the first key, the first value, and the block feature vector of the first block, the block latent vector of the first block is determined.

[0254] In some embodiments, the first generating unit 2420 is further configured to:

[0255] The block feature vector of the first block is normalized to obtain the normalized block feature vector;

[0256] The self-attention calculation result of the query weight matrix, the first key, the first value, and the normalized block feature vector is added to the block feature vector of the first block, and the block latent vector of the first block is determined based on the addition result.

[0257] In some embodiments, the first generating unit 2420 is further configured to:

[0258] Based on the self-attention calculation results of the query weight matrix, the first key, the first value, and the block feature vector of the first block, the intermediate hidden vector is determined.

[0259] Obtain the last vector from the implicit vector of the preceding block corresponding to the second block before the first block;

[0260] The block latent vector of the first block is determined based on the last latent vector and the middle latent vector.

[0261] In some embodiments, the first neural network, the second neural network, and the audio generation network are jointly trained through the following process:

[0262] Obtain the first sample audio data, the first sample target audio, the first sample audio features, and the first sample audio frame latent vector;

[0263] The first sample audio data is input into the first neural network for feature extraction, and the first predicted audio frame features of each audio frame in the first sample target audio frame and the first predicted audio features of the first sample target audio are obtained.

[0264] Using the second neural network, the first predicted block latent vector of each block is obtained based on the first predicted audio frame features corresponding to each block.

[0265] The latent vector of the first sample audio frame is divided into multiple blocks to obtain the first sample block latent vector of multiple blocks. The first sample block latent vector is then input into the audio generation network in the order of the blocks to obtain the first predicted target audio.

[0266] A first loss function is calculated based on the first predicted audio features and the first sample audio features; a second loss function is calculated based on the first predicted block latent vector and the first sample audio frame latent vector of each block; and a third loss function is calculated based on the first predicted target audio and the first sample target audio.

[0267] The first neural network, the second neural network, and the audio generation network are trained using the first loss function, the second loss function, and the third loss function.

[0268] In some embodiments, the first generating unit 2420 is further configured to:

[0269] The audio data to be synthesized is input into the third neural network for feature extraction to obtain the audio frame features of each audio frame in the target audio.

[0270] The audio frame features of each audio frame are input into the fourth neural network for convolution to obtain the audio frame latent vector corresponding to each audio frame.

[0271] Based on the correspondence between reference blocks and audio frames, block latent vectors are generated from the latent vectors of audio frames.

[0272] In some embodiments, the second generation unit 2430 is further configured to:

[0273] For the first segment in the target audio, based on the latent vector of the first segment, generate the audio frame of the first segment and add it to the target playback queue for playback;

[0274] For subsequent blocks after the first block, audio frames of the subsequent blocks are generated based on the block latent vectors of the subsequent blocks and the block latent vectors of the preceding blocks, and added to the target playback queue for playback. The generation of audio frames of the subsequent blocks and their addition to the target playback queue are performed simultaneously with the playback of the preceding blocks.

[0275] In some embodiments, the second generation unit 2430 is further configured to:

[0276] The target block latent vector of the segment is filled with the block latent vector of the segment using the block latent vector of the preceding segment to obtain the extended audio frame latent vector.

[0277] Based on the extended audio frame latent vector, segmented audio frames are generated, and the generated audio frames are added to the target playback queue for playback.

[0278] In some embodiments, the second generation unit 2430 is further configured to:

[0279] The padding latent vector is obtained from the block latent vector of the preceding block according to the predetermined padding length;

[0280] The padding latent vector is filled to the left of the target block latent vector of the block to obtain the augmented audio frame latent vector.

[0281] In some embodiments, the second generation unit 2430 is further configured to:

[0282] Input the block and its multiple preceding blocks into the semantic association prediction model to obtain the semantic association between the block and each preceding block.

[0283] Based on semantic relevance, the target block is determined among multiple preceding blocks of the block.

[0284] The target block latent vector of the segment is filled with the target block latent vector based on the target block to obtain the extended audio frame latent vector.

[0285] In some embodiments, the second generation unit 2430 is further configured to:

[0286] The weight of the target block is determined based on the semantic relevance between the target blocks and the blocks.

[0287] Based on the target block's latent vector and weight, the target block's latent vector is filled to obtain the augmented audio frame latent vector.

[0288] Referring to Figure 25, which is a partial structural block diagram of a terminal 110 implementing the audio processing method of this application embodiment, the terminal includes: a radio frequency (RF) circuit 2510, a memory 2515, an input unit 2530, a display unit 2540, a sensor 2550, an audio circuit 2560, a wireless fidelity (WiFi) module 2570, a processor 2580, and a power supply 2590, etc. Those skilled in the art will understand that the structure of the terminal 110 shown in Figure 25 does not constitute a limitation on a mobile phone or computer, and may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0289] The RF circuit 2510 is configured to receive and transmit signals during information transmission or calls. Specifically, it receives downlink information from the base station and processes it with the processor 2580; in addition, it transmits uplink data to the base station.

[0290] The memory 2515 is configured to store software programs and modules, and the processor 2580 executes various functional applications and data processing of the content terminal by running the software programs and modules stored in the memory 2515.

[0291] The input unit 2530 is configured to receive input numeric or character information and generate key signal inputs related to the settings and function control of the content terminal. Specifically, the input unit 2530 may include a touch panel 2531 and other input devices 2532.

[0292] Display unit 2540 is configured to display input or provided information, as well as various menus of the content terminal. Display unit 2540 may include display panel 2541.

[0293] Audio circuitry 2560, speaker 2561, and microphone 2562 provide an audio interface.

[0294] In this embodiment, the processor 2580 included in the terminal 110 can execute the audio processing method of the previous embodiment.

[0295] The terminal 110 in this application embodiment includes, but is not limited to, mobile phones, computers, intelligent voice interaction devices, smart home appliances, vehicle terminals, aircraft, etc. This invention embodiment can be applied to various scenarios, including but not limited to song generation, voice generation, etc.

[0296] Figure 26 is a partial structural block diagram of a server 140 implementing the audio processing method of this application embodiment. The server 140 can vary significantly due to different configurations or performance, and may include one or more central processing units (CPUs) 2622 (e.g., one or more processors) and memory 2632, and one or more storage media 2630 (e.g., one or more mass storage devices) storing application programs 2642 or data 2644. The memory 2632 and storage media 2630 may be temporary or persistent storage. The program stored in the storage media 2630 may include one or more modules (not shown in the figure), each module including a series of instruction operations on the server. Furthermore, the CPU 2622 may be configured to communicate with the storage media 2630 and execute the series of instruction operations in the storage media 2630 on the server.

[0297] Server 140 may also include one or more power supplies 2626, one or more wired or wireless network interfaces 2650, one or more input / output interfaces 2658, and / or one or more operating systems 2641, such as Windows Server™, Mac OS X™, Unix™, Linux™, FreeBSD™, etc.

[0298] The central processing unit 2622 in server 140 can be used to execute the audio processing method of the embodiments of this application.

[0299] This application also provides a computer-readable storage medium for storing program code for executing the audio processing methods of the foregoing embodiments.

[0300] This application also provides a computer program product, which includes a computer program. A processor of a computer device reads and executes the computer program, causing the computer device to perform the audio processing method described above.

[0301] The terms "first," "second," "third," "fourth," etc. (if present) in the specification and accompanying drawings of the embodiments of this application are used to distinguish similar content and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that embodiments of the present application described herein can be implemented, for example, in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "including," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that includes a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatuses.

[0302] It should be understood that in the embodiments of this application, "at least one (item)" refers to one or more, and "more than one" refers to two or more. "And / or" is used to describe the relationship between related content, indicating that three relationships can exist. For example, "A and / or B" can represent three cases: only A exists, only B exists, and both A and B exist simultaneously, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related content are in an "or" relationship. "At least one (item) of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one (item) of a, b, or c can represent: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be single or multiple.

[0303] It should be understood that in the description of the embodiments of this application, "multiple" means two or more, "greater than", "less than", "exceeding" etc. are understood to exclude the number itself, and "above", "below", "within" etc. are understood to include the number itself.

[0304] In this application embodiment, the terms "module" or "unit" refer to a computer program or part of a computer program that has a predetermined function and works with other related parts to achieve a predetermined goal, and can be implemented wholly or partially using software, hardware (such as processing circuitry or memory), or a combination thereof. Similarly, a processor (or multiple processors or memory) can be used to implement one or more modules or units. Furthermore, each module or unit can be part of an overall module or unit that includes the functionality of that module or unit.

[0305] In the embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces, or indirect coupling or communication connection between apparatuses or units, and may be electrical, mechanical, or other forms.

[0306] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0307] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0308] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solutions of the embodiments of this application, or the parts that contribute to related technologies, or all or part of the technical solutions, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0309] It should also be understood that the various implementation methods provided in this application can be combined arbitrarily to achieve different technical effects.

[0310] The above is a detailed description of the implementation methods of the embodiments of this application. However, the embodiments of this application are not limited to the above implementation methods. Those skilled in the art can make various equivalent modifications or substitutions without departing from the spirit of the embodiments of this application. All such equivalent modifications or substitutions are included within the scope defined by the claims of the embodiments of this application.

Claims

1. An audio processing method, wherein, The method is performed by an electronic device and includes: Receive an audio synthesis request, wherein the audio synthesis request includes audio data to be synthesized; Based on the audio data to be synthesized, a block latent vector is generated for multiple blocks, wherein each block includes multiple consecutive audio frames in the target audio, the target audio being the audio synthesized based on the audio synthesis request, and the multiple blocks are sorted according to the order of the audio frames included in the block, with the audio frames of two adjacent blocks being consecutive. The multiple block-level latent vectors are input into the audio generation network in the order of the blocks, wherein the audio generation network is used to perform the following processing: For each segment in the target audio, while playing the preceding segments, the audio frame of the segment is generated based on the segment latent vector of the segment and the segment latent vector of the preceding segments, and the generated audio frame is added to the target playback queue for playback.

2. The audio processing method according to claim 1, wherein, The step of generating multiple block latent vectors based on the audio data to be synthesized includes: The audio data to be synthesized is input into the first neural network for feature extraction to obtain the audio frame features of each audio frame in the target audio. Referring to the correspondence between the blocks and the audio frames, a block feature vector is generated based on the audio frame features; The feature vectors of each block are input into the second neural network for convolution to obtain the hidden vectors of each block.

3. The audio processing method according to claim 2, wherein, The step of inputting the block feature vectors of each block into the second neural network for convolution to obtain the block latent vectors of each block includes: For the first block among the plurality of blocks, obtain the first intermediate parameters generated in the second neural network by the second block preceding the first block; Based on the block feature vectors of the first intermediate parameter, the first block, and the third block after the first block, the block latent vector of the first block is obtained by convolution using the second neural network.

4. The audio processing method according to claim 3, wherein, The step of inputting multiple block-based latent vectors into the audio generation network in the order of the blocks includes: When the hidden vector of the first block is generated, the hidden vector of the block is input into the audio generation network, wherein the step of inputting the hidden vector of the block into the audio generation network is before the generation of the third block after the first block in the plurality of blocks, or after the generation of the third block after the first block in the plurality of blocks.

5. The audio processing method according to claim 3, wherein, The step of obtaining the block latent vector of the first block by convolution using the second neural network based on the block feature vector of the first intermediate parameter, the first block, and the third block after the first block includes: Obtain the key weight matrix, value weight matrix, and query weight matrix of the feature vector of the first segment; Based on the first intermediate parameter, the block feature vector of the first block and the third block after the first block, the key weight matrix and the value weight matrix, the first key and the first value corresponding to the first block are determined; Based on the self-attention calculation results of the query weight matrix, the first key, the first value, and the block feature vector of the first block, the block latent vector of the first block is determined.

6. The audio processing method according to claim 3, wherein, The method further includes: When the first block is the first block among the plurality of blocks, the block latent vector of the first block is obtained by convolution using the second neural network based on the block feature vectors of the first block and the third block after the first block.

7. The audio processing method according to claim 6, wherein, The step of determining the latent vector of the first block based on the self-attention calculation result of the query weight matrix, the first key, the first value, and the block feature vector of the first block includes: The block feature vector of the first block is normalized to obtain a normalized block feature vector. The self-attention calculation result of the query weight matrix, the first key, the first value, and the normalized block feature vector is added to the block feature vector of the first block, and the block latent vector of the first block is determined based on the addition result.

8. The audio processing method according to claim 6, wherein, The step of determining the latent vector of the first block based on the self-attention calculation result of the query weight matrix, the first key, the first value, and the block feature vector of the first block includes: Based on the self-attention calculation results of the query weight matrix, the first key, the first value, and the block feature vector of the first block, the intermediate hidden vector is determined; Obtain the last vector from the implicit vector of the preceding block corresponding to the second block before the first block; The block latent vector of the first block is determined based on the last vector and the intermediate latent vector.

9. The audio processing method according to claim 8, wherein, Determining the block latent vector of the first block based on the last vector and the intermediate latent vector includes: The last vector is concatenated to the left side of the middle implicit vector to obtain the first concatenated vector; Delete the subvector of a predetermined length starting from the last position of the first concatenated vector to obtain the second concatenated vector; The second concatenated vector is subjected to causal convolution to obtain the block latent vector of the first block.

10. The audio processing method according to any one of claims 2 to 9, wherein, The first neural network, the second neural network, and the audio generation network are jointly trained through the following process: Obtain the first sample audio data, the first sample target audio, the first sample audio features, and the first sample audio frame latent vector; The first sample audio data is input into the first neural network for feature extraction to obtain the first predicted audio frame feature of each audio frame in the first sample target audio frame, and the first predicted audio feature of the first sample target audio. Using a second neural network, the first predicted block latent vector of each block is obtained based on the features of the first predicted audio frame included in each block; The first sample audio frame latent vector is divided into multiple blocks to obtain the first sample block latent vector of multiple blocks. The first sample block latent vector is then input into the audio generation network in the order of the blocks to obtain the first predicted target audio. A first loss function is calculated based on the first predicted audio feature and the first sample audio feature; a second loss function is calculated based on the first predicted block latent vector of each block and the first sample audio frame latent vector; and a third loss function is calculated based on the first predicted target audio and the first sample target audio. The first neural network, the second neural network, and the audio generation network are trained using the first loss function, the second loss function, and the third loss function.

11. The audio processing method according to any one of claims 1 to 10, wherein, The step of generating multiple block latent vectors based on the audio data to be synthesized includes: The audio data to be synthesized is input into a third neural network for feature extraction to obtain the audio frame features of each audio frame in the target audio. The audio frame features of each audio frame are input into the fourth neural network for convolution to obtain the audio frame latent vector corresponding to each audio frame. Referring to the correspondence between the blocks and the audio frames, block latent vectors are generated based on the audio frame latent vectors.

12. The audio processing method according to any one of claims 1 to 11, wherein, For each segment in the target audio, while playing the preceding segments, the audio frame of the segment is generated based on the segment's latent vector and the latent vectors of the preceding segments, and the generated audio frame is added to the target playback queue for playback, including: For the first block in the target audio, based on the latent vector of the first block, the audio frame of the first block is generated and added to the target playback queue for playback; For subsequent blocks after the first block, the audio frames of the subsequent blocks are generated and added to the target playback queue based on the block latent vector of the subsequent block and the block latent vector of the preceding block of the subsequent block. The generation of the audio frames of the subsequent blocks and their addition to the target playback queue are performed simultaneously with the playback of the preceding blocks of the subsequent blocks.

13. The audio processing method according to any one of claims 1 to 12, wherein, The process of generating the audio frame of the segment based on the segment's latent vector and the segment's preceding latent vector, and adding the generated audio frame to the target playback queue for playback, includes: The target block latent vector of the block is filled with the block latent vector of the block using the block latent vector of the preceding block to obtain the extended audio frame latent vector. The audio frames of the blocks are generated based on the extended audio frame latent vectors, and the generated audio frames are added to the target playback queue for playback.

14. The audio processing method according to claim 13, wherein, The step of filling the target block latent vector of the segment with the block latent vector of the preceding segment to obtain the extended audio frame latent vector includes: The fill hidden vector is obtained from the block hidden vector of the preceding block according to the predetermined fill length; The padding latent vector is filled to the left of the target block latent vector of the block to obtain the augmented audio frame latent vector.

15. The audio processing method according to claim 13, wherein, The step of filling the target block latent vector of the segment with the block latent vector of the preceding segment to obtain the extended audio frame latent vector includes: The block and its multiple preceding blocks are input into the semantic association prediction model to obtain the semantic association between the block and each of its preceding blocks. Based on the semantic relevance, the target block is determined among the multiple preceding blocks of the block; The target block latent vector of the target block is filled with the block latent vector of the target block to obtain the extended audio frame latent vector.

16. The audio processing method according to claim 15, wherein, The process of filling the target block latent vector of the segment with the block latent vector based on the target segment to obtain the augmented audio frame latent vector includes: The weight of the target block is determined based on the semantic relevance between the target block and the block. Based on the target block's hidden vector and its weight, the target block's hidden vector is filled to obtain the extended audio frame hidden vector.

17. An audio processing apparatus, comprising: The receiving unit is configured to receive an audio synthesis request, wherein the audio synthesis request includes audio data to be synthesized; The first generation unit is configured to generate multiple block latent vectors based on the audio data to be synthesized, each block including multiple consecutive audio frames in the target audio, the target audio being audio synthesized based on the audio synthesis request, and the order of the multiple blocks being based on the order of the audio frames included in the block. The second generation unit is configured to input the multiple block latent vectors into the audio generation network in the order of the blocks, wherein the audio generation network is used to perform the following processing: for each block in the target audio, while playing the preceding blocks of the block, the audio frame of the block is generated based on the block latent vector of the block and the block latent vector of the preceding blocks of the block, and the generated audio frame is added to the target playback queue for playback.

18. An electronic device comprising a memory and a processor, wherein the memory stores a computer program, wherein, When the processor executes the computer program, it implements the audio processing method according to any one of claims 1 to 16.

19. A computer-readable storage medium storing a computer program, wherein, When the computer program is executed by a processor, it implements the audio processing method according to any one of claims 1 to 16.

20. A computer program product comprising a computer program that is read and executed by a processor of a computer device, causing the computer device to perform the audio processing method according to any one of claims 1 to 16.

Images