Methods, devices, computer equipment, and storage media for generating piano performance audio

Abstract

This invention relates to the field of artificial intelligence, and more particularly to a method, apparatus, computer device, and storage medium for generating piano performance audio. The method includes: acquiring piano performance event data; inputting the piano performance event data into an event encoder for event encoding to obtain piano performance event codes; performing code conversion on the piano performance event codes using an audio code conversion model to obtain Mel spectrum codes; decoding the Mel spectrum codes using a Mel spectrum generator to obtain Mel spectrum data; and inputting the Mel spectrum data into a vocoder for reconstruction to obtain piano performance audio. This invention transforms piano playing events into piano playing audio, taking into account the sound produced by the resonance between piano strings, making the obtained piano playing audio more vivid and realistic in terms of overtones.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence, and in particular to a method, apparatus, computer device, and storage medium for generating piano performance audio. Background Technology

[0002] With the development of electronic music, methods for producing music using electronic instruments or electronic music production techniques have become increasingly sophisticated. In electronic music, some timbres are not produced by playing real instruments, but rather synthesized using synthesizers.

[0003] Regarding piano tone synthesis, existing tone synthesizers typically employ the following two technologies:

[0004] 1. Record a real sample of each note, and then combine the recorded samples into a piano performance audio (e.g., various piano timbres in Kontakt, a timbre sampler tool);

[0005] 2. Using physical modeling, the operation of musical instruments is simulated through physical function algorithms (for example, Pianoteq, a plugin that allows users to customize piano sounds) to calculate the piano performance audio.

[0006] However, while individual timbres in the piano performance audio synthesized by Technique 1 closely resemble the timbre of a real piano, it fails to accurately simulate the resonance between piano strings, resulting in a lack of overtones in the synthesized audio. Secondly, although Technique 2 is superior to Technique 1 in simulating the resonance of individual piano parts and strings, each piano timbre is generated through simulation, leading to a discrepancy between the simulated and real piano timbres. Summary of the Invention

[0007] Therefore, it is necessary to provide a method, apparatus, computer equipment, and storage medium for generating piano performance audio, in order to solve the problem of the lack of overtones in piano performance audio generated by existing audio synthesis technology.

[0008] A method for generating piano performance audio, comprising:

[0009] Retrieve piano performance event data;

[0010] The piano performance event data is input into the event encoder for event encoding to obtain the piano performance event code.

[0011] The piano performance event encoding is converted using an audio code conversion model to obtain the Mel spectrum encoding corresponding to the piano performance event encoding.

[0012] The Mel spectrum code is decoded by a Mel spectrum generator to obtain Mel spectrum data corresponding to the piano performance event code;

[0013] The Mel spectrum data is input into a vocoder for reconstruction to obtain the piano performance audio.

[0014] A piano performance audio generation device includes:

[0015] The piano performance event data module is used to acquire piano performance event data.

[0016] The piano performance event encoding module is used to input the piano performance event data into the event encoder for event encoding to obtain the piano performance event code.

[0017] The Mel spectrum encoding module is used to convert the piano performance event encoding through an audio code conversion model to obtain the Mel spectrum encoding corresponding to the piano performance event encoding.

[0018] The Mel spectrum data module is used to decode the Mel spectrum code through the Mel spectrum generator to obtain Mel spectrum data corresponding to the piano performance event code;

[0019] The piano performance audio module is used to input the Mel spectrum data into a vocoder for reconstruction, thereby obtaining the piano performance audio.

[0020] A computer device includes a memory, a processor, and computer-readable instructions stored in the memory and executable on the processor, wherein the processor, when executing the computer-readable instructions, implements the above-described method for generating piano performance audio.

[0021] One or more readable storage media storing computer-readable instructions, which, when executed by one or more processors, cause the one or more processors to perform the piano performance audio generation method described above.

[0022] The aforementioned piano performance audio generation method, apparatus, computer equipment, and storage medium acquire piano performance event data; input the piano performance event data into an event encoder for event encoding to obtain a piano performance event code; perform code conversion on the piano performance event code using an audio code conversion model to obtain a Mel spectrum code corresponding to the piano performance event code; decode the Mel spectrum code using a Mel spectrum generator to obtain Mel spectrum data corresponding to the piano performance event code; and input the Mel spectrum data into a vocoder for restoration to obtain piano performance audio. This invention can convert piano playing events into piano playing audio. The piano playing audio obtained from piano playing event data considers all event data at every moment during the piano performance, that is, it considers the sound produced by the resonance between the piano strings, making the obtained piano playing audio more vivid and realistic in terms of overtones. Furthermore, by restoring the Mel spectrum using a vocoder, the generated piano playing audio is made closer to the sound of a real piano performance. Attached Figure Description

[0023] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the description of the embodiments of the present invention will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0024] Figure 1 This is a schematic diagram of an application environment for a piano performance audio generation method according to an embodiment of the present invention;

[0025] Figure 2 This is a flowchart illustrating a method for generating piano performance audio according to an embodiment of the present invention;

[0026] Figure 3 This is a schematic diagram of piano performance event data in one embodiment of the present invention;

[0027] Figure 4 This is a schematic diagram of a piano performance audio generation device according to an embodiment of the present invention;

[0028] Figure 5 This is a schematic diagram of a computer device according to an embodiment of the present invention. Detailed Implementation

[0029] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0030] The piano performance audio generation method provided in this embodiment can be applied to, for example, Figure 1 In this application environment, the client communicates with the server. Clients include, but are not limited to, various personal computers, laptops, smartphones, tablets, and portable wearable devices. The server can be implemented using a standalone server or a server cluster consisting of multiple servers.

[0031] In one embodiment, such as Figure 2 As shown, a method for generating piano performance audio is provided, which can be applied to... Figure 1 Taking the server-side as an example, the explanation includes the following steps:

[0032] S10, Obtain piano performance event data.

[0033] Understandably, piano performance event data refers to event data during piano performance. This event data is used to record played notes, special musical techniques, etc., including key press events, sustain pedal events, soft pedal events, and so on. For example... Figure 3 The diagram illustrates piano performance event data. Events in the data are numbered 0-89, where 0 and 1 represent Una-Corda (soft pedal event) and Pedal (pedal) events, respectively; 2 to 89 represent the events of the 88 keys on the piano. The timeline during piano performance can be represented as (t1, t2, t3, ..., t...). i ).

[0034] S20. Input the piano performance event data into the event encoder for event encoding to obtain the piano performance event code.

[0035] Understandably, an event encoder is an encoder that encodes piano performance event data. Piano performance event encoding refers to the encoding of piano performance events, which can be composed of several characters. Specifically, by encoding the event data at each moment in the piano performance event data using an event encoder, the encoding for each moment is obtained. Then, the encodings for each moment are combined to obtain the piano performance event encoding. For example, at a certain moment t... i The encoding can be achieved through array r i This indicates that the piano performance event is encoded as (r1, r2, r3, ..., r... i ). In r i In the expression, when a position does not contain an event, that position is set to "0"; when a position contains an event, that position is set to a non-zero value, such as a non-zero value representing the key press intensity. Figure 3 In the code, t3 can be represented as:

[0036]

[0037] In the expression for r3 above, The value "127" indicates the intensity of the event; "1" indicates a Pedal event; and "0" indicates that the position does not contain an event.

[0038] S30. The piano performance event encoding is converted using an audio code conversion model to obtain the Mel spectrum encoding corresponding to the piano performance event encoding.

[0039] Understandably, an audio code-transformation model is a trained neural network model used to convert the encoding of piano performance events into Mel-spectral encoding. This audio code-transformation model can be a GPT2 model (language model), which is a type of Transformer (self-attention network) model. Code-transformation refers to the process of converting the encoding of piano performance events into Mel-spectral encoding. Mel-spectral encoding refers to the encoding of audio obtained through the conversion of piano performance event encoding.

[0040] S40. The Mel spectrum code is decoded by the Mel spectrum generator to obtain the Mel spectrum data corresponding to the piano performance event code.

[0041] Understandably, a Mel spectrum generator is a trained neural network model used to decode Mel spectrum encoding, converting it into Mel spectrum data. For example, a Mel spectrum generator could be a VqGAN (Vector Quantised General Adversarial Network, an image generation model) model, which generates corresponding images based on input data. Decoding refers to the process of converting Mel spectrum encoding into Mel spectrum data. Mel spectrum data can be a Mel spectrogram. Specifically, this Mel spectrum data could be the Mel spectrogram of music generated when a piano is played.

[0042] S50. Input the Mel spectrum data into the vocoder for restoration to obtain the piano performance audio.

[0043] Understandably, a vocoder is used to convert Mel spectrum data into audio data. This vocoder can be a FreGAN vocoder, which uses an RCG (resolution-connected generator) and resolution-wise discriminators to learn different frequency bands on the spectrum, enabling it to output high-quality audio data.

[0044] In steps S10-S50, piano performance event data is acquired; the piano performance event data is input into an event encoder for event encoding to obtain a piano performance event code; the piano performance event code is converted using an audio code conversion model to obtain a Mel spectrum code corresponding to the piano performance event code; the Mel spectrum code is decoded using a Mel spectrum generator to obtain Mel spectrum data corresponding to the piano performance event code; the Mel spectrum data is input into a vocoder for restoration to obtain piano performance audio. In this embodiment, piano playing events can be converted into piano playing audio. The piano playing audio obtained from the piano playing event data considers all event data at every moment during the piano performance, that is, it considers the sound produced by the resonance between the piano strings, making the obtained piano playing audio more vivid and realistic in terms of overtones. Furthermore, the Mel spectrum is restored using a vocoder, making the generated piano playing audio closer to the sound of a real piano performance.

[0045] Optionally, before step S30, that is, before the code conversion of the piano performance event encoding through the audio code conversion model to obtain the Mel spectrum encoding corresponding to the piano performance event encoding, the following steps are included:

[0046] S301. Obtain a set of piano performance audio samples;

[0047] S302. Encode the piano performance audio sample set to obtain piano performance event sample code and first Mel spectrum code;

[0048] S303. Input the encoded piano performance event sample into the initial audio code conversion model to obtain the target Mel spectrum code;

[0049] S304. Determine the Mel spectrum loss value based on the first Mel spectrum code and the target Mel spectrum code;

[0050] S305. When the Mel spectrum loss value does not reach the preset convergence condition, iteratively update the initial parameters of the initial audio code conversion model until the Mel spectrum loss value reaches the preset convergence condition, and then use the converged initial audio code conversion model as the audio code conversion model.

[0051] Understandably, the piano performance audio sample set includes several piano performance audio samples and several piano performance event samples corresponding to the piano performance audio samples. Encoding the piano performance audio sample set refers to obtaining piano performance event samples and the first piano performance audio samples corresponding to those event samples from the set, and encoding both the event samples and the first audio samples to obtain the piano performance event sample code and the first Mel-spectral code. An initial audio code conversion model is used to train and learn the first Mel-spectral code and the piano performance event sample code, ultimately generating an audio code conversion model that can convert the piano performance event sample code into Mel-spectral codes. The initial audio code conversion model can be a GPT2 model (language model), a type of Transformer (self-attention network) model that can predict output results based on input data. The initial audio code conversion model has relatively weak code conversion capabilities and requires continuous iterative updates to eventually obtain an audio code conversion model with strong conversion capabilities that meets the convergence conditions. The target Mel-spectral code is the Mel-spectral code obtained by converting the piano performance event sample code using the initial audio code conversion model. The Mel-spectral loss value refers to the loss between the first Mel-spectral code and the target Mel-spectral code. A smaller Mel-spectral loss value indicates that the target Mel-spectral code is closer to the first Mel-spectral code, representing a stronger code conversion capability of the initial audio code conversion model. Preferably, the Mel-spectral loss value can be obtained through a loss function based on cross-entropy.

[0052] In this embodiment, the initial audio code conversion model is trained based on a set of piano performance audio samples, so that the resulting audio code conversion model has a strong code conversion capability and improves the conversion accuracy of the initial audio code conversion model.

[0053] Optionally, in step S303, i.e., inputting the encoded piano performance event sample into the initial audio code conversion model to obtain the target Mel spectrum code, the following steps are included:

[0054] S3031. Combine the piano performance event sample code and the first Mel spectrum code to obtain the combined code;

[0055] S3032. The concatenated code is trained by misalignment through the initial audio code conversion model to obtain the target Mel spectrum code.

[0056] Understandably, the piano performance event sample code can be a code consisting of several characters. The first Mel spectrum code can also be a code consisting of several characters. The concatenated code is the code obtained by combining the piano performance event sample code and the first Mel spectrum code. For example, the concatenated code W = {w1, w2, ..., w...} n Misaligned training refers to model training that utilizes the misaligned relationship between input and output data. For example, input data W... a ={w1, w2, ..., w n-1 Output data W b ={w2, w3, ..., w n By using misaligned training, the initial audio code-changing model is made more capable of using preceding information to predict subsequent information, thus improving the predictive ability of the audio code-changing model.

[0057] Optionally, in step S302, i.e., encoding the piano performance audio samples to obtain piano performance event sample encoding and first Mel spectrum encoding, the following steps are included:

[0058] S3021. Obtain piano performance event samples and a first piano performance audio sample corresponding to the piano performance event samples from the piano performance audio sample set;

[0059] S3022. Input the first piano performance audio sample into the Mel spectrum encoder to obtain the first Mel spectrum code;

[0060] S3023. Input the piano performance event sample into the event encoder to obtain the piano performance event sample code.

[0061] Understandably, a Mel spectrum encoder refers to an encoder that encodes a first piano performance audio sample. Preferably, the first piano performance audio sample can be either audio data or Mel spectrum data. When the first piano performance audio sample is Mel spectrum data, it can be directly encoded using a Mel spectrum encoder. When the first piano performance audio sample is audio data, the audio data needs to be converted to Mel spectrum data before encoding. The piano performance audio sample set includes several piano performance audio samples and several piano performance event samples corresponding to the piano performance audio samples. Obtaining the piano performance audio samples and the corresponding piano performance event samples from the piano performance audio sample set, and encoding them separately, improves the processing speed and accuracy of the sample data, and allows for the rapid acquisition of the first Mel spectrum code and the piano performance event sample code.

[0062] Optionally, before step S40, that is, before decoding the Mel spectrum code using the Mel spectrum generator to obtain the Mel spectrum data corresponding to the piano performance event code, the following steps are included:

[0063] S401. Obtain the second piano performance audio sample;

[0064] S402. Input the second piano performance audio sample into the Mel spectrum encoder to obtain the second Mel spectrum code;

[0065] S403. Quantize the second Mel spectrum code to obtain the quantized Mel spectrum code;

[0066] S404. Input the quantized Mel spectrum encoding into the initial Mel spectrum generator for decoding to obtain the target Mel spectrum data;

[0067] S405. Determine the coding loss value based on the second piano performance audio sample, the second Mel spectrum code, the quantized Mel spectrum code, and the target Mel spectrum data;

[0068] S406. When the coding loss value does not reach the preset convergence condition, iteratively update the initial parameters of the initial Mel spectrum generator until the coding loss value reaches the preset convergence condition, and then use the converged initial Mel spectrum generator as the Mel spectrum generator.

[0069] Understandably, the second piano performance audio sample refers to the audio data obtained from the set of piano playing video samples. The second Mel spectrum encoding refers to the Mel spectrum encoding obtained by encoding the second piano performance audio sample using a Mel spectrum encoder. Quantization processing refers to the process of querying the Mel spectrum encoding corresponding to the second Mel spectrum encoding in the codebook using a quantization function. The codebook is an existing code library containing several Mel spectrum encodings. Quantized Mel spectrum encoding refers to the Mel spectrum encoding corresponding to the second Mel spectrum encoding found in the codebook using a quantization function. The initial Mel spectrum generator can be a VqGAN model (Vector Quantized General Adversarial Network, image generation model), which can generate corresponding images based on input data. The initial Mel spectrum generator can decode the quantized Mel spectrum encoding into Mel spectrum data, i.e., the target Mel spectrum data. Furthermore, based on the second piano performance audio sample, the second Mel spectrum encoding, the quantized Mel spectrum encoding, and the target Mel spectrum data, the encoding loss value of the initial Mel spectrum generator is determined, and the initial parameters of the initial Mel spectrum generator are iteratively updated based on this encoding loss value until the initial Mel spectrum generator meets the convergence condition, thus obtaining the Mel spectrum generator.

[0070] In this embodiment, the initial Mel spectrum generator is trained based on the second piano performance audio sample, so that the obtained Mel spectrum generator has a strong generation capability and improves the accuracy of the Mel spectrum generator.

[0071] Optionally, in step S405, namely, determining the coding loss value based on the second piano performance audio sample, the second Mel-spectral encoding, the quantized Mel-spectral encoding, and the target Mel-spectral data, includes:

[0072] S4051. Determine a first loss value based on the second piano performance audio sample and the target Mel spectrum data;

[0073] S4052. Determine the second loss value based on the second Mel spectrum encoding and the quantized Mel spectrum encoding;

[0074] S4053. Determine the encoding loss value based on the first loss value and the second loss value.

[0075] Understandably, the first loss value refers to the loss between the second piano performance audio sample and the target Mel-spectral data, which can be calculated by a discriminator. Similarly, the second loss value refers to the loss between the second Mel-spectral encoding and the quantized Mel-spectral encoding. The second loss value can be calculated based on Euclidean distance. Furthermore, the sum of the first and second loss values ​​can be determined as the encoding loss value.

[0076] In this embodiment, the encoding loss value considers the loss values ​​between the second piano performance audio sample and the target Mel spectrum data, as well as between the second Mel spectrum encoding and the quantized Mel spectrum encoding. This allows the initial Mel spectrum generator to fully learn the information between the real Mel spectrum and the codebook library during model training, making the Mel spectrum data generated by the Mel spectrum generator more accurate.

[0077] It should be understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0078] In one embodiment, a piano performance audio generation device is provided, which corresponds one-to-one with the piano performance audio generation method described in the above embodiments. For example... Figure 4 As shown, the piano performance audio generation device includes a piano performance event data module 10, a piano performance event encoding module 20, a Mel spectrum encoding module 30, a Mel spectrum data module 40, and a piano performance audio module 50. Detailed descriptions of each functional module are as follows:

[0079] Piano performance event data module 10 is used to acquire piano performance event data;

[0080] The piano performance event encoding module 20 is used to input the piano performance event data into the event encoder for event encoding to obtain the piano performance event code.

[0081] Mel spectrum encoding module 30 is used to perform code conversion on the piano performance event encoding through an audio code conversion model to obtain Mel spectrum encoding corresponding to the piano performance event encoding;

[0082] Mel spectrum data module 40 is used to decode the Mel spectrum code through a Mel spectrum generator to obtain Mel spectrum data corresponding to the piano performance event code;

[0083] The piano performance audio module 50 is used to input the Mel spectrum data into a vocoder for reconstruction to obtain the piano performance audio.

[0084] Optionally, the piano performance audio generation device further includes:

[0085] The piano performance audio sample set module is used to obtain a piano performance audio sample set;

[0086] The encoding processing module is used to encode the piano performance audio sample set to obtain piano performance event sample encoding and first Mel spectrum encoding;

[0087] The target Mel spectrum encoding module is used to input the encoded piano performance event samples into the initial audio code conversion model to obtain the target Mel spectrum encoding.

[0088] The Mel spectrum loss module is used to determine the Mel spectrum loss value based on the first Mel spectrum code and the target Mel spectrum code.

[0089] The audio code conversion model module is used to iteratively update the initial parameters of the initial audio code conversion model when the Mel spectrum loss value does not reach the preset convergence condition, until the Mel spectrum loss value reaches the preset convergence condition, and then use the converged initial audio code conversion model as the audio code conversion model.

[0090] Optionally, the target Mel spectrum coding module includes:

[0091] The concatenation coding unit is used to concatenate the piano performance event sample code and the first Mel spectrum code to obtain the concatenation code;

[0092] The target Mel-spectrum coding unit is used to perform misalignment training on the concatenated coding through the initial audio code conversion model to obtain the target Mel-spectrum coding.

[0093] Optionally, the encoding processing module includes:

[0094] The sample data acquisition unit is used to acquire piano performance event samples and first piano performance audio samples corresponding to the piano performance event samples from the piano performance audio sample set.

[0095] The first Mel spectrum encoding unit is used to input the first piano performance audio sample into the Mel spectrum encoder to obtain the first Mel spectrum code;

[0096] A piano performance event sample encoding unit is used to input the piano performance event sample into the event encoder to obtain the piano performance event sample encoding.

[0097] Optionally, the piano performance audio generation device further includes:

[0098] The second piano performance audio sample module is used to obtain second piano performance audio samples.

[0099] The second Mel spectrum encoding module is used to input the second piano performance audio sample into the Mel spectrum encoder to obtain the second Mel spectrum code;

[0100] The quantization Mel spectrum coding module is used to quantize the second Mel spectrum code to obtain the quantized Mel spectrum code;

[0101] The target Mel spectrum data module is used to input the quantized Mel spectrum encoding into the initial Mel spectrum generator for decoding processing to obtain the target Mel spectrum data;

[0102] The coding loss module is used to determine the coding loss value based on the second piano performance audio sample, the second Mel spectrum coding, the quantized Mel spectrum coding, and the target Mel spectrum data.

[0103] The Mel spectrum generator module is used to iteratively update the initial parameters of the initial Mel spectrum generator when the coding loss value does not reach the preset convergence condition, until the coding loss value reaches the preset convergence condition, and then use the converged initial Mel spectrum generator as the Mel spectrum generator.

[0104] Optionally, the encoding loss value module includes:

[0105] The first loss value unit is used to determine a first loss value based on the second piano performance audio sample and the target Mel spectrum data;

[0106] The second loss value unit is used to determine a second loss value based on the second Mel spectrum coding and the quantized Mel spectrum coding;

[0107] The encoding loss value unit is used to determine the encoding loss value based on the first loss value and the second loss value.

[0108] Specific limitations regarding the piano performance audio generation device can be found in the limitations of the piano performance audio generation method described above, and will not be repeated here. Each module in the aforementioned piano performance audio generation device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device in hardware form, or stored in the memory of a computer device in software form, so that the processor can call and execute the corresponding operations of each module.

[0109] In one embodiment, a computer device is provided, which may be a terminal, and its internal structure diagram may be as follows: Figure 5 As shown, the computer device includes a processor, memory, network interface, display screen, and input device connected via a system bus. The processor provides computing and control capabilities. The memory includes a readable storage medium and internal memory. The non-volatile storage medium stores an operating system and computer-readable instructions. The internal memory provides an environment for the operation of the operating system and computer-readable instructions in the readable storage medium. The network interface is used to communicate with an external server via a network connection. When the computer-readable instructions are executed by the processor, they implement a method for generating piano performance audio. The readable storage medium provided in this embodiment includes both non-volatile and volatile readable storage media.

[0110] In one embodiment, a computer device is provided, including a memory, a processor, and computer-readable instructions stored in the memory and executable on the processor, wherein the processor performs the following steps when executing the computer-readable instructions:

[0111] Retrieve piano performance event data;

[0112] The piano performance event data is input into the event encoder for event encoding to obtain the piano performance event code.

[0113] The piano performance event encoding is converted using an audio code conversion model to obtain the Mel spectrum encoding corresponding to the piano performance event encoding.

[0114] The Mel spectrum code is decoded by a Mel spectrum generator to obtain Mel spectrum data corresponding to the piano performance event code;

[0115] The Mel spectrum data is input into a vocoder for reconstruction to obtain the piano performance audio.

[0116] In one embodiment, one or more computer-readable storage media storing computer-readable instructions are provided. The readable storage media provided in this embodiment include non-volatile readable storage media and volatile readable storage media. The readable storage media stores computer-readable instructions, which, when executed by one or more processors, perform the following steps:

[0117] Retrieve piano performance event data;

[0118] The piano performance event data is input into the event encoder for event encoding to obtain the piano performance event code.

[0119] The piano performance event encoding is converted using an audio code conversion model to obtain the Mel spectrum encoding corresponding to the piano performance event encoding.

[0120] The Mel spectrum code is decoded by a Mel spectrum generator to obtain Mel spectrum data corresponding to the piano performance event code;

[0121] The Mel spectrum data is input into a vocoder for reconstruction to obtain the piano performance audio.

[0122] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by instructing related hardware with computer-readable instructions. These computer-readable instructions can be stored in a non-volatile readable storage medium or a volatile readable storage medium. When executed, these computer-readable instructions can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in a variety of forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), RAMbus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0123] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is used as an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above.

[0124] The above-described embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be included within the protection scope of the present invention.

Claims

1. A method for generating piano performance audio, characterized in that, include: Retrieve piano performance event data; The piano performance event data is input into the event encoder for event encoding to obtain the piano performance event code. The event encoder refers to an encoder that encodes piano performance event data. The piano performance event encoding is the encoding of a piano performance event, which consists of several characters. Specifically, the event encoder encodes the event data at each moment in the piano performance event data to obtain the encoding for each moment. These encodings are then combined to obtain the piano performance event encoding. The piano performance event encoding is (r1, r2, r3, ..., ri). In the expression for ri, if a position does not contain an event, that position is set to "0"; if a position contains an event, that position is set to a non-zero value. The piano performance event encoding is converted into Mel spectral encoding by an audio code conversion model to obtain Mel spectral encoding corresponding to the piano performance event encoding. The audio code conversion model is a self-attention network model. Code conversion refers to the process of converting the piano performance event encoding into Mel spectral encoding. Mel spectral encoding refers to the encoding of audio obtained by converting the piano performance event encoding. The Mel spectrum code is decoded by a Mel spectrum generator to obtain Mel spectrum data corresponding to the piano performance event code; The Mel spectrum data is input into a vocoder for reconstruction to obtain the piano performance audio. Before performing code conversion on the piano performance event encoding using the audio code conversion model to obtain the Mel spectrum encoding corresponding to the piano performance event encoding, the following steps are included: Obtain a set of piano performance audio samples; The piano performance audio sample set is encoded to obtain piano performance event sample code and first Mel spectrum code; The piano performance event samples are encoded and input into the initial audio code conversion model to obtain the target Mel spectrum code; The Mel spectrum loss value is determined based on the first Mel spectrum code and the target Mel spectrum code; When the Mel spectrum loss value does not reach the preset convergence condition, the initial parameters of the initial audio code conversion model are iteratively updated until the Mel spectrum loss value reaches the preset convergence condition. Then, the converged initial audio code conversion model is used as the audio code conversion model. The step of inputting the encoded piano performance event samples into the initial audio code conversion model to obtain the target Mel spectrum code includes: The piano performance event sample code and the first Mel spectrum code are combined to obtain the combined code; The target Mel-spectral code is obtained by performing misalignment training on the concatenated code using the initial audio code conversion model.

2. The piano performance audio generation method as described in claim 1, characterized in that, The encoding process of the piano performance audio samples to obtain piano performance event sample encoding and first Mel spectrum encoding includes: Obtain piano performance event samples and first piano performance audio samples corresponding to the piano performance event samples from the piano performance audio sample set; The first piano performance audio sample is input into the Mel spectrum encoder to obtain the first Mel spectrum code; The piano performance event sample is input into the event encoder to obtain the piano performance event sample code.

3. The piano performance audio generation method as described in claim 1, characterized in that, Before decoding the Mel spectrum encoding using a Mel spectrum generator to obtain the Mel spectrum data corresponding to the piano performance event encoding, the process includes: Obtain audio samples of the second piano performance; The second piano performance audio sample is input into the Mel spectrum encoder to obtain the second Mel spectrum code; The second Mel spectrum code is quantized to obtain the quantized Mel spectrum code; The quantized Mel spectrum encoding is input into the initial Mel spectrum generator for decoding to obtain the target Mel spectrum data; The coding loss value is determined based on the second piano performance audio sample, the second Mel spectrum encoding, the quantized Mel spectrum encoding, and the target Mel spectrum data; When the coding loss value does not reach the preset convergence condition, the initial parameters of the initial Mel spectrum generator are iteratively updated until the coding loss value reaches the preset convergence condition. Then, the converged initial Mel spectrum generator is used as the Mel spectrum generator.

4. The piano performance audio generation method as described in claim 3, characterized in that, The step of determining the coding loss value based on the second piano performance audio sample, the second Mel-spectral encoding, the quantized Mel-spectral encoding, and the target Mel-spectral data includes: A first loss value is determined based on the second piano performance audio sample and the target Mel spectrum data; The second loss value is determined based on the second Mel spectrum encoding and the quantized Mel spectrum encoding; The encoding loss value is determined based on the first loss value and the second loss value.

5. A piano performance audio generation device, characterized in that, include: The piano performance event data module is used to acquire piano performance event data. The piano performance event encoding module is used to input the piano performance event data into an event encoder for event encoding to obtain the piano performance event code. The event encoder is an encoder that encodes the piano performance event data, and the piano performance event code is the encoding of the piano performance event, which is a code composed of several characters. Specifically, the event encoder encodes the event data at each moment in the piano performance event data to obtain the code for each moment. The codes for each moment are then combined to obtain the piano performance event code. The piano performance event code is (r1, r2, r3, ..., ri). In the expression for ri, when a position does not contain an event, that position is set to "0"; when a position contains an event, that position is set to a non-zero value. The Mel spectrum encoding module is used to convert the piano performance event encoding into Mel spectrum encoding through an audio code conversion model to obtain Mel spectrum encoding corresponding to the piano performance event encoding. The audio code conversion model is a self-attention network model. Code conversion refers to the process of converting the piano performance event encoding into Mel spectrum encoding. Mel spectrum encoding refers to the encoding of audio obtained by converting the piano performance event encoding. The Mel spectrum data module is used to decode the Mel spectrum code through the Mel spectrum generator to obtain Mel spectrum data corresponding to the piano performance event code; The piano performance audio module is used to input the Mel spectrum data into a vocoder for reconstruction to obtain the piano performance audio. The piano performance audio generation device further includes: The piano performance audio sample set module is used to obtain a piano performance audio sample set; The encoding processing module is used to encode the piano performance audio sample set to obtain piano performance event sample encoding and first Mel spectrum encoding; The target Mel spectrum encoding module is used to input the encoded piano performance event samples into the initial audio code conversion model to obtain the target Mel spectrum encoding. The Mel spectrum loss module is used to determine the Mel spectrum loss value based on the first Mel spectrum code and the target Mel spectrum code. The audio code conversion model module is used to iteratively update the initial parameters of the initial audio code conversion model when the Mel spectrum loss value does not reach the preset convergence condition, until the Mel spectrum loss value reaches the preset convergence condition, and then use the converged initial audio code conversion model as the audio code conversion model. The target Mel spectrum coding module includes: The concatenation coding unit is used to concatenate the piano performance event sample code and the first Mel spectrum code to obtain the concatenation code; The target Mel-spectrum coding unit is used to perform misalignment training on the concatenated coding through the initial audio code conversion model to obtain the target Mel-spectrum coding.

6. A computer device comprising a memory, a processor, and computer-readable instructions stored in the memory and executable on the processor, characterized in that, When the processor executes the computer-readable instructions, it implements the piano performance audio generation method as described in any one of claims 1 to 4.

7. One or more readable storage media storing computer-readable instructions, characterized in that, When the computer-readable instructions are executed by one or more processors, the one or more processors cause the piano performance audio generation method as described in any one of claims 1 to 4 to be performed.

Images