Piano playing video generation method and device, computer device and storage medium

Abstract

The present application relates to the field of voice analysis, and particularly relates to a piano playing video generation method and device, computer equipment and storage medium. The method comprises: obtaining audio stream data; inputting the audio stream data into an audio encoder for encoding processing to obtain audio encoding; performing code conversion on the audio encoding through a piano video code conversion model to obtain a piano video code thin sequence; performing decoding processing on the piano video code thin sequence through a piano video code thin decoder to obtain piano video stream data; the piano video stream data refers to a video stream of a hand playing music corresponding to the audio encoding on a piano; and the piano video stream data and the audio stream data are merged to obtain a piano playing video. The present application converts the audio stream data into video stream data with a picture of a hand playing a piano, and finally generates a video containing both audio and a hand playing the audio on a piano, so that the video effect and quality are better, and user experience is improved.

Description

Technical Field

[0001] This invention relates to the field of speech analysis, and in particular to a method, apparatus, computer device, and storage medium for generating piano playing videos. Background Technology

[0002] With the development of technology and the entertainment industry, people's entertainment methods have shifted from simple text reading and image display to multimedia such as audio and video. Especially with the rise of short videos, the demand for music videos is increasing. Currently, music is usually obtained through music software, but this often only provides the audio of a piano being played, not a complete video showing a person playing the piano. Furthermore, even when corresponding piano videos are generated for some piano music, these videos are usually generated by using speech to drive the virtual human's movements, resulting in less smooth and lower-quality videos that negatively impact the user experience. Summary of the Invention

[0003] Therefore, it is necessary to provide a method, apparatus, computer equipment, and storage medium for generating piano playing videos to address the aforementioned technical problems and solve the issue of poor video quality in existing piano playing video generation technologies.

[0004] A method for generating piano playing videos includes:

[0005] Acquire audio stream data;

[0006] The audio stream data is input into an audio encoder for encoding processing to obtain the audio code;

[0007] The audio encoding is converted using a piano video code conversion model to obtain a piano video codebook sequence corresponding to the audio encoding.

[0008] The piano video codebook sequence is decoded by a piano video codebook decoder to obtain piano video stream data; the piano video stream data refers to a video stream of a person playing music on a piano that corresponds to the audio encoding.

[0009] The piano video stream data and the audio stream data are merged to obtain a piano playing video.

[0010] A piano playing video generation device includes:

[0011] The audio stream data module is used to acquire audio stream data;

[0012] The audio encoding module is used to input the audio stream data into the audio encoder for encoding processing to obtain the audio encoding;

[0013] The piano video codebook sequence module is used to convert the audio encoding using a piano video code conversion model to obtain a piano video codebook sequence corresponding to the audio encoding.

[0014] The piano video stream data module is used to decode the piano video codebook sequence through a piano video codebook decoder to obtain piano video stream data; the piano video stream data refers to a video stream of a person playing music on a piano that corresponds to the audio encoding.

[0015] The piano playing video module is used to merge the piano video stream data and the audio stream data to obtain a piano playing video.

[0016] A computer device includes a memory, a processor, and computer-readable instructions stored in the memory and executable on the processor, wherein the processor executes the computer-readable instructions to implement the above-described piano playing video generation method.

[0017] One or more readable storage media storing computer-readable instructions, which, when executed by one or more processors, cause the one or more processors to perform the piano playing video generation method described above.

[0018] The aforementioned piano playing video generation method, apparatus, computer equipment, and storage medium acquire audio stream data; input the audio stream data into an audio encoder for encoding processing to obtain audio encoding; perform code conversion on the audio encoding using a piano video code conversion model to obtain a piano video codebook sequence corresponding to the audio encoding; perform decoding processing on the piano video codebook sequence using a piano video codebook decoder to obtain piano video stream data; and merge the piano video stream data and the audio stream data to obtain a piano playing video. This invention encodes audio data and converts it into a piano video codebook sequence, then decodes the piano video codebook sequence into piano video stream data (this piano video stream data refers to a video stream of a person playing music corresponding to the audio encoding on a piano). The piano video stream data and audio stream data are then merged to obtain a piano playing video, ultimately converting audio stream data into a piano playing video that includes both audio and the sound of a person playing that audio on a piano. The piano playing video generated from the audio stream data is smoother, resulting in better video effects and quality, and improved user experience. Attached Figure Description

[0019] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the description of the embodiments of the present invention will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0020] Figure 1 This is a schematic diagram of an application environment for a piano playing video generation method according to an embodiment of the present invention;

[0021] Figure 2 This is a flowchart illustrating a method for generating a piano playing video according to an embodiment of the present invention;

[0022] Figure 3 This is a schematic diagram of a piano playing video generation device according to an embodiment of the present invention;

[0023] Figure 4 This is a schematic diagram of a computer device according to an embodiment of the present invention. Detailed Implementation

[0024] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0025] The piano playing video generation method provided in this embodiment can be applied to, for example... Figure 1 In this application environment, the client communicates with the server. Clients include, but are not limited to, various personal computers, laptops, smartphones, tablets, and portable wearable devices. The server can be implemented using a standalone server or a server cluster consisting of multiple servers.

[0026] In one embodiment, such as Figure 2 As shown, a method for generating piano playing videos is provided, which can be applied to... Figure 1 Taking the server-side as an example, the explanation includes the following steps:

[0027] S10, Obtain audio stream data.

[0028] As you can understand, audio streaming data refers to audio data. For example, music data obtained from a music platform.

[0029] S20. Input the audio stream data into the audio encoder for encoding processing to obtain the audio code.

[0030] As we understand it, an audio encoder is an encoder that encodes audio data. Audio encoding refers to the encoding of audio data, which can be an encoding composed of several characters. Specifically, before inputting audio stream data into the audio encoder, the audio stream data can be converted into Mel-spectrum data, and the Mel-spectrum characteristics of the audio stream data can be obtained based on the Mel-spectrum data. These Mel-spectrum characteristics are then input into the audio encoder for encoding processing.

[0031] S30. The audio encoding is converted using a piano video code conversion model to obtain a piano video codebook sequence corresponding to the audio encoding.

[0032] Understandably, a piano video code-transfer model is a trained neural network model used to convert audio encoding into a sequence of codebooks representing a person playing the piano. This model can be a GPT2 model (a language model), which is a type of Transformer (self-attention network) model. Code-transfer refers to the process of converting audio encoding into a sequence of codebooks for a piano video, that is, converting audio encoding into video encoding. The piano video codebook sequence is the video encoding obtained through code-transferring audio encoding. This sequence refers to the codebook sequence of a video showing a person playing music on a piano, corresponding to the audio encoding.

[0033] S40. The piano video codebook sequence is decoded by a piano video codebook decoder to obtain piano video stream data; the piano video stream data refers to the video stream of a person playing music on a piano that corresponds to the audio encoding.

[0034] Understandably, a piano video codebook decoder is a trained neural network model used to decode piano video codebook sequences into piano video stream data. For example, a piano video codebook decoder could be a VqGAN model (Vector Quantized General Adversarial Network, an image generation model), which can generate corresponding images based on input data. Decoding refers to the process of decoding the piano video codebook sequence into piano video stream data. Piano video stream data refers to data containing several video frames. Specifically, the piano video stream data refers to a video stream of a person playing music corresponding to the audio encoding on a piano.

[0035] S50. Merge the piano video stream data and the audio stream data to obtain a piano playing video.

[0036] Understandably, after obtaining the video stream of a person playing music on a piano that corresponds to the audio encoding, the video stream and audio stream data of the person playing music on a piano that corresponds to the audio encoding are merged to obtain a video that contains both the audio and the person playing the audio on the piano, i.e., a piano playing video.

[0037] In steps S10-S50, audio stream data is acquired; the audio stream data is input into an audio encoder for encoding processing to obtain audio encoding; the audio encoding is converted into code using a piano video code conversion model to obtain a piano video codebook sequence corresponding to the audio encoding; the piano video codebook sequence is decoded using a piano video codebook decoder to obtain piano video stream data; the piano video stream data and the audio stream data are merged to obtain a piano playing video. This embodiment encodes the audio data and converts it into a piano video codebook sequence, then decodes the piano video codebook sequence into piano video stream data (this piano video stream data refers to a video stream of a person playing music corresponding to the audio encoding on a piano). The piano video stream data and the audio stream data are then merged to obtain a piano playing video, ultimately converting audio stream data into a piano playing video that includes both audio and the sound of a person playing that audio on a piano. The piano playing video generated from the audio stream data is smoother, resulting in better video effects and quality, and improved user experience.

[0038] Optionally, before step S30, i.e. before the audio encoding is converted using the piano video code conversion model to obtain the piano video codebook sequence corresponding to the audio encoding, the following steps are included:

[0039] S301. Obtain a piano playing video sample;

[0040] S302. Encode the piano playing video sample to obtain a first video codebook sequence and audio sample encoding;

[0041] S303. Input the encoded audio samples into the initial piano video code conversion model to obtain the target video codebook sequence;

[0042] S304. Determine the video loss value based on the first video codebook sequence and the target video codebook sequence;

[0043] S305. When the video loss value does not reach the preset convergence condition, iteratively update the initial parameters of the initial piano video code conversion model until the video loss value reaches the preset convergence condition, and then use the converged initial piano video code conversion model as the piano video code conversion model.

[0044] Understandably, piano playing video samples refer to videos collected for training the piano video code conversion model, containing both audio and the sound of a person playing the piano. Encoding the piano playing video samples involves encoding them into a first video codebook sequence and audio sample codes. Specifically, the piano playing video samples are first split into first piano video stream samples and audio stream samples; then, the first piano video stream samples and audio stream samples are encoded separately to obtain a first video codebook sequence and audio sample codes. An initial piano video code conversion model is used to train and learn from the first video codebook sequence and audio sample codes, ultimately generating a piano video code conversion model that can convert audio sample codes into video codebook sequences. The initial piano video code conversion model can be a GPT2 model (language model), a type of Transformer (self-attention network) model that predicts output results based on input data. The initial piano video code conversion model has relatively weak code conversion capabilities and requires continuous iterative updates to eventually obtain a piano video code conversion model with strong conversion capabilities that meets convergence conditions. The target video codebook sequence is obtained by converting the audio sample encoding using the initial piano video code conversion model. The video loss value refers to the difference between the first and target video codebook sequences. A smaller video loss value indicates that the target video codebook sequence is closer to the first video codebook sequence, representing a stronger code conversion capability of the initial piano video code conversion model. Preferably, the video loss value can be obtained using a loss function based on cross-entropy.

[0045] In this embodiment, the initial piano video code conversion model is trained based on piano playing video samples, so that the obtained piano video code conversion model has a strong code conversion capability and improves the conversion accuracy of the piano video code conversion model.

[0046] Optionally, in step S303, the step of encoding the audio samples and inputting them into the initial piano video code conversion model to obtain the target video codebook sequence includes:

[0047] S3031. The audio sample encoding is used as the first input data and input into the initial piano video code conversion model to obtain the first frame video codebook sequence;

[0048] S3032. The audio sample encoding and the first frame video codebook sequence are concatenated as second input data and input into the initial piano video code conversion model to obtain the second frame video codebook sequence.

[0049] S3033. The audio sample encoding, the first frame video codebook sequence and the second frame video codebook sequence are concatenated as third input data and input into the initial piano video code conversion model to obtain the third frame video codebook sequence.

[0050] S3034. When all the audio sample encodings are converted into a video codebook sequence, the target video codebook sequence is obtained.

[0051] Understandably, audio sample encoding can be an encoding composed of several characters. Similarly, video codebook sequences can be encodings composed of several characters. After inputting the entire audio sample encoding as the first input data into the initial piano video code conversion model, the initial piano video code conversion model first predicts the code sequence of one video frame when converting the audio sample encoding, obtaining a video codebook sequence for that video frame. This video codebook sequence is denoted as the first frame video codebook sequence. Then, the predicted first frame video codebook sequence and the audio sample encoding are concatenated as the second input data to predict the video codebook sequence of the next video frame, which is denoted as the second frame video codebook sequence. Likewise, the audio sample encoding, the first frame video codebook sequence, and the second frame video codebook sequence are concatenated as the third input data and input into the initial piano video code conversion model to obtain the third frame video codebook sequence. This process continues until all characters of the audio sample encoding have been converted into video codebook sequences, at which point the target video codebook sequence is obtained. For example, when the first input data is (a1, a2, a3, ..., ... <bos>When ), the corresponding output data can be (a2, a3, a4, ..., <bos>(v1). Where (a1, a2, a3, ...) are the audio sample codes. <bos>Let v1 be the delimiter and v1 be the sequence of the first frame video codebook; then the second input data is (a1, a2, a3, ..., ...). <bos>The corresponding output data can be (a3, a4, a5, ..., v1). <bos>The third input data is (a1, a2, a3, ..., v1, v2), where v2 is the second frame video codebook sequence; the third input data is (a1, a2, a3, ..., v1, v2). <bos>The corresponding output data can be (a4, a5, a6, ..., v1, v2). <bos>v1, v2, v3), where v3 is the third frame video codebook sequence; when all audio sample encodings are converted into video codebook sequences, the corresponding output data can be (v1, v2, v3, ...), where (v1, v2, v3, ...) is the target video codebook sequence.

[0052] In this embodiment, the video codebook sequence of the first frame is predicted by the entire audio encoding. Then, the video codebook sequence of the next video frame is predicted based on the entire audio encoding and the obtained video codebook sequence. This allows the video codebook sequence of each video frame to fully learn the content of the entire audio encoding and the video codebook sequences of all video frames before that video frame, thereby improving the accuracy of the target video codebook sequence.

[0053] Optionally, in step S302, i.e., encoding the piano playing video sample to obtain the first video codebook sequence and audio sample encoding, the following steps are included:

[0054] S3021. The piano playing video sample is split into two streams to obtain a first piano video stream sample and an audio stream sample.

[0055] S3022. Input the first piano video stream sample into the codebook encoder to obtain the first video codebook sequence;

[0056] S3023. Input the audio stream sample into the audio encoder to obtain the audio sample encoding.

[0057] Understandably, stream splitting refers to the process of dividing a piano playing video sample into a first piano video stream sample and an audio stream sample using video splitting technology. The first piano video stream sample is the video stream segmented from the piano playing video sample, containing several video frames of a person playing the piano. The audio stream sample is the audio stream segmented from the piano playing video sample. A codebook encoder is an encoder that encodes the video stream. The codebook encoder encodes the first piano video stream sample into a first video codebook sequence. An audio encoder is an encoder that encodes audio data. The audio encoder encodes the audio stream sample into an audio sample code.

[0058] In this embodiment, the piano playing video samples are split into two streams, and the video stream and audio stream are encoded differently. This allows for the rapid and accurate acquisition of the first video codebook sequence and audio sample encoding, thereby improving the efficiency of model training.

[0059] Optionally, before step S40, before decoding the piano video codebook sequence using the piano video codebook decoder to obtain piano video stream data, the following steps are included:

[0060] S401. Obtain the second piano video stream sample;

[0061] S402. Extract the first skeletal key point video stream from the second piano video stream sample using the hand model;

[0062] S403. The second piano video stream sample is encoded using a codebook encoder to obtain a second video codebook sequence;

[0063] S404. Quantize the second video codebook sequence to obtain a video quantized codebook sequence;

[0064] S405. Input the video quantization codebook sequence into the initial piano video codebook decoder for decoding processing to obtain the target piano video stream and the second skeletal keypoint video stream.

[0065] S406. Determine the total loss value based on the second piano video stream sample, the target piano video stream, the first skeletal keypoint video stream, the second skeletal keypoint video stream, the second video codebook sequence, and the video quantization codebook sequence.

[0066] S407. When the total loss value does not reach the preset convergence condition, iteratively update the initial parameters of the initial piano video codebook decoder until the total loss value reaches the preset convergence condition, and then use the converged initial piano video codebook decoder as the piano video codebook decoder.

[0067] Understandably, the second piano video stream sample refers to a video stream segmented from the piano playing video sample, containing several video frames of a person playing the piano with their hands. The hand model refers to a trained neural network model used to identify hands from the input data. The hand model extracts the skeletal keypoints of the hand in each video frame of the second piano video stream sample, and merges these keypoints into a single video, outputting the first skeletal keypoint video stream. The second video codebook sequence refers to the video codebook sequence obtained by encoding the second piano video stream sample using a codebook encoder. Quantization processing refers to the process of querying the codebook library for the video codebook sequence corresponding to the second video codebook sequence using a quantization function. The codebook library is an existing library containing several piano video codes. The video quantized codebook sequence refers to the video codebook sequence corresponding to the second video codebook sequence found in the codebook library using a quantization function. The initial piano video codebook decoder can be a VqGAN model (Vector Quantized General Adversarial Network, an image generation model), which can generate corresponding images based on the input data. The initial piano video codebook decoder decodes the video quantization codebook sequence into a video stream consisting of several images, i.e., the target piano video stream. Similarly, referring to the method for obtaining the first skeletal keypoint video stream, the second skeletal keypoint video stream can be obtained from the target piano video stream. Furthermore, based on the second piano video stream samples, the target piano video stream, the first skeletal keypoint video stream, the second skeletal keypoint video stream, the second video codebook sequence, and the video quantization codebook sequence, the total loss value of the initial piano video codebook decoder is determined. The initial parameters of the initial piano video codebook decoder are then iteratively updated based on this total loss value until the initial piano video codebook decoder meets the convergence condition, thus obtaining the piano video codebook decoder.

[0068] In this embodiment, the initial piano video codebook decoder is trained based on the second piano video stream sample, so that the obtained piano video codebook decoder has a strong decoding capability and improves the decoding accuracy of the piano video codebook decoder.

[0069] Optionally, in step S406, namely, determining the total loss value based on the second piano video stream sample, the target piano video stream, the first skeletal keypoint video stream, the second skeletal keypoint video stream, the second video codebook sequence, and the video quantization codebook sequence, includes:

[0070] S4061. Determine the first loss value based on the second piano video stream sample and the target piano video stream;

[0071] S4062. Determine the second loss value based on the first skeletal keypoint video stream and the second skeletal keypoint video stream;

[0072] S4063. Determine a third loss value based on the second video codebook sequence and the video quantization codebook sequence;

[0073] S4064. Determine the total loss value based on the first loss value, the second loss value, and the third loss value.

[0074] Understandably, the first loss value refers to the loss between the second piano video stream sample and the target piano video stream, which can be calculated by a discriminator. Similarly, the second loss value refers to the loss between the second video codebook sequence and the video quantization codebook sequence, and the third loss value refers to the loss between the second video codebook sequence and the video quantization codebook sequence. The second and third loss values ​​can be calculated based on Euclidean distance. Furthermore, the sum of the first, second, and third loss values ​​can be determined as the total loss value.

[0075] In this embodiment, the total loss value considers the loss values ​​between the second piano video stream sample and the target piano video stream, between the first skeletal keypoint video stream and the second skeletal keypoint video stream, and between the second video codebook sequence and the video quantization codebook sequence. This ensures that the initial piano video codebook decoder fully learns the hand skeletal keypoints, the video quantization codebook sequence, and the piano video stream during model training, resulting in clearer hands and smoother video playing in the generated piano playing video, thus improving the user experience.

[0076] It should be understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0077] In one embodiment, a piano playing video generation device is provided, which corresponds one-to-one with the piano playing video generation method described in the above embodiments. For example... Figure 3 As shown, the piano playing video generation device includes an audio stream data module 10, an audio encoding module 20, a piano video codebook sequence module 30, a piano video stream data module 40, and a piano playing video module 50. Detailed descriptions of each functional module are as follows:

[0078] Audio stream data module 10 is used to acquire audio stream data;

[0079] Audio encoding module 20 is used to input the audio stream data into an audio encoder for encoding processing to obtain audio encoding;

[0080] The piano video codebook sequence module 30 is used to perform code conversion on the audio encoding through a piano video code conversion model to obtain a piano video codebook sequence corresponding to the audio encoding.

[0081] The piano video stream data module 40 is used to decode the piano video codebook sequence through the piano video codebook decoder to obtain piano video stream data;

[0082] The piano playing video module 50 is used to merge the piano video stream data and the audio stream data to obtain a piano playing video; the piano video stream data refers to the video stream of a person playing music corresponding to the audio encoding on a piano.

[0083] Optionally, the piano playing video generation device further includes:

[0084] The piano playing video sample module is used to obtain piano playing video samples.

[0085] The encoding processing module is used to encode the piano playing video samples to obtain a first video codebook sequence and audio sample encoding;

[0086] The target video codebook sequence module is used to encode the audio samples and input them into the initial piano video code conversion model to obtain the target video codebook sequence.

[0087] The video loss value module is used to determine a video loss value based on the first video codebook sequence and the target video codebook sequence;

[0088] The piano video code conversion model module is used to iteratively update the initial parameters of the initial piano video code conversion model when the video loss value does not reach the preset convergence condition, until the video loss value reaches the preset convergence condition, and then use the converged initial piano video code conversion model as the piano video code conversion model.

[0089] Optionally, the target video codebook sequence module includes:

[0090] The first frame video codebook sequence unit is used to input the audio sample encoding as the first input data into the initial piano video code conversion model to obtain the first frame video codebook sequence.

[0091] The second frame video codebook sequence unit is used to concatenate the audio sample encoding and the first frame video codebook sequence as second input data, and input them into the initial piano video code conversion model to obtain the second frame video codebook sequence.

[0092] The third frame video codebook sequence unit is used to concatenate the audio sample encoding, the first frame video codebook sequence, and the second frame video codebook sequence as third input data, and input them into the initial piano video code conversion model to obtain the third frame video codebook sequence.

[0093] The target video codebook sequence unit is used to obtain the target video codebook sequence when all the audio sample encodings are converted into a video codebook sequence.

[0094] Optionally, the encoding processing module includes:

[0095] The splitting processing unit is used to split the piano playing video sample to obtain a first piano video stream sample and an audio stream sample.

[0096] The first video codebook sequence unit is used to input the first piano video stream sample into the codebook encoder to obtain the first video codebook sequence.

[0097] An audio sample encoding unit is used to input the audio stream sample into the audio encoder to obtain the audio sample encoding.

[0098] Optionally, the piano playing video generation device further includes:

[0099] The second piano video stream sample module is used to obtain the second piano video stream sample.

[0100] The first skeletal key point video stream module is used to extract the first skeletal key point video stream from the second piano video stream sample through the hand model;

[0101] The second video codebook sequence module is used to encode the second piano video stream sample through a codebook encoder to obtain the second video codebook sequence.

[0102] The video quantization codebook sequence module is used to quantize the second video codebook sequence to obtain a video quantization codebook sequence.

[0103] The decoding processing module is used to input the video quantization codebook sequence into the initial piano video codebook decoder for decoding processing to obtain the target piano video stream and the second skeletal keypoint video stream.

[0104] The total loss value module is used to determine the total loss value based on the second piano video stream sample, the target piano video stream, the first skeletal keypoint video stream, the second skeletal keypoint video stream, the second video codebook sequence, and the video quantization codebook sequence.

[0105] The piano video codebook decoder module is used to iteratively update the initial parameters of the initial piano video codebook decoder when the total loss value does not reach the preset convergence condition, until the total loss value reaches the preset convergence condition, and then use the converged initial piano video codebook decoder as the piano video codebook decoder.

[0106] Optionally, the total loss value module includes:

[0107] The first loss value unit is used to determine a first loss value based on the second piano video stream sample and the target piano video stream;

[0108] The second loss value unit is used to determine a second loss value based on the first skeletal keypoint video stream and the second skeletal keypoint video stream.

[0109] The third loss value unit is used to determine a third loss value based on the second video codebook sequence and the video quantization codebook sequence;

[0110] The total loss value unit is used to determine the total loss value based on the first loss value, the second loss value, and the third loss value.

[0111] Specific limitations regarding the piano playing video generation device can be found in the limitations of the piano playing video generation method described above, and will not be repeated here. Each module in the aforementioned piano playing video generation device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device in hardware form, or stored in the memory of a computer device in software form, so that the processor can call and execute the corresponding operations of each module.

[0112] In one embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be as follows: Figure 4 As shown, the computer device includes a processor, memory, network interface, and database connected via a system bus. The processor provides computing and control capabilities. The memory includes a readable storage medium and internal memory. The readable storage medium stores an operating system, computer-readable instructions, and a database. The internal memory provides an environment for the operation of the operating system and computer-readable instructions in the readable storage medium. The database stores data related to the piano playing video generation method. The network interface communicates with external terminals via a network connection. When the computer-readable instructions are executed by the processor, a piano playing video generation method is implemented. The readable storage medium provided in this embodiment includes both non-volatile and volatile readable storage media.

[0113] In one embodiment, a computer device is provided, including a memory, a processor, and computer-readable instructions stored in the memory and executable on the processor, wherein the processor performs the following steps when executing the computer-readable instructions:

[0114] Acquire audio stream data;

[0115] The audio stream data is input into an audio encoder for encoding processing to obtain the audio code;

[0116] The audio encoding is converted using a piano video code conversion model to obtain a piano video codebook sequence corresponding to the audio encoding.

[0117] The piano video codebook sequence is decoded by a piano video codebook decoder to obtain piano video stream data; the piano video stream data refers to a video stream of a person playing music on a piano that corresponds to the audio encoding.

[0118] The piano video stream data and the audio stream data are merged to obtain a piano playing video.

[0119] In one embodiment, one or more computer-readable storage media storing computer-readable instructions are provided. The readable storage media provided in this embodiment include non-volatile readable storage media and volatile readable storage media. The readable storage media stores computer-readable instructions, which, when executed by one or more processors, perform the following steps:

[0120] Acquire audio stream data;

[0121] The audio stream data is input into an audio encoder for encoding processing to obtain the audio code;

[0122] The audio encoding is converted using a piano video code conversion model to obtain a piano video codebook sequence corresponding to the audio encoding.

[0123] The piano video codebook sequence is decoded by a piano video codebook decoder to obtain piano video stream data; the piano video stream data refers to a video stream of a person playing music on a piano that corresponds to the audio encoding.

[0124] The piano video stream data and the audio stream data are merged to obtain a piano playing video.

[0125] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by instructing related hardware with computer-readable instructions. These computer-readable instructions can be stored in a non-volatile readable storage medium or a volatile readable storage medium. When executed, these computer-readable instructions can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in a variety of forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), RAMbus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0126] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is used as an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above.

[0127] The above-described embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be included within the protection scope of the present invention.< / bos> < / bos> < / bos> < / bos> < / bos> < / bos> < / bos>

Claims

1. A method for generating piano playing videos, characterized in that, include: Acquire audio stream data; The audio stream data is input into an audio encoder for encoding processing to obtain the audio code; The audio encoding is converted using a piano video code conversion model to obtain a piano video codebook sequence corresponding to the audio encoding. The piano video codebook sequence is decoded using a piano video codebook decoder to obtain piano video stream data; The piano video stream data refers to a video stream of a person playing music on a piano that corresponds to the audio encoding. The piano video stream data and the audio stream data are merged to obtain a piano playing video; Before decoding the piano video codebook sequence using a piano video codebook decoder to obtain piano video stream data, the process includes: Obtain a sample of the second piano video stream; The first skeletal keypoint video stream is extracted from the second piano video stream sample using a hand model; The second piano video stream sample is encoded using a codebook encoder to obtain a second video codebook sequence; The second video codebook sequence is quantized to obtain a video quantized codebook sequence; The video quantization codebook sequence is input into the initial piano video codebook decoder for decoding processing to obtain the target piano video stream and the second skeletal keypoint video stream. The total loss value is determined based on the second piano video stream sample, the target piano video stream, the first skeletal keypoint video stream, the second skeletal keypoint video stream, the second video codebook sequence, and the video quantization codebook sequence. When the total loss value does not reach the preset convergence condition, the initial parameters of the initial piano video codebook decoder are iteratively updated until the total loss value reaches the preset convergence condition. Then, the converged initial piano video codebook decoder is used as the piano video codebook decoder.

2. The method for generating piano playing videos as described in claim 1, characterized in that, Before performing code conversion on the audio encoding using the piano video code conversion model to obtain the piano video codebook sequence corresponding to the audio encoding, the following steps are included: Obtain piano playing video samples; The piano playing video samples are encoded to obtain a first video codebook sequence and audio sample encoding; The audio sample encoding is input into the initial piano video code conversion model to obtain the target video codebook sequence; The video loss value is determined based on the first video codebook sequence and the target video codebook sequence; When the video loss value does not reach the preset convergence condition, the initial parameters of the initial piano video code conversion model are iteratively updated until the video loss value reaches the preset convergence condition. Then, the converged initial piano video code conversion model is used as the piano video code conversion model.

3. The method for generating piano playing videos as described in claim 2, characterized in that, The step of inputting the encoded audio samples into the initial piano video code conversion model to obtain the target video codebook sequence includes: The audio sample encoding is used as the first input data and input into the initial piano video code conversion model to obtain the first frame video codebook sequence; The audio sample encoding and the first frame video codebook sequence are concatenated as second input data and input into the initial piano video code conversion model to obtain the second frame video codebook sequence. The audio sample encoding, the first frame video codebook sequence, and the second frame video codebook sequence are concatenated as third input data and input into the initial piano video code conversion model to obtain the third frame video codebook sequence. When all the audio sample encodings are converted into a video codebook sequence, the target video codebook sequence is obtained.

4. The method for generating piano playing videos as described in claim 2, characterized in that, The encoding process of the piano playing video sample to obtain a first video codebook sequence and audio sample encoding includes: The piano playing video sample is split into two streams to obtain a first piano video stream sample and an audio stream sample. The first piano video stream sample is input into the codebook encoder to obtain the first video codebook sequence; The audio stream sample is input into the audio encoder to obtain the audio sample encoding.

5. The method for generating piano playing videos as described in claim 4, characterized in that, The step of determining the total loss value based on the second piano video stream sample, the target piano video stream, the first skeletal keypoint video stream, the second skeletal keypoint video stream, the second video codebook sequence, and the video quantization codebook sequence includes: A first loss value is determined based on the second piano video stream sample and the target piano video stream; The second loss value is determined based on the first skeletal keypoint video stream and the second skeletal keypoint video stream; A third loss value is determined based on the second video codebook sequence and the video quantization codebook sequence; The total loss value is determined based on the first loss value, the second loss value, and the third loss value.

6. A device for generating piano playing videos, characterized in that, include: The audio stream data module is used to acquire audio stream data; The audio encoding module is used to input the audio stream data into the audio encoder for encoding processing to obtain the audio encoding; The piano video codebook sequence module is used to convert the audio encoding using a piano video code conversion model to obtain a piano video codebook sequence corresponding to the audio encoding. The piano video stream data module is used to decode the piano video codebook sequence through a piano video codebook decoder to obtain piano video stream data; The piano video stream data refers to a video stream of a person playing music on a piano that corresponds to the audio encoding. The piano playing video module is used to merge the piano video stream data and the audio stream data to obtain a piano playing video. The piano playing video generation device further includes: The second piano video stream sample module is used to obtain the second piano video stream sample. The first skeletal key point video stream module is used to extract the first skeletal key point video stream from the second piano video stream sample through the hand model; The second video codebook sequence module is used to encode the second piano video stream sample through a codebook encoder to obtain the second video codebook sequence. The video quantization codebook sequence module is used to quantize the second video codebook sequence to obtain a video quantization codebook sequence. The decoding processing module is used to input the video quantization codebook sequence into the initial piano video codebook decoder for decoding processing to obtain the target piano video stream and the second skeletal keypoint video stream. The total loss value module is used to determine the total loss value based on the second piano video stream sample, the target piano video stream, the first skeletal keypoint video stream, the second skeletal keypoint video stream, the second video codebook sequence, and the video quantization codebook sequence. The piano video codebook decoder module is used to iteratively update the initial parameters of the initial piano video codebook decoder when the total loss value does not reach the preset convergence condition, until the total loss value reaches the preset convergence condition, and then use the converged initial piano video codebook decoder as the piano video codebook decoder.

7. The piano playing video generation device as described in claim 6, characterized in that, Prior to the piano video codebook sequence module, the following is included: The piano playing video sample module is used to obtain piano playing video samples. The encoding processing module is used to encode the piano playing video samples to obtain a first video codebook sequence and audio sample encoding; The target video codebook sequence module is used to encode the audio samples and input them into the initial piano video code conversion model to obtain the target video codebook sequence. The video loss value module is used to determine a video loss value based on the first video codebook sequence and the target video codebook sequence; The piano video code conversion model module is used to iteratively update the initial parameters of the initial piano video code conversion model when the video loss value does not reach the preset convergence condition, until the video loss value reaches the preset convergence condition, and then use the converged initial piano video code conversion model as the piano video code conversion model.

8. A computer device comprising a memory, a processor, and computer-readable instructions stored in the memory and executable on the processor, characterized in that, When the processor executes the computer-readable instructions, it implements the piano playing video generation method as described in any one of claims 1 to 5.

9. One or more readable storage media storing computer-readable instructions, characterized in that, When the computer-readable instructions are executed by one or more processors, the one or more processors cause the piano playing video generation method as described in any one of claims 1 to 5 to be performed.

Images