Piano playing audio and video joint generation model based on multi-modal MIDI guidance
By using a multimodal MIDI-guided audio-video co-generation model, the problems of inaccurate finger-key correspondence and audio-video asynchrony in existing technologies have been solved, enabling the generation of high-quality, synchronized piano performance videos and improving the credibility of teaching and virtual performances.
Patent Information
- Application Number
- CN202511711082.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-11-20
- Publication Date
- 2026-03-06
AI Technical Summary
Existing music visualization generation methods cannot achieve precise correspondence between fingers and piano keys, have poor synchronization between audio and video, and lack multimodal training data, resulting in inconsistencies between the generated video and audio, which affects the credibility of teaching and performance visualization.
A multimodal MIDI-guided piano performance audio-video joint generation model is adopted. The rhythm features and keyboard geometry information are extracted through the MIDI-to-Pose module, and the Pose-to-Video generation module generates synchronized video. The Symbolic-to-Audio generation module generates audio aligned with the video frames. The diffusion model and Transformer architecture are used to improve the generation quality.
It achieves strict temporal and semantic alignment between hand movements and key strikes, resulting in highly synchronized video and audio with an error of less than 25 milliseconds. Visual quality and audio fidelity are significantly improved, reducing the cost of artificial synthesis and making it suitable for scenarios such as music education and virtual performance.
Smart Images
Figure CN121619477A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of multimodal generation and audio-visual synthesis technology, and in particular to a joint generation model of piano performance audio and video based on multimodal MIDI guidance. Background Technology
[0002] Existing music visualization generation methods mostly rely on audio or text prompts, only generating videos that superficially "look like playing the piano," lacking a precise correspondence with actual key presses and note rhythms. For example, existing technologies have the following drawbacks:
[0003] 1. Existing video generation models are limited to general scenarios: such as Sora2 and Veo3, although they can generate high-quality images, they cannot control the physical interaction between fingers and piano keys.
[0004] 2. Inaccurate music-driven gesture generation: Existing sound-driven or gesture-driven generation only focuses on rhythm consistency and lacks hand key alignment constraints, making it difficult to reproduce realistic performance.
[0005] 3. Independent audio and video generation leads to asynchrony: Traditional methods generate audio and video in separate steps, which can easily cause audio-visual misalignment and rhythm discrepancies, affecting the credibility of teaching and performance visualization.
[0006] 4. Lack of multimodal training data: Existing datasets mostly contain only audio or pose sequences, lacking high-quality synchronous MIDI, video, and 3D hand pose annotation data.
[0007] Therefore, it is necessary to provide a piano performance audio-video joint generation model based on multimodal MIDI guidance, so that the generated hand movements, key strikes and piano sounds are strictly aligned in time and semantics, so as to achieve the generation of realistic, synchronous and physically consistent piano performance videos. Summary of the Invention
[0008] The purpose of this invention is to provide a piano performance audio-video joint generation model based on multimodal MIDI guidance, which ensures that the generated hand movements, key strikes and piano sounds are strictly aligned in time and semantics, thereby achieving the generation of realistic, synchronous and physically consistent piano performance videos.
[0009] To address the problems existing in the prior art, this invention provides a piano performance audio-video joint generation model based on multimodal MIDI guidance, comprising:
[0010] The dataset module contains a multimodal dataset of piano performance materials, including video, MIDI, audio, and 3D gesture annotations.
[0011] The MIDI-to-Pose module is configured to use a MIDI encoder to extract rhythm features, pitch features, and velocity features; a spatial encoder to extract keyboard geometry information and embed it into a unified latent space; after fusing the two through cross-attention, the global gesture trajectory is output by the position decoder, and then a physically reasonable 3D gesture sequence is generated by the gesture U-Net based on the diffusion model; during training, position loss, velocity loss, and diffusion noise prediction loss are combined to ensure spatial and temporal consistency.
[0012] The Pose-to-Video generation module is configured to introduce a pose sequence encoder and a reference pose encoder. The pose sequence encoder and the reference pose encoder are used to encode the gesture pose sequence generated in the previous stage and the reference gesture pose into the latent variable space, and inject the gesture-driven features into the diffusion video generation backbone. Based on the pre-trained video diffusion model, fine-tuning is performed to generate a coherent video frame by frame through conditional latent variable diffusion.
[0013] The Symbolic-to-Audio generation module is configured to build a lightweight MIDI guide encoder that maps MIDI events to time embeddings. It inputs time embeddings, gesture video features, and text prompts into a DiT-based audio generation network. It learns the acoustic velocity field through stream matching targets and finally generates piano performance audio that is strictly aligned with video frames and consistent with MIDI.
[0014] Optionally, in the multimodal MIDI-guided piano performance audio-video joint generation model, the MIDI-to-Pose module takes the following inputs: a structured MIDI sequence and video spatial metadata, including resolution and key position coordinates; and outputs: predicted hand 3D translation vectors and joint pose parameters.
[0015] Optionally, in the multimodal MIDI-guided piano performance audio-video joint generation model, the input to the Pose-to-Video generation module is the predicted gesture sequence and a single-frame reference image.
[0016] Optionally, in the multimodal MIDI-guided piano performance audio-video joint generation model, the optimization objective of the Pose-to-Video generation module is noise prediction loss.
[0017] Optionally, in the multimodal MIDI-guided piano performance audio-video joint generation model, the input to the Symbolic-to-Audio generation module is a MIDI event sequence, gesture video features, and text prompts.
[0018] Compared with the prior art, the present invention has the following advantages:
[0019] (1) The present invention aims to provide a piano performance audio and video joint generation model based on multimodal MIDI guidance, so that the generated hand movements, key strikes and piano sounds are strictly aligned in time and semantics, so as to realize the generation of realistic, synchronous and physically consistent piano performance videos.
[0020] (2) This invention has a wide range of applications and can be used in music education, virtual performance, interpretable music generation and performance analysis and other scenarios.
[0021] (3) This invention can significantly reduce the cost of manual post-production synthesis and improve the production efficiency of interactive music content.
[0022] (4) Significantly improved physical and temporal synchronization: Generated hand gestures correspond one-to-one with key presses, with an audio-visual synchronization error of less than 25 milliseconds. High visual quality: The video's FID / FVD metrics are comparable to mainstream video diffusion models, while maintaining better keyboard geometric consistency. Superior audio fidelity: The Frechet Distance under VGG / PANN / PASST features is significantly lower than the baseline model, and the KL divergence is reduced by more than 60%. Attached Figure Description
[0023] Figure 1 A module diagram of the piano performance audio-visual joint generation model provided in an embodiment of the present invention;
[0024] Figure 2 This is a schematic diagram of the overall structure of the model provided in an embodiment of the present invention. Detailed Implementation
[0025] The specific embodiments of the present invention will now be described in more detail with reference to the accompanying drawings. The advantages and features of the present invention will become clearer from the following description. It should be noted that the drawings are all in a very simplified form and use non-precise proportions, and are only used to facilitate and clarify the illustration of the embodiments of the present invention.
[0026] In the description of this application, it should be understood that the terms "center", "longitudinal", "lateral", "length", "width", "thickness", "upper", "lower", "front", "rear", "left", "right", "vertical", "horizontal", "top", "bottom", "inner", "outer", "clockwise", "counterclockwise", etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. They are used only for the convenience of describing this application and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limitations on this application.
[0027] In the following, if the methods described herein include a series of steps, the order of these steps presented herein is not necessarily the only order in which these steps can be performed, and some of the steps described may be omitted and / or some other steps not described herein may be added to the method.
[0028] Existing music visualization generation methods mostly rely on audio or text prompts, which can only generate videos that superficially "look like playing the piano," but lack a precise correspondence with real key presses and note rhythms.
[0029] To address the problems existing in the prior art, this invention provides a piano performance audio-video joint generation model based on multimodal MIDI guidance. Its overall structure includes four modules: ① a dataset module; ② a MIDI-to-Pose module; ③ a Pose-to-Video generation module; and ④ a Symbol-to-Audio generation module. Among these, the MIDI-to-Pose, Pose-to-Video, and Symbol-to-Audio generation modules are the core modules.
[0030] Specifically, such as Figure 1 and 2 The piano performance audio-visual joint generation model includes:
[0031] The dataset module contains a multimodal dataset of piano performance materials, including video, MIDI, audio, and 3D gesture annotations. MIDI, short for Musical Instrument Digital Interface, is a computer science term and the most widely used music standard format in the music production industry, often referred to as "computer-understandable sheet music." It uses digital control signals for musical notes to record music.
[0032] The MIDI-to-Pose module is configured to extract rhythm, pitch, and velocity features using a MIDI encoder; extract keyboard geometry information using a spatial encoder and embed it into a unified latent space; fuse the two through cross-attention, output the global gesture trajectory via a position decoder, and then generate a physically plausible 3D gesture sequence using a gesture U-Net based on a diffusion model; during training, a position loss L is incorporated. p Speed loss L v With the predicted loss of diffused noise L diff This ensures consistency between space and time; among them, U-Net is a deep learning model commonly used for image segmentation, which can also be used for gesture segmentation.
[0033] In the MIDI-to-Pose module, the inputs are a structured MIDI sequence M and video spatial metadata S, including resolution and key position coordinates; the outputs are the predicted two-hand 3D translation vectors T and joint pose parameters θ.
[0034] The Pose-to-Video generation module is configured to incorporate a pose sequence encoder and a reference pose encoder. These encoders encode the gesture pose sequence generated in the previous stage and the reference gesture pose into the latent variable space, injecting gesture-driven features into the diffusion-based video generation backbone. Based on a pre-trained video diffusion model (such as WAN2.1), fine-tuning is performed to generate coherent video frame-by-frame through conditional latent variable diffusion. The optimization objective of the Pose-to-Video generation module is noise prediction loss, thereby ensuring spatial coherence and temporal smoothness of the generated video. The generated video can reproduce the correspondence between keyboard geometry and hand trajectories, achieving high-fidelity visualization of performance.
[0035] The input to the Pose-to-Video generation module is the predicted gesture sequence and a single-frame reference image.
[0036] The Symbolic-to-Audio generation module is configured to build a lightweight MIDI guide encoder that maps MIDI events to time embeddings. It then feeds the time embeddings, gesture video features, and text cues into a DiT-based audio generation network. Through stream matching, it learns the acoustic velocity field and ultimately generates piano performance audio that is strictly aligned with video frames and consistent with MIDI. The inputs to the Symbolic-to-Audio generation module are MIDI event sequences, gesture video features, and text cues.
[0037] DiT (Diffusion Transformer) is a generative model that combines diffusion models with the Transformer architecture. Through multimodal feature encoding and global temporal modeling capabilities, it significantly improves generation quality and computational efficiency.
[0038] In this invention, the pre-trained Video DiT and Audio DiT backbones are frozen during training, and only the pose, MIDI encoder, and adapter layers are trained, achieving efficient fine-tuning and resource saving.
[0039] In summary, compared with the prior art, the present invention has the following advantages:
[0040] (1) The present invention aims to provide a piano performance audio and video joint generation model based on multimodal MIDI guidance, so that the generated hand movements, key strikes and piano sounds are strictly aligned in time and semantics, so as to realize the generation of realistic, synchronous and physically consistent piano performance videos.
[0041] (2) This invention has a wide range of applications and can be used in music education, virtual performance, interpretable music generation and performance analysis and other scenarios.
[0042] (3) This invention can significantly reduce the cost of manual post-production synthesis and improve the production efficiency of interactive music content.
[0043] (4) Significantly improved physical and temporal synchronization: Generated hand gestures correspond one-to-one with key presses, with an audio-visual synchronization error of less than 25 milliseconds. High visual quality: The video's FID / FVD metrics are comparable to mainstream video diffusion models, while maintaining better keyboard geometric consistency. Superior audio fidelity: The Frechet Distance under VGG / PANN / PASST features is significantly lower than the baseline model, and the KL divergence is reduced by more than 60%.
[0044] The above are merely preferred embodiments of the present invention and do not constitute any limitation on the present invention. Any equivalent substitutions or modifications made by those skilled in the art to the technical solutions and content disclosed in the present invention without departing from the scope of the present invention shall be deemed to have remained within the protection scope of the present invention.
Claims
1. A piano performance audio-video joint generation model based on multi-modal MIDI guidance, characterized in that, Comprise: A dataset module containing piano performance multi-modal dataset with video, MIDI, audio and 3D gesture annotation; A MIDI-to-Pose module configured to extract rhythm features, pitch features and dynamics features using a MIDI encoder; A spatial encoder extracts keyboard geometry information and embeds it into a unified latent space; after fusion through cross-attention, the global gesture trajectory is output through a positional decoder, and then a physically reasonable 3D gesture sequence is generated through a gesture U-Net based on a diffusion model; during training, position loss, velocity loss and diffusion noise prediction loss are combined to ensure spatial and temporal consistency; A Pose-to-Video generation module configured to introduce a pose sequence encoder and a reference pose encoder, which are used to encode the gesture pose sequence generated in the previous stage and the reference gesture pose into the latent variable space, and inject gesture driving features into the diffusion-based video generation backbone; based on a pre-trained video diffusion model, a coherent video is generated frame by frame through conditional latent variable diffusion; A Symbolic-to-Audio generation module configured to build a lightweight MIDI guided encoder to map MIDI events to temporal embeddings; input the temporal embeddings, gesture video features and text prompts into an audio generation network based on the DiT structure; Through flow matching target learning acoustic velocity field, finally generate piano performance audio strictly aligned with video frames and consistent with MIDI.
2. The piano performance audio-video joint generation model based on multi-modal MIDI guidance of claim 1, wherein, In the MIDI-to-Pose module, the input is structured MIDI sequence and video spatial meta information, including resolution and key position coordinates; the output is predicted three-dimensional translation vector and joint pose parameter of both hands.
3. The multi-modal MIDI-guided piano performance audio-video joint generation model of claim 1, wherein, The input of the Pose-to-Video generation module is the predicted gesture sequence and single frame reference image.
4. The multi-modal MIDI-guided piano performance audio-video joint generation model of claim 1, wherein, The optimization goal of the Pose-to-Video generation module is the noise prediction loss.
5. The multi-modal MIDI-guided piano performance audio-video joint generation model of claim 1, wherein, The input of the Symbolic-to-Audio generation module is MIDI event sequence, gesture video feature and text prompt.