Method for generating professional piano sound

Abstract

The invention relates to a method for generating professional piano sound, which comprises the following steps of: constructing a data set which comprises professional data of 10-hour piano sound; a training strategy of a multi-stage stream matching framework is adopted for training, and the stages are as follows: the first stage is supervised training: a multi-stream DiT framework is constructed based on stream matching, and video CLIP features and text FLAN-T5 codes are fused; performing cross-modal feature mapping in the Encodec audio potential space; the second stage is CoP guided learning: introducing an MIDI feature control generation module and a playing video to an MIDI feature predictor; the step-by-step generation comprises the following steps of: generating a top view, a multi-view and stylized audio; in the third stage, factorization contrast learning is adopted, audio and video sharing features and independent features are decoupled, and mutual information boundaries are optimized; a direct preference optimization method is adopted, fine tuning is carried out based on piano playing style preference data, and style controllability is improved. According to the invention, the professional piano sound can be generated based on the video-guided sound effect generation model.

Description

Technical Field

[0001] The present invention relates to the technical field of multimedia audio - video generation, and particularly to a method for generating professional piano sounds. Background Art

[0002] Existing video - to - audio generation technology (V2A) and professional musical instrument video sound generation have the following defects:

[0003] (1) Insufficient alignment ability: Existing V2A models have deviations in audio - video semantics and temporal alignment, and the model is highly sensitive to audio - video asynchrony.

[0004] (2) Lack of professional audio generation: There is a lack of datasets and step - by - step generation guidance mechanisms for professional audio such as piano and violin.

[0005] (3) Limited style control: Traditional supervised learning cannot capture the subtle differences in playing styles.

[0006] Therefore, it is necessary to provide a method for generating professional piano sounds, which generates professional piano sounds based on a video - guided sound effect generation model. Summary of the Invention

[0007] The purpose of the present invention is to provide a method for generating professional piano sounds. Through multi - modal contrast learning and combined preference optimization technology, on the basis of ensuring general sound effect generation, high - quality generation of professional - level audio such as piano performance and general sound effects is achieved.

[0008] To solve the problems existing in the prior art, the present invention provides a method for generating professional piano sounds, including the following steps:

[0009] Construct a dataset, where the dataset contains 10 hours of professional data of piano sounds;

[0010] Adopt a training strategy of a multi - stage flow matching framework for training, and the stages are as follows:

[0011] The first stage is supervised training: Based on flow matching, construct a multi - flow DiT architecture, fuse video CLIP features and text FLAN - T5 encodings; perform cross - modal feature mapping in the Encodec audio latent space;

[0012] The second stage is CoP - guided learning: Introduce a MIDI feature control generation module and a performance video - to - MIDI feature predictor; generate the following processes step - by - step: top - view, multi - view, and stylized audio generation;

[0013] The third stage is combined optimization: Adopt factorized contrast learning to decouple audio - video shared features and independent features, and optimize the mutual information boundary; adopt the direct preference optimization method to fine - tune based on the pianist's playing style preference data to improve style controllability.

[0014] Optionally, in the method for generating professional piano sounds, the content in the dataset includes five-perspective piano performance videos, annotated multi-granularity MIDI information, and differential style annotations of at least two performers, where MIDI is Musical Instrument Digital Interface.

[0015] Optionally, in the method for generating professional piano sounds, the five-perspective piano performance videos show complete piano keys.

[0016] Optionally, in the method for generating professional piano sounds, the multi-granularity MIDI information includes pitch, time, velocity, and sustain.

[0017] Optionally, in the method for generating professional piano sounds,

[0018] The top view, i.e., the coarse-grained midi, contains pitch and timestamps;

[0019] The multi-view, i.e., the fine-grained midi, contains pitch, timestamps, velocity, and sustain.

[0020] Compared with the prior art, the present invention has the following advantages:

[0021] (1) The present invention can solve the deficiencies of existing video-to-audio generation models in semantic alignment, time synchronization, and professional audio generation quality, and proposes a Flow-matching generation framework based on multi-stage chain reasoning (Chain-of-Perform, CoP). Through multi-modal contrast learning and combined preference optimization techniques, on the basis of ensuring the generation of general sound effects, high-quality generation of professional-level audio such as piano performance and general sound effects is achieved.

[0022] (2) The present invention constructs a professional CoT-like dataset, predicts MIDI from coarse-grained to fine-grained through the way of multi-video perspective information fusion, and fuses it into the original V2A network through the newly added DiT network, but the original V2A effect does not decline, achieving pluggable acceptance of various professional sound effect generation tasks. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] Figure 1 It is a flowchart of the method for generating professional piano sounds provided by the embodiment of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0024] The specific embodiments of the present invention will be described in more detail below with reference to the schematic diagrams. According to the following description, the advantages and features of the present invention will be clearer. It should be noted that the drawings are all in a very simplified form and use non-precise scales, only for the purpose of facilitating and clearly assisting in explaining the objectives of the embodiments of the present invention.

[0025] In the description of the present application, it should be understood that the orientation or positional relationships indicated by the terms "center", "longitudinal", "transverse", "length", "width", "thickness", "upper", "lower", "front", "rear", "left", "right", "vertical", "horizontal", "top", "bottom", "inner", "outer", "clockwise", "counterclockwise", etc. are based on the orientation or positional relationships shown in the drawings. These are only for the convenience of describing the present application and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation. Therefore, it should not be construed as a limitation to the present application.

[0026] The existing video-guided audio generation technology (V2A) and the generation of professional musical instrument video sounds have the following defects: (1) Insufficient alignment ability: There are deviations in the semantic and temporal alignment of audio and video in the existing V2A models, and the models are highly sensitive to audio-visual asynchrony; (2) Lack of professional audio generation: There is a lack of datasets and step-by-step generation guidance mechanisms for professional audio such as piano and violin; (3) Limited style control: Traditional supervised learning cannot capture the subtle differences in playing styles.

[0027] To solve the problems existing in the prior art, the present invention provides a method for generating professional piano sounds, as Figure 1 shown, the method includes the following steps:

[0028] S1: Construct a dataset, which contains 10 hours of professional data of piano sounds;

[0029] The content in the dataset includes five-perspective piano performance videos, annotated multi-granularity MIDI information, and differential style annotations of at least two performers, where MIDI (Musical Instrument Digital Interface) is the Musical Instrument Digital Interface.

[0030] Preferably, the five-perspective piano performance videos show complete piano keys. The multi-granularity MIDI information includes pitch, time, dynamics, and sustain. The differential style annotations of at least two performers include CoP step-by-step guidance.

[0031] S2: Adopt a training strategy of a multi-stage flow matching framework for training, and the stages are as follows:

[0032] The first stage is supervised training: Based on flow matching, construct a multi-flow DiT architecture, fuse video CLIP features and text FLAN-T5 encodings; perform cross-modal feature mapping in the Encodec audio latent space;

[0033] Among them, DiT (Diffusion Transformer) is a diffusion model based on the Transformer architecture for image and video generation, which achieves higher scalability and generation quality by replacing the traditional U-Net structure.

[0034] CLIP (Contrastive Language-Image Pre-training) is a pre-training method or model based on contrastive text-image pairs. The working principle of CLIP: The workflow of CLIP includes the following steps: Feature extraction: Use a text encoder and an image encoder to extract the features of text and images respectively. Feature matching: Calculate the cosine similarity to match the image features with the text features and find the most similar text-image matching pairs. Training process: Use the softmax function to calculate probabilities and optimize the model parameters through backpropagation, enabling the model to learn better image and text representations.

[0035] FLAN-T5 is a large pre-trained language model based on the T5 (Text-To-Text Transfer Transformer) architecture, developed by Google's research team. It aims to be trained on a large scale of text pairs data to achieve a wide range of natural language processing tasks, including but not limited to text generation, question answering, translation, summarization, etc.

[0036] EnCodec is a neural network-based audio coding scheme designed to significantly reduce the size of audio files while maintaining high sound quality.

[0037] The second stage is CoP (Chain-of-Perform) guided learning: Introduce the MIDI feature control generation module (Extra-DiT) and the performance video to MIDI feature predictor (MIDI-Predictor); Generate the following processes step by step: top view, multi-view, and stylized audio generation (DPO optimization); The top view is the coarse-grained midi, including pitch and timestamp; The multi-view is the fine-grained midi, including pitch, timestamp, velocity, and sustain.

[0038] The third stage is combined optimization: Adopt factorized contrastive learning (FactorCL) to decouple the audio-visual shared features and independent features and optimize the mutual information boundary; Adopt the direct preference optimization method (DPO) to fine-tune based on the pianist's performance style preference data to improve style controllability.

[0039] Specifically, the principle of the present invention is as Figure 1As shown in the figure, to support both video-to-audio and video-to-piano generation, we employ a multi-stream DiT architecture to process video, audio, and music information separately and integrate them within the network. The original music information is directly extracted from the video using a MIDI prediction module, supporting multi-dimensional performance information such as note pitch, timestamps, dynamics, and sustain, resulting in rich, high-quality music audio.

[0040] In summary, the present invention has the following advantages compared with the prior art:

[0041] (1) The present invention can solve the shortcomings of existing video-to-audio generation models in semantic alignment, temporal synchronization, and professional audio generation quality. It proposes a Flow-matching generation framework based on multi-stage chain-of-Perform (CoP). Through multimodal contrastive learning and combination preference optimization technology, it can achieve high-quality generation of professional-grade audio such as piano performances and general sound effects while ensuring the generation of general sound effects.

[0042] (2) The present invention constructs a professional CoT-like dataset and predicts MIDI from coarse-grained to fine-grained by fusing multi-video perspective information. The newly added DiT network is then integrated into the original V2A network without reducing the original V2A effect, making it pluggable to undertake various professional sound effect generation tasks.

[0043] The above description is merely a preferred embodiment of the present invention and does not limit the present invention in any way. Any person skilled in the art who, without departing from the scope of the present invention, makes any equivalent substitution, modification, or other changes to the technical solution and technical content disclosed in the present invention shall be deemed to be within the scope of the present invention and still fall within the scope of protection of the present invention.

Claims

1. A method for generating professional piano sounds, characterized in that, Including the following steps: Construct a dataset, which contains 10 hours of professional data of piano sounds; Adopt the training strategy of the multi-stage flow matching framework for training, and the stages are as follows: The first stage is supervised training: construct a multi-flow DiT architecture based on flow matching, fuse video CLIP features and text FLAN-T5 encodings; perform cross-modal feature mapping in the Encodec audio latent space; The second stage is CoP-guided learning: introduce a MIDI feature control generation module and a performance video to MIDI feature predictor; Generate the following processes step by step: top view, multi-view, and stylized audio generation; The third stage is combinatorial optimization: adopt factored contrastive learning to decouple audio-visual shared features and independent features, and optimize the mutual information boundary; adopt the direct preference optimization method to fine-tune based on the pianist's performance style preference data to improve style controllability.

2. The method for generating a professional piano sound according to claim 1, wherein, The content in the dataset includes five-view piano performance videos, annotated multi-granularity MIDI information, and differential style annotations of at least two performers, where MIDI is Musical Instrument Digital Interface.

3. The method for generating a professional piano sound according to claim 2, wherein The five-view piano performance videos show complete piano keys.

4. The method for generating a professional piano sound according to claim 2, characterized in that, The multi-granularity MIDI information includes pitch, time, velocity, and sustain.

5. The method for generating professional piano sounds according to claim 1, wherein The top view is coarse-grained midi, including pitch and timestamps; The multi-view is fine-grained midi, including pitch, timestamps, velocity, and sustain.

Images