Audio and video beautifying method and system for musical instrument playing and readable computer storage medium

Abstract

The invention discloses a musical instrument playing audio and video beautifying method and system based on artificial intelligence and a readable storage medium, and aims to solve the problems that in the prior art, wrong playing tones cannot be automatically repaired, the rhythm cannot be optimized, and audio and video synchronization cannot be ensured. The method comprises the following steps: after transcribing a playing audio into MIDI data, carrying out wrong tone repair (correcting wrong notes, supplementing leaked tones and deleting redundant tones) and rhythm beautification (adjusting a note time sequence) through music score alignment or a generative large model; for video beautification, after audio is extracted from the video and beautified, audio and picture synchronization is ensured by dynamically controlling the replacement amplitude. The method can significantly improve the accuracy, fluency and hearing quality of playing audios and videos, and is suitable for the fields of music education, performance later stage, intelligent musical instruments and the like.

Description

Technical Field

[0001] The present invention relates to the technical fields of artificial intelligence and audio processing, and specifically refers to a method, system and readable computer storage medium for beautifying the audio and video of musical instrument performances. Background Art

[0002] The existing technologies for beautifying the audio and video of musical instrument performances mainly focus on timbre optimization and background noise removal. For example, the timbre is adjusted through an equalizer or environmental noise is eliminated using a noise reduction algorithm. However, these methods cannot repair wrong notes (such as playing the wrong note, missing a note or having multiple notes) during the performance, nor can they adjust the rhythm deviation, resulting in obvious performance errors still existing in the beautified audio and video, which affects the overall listening experience.

[0003] In addition, the existing technologies usually only process the audio and ignore the need for synchronous beautification of the performance video. If the beautified audio is directly replaced into the original video, it may cause out-of-sync audio and video due to rhythm or note changes, affecting the viewing experience.

[0004] Therefore, there is an urgent need for a beautification method that can automatically repair performance errors, optimize the rhythm, and ensure audio and video synchronization. Summary of the Invention

[0005] To solve the above technical problems, the technical solution provided by the present invention is as follows: A method, system and readable computer storage medium for beautifying the audio and video of musical instrument performances, including the following steps:

[0006] (1) Transcribe the performance audio into MIDI data, where the MIDI data includes pitch, duration, dynamics and pedal information;

[0007] (2) Determine whether the musical score corresponding to the performance audio is known:

[0008] (2-1) If the musical score is known, align the MIDI data with the musical score through a musical score alignment algorithm to generate an alignment result;

[0009] (2-2) If the musical score is unknown, search for the top n candidate musical scores with the highest matching degree with the MIDI data in the musical score library, and select the musical score with a matching degree exceeding the preset threshold as the alignment musical score;

[0010] (3) Perform beautification processing on the MIDI data based on the alignment result, including at least one of the following:

[0011] (3-1) Wrong note repair: Correct the notes in the MIDI data according to the error types (wrong, miss, redundant) in the alignment result;

[0012] (3-2) Rhythm beautification: Adjust the note start time of the MIDI data according to the tempo fluctuation value in the alignment result;

[0013] (4) Render the beautified MIDI data through a high-quality sound library to generate the target audio file.

[0014] Further, the wrong note repair and rhythm beautification described in step 3 are implemented by any of the following methods:

[0015] Method 1: Directly correct the MIDI data based on the score alignment result;

[0016] Method 2: Convert the MIDI data into a token sequence, input it into a pre-trained large model for wrong note repair and rhythm beautification for processing, and output the beautified MIDI token sequence.

[0017] Further, the execution of wrong note repair and rhythm beautification in step 3 needs to meet the following conditions:

[0018] The proportion of wrong notes played by the user is lower than the preset threshold;

[0019] The user selects to enable the wrong note repair or rhythm beautification function through the interaction interface.

[0020] The advantages of the invention compared with the prior art are as follows:

[0021] 1. Accurately repair wrong notes: Correct wrong notes through score alignment or a large model to improve the performance accuracy.

[0022] 2. Naturally optimize the rhythm: Adjust the note timing to make the performance smoother.

[0023] 3. High-fidelity sound: Render using a professional sound library to achieve a concert hall-level sound effect.

[0024] 4. Audio-video consistency: Dynamically control the beautification amplitude to ensure the synchronization of video actions and audio. Brief Description of the Drawings

[0025] Figure 1 It is a schematic diagram of the user interaction process for the audio beautification function of instrument performance;

[0026] Figure 2 It is a schematic diagram of the user interaction process for the video beautification function of instrument performance;

[0027] Figure 3 It is a schematic diagram of the processing process for the audio beautification of instrument performance;

[0028] Figure 4 It is a schematic diagram of the processing process for the video beautification of instrument performance. Detailed Description of the Invention

[0029] The present invention will be further described in detail below with reference to the accompanying drawings.

[0030] The present invention will be introduced in detail with reference to the accompanying drawings.

[0031] When the present invention is specifically implemented, it provides a method, system and readable computer storage medium for beautifying the audio and video of musical instrument performance, including the following steps:

[0032] (1) Transcribe the performance audio into MIDI data, where the MIDI data includes pitch, duration, velocity and pedal information;

[0033] (2) Determine whether the musical score corresponding to the performance audio is known:

[0034] (2-1) If the musical score is known, align the MIDI data with the musical score through a musical score alignment algorithm to generate an alignment result;

[0035] (2-2) If the musical score is unknown, search for the top n candidate musical scores with the highest matching degree with the MIDI data from the musical score library, and select the musical score with a matching degree exceeding the preset threshold as the alignment musical score;

[0036] (3) Perform beautification processing on the MIDI data based on the alignment result, including at least one of the following:

[0037] (3-1) Wrong note repair: Correct the notes in the MIDI data according to the error types (wrong, miss, redundant) in the alignment result;

[0038] (3-2) Rhythm beautification: Adjust the note start time of the MIDI data according to the tempo fluctuation value in the alignment result;

[0039] (4) Render the beautified MIDI data through a high-quality tone library to generate a target audio file.

[0040] Example 1:

[0041] Piano performance audio beautification (known musical score)

[0042] 1. User input

[0043] The user uploads a piano performance audio file (format: MP3, duration: 2 minutes and 30 seconds), and at the same time provides the standard musical score (MusicXML format) of this piece.

[0044] 2. Audio transcription into MIDI

[0045] The system uses an audio-to-MIDI tool (such as PrettyMIDI) to analyze the uploaded audio and extract the following MIDI information:

[0046] Note sequence (pitch, start time, duration)

[0047] Velocity

[0048] Sustain pedal control signal

[0049] 3. Score alignment

[0050] Align the MIDI data with the score provided by the user and calculate the matching status of each note:

[0051] Right: The pitch and rhythm are consistent with the score (accounting for 85%).

[0052] Wrong: The pitch does not match (e.g., C4 is misplayed as C#4, a total of 5 places).

[0053] Miss: Present in the score but missing in the performance (2 places).

[0054] Redundant: Present in the performance but not marked in the score (3 places).

[0055] 4. MIDI beautification processing

[0056] (1) Wrong note repair

[0057] The system automatically corrects the MIDI data according to the alignment result:

[0058] Replace the 5 Wrong notes with the correct pitch marked in the score.

[0059] Insert the missing notes at the Miss positions (pitch and velocity are generated according to the score).

[0060] Delete the 3 Redundant notes.

[0061] (2) Rhythm beautification

[0062] Analyze the rhythm fluctuations of the original performance:

[0063] The score is marked with a speed of 120 BPM, but the actual performance fluctuates between 112 - 128 BPM.

[0064] The system smoothly adjusts the note start times to make the overall approach 120 BPM (the maximum adjustment range is ±5%).

[0065] 5. Audio generation

[0066] Use FluidSynth to load the "Steinway Grand Piano" sound library and render the beautified MIDI into high-quality MP3 audio.

[0067] 6. Output result

[0068] User audition comparison:

[0069] Original audio: There are obvious wrong notes (such as the dissonant note C#4 at the 45th second) and unstable rhythm.

[0070] Audio after beautification: Wrong notes are corrected, rhythm is stable, and timbre is full.

[0071] Example 2: Beautification of guitar performance video (unknown sheet music)

[0072] 1. User input

[0073] The user uploads a guitar-playing and singing video (MP4 format, 3 minutes long) without providing sheet music.

[0074] 2. Audio extraction and transcription

[0075] Use FFmpeg to extract the audio (WAV format) from the video.

[0076] Generate MIDI data for the guitar track through an audio-to-MIDI tool (only retain the melodic pitch and ignore the vocals).

[0077] 3. Sheet music search and matching

[0078] Search for similar sheet music in the sheet music library (Top 5 candidates):

[0079] Candidate 1: A fragment of "Romance de Amor", matching degree 88%

[0080] Candidate 2: The theme of "Castle in the Sky", matching degree 76%

[0081] ... (The remaining matching degrees < 70%)

[0082] Select "Romance de Amor" with the highest matching degree as the alignment sheet music.

[0083] 4. MIDI beautification

[0084] (1) Wrong note repair

[0085] Correct 3 Wrong notes (for example, F3 at the 1 minute and 12 seconds should be G3).

[0086] Supplement 1 Miss note (the rest note at the 2 minute and 05 seconds was misplayed as a sustained note).

[0087] (2) Rhythm optimization

[0088] The original performance has a relatively large fluctuation in the free rhythm (Rubato), and the system fine-tunes the note durations according to the 80 BPM marked in the sheet music (adjustment range ±3%).

[0089] 5. Audio-video synchronization

[0090] The length of the beautified audio is the same as the original audio (3 minutes) and is directly replaced in the video.

[0091] Because the rhythm adjustment is small (3%), the guitar fingering movements are perfectly matched to the audio.

[0092] 6. Output results

[0093] Users gain:

[0094] Beautify the video: eliminate audio errors, make the rhythm more stable, and make the original video look harmonious.

[0095] Optional output: a version that only beautifies the tone (without correcting wrong notes / rhythm).

[0096] Example 3: Beautification of cello solo (with the help of large model)

[0097] 1. Special scenarios

[0098] The user uploaded an audio clip of an impromptu cello performance (without music score, but with complex embellishments).

[0099] 2. Large model processing

[0100] Convert MIDI data into a token sequence and input it into the pre-trained beautification model:

[0101] Mis-note repair model: Identify and correct off-tune notes (such as dissonances in improvisations).

[0102] Rhythm beautification model: optimize the duration distribution of ornaments to make the flow stronger.

[0103] 3. Results comparison

[0104] Original audio: The rhythm of the improvisation section is loose, and some pitches deviate from the main tune.

[0105] After beautification: The improvisational style is retained, but the pitch is more harmonious and the rhythm is more dynamic.

[0106] As a further elaboration of the present invention, the wrong note repair and rhythm beautification in step 3 are achieved by any of the following methods:

[0107] Method 1: Directly modify MIDI data based on score alignment results;

[0108] Method 2: Convert MIDI data into a token sequence, input the pre-trained wrong note repair model and rhythm beautification model for processing, and output the beautified MIDI token sequence.

[0109] As a further elaboration of the present invention, the execution of wrong note repair and rhythm beautification in step 3 needs to meet the following conditions:

[0110] The proportion of incorrect notes played by the user is lower than the preset threshold;

[0111] The user selects to enable the incorrect note repair or rhythm beautification function through the interaction interface.

[0112] A method for beautifying an instrument performance video, comprising the following steps:

[0113] (1) Extract the original audio from the performance video;

[0114] (2) Perform beautification processing on the original audio by using the method described in any one of claims 1-3 to generate beautified audio;

[0115] (3) Replace the original audio in the original video with the beautified audio and ensure that the audio durations are the same.

[0116] For a method for beautifying an instrument performance video, before replacement in step 3, it is necessary to judge the difference degree between the beautified audio and the original audio. If the difference degree exceeds the preset threshold, only perform timbre beautification and background noise denoising processing.

[0117] An audio-visual beautification system, comprising:

[0118] An audio transcription module: used to convert the audio extracted from the performance audio or video into MIDI data;

[0119] A score alignment module: used to match the MIDI data with the score library and generate an alignment result;

[0120] A beautification processing module: used to perform incorrect note repair and rhythm beautification, supporting the score alignment method or the large model generation method;

[0121] A rendering output module: used to generate audio from the beautified MIDI data or replace it in the original video.

[0122] An audio-visual beautification system, the beautification processing module includes:

[0123] (1) An incorrect note repair sub-module: correcting notes based on the alignment result or an incorrect note repair large model;

[0124] (2) A rhythm beautification sub-module: adjusting the note timings based on the alignment result or a rhythm beautification large model.

[0125] A computer-readable storage medium, on which a computer program is stored, and when the program is executed by a processor, it implements the method described in any one of claims 1-5.

[0126] The user interaction process in the present invention:

[0127] (1) The user plays the music according to the score, and the system recognizes the sound of the user's performance in real time. After the performance is completed, the system automatically submits the audio and provides feedback to the user on the scores of various dimensions. The user can choose whether to perform audio beautification processing based on the scores and his or her own wishes. The audio beautification function beautifies the timbre and denoises the background sound by default. There are also two other options for the user to choose from, one is wrong note repair and the other is rhythm beautification. The user can choose neither, only wrong note repair, or both wrong note repair and rhythm beautification. The user clicks to submit the audio beautification task, and the system performs the corresponding beautification process in the background according to the user's selection, and provides feedback to the user on the processing progress through a progress bar. After the processing is completed, the user can click to play the beautified audio and can share the link of the beautified audio.

[0128] (2) Users can directly upload performance videos on their mobile phones or pads for beautification. The video beautification function will also beautify the tone and denoise the background sound by default. There are also two options for users to choose from, one is wrong note repair and the other is rhythm beautification. Users can choose only wrong note repair, or both wrong note repair and rhythm beautification, or neither. When the user clicks to submit the video beautification task, the system will perform the corresponding beautification process in the background according to the user's selection, and feedback the processing progress to the user through a progress bar. After the processing is completed, the user can click to play the beautified video and save it to the album.

[0129] The present invention and its embodiments are described above, and such description is not restrictive. The drawings show only one embodiment of the present invention, and the actual structure is not limited thereto. In short, if ordinary technicians in the field are inspired by it, without departing from the purpose of the invention, they can design a structure and embodiment similar to the technical solution without creativity, which should belong to the protection scope of the present invention.

Claims

1. A method for beautifying the audio of musical instrument performance, characterized in that, It includes the following steps: (1) Transcribe the performance audio into MIDI data, where the MIDI data includes pitch, duration, velocity, and pedal information; (2) Determine whether the sheet music corresponding to the performance audio is known: (2-1) If the sheet music is known, align the MIDI data with the sheet music through a sheet music alignment algorithm to generate an alignment result; (2-2) If the sheet music is unknown, search the sheet music library for the top n candidate sheet music with the highest matching degree with the MIDI data, and select the sheet music with a matching degree exceeding the preset threshold as the alignment sheet music; (3) Perform beautification processing on the MIDI data based on the alignment result, including at least one of the following: (3-1) Wrong note repair: Correct the notes in the MIDI data according to the error types (wrong, miss, redundant) in the alignment result; (3-2) Rhythm beautification: Adjust the note start time of the MIDI data according to the tempo fluctuation value in the alignment result; (4) Render the beautified MIDI data through a high-quality sound library to generate a target audio file.

2. The method for beautifying musical instrument performance audio according to claim 1, wherein The wrong note repair and rhythm beautification in step 3 are implemented in any of the following ways: Method 1: Directly correct the MIDI data based on the sheet music alignment result; Method 2: Convert the MIDI data into a token sequence, input it into a pre-trained wrong note repair large model and a rhythm beautification large model for processing, and output a beautified MIDI token sequence.

3. A method for beautifying musical instrument performance audio according to claim 1, characterized in that, The execution of wrong note repair and rhythm beautification in step 3 needs to meet the following conditions: The proportion of wrong notes played by the user is lower than the preset threshold; The user selects to enable the wrong note repair or rhythm beautification function through the interaction interface.

4. A method for beautifying a musical instrument performance video, characterized in that, It includes the following steps: (1) Extract the original audio from the performance video; (2) Perform beautification processing on the original audio by using the method described in any one of claims 1-3 to generate beautified audio; (3) Replace the original audio in the original video with the beautified audio and ensure that the audio durations are the same.

5. The method according to claim 4, characterized in that, Before replacement in step 3, it is necessary to judge the difference degree between the beautified audio and the original audio. If the difference degree exceeds the preset threshold, only perform timbre beautification and background noise denoising processing.

6. An audio-video beautification system, characterized in that, It includes: Audio transcription module: Used to convert the performance audio or the audio extracted from the video into MIDI data; Sheet music alignment module: Used to match the MIDI data with the sheet music library and generate an alignment result; Beautification processing module: Used to perform wrong note repair and rhythm beautification, supporting the sheet music alignment method or the large model generation method; Rendering output module: Used to generate audio from the beautified MIDI data or replace it into the original video.

7. An audio - video beautification system according to claim 6, wherein, The beautification processing module includes: (1) Wrong note repair sub-module: Correct the notes based on the alignment result or the wrong note repair large model; (2) Rhythm beautification sub-module: Adjust the note timing based on the alignment result or the rhythm beautification large model.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by a processor, it implements the method described in any one of claims 1-5.

Images