Audio-visual synchronous singing assisting method and terminal

Abstract

The application provides a singing picture synchronization singing assisting method and a terminal. The method comprises the following steps: obtaining audio data and video data of a song; separating real-time singing voice and accompaniment of the audio data to obtain a original singing voice track and predict a processing time length of the real-time singing voice and accompaniment separation process; applying active delay compensation to the video data of the song according to the processing time length to obtain a target video frame and output the target video frame. The application realizes active prediction of the real-time singing voice separation processing delay and accurate compensation of the video picture. The processing time length is predicted, and active delay is applied to the video frame, so that the target video frame and the audio after the separation processing are accurately aligned in time sequence, and the consistency of the mouth shape and the voice in the singing process is ensured. The application actively compensates from the video side, and shifts the control object of the audio and video synchronization from the audio side to the video side, accurately resets the time sequence deviation of the audio and the picture, and provides a professional singing experience for the user.

Description

Technical Field

[0001] This invention relates to the field of audio and video processing technology, and in particular to a method and terminal for synchronized audio and video singing. Background Technology

[0002] Existing intelligent singing assistance solutions mainly include pure accompaniment mode and original vocal assistance mode. Pure accompaniment mode lacks singing reference, making it easy for beginners to sing off-key; original vocal assistance mode plays the original vocals at a constant volume, which can cause auditory interference for advanced users. Neither mode can adaptively adjust the guidance intensity according to the user's singing level.

[0003] In terms of intelligent vocal assistance, traditional systems rely on preset voice information to switch playback at pre-defined times, failing to dynamically adjust the original vocal output based on the user's actual singing state. Furthermore, during quiet sections like transitions and interludes, the original vocals may be accidentally triggered by the user's breathing, resulting in abrupt auditory transitions. Regarding digital pitch shifting, most methods shift the pitch of the entire output audio, easily introducing sound quality degradation, and cannot perform independent pitch shifting for the separated, pure vocals.

[0004] Furthermore, before the accompanying vocals, real-time vocal separation of the original singer's voice and the accompaniment is required. During this real-time vocal separation process, the inference time of the deep learning model causes the audio to lag behind the video. Existing solutions only compensate for the delay in the audio link, which cannot fundamentally eliminate timing discrepancies and severely weakens the real-time performance experience. Summary of the Invention

[0005] The technical problem to be solved by the present invention is to provide a method and terminal for audio-visual synchronized singing, which can actively compensate for the processing delay of real-time voice separation and realize audio-visual synchronization.

[0006] To solve the above-mentioned technical problems, the technical solution adopted by the present invention is as follows: A method for synchronized audio and video singing, characterized by comprising: Obtain the audio and video data of the song; The audio data is subjected to real-time vocal separation and accompaniment separation to obtain the original vocal track, and the processing time of the real-time vocal and accompaniment separation process is predicted. Based on the processing time, active delay compensation is applied to the video data of the song to obtain the target video frame and output it.

[0007] To solve the above-mentioned technical problems, another technical solution adopted by the present invention is as follows: A synchronized audio-visual singing aid terminal includes a memory, a processor, and a computer program stored in the memory and running on the processor. When the processor executes the computer program, it implements the various steps of the aforementioned synchronized audio-visual singing aid method.

[0008] The beneficial effects of this invention are as follows: By detecting the processing time of the real-time vocal and accompaniment separation process and applying active delay compensation to the video data output, the target video frame and the separated audio are aligned in time, solving the audio-visual asynchrony problem caused by real-time vocal separation; by actively detecting the processing time of the separation module and using this time as the basis for video delay compensation, rather than passively waiting or ignoring the delay, a closed loop is formed between processing time detection and delay compensation. The longer the processing time, the longer the waiting time for the video frame, and finally the time difference between the target video frame and the audio is precisely reduced to zero; this invention performs delay compensation on the video frame, fundamentally eliminating audio-visual timing deviation, providing a quantitative basis for detecting the processing time, and providing a synchronous execution means for applying active delay compensation to the video frame. The two work together to achieve optimization from audio processing priority to audio-visual co-synchronization, ensuring the consistency between the displayed image and audio during the performance and improving the user's singing experience. Attached Figure Description

[0009] Figure 1 This is a flowchart of a method for synchronized audio-visual singing according to an embodiment of the present invention; Figure 2 This is a flowchart illustrating the information interaction and processing of a method for synchronized audio-visual singing assistance according to an embodiment of the present invention. Figure 3 This is a schematic diagram of a terminal for a synchronized audio-visual singing aid method according to an embodiment of the present invention. Detailed Implementation

[0010] To explain in detail the technical content, objectives, and effects of the present invention, the following description is provided in conjunction with the embodiments and accompanying drawings.

[0011] Existing intelligent singing assistance solutions have the following three technical limitations: 1. The bootloader mode is rigid and cannot adapt to changes. Existing intelligent vocal assistance solutions mainly include a pure accompaniment mode and a vocal rendition mode. The pure accompaniment mode lacks a singing reference, making it easy for beginners to sing off-key; the vocal rendition mode plays the original vocals at a constant volume, which can be auditoryly disruptive to advanced users. Neither mode can adaptively adjust the guidance intensity based on the user's singing level. When using intelligent vocal assistance, traditional systems rely on preset voice information to switch playback at pre-set times, failing to dynamically adjust the original vocal output based on the user's actual singing state. Furthermore, during transitions and interludes where the original vocals are silent, the user's breathing may accidentally trigger the original vocals to fill in, creating an abrupt auditory experience.

[0012] 2. Digital pitch shifting can easily introduce sound quality damage. Existing digital pitch shifting methods mostly process the accompaniment and vocals as a whole, lacking independent operation for the vocal track, which easily introduces perceptible sound quality degradation.

[0013] III. Real-time processing suffers from audio and video timing deviations. In real-time voice separation, the inference time of the deep learning model causes the audio to lag behind the video. Existing solutions only compensate for the delay in the audio link, which cannot fundamentally eliminate timing deviations and severely degrades the real-time singing experience.

[0014] To at least address the aforementioned problems, this disclosure provides a method and terminal for synchronized audio-visual singing assistance. First, the audio and video data of the song are acquired. The audio data is then separated into vocal and accompaniment tracks in real time to obtain the original vocal track, and the processing time for this separation is predicted. Based on this time, active delay compensation is applied to the video data to align the target video frame with the separated audio in time. In this way, precise time alignment between the video frame and the separated audio can be achieved, ensuring consistency between lip movements and vocals during singing. The following details one such method for synchronized audio-visual singing assistance; please refer to... Figure 1 The method 100 includes steps 110 to 130.

[0015] Step 110: Obtain the audio and video data of the song.

[0016] For example, when a user selects a song through a singing app on a smart device, the system retrieves the corresponding audio and video data from local storage or the network. The audio data includes a mixed audio track of the original vocals and accompaniment, and the video data includes video footage or lyrics subtitles synchronized with the song.

[0017] Step 120: Perform real-time separation of vocals and accompaniment on the audio data to obtain the original vocal track, and predict the processing time of the real-time separation process.

[0018] For example, the system will separate the acquired audio data into vocals and accompaniment, and output independent original vocal tracks and accompaniment tracks; at the same time, the system will predict the processing time required for this separation process in real time based on the current device's computing resources.

[0019] Step 130: Apply active delay compensation to the video data of the song according to the processing time to obtain the target video frame and output it.

[0020] For example, the system uses the processing time predicted in step 120 as the basis for active delay compensation of video data, applies the same delay to the output of video data, and makes the video frame wait for the audio separation processing to be completed and output synchronously, so as to obtain the target video frame and output it, thereby ensuring that the final displayed picture and the separated audio heard by the user are precisely aligned in time.

[0021] As described above, by acquiring the audio and video data of the song, a complete audio-visual material foundation is provided for the singing assistance; by performing real-time separation of vocals and accompaniment from the audio data, the original vocal track is obtained, achieving decoupling between the original vocals and accompaniment, providing a prerequisite for subsequent independent adjustment of the original vocal volume; by predicting the processing time of the real-time vocal and accompaniment separation process, a delay quantification basis is provided for audio-visual synchronization; by applying active delay compensation to the video data according to the processing time, the target video frame is obtained and output, aligning the target video frame with the separated audio in time, solving the audio-visual asynchrony problem caused by audio lagging behind video in existing solutions. This invention shifts the control object of delay compensation from the audio side to the video side, forming a closed loop between processing time prediction and video delay compensation. The longer the processing time, the longer the video frame waiting time, ultimately achieving precise zeroing of audio-visual timing deviation, ensuring the consistency between the displayed image and audio during the performance.

[0022] In one alternative implementation, step 120 obtains the original singer's voice track, including steps 121 to 123.

[0023] Step 121: Real-time acquisition of the user's dry audio track and detection of the instantaneous volume of the user's dry audio track.

[0024] For example, the system uses a microphone to capture the user's dry audio track (i.e., the original human voice without reverberation processing) in real time, and continuously calculates the instantaneous volume value of the currently captured dry audio track, using this instantaneous volume value as the basis for determining whether the user is currently singing.

[0025] Step 122: If the instantaneous volume of the user's dry audio track is lower than the preset mute threshold, increase the output volume of the original vocal track to the preset guide volume.

[0026] For example, when the system detects that the instantaneous volume of the user's dry audio track is lower than the preset mute threshold, it indicates that the user is not currently singing or is in a singing gap. The system will gradually increase the output volume of the original vocal track to the preset guide volume, and the original vocal will fill the singing gap to achieve the effect of "original vocal filling in and leading the singing".

[0027] Step 123: If the instantaneous volume of the user's dry audio track is higher than the first preset threshold, reduce the output volume of the original vocal track to mute.

[0028] For example, when the system detects that the instantaneous volume of the user's dry audio track is higher than the first preset threshold, it indicates that the user is currently singing. The system then reduces the output volume of the original vocal track to mute to avoid interference from the original vocals on the user's singing, thus achieving the "user lead vocal" effect.

[0029] As described above, by real-time acquisition of the user's vocal track and detection of its instantaneous volume, the system can accurately determine the user's singing state. When the instantaneous volume of the user's vocal track is lower than a preset mute threshold, the output volume of the original vocal track is increased to a preset guide volume, enabling the original singer to automatically fill in and lead the singing during the user's singing breaks, avoiding any sense of interruption caused by the user's breathing or musical phrase breaks. When the instantaneous volume of the user's vocal track is higher than a first preset threshold, the output volume of the original vocal track is reduced to mute, preventing auditory interference from the original singer to the user's current singing. Through a state switching mechanism based on instantaneous volume, an intelligent duet effect is achieved where the original singer is muted when the user sings and fills in when the user stops, without requiring the user to manually switch modes, thus improving the interactivity and smoothness of the performance.

[0030] In one alternative implementation, after obtaining the original singer's voice track in step 120, steps 1201 and 1202 are also included.

[0031] Step 1201: Real-time detection of whether the original singer's voice track has a valid signal.

[0032] For example, while the system performs real-time separation of vocals and accompaniment to obtain the original vocal track, it continuously monitors the audio signal energy of the original vocal track to determine whether the original vocals are actually present at the current moment. The original vocal track may have different signal states in different sections of the song. For example, in the verses and choruses of a song, the original vocals have a valid signal; in the transitions, interludes, intros, or outros of a song, since the music itself does not contain vocals, the original vocal track does not have a valid signal.

[0033] Step 1202: When it is detected that the original vocal track does not have a valid signal and the duration of the absence of a valid signal reaches the second preset threshold, it is determined that the original vocal track is a section that does not require vocal guidance processing, and the operation of increasing the output volume of the original vocal track to the preset guiding volume is not performed.

[0034] For example, the system detects the signal energy of the original vocal track in real time. When it detects that the original vocal track does not have a valid signal and this state lasts for 200 milliseconds, the system determines that the current segment is a transition, interlude, or other segment in the song where the original vocals do not exist. At this time, even if the user is in a paused state, the system will not perform the operation of increasing the output volume of the original vocal track to the preset guide volume, so as to avoid mistakenly starting the original vocals during the time period when the original vocals are not needed.

[0035] As described above, by detecting whether the original vocal track has a valid signal in real time, the system can accurately identify the actual presence of the original vocals. When the original vocal track is found to lack a valid signal and continuously reaches the second preset threshold, it is determined to be a section that does not require vocal guidance and the original vocal enhancement operation is not performed. This avoids unnecessary vocal enhancement being mistakenly triggered in sections where the original vocals do not exist, such as song transitions or interludes, due to the user's brief breath or musical phrase gaps. This prevents abrupt auditory feedback and improves the accuracy of intelligent vocal enhancement control and the naturalness of the user experience.

[0036] In an alternative implementation, the method 100 further includes steps 140 and 141.

[0037] Step 140: Collect the user's dry vocal track in real time and calculate the real-time singing score based on the user's dry vocal track.

[0038] For example, the system collects the user's dry audio track in real time through a microphone, compares the user's dry audio track with the original singer's audio track separated in step 120, extracts multi-dimensional acoustic features such as pitch, rhythm, technique and emotion based on word-level temporal alignment, calculates scores for dimensions such as pitch matching degree and rhythm matching degree, and generates a real-time singing score through weighted fusion.

[0039] Step 141: When the real-time singing score is between the low score threshold and the high score threshold, the output volume of the original singer's voice track decreases smoothly as the real-time singing score increases.

[0040] For example, the system presets low score thresholds and high score thresholds. When the calculated real-time singing score is between the two, the system dynamically adjusts the output volume of the original singer's vocal track according to the real-time singing score, so that the output volume decreases smoothly as the score increases. The higher the score, the weaker the original singer's guidance, and the lower the score, the stronger the original singer's guidance, thus achieving a smooth transition.

[0041] As described above, by collecting the user's dry vocal track in real time and comparing it with the original singer's vocal track to calculate the real-time singing score, the user's singing level can be objectively quantified. When the real-time singing score is between the low and high score thresholds, the output volume of the original singer's vocal track decreases smoothly as the score increases, achieving a flexible transition in guidance intensity. This avoids the auditory jump caused by sudden changes in volume and ensures that users of different singing levels can obtain a suitable guidance experience.

[0042] In one alternative implementation, step 141 includes step 1411, in which the output volume of the original vocal track decreases smoothly as the real-time performance score increases.

[0043] Step 1411,

[0044] Where V(t) is the output volume of the original vocal track at the current time t. Where is the initial volume or maximum volume, T is the total transition time, and π is the constant pi.

[0045] For example, the system uses a smooth curve decreasing formula to dynamically adjust the output volume of the original vocal track. When the real-time performance score just exceeds the low score threshold, t approaches 0. =1, at this time, V(t)= The original vocal track is output at maximum volume to provide the strongest guidance for users; as the real-time performance score gradually increases, t gradually increases and approaches T. At this point, V(t) approaches 0, and the output volume of the original vocal track gradually decreases to mute. Throughout the transition from t=0 to t=T, the volume decreases smoothly along a curve, with zero slope at both ends. The decrease speed in the middle region is slow at first, then fast, and then slows down again, simulating the auditory effect of the sound fading out naturally.

[0046] As can be seen from the above description, through The formula uses a smooth curve to gradually decrease the output volume of the original vocal track, with the volume decreasing from... The entire change process from 0 to 0 is continuous and smooth, avoiding the abruptness that may be caused by linear decrease, ensuring the naturalness and comfort of the volume transition, and improving the auditory experience of sound output during singing.

[0047] In an alternative implementation, the method 100 further includes steps 150 to 152.

[0048] Step 150: Collect the user's dry vocal track in real time and calculate the real-time singing score based on the user's dry vocal track.

[0049] For example, the system collects the user's dry audio track in real time through a microphone, compares the user's dry audio track with the original singer's audio track separated in step 120, extracts multi-dimensional acoustic features such as pitch and rhythm based on word-level temporal alignment, calculates scores for dimensions such as pitch matching degree and rhythm matching degree, and generates a real-time singing score through weighted fusion.

[0050] Step 151: When the real-time singing score is lower than the low score threshold, increase the output volume of the original singer's voice track to the preset guide volume.

[0051] For example, the system presets low score thresholds and high score thresholds. When the calculated real-time singing score is lower than the low score threshold, it indicates that the user's current singing level is weak. The system will increase the output volume of the original singer's voice track to the preset guide volume, using the original singer as the lead singer to provide strong assistance and help the user sing along.

[0052] Step 152: When the real-time singing score is higher than the high score threshold, reduce the output volume of the original singer's voice track to mute.

[0053] For example, when the calculated real-time singing score is higher than the high score threshold, it indicates that the user's current singing level is high. The system will reduce the output volume of the original vocal track to mute, completely removing the original vocal interference, so that the user can enjoy an immersive singing experience that is almost pure accompaniment.

[0054] As described above, by collecting users' raw vocal tracks in real time and calculating real-time performance scores, the user's singing level can be objectively quantified. When the real-time performance score is below a low score threshold, the output volume of the original vocal track is increased to a preset guiding volume, providing clear original vocal lead singing assistance for users with weaker singing abilities. When the real-time performance score is above a high score threshold, the output volume of the original vocal track is reduced to mute, removing original vocal interference for high-level users. This achieves an adaptive singing assistance mechanism of "strong guidance for low scores and weak interference for high scores," improving the singing experience for users with different singing levels.

[0055] In one alternative implementation, after obtaining the original singer's voice track in step 120, steps 160 to 162 are also included.

[0056] Step 160: When a vocal accompaniment effect is needed, input the separated original vocal track into the pitch shifter.

[0057] For example, after a user selects the accompaniment mode through the interactive interface of a smart terminal, the system uses the separated original vocal track as the object for pitch shifting and inputs it into the pitch shifter. This pitch shifter is built on an improved SoundTouch audio library (LGPL-2.1 protocol), which can independently adjust the pitch without affecting the duration and rhythm, only changing the original vocal tone while keeping the accompaniment track unchanged.

[0058] Step 161: Perform digital pitch shifting on the original vocal track to generate a pitch-shifted vocal track.

[0059] For example, the system determines the direction and magnitude of pitch shifting based on the user-selected vocal effect type. Digital pitch shifting refers to raising or lowering the pitch of the original vocal track, such as raising the pitch of a male voice to a female voice or a child's voice, or lowering the pitch of a female voice to a male voice or a retro effect. When simulating a female or child's voice, the original vocal track is raised by a semitone; when simulating a male voice or a retro effect, the original vocal track is lowered by a semitone. During the pitch shifting process, the duration and rhythm of the original vocal track remain unchanged; only the pitch parameter changes, generating a pitch-shifted vocal track.

[0060] Step 162: Mix and output the pitch-shifted vocal track with the accompaniment track obtained by separating the real-time vocals and accompaniment.

[0061] For example, the system mixes the pitch-shifted vocal track generated in step 161 with the accompaniment track separated in step 120, so that the original vocal track after pitch shifting and the accompaniment track are output synchronously, forming a "accompaniment + pitch-shifted original vocal" accompaniment listening experience, and achieving the accompaniment effect.

[0062] As described above, by inputting the separated original vocal track into the pitch-shifting processor, independent digital pitch-shifting of the pure vocals is achieved, avoiding the sound quality loss caused by overall pitch shifting of the accompaniment and vocals in traditional methods. By raising or lowering the pitch of the original vocal track to generate a pitch-shifted vocal track, and maintaining the same duration and rhythm during the pitch shifting process, various choral effects such as gender and style changes are achieved. By mixing the pitch-shifted vocal track with the accompaniment track and outputting the result, a pitch-shifted original vocal track is created to assist the user in singing, enriching the interactivity and fun of the performance. Simultaneously, because the digital pitch-shifting processing only affects the separated pure vocals, the accompaniment track remains unchanged, avoiding the impact of pitch shifting on the accompaniment's sound quality and ensuring the overall purity of the output sound quality.

[0063] In one alternative implementation, after obtaining the original singer's voice track in step 120, steps 170 to 172 are also included.

[0064] Step 170: When a vocal accompaniment effect is needed, input the separated original vocal track into the pitch shifter.

[0065] For example, after a user selects the accompaniment mode through the interactive interface of a smart terminal, the system uses the original vocal track separated in step 120 as the object of voiceprint conversion processing and inputs it into the pitch shifter. This pitch shifter is built on an improved SoundTouch audio processing library and supports independent voiceprint feature conversion of the input pure vocal track without affecting the audio's duration and rhythm. Since the input is the separated pure vocal track, the voiceprint conversion processing only applies to the original vocal track itself, while the accompaniment track remains unchanged.

[0066] Step 171: Perform voiceprint conversion processing on the original singer's voice track to generate a pitch-shifted voice track based on the target assistant singer's voiceprint.

[0067] For example, the system determines the target for voiceprint conversion based on the user's selected vocal assistant role. When the user chooses to convert the original singer A's voice to the voice of singer B, the system extracts the voiceprint feature parameters of the target singer B (including fundamental frequency distribution, formant position, timbre envelope, etc.), maps the original voiceprint features of the original vocal track to the target voiceprint features through a voiceprint conversion algorithm, and generates a transposed vocal track with the target vocalist's voiceprint. During the voiceprint conversion process, the lyrics, duration, and rhythm of the original vocal track remain unchanged; only the voiceprint features (timbre attributes) change.

[0068] Step 172: Mix and output the pitch-shifted vocal track with the accompaniment track obtained by separating the real-time vocals and accompaniment.

[0069] For example, the system mixes the generated pitch-shifted vocal track with the target singer's voiceprint with the accompaniment track, so that the original vocal track after voiceprint conversion and the accompaniment track are output synchronously, forming a "accompaniment + target voiceprint original singer" singing effect, and realizing the singing effect of role change.

[0070] As described above, by performing voiceprint conversion on the separated original vocal track, a pitch-shifted vocal track with the target backing vocalist's voiceprint is generated. This achieves a timbre conversion from the original singer's voice to the target singer's voice, avoiding the limitation of traditional pitch-shifting methods that only change pitch but not timbre. By mixing the pitch-shifted vocal track with the target voiceprint with the accompaniment track, an effect of the accompaniment and the target singer singing together is created, enriching the diversity and fun of backing vocals. Simultaneously, since the voiceprint conversion only applies to the separated pure vocals, the accompaniment track remains unchanged, ensuring the overall sound quality purity and enhancing the user's singing experience.

[0071] In one alternative implementation, after obtaining the original singer's voice track in step 120, step 173 is also included.

[0072] Step 173: Mix the accompaniment track, the original vocal track with adjusted output volume, or the vocal track with pitch shift, and output them synchronously with the target video frame after active delay compensation.

[0073] For example, the system mixes the accompaniment track separated in step 120 with the original vocal track or the vocal track with adjusted output volume to obtain mixed audio data. Simultaneously, the mixed audio data is synchronized with the target video frame obtained after active delay compensation in step 130, ensuring that the output audio and video are in sync in terms of timing. Finally, the audio is output synchronously through speakers and the display, achieving intelligent audio-visual synchronization for assisted singing.

[0074] As described above, by mixing the accompaniment track, the original vocal track with adjusted output volume, or the vocal track with pitch shift, it is possible to adaptively generate mixed audio containing different original vocal guidance intensities based on the user's actual singing state and level. By synchronizing the output with the target video frame after active delay compensation, the output audio and video images are kept in chronological order, ensuring that the user's visuals and audio are synchronized during the performance. This avoids a decline in experience due to audio-visual timing discrepancies, and achieves a complete audio-visual synchronized intelligent singing assistance solution.

[0075] Please refer to Figure 2This invention can apply the above-mentioned solutions to intelligent singing assistance scenarios, especially those requiring real-time voice separation, audio-visual synchronization compensation, dynamic volume control, and pitch shifting, such as singing, online singing, and music teaching. It includes steps S201 to S207.

[0076] S201. User Audio Acquisition. The system acquires the user's dry audio track (i.e., pure human voice without reverberation processing) in real time through a microphone, which serves as input data for subsequent real-time singing scoring and dynamic volume control. This is equivalent to the user's dry audio acquisition in steps 121 and 140 above.

[0077] S202, AI Real-Time Scoring Module (Multi-Dimensional Scoring of Pitch / Rhythm / Breath). The system compares the user's dry audio track collected in S201 with the separated original vocal track. Based on word-level temporal alignment, it extracts pitch sequence (fundamental frequency F0), note duration, and lyric time alignment information, calculates scores for dimensions such as pitch matching degree, rhythm matching degree, and breath continuity, and continuously updates the real-time performance score using a sliding time window. This is equivalent to steps 140 and 150 above.

[0078] S203, Vocal Separation Module (Original Vocal Track + Accompaniment Track). The system performs real-time vocal and accompaniment separation on the acquired song audio data, and uses a deep learning model to output independent original vocal and accompaniment tracks. Simultaneously, the system predicts the processing time of this separation process under the current device's computing resources and outputs this time to the audio-visual synchronization delay compensation module. This is equivalent to step 120 above.

[0079] S204, Song Vocal Volume Control Module (Dynamically Adjusting Original Vocal Track Volume). Based on the real-time performance score output by the AI ​​real-time scoring module in S202, the system dynamically adjusts the output volume of the original vocal track. When the real-time performance score is below a low threshold, the output volume of the original vocal track is increased to a preset strong guiding volume; when the score is between the low and high thresholds, the volume of the original vocal track decreases smoothly as the score increases; when the score is above the high threshold, the output volume of the original vocal track is muted. This is equivalent to steps 141, 151, and 152 above.

[0080] S205, Vocal Volume Control Module (User Pitch Shifting Settings: Male / Female). When the user selects a chorus effect, the system inputs the separated original vocal track into the pitch shifting processor. Based on the user's selected pitch shifting settings (e.g., raising the pitch by a semitone when simulating a female / child's voice, lowering it by a semitone when simulating a male / retro effect), the system shifts the pitch of the original vocal track, generating a pitch-shifted vocal track. During the pitch shifting process, the duration and rhythm of the original vocal track remain unchanged, while the accompaniment track remains unchanged. This is equivalent to steps 1203 and 1204 above.

[0081] S206, Mixing Output (Adjusted Vocals + Accompaniment Track). The system mixes the accompaniment track separated in S203, the original vocal track after dynamic volume adjustment in S204, or the pitch-shifted vocal track generated in S205 to obtain mixed audio data, which is then output to the speaker device. This is equivalent to the mixing part in step 173 above.

[0082] S207, Audio-visual synchronization delay compensation (compensating for all calculation delays). Based on the processing time predicted by the vocal separation module in S203, the system applies active delay compensation to the video data of the song, ensuring that the video frames are output synchronously after the audio separation processing is completed, guaranteeing that the final output mixed audio and video images are precisely aligned in timing. This is equivalent to step 130 above.

[0083] Through the above steps, user audio is first collected and scored in real time. Simultaneously, vocal separation is performed on the song audio, and the processing time is predicted. Then, the volume of the original vocal track is dynamically adjusted based on the real-time performance score, or pitch shifting is applied according to user settings. Finally, the accompaniment and the adjusted original vocal track (or pitch-shifted vocals) are mixed and output synchronously with the delayed-compensated video footage. This process forms a complete intelligent singing assistance chain, from audio acquisition, vocal separation, intelligent scoring, dynamic volume control / pitch shifting to synchronized audio-visual output, providing a reliable technical foundation and implementation framework for subsequent adaptive singing assistance, intelligent duet singing, and choral effects.

[0084] Please refer to Figure 3 The present invention also provides an audio-visual synchronized singing terminal 300, including a memory 301, a processor 302, and a computer program stored in the memory 301 and executable on the processor 302. When the processor 302 executes the computer program, it implements the various steps of the audio-visual synchronized singing method described above.

[0085] In summary, this invention provides a method and terminal for synchronized audio and video singing assistance. By separating vocals and accompaniment in real time, predicting processing time and actively compensating for video delay, combined with user singing status detection, real-time singing scoring and dynamic volume control, it achieves intelligent singing assistance with synchronized audio and video.

[0086] The system first acquires the audio and video data of the song. It performs real-time separation of vocals and accompaniment in the audio data to obtain the original vocal track and predicts the processing time of the separation process. Based on this time, it applies active delay compensation to the video data, ensuring precise temporal alignment between the target video frame and the separated audio. Simultaneously, the system acquires the user's dry vocal track in real time and detects its instantaneous volume. When the original vocal track is muted during the user's performance, or when the user pauses and a valid signal is present in the original vocal track, the original vocal track fills in, achieving intelligent duet. Based on the comparison between the user's dry vocal track and the original vocal track, a real-time performance score is calculated. When the score is between a low and high threshold, the volume of the original vocal track decreases smoothly as the score increases, achieving adaptive singing assistance with strong guidance for low scores and weak interference for high scores. When a chorus effect is needed, the separated original vocal track undergoes pitch shifting or voiceprint conversion processing to generate a pitch-shifted vocal track, which is then mixed with the user's dry vocal track for output, enriching the interactivity and fun of the performance.

[0087] This invention shifts the control of audio-visual synchronization from the audio side to the video side by predicting the separation processing time and applying active delay compensation to the video frame. This achieves precise zeroing of audio-visual timing deviations, ensuring consistency between the displayed image and audio during performance. Through a state switching mechanism based on instantaneous volume, it achieves an intelligent duet effect where the original vocals are muted when the user sings and filled in when the user stops, eliminating the need for manual mode switching. Dynamic volume control based on real-time performance scoring enables a smooth transition of guidance intensity, avoiding auditory jumps caused by sudden volume changes. Independent pitch shifting processing for the separated pure vocals avoids the sound quality damage caused by overall pitch shifting of accompaniment and vocals in traditional methods. This invention solves the problems of audio-visual asynchrony, inability to adaptively adjust guidance volume according to the user's singing state and level, and pitch shifting sound quality damage in existing technologies. It achieves an intelligent assisted singing solution with audio-visual synchronization, adaptive assisted singing, and high-quality pitch shifting, enhancing the user's singing experience.

[0088] The above description is merely an embodiment of the present invention and does not limit the patent scope of the present invention. Any equivalent modifications made based on the content of the present invention specification and drawings, or direct or indirect applications in related technical fields, are similarly included within the patent protection scope of the present invention.

Claims

1. A method for synchronized audio and video singing, characterized in that, include: Obtain the audio and video data of the song; The audio data is separated into vocals and accompaniment in real time to obtain the original vocal track, and the processing time of the real-time vocal and accompaniment separation process is predicted. Based on the processing time, active delay compensation is applied to the video data of the song to obtain the target video frame and output it.

2. The method for synchronized audio-visual singing according to claim 1, characterized in that, The obtained original vocal track includes: Real-time acquisition of user's dry audio track, and detection of the instantaneous volume of the user's dry audio track; If the instantaneous volume of the user's dry audio track is lower than a preset mute threshold, the output volume of the original vocal track will be increased to a preset guide volume. If the instantaneous volume of the user's dry audio track is higher than a first preset threshold, the output volume of the original vocal track will be reduced to mute.

3. The method for synchronized audio-visual singing assistance according to claim 2, characterized in that, After obtaining the original singer's voice track, the process also includes: Real-time detection of whether the original singer's voice track has a valid signal; When it is detected that the original vocal track does not have the valid signal and the duration of the absence of the valid signal reaches the second preset threshold, it is determined that the original vocal track is a section that does not require introductory vocal processing, and the operation of increasing the output volume of the original vocal track to the preset introductory volume is not performed.

4. The method for synchronized audio-visual singing according to claim 1, characterized in that, Also includes: Real-time acquisition of user's dry vocal tracks, and calculation of real-time vocal scores based on the user's dry vocal tracks; When the real-time singing score is between a low score threshold and a high score threshold, the output volume of the original singer's voice track decreases smoothly as the real-time singing score increases.

5. The method for synchronized audio-visual singing assistance according to claim 4, characterized in that, The output volume of the original vocal track decreases smoothly with increasing real-time performance score, including: Where V(t) is the output volume of the original vocal track at the current time t. Where is the initial volume or maximum volume, T is the total transition time, and π is the constant pi.

6. The method for synchronized audio-visual singing according to claim 1, characterized in that, Also includes: Real-time acquisition of user's dry vocal tracks, and calculation of real-time vocal scores based on the user's dry vocal tracks; When the real-time singing score is lower than the low score threshold, the output volume of the original vocal track is increased to a preset guide volume; When the real-time singing score is higher than the high score threshold, the output volume of the original singer's voice track is reduced to mute.

7. The method for synchronized audio-visual singing assistance according to claim 2, characterized in that, After obtaining the original singer's voice track, the process also includes: When a vocal accompaniment effect is needed, the separated original vocal track is input into the pitch shifter. The original vocal track is digitally pitch-shifted to generate a pitch-shifted vocal track. The pitch-shifted vocal track is mixed and output with the accompaniment track obtained by separating the real-time vocals and accompaniment.

8. The method for synchronized audio-visual singing assistance according to claim 2, characterized in that, After obtaining the original singer's voice track, the process also includes: When a vocal accompaniment effect is needed, the separated original vocal track is input into the pitch shifter. The original singer's voice track is processed by voiceprint conversion to generate a pitch-shifted voice track based on the voiceprint of the target assistant singer. The pitch-shifted vocal track is mixed and output with the accompaniment track obtained by separating the real-time vocals and accompaniment.

9. A method for synchronized audio-visual singing assistance according to any one of claims 2, 4, 6, 7, and 8, characterized in that, After obtaining the original singer's voice track, the process also includes: The accompaniment track, the original vocal track or the pitch-shifted vocal track with the output volume adjusted are mixed and output synchronously with the target video frame after active delay compensation.

10. A synchronized audio-visual singing terminal, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements each step of the audio-visual synchronized singing method according to any one of claims 1 to 9.

Images