A karaoke video generation method

Abstract

A karaoke video generation method, comprising the following steps: step S1, video type selection; step S2, material collection; step S3, material processing; step S4, establishing a digital feature file; step S5, song content analysis; step S6, video generation. This karaoke video generation method can automatically generate karaoke videos in real time while karaoke, that is, video, audio recording and scene selection can be completed, and corresponding videos can be generated according to the rhythm of the song, greatly improving the user experience.

Description

Technical Field

[0001] This invention relates to the field of audio equipment technology, and in particular to a method for generating karaoke videos. Background Technology

[0002] Existing karaoke products typically include a microphone, speakers, a display screen, or a camera, enabling them to play accompaniment, display lyrics, and record the user's live audio or video while they sing. Some mobile karaoke apps or karaoke devices with screens can also apply beauty filters, add subtitles, reverb, and edit the recordings, allowing users to save or share the processed content to social media platforms.

[0003] On the other hand, existing AI video generation platforms can generate virtual scene videos based on text, images, or portraits. Some platforms support digital humans, face swapping, lip-syncing, voice cloning, or templated MV generation. However, these platforms are usually independent of the karaoke device. Users need to export audio or video from the karaoke device first, and then upload it to an external platform to complete portrait processing, audio processing, scene selection, video generation, and downloading and sharing separately.

[0004] Existing technologies include solutions such as using pre-made MV backgrounds, green screen keying, or static templates to replace home backgrounds. However, these primarily involve overlaying the user's image onto a fixed background, making it difficult to automatically generate a complete video with continuous shots, stage lighting, audience atmosphere, and professional camera work based on the song's rhythm, lyrics, user actions, and singing state. Therefore, it is necessary to provide a method for generating karaoke videos. Summary of the Invention

[0005] The purpose of this invention is to provide a method for generating karaoke videos, comprising the following steps: Step S1, Video Type Selection: Select the song and target video type on the karaoke device terminal; Step S2, Material Acquisition: While the karaoke device terminal is playing the accompaniment of a song and displaying the lyrics, the user's live singing video and audio are simultaneously acquired, and a unified time reference is written for the acquired video data, audio data, and the accompaniment and lyrics of the song. Step S3, Material Processing: Perform face detection, human body detection and background segmentation on the captured live performance video, perform noise reduction, echo cancellation and voice extraction on the performance audio, and perform audio-visual synchronization correction. Step S4, establish a digital feature profile: After obtaining user authorization, extract user avatar features and / or human motion features from the live performance video, extract user voice features from the performance audio, and establish a digital feature profile associated with the user account; Step S5, Song Content Analysis: Perform content analysis on the song and determine the rhythms of the song based on the song content; Step S6, Video Generation: Based on the target video type selected by the user, the analysis results of the song content, and the user characteristics in the digital feature file, generate a target karaoke video that retains the user's identity characteristics and singing voice characteristics.

[0006] Based on the above scheme and as a preferred option, the target video types in step S1 include concerts, professional music videos, music festivals, bar live performances, animation, traditional Chinese style videos, futuristic technology videos, and narrative music videos.

[0007] Based on the above scheme and as a preferred embodiment of the above scheme, in step S2, the user's live singing video is captured by a camera device, and the user's live singing audio is captured by a microphone.

[0008] Based on the above scheme and as a preferred embodiment of the above scheme, the step S4 of establishing a digital feature profile includes: extracting avatar, facial key points, expression and action features from the singing video, extracting authorized voiceprint or timbre features from the singing audio, and forming a digital feature profile associated with the user account.

[0009] Based on the above scheme and as a preferred option, step S5, which determines the rhythms of the song according to the song content, includes determining the intro, verse, chorus, interlude, climax, and ending based on the semantics of the lyrics, the timeline of the lyrics, the beat, the tempo, the paragraph structure, and the emotional changes.

[0010] Based on the above scheme and as a preferred embodiment of the above scheme, step S6 further includes a singing synchronization step: using the user's actual singing audio and lyrics phonemes as driving signals to synchronize the lip movements of the person in the digital feature file with the singing content.

[0011] Based on the above scheme and as a preferred option, the method further includes step S7, video detection: performing face consistency detection, lip-sync detection, audio-visual time deviation detection, character edge detection, image continuity detection, and audio quality detection on the generated result; if the preset threshold is not met, the local segment is automatically regenerated, the timeline is adjusted, or the user is prompted to select other materials.

[0012] Based on the above scheme and as a preferred option, step S8 is also included: video settings: after the karaoke video is generated, the user can set the template, scene, clothing, lens, subtitle style, cover, segment order, and music mixing parameters on the karaoke device screen.

[0013] Based on the above scheme and as a preferred option, the set fragment can be regenerated and recombined with the unset fragment.

[0014] Based on the above solution and as a preferred solution, step S9 is also included: video sharing: after selecting to share a video on the karaoke device screen, a corresponding sharing link is automatically generated on the karaoke device screen.

[0015] The beneficial effects of the karaoke video generation method disclosed in this invention are: this karaoke video generation method can automatically generate karaoke videos in real time while singing karaoke, that is, it can complete video and audio recording and scene selection, and can generate corresponding videos according to the rhythm of the song, which greatly improves the user experience. Attached Figure Description

[0016] Figure 1 This is a flowchart of a karaoke video generation method according to the present invention.

[0017] As shown in the figure, specific structures and devices are labeled in the figure to clearly illustrate the structure of the embodiments of the present invention. However, this is only for illustrative purposes and is not intended to limit the present invention to the specific structure, device and environment. Those skilled in the art can adjust or modify these devices and environments according to specific needs, and such adjustments or modifications are still included in the scope of the appended claims. Detailed Implementation

[0018] The present invention will be further described and illustrated below with reference to specific embodiments and the accompanying drawings. It should also be noted that, in order to make the embodiments more detailed, the following embodiments are the best and preferred embodiments, and those skilled in the art can also use other alternative methods to implement some well-known technologies; moreover, the accompanying drawings are only for more specific description of the embodiments and are not intended to specifically limit the present invention.

[0019] It should be noted that embodiments referred to in the specification as "an embodiment," "an embodiment," "an exemplary embodiment," "some embodiments," etc., may include specific features, structures, or characteristics, but not every embodiment necessarily includes that specific feature, structure, or characteristic. Furthermore, when a specific feature, structure, or characteristic is described in connection with an embodiment, implementing such a feature, structure, or characteristic in conjunction with other embodiments (whether explicitly described or not) should be within the knowledge of those skilled in the art.

[0020] Generally, terms can be understood at least partly from their use in context. For example, depending at least partly on the context, the term "one or more" as used herein can be used to describe any feature, structure, or characteristic in a singular sense, or a combination of features, structures, or characteristics in a plural sense. Additionally, the term "based on" can be understood not necessarily to convey an exclusive set of factors, but rather, alternatively, depending at least partly on the context, to allow for the presence of other factors that are not necessarily explicitly described.

[0021] It is understood that the meanings of “on”, “above” and “above” in this disclosure should be interpreted in the broadest sense, such that “on” means not only “directly on” something, but also includes something with an intermediary feature or layer, and that “above” or “above” means not only “on” something, but also includes something “above” or “above” without an intermediary feature or layer.

[0022] Furthermore, spatially related terms such as “below,” “under,” “lower,” “above,” and “upper” are used herein for convenience to describe the relationship of one element or feature to one or more other elements or features, as illustrated in the accompanying drawings. Spatially related terms are intended to cover different orientations in the use or operation of the device other than those depicted in the accompanying drawings. The device may be oriented in other ways, and the spatially related descriptive terms used herein can be interpreted similarly.

[0023] like Figure 1 As shown, a method for generating karaoke videos includes the following steps: Step S1, Video Type Selection: Select the song and target video type on the karaoke device terminal. The target video type in Step S1 includes concert, professional MV, music festival, bar live, animation, ancient style, future technology, drama MV, movie clip, game world, city night scene, multi-person chorus, or drama video automatically generated based on lyrics.

[0024] Step S2, Material Acquisition: While the karaoke device terminal is playing the accompaniment and displaying the lyrics, the user's live singing video and audio are simultaneously acquired, and a unified time reference is written for the acquired video data, audio data, and the accompaniment and lyrics of the song. Specifically, the user's live singing video can be acquired using a camera device, which can be a built-in camera above the screen, an external USB camera, a mobile phone camera, multiple cameras, or a rotatable camera; multiple cameras can generate multi-camera shots. The user's live singing audio is acquired using a microphone, which can be a wireless microphone, a wired microphone, an array microphone, or an external audio device; the user's mixed audio can be used, or the dry audio, accompaniment, and ambient sound can be acquired separately and then mixed.

[0025] Step S3, Material Processing: Perform face detection, human body detection and background segmentation on the captured live performance video, perform noise reduction, echo cancellation and voice extraction on the performance audio, and perform audio-visual synchronization correction. Step S4, Establish a digital feature profile: After obtaining user authorization, extract user avatar features and / or human motion features from the live performance video, and extract user voice features from the performance audio to establish a digital feature profile associated with the user account. Specifically, extract avatar, facial key points, expression, and motion features from the performance video, and extract authorized voiceprint or timbre features from the performance audio to form a digital feature profile associated with the user account. The user's digital feature profile can be stored locally, in the cloud, on a removable storage device, or in a secure area, or it can be temporarily extracted and used in a single generation task without long-term storage.

[0026] Step S5, Song Content Analysis: Perform content analysis on the song and determine the rhythms of the song based on the song content; specifically, determine the intro, verse, chorus, interlude, climax and ending based on the semantics of the lyrics, the timeline of the lyrics, beat, tempo, paragraph structure and emotional changes.

[0027] Step S6, Video Generation: Based on the target video type selected by the user, the song content analysis results, and the user characteristics in the digital feature file, a target karaoke video that retains the user's identity characteristics and singing voice characteristics is generated. This also includes a singing synchronization step: using the user's actual singing audio and lyrics phonemes as driving signals, the lip movements in the digital feature file are synchronized with the singing content.

[0028] Step S7, Video Detection: Perform face consistency detection, lip-sync detection, audio-visual time deviation detection, character edge detection, image continuity detection, and audio quality detection on the generated results; if the preset threshold is not met, automatically regenerate the local segment, adjust the timeline, or prompt the user to select other materials.

[0029] Step S8, Video Settings: After the karaoke video is generated, the user can set the template, scene, clothing, camera angles, subtitle style, cover, segment order, and music mixing parameters on the karaoke device screen. The set segments can be regenerated and then recombined with segments that have not been set.

[0030] Step S9, Video Sharing: After selecting to share a video on the karaoke device screen, the corresponding sharing link will be automatically generated on the karaoke device screen.

[0031] This karaoke video generation method can automatically generate karaoke videos in real time while singing karaoke. It can complete video and audio recording and scene selection, and can generate corresponding videos according to the rhythm of the song, which greatly improves the user experience.

[0032] In addition, it also has the following technical effects: Balancing realism with professional visual effects: The digital character is driven by the user's real singing audio and facial movements, fully preserving the user's original voice and identity characteristics. At the same time, it generates concerts, story music videos, and professional virtual stages, solving the problem of monotonous visuals in ordinary karaoke recordings.

[0033] High audio-visual matching: The song's beat, lyrics, lip movements, and camera lighting are bound together with a unified global timeline. The rendering script is automatically planned based on the song's semantics, and the camera transitions, special effects, lip movements and singing are highly consistent with each other without any obvious sense of disconnect.

[0034] Efficient local regeneration mechanism: When there are defects in the quality of the material or when the user makes editing modifications, only the corresponding segment is regenerated, without the need for overall re-rendering, which reduces the consumption of terminal computing power and shortens the generation waiting time.

[0035] Comprehensive user digital feature privacy authorization and control: Establish digital feature profiles that are bound to accounts, have authorization periods, and have scope of use, support one-time collection for multiple creations, and allow authorization to be withdrawn and feature data to be deleted at any time, balancing the convenience of creation with the security of user biometrics.

[0036] Automatically adapts to distribution across multiple social media platforms: Automatically matches video ratios, encoding, and cover specifications for various short video platforms, enabling one-click publishing and streamlining the entire process from content creation to social media dissemination.

[0037] Flexible deployment architecture: Supports local, cloud, and hybrid AI rendering, adaptable to various usage scenarios such as low-computing-power home karaoke machines, high-computing-power large-screen devices, and unstable networks.

[0038] Highly scalable: Supports solo / multi-person chorus, multi-camera shooting, various digital human reconstruction models, and multi-format output, adapting to diverse creative needs such as short videos, long music videos, and dynamic covers.

[0039] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit the scope of protection of the present invention. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the essence and scope of the technical solutions of the present invention.

Claims

1. A method for generating karaoke videos, characterized in that, Includes the following steps: Step S1, Video Type Selection: Select the song and target video type on the karaoke device terminal; Step S2, Material Acquisition: While the karaoke device terminal is playing the accompaniment of a song and displaying the lyrics, the user's live singing video and audio are simultaneously acquired, and a unified time reference is written for the acquired video data, audio data, and the accompaniment and lyrics of the song. Step S3, Material Processing: Perform face detection, human body detection and background segmentation on the captured live performance video, perform noise reduction, echo cancellation and voice extraction on the performance audio, and perform audio-visual synchronization correction. Step S4, establish a digital feature profile: After obtaining user authorization, extract user avatar features and / or human motion features from the live performance video, extract user voice features from the performance audio, and establish a digital feature profile associated with the user account; Step S5, Song Content Analysis: Perform content analysis on the song and determine the rhythms of the song based on the song content; Step S6, Video Generation: Based on the target video type selected by the user, the analysis results of the song content, and the user characteristics in the digital feature file, generate a target karaoke video that retains the user's identity characteristics and singing voice characteristics.

2. The karaoke video generation method as described in claim 1, characterized in that: The target video types in step S1 include concerts, professional music videos, music festivals, bar live performances, anime, traditional Chinese style videos, futuristic technology videos, and narrative music videos.

3. The karaoke video generation method as described in claim 1, characterized in that: In step S2, the user's live singing video is captured by a camera device, and the user's live singing audio is captured by a microphone.

4. The karaoke video generation method as described in claim 1, characterized in that: The step S4 of establishing a digital feature profile includes: extracting avatar, facial key points, expression and action features from the singing video, extracting authorized voiceprint or timbre features from the singing audio, and forming a digital feature profile associated with the user account.

5. The karaoke video generation method as described in claim 1, characterized in that: In step S5, determining the rhythms of the song based on its content includes: determining the intro, verse, chorus, interlude, climax, and ending based on the semantics of the lyrics, the timeline of the lyrics, the beat, the tempo, the paragraph structure, and the emotional changes.

6. The karaoke video generation method as described in claim 1, characterized in that, Step S6 also includes a singing synchronization step: using the user's actual singing audio and lyrics phonemes as driving signals to synchronize the lip movements of the person in the digital feature file with the singing content.

7. The karaoke video generation method as described in claim 1, characterized in that, It also includes step S7, video detection: performing face consistency detection, lip-sync detection, audio-visual time deviation detection, character edge detection, image continuity detection, and audio quality detection on the generated results; if the preset threshold is not met, the local segment will be automatically regenerated, the timeline will be adjusted, or the user will be prompted to select other materials.

8. The karaoke video generation method as described in claim 7, characterized in that: It also includes step S8, video settings: After the karaoke video is generated, the user can set the template, scene, clothing, lens, subtitle style, cover, segment order, and music mixing parameters on the karaoke device screen.

9. The karaoke video generation method as described in claim 8, characterized in that: The set fragment can be regenerated and recombined with the unset fragment.

10. The karaoke video generation method as described in claim 8, characterized in that: It also includes step S9, video sharing: After selecting to share a video on the karaoke device screen, the corresponding sharing link is automatically generated on the karaoke device screen.

Images