Intelligent karaoke system and karaoke video recording method

Abstract

An intelligent karaoke system and a karaoke video recording method, wherein the intelligent karaoke system comprises a sound equipment, a microphone, a projection device, a camera device, an audio acquisition module, a main singer identification module, a lens control module and a video generation module; the main singer identification module is used for identifying the main singer in the current environment, the lens control module is used for controlling the camera device to shoot the picture and shoot the singing expression and action of the main singer identified by the main singer identification module; and the video generation module is used for generating a video file from the picture shot by the camera device. Through the main singer identification and intelligent tracking, the device can automatically highlight the main singer in the main song or solo stage, so that the character is kept in the picture center or the best composition area, and the problems of the subject deviation, the character being too small and the picture lacking the key in the fixed camera recording are improved.

Description

Technical Field

[0001] This invention relates to the field of audio equipment technology, and in particular to an intelligent karaoke system and a karaoke video recording method. Background Technology

[0002] Currently, some karaoke devices, projectors, smart speakers, or karaoke machines with screens can record the karaoke process via mobile phone, external camera, or built-in camera; some video platform apps can also edit, add music, and publish the footage after the user manually records it. Common solutions mainly include: recording the scene with a fixed-angle camera; shooting the scene with a mobile phone and editing it afterward; using a wide-angle camera to cover a large area at once; or shooting in real time with the user's manual operation through live streaming / short video apps.

[0003] In projectors or karaoke projectors, cameras are primarily used for video calls, facial recognition, autofocus, automatic keystone correction, motion control, or simple recording. Even when recording is available, it typically involves continuous recording based on user start / stop commands, with a generally fixed camera angle. The recorded content cannot automatically adapt to changes in song structure, lead singer position, or chorus atmosphere.

[0004] In karaoke entertainment scenarios, there are usually multiple participants, including lead singers, backup singers, onlookers, and dancers. During the verse, users want to highlight the lead singer's performance, while during the chorus or chorus, they want to record the overall interactive atmosphere. Existing devices often cannot identify "who is currently the lead singer," nor can they recognize "when the song enters the chorus or chorus," and they cannot automatically generate video files suitable for social media posting after a single song ends. Therefore, it is necessary to provide an intelligent karaoke system and a karaoke video recording method. Summary of the Invention

[0005] One objective of this invention is to provide an intelligent karaoke system, comprising: audio equipment, microphone, projection equipment, camera equipment, audio acquisition module, lead singer recognition module, lens control module, and video generation module; The audio equipment is used for audio output during karaoke; The microphone is used for audio input during karaoke, and the microphone can communicate with audio equipment. The projection device and camera device are integrated into the audio equipment. The projection device is used to project the audio playback image onto a wall or screen; the camera device is used to record the image of the current karaoke scene in real time. The audio acquisition module is used to acquire sound information from audio equipment, microphones, and the environment. The lead singer recognition module is used to identify the lead singer in the current environment; The lens control module is used to control the camera equipment to capture images and capture the singing expressions and movements of the lead singer identified by the lead singer recognition module. The video generation module is used to generate video files from the footage captured by the camera device.

[0006] Based on the above solution and as a preferred solution, the projection device can also be used to project lyrics, music videos, recording status, final previews, QR codes, or sharing portals.

[0007] Based on the above scheme and as a preferred embodiment of the above scheme, the lead singer recognition module identifies the lead singer based on microphone position information.

[0008] Based on the above scheme and as a preferred embodiment of the above scheme, the lead singer recognition module identifies the lead singer based on the audio energy acquired by the audio acquisition module.

[0009] Based on the above scheme and as a preferred embodiment of the above scheme, the lead singer recognition module identifies the lead singer based on facial expressions and movements captured by the camera device.

[0010] Based on the above scheme and as a preferred embodiment of the above scheme, it also includes a song information recognition module, which is used to identify or obtain the lyrics timeline and music library segment information of the currently playing song.

[0011] Based on the above scheme and as a preferred embodiment of the above scheme, the song information recognition module can identify each singing stage of the current song, and the singing stage includes at least one of the verse stage, chorus stage and chorus stage.

[0012] Based on the above scheme and as a preferred option, when the verse is being filmed, the camera will focus on capturing the lead singer's expressions and movements; when the chorus or chorus is being filmed, the camera will switch to panoramic recording mode, wide-angle mode, or multi-person interactive composition mode.

[0013] Based on the above scheme and as a preferred embodiment of the above scheme, the video generated by the video generation module includes one or more of the following: lead vocal follow-up segment, chorus panoramic segment, lyrics subtitles, song title, singer's name, timestamp, device watermark, cover image, and end credits sharing information.

[0014] The second objective of this invention is to provide a method for recording karaoke videos, comprising the following steps: Step S1: Start karaoke recording: The song information recognition module acquires the song's lyrics timeline, song library segment information, and accompaniment; the audio acquisition module acquires microphone audio information; Step S2: Identify the song structure: Determine the lyrics tags and audio features based on the song information obtained by the song information identification module and the audio information obtained by the audio acquisition module, or determine the various singing stages of the song based on the cloud music library; Step S3: Identify the lead singer using one or a combination of the following methods: 1. Identify the lead singer based on microphone position information; 2. Identify the lead singer based on the audio energy acquired by the audio acquisition module; 3. Identify the lead singer based on facial expressions and movements captured by camera equipment; Step S4, Intelligent Tracking of Lead Vocal Section: Drive the gimbal or electronic cropping to keep the lead vocalist in the center of the frame and record stably; Step S5, Switch to Panoramic View for Chorus: Switch to 360° panoramic or wide-angle chorus view to record the atmosphere of multi-person interaction on site; Step S6, Audio and Video Combination: Align timestamps, mix vocals / accompaniment / ambient sound, overlay lyrics, lighting effects borders, and segmented shots; Step S7: Automatic video generation upon completion of a single track: Generates a video file, cover art, and sharing link / QR code; Step S8: Users download or share with one click to Moments, short video platforms, family sharing, or local photo albums.

[0015] The beneficial effects of the intelligent karaoke system disclosed in this invention are as follows: This intelligent karaoke system, through lead singer recognition and intelligent tracking, can automatically highlight the main singer during the verse or solo phase, keeping the person in the center of the frame or the optimal composition area, thus improving the problems of subject deviation, people being too small, and lack of focus in the image when recording with a fixed camera; by recognizing the chorus or chorus and switching to 360° panoramic recording, the device can record more people and interactive atmosphere during the climax of the song, making the video content more in line with the entertainment and dissemination needs of karaoke parties. Attached Figure Description

[0016] Figure 1 This is a flowchart of a karaoke video recording method according to the present invention. Detailed Implementation

[0017] The present invention will be further described and illustrated below with reference to specific embodiments and the accompanying drawings. It should also be noted that, in order to make the embodiments more detailed, the following embodiments are the best and preferred embodiments, and those skilled in the art can also use other alternative methods to implement some well-known technologies; moreover, the accompanying drawings are only for more specific description of the embodiments and are not intended to specifically limit the present invention.

[0018] It should be noted that embodiments referred to in the specification as "an embodiment," "an embodiment," "an exemplary embodiment," "some embodiments," etc., may include specific features, structures, or characteristics, but not every embodiment necessarily includes that specific feature, structure, or characteristic. Furthermore, when a specific feature, structure, or characteristic is described in connection with an embodiment, implementing such a feature, structure, or characteristic in conjunction with other embodiments (whether explicitly described or not) should be within the knowledge of those skilled in the art.

[0019] Generally, terms can be understood at least partly from their use in context. For example, depending at least partly on the context, the term "one or more" as used herein can be used to describe any feature, structure, or characteristic in a singular sense, or a combination of features, structures, or characteristics in a plural sense. Additionally, the term "based on" can be understood not necessarily to convey an exclusive set of factors, but rather, alternatively, depending at least partly on the context, to allow for the presence of other factors that are not necessarily explicitly described.

[0020] It is understood that the meanings of “on”, “above” and “above” in this disclosure should be interpreted in the broadest sense, such that “on” means not only “directly on” something, but also includes something with an intermediary feature or layer, and that “above” or “above” means not only “on” something, but also includes something “above” or “above” without an intermediary feature or layer.

[0021] Furthermore, spatially related terms such as “below,” “under,” “lower,” “above,” and “upper” are used herein for convenience to describe the relationship of one element or feature to one or more other elements or features, as illustrated in the accompanying drawings. Spatially related terms are intended to cover different orientations in the use or operation of the device other than those depicted in the accompanying drawings. The device may be oriented in other ways, and the spatially related descriptive terms used herein can be interpreted similarly.

[0022] A smart karaoke system includes: audio equipment, microphone, projection equipment, camera equipment, audio acquisition module, lead singer recognition module, lens control module, and video generation module; The audio equipment is used for audio output during karaoke; The microphone is used for audio input during karaoke, and the microphone can communicate with audio equipment. The projection device and camera device are integrated into the audio equipment. The projection device is used to project the audio playback image onto a wall or screen, and can also be used to project lyrics, music videos, recording status, preview of the finished product, QR codes, or sharing portals. The camera device is used to record the scene in the current karaoke session in real time. The camera device can be a rotatable PTZ camera, a wide-angle camera, a fisheye camera, a 360° panoramic camera, a multi-camera array, or a high-resolution camera with electronic cropping capabilities, used to capture live video.

[0023] The audio acquisition module is used to acquire sound information from audio equipment, microphones, and the environment. The lead singer recognition module is used to identify the lead singer in the current environment; it can use one or a combination of the following methods to perform lead singer recognition: 1. Microphone ID-based identification. For example, when a user picks up or activates a wireless microphone, the device sets that microphone as the lead vocal microphone and determines the lead vocal position by combining the direction of the sound source or the user's position.

[0024] 2. Based on audio energy recognition. That is, when a certain direction or microphone input is detected to be significantly higher than other inputs in the human voice frequency band, the corresponding person is identified as the lead singer.

[0025] 3. Based on visual recognition. This means identifying the person singing by observing their face, body shape, mouth opening and closing, microphone holding posture, and the direction they are facing.

[0026] Combining the above methods can improve recognition accuracy. The confidence scores of microphone ID recognition, audio energy recognition, and visual recognition results are calculated, and the object with the highest confidence score is selected as the lead singer.

[0027] The lens control module controls the camera equipment to capture images, including the vocalist's expressions and movements as identified by the vocalist recognition module. After identifying the vocalist, the camera equipment can perform mechanical or electronic tracking. Mechanical tracking involves the camera pan / tilt unit automatically rotating based on the vocalist's position in the frame, keeping them within a preset composition area. Electronic tracking involves the camera capturing a wide field of view or high-resolution image, then cropping, scaling, and stabilizing the image to output a close-up or half-body shot of the vocalist. The two methods can also be used in combination; for example, the pan / tilt unit can handle wide-area tracking, while electronic cropping handles finer composition.

[0028] The video generation module is used to generate video files from footage captured by the camera. By automatically generating video files, cover art, and sharing links after a single song ends, users can download or forward the video without manually editing and exporting, enhancing the instant sharing experience after karaoke.

[0029] This intelligent karaoke system uses lead singer recognition and intelligent tracking to automatically highlight the main singer during verses or solos, keeping the subject in the center of the frame or in the optimal composition area. This improves upon the issues of subject deviation, subjects being too small, and lack of focus in fixed camera recordings. By integrating projection display, recording, intelligent camera movement, automatic video editing, and social sharing into a single device, the projector is upgraded from a simple display device to a karaoke content creation terminal, enhancing the product's intelligence, entertainment value, and differentiation.

[0030] Furthermore, it also includes a song information recognition module, which is used to identify or obtain the lyrics timeline and song library segment information of the currently playing song. Specifically, it includes the following: obtaining the lyrics, accompaniment, timeline, and verse / chorus / interlude / outro tags of the current song, or inferring song segments in real time through audio features. By uniformly timestamping and encoding the vocals, accompaniment, ambient sound, and video, it can reduce problems such as audio-visual asynchrony, lip-syncing errors, and imbalances in the ratio of accompaniment to vocals, thus improving the quality of the final product.

[0031] The song information recognition module can identify each stage of the current song's performance, including at least one of the verse, chorus, and chorus stages. During the verse, the lead singer is identified and the camera is controlled to record. During the chorus, the lens control module switches the camera to 360° panoramic mode, ultra-wide-angle mode, or multi-camera stitching mode to cover more people and space. If a fisheye or 360° lens is used, the device can unfold the panoramic view into equidistant columnar, surround, split-screen, or intelligently cropped images. If multiple ordinary cameras are used, multi-channel image stitching or selection of the optimal wide-angle view can be performed. By recognizing the chorus or chorus and switching to 360° panoramic recording, the device can record more people and interactive atmosphere during the song's climax, making the video content more suitable for the entertainment and dissemination needs of karaoke parties.

[0032] Based on the above, the video generation module generates videos including one or more of the following: lead vocal follow-up segment, chorus panoramic segment, lyrics subtitles, song title, singer's name, timestamp, device watermark, cover image, and end credits sharing information.

[0033] Furthermore, it also includes a display module installed on the audio equipment, which can be used to display the audio equipment's control interface, song playback interface, images captured by the camera equipment, and links to generated video files.

[0034] like Figure 1 As shown, a method for recording karaoke videos includes the following steps: Step S1: Start karaoke recording: The song information recognition module acquires the song's lyrics timeline, song library segment information, and accompaniment; the audio acquisition module acquires microphone audio information; Step S2: Identify the song structure: Determine the lyrics tags and audio features based on the song information obtained by the song information identification module and the audio information obtained by the audio acquisition module, or determine the various singing stages of the song based on the cloud music library; Step S3: Identify the lead singer using one or a combination of the following methods: 1. Identify the lead singer based on microphone position information; 2. Identify the lead singer based on the audio energy acquired by the audio acquisition module; 3. Identify the lead singer based on facial expressions and movements captured by camera equipment; Step S4, Intelligent Tracking of Lead Vocal Section: Drive the gimbal or electronic cropping to keep the lead vocalist in the center of the frame and record stably; Step S5, Switch to Panoramic View for Chorus: Switch to 360° panoramic or wide-angle chorus view to record the atmosphere of multi-person interaction on site; Step S6, Audio and Video Combination: Align timestamps, mix vocals / accompaniment / ambient sound, overlay lyrics, lighting effects borders, and segmented shots; Step S7: Automatic video generation upon completion of a single track: Generates a video file, cover art, and sharing link / QR code; Step S8: Users download or share with one click to Moments, short video platforms, family sharing, or local photo albums.

[0035] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit the scope of protection of the present invention. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the essence and scope of the technical solutions of the present invention.

Claims

1. An intelligent karaoke system, characterized in that, include: Audio equipment, microphones, projection equipment, camera equipment, audio acquisition module, lead singer recognition module, lens control module, and video generation module; The audio equipment is used for audio output during karaoke; The microphone is used for audio input during karaoke, and the microphone can communicate with audio equipment. The projection device and camera device are integrated into the audio equipment. The projection device is used to project the audio playback image onto a wall or screen; the camera device is used to record the image of the current karaoke scene in real time. The audio acquisition module is used to acquire sound information from audio equipment, microphones, and the environment. The lead singer recognition module is used to identify the lead singer in the current environment; The lens control module is used to control the camera equipment to capture images and capture the singing expressions and movements of the lead singer identified by the lead singer recognition module. The video generation module is used to generate video files from the footage captured by the camera device.

2. The intelligent karaoke system as described in claim 1, characterized in that: The projection device can also be used to project lyrics, music videos, recording status, final previews, QR codes, or sharing links.

3. The intelligent karaoke system as described in claim 1, characterized in that: The lead singer recognition module identifies the lead singer based on microphone position information.

4. The intelligent karaoke system as described in claim 1, characterized in that: The lead singer recognition module identifies the lead singer based on the audio energy acquired by the audio acquisition module.

5. The intelligent karaoke system as described in claim 1, characterized in that: The lead singer recognition module identifies the lead singer based on facial expressions and movements captured by the camera device.

6. The intelligent karaoke system as described in claim 1, characterized in that: It also includes a song information recognition module, which is used to identify or obtain the lyrics timeline and music library segment information of the currently playing song.

7. The intelligent karaoke system as described in claim 6, characterized in that: The song information recognition module can identify each singing stage of the current song, including at least one of the verse stage, chorus stage, and chorus stage.

8. The intelligent karaoke system as described in claim 7, characterized in that: During the verse, the camera will focus on capturing the lead singer's expressions and movements; during the chorus or chorus, the camera will switch to panoramic recording mode, wide-angle mode, or multi-person interactive composition mode for shooting.

9. The intelligent karaoke system as described in claim 1, characterized in that: The video generated by the video generation module includes one or more of the following: lead vocal follow-up, chorus panoramic, lyrics subtitles, song title, singer's name, timestamp, device watermark, cover image, and end credits sharing information.

10. A method for recording karaoke videos, characterized in that, Includes the following steps: Step S1: Start karaoke recording: The song information recognition module acquires the song's lyrics timeline, song library segment information, and accompaniment; the audio acquisition module acquires microphone audio information; Step S2: Identify the song structure: Determine the lyrics tags and audio features based on the song information obtained by the song information identification module and the audio information obtained by the audio acquisition module, or determine the various singing stages of the song based on the cloud music library; Step S3: Identify the lead singer using one or a combination of the following methods:

1. Identify the lead singer based on microphone position information; 2. Identify the lead singer based on the audio energy acquired by the audio acquisition module; 3. Identify the lead singer based on facial expressions and movements captured by camera equipment; Step S4, Intelligent Tracking of Lead Vocal Section: Drive the gimbal or electronic cropping to keep the lead vocalist in the center of the frame and record stably; Step S5, Switch to Panoramic View for Chorus: Switch to 360° panoramic or wide-angle chorus view to record the atmosphere of multi-person interaction on site; Step S6, Audio and Video Combination: Align timestamps, mix vocals / accompaniment / ambient sound, overlay lyrics, lighting effects borders, and segmented shots; Step S7: Automatic video generation upon completion of a single track: Generates a video file, cover art, and sharing link / QR code; Step S8: Users download or share with one click to Moments, short video platforms, family sharing, or local photo albums.

Images