Karaoke processing method and device, electronic equipment and storage medium

Abstract

The invention relates to the technical field of audio and video processing, and discloses a karaoke processing method and device, electronic equipment and a storage medium. The method comprises the following steps: acquiring a song signal corresponding to a current song to be karaoke, wherein the song signal comprises an audio signal and an image sequence signal; obtaining a human voice signal and an accompaniment signal in the song according to the audio signal; acquiring an image synchronized with the audio signal according to the image sequence signal; fusing the human voice signal, the accompaniment signal and the corresponding image to obtain a first digital signal; fusing the accompaniment signal with a corresponding image to obtain a second digital signal; and generating a karaoke mode corresponding to the song based on the first digital signal and the second digital signal. According to the method and the device, the tracks of the karaoke song wanted by the user can be processed in real time, some modes facilitating karaoke song of the user are generated, the user can select the wanted karaoke song mode conveniently, more convenient and personalized karaoke song experience is provided for the user, and the degree of freedom and real-time performance of karaoke song of the user are improved.

Description

Technical Field

[0001] This application relates to the field of audio and video processing technology, and in particular to a karaoke processing method, apparatus, electronic device, and storage medium. Background Technology

[0002] With the widespread adoption of internet TVs, televisions are fulfilling more of the family's entertainment needs, such as karaoke. With a dedicated microphone and corresponding karaoke software installed on an internet TV, you can sing karaoke at home.

[0003] Currently, the songs in karaoke apps are usually pre-produced, including the original song, the backing track, and a mix of both. When a user wants to sing, they select the song and choose their preferred singing style, such as just the backing track or a mix of the original and the backing track.

[0004] In the process of implementing the embodiments of this application, the inventors discovered that the current karaoke method has some limitations on the songs that users can choose to sing, and the song library may not contain the songs that the user wants to sing. Summary of the Invention

[0005] One objective of this application is to provide a karaoke processing method, apparatus, and electronic device to solve the technical problem of limited song selection when users sing karaoke, thereby improving the freedom and real-time nature of users' karaoke experience.

[0006] To address the aforementioned technical problems, one technical solution adopted in this application is to provide a karaoke processing method, comprising: acquiring a song signal corresponding to a song to be sung, the song signal including an audio signal and an image sequence signal; acquiring a vocal signal and an accompaniment signal from the song based on the audio signal; acquiring an image synchronized with the audio signal based on the image sequence signal; fusing the vocal signal, the accompaniment signal, and the corresponding image to obtain a first digital signal; fusing the accompaniment signal and the corresponding image to obtain a second digital signal; and generating a karaoke mode corresponding to the song based on the first digital signal and the second digital signal, the karaoke mode including an original vocal mode corresponding to the first digital signal and an accompaniment mode corresponding to the second digital signal. This method acquires the song signal of a song input by the user, separates the original vocals and accompaniment in real time based on the song signal, obtains the corresponding image, and then generates an accompaniment mode (i.e., a mode with only accompaniment) and an original vocal mode (i.e., a mode including both accompaniment and original vocals) based on the original vocals, accompaniment, and the corresponding image, thereby facilitating karaoke for users. This method can process the songs that users want to sing in real time and generate some modes that are convenient for users to sing. It makes it easier for users to choose the karaoke mode they want, providing users with a more convenient and personalized karaoke experience and improving the freedom and real-time nature of users' karaoke.

[0007] Optionally, obtaining the vocal signal and accompaniment signal in the song based on the audio signal includes: performing a Fourier transform on the audio signal to obtain a complex spectrum, the complex spectrum including a real part and an imaginary part; passing the real part and the imaginary part as input features of two paths to the encoding module of a preset neural network model; the preset neural network model includes an encoding module, a decoding module, and an intermediate layer module; extracting and learning features from the input real part and the imaginary part through the encoding module to obtain first feature information; processing the first feature information through the intermediate layer module to obtain second feature information; and processing the first feature information and the second feature information through the decoding module to output the vocal signal corresponding to the real part and the imaginary part, and the accompaniment signal corresponding to the real part and the imaginary part. This method specifically illustrates how to separate the original vocals and accompaniment from an audio signal. It involves converting the audio signal into a spectrum using Fourier transform, which enables better extraction of audio features; using a neural network model to process the audio signal allows for the learning of complex feature representations; the intermediate layer module can extract the temporal features of the audio signal, which helps to better understand the audio data; and finally, the decoding module effectively separates the vocal signal and the accompaniment signal, providing more possibilities for audio processing.

[0008] Optionally, when the song includes multiple original singers, after obtaining the vocal signal and accompaniment signal from the song based on the audio signal, the method further includes: based on the obtained vocal signal from the song, separating multiple independent vocal signals, and ensuring that each separated vocal signal corresponds one-to-one with each original singer in the song. Here, the obtained original singers of the song include vocal signals from different original singers. By extracting the vocal signals from these different original singers, it is possible to distinguish which specific segment of the song each original singer is performing, allowing each original singer's voice to be processed and analyzed independently, thereby bringing more possibilities for creation and application.

[0009] Optionally, the step of separating multiple independent vocal signals based on the obtained vocal signals in the song, and ensuring that each separated vocal signal corresponds one-to-one with each original singer in the song, includes: encoding the obtained vocal signals in the song to obtain audio features; determining the number of original singers corresponding to the multiple original singers; copying the audio features to obtain multiple audio features, wherein the number of copies is equal to the number of original singers; decoding the copied multiple audio features to obtain feature vectors corresponding to the multiple original singers, each feature vector corresponding to one original singer; combining the audio features and the feature vectors to obtain combined features corresponding to the multiple original singers; inputting the combined features into a preset activation function to obtain the probability value corresponding to each original singer in each frame of the combined features; and determining whether each original singer is vocalizing in each frame of the combined features based on the probability value, thereby obtaining vocal signals of different original singers. This method has high real-time performance, making it suitable for application scenarios that require immediate processing and feedback; after obtaining the different segments corresponding to multiple original singers' voices in a song, personalized audio processing can be performed on each original singer, expanding the diversity of audio applications; moreover, the automated processing flow can improve production efficiency and reduce manual intervention and processing time.

[0010] Optionally, the karaoke mode further includes a duet mode, and the method further includes: receiving a vocal cancellation signal sent by the user; in response to the vocal cancellation signal, performing a vocal cancellation operation on a specified original singer among the plurality of original singers to obtain an uncancelled vocal signal among the plurality of original singers; and generating a duet mode based on the uncancelled vocal signal. The method of this embodiment is a duet function in the karaoke mode. Through vocal cancellation and vocal signal processing, as well as synthesis technology, it allows users to generate a duet effect between vocal signals from different original singers, thereby enhancing the user's singing experience and allowing them to enjoy more creative and expressive fun in karaoke applications.

[0011] Optionally, acquiring an image synchronized with the audio signal based on the image sequence signal includes: acquiring the delay time corresponding to each of the voice signal and the accompaniment signal; adjusting the playback time corresponding to the image sequence signal according to the delay time, so that the timestamp of the image sequence signal is synchronized with the timestamp of the audio signal. This embodiment's method ensures that while displaying a video or image sequence, the associated audio signal can match in time and content, thereby providing a smoother and more consistent karaoke experience.

[0012] To solve the above-mentioned technical problems, one technical solution adopted in this application is: providing a karaoke processing device, comprising: a song acquisition module, used to acquire the song signal corresponding to the song to be sung, the song signal including an audio signal and an image sequence signal; a vocal and accompaniment separation module, used to acquire the vocal signal and accompaniment signal in the song according to the audio signal; an image processing module, used to acquire an image synchronized with the audio signal according to the image sequence signal; a first fusion processing module, used to fuse the vocal signal, the accompaniment signal and the corresponding image to obtain a first digital signal; a second fusion processing module, used to fuse the accompaniment signal and the corresponding image to obtain a second digital signal; and a karaoke mode determination module, used to generate a karaoke mode corresponding to the song based on the first digital signal and the second digital signal, the karaoke mode including the original vocal mode corresponding to the first digital signal and the accompaniment mode corresponding to the second digital signal.

[0013] To address the aforementioned technical problems, one technical solution adopted in this application is to provide an electronic device, including a memory and a processor. The memory is connected to the processor, and the processor is configured to execute one or more computer programs stored in the memory. When the processor executes the one or more computer programs, it enables the electronic device to implement a karaoke processing method applicable to the electronic device. This electronic device possesses the beneficial effects corresponding to the aforementioned karaoke processing method applicable to the electronic device.

[0014] To solve the above-mentioned technical problems, one technical solution adopted in this application is to provide a non-volatile computer-readable storage medium that stores computer-executable instructions. When the computer-executable instructions are executed by an electronic device, the electronic device performs the karaoke processing method as described above.

[0015] To solve the above-mentioned technical problems, one technical solution adopted in this application is: to provide a computer program product, the computer program product including a computer program stored on a non-volatile computer-readable storage medium, the computer program including program instructions, and when the program instructions are executed by an electronic device, causing the electronic device to perform the karaoke processing method as described above. Attached Figure Description

[0016] To more clearly illustrate the technical solutions of the embodiments of this application, the drawings used in the description of the embodiments of this application will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0017] Figure 1 This is a flowchart of a karaoke processing method provided in an embodiment of this application;

[0018] Figure 2 This is a flowchart of a method for obtaining the original vocals and accompaniment in a song based on the audio signal, provided in an embodiment of this application.

[0019] Figure 3 This is a schematic diagram of a model for separating the original vocals and accompaniment provided in an embodiment of this application;

[0020] Figure 4 This is a schematic diagram of a model for separating different voices from multiple original vocals, provided in an embodiment of this application.

[0021] Figure 5 This is a flowchart of a karaoke processing method provided in another embodiment of this application;

[0022] Figure 6 This is a schematic diagram of the structure of a karaoke processing device provided in an embodiment of this application;

[0023] Figure 7 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0024] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application. All other embodiments obtained by those skilled in the art based on the embodiments in this application without inventive effort are within the scope of protection of this application.

[0025] It should be noted that, unless there is a conflict, the various features in the embodiments of this application can be combined with each other, all of which are within the protection scope of this application. Furthermore, although functional modules are divided in the device schematic diagram and a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in a different order than the module division in the device or the order in the flowchart. Moreover, the terms "first," "second," and "third" used in this application do not limit the data or execution order, but only distinguish identical or similar items with essentially the same function and effect.

[0026] It should be noted that in the various embodiments described below, there is no necessarily a certain order between the steps. Those skilled in the art can understand from the description of the embodiments of this application that the steps may have different execution orders in different embodiments, that is, they may be executed in parallel or in interchange, etc.

[0027] With the widespread adoption of internet TVs, the demand for home entertainment is constantly growing, among which TV karaoke has become a popular form. Users can enjoy karaoke at home by equipping themselves with a dedicated microphone and installing corresponding karaoke software. However, current karaoke software usually provides pre-made tracks, including original vocals, backing tracks, and a mix of original vocals and backing tracks. Users can only choose from these options, which has certain limitations because the song library may not contain the songs the user wants to sing.

[0028] Based on this, this application proposes a technical solution for real-time separation of vocals and accompaniment, and real-time differentiation of multiple singers, specifically for TV karaoke scenarios, enabling any existing song to be used for karaoke. By using real-time audio processing technology combined with a deep learning model, the input audio stream is analyzed and processed. Using acoustic models and deep learning algorithms, real-time separation of vocals (i.e., the original singer) and accompaniment signals is achieved, allowing for independent extraction of both. A real-time parameter adjustment function is also introduced, allowing users to adjust the ratio of vocals to accompaniment according to their needs, satisfying their personalized requirements for music mixing. Furthermore, through acoustic models and deep learning technology, the voices of different singers in multi-person choruses are differentiated, allowing users to selectively remove or highlight the voices of specific singers for customized multi-person choruses. The processed audio stream can be output to the TV karaoke software in real-time, allowing users to hear the separated vocals and accompaniment during live singing and to select the appropriate karaoke mode as needed.

[0029] The karaoke processing method and apparatus of this application separate vocals and accompaniment in real time, adjust the ratio of vocals to accompaniment, and realize personalized processing for multi-person chorus singing. This technical solution brings a more free, flexible, and innovative singing experience to TV karaoke.

[0030] Please see Figure 1 , Figure 1 This is a flowchart of a karaoke processing method provided in an embodiment of this application. The method includes the following steps:

[0031] Step S11: Obtain the song signal corresponding to the song to be sung, wherein the song signal includes audio signal and image sequence signal.

[0032] A song signal refers to the data signal generated by a song, including audio signals and image sequence signals. Audio signals are the electrical signals transmitted by the sounds in the song. In digital audio, audio signals can be represented digitally, and analog audio signals can be converted into digital signals through sampling and quantization. Audio signals contain audio information such as vocals and instrument sounds from the song. Image sequence signals are signals composed of a series of image frames from a song's video. When a song is accompanied by video, the image sequence in the video changes over time, forming a video signal. These image sequences can showcase the singer's performance, scenes from the music video, and other content.

[0033] Audio signals, typically represented as digital signals, can be extracted from song files using audio processing techniques. Image sequence signals can be sequences of images from music videos, which can be extracted from video files using video processing techniques.

[0034] The input song signal can also be decoded into audio and image sequence signals using a decoder. Audio signal decoding can be achieved through audio processing techniques and corresponding decoding algorithms. If the song signal is input as an audio file, audio decoding libraries (such as FFmpeg, LibAV, etc.) can be used to decode the audio data and convert it into a digital audio signal. Image sequence signal decoding can consider the parsing and extraction of video data from the song signal. If the song signal contains video data, video decoding libraries (such as FFmpeg, OpenCV, etc.) can be used to parse the video data and extract the image sequence. When designing the decoder module, error handling mechanisms should be considered to ensure that it can handle abnormal situations and erroneous input data. Compatibility with different audio and video formats should also be considered to ensure that the decoder can adapt to the decoding requirements of various song signal formats.

[0035] Step S12: Obtain the vocal signal and accompaniment signal in the song based on the audio signal.

[0036] The vocal signal refers to the original vocal part of a song. The accompaniment refers to the musical part of a song excluding the original vocals, including background music, instrumental performances, etc. The accompaniment contains elements such as harmony and rhythm, but does not include the original vocals. Extracting the accompaniment signal means separating the musical elements from the original vocals in the song, so that the musical part can be played separately without the original vocals.

[0037] One method is to extract vocal signals from song audio using audio separation techniques, which can employ audio signal processing techniques and machine learning algorithms. Similarly, extracting accompaniment signals from song audio can also utilize audio signal processing techniques, such as differential processing based on the extracted vocal signals.

[0038] After obtaining the vocal and instrumental signals from a song, users can freely adjust the volume ratio of the original vocals and the instrumental, thus achieving different mixing effects. For example, an audio mixing system can be designed that accepts user input and operation to adjust the volume ratio of the original vocals and the instrumental. The user interface can include control elements such as sliders, buttons, or knobs, allowing users to adjust the volume ratio of the original vocals and the instrumental. When the user adjusts the ratio, the system can process the audio signals of the original vocals and the instrumental in real time, performing mixing according to the user-defined ratio. This includes weighted mixing of the original vocals and the instrumental audio signals according to the set volume ratio to generate the final mixing result. Furthermore, during the user's volume adjustment process, the system can provide real-time feedback on the mixing effect, allowing users to hear the mixing effect at different ratios and providing real-time audio playback functionality, enabling users to hear the adjusted music mix at any time. Therefore, by combining audio processing and user interaction design, an intuitive operating interface and real-time mixing effects are provided, thus meeting users' personalized requirements for music mixing.

[0039] Step S13: Obtain an image synchronized with the audio signal based on the image sequence signal.

[0040] An image sequence signal refers to a video signal, which is a signal composed of a series of consecutive image frames. Each frame represents a still image in the video, and by playing these frames quickly at a certain frame rate (frames per second), a continuously moving video can be presented.

[0041] Understandably, audio signals and image sequence signals are generally synchronized in terms of timestamps. However, due to the complexity of processing vocals and accompaniment separately, the audio signal may experience delays after passing through certain processing modules. To ensure that the audio and video signals are synchronized, i.e., that the playback of audio matches the playback of video, the delay times corresponding to the vocal and accompaniment signals can be obtained. The playback time of the image sequence signal is then adjusted based on these delay times to synchronize the timestamps of the image sequence signal and the audio signal. This method ensures that while displaying a video or image sequence, the associated audio signal is matched in both time and content, thus providing a smoother and more consistent karaoke experience.

[0042] For example, if audio processing introduces a delay of 100 milliseconds (4 frames), and the image frame rate is 30 frames per second, then in this case, the image output can be delayed by 3 frames (100 milliseconds) based on the delay information. In this way, during playback, the video image will play together with the audio according to the adjusted timestamp, keeping the audio and video synchronized.

[0043] Step S14: Fuse the human voice signal, the accompaniment signal and the corresponding image to obtain the first digital signal.

[0044] The process involves audio processing of the vocal and accompaniment signals, such as mixing techniques, to combine them into a single audio signal. This may include adjusting parameters like volume and frequency to ensure a balanced and harmonious sound. Next, the image is processed to obtain an image signal, ensuring correspondence between the image and audio signals. This process can synchronously process the image's timing information to guarantee correct alignment. The processed audio and image signals are then fused, which may involve combining the audio waveform with visual elements of the image. This can be achieved using video editing software or specific mixing techniques. Finally, the fused audio and image signals are combined into a new digital signal, the first digital signal. This first digital signal contains integrated information from the vocals, accompaniment, and image, and is used for subsequent karaoke mode generation or other processing.

[0045] Step S15: Fuse the accompaniment signal with the corresponding image to obtain a second digital signal.

[0046] The process involves several steps: First, the accompaniment signal undergoes audio processing, such as volume balancing and audio effect adjustments. Then, the corresponding image is processed to ensure a match between the processed image and the accompaniment signal. For example, the timeline of the accompaniment signal is aligned with the selected image frames. Next, the processed accompaniment signal and the corresponding image frames are synthesized, including overlaying or mixing the accompaniment audio and image. Finally, the synthesized audio and image are integrated into a new digital signal, the second digital signal. This second digital signal contains the fusion of the accompaniment and image, used for subsequent karaoke mode generation or other processing.

[0047] Step S16: Generate a karaoke mode corresponding to the song based on the first digital signal and the second digital signal. The karaoke mode includes the original sound mode corresponding to the first digital signal and the accompaniment mode corresponding to the second digital signal.

[0048] The "Karaoke Mode" refers to an audio and video fusion experience. In this embodiment, users can generate a Karaoke Mode for the songs they want to sing in real time. The "Background Music Mode" plays the background music of a song without the original vocals. In this mode, users can hear the instrumental track but not the original vocals. They can usually see lyrics or music-related images or videos to assist them in singing. The "Original Audio Mode" plays both the original vocals and background music simultaneously. In this mode, users can hear the full version of the song, including both the original vocals and the instrumental track. They can usually also see lyrics or related images or videos so they can sing along with the original vocals.

[0049] The method for generating the karaoke mode may include: fusing the separated vocal and accompaniment signals with corresponding synchronized images, and outputting the fusion result to the TV karaoke software in real time. This process may use encoding and transmission protocols to ensure that the signal is transmitted to the TV screen with optimal quality. Furthermore, a user-friendly interface can be created within the TV karaoke software, allowing users to select either an accompaniment mode or an original audio mode, and providing lyrics or images to assist singing. In accompaniment mode, only the accompaniment is played and related images are displayed; in original audio mode, the original vocals, accompaniment, and related images are played simultaneously. This achieves real-time fusion of the separated original vocals and accompaniment with synchronized images, and outputs the result to the TV karaoke software, providing users with an immersive karaoke experience.

[0050] This application provides a karaoke processing method that can process the songs a user wants to sing in real time. The songs are not limited by the music library, and the method can generate the corresponding karaoke mode for the user in real time, making it easier for the user to choose the desired karaoke mode. This provides the user with a more convenient and personalized karaoke experience, and improves the user's freedom and real-time performance in karaoke.

[0051] Understandably, one of the key aspects of enabling users to sing karaoke in real time is to separate the original vocals and accompaniment in a song in real time. Accurately separating the original vocals and accompaniment in a song can provide users with a better audio experience for singing karaoke in real time.

[0052] Therefore, in some embodiments, please refer to Figure 2 Step S12, obtaining the vocal signal and accompaniment signal in the song based on the audio signal, includes:

[0053] Step S121: Perform a Fourier transform on the audio signal to obtain a complex spectrum, which includes a real part and an imaginary part. The real part represents the amplitude of the cosine sequence, and the imaginary part represents the amplitude of the sine sequence.

[0054] Step S122: The real part and the imaginary part are respectively passed as input features of two paths to the encoding module of a preset neural network model; the preset neural network model includes an encoding module, a decoding module and an intermediate layer module.

[0055] Step S123: The encoding module extracts and learns features from the input real and imaginary parts to obtain first feature information. This first feature information contains abstract features learned by the audio signal spectrum during the neural network learning process, and can be understood as a high-level representation of the input spectrum.

[0056] Step S124: Process the first feature information through the intermediate layer module to obtain the second feature information. The intermediate layer module further processes and refines the first feature information to obtain the second feature information, which contains a higher-level abstract feature representation. Specifically, the intermediate layer module can be a Long Short-Term Memory (LSTM) network, and the second feature information can be time-series information learned by the LSTM.

[0057] The aforementioned first and second feature information can include various features of the audio signal spectrum. Through learning and extraction by the neural network model, the model can better reproduce the human voice signal and the accompaniment signal.

[0058] Step S125: Process the first feature information and the second feature information through the decoding module, and output the human voice signal corresponding to the real part and the imaginary part, as well as the accompaniment signal corresponding to the real part and the imaginary part.

[0059] This method converts audio signals into a spectrum using Fourier transform, enabling better extraction of frequency information; it uses a neural network model to process the audio signals, allowing for the learning of complex feature representations; the intermediate layer module can extract the temporal features of the audio signals, facilitating a better understanding of the audio data; and finally, the decoding module effectively separates the original vocals and accompaniment signals, providing more possibilities for audio processing.

[0060] For example, please see Figure 3 , Figure 3This paper presents a deep learning model architecture for audio signal separation, mainly consisting of an encoder, a decoder, and an intermediate layer module (LSTM). The audio signal output from the decoder undergoes a short-time Fourier transform to obtain a complex spectrum, using the real and imaginary parts as input features for two separate paths. The output is a mask matrix, which, multiplied by the complex spectrum of the original signal, yields the complex spectrum of the output signal. A human voice signal can then be obtained through an inverse short-time Fourier transform. The mask matrix has the same dimensionality as the input signal's spectrum and guides the separation of different components of the mixed signal. Multiplying the mask matrix point-by-point with the complex spectrum of the original signal allows for control over the weighting of different frequency components, thus achieving signal separation. Multiplying the complex spectrum of the original signal by the mask matrix yields the complex spectrum of the output signal, which contains information about the separated target signal (e.g., human voice). The final step is to convert the complex spectrum of the output signal back to the time domain through the inverse short-time Fourier transform, thereby obtaining the final audio signal. This process is the operation of converting the frequency domain signal into a time domain signal, so that the target signal (such as a human voice signal) can be separated.

[0061] The encoder consists of four stacked encoder blocks. Each encoder block comprises a convolutional layer, a normalization layer, and an activation layer, using causal convolutions. The kernel size can be (3×3) with a stride of (2, 1). The number of channels in each encoder block is given in the diagram; for example, "2->16" indicates 2 input channels and 16 output channels, with one channel used for the real part of the input and the other for the imaginary part. The decoder consists of four stacked decoder blocks, similar in structure to the encoder. Each decoder block comprises a transposed convolutional layer, a normalization layer, and an activation layer. The transposed convolutional kernel size can be (3×3) with a stride of (2, 1). The intermediate layer module is represented by a Long Short-Term Memory (LSTM) network and connects the encoder and decoder. To meet real-time requirements, the latency of the entire model is set to 1 frame (e.g., divided into 25ms per frame). In addition, considering real-time performance, all convolutions can be causal convolutions to ensure that the model can accurately process audio signals in real time.

[0062] Figure 3In the model shown, the encoder and decoder gradually extract and recover features from the audio signal through multiple stacked blocks. Causal convolution ensures the model's real-time performance and stability, avoiding information leakage and the exploitation of future information. The LSTM structure helps the model capture long-term temporal dependencies, improving separation performance. By using this model, the original vocals and accompaniment can be effectively separated from the audio signal while maintaining low latency and high accuracy.

[0063] In choral songs, multiple singers typically perform simultaneously. When recording such songs, each singer's voice appears in different parts of the song. However, sometimes it's desirable to selectively remove the voice of a particular singer to achieve the effect of a duet with the other original singers.

[0064] Therefore, in some embodiments, when the song includes multiple original singers, after obtaining the vocal signal and accompaniment signal in the song based on the audio signal, the method further includes: based on the obtained vocal signal in the song, separating multiple independent vocal signals, and ensuring that each separated vocal signal corresponds one-to-one with each original singer in the song. By extracting the vocal signals of multiple original singers, the vocal signals of different original singers in the song can be obtained, and it is possible to distinguish which specific segment of the song each original singer is performing. This allows the voice of each original singer to be processed and analyzed independently, thereby bringing more possibilities for creation and application.

[0065] Specifically, based on the obtained vocal signals in the song, separating multiple independent vocal signals and ensuring a one-to-one correspondence between each separated vocal signal and each original singer in the song includes: encoding the obtained vocal signals in the song to obtain audio features; determining the number of original singers corresponding to the multiple original singers; copying the audio features to obtain multiple audio features, wherein the number of copies is equal to the number of original singers; decoding the copied multiple audio features to obtain feature vectors corresponding to the multiple original singers, each feature vector corresponding to one original singer; combining the audio features and the feature vectors to obtain combined features corresponding to the multiple original singers; inputting the combined features into a preset activation function to obtain the probability value corresponding to each original singer in each frame of the combined features; and determining whether each original singer is vocalizing in each frame of the combined features based on the probability value, thereby obtaining vocal signals of different original singers.

[0066] The obtained vocal signal from the original singer in the song contains the voices of multiple different original singers. At this stage, the vocal signal has not been processed, and the voices of the different original singers are not distinguished. The vocal signal can be encoded to obtain corresponding audio features. These audio features can be a set of feature vectors or sequences that capture important information in the vocal signal. Specifically, these audio features can be spectral features, time-frequency domain features, or other types of features. The encoding process may include feature dimensionality reduction or other processing to better represent the vocal features of multiple original singers.

[0067] Determining the number of original singers corresponding to multiple original singers is crucial for accurately copying and processing audio features in subsequent processing. For example, if there are two original singers, the audio features are copied twice to obtain two identical sets of audio features.

[0068] The copied audio features are decoded to obtain feature vectors corresponding to multiple original singers. This decoding process includes restoring the encoded features to their original representations. Each original singer corresponds to one feature vector. Each original singer's vocal signal is represented as a sequence of feature vectors, with each feature vector corresponding to a specific time period or frame of the audio signal. Therefore, decoding the audio features of each original singer yields a sequence of feature vectors equal in number to the original singers. Each feature vector in the sequence represents the sonic feature representation of the corresponding original singer within a specific time period or frame. This representation method facilitates more refined and accurate modeling of the original singers' vocal signals, thereby enabling effective separation and processing of vocal signals from different original singers.

[0069] By combining audio features and feature vectors, combined features corresponding to multiple original singers can be obtained. For example, audio features and feature vectors can be added together or combined in other ways to obtain an integrated feature representation. Alternatively, there may be a one-to-one correspondence between audio features and feature vectors, meaning each audio feature is associated with a feature vector. In this case, these two types of features can be fused or combined to obtain combined features corresponding to multiple original singers. This method helps maintain the information independence of each original singer and better distinguishes the vocal features of different original singers during processing.

[0070] The combined features are input into a preset activation function to obtain the probability value of each original singer's combined features in each frame. This probability value represents the vocal activity of each singer in each time frame. Finally, based on the probability value, it is determined whether each singer is vocalizing in each time frame, thus obtaining the vocal signals of different singers. For example, if the probability value is greater than a preset threshold, it means that the singer is vocalizing in that time frame, and that time frame belongs to the singer's singing segment; if the probability value is less than or equal to the preset threshold, it means that the singer is not vocalizing in that time frame, and that time frame does not belong to the singer's singing segment.

[0071] For example, please see Figure 4 This provides an audio signal processing model that combines an Encoder-Decoder structure for the separation and recognition of multi-singer vocal signals. It mainly includes:

[0072] Input processing: The human voice signal is divided into T frames, each containing 25ms of audio data. The frame-level audio features are extracted by an encoder. e1 ...e t ...e T .

[0073] Multi-singer processing: Assuming a song is sung by S singers, the frame-level audio features are copied S times and then fed into a decoder to obtain feature vectors representing the original singers. as,1 ...a s,t ...a s,T .

[0074] Feature fusion: The original frame-level audio features are multiplied with the corresponding singer's feature vector to obtain the probability value of each frame for each original singer.

[0075] Output processing: The product result is mapped to a probability value using the Sigmoid (hyperbolic tangent) function. It represents the probability of each original singer's voice appearing in each frame. A threshold is set, when... If the value is greater than the threshold, it means that the s-th singer is singing at frame t. If the threshold is less than or equal to, it means that the s-th singer did not sing at frame t.

[0076] Attention Mechanism: The Encoder and Decoder in the model are primarily Transformer structures, which need to meet real-time requirements and can rely on historical information (or partial historical information) for attention calculation. This method can rely on information from the next 0 to 5 frames, achieving a minimum latency of 1 frame and a maximum latency of 6 frames, thereby improving the model's discrimination accuracy. The Transformer is a deep learning model architecture based on a self-attention mechanism.

[0077] After separating the vocal signals of multiple original singers through the above embodiments, a personalized karaoke mode can be customized for the user. For example, the karaoke mode also includes a duet mode. The method further includes: receiving an original vocal cancellation signal sent by the user; in response to the original vocal cancellation signal, performing a cancellation operation on the vocal signal of a specified original singer among the multiple original singers to obtain the uncancelled vocal signal among the multiple original singers; and generating a duet mode based on the uncancelled vocal signal.

[0078] For example, suppose there's a song sung by a male singer (original singer A) and a female singer (original singer B). A user wants to keep the accompaniment and remove the female singer's voice so they can sing the female singer's part in karaoke, achieving a duet effect with the male singer. The original audio contains the voices of both singers. In this case, the method described above can be used to extract the part containing the female singer's voice. Through voice removal, the female singer's voice can be eliminated, while the male singer's voice and accompaniment are retained. Based on the audio obtained after removing the female singer's voice, a duet mode can be generated. In this mode, the user can sing the female singer's part while simultaneously hearing the male singer's voice and accompaniment, achieving a duet effect with the male singer. When the user selects the duet mode for this song in karaoke, the system will play the track that retains the male singer's voice and accompaniment, while the user can record the female singer's part using a microphone. The duet mode will mix the user's recording with the male singer's voice in real time, allowing the user to achieve a duet effect with the male singer while singing the female singer's part. In the final output, the user will hear a duet effect between the male and female original singers, while the female original singer's voice has been successfully removed and replaced with the user's own singing voice.

[0079] The method provided in this embodiment allows users to customize their karaoke experience according to their preferences and needs. By removing specific original vocals and generating a chorus mode, users can selectively participate in the chorus, singing their favorite parts and achieving personalized musical expression.

[0080] Please see Figure 5 , Figure 5This is a flowchart of a karaoke processing method provided in another embodiment of this application. It mainly includes: the input song is decoded into an audio signal and an image sequence signal by a decoder. A vocal and accompaniment separation module processes the audio signal, which can use a U-shaped residual network to achieve low-latency separation of the original vocals and accompaniment, obtaining vocal and accompaniment signals. When multiple original vocals exist in the vocal signal, real-time vocal differentiation can be achieved, distinguishing different original singers in the song and determining whether each original singer is singing at each time frame. This is mainly achieved by copying audio features and combining them with the decoded original singer features, and using a threshold determination method to achieve real-time vocal differentiation and vocal determination. Through the above vocal differentiation, the original vocals of each original singer at their respective time frames can be obtained, and then the obtained accompaniment signal is added in, i.e., vocals and accompaniment are mixed. In this process, a specific original vocal can be eliminated or suppressed.

[0081] In addition, the image sequence signal obtained above can be buffered in the image sequence buffer module. The audio signal may experience delays after processing by the voice and accompaniment separation and voice differentiation modules. To achieve audio and video synchronization, a buffer is introduced into the image sequence processing flow to store the image frames to be output. When the audio signal output from the audio processing module is processed by the voice and accompaniment separation and voice differentiation modules, a delayed audio signal will be obtained. Simultaneously, this delay information can be passed to the image sequence buffer module. The image sequence buffer module can store the received image frames in the buffer based on the delay information and output the buffered image frames at appropriate times.

[0082] The karaoke processing method provided in this embodiment has two main advantages. First, it can separate vocals and accompaniment in real time. Compared to traditional karaoke solutions, it eliminates the need for pre-prepared content and allows for instant separation of vocals and accompaniment from any song. This enables users to choose any song for karaoke, increasing freedom and real-time performance. Second, it can distinguish singers in real time, even when there are multiple original singers. By copying audio features and combining them with decoded original singer features, and using a threshold method, it determines whether each original singer is singing at each time frame. This real-time vocal differentiation technology allows for targeted elimination of specific singers in multi-person duets, achieving the effect of singing with other original singers. Finally, it also allows for adjustment of the vocal and accompaniment ratio. By mixing the separated vocal signal with the accompaniment signal, the relative intensity of the vocals and accompaniment can be flexibly controlled to suit the needs and preferences of different users. The vocal signal is the original singer's audio signal.

[0083] The karaoke processing method in the above embodiments is a real-time karaoke solution. In some embodiments, a buffering mechanism can also be introduced, combined with audio signal processing and voiceprint recognition technologies to achieve a karaoke method. When the application software downloads the song to be sung in real time, it not only downloads the song itself but also caches the complete song signal. The application stores the audio data of the entire song, rather than just providing online streaming playback. Using the separation technology in the above embodiments, the audio information in the song is processed to extract the original vocals and accompaniment. When there are multiple original vocals, voiceprint and clustering technologies can be used on the vocal signal to distinguish the corresponding speaker in different sections. This is a non-real-time processing process, meaning that it may take some time to process the vocal signal of the entire song. After completing the vocal separation and speaker identification, the user can perform the regular karaoke process, including selecting the part to sing, using effects, and adjusting the timbre. This application embodiment improves the quality and accuracy of audio processing by caching the complete song signal and using non-real-time voiceprint recognition, resulting in a better karaoke experience.

[0084] Please see Figure 6 , Figure 6 This is a schematic diagram of a karaoke processing device provided in an embodiment of this application. The device 20 includes:

[0085] The song acquisition module 201 is used to acquire the song signal corresponding to the song to be sung in karaoke, the song signal including audio signal and image sequence signal;

[0086] The original vocal and accompaniment separation module 202 is used to obtain the vocal signal and accompaniment signal in the song based on the audio signal;

[0087] Image processing module 203 is used to acquire an image synchronized with the audio signal based on the image sequence signal;

[0088] The first fusion processing module 204 is used to fuse the human voice signal, the accompaniment signal and the corresponding image to obtain a first digital signal;

[0089] The second fusion processing module 205 is used to fuse the accompaniment signal with the corresponding image to obtain a second digital signal;

[0090] Karaoke mode determination module 206 is used to generate a karaoke mode corresponding to the song based on the first digital signal and the second digital signal. The karaoke mode includes the original sound mode corresponding to the first digital signal and the accompaniment mode corresponding to the second digital signal.

[0091] The aforementioned karaoke processing device 20 can be a software module, which includes several instructions stored in a memory. The processor can access the memory, call the instructions, and execute them to complete the karaoke processing method described in the above embodiments.

[0092] In some embodiments, the karaoke processing device 20 can be constructed from hardware devices. For example, the karaoke processing device 20 can be constructed from one or more chips, which can work in coordination to complete the karaoke processing method described in the various embodiments. As another example, the karaoke processing device 20 can also be constructed from various logic devices, such as general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), microcontrollers, ARM (Acorn RISC Machine) or other programmable logic devices, discrete gate or transistor logic, discrete hardware components, or any combination of these components.

[0093] It should be noted that the karaoke processing device 20 described above can execute the karaoke processing method for electronic devices provided in the embodiments of this application, and has the corresponding functional modules and beneficial effects for executing the method. Technical details not described in detail in the embodiments of the karaoke processing device 20 can be found in the karaoke processing method for electronic devices provided in the embodiments of this application.

[0094] Please see Figure 7 , Figure 7 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. The electronic device 30 includes one or more processors 31 and a memory 32. The memory 32 is connected to one or more processors 31, for example, via a bus.

[0095] Processor 31 is configured to support the electronic device 30 in performing the corresponding functions in the methods described in the above method embodiments. The processor may be a central processing unit (CPU), a network processor (NP), a hardware chip, or any combination thereof. The aforementioned hardware chip may be an application-specific integrated circuit (ASIC), a programmable logic device (PLD), or a combination thereof. The aforementioned PLD may be a complex programmable logic device (CPLD), a field-programmable gate array (FPGA), a generic array logic (GAL), or any combination thereof.

[0096] Memory 32 is used to store program code, etc. Memory 32 may include volatile memory (VM), such as random access memory (RAM); memory 32 may also include non-volatile memory (NVM), such as read-only memory (ROM), flash memory, hard disk drive (HDD), or solid-state drive (SSD); memory 32 may also include combinations of the above types of memory.

[0097] The memory 32 can be used to store non-volatile software programs, non-volatile computer-executable programs, and modules, such as the program instructions / modules corresponding to the karaoke processing method in the embodiments of this application. The processor 31 executes various functional applications and data processing of the karaoke processing method and karaoke processing device by running the non-volatile software programs, instructions, and modules stored in the memory 32, that is, it realizes the functions of the various modules or units of the karaoke processing method and karaoke processing device provided in the above method embodiments.

[0098] The memory 32 may include a program storage area and a data storage area, wherein the program storage area may store the operating system and applications required for at least one function. The data storage area may store data created based on the use of the karaoke processing device. In some embodiments, the memory 32 may include memory remotely located relative to the processor 31, and such remote memory may be connected to the karaoke processing device via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0099] The one or more modules are stored in the memory 32. When executed by the one or more processors 31, they perform the karaoke processing method in any of the above method embodiments. For example, they perform the method steps described in the above method embodiments to realize the functions of the modules described in the above device embodiments.

[0100] The electronic devices in this application embodiment may specifically be: televisions, computers, smartphones, tablets, professional audio equipment, servers / cloud platforms, etc.

[0101] This application also provides a non-volatile computer-readable storage medium storing a computer program, the computer program including program instructions, which, when executed by the electronic device 30, cause the electronic device 30 to perform the method described in the foregoing embodiments.

[0102] This application also provides a computer program product, which includes a computer program stored on a non-volatile computer-readable storage medium. The computer program includes program instructions that, when executed by an electronic device 30, cause the electronic device 30 to perform the method described in the foregoing embodiments.

[0103] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The program can be stored in a computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. The storage medium can be a magnetic disk, optical disk, read-only memory (ROM), or random access memory (RAM), etc.

[0104] The above-disclosed embodiments are merely preferred embodiments of this application and should not be construed as limiting the scope of this application. Therefore, any equivalent variations made in accordance with the claims of this application shall still fall within the scope of this application.

Claims

1. A karaoke processing method, characterized in that, include: Obtain the song signal corresponding to the song to be sung in karaoke, wherein the song signal includes audio signal and image sequence signal; The vocal and accompaniment signals in the song are obtained based on the audio signal; Based on the image sequence signal, obtain an image synchronized with the audio signal; The human voice signal, the accompaniment signal, and the corresponding image are fused to obtain the first digital signal; The accompaniment signal is fused with the corresponding image to obtain a second digital signal; The karaoke mode corresponding to the song is generated based on the first digital signal and the second digital signal. The karaoke mode includes the original sound mode corresponding to the first digital signal and the accompaniment mode corresponding to the second digital signal.

2. The method according to claim 1, characterized in that, The step of obtaining the vocal signal and accompaniment signal in the song based on the audio signal includes: The audio signal is subjected to a Fourier transform to obtain a complex spectrum, which includes a real part and an imaginary part; The real part and the imaginary part are respectively passed as input features of two pathways to the encoding module of a preset neural network model; the preset neural network model includes an encoding module, a decoding module, and an intermediate layer module; The encoding module extracts and learns features from the input real and imaginary parts to obtain first feature information. The second feature information is obtained by processing the first feature information through the intermediate layer module; The decoding module processes the first feature information and the second feature information to output the human voice signal corresponding to the real part and the imaginary part, as well as the accompaniment signal corresponding to the real part and the imaginary part.

3. The method according to claim 1 or 2, characterized in that, When the song includes multiple original singers, after obtaining the vocal signal and accompaniment signal from the song based on the audio signal, the method further includes: Based on the obtained vocal signals in the song, multiple independent vocal signals are separated, and each separated vocal signal corresponds one-to-one with each original singer in the song.

4. The method according to claim 3, characterized in that, Based on the obtained vocal signals in the song, multiple independent vocal signals are separated, and each separated vocal signal corresponds one-to-one with each original singer in the song, including: The obtained vocal signal in the song is encoded to obtain audio features; Determine the number of original singers corresponding to the multiple original singers; The audio features are copied to obtain multiple audio features, wherein the number of copies is the same as the number of original vocals; Decode the multiple copied audio features to obtain feature vectors corresponding to the multiple original singers, each feature vector corresponding to an original singer. By combining the audio features and the feature vectors, the combined features corresponding to the multiple original singers are obtained; The combined features are input into a preset activation function to obtain the probability value of each original singer in each frame on the combined features; Based on the probability value, determine whether each original singer produces sound in the combination features of each frame, thereby obtaining the vocal signals of different original singers.

5. The method according to claim 3, characterized in that, The karaoke mode also includes a duet mode, and the method further includes: Receive the original sound cancellation signal sent by the user; In response to the original vocal cancellation signal, the vocal signal of a specified original singer among the plurality of original singers is cancelled to obtain the uncancelled vocal signal among the plurality of original singers; A chorus pattern is generated based on the unremoved vocal signals.

6. The method according to claim 1, characterized in that, The step of acquiring an image synchronized with the audio signal based on the image sequence signal includes: Obtain the delay time corresponding to the voice signal and the accompaniment signal respectively; The playback time corresponding to the image sequence signal is adjusted according to the delay time so that the timestamp of the image sequence signal is synchronized with the timestamp of the audio signal.

7. A karaoke processing device, characterized in that, include: The song acquisition module is used to acquire the song signal corresponding to the song to be sung in karaoke, the song signal including audio signal and image sequence signal; The original vocal and accompaniment separation module is used to obtain the vocal signal and accompaniment signal in the song based on the audio signal; An image processing module is used to acquire an image synchronized with the audio signal based on the image sequence signal; The first fusion processing module is used to fuse the human voice signal, the accompaniment signal and the corresponding image to obtain a first digital signal; The second fusion processing module is used to fuse the accompaniment signal with the corresponding image to obtain a second digital signal; The karaoke mode determination module is used to generate a karaoke mode corresponding to the song based on the first digital signal and the second digital signal. The karaoke mode includes the original sound mode corresponding to the first digital signal and the accompaniment mode corresponding to the second digital signal.

8. An electronic device, characterized in that, include: A memory and a processor, the memory being connected to the processor, the processor being configured to execute one or more computer programs stored in the memory, the processor causing the electronic device to implement the karaoke processing method as described in any one of claims 1-6 when executing the one or more computer programs.

9. A non-volatile computer-readable storage medium, characterized in that, The non-volatile computer-readable storage medium stores computer-executable instructions, which, when executed by an electronic device, cause the electronic device to perform the karaoke processing method as described in any one of claims 1-6.

10. A computer program product, characterized in that, The computer program product includes a computer program stored on a non-volatile computer-readable storage medium, the computer program including program instructions that, when executed by an electronic device, cause the electronic device to perform the karaoke processing method as described in any one of claims 1-6.

Images