Song recording method, electronic equipment and computer readable storage medium

Abstract

The invention discloses a song recording method, electronic equipment and a computer readable storage medium, and the method comprises the steps: responding to a recording command of a target song triggered by a song application after a user account logs in the song application, and playing the audio of the target song containing the original singing voice; in the process of playing the target song, obtaining an input audio through an audio input device; extracting a target voiceprint audio matched with the voiceprint feature of the user account from the input audio; and synthesizing the accompaniment audio of the target song and the target dry sound audio to obtain a recorded audio. In the recording process of the target song, the target vocal audio conforming to the vocal print characteristics of the user and the music melody of the target song is extracted from the input audio collected by the audio input device, and the original singing voice of the target song in the input audio is eliminated; when songs are synthesized, the accompaniment audio of the target song and the target dry audio are subjected to chorus, so that the original human voice in the final synthesized audio is eliminated, and the user experience is improved.

Description

Technical Field

[0001] The present application relates to the field of music technology, and more specifically, to a song recording method, an electronic device, and a computer-readable storage medium. Background Art

[0002] In the related technology, song recording software supports playing accompaniment including the original singer's vocals during the recording process. However, when the microphone collects the singer's singing voice, it will also collect the original singer's vocals. When synthesizing and mixing the song, the original singer's vocals in the accompaniment cannot be eliminated, resulting in the final synthesized audio containing the original singer's vocals.

[0003] Therefore, how to support the playback of the accompaniment containing the original singer's vocals while eliminating the original singer's vocals in the final synthesized audio during the song recording process is a technical problem that technicians in this field need to solve. Summary of the invention

[0004] The purpose of the present application is to provide a song recording method, electronic device, storage medium and computer program product, which, during the song recording process, support playing an accompaniment containing the original singer's vocals while eliminating the original singer's vocals in the final synthesized audio.

[0005] To achieve the above-mentioned purpose, the first aspect of the present application provides a song recording method, comprising:

[0006] After the user account logs into the song application, in response to a recording command of a target song triggered in the song application, playing the audio of the target song including the original singer's voice;

[0007] During the playing of the target song, obtaining input audio through an audio input device;

[0008] Extracting a target dry voice audio that matches the voiceprint feature of the user account from the input audio;

[0009] The accompaniment audio of the target song and the target dry sound audio are synthesized to obtain the recorded audio.

[0010] Among them, it also includes:

[0011] Obtaining a baseline dry voice audio of the user account; wherein the baseline dry voice audio includes a cappella dry voice audio and / or a reading dry voice audio, and the a cappella dry voice audio includes a cappella dry voice audio of different pitches;

[0012] The reference dry audio is input into an audio feature extraction model to extract the voiceprint features of the user account.

[0013] The step of extracting a target dry audio that matches the voiceprint feature of the user account from the input audio includes:

[0014] Extracting all dry audios from the input audio, and matching target dry audios that match the voiceprint features of the user account among all dry audios;

[0015] Determining whether the target dry audio matches the music melody of the target song;

[0016] If so, the step of synthesizing the accompaniment audio of the target song and the target dry sound audio to obtain the recorded audio is entered.

[0017] The step of judging whether the target dry audio matches the music melody of the target song includes:

[0018] At least one audio matching method is used to determine whether the target dry sound audio matches the music melody of the target song; wherein the audio matching method includes an audio fingerprint matching method, a baseband midi matching method, and a lyrics matching method;

[0019] If multiple audio matching methods are used, a comprehensive judgment is made based on the judgment results of the multiple audio matching methods as to whether the target dry sound audio matches the musical melody of the target song.

[0020] The method of using an audio fingerprint matching method to determine whether the target dry sound audio matches the music melody of the target song includes:

[0021] Extracting a target audio fingerprint of the target dry audio;

[0022] Comparing the target audio fingerprint with the standard audio fingerprint of the target song, determining whether the similarity between the target audio fingerprint and the standard audio fingerprint of the target song is greater than or equal to a preset value;

[0023] If so, it is determined whether the target dry audio matches the musical melody of the target song.

[0024] Wherein, the baseband midi matching method is used to determine whether the target dry sound audio matches the music melody of the target song, including:

[0025] Extracting a target fundamental frequency midi of the target dry audio;

[0026] Compare the target fundamental frequency midi with the standard target fundamental frequency midi of the target song, and determine whether the similarity between the target fundamental frequency midi and the standard target fundamental frequency midi of the target song is greater than or equal to a preset value;

[0027] If so, it is determined whether the target dry audio matches the musical melody of the target song.

[0028] The method of using lyrics matching to determine whether the target dry audio matches the music melody of the target song includes:

[0029] Extract target lyrics of the target dry audio;

[0030] Comparing the target lyrics with the standard lyrics of the target song, and determining whether the similarity between the target lyrics and the standard lyrics of the target song is greater than or equal to a preset value;

[0031] If so, it is determined whether the target dry audio matches the musical melody of the target song.

[0032] After determining whether the target dry sound audio matches the music melody of the target song, the method further includes:

[0033] If the target dry audio matches the music melody of the target song, performing denoising and dereverberation operations on the target dry audio to obtain the dry audio to be synthesized;

[0034] Accordingly, the synthesizing the accompaniment audio of the target song and the target dry sound audio to obtain the recorded audio includes:

[0035] The accompaniment audio of the target song and the dry audio to be synthesized are synthesized to obtain the recorded audio.

[0036] To achieve the above-mentioned purpose, the second aspect of the present application provides an electronic device, including:

[0037] Memory for storing computer programs;

[0038] A processor is used to implement the steps of the song recording method as described above when executing the computer program.

[0039] To achieve the above-mentioned purpose, the third aspect of the present application provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the song recording method as described above are implemented.

[0040] To achieve the above objectives, the fourth aspect of the present application provides a computer program product, including a computer program, which implements the steps of the above song recording method when executed by a processor.

[0041] It can be seen from the above scheme that a song recording method provided by the present application includes: after a user account logs in to a song application, in response to a recording command of a target song triggered in the song application, playing the audio of the target song containing the original singer's vocals; in the process of playing the target song, obtaining input audio through an audio input device; extracting target dry voice audio that matches the voiceprint feature of the user account from the input audio; and synthesizing the accompaniment audio of the target song and the target dry voice audio to obtain the recorded audio.

[0042] The song recording method provided by the present application obtains the voiceprint features of the user account in advance, and during the recording of the target song, extracts the target dry audio that matches the voiceprint features and the music melody of the target song from the input audio collected by the audio input device, removes the original vocals of the target song in the input audio, and sings the accompaniment audio of the target song with the target dry audio during song synthesis, eliminating the original vocals in the final synthesized audio, thereby improving the user experience. The present application also discloses an electronic device, a computer-readable storage medium, and a computer program product, which can also achieve the above technical effects.

[0043] It should be understood that the foregoing general description and the following detailed description are exemplary only and are not restrictive of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following is a brief introduction to the drawings required for use in the embodiments or the prior art descriptions. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work. The drawings are used to provide a further understanding of the present disclosure and constitute a part of the specification. Together with the following specific implementation methods, they are used to explain the present disclosure, but do not constitute a limitation to the present disclosure. In the drawings:

[0045] Figure 1 A flowchart of a song playing software in the related art;

[0046] Figure 2 It is a flowchart of a recording and singing software in the related art;

[0047] Figure 3 An architecture diagram of a song recording system provided in an embodiment of the present application;

[0048] Figure 4 A flowchart of a song recording method provided in an embodiment of the present application;

[0049] Figure 5A flowchart of another song recording method provided in an embodiment of the present application;

[0050] Figure 6 A preprocessing flow chart of an application embodiment provided in an embodiment of the present application;

[0051] Figure 7 A song recording flow chart of an application embodiment provided in the embodiment of the present application;

[0052] Figure 8 A structural diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0053] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0054] There are two main types of music software in the related art: software that focuses on song playback and software that focuses on recording and singing. Figure 1 As shown in the figure, the song file is downloaded from the server or CDK (Content Delivery Network) acceleration node to the music player client, and the client plays the complete music work locally, including the original singer's voice and accompaniment. Figure 2 As shown, the accompaniment file is downloaded from the server or CDK acceleration node to the recording software client, and the client plays the accompaniment locally to guide the user to sing along, while obtaining the user's singing voice input from the microphone and synthesizing the singing voice with the accompaniment. Although the recording software also supports playing the original singer's voice during the recording process, the microphone will collect the original singer's voice while collecting the user's singing voice. When the song is synthesized and mixed, the original singer's voice in the accompaniment cannot be eliminated, resulting in the final synthesized audio containing the original singer's voice.

[0055] Therefore, in the present application, the voiceprint features of the user account are obtained in advance, and during the recording process of the target song, the target dry audio that conforms to the voiceprint features and the music melody of the target song is extracted from the input audio collected by the audio input device, and the original vocals of the target song in the input audio are eliminated. When the song is synthesized, the accompaniment audio of the target song is sung together with the target dry audio, thereby eliminating the original vocals in the final synthesized audio and improving the user experience.

[0056] In order to facilitate understanding of the song recording method provided by this application, the system used by the method is introduced below. Figure 3 , which shows an architecture diagram of a song recording system provided by an embodiment of the present application, such as Figure 3 As shown, it includes a server 10 and a terminal 20.

[0057] The terminal 20 may be a fixed terminal, a mobile terminal, an IoT device, a wearable device, etc., a fixed terminal such as a PC (Chinese full name: personal computer, English full name: Personal Computer), etc., a mobile terminal such as a mobile phone, etc., an IoT device such as a smart speaker, a smart TV, a smart air conditioner, a smart car, etc., a wearable device such as a smart watch, a smart bracelet, a head-mounted device, etc. The terminal 20 includes an audio input device and an audio output device, the audio input device is used to collect the reference dry audio of the user account, which may include a cappella dry audio and a reading dry audio, and the reference dry audio is used to generate the voiceprint feature of the user account. The terminal 20 may generate the voiceprint feature of the user account after collecting the reference dry audio of the user account, or may send the voiceprint feature of the user account to the server 10, and the server 10 generates the voiceprint feature based on the reference dry audio of the user account. The audio output device is used to play the audio of the target song containing the original singer's voice when receiving the recording command of the target song, and the audio is downloaded and stored locally by the terminal 20 from the server 10. The audio input device is also used to collect input audio during the recording process.

[0058] Furthermore, the terminal 20 or the server 10 is used to extract the target dry audio corresponding to the target song from the input audio based on the voiceprint feature of the user account, and synthesize the accompaniment audio of the target song and the target dry audio to obtain the recorded audio. The server 10 is also used to store the recorded audio, and the recorded audio can be output through the audio output device of the terminal 20.

[0059] The embodiment of the present application discloses a song recording method, which supports playing an accompaniment including the original singer's vocals while eliminating the original singer's vocals in the final synthesized audio during the song recording process.

[0060] See also Figure 4 , a flowchart of a song recording method provided in an embodiment of the present application, such as Figure 4 As shown, including:

[0061] S101: After a user account logs into a song application, in response to a recording command of a target song triggered in the song application, playing the audio of the target song including the original singer's voice;

[0062] In a specific implementation, the user logs in to the song application through the user account, selects the target song to be recorded in the song application, triggers the recording command of the target song, and starts playing the audio of the target song including the original singer's vocals through the audio output device, so that the user can hear the original singer of the target song and guide the user to record the target song.

[0063] S102: Acquiring input audio through an audio input device during the playing of the target song;

[0064] In a specific implementation, during the process of playing the target song, the user records the singing voice through the audio input device. The audio input device refers to a device used to capture sound signals and convert them into electrical signals, such as a microphone, which can be a built-in microphone in an electronic device running a song application, or an external microphone or a wireless microphone connected to the electronic device. The input audio collected by the audio input device includes not only the singing voice of the user, but also the original singer's voice.

[0065] S103: extracting a target dry audio that matches the voiceprint feature of the user account from the input audio;

[0066] In a specific implementation, since the input audio collected by the audio input device includes not only the singing voice of the user but also the original singer's voice, in this step, the target dry audio that matches the voiceprint features of the user account and the music melody of the target song is extracted from the input audio, and the original singer's voice of the target song in the input audio is eliminated.

[0067] As a feasible implementation, this embodiment further includes: obtaining the voiceprint features of the user account. In a specific implementation, the voiceprint features of the user are obtained in advance. The voiceprint features are used to describe the unique attributes of an individual's voice, including pitch, timbre, rhythm, etc. By analyzing the voiceprint features, a digital model representing the user's voice can be created for identifying and processing the user's voice.

[0068] As a feasible implementation method, the method of obtaining the voiceprint features of the user account includes: obtaining a baseline dry audio of the user account; wherein the baseline dry audio includes a cappella dry audio and / or reading dry audio, and the a cappella dry audio includes a cappella dry audio of different pitches; inputting the baseline dry audio into an audio feature extraction model to extract the voiceprint features of the user account. The audio feature extraction model can be a variety of types of models, including traditional machine learning models and deep learning models, such as the Wav2Vec model, which can be obtained through pre-training. The main function of the audio feature extraction model is to extract voiceprint features from the baseline dry audio that can reflect the essential characteristics of the sound and be distinguished from other dry audio, which can include the frequency, energy, pitch, rhythm, etc. of the audio. The process of extracting voiceprint features by the audio feature extraction model usually includes the following steps: preprocessing, including normalization, pre-emphasis, windowing and framing, etc., to eliminate the size differences between different sample sounds and extract their short-term characteristics; extracting sound features, such as MFCC (Mel-Frequency Cepstral Coefficients) features and LPC (Linear Predictive Coding) features; using deep learning models, such as Wav2Vec, you can directly extract features from the original audio. These features contain complex information about the sound and are suitable for tasks such as voiceprint recognition.

[0069] In a specific implementation, after the song recording software is started, the user is prompted whether to turn on the automatic song synthesis function. If it is turned on, the user is prompted to input reference dry audio, which may include dry audio for a cappella singing, dry audio covering high, medium and low pitches, and dry audio for text reading. The reference dry audio is input into the audio feature extraction model to extract the user's voiceprint features.

[0070] As a feasible implementation method, extracting target dry audio that matches the voiceprint features of the user account from the input audio includes: extracting all dry audios from the input audio, matching the target dry audio that matches the voiceprint features of the user account among all dry audios; determining whether the target dry audio matches the musical melody of the target song; and if so, entering the step of synthesizing the accompaniment audio of the target song with the target dry audio to obtain the recorded audio.

[0071] In a specific implementation, stereo echo cancellation technology or vocal separation technology is used to extract all dry audio from the input audio, and then all dry audio is matched with the voiceprint features of the user account, and the similarity between the dry audio extracted from the input audio and the user's voiceprint features is calculated, and the dry audio with a similarity greater than a preset value is used as a candidate dry audio, and then the candidate dry audio with the highest similarity is taken as the target dry audio, that is, the target dry audio is the user's voice. It is determined whether the target dry audio matches the melody and rhythm of the target song. If it matches, it means that the target dry audio is the dry voice of the user singing the target song. If it does not match, it means that the user is not singing the target song.

[0072] As a feasible implementation method, the determination of whether the target dry sound audio conforms to the music melody of the target song includes: using at least one audio matching method to determine whether the target dry sound audio conforms to the music melody of the target song; wherein the audio matching method includes an audio fingerprint matching method, a fundamental frequency midi matching method, and a lyrics matching method; if multiple audio matching methods are used, a comprehensive determination of whether the target dry sound audio conforms to the music melody of the target song is made based on the judgment results of the multiple audio matching methods.

[0073] In a specific implementation, one or more audio matching methods are used to determine whether the target dry audio meets the music melody of the target song. The audio matching methods may include audio fingerprint matching methods, baseband midi (Musical Instrument Digital Interface, music device digital interface protocol) matching methods, lyrics matching methods, etc. If multiple audio matching methods are used, the target dry audio is judged to be consistent with the music melody of the target song based on the judgment results of the multiple audio matching methods. For example, if an odd number of audio matching methods are used, a majority voting mechanism can be used. If the majority judgment result determines that the target dry audio meets the music melody of the target song, then the target dry audio is considered to meet the music melody of the target song. For another example, the judgment result of each audio matching method can be digitally represented, and the judgment result of compliance is represented as 1, and the judgment result of non-compliance is represented as 0. Then, a corresponding weight is assigned to each audio matching method, and the judgment results of all audio matching methods are weighted averaged. If the weighted average value is greater than a preset threshold, the target dry audio is judged to meet the music melody of the target song, otherwise the target dry audio is judged to be inconsistent with the music melody of the target song.

[0074] As a feasible implementation method, an audio fingerprint matching method is used to determine whether the target dry audio conforms to the music melody of the target song, including: extracting the target audio fingerprint of the target dry audio; comparing the target audio fingerprint with the standard audio fingerprint of the target song, and determining whether the similarity between the target audio fingerprint and the standard audio fingerprint of the target song is greater than or equal to a preset value; if so, determining whether the target dry audio conforms to the music melody of the target song.

[0075] In a specific implementation, first, the target audio fingerprint of the target dry audio is extracted. The audio fingerprint is a digital identifier representing the unique characteristics of the audio. It is generated by analyzing the characteristics of the audio and is used to identify and compare the audio content. Then, the target audio fingerprint is compared with the standard audio fingerprint of the target song stored in advance. By calculating the similarity between the two, it can be determined whether the target dry audio matches the melody of the target song. If the similarity is greater than or equal to the preset value, then it is determined that the target dry audio matches the musical melody of the target song, otherwise it is determined that the target dry audio does not match the musical melody of the target song.

[0076] As a feasible implementation method, a fundamental frequency midi matching method is used to determine whether the target dry sound audio conforms to the music melody of the target song, including: extracting the target fundamental frequency midi of the target dry sound audio; comparing the target fundamental frequency midi with the standard target fundamental frequency midi of the target song, and determining whether the similarity between the target fundamental frequency midi and the standard target fundamental frequency midi of the target song is greater than or equal to a preset value; if so, determining whether the target dry sound audio conforms to the music melody of the target song.

[0077] In a specific implementation, first, the target dry audio is analyzed to extract its fundamental frequency information, and converted into MIDI data, namely, target fundamental frequency MIDI, which can represent the pitch and rhythm of the target dry audio. Then, the target fundamental frequency MIDI is compared with the standard fundamental frequency MIDI of the target song stored in advance. By calculating the similarity between the two, it can be determined whether the target dry audio matches the melody of the target song. If the similarity is greater than or equal to the preset value, then it is determined that the target dry audio matches the music melody of the target song, otherwise it is determined that the target dry audio does not match the music melody of the target song.

[0078] As a feasible implementation method, a lyrics matching method is used to determine whether the target dry voice audio conforms to the music melody of the target song, including: extracting target lyrics of the target dry voice audio; comparing the target lyrics with the standard lyrics of the target song, and determining whether the similarity between the target lyrics and the standard lyrics of the target song is greater than or equal to a preset value; if so, determining whether the target dry voice audio conforms to the music melody of the target song.

[0079] In a specific implementation, first, the target lyrics of the target dry audio are extracted. Then, the target lyrics are compared with the standard lyrics of the target song stored in advance. By calculating the similarity between the two, it can be determined whether the target dry audio matches the melody of the target song. If the similarity is greater than or equal to a preset value, then it is determined that the target dry audio matches the music melody of the target song, otherwise it is determined that the target dry audio does not match the music melody of the target song.

[0080] As a preferred implementation, after determining whether the target dry audio matches the music melody of the target song, it further includes: if the target dry audio matches the music melody of the target song, performing denoising and dereverberation operations on the target dry audio to obtain the dry audio to be synthesized. In a specific implementation, the denoising and dereverberation operations are performed on the target dry audio to filter the background sound and noise to obtain a pure dry audio, i.e., the dry audio to be synthesized.

[0081] S104: synthesizing the accompaniment audio of the target song and the target dry sound audio to obtain the recorded audio.

[0082] In this step, the extracted target dry audio recorded by the user is synthesized with the accompaniment audio of the target song to complete the recording of the target song and obtain the recorded audio. If the target dry audio is subjected to denoising and reverberation operations to obtain the dry audio to be synthesized, then in this step, the accompaniment audio of the target song and the dry audio to be synthesized are synthesized to obtain the recorded audio. After the recording is completed, the user can be prompted to select whether to save or publish the recorded audio, or the audio can be automatically saved or published according to the user's pre-set settings.

[0083] The song recording method provided in the embodiment of the present application obtains the voiceprint features of the user account in advance, and during the recording process of the target song, extracts the target dry audio that conforms to the voiceprint features and the music melody of the target song from the input audio collected by the audio input device, eliminates the original vocals of the target song in the input audio, and combines the accompaniment audio of the target song with the target dry audio during song synthesis, thereby eliminating the original vocals in the final synthesized audio and improving the user experience.

[0084] The embodiment of the present application discloses a song recording method. Compared with the above embodiment, this embodiment further explains and optimizes the technical solution. Specifically:

[0085] See also Figure 5 , a flowchart of another song recording method provided in an embodiment of the present application, such as Figure 5 As shown, including:

[0086] S201: Obtaining a baseline dry voice audio of a user account; wherein the baseline dry voice audio includes a cappella dry voice audio and / or a reading dry voice audio, and the cappella dry voice audio includes a cappella dry voice audio of different pitches;

[0087] S202: Inputting the reference dry audio into an audio feature extraction model to extract the voiceprint feature of the user account;

[0088] In a specific implementation, after the song recording software is started, a user input baseline dry voice audio is obtained, which may include dry voice audio of a cappella singing, dry voice audio covering different pitches, and dry voice audio of text reading. The baseline dry voice audio is input into an audio feature extraction model to extract the user's voiceprint features.

[0089] S203: After the user account logs into the song application, in response to a recording command of a target song triggered in the song application, playing the audio of the target song including the original singer's voice;

[0090] In the specific implementation, after the user selects the target song he wants to record, the audio of the target song including the original singer's vocals will start to be played through the audio output device. It supports playing the target music in the form of external speakers, external speakers, etc., so that the user can hear the original singer of the target song and guide the user to record the target song.

[0091] S204: Acquiring input audio through an audio input device during the playing of the target song;

[0092] S205: extracting all dry audios from the input audio, and matching target dry audios that meet the voiceprint feature among all dry audios;

[0093] In a specific implementation, stereo echo cancellation technology or voice separation technology is used to extract all dry audio from the input audio, and then all dry audio is matched with the user's voiceprint features, and the dry audio with the highest matching degree is selected as the target dry audio.

[0094] S206: extracting the target audio fingerprint of the target dry audio, comparing the target audio fingerprint with the standard audio fingerprint of the target song, and determining whether the similarity between the target audio fingerprint and the standard audio fingerprint of the target song is greater than or equal to a preset value; if so, proceeding to S207;

[0095] In a specific implementation, first, the target audio fingerprint of the target dry audio is extracted. The audio fingerprint is a digital identifier representing the unique characteristics of the audio. It is generated by analyzing the characteristics of the audio and is used to identify and compare the audio content. Then, the target audio fingerprint is compared with the standard audio fingerprint of the target song stored in advance. By calculating the similarity between the two, it can be determined whether the target dry audio matches the melody of the target song. If the similarity is greater than or equal to the preset value, then it is determined that the target dry audio matches the musical melody of the target song, otherwise it is determined that the target dry audio does not match the musical melody of the target song.

[0096] S207: performing a denoising operation and a dereverberation operation on the target dry audio to obtain a dry audio to be synthesized;

[0097] In a specific implementation, a denoising operation and a dereverberation operation are performed on the target dry audio to filter the background sound and noise to obtain a pure dry audio, that is, the dry audio to be synthesized.

[0098] S208: synthesizing the accompaniment audio of the target song and the dry audio to be synthesized to obtain the recorded audio.

[0099] It can be seen that the present embodiment can support the playback of target music in the form of external speakers, external speakers, etc., and automatically monitor and identify the target dry sound audio of the user account that matches the melody of the target music during the music playback process, and perform denoising and de-reverberation operations on it to improve the quality of the target dry sound audio, and synthesize the target dry sound audio with the accompaniment audio into a musical work. The synthesized musical work does not include the original singer's vocals, and has a better user experience.

[0100] The following describes an application example provided by the present application. The preprocessing process is as follows: Figure 6 As shown, the following steps are included: application startup and user registration; determining whether to enable automatic recording and synthesis; prompting the user to input dry voice and record audio input; using dry voice pre-training to establish a user voice model; and the pre-processing is completed.

[0101] Song recording process Figure 7 As shown, the following steps are included:

[0102] Step 1: Start the app and play music;

[0103] Step 2: Monitor the audio to determine whether there is dry sound input; if so, proceed to step 3;

[0104] Step 3: Detect dry sound and determine whether it matches the pre-trained sound model; if so, proceed to step 4;

[0105] Step 4: Audio fingerprint detection to determine whether it is similar to the music being played; if so, proceed to step 5;

[0106] Step 5: Use the pre-trained model to eliminate background noise and retain only the target user’s dry voice;

[0107] Step 6: Pull the unvoiced accompaniment and synthesize it with the user's dry voice;

[0108] Step 7: Prompt the user to choose whether to save or publish the composite work;

[0109] Step 8: Save, publish or cancel to end.

[0110] A song recording device provided in an embodiment of the present application is introduced below. The song recording device described below and the song recording method described above can be referred to each other. The song recording device includes:

[0111] A playing module, for playing the audio of the target song including the original singer's voice in response to a recording command of the target song triggered by the song application after the user account logs in to the song application;

[0112] A first acquisition module, used for acquiring input audio through an audio input device during the playing of the target song;

[0113] An extraction module, configured to extract a target dry voice audio that matches the voiceprint feature of the user account from the input audio;

[0114] A synthesis module is used to synthesize the accompaniment audio of the target song and the target dry sound audio to obtain the recorded audio.

[0115] The song recording device provided in the embodiment of the present application obtains the voiceprint features of the user account in advance, and during the process of recording the target song, extracts the target dry audio that conforms to the voiceprint features and the music melody of the target song from the input audio collected by the audio input device, eliminates the original vocals of the target song in the input audio, and combines the accompaniment audio of the target song with the target dry audio during song synthesis, thereby eliminating the original vocals in the final synthesized audio and improving the user experience.

[0116] Based on the above embodiment, as a preferred implementation, it also includes:

[0117] The second acquisition module is used to obtain the voiceprint feature of the user account.

[0118] On the basis of the above embodiments, as a preferred implementation mode, the second acquisition module is specifically used to: obtain a baseline dry voice audio of the user account; wherein the baseline dry voice audio includes a cappella dry voice audio and / or a reading dry voice audio, and the a cappella dry voice audio includes a cappella dry voice audio of different pitches; input the baseline dry voice audio into an audio feature extraction model to extract the voiceprint features of the user account.

[0119] On the basis of the above embodiments, as a preferred implementation mode, the extraction module is specifically used to: extract all dry audios from the input audio, and match target dry audios that match the voiceprint features of the user account among all dry audios; determine whether the target dry audio matches the musical melody of the target song; if so, enter the step of synthesizing the accompaniment audio of the target song and the target dry audio to obtain the recorded audio.

[0120] On the basis of the above embodiments, as a preferred implementation mode, the extraction module is specifically used to: use at least one audio matching method to determine whether the target dry sound audio conforms to the music melody of the target song; wherein the audio matching method includes an audio fingerprint matching method, a fundamental frequency midi matching method, and a lyrics matching method; if multiple audio matching methods are used, then a comprehensive judgment is made as to whether the target dry sound audio conforms to the music melody of the target song based on the judgment results of multiple audio matching methods.

[0121] Based on the above embodiments, as a preferred implementation mode, the extraction module is specifically used to: extract the target audio fingerprint of the target dry sound audio; compare the target audio fingerprint with the standard audio fingerprint of the target song, and determine whether the similarity between the target audio fingerprint and the standard audio fingerprint of the target song is greater than or equal to a preset value; if so, determine whether the target dry sound audio conforms to the musical melody of the target song.

[0122] On the basis of the above embodiments, as a preferred implementation mode, the extraction module is specifically used to: extract the target fundamental frequency midi of the target dry sound audio; compare the target fundamental frequency midi with the standard target fundamental frequency midi of the target song, and determine whether the similarity between the target fundamental frequency midi and the standard target fundamental frequency midi of the target song is greater than or equal to a preset value; if so, determine whether the target dry sound audio conforms to the musical melody of the target song.

[0123] On the basis of the above embodiments, as a preferred implementation mode, the extraction module is specifically used to: extract target lyrics of the target dry voice audio; compare the target lyrics with the standard lyrics of the target song to determine whether the similarity between the target lyrics and the standard lyrics of the target song is greater than or equal to a preset value; if so, determine whether the target dry voice audio conforms to the music melody of the target song.

[0124] Based on the above embodiment, as a preferred implementation, it also includes:

[0125] An operation module, configured to perform a denoising operation and a dereverberation operation on the target dry audio to obtain the dry audio to be synthesized;

[0126] Correspondingly, the synthesis module is specifically used to synthesize the accompaniment audio of the target song and the dry sound audio to be synthesized to obtain the recorded audio.

[0127] The present application also provides an electronic device, see Figure 8 , a structural diagram of an electronic device provided in an embodiment of the present application, such as Figure 8 As shown, a processor 11 and a memory 12 may be included.

[0128] Among them, the processor 11 may include one or more processing cores, such as a 4-core processor, an 8-core processor, etc. The processor 11 can be implemented in at least one hardware form of DSP (Digital Signal Processing), FPGA (Field-Programmable Gate Array), and PLA (Programmable Logic Array). The processor 11 may also include a main processor and a coprocessor. The main processor is a processor for processing data in the awake state, also known as a CPU (Central Processing Unit); the coprocessor is a low-power processor for processing data in the standby state. In some embodiments, the processor 11 may be integrated with a GPU (Graphics Processing Unit), which is responsible for rendering and drawing the content to be displayed on the display screen. In some embodiments, the processor 11 may also include an AI (Artificial Intelligence) processor, which is used to process computing operations related to machine learning.

[0129] The memory 12 may include one or more computer-readable storage media, which may be non-transitory. The memory 12 may also include a high-speed random access memory, and a non-volatile memory, such as one or more disk storage devices, flash memory storage devices. In this embodiment, the memory 12 is at least used to store the following computer program 121, wherein, after the computer program is loaded and executed by the processor 11, it can implement the relevant steps in the song recording method performed by the electronic device side disclosed in any of the aforementioned embodiments. In addition, the resources stored in the memory 12 may also include an operating system 122 and data 123, etc., and the storage method may be short-term storage or permanent storage. Among them, the operating system 122 may include Windows, Unix, Linux, etc.

[0130] In some embodiments, the electronic device may further include a display screen 13 , an input / output interface 14 , a communication interface 15 , a sensor 16 , a power source 17 , and a communication bus 18 .

[0131] certainly, Figure 8 The structure of the electronic device shown does not constitute a limitation on the electronic device in the embodiments of the present application. In actual applications, the electronic device may include Figure 8 More or fewer components than shown, or combinations of certain components.

[0132] In another exemplary embodiment, a computer-readable storage medium including program instructions is also provided. When the program instructions are executed by a processor, the steps of the song recording method performed by the electronic device in any of the above embodiments are implemented.

[0133] In another exemplary embodiment, a computer program product including a computer program is also provided. When the computer program is executed by a processor, the steps of the song recording method performed by the electronic device in any of the above embodiments are implemented.

[0134] The various embodiments in the specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments, and the same and similar parts between the various embodiments can be referred to each other. For the device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part description. It should be pointed out that for ordinary technicians in this technical field, without departing from the principles of this application, several improvements and modifications can be made to the present application, and these improvements and modifications also fall within the scope of protection of the claims of this application.

[0135] It should also be noted that, in this specification, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "comprise", "include" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the statement "comprises a ..." does not exclude the presence of other identical elements in the process, method, article or device including the element.

Claims

1. A song recording method, characterized in that: include: After the user account logs into the song application, in response to a recording command of a target song triggered in the song application, playing the audio of the target song including the original singer's voice; During the playing of the target song, obtaining input audio through an audio input device; Extracting a target dry voice audio that matches the voiceprint feature of the user account from the input audio; The accompaniment audio of the target song and the target dry sound audio are synthesized to obtain the recorded audio.

2. The song recording method according to claim 1, characterized in that: Also includes: Obtaining a baseline dry voice audio of the user account; wherein the baseline dry voice audio includes a cappella dry voice audio and / or a reading dry voice audio, and the a cappella dry voice audio includes a cappella dry voice audio of different pitches; The reference dry audio is input into an audio feature extraction model to extract the voiceprint features of the user account.

3. The song recording method according to claim 1, characterized in that: Extracting target dry audio that matches the voiceprint feature of the user account from the input audio includes: Extracting all dry audios from the input audio, and matching target dry audios that match the voiceprint features of the user account among all dry audios; Determining whether the target dry audio matches the music melody of the target song; If so, the step of synthesizing the accompaniment audio of the target song and the target dry sound audio to obtain the recorded audio is entered.

4. The song recording method according to claim 3, characterized in that: The determining whether the target dry audio matches the music melody of the target song includes: At least one audio matching method is used to determine whether the target dry sound audio matches the music melody of the target song; wherein the audio matching method includes an audio fingerprint matching method, a baseband midi matching method, and a lyrics matching method; If multiple audio matching methods are used, a comprehensive judgment is made based on the judgment results of the multiple audio matching methods as to whether the target dry sound audio matches the musical melody of the target song.

5. The song recording method according to claim 4, characterized in that: The audio fingerprint matching method is used to determine whether the target dry audio matches the music melody of the target song, including: Extracting a target audio fingerprint of the target dry audio; Comparing the target audio fingerprint with the standard audio fingerprint of the target song, determining whether the similarity between the target audio fingerprint and the standard audio fingerprint of the target song is greater than or equal to a preset value; If so, it is determined whether the target dry audio matches the musical melody of the target song.

6. The song recording method according to claim 4, characterized in that: The baseband midi matching method is used to determine whether the target dry sound audio matches the music melody of the target song, including: Extracting a target fundamental frequency midi of the target dry audio; Compare the target fundamental frequency midi with the standard target fundamental frequency midi of the target song, and determine whether the similarity between the target fundamental frequency midi and the standard target fundamental frequency midi of the target song is greater than or equal to a preset value; If so, it is determined whether the target dry audio matches the musical melody of the target song.

7. The song recording method according to claim 4, characterized in that: Using a lyrics matching method to determine whether the target dry audio matches the music melody of the target song includes: Extract target lyrics of the target dry audio; Comparing the target lyrics with the standard lyrics of the target song, and determining whether the similarity between the target lyrics and the standard lyrics of the target song is greater than or equal to a preset value; If so, it is determined whether the target dry audio matches the musical melody of the target song.

8. The song recording method according to claim 3, characterized in that: After determining whether the target dry audio matches the music melody of the target song, the method further includes: If the target dry audio matches the music melody of the target song, performing denoising and dereverberation operations on the target dry audio to obtain the dry audio to be synthesized; Accordingly, the synthesizing the accompaniment audio of the target song and the target dry sound audio to obtain the recorded audio includes: The accompaniment audio of the target song and the dry audio to be synthesized are synthesized to obtain the recorded audio.

9. An electronic device, characterized in that: include: Memory for storing computer programs; A processor, used to implement the steps of the song recording method as claimed in any one of claims 1 to 8 when executing the computer program.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, which, when executed, implements the steps of the song recording method according to any one of claims 1 to 8.

Images