Real-time accompaniment tuning method and device, storage medium and program product

Abstract

The invention discloses a real-time accompaniment tuning method and device, a storage medium and a program product. The method comprises the following steps: in response to a playing instruction for a target song, obtaining musical instrument audio data of each musical instrument corresponding to the target song; the musical instrument audio data is obtained by performing sound source separation on a music file of the target song according to musical instrument types; obtaining an adjustment strategy of a user for a target musical instrument in the target song in real time; performing adjustment processing on the musical instrument audio data corresponding to the target musical instrument according to the adjustment strategy to obtain adjusted musical instrument audio data; and based on the adjusted musical instrument audio data and the musical instrument audio data corresponding to the other musical instruments, obtaining customized accompaniment audio corresponding to the music file through sound mixing, and playing the customized accompaniment audio. Customized accompaniment audio meeting user requirements can be generated in real time.

Description

Technical Field

[0001] The present invention relates to the technical field of audio data processing, and in particular to a real-time accompaniment tuning method, equipment, storage medium and program product. Background Art

[0002] At present, it is a common music entertainment activity for users to sing songs online or offline through mobile phones, car stereos, jukeboxes, large-screen TVs and other terminals. During the singing process, the accompaniment file of the song will be played to obtain the rhythm reference and atmosphere rendering effect of the singing. The effect of the accompaniment will greatly affect the effect of the singing, and the expectations of the accompaniment effect will also change with the user's personal preferences and on-the-spot emotions. Usually, the accompaniment of a song is fixed. Therefore, how to achieve real-time customized accompaniment tuning and improve the user experience is a problem that needs to be solved urgently. Summary of the invention

[0003] In view of this, the purpose of the present invention is to provide a real-time accompaniment tuning method, device, storage medium and program product, which can generate customized accompaniment audio that meets user needs in real time. The specific scheme is as follows:

[0004] In a first aspect, the present application discloses a real-time accompaniment tuning method, comprising:

[0005] In response to a play instruction for a target song, obtain instrument audio data of each instrument corresponding to the target song; the instrument audio data is obtained by separating the sound sources of the music file of the target song according to the type of instrument;

[0006] Acquire the user's adjustment strategy for the target instrument in the target song in real time;

[0007] Performing adjustment processing on the instrument audio data corresponding to the target instrument according to the adjustment strategy to obtain adjusted instrument audio data;

[0008] Based on the adjusted instrument audio data and the instrument audio data corresponding to the remaining instruments, the customized accompaniment audio corresponding to the music file is obtained by mixing, and the customized accompaniment audio is played.

[0009] Optionally, before obtaining the instrument audio data of each instrument corresponding to the target song, the method further includes:

[0010] Separating the music files of the target song according to the types of musical instruments to obtain initial musical instrument audio data corresponding to each musical instrument;

[0011] The initial musical instrument audio data is optimized to obtain the musical instrument audio data of each musical instrument; the optimization includes at least one of alignment and noise reduction; the alignment includes alignment verification of all the initial musical instrument audio data corresponding to the same music file; the noise reduction includes noise reduction of the initial musical instrument audio data.

[0012] Optionally, the optimization process includes alignment process, and the optimization process is performed on the initial instrument audio data to obtain the instrument audio data of each instrument, including:

[0013] Performing alignment check on all the initial instrument audio data corresponding to the same music file;

[0014] For the initial instrument audio data that fails the alignment check, blank data of corresponding duration is added to obtain the instrument audio data of each instrument.

[0015] Optionally, after obtaining the instrument audio data of each instrument corresponding to the target song, the method further includes:

[0016] Encoding the musical instrument audio data simultaneously by using multiple threads to obtain digital format audio corresponding to the musical instrument audio data;

[0017] The storage and the adjustment processing are performed based on the digital format audio.

[0018] Optionally, the adjustment strategy includes any one or more of loudness adjustment, equalization processing, reverberation processing, and timbre conversion synthesis.

[0019] Optionally, the real-time acquisition of the user's adjustment strategy for the target instrument in the target song includes:

[0020] Displaying tuning options of each musical instrument corresponding to the music file through a human-computer interaction interface;

[0021] According to the user's selection instruction for the target tuning option, a user's adjustment strategy for the target instrument is generated.

[0022] Optionally, after obtaining the user's adjustment strategy for the target instrument in the target song in real time, the method further includes:

[0023] Saving the adjustment strategy as a historical adjustment strategy;

[0024] When the real-time accompaniment tuning demand of the same user for the music file is obtained again, the instrument audio data corresponding to the music file is adjusted according to the historical adjustment strategy.

[0025] Optionally, the adjusting the instrument audio data corresponding to the target instrument according to the adjustment strategy to obtain adjusted instrument audio data includes:

[0026] Based on the playing progress of the target song, determine the to-be-played segment in the instrument audio data of the target instrument;

[0027] Adjust the to-be-played segment according to the adjustment strategy to obtain an adjusted to-be-played segment;

[0028] The step of obtaining the customized accompaniment audio corresponding to the target song by mixing the adjusted instrument audio data and the instrument audio data corresponding to other instruments includes:

[0029] Based on the adjusted to-be-played segment and the to-be-played segments in the instrument audio data of other instruments, obtaining the to-be-played customized accompaniment audio segment by mixing;

[0030] The playing of the customized accompaniment audio includes:

[0031] When the playing progress of the target song reaches the target time point, the customized accompaniment audio clip is played.

[0032] Optionally, the step of obtaining a customized accompaniment audio corresponding to the music file by mixing based on the adjusted instrument audio data and the instrument audio data corresponding to other instruments, and playing the customized accompaniment audio includes:

[0033] Obtain a first timestamp corresponding to the current playing point of the target song;

[0034] According to the first timestamp, locating a second timestamp corresponding to the first timestamp from the musical instrument audio data corresponding to each musical instrument or the adjusted musical instrument audio data;

[0035] Based on the adjusted instrument audio data and the instrument audio data corresponding to the remaining instruments, starting from the second timestamp, the customized accompaniment audio corresponding to the music file is obtained by mixing.

[0036] In a second aspect, the present application discloses an electronic device, comprising:

[0037] Memory, used to store computer programs;

[0038] The processor is used to execute the computer program to implement the above-mentioned real-time accompaniment tuning method.

[0039] In a third aspect, the present application discloses a computer-readable storage medium for storing a computer program; wherein the computer program implements the aforementioned real-time accompaniment tuning method when executed by a processor.

[0040] In a fourth aspect, the present application discloses a computer program product, including a computer program / instruction; wherein the computer program / instruction implements the aforementioned real-time accompaniment tuning method when executed by a processor.

[0041] In the present application, in response to a play instruction for a target song, the instrument audio data of each instrument corresponding to the target song is obtained; the instrument audio data is obtained by separating the sound source of the music file of the target song according to the type of instrument; the user's adjustment strategy for the target instrument in the target song is obtained in real time; the instrument audio data corresponding to the target instrument is adjusted according to the adjustment strategy to obtain the adjusted instrument audio data; based on the adjusted instrument audio data and the instrument audio data corresponding to the remaining instruments, the customized accompaniment audio corresponding to the music file is obtained by mixing, and the customized accompaniment audio is played. It can be seen that by dividing the accompaniment into multiple instrument audio data according to the instrument type, the user can adjust and process various instruments in real time during the singing process, and finally, according to the adjusted instrument audio data corresponding to the target instrument and the instrument audio data corresponding to the remaining instruments, the customized accompaniment audio file that meets the user's needs is mixed, so as to realize flexible tuning of the instruments in the accompaniment and provide the user with a personalized singing experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying creative work.

[0043] FIG1 is a flow chart of a real-time accompaniment tuning method provided by the present application;

[0044] FIG2 is a flow chart of a specific real-time accompaniment tuning method provided by the present application;

[0045] Figure 3 A schematic diagram of a specific real-time accompaniment tuning interface provided by this application;

[0046] Figure 4 A schematic diagram of a specific real-time accompaniment tuning system framework provided in this application;

[0047] FIG5 is a structural diagram of an electronic device provided in this application. DETAILED DESCRIPTION

[0048] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0049] In the prior art, it is a common music entertainment activity for users to sing songs online or offline through terminals such as mobile phones, car stereos, jukeboxes, and large-screen TVs. During the singing process, the accompaniment file of the song will be played to obtain the rhythm reference and atmosphere rendering effect of the singing. The effect of the accompaniment will greatly affect the effect of the singing, and the expectations for the accompaniment effect will also change with the user's personal preferences and on-the-spot emotions. Usually, the accompaniment of a song is fixed. Therefore, how to achieve real-time customized accompaniment tuning and improve the user's experience is a problem that urgently needs to be solved. In order to overcome the above-mentioned technical problems, the present application proposes a real-time accompaniment tuning method that can generate customized accompaniment audio that meets user needs in real time.

[0050] The present application embodiment discloses a real-time accompaniment tuning method, as shown in FIG1 , the method may include the following steps:

[0051] Step S11: in response to a play instruction for a target song, obtain instrument audio data of each instrument corresponding to the target song; the instrument audio data is obtained by separating the sound sources of the music file of the target song according to the type of instrument.

[0052] First, obtain the instrument audio data corresponding to the target song that needs to be customized for accompaniment. The instrument audio data can be obtained by separating the sound source of the music file of the target song according to the type of instrument in advance. Music Source Separation (MSS) refers to the technology of separating the signals of different sound sources (such as human voice, musical instrument, accompaniment, etc.) mixed in music. Its main approach is to extract the sound source of individual instruments and human voices from an audio file containing multiple music; such as separating the piano audio, guitar audio and drum audio of the target song. The music file of the above-mentioned target song can be a music file of a complete song or a music file of an accompaniment version. The music file of a complete song refers to a music file including all instruments, sound effects and lead singer's voice. The music file of the accompaniment version only contains the instrument part, but does not include the lead singer's voice. In the specific implementation process, it can be firstly obtained from the user's singing instruction for the target song, and the instrument audio data corresponding to the target song is actively obtained according to the singing instruction.

[0053] Before obtaining the instrument audio data of each instrument corresponding to the target song, the music file can be separated in advance to obtain the instrument audio data. Specifically, the music file of the target song is separated according to the type of instrument to obtain the initial instrument audio data corresponding to each instrument. For example, based on the existing song or accompaniment audio, the audio is separated into multiple sound sources according to the instrument using separation models such as Demucs, Spleeter, and BSRNN. Under normal circumstances, the performance of a song includes piano, guitar, bass, drums, violin and other instruments. After separation, the instrument audio data of each instrument is an independent audio data. Further, the initial instrument audio data is optimized to obtain the instrument audio data of each instrument; the optimization process includes at least one of alignment processing and noise reduction processing; the alignment process includes alignment verification of all the initial instrument audio data corresponding to the same music file; the noise reduction process includes noise reduction processing of the initial instrument audio data; that is, it is necessary to perform alignment verification on all the instrument audio data corresponding to the same music file, and the alignment verification is specifically a time alignment verification. It can be understood that the problem of rhythm disorder after the mixing of the instrument audio data is avoided by time alignment.

[0054] Furthermore, for the initial instrument audio data that fails the alignment check, blank data of corresponding duration is added to obtain the instrument audio data of each instrument, so as to ensure that all the instrument audio data corresponding to the same music file are aligned in time. It is understandable that, since some instruments appear at the beginning of the music playing, while some instruments appear after a period of time, the instrument audio data of such instruments may not be aligned. Therefore, for the instrument audio data that is not aligned in time, the music file is used as a reference, and blank data of corresponding duration is added to align it with the music file and the instrument audio data in time.

[0055] In another embodiment, an alignment check may also be performed after the musical instrument audio data is acquired. For example, as shown in FIG. 2 , an alignment check is performed based on the timestamp information of the musical instrument audio data. For musical instrument audio data that are not aligned in time, a data buffer of a corresponding size is set to keep the decoding progress of different musical instrument audio data consistent, so that different musical instrument audio data can correspond to the accompaniment content at the same time point when being played.

[0056] In addition, the sampling rate and bit depth of all the instrument audio data corresponding to the same music file can be checked. By keeping the instrument audio data at similar sampling rates and bit depths, the quality of the audio after mixing the instrument audio data can be guaranteed. In some embodiments, the instrument audio data can also be subjected to noise reduction processing. That is, noise points that may appear in each audio track are subjected to noise reduction processing to eliminate data defects.

[0057] In some embodiments, after obtaining the instrument audio data of each instrument corresponding to the target song, the method may further include: encoding the instrument audio data using multiple threads at the same time to obtain the digital format audio corresponding to the instrument audio data; and performing storage and adjustment processing based on the digital format audio. That is, the storage of the instrument audio data and the subsequent adjustment processing are performed based on the digital format audio. Therefore, it is necessary to convert the acquired audio data into digital format audio first. Specifically, in order to improve efficiency, the instrument audio data may be encoded using multiple threads at the same time. When storing, each type of instrument audio data may be saved as an independent audio file, or encapsulated in an audio file in the form of multiple tracks. The one or more files finally generated may be placed on the server or stored in the local device of the singing software for singing. Then, when the user needs to sing, the various types of instrument audio data corresponding to the song are read. If the instrument audio data is stored on the server, it is pulled from the server in real time. If the instrument audio data is stored locally, it is read from the local file system. Of course, the instrument audio data may also be stored on the server and locally at the same time to ensure that the instrument audio data is read normally.

[0058] Step S12: Acquire the user's adjustment strategy for the target instrument in the target song in real time.

[0059] Obtain the user's adjustment strategy for one or more musical instruments contained in the music file, wherein the adjustment strategy includes but is not limited to any one or more of loudness adjustment, equalization processing, reverberation processing, and timbre conversion synthesis. Specifically, the user's adjustment strategy for the target instrument in the target song can be obtained in real time through a preset tuning interface. Among them, the above-mentioned real-time acquisition of the user's adjustment strategy for the target instrument in the target song can specifically include: displaying the tuning options of each musical instrument corresponding to the music file through a human-computer interaction interface; the tuning options include loudness adjustment, equalization processing, reverberation processing, and timbre conversion synthesis; according to the user's selection instructions for the target tuning options, generate the user's adjustment strategy for the target instrument. Tone conversion synthesis refers to the effect of generating different timbres of the same type of musical instruments based on the conversion of musical instrument audio data. For example, if the instrument category of the accompaniment separation is an acoustic piano, the piano instrument is converted and synthesized into an electric piano timbre in real time during singing.

[0060] That is, when a user initiates a song, the singing software pulls the audio data of each instrument corresponding to the music from the local or cloud; when the audio data of each instrument starts playing, a real-time adjustment interface is displayed, and the software displays the audio of various instruments and related real-time tuning options to the user. Figure 3As shown in the figure, each instrument track displays the corresponding volume, equalizer and reverberation operation interface; for volume, the default volume is adjusted to the middle of the drag bar. When it is set to the minimum value, the overall volume of the instrument is 0, which is equivalent to removing the sound of the instrument in the accompaniment; similar to the conventional mixing link, the equalizer performs corresponding loudness enhancement or reduction processing on the signal of the instrument in different frequency bands; reverberation is a reflected sound effect with superimposed specific sound attenuation and delay.

[0061] It should be noted that the user can obtain the adjustment strategy for the music file in real time. For example, as shown in Figure 2, if the adjustment strategy is obtained before the audio data of each instrument is played, the instrument audio data is adjusted based on the adjustment strategy, and then the mixing is performed to obtain a customized accompaniment; or the mixing and playing can be performed directly based on the obtained instrument audio data, and the adjustment strategy is obtained during the playback process, and the remaining unplayed content is played after being processed according to the adjustment strategy; and the user can issue multiple adjustment strategies during the playback process and make multiple adjustments.

[0062] Step S13: adjusting the instrument audio data corresponding to the target instrument according to the adjustment strategy to obtain adjusted instrument audio data.

[0063] According to the user's corresponding adjustment strategies for different target instruments in the music file, the instrument audio data corresponding to the target instrument is adjusted and processed, and then the adjusted instrument audio data corresponding to the target instrument is obtained. Specifically, if the adjustment strategy is obtained during the song playback process, the instrument audio data can be fully adjusted to obtain complete adjusted instrument audio data, and then the remaining part is played based on the adjusted instrument audio data according to the current playback time point; it can also be based on the current playback time point, only the remaining unplayed fragments in the instrument audio data are adjusted to obtain the customized accompaniment audio fragment to be played, and play it; that is, it can be seen that the adjustment processing of the instrument audio data in this application can be the adjustment processing of the entire instrument audio data, or it can be the processing of the to-be-played fragments in the instrument audio data. Which specific solution to adopt can be selected by user needs. If the user needs to save the complete customized accompaniment audio of the target song under the current adjustment strategy, then choose to adjust all the instrument audio data. However, if the user triggers multiple adjustments while playing a song, that is, multiple adjustment strategies are generated, the method of only adjusting the remaining unplayed segments in the instrumental audio data is adopted. This method can avoid processing all the audio data each time and avoid wasting computing resources.

[0064] In some embodiments, the instrument audio data corresponding to the target instrument is adjusted according to the adjustment strategy to obtain adjusted instrument audio data, including: based on the playback progress of the target song, determining the to-be-played segment in the instrument audio data of the target instrument; adjusting the to-be-played segment according to the adjustment strategy to obtain the adjusted to-be-played segment; accordingly, based on the adjusted instrument audio data and the instrument audio data corresponding to the remaining instruments, obtaining the customized accompaniment audio corresponding to the target song by mixing, including: based on the adjusted to-be-played segment and the to-be-played segment in the instrument audio data of the remaining instruments, obtaining the customized accompaniment audio segment to be played by mixing; accordingly, playing the customized accompaniment audio includes: when the playback progress of the target song reaches the target time point, playing the customized accompaniment audio segment. Among them, the above-mentioned segment to be played is at least a part of the remaining unplayed segments, and specifically can be a segment that is separated from the current playback time point by a preset time interval and has a preset length of segment, and the preset length can be set as needed; for example, segment A that is 60 seconds continuous after the current time point is used as the segment to be played. After segment A is adjusted and played, if the user does not adjust it again during the playback of segment A, the next segment to be played will be adjusted based on the adjustment strategy for segment A.

[0065] After obtaining the user's adjustment strategy for the target instrument in the target song in real time, it can also include: saving the adjustment strategy as a historical adjustment strategy; when the same user's real-time accompaniment tuning requirements for the music file are obtained again, the instrument audio data corresponding to the music file is adjusted and processed according to the historical adjustment strategy. That is, considering that the same user may have fixed singing habits, after obtaining the user's adjustment strategy for the target instrument, the obtained adjustment strategy can be saved as a historical adjustment strategy. When the same user's real-time accompaniment tuning requirements for a certain music file are obtained again, if the adjustment strategy for the music file is not obtained, a prompt can be generated to indicate whether to adopt the historical adjustment strategy. If the user agrees to adopt the historical adjustment strategy, the historical adjustment strategy can be used to process the instrument audio data of the music file.

[0066] Step S14: Based on the adjusted instrument audio data and the instrument audio data corresponding to the remaining instruments, a customized accompaniment audio corresponding to the music file is obtained by mixing, and the customized accompaniment audio is played.

[0067] Finally, based on the adjusted instrument audio data corresponding to the target instrument and the remaining unprocessed instrument audio data corresponding to the music file, a customized accompaniment audio file corresponding to the music file is generated through mixing. Mixing is to mix and integrate sounds from multiple sources, such as different instruments, human voices or environmental sounds, into a stereo track or a mono track for playback by a player.

[0068] By using the sound source separation algorithm in advance, the song or accompaniment file is separated into multiple instrument tracks, which are saved as independent instrument audio files, and these audio files are marked as a group and associated with the accompaniment. When the user needs to sing a song, the multiple instrument audio files of the song are read, the basic audio information of each instrument is displayed, and each type of instrument is adjusted independently. The adjustment processing includes operations such as loudness enhancement, attenuation or zeroing, as well as independent sound effects and equalization processing for each type of instrument; finally, multi-track mixing is performed to achieve the overall accompaniment listening experience expected by the user. Of course, on the basis of real-time mixing playback, the user can also save the audio containing the singing life and customized accompaniment audio files as a customized accompaniment singing work file, that is, after the singing is over, the accompaniment audio adjusted and output during the singing process is used as the material, and mixed with the user's dry voice to obtain the final singing work.

[0069] In some embodiments, the step of obtaining the customized accompaniment audio corresponding to the music file through mixing based on the adjusted instrument audio data and the instrument audio data corresponding to the remaining instruments, and playing the customized accompaniment audio may include: obtaining a first timestamp corresponding to the current playback point of the target song; locating a second timestamp corresponding to the first timestamp from the instrument audio data corresponding to each instrument or the adjusted instrument audio data according to the first timestamp; obtaining the customized accompaniment audio corresponding to the music file through mixing based on the adjusted instrument audio data and the instrument audio data corresponding to the remaining instruments, starting from the second timestamp, and playing the customized accompaniment audio. That is, if an adjustment strategy is obtained during the audio playback process, the remaining unplayed audio needs to be played after processing according to the adjustment strategy. At this time, it is necessary to ensure the synchronization of the adjusted instrument audio data and the instrument audio data playback point. Specifically, this can be done by first determining the first timestamp corresponding to the current audio playback point, that is, the first timestamp of the current audio playback point in the music file, and then locating the corresponding second timestamp in each instrument audio data or the adjusted instrument audio data based on the first timestamp, and mixing and playing the instrument audio data and the adjusted instrument audio data starting from the second timestamp, thereby ensuring the continuity and accuracy of the music.

[0070] To further ensure the alignment of the time points, further judgment can be made after determining the second timestamp, that is, it is necessary to ensure that the second timestamp of the adjusted instrument audio data is later than the current playback point. For example, if the second timestamp is less than the playback timestamp, the audio data before the second timestamp is discarded, so that the second timestamp is slightly larger than the playback point timestamp, and the remaining data and the corresponding timestamp are written back to the corresponding storage area; otherwise, the adjusted instrument audio data and timestamp are directly written to the corresponding storage area.

[0071] As can be seen from the above, in this embodiment, in response to the play instruction for the target song, the instrument audio data of each instrument corresponding to the target song is obtained; the instrument audio data is obtained by separating the sound source of the music file of the target song according to the type of instrument; the user's adjustment strategy for the target instrument in the target song is obtained in real time; the instrument audio data corresponding to the target instrument is adjusted according to the adjustment strategy to obtain the adjusted instrument audio data; based on the adjusted instrument audio data and the instrument audio data corresponding to the remaining instruments, the customized accompaniment audio corresponding to the music file is obtained by mixing, and the customized accompaniment audio is played. It can be seen that by dividing the accompaniment into multiple instrument audio data according to the instrument type, the user can adjust and process various instruments in real time during the singing process, and finally, according to the adjusted instrument audio data corresponding to the target instrument and the instrument audio data corresponding to the remaining instruments, the customized accompaniment audio file that meets the user's needs is mixed, so as to realize flexible tuning of the instruments in the accompaniment and provide the user with a personalized singing experience.

[0072] In the real-time accompaniment tuning solution of this application, the system framework used can be specifically referred to Figure 4 As shown, it may specifically include: a backend server and a number of user terminals that establish communication connections with the backend server. Among them, the user terminal includes but is not limited to a tablet computer, a laptop computer, a smart phone, a personal computer, a car audio system, a large-screen TV, and a jukebox, etc., which are not limited here. Taking the storage of instrument audio data in the backend server as an example, in this application, the user terminal is used to respond to the play instruction for the target song, obtain the instrument audio data of each instrument corresponding to the target song from the backend server, and the user terminal performs the steps of the real-time accompaniment tuning method, including obtaining the instrument audio data of each instrument corresponding to the target song; the instrument audio data is obtained by separating the sound source of the music file of the target song according to the type of instrument; obtaining the user's adjustment strategy for the target instrument in the target song in real time; adjusting the instrument audio data corresponding to the target instrument according to the adjustment strategy to obtain the adjusted instrument audio data; based on the adjusted instrument audio data and the instrument audio data corresponding to the remaining instruments, the customized accompaniment audio corresponding to the music file is obtained by mixing, and the customized accompaniment audio is played.

[0073] Below, taking a certain music APP as an example, the technical solution of this application is explained. Assuming that the user has installed this music APP on the user side, when he wants to sing a song and needs customized accompaniment, he enters the singing function of this APP, queries the song information and initiates singing; the user side obtains the instrument audio data corresponding to the song from the local or server, and at the same time, the user side obtains the user's adjustment strategy for the target instrument in the target song in real time, and the target instrument can be one or more; then the instrument audio data corresponding to the target instrument is adjusted according to the adjustment strategy to obtain the adjusted instrument audio data; finally, based on the adjusted instrument audio data corresponding to the target instrument and the instrument audio data corresponding to the remaining instruments, the customized accompaniment audio file corresponding to the music file is obtained through mixing.

[0074] Correspondingly, the embodiment of the present application also discloses a real-time accompaniment tuning device, which includes:

[0075] The instrument audio data acquisition module 11 is used to respond to the play instruction for the target song and acquire the instrument audio data of each instrument corresponding to the target song; the instrument audio data is obtained by separating the sound source of the music file of the target song according to the type of instrument;

[0076] An adjustment strategy acquisition module 12 is used to acquire the user's adjustment strategy for the target instrument in the target song in real time;

[0077] An adjustment processing module 13, configured to perform adjustment processing on the instrument audio data corresponding to the target instrument according to the adjustment strategy to obtain adjusted instrument audio data;

[0078] The mixing module 14 is used to obtain the customized accompaniment audio corresponding to the target song through mixing based on the adjusted instrument audio data and the instrument audio data corresponding to other instruments, and play the customized accompaniment audio.

[0079] As can be seen from the above, in this embodiment, in response to the play instruction for the target song, the instrument audio data of each instrument corresponding to the target song is obtained; the instrument audio data is obtained by separating the sound source of the music file of the target song according to the type of instrument; the user's adjustment strategy for the target instrument in the target song is obtained in real time; the instrument audio data corresponding to the target instrument is adjusted according to the adjustment strategy to obtain the adjusted instrument audio data; based on the adjusted instrument audio data and the instrument audio data corresponding to the remaining instruments, the customized accompaniment audio corresponding to the music file is obtained by mixing, and the customized accompaniment audio is played. It can be seen that by dividing the accompaniment into multiple instrument audio data according to the instrument type, the user can adjust and process various instruments in real time during the singing process, and finally, according to the adjusted instrument audio data corresponding to the target instrument and the instrument audio data corresponding to the remaining instruments, the customized accompaniment audio file that meets the user's needs is mixed, so as to realize flexible tuning of the instruments in the accompaniment and provide the user with a personalized singing experience.

[0080] In some specific embodiments, the real-time accompaniment tuning device may specifically include:

[0081] A sound source separation unit is used to separate the sound sources of the music file of the target song according to the type of musical instrument before obtaining the musical instrument audio data of each musical instrument corresponding to the target song, so as to obtain the initial musical instrument audio data corresponding to each musical instrument;

[0082] An alignment check unit, used for performing alignment check on all the musical instrument audio data corresponding to the same music file;

[0083] The noise reduction processing unit is used to perform noise reduction processing on the musical instrument audio data.

[0084] In some specific embodiments, the alignment check unit can be specifically used to add blank data of corresponding duration for the instrument audio data that fails the alignment check, so as to ensure that all the instrument audio data corresponding to the same music file are aligned in time.

[0085] In some specific embodiments, the real-time accompaniment tuning device may specifically include:

[0086] An encoding unit, for, after acquiring the instrument audio data of each instrument corresponding to the target song, encoding the instrument audio data simultaneously by using multiple threads to obtain digital format audio corresponding to the instrument audio data;

[0087] A storage unit is used for performing storage and adjustment processing based on the digital format audio.

[0088] In some specific embodiments, the adjustment strategy may specifically include any one or more of loudness adjustment, equalization processing, reverberation processing, and timbre conversion synthesis.

[0089] In some specific embodiments, the adjustment strategy acquisition module 12 may specifically include:

[0090] A tuning option display unit, used to display the tuning options of each musical instrument corresponding to the music file through a human-computer interaction interface;

[0091] The adjustment strategy generating unit is used to generate the user's adjustment strategy for the target musical instrument according to the user's selection instruction for the target tuning option.

[0092] In some specific embodiments, the adjustment processing module 13 may include:

[0093] A to-be-played segment determination unit, used to determine the to-be-played segment in the instrument audio data of the target instrument based on the playing progress of the target song;

[0094] An adjustment unit, configured to adjust the to-be-played segment according to the adjustment strategy to obtain an adjusted to-be-played segment;

[0095] Accordingly, the mixing module 14 includes:

[0096] A mixing unit, configured to obtain a customized accompaniment audio segment to be played by mixing the adjusted segment to be played and the segment to be played in the instrument audio data of other instruments;

[0097] The playing unit is used to play the customized accompaniment audio clip when the playing progress of the target song reaches the target time point.

[0098] In some specific embodiments, the real-time accompaniment tuning device may specifically include:

[0099] A strategy saving unit, configured to save the adjustment strategy of the user for the target instrument in the target song as a historical adjustment strategy after obtaining the adjustment strategy of the user for the target instrument in the target song in real time;

[0100] The adjustment processing unit is used to adjust the instrument audio data corresponding to the music file according to the historical adjustment strategy when the real-time accompaniment tuning demand of the same user for the music file is obtained again.

[0101] In some specific embodiments, the mixing module 14 may specifically include:

[0102] A first timestamp determination unit, used to obtain a first timestamp corresponding to the current playing point of the target song;

[0103] A second timestamp determining unit, configured to locate, based on the first timestamp, a second timestamp corresponding to the first timestamp from the musical instrument audio data corresponding to each musical instrument or the adjusted musical instrument audio data;

[0104] A mixing unit is used to obtain the customized accompaniment audio corresponding to the music file through mixing based on the adjusted instrument audio data and the instrument audio data corresponding to the remaining instruments, starting from the second timestamp, and play the customized accompaniment audio.

[0105] Furthermore, an embodiment of the present application also discloses an electronic device, as shown in FIG5 . The content in the figure cannot be regarded as any limitation on the scope of use of the present application.

[0106] FIG5 is a schematic diagram of the structure of an electronic device 20 provided in an embodiment of the present application. The electronic device 20 may specifically include: at least one processor 21, at least one memory 22, a power supply 23, a communication interface 24, an input / output interface 25, and a communication bus 26. The memory 22 is used to store a computer program, which is loaded and executed by the processor 21 to implement the relevant steps in the real-time accompaniment tuning method disclosed in any of the aforementioned embodiments.

[0107] In this embodiment, the power supply 23 is used to provide working voltage for each hardware device on the electronic device 20; the communication interface 24 can create a data transmission channel between the electronic device 20 and the external device, and the communication protocol it follows is any communication protocol that can be applied to the technical solution of the present application, and is not specifically limited here; the input and output interface 25 is used to obtain external input data or output data to the outside world, and its specific interface type can be selected according to specific application needs and is not specifically limited here.

[0108] In addition, the memory 22, as a carrier for storing resources, can be a read-only memory, a random access memory, a disk or an optical disk, etc. The resources stored thereon include an operating system 221, a computer program 222, and data 223 including musical instrument audio data, etc. The storage method can be temporary storage or permanent storage.

[0109] The operating system 221 is used to manage and control the hardware devices and computer programs 222 on the electronic device 20, so as to realize the operation and processing of the massive data 223 in the memory 22 by the processor 21, and it can be Windows Server, Netware, Unix, Linux, etc. In addition to including a computer program that can be used to complete the real-time accompaniment tuning method performed by the electronic device 20 disclosed in any of the aforementioned embodiments, the computer program 222 can further include a computer program that can be used to complete other specific tasks.

[0110] Furthermore, an embodiment of the present application also discloses a computer program product, including a computer program, which, when executed by a processor, implements the steps of the real-time accompaniment tuning method disclosed in any of the aforementioned embodiments.

[0111] Furthermore, an embodiment of the present application also discloses a computer storage medium, in which computer executable instructions are stored. When the computer executable instructions are loaded and executed by a processor, the steps of the real-time accompaniment tuning method disclosed in any of the aforementioned embodiments are implemented.

[0112] In this specification, each embodiment is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the embodiments can be referred to each other. For the device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part.

[0113] The steps of the method or algorithm described in conjunction with the embodiments disclosed herein may be implemented directly using hardware, a software module executed by a processor, or a combination of the two. The software module may be placed in a random access memory (RAM), a memory, a read-only memory (ROM), an electrically programmable ROM, an electrically erasable programmable ROM, a register, a hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art.

[0114] Finally, it should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the presence of other identical elements in the process, method, article or device including the elements.

[0115] The above is a detailed introduction to a real-time accompaniment tuning method, device, storage medium and program product provided by the present invention. Specific examples are used in this article to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea. At the same time, for those skilled in the art, according to the idea of ​​the present invention, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as limiting the present invention.

Claims

1. A real-time accompaniment tuning method, characterized in that: include: In response to a play instruction for a target song, obtaining instrument audio data of each instrument corresponding to the target song; The musical instrument audio data is obtained by separating the sound sources of the music file of the target song according to the types of musical instruments; Acquire the user's adjustment strategy for the target instrument in the target song in real time; Performing adjustment processing on the instrument audio data corresponding to the target instrument according to the adjustment strategy to obtain adjusted instrument audio data; Based on the adjusted instrument audio data and the instrument audio data corresponding to the remaining instruments, the customized accompaniment audio corresponding to the target song is obtained by mixing, and the customized accompaniment audio is played.

2. The real-time accompaniment tuning method according to claim 1, characterized in that: Before obtaining the instrument audio data of each instrument corresponding to the target song, the method further includes: Separating the music files of the target song according to the types of musical instruments to obtain initial musical instrument audio data corresponding to each musical instrument; The initial musical instrument audio data is optimized to obtain the musical instrument audio data of each musical instrument; the optimization includes at least one of alignment and noise reduction; the alignment includes alignment verification of all the initial musical instrument audio data corresponding to the same music file; the noise reduction includes noise reduction of the initial musical instrument audio data.

3. The real-time accompaniment tuning method according to claim 2, characterized in that: The optimization process includes alignment process, and the optimization process is performed on the initial instrument audio data to obtain the instrument audio data of each instrument, including: Performing alignment check on all the initial instrument audio data corresponding to the same music file; For the initial instrument audio data that fails the alignment check, blank data of corresponding duration is added to obtain the instrument audio data of each instrument.

4. The real-time accompaniment tuning method according to claim 1, characterized in that: After obtaining the instrument audio data of each instrument corresponding to the target song, the method further includes: Encoding the musical instrument audio data simultaneously by using multiple threads to obtain digital format audio corresponding to the musical instrument audio data; The storage and the adjustment processing are performed based on the digital format audio.

5. The real-time accompaniment tuning method according to claim 1, characterized in that: The adjustment strategy includes any one or more of loudness adjustment, equalization processing, reverberation processing, and timbre conversion synthesis.

6. The real-time accompaniment tuning method according to claim 1, characterized in that: The real-time acquisition of the user's adjustment strategy for the target instrument in the target song includes: Displaying tuning options of each musical instrument corresponding to the music file through a human-computer interaction interface; According to the user's selection instruction for the target tuning option, a user's adjustment strategy for the target instrument is generated.

7. The real-time accompaniment tuning method according to claim 1, characterized in that: After the real-time acquisition of the user's adjustment strategy for the target instrument in the target song, the method further includes: Saving the adjustment strategy as a historical adjustment strategy; When the real-time accompaniment tuning demand of the same user for the music file is obtained again, the instrument audio data corresponding to the music file is adjusted according to the historical adjustment strategy.

8. The method according to any one of claims 1 to 7, characterized in that: The step of adjusting the instrument audio data corresponding to the target instrument according to the adjustment strategy to obtain adjusted instrument audio data includes: Based on the playing progress of the target song, determine the to-be-played segment in the instrument audio data of the target instrument; Adjust the to-be-played segment according to the adjustment strategy to obtain an adjusted to-be-played segment; The step of obtaining the customized accompaniment audio corresponding to the target song by mixing the adjusted instrument audio data and the instrument audio data corresponding to other instruments includes: Based on the adjusted to-be-played segment and the to-be-played segments in the instrument audio data of other instruments, obtaining the to-be-played customized accompaniment audio segment by mixing; The playing of the customized accompaniment audio includes: When the playing progress of the target song reaches the target time point, the customized accompaniment audio clip is played.

9. The real-time accompaniment tuning method according to any one of claims 1 to 7, characterized in that: The step of obtaining the customized accompaniment audio corresponding to the target song by mixing based on the adjusted instrument audio data and the instrument audio data corresponding to other instruments includes: Obtain a first timestamp corresponding to the current playing point of the target song; According to the first timestamp, locating a second timestamp corresponding to the first timestamp from the musical instrument audio data corresponding to each musical instrument or the adjusted musical instrument audio data; Based on the adjusted instrument audio data and the instrument audio data corresponding to the remaining instruments, starting from the second timestamp, the customized accompaniment audio corresponding to the target song is obtained through mixing.

10. An electronic device, characterized in that: include: Memory, used to store computer programs; A processor, configured to execute the computer program to implement the real-time accompaniment tuning method as claimed in any one of claims 1 to 9.

11. A computer-readable storage medium, characterized in that: Used to store computer programs; wherein when the computer programs are executed by a processor, the real-time accompaniment tuning method as described in any one of claims 1 to 9 is implemented.

12. A computer program product, characterized in that The method comprises a computer program / instruction; wherein when the computer program / instruction is executed by a processor, the real-time accompaniment tuning method as claimed in any one of claims 1 to 9 is implemented.

Images