Mother band audio generation method and terminal

Abstract

The invention provides a mother band audio generation method and a terminal, and relates to the technical field of audios, the method is applied to the terminal, and the method comprises the following steps: obtaining a to-be-processed audio of a target song; performing sub-band segmentation on the audio to be processed to obtain a plurality of sub-band audios; inputting the plurality of sub-band audios into a mother band audio generation model to obtain a plurality of audios to be synthesized; and performing sub-band synthesis on the plurality of to-be-synthesized audios to obtain a mother band audio of the target song. By adopting the method and the device, the terminal can complete the acquisition of the mother band audio under the condition that the terminal does not need to interact with the server, and the bandwidth occupation generated by the interaction with the server is saved.

Description

Technical Field

[0001] The present application relates to the field of audio technology, and in particular to a method and a terminal for generating master audio. Background Art

[0002] The master audio can restore the sound quality of the song most accurately and show the highest sound quality of the music. Common lossless audio is obtained by compressing the master audio.

[0003] Currently, the method for users to obtain master audio through a terminal is usually as follows: the user initiates a master audio acquisition request for a specified song to a server at the terminal, and the server sends the master audio of the specified song to the user's terminal.

[0004] However, the data volume of the master audio of a song is usually large, generally reaching hundreds of megabytes (Mbyte). During the transmission from the server to the terminal, a large amount of bandwidth will be occupied, increasing the bandwidth cost. Summary of the invention

[0005] The embodiment of the present application provides a method and terminal for generating master audio, which can solve the problem of bandwidth occupation caused by the terminal obtaining master audio through a server. The technical solution is as follows:

[0006] In a first aspect, a method for generating a master audio is provided, the method being applied to a terminal, the method comprising:

[0007] Get the audio to be processed of the target song;

[0008] Dividing the audio to be processed into sub-bands to obtain multiple sub-band audios;

[0009] Inputting the plurality of sub-band audios into a master audio generation model to obtain a plurality of audios to be synthesized;

[0010] Sub-band synthesis is performed on the multiple audios to be synthesized to obtain the master audio of the target song.

[0011] In a possible implementation, dividing the audio to be processed into sub-bands to obtain a plurality of sub-band audios includes:

[0012] Convolving the audio to be processed with a plurality of analysis filters respectively to obtain a plurality of analysis filtered signals;

[0013] The multiple filtered signals are downsampled to obtain multiple sub-band audios.

[0014] In a possible implementation, performing sub-band synthesis on the multiple audios to be synthesized to obtain the master audio of the target song includes:

[0015] Convolving the multiple audios to be synthesized with multiple synthesis filters respectively to obtain multiple synthesis filtered signals;

[0016] Upsampling the multiple synthetic filtered signals to obtain multiple upsampled signals;

[0017] The multiple up-sampled signals are added together to obtain the master audio of the target song.

[0018] In a possible implementation, the sampling rates of the audio to be processed and the master audio are the same.

[0019] In a possible implementation, obtaining the audio to be processed of the target song includes:

[0020] Acquire the audio to be improved in sound quality of the target song, wherein the sampling rate of the audio to be improved in sound quality is lower than the sampling rate of the master audio;

[0021] The audio to be improved in sound quality is up-sampled to obtain audio to be processed.

[0022] In a possible implementation, the master audio generation model is an autoencoder AutoEncoder model.

[0023] In a possible implementation, before obtaining the audio to be processed of the target song, the method further includes:

[0024] In the playback interface of the target song, a user's selection operation on the master audio playback option of the target song is detected.

[0025] In a possible implementation, the method further includes:

[0026] Get the original master audio of the sample song;

[0027] Downsampling the original master audio to audio at a first sampling rate, wherein the first sampling rate is lower than the sampling rate of the original master audio;

[0028] Upsampling the audio at the target sampling rate to audio at a second sampling rate, wherein the second sampling rate is the sampling rate of the original master audio;

[0029] Dividing the audio at the second sampling rate into sub-bands to obtain a plurality of sample sub-band audios;

[0030] Inputting the plurality of sample sub-band audios into a master audio generation model to be trained to obtain a plurality of sample audios to be synthesized;

[0031] Performing sub-band synthesis on the plurality of sample audios to be synthesized to obtain a predicted master audio;

[0032] A loss value is calculated based on the predicted master audio and the original master audio, and the master audio generation model to be trained is trained based on the loss value.

[0033] In a second aspect, a device for generating a master audio is provided, the device being applied to a terminal, the device comprising:

[0034] An acquisition module is used to acquire the audio to be processed of the target song;

[0035] A segmentation module, used for segmenting the audio to be processed into sub-bands to obtain multiple sub-band audios;

[0036] An inference module, used for inputting the plurality of sub-band audios into a master audio generation model to obtain a plurality of audios to be synthesized;

[0037] The synthesis module is used to perform sub-band synthesis on the multiple audios to be synthesized to obtain the master audio of the target song.

[0038] In a possible implementation, the segmentation module is used to:

[0039] Convolving the audio to be processed with a plurality of analysis filters respectively to obtain a plurality of analysis filtered signals;

[0040] The multiple filtered signals are downsampled to obtain multiple sub-band audios.

[0041] In a possible implementation, the synthesis module is used to:

[0042] Convolving the multiple audios to be synthesized with multiple synthesis filters respectively to obtain multiple synthesis filtered signals;

[0043] Upsampling the multiple synthetic filtered signals to obtain multiple upsampled signals;

[0044] The multiple up-sampled signals are added together to obtain the master audio of the target song.

[0045] In a possible implementation, the sampling rates of the audio to be processed and the master audio are the same.

[0046] In a possible implementation, the acquisition module is used to:

[0047] Acquire the audio to be improved in sound quality of the target song, wherein the sampling rate of the audio to be improved in sound quality is lower than the sampling rate of the master audio;

[0048] The audio to be improved in sound quality is up-sampled to obtain audio to be processed.

[0049] In a possible implementation, the master audio generation model is an autoencoder AutoEncoder model.

[0050] In a possible implementation, the device further includes a checking module, which is used to:

[0051] In the playback interface of the target song, a user's selection operation on the master audio playback option of the target song is detected.

[0052] In a possible implementation, the device further includes a training module, which is used to:

[0053] Get the original master audio of the sample song;

[0054] Downsampling the original master audio to audio at a first sampling rate, wherein the first sampling rate is lower than the sampling rate of the original master audio;

[0055] Upsampling the audio at the target sampling rate to audio at a second sampling rate, wherein the second sampling rate is the sampling rate of the original master audio;

[0056] Dividing the audio at the second sampling rate into sub-bands to obtain a plurality of sample sub-band audios;

[0057] Inputting the plurality of sample sub-band audios into a master audio generation model to be trained to obtain a plurality of sample audios to be synthesized;

[0058] Performing sub-band synthesis on the plurality of sample audios to be synthesized to obtain a predicted master audio;

[0059] A loss value is calculated based on the predicted master audio and the original master audio, and the master audio generation model to be trained is trained based on the loss value.

[0060] In a third aspect, a terminal is provided, comprising a processor and a memory, wherein the memory stores at least one instruction, and the instruction is loaded and executed by the processor to implement the operations performed by the method for generating master audio as described in the first aspect and any possible implementation of the first aspect.

[0061] In a fourth aspect, a computer-readable storage medium is provided, wherein at least one instruction is stored in the storage medium, and the instruction is loaded and executed by a processor to implement the operations performed by the method for generating master audio as described in the first aspect and any possible implementation of the first aspect.

[0062] In a fifth aspect, a computer program product is provided, wherein at least one instruction is stored in the computer program product, and the instruction is loaded and executed by a processor to implement the operations performed by the method for generating master audio as described in the first aspect and any possible implementation of the first aspect.

[0063] The beneficial effects of the technical solution provided by the embodiment of the present application are:

[0064] In the technical solution provided by the embodiment of the present application, a master audio generation model is deployed on the terminal side. When the user wants to listen to the master audio of the target song, the user can directly divide the processed audio of the target song into sub-bands through the terminal, and process the divided sub-band audios through the master audio generation model, and then synthesize the multiple processed audios to be synthesized to obtain the master audio of the target song. The entire process is completed by the terminal without the need to interact with the server, saving the bandwidth cost of the terminal. BRIEF DESCRIPTION OF THE DRAWINGS

[0065] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0066] Figure 1 It is a flow chart of a method for generating master audio provided by an embodiment of the present application;

[0067] Figure 2 is a schematic diagram of a playback interface provided in an embodiment of the present application;

[0068] Figure 3 is a schematic diagram of sub-band segmentation provided in an embodiment of the present application;

[0069] Figure 4 is a schematic diagram of a sub-band synthesis provided in an embodiment of the present application;

[0070] Figure 5 It is a schematic diagram of the structure of a device for generating master audio provided in an embodiment of the present application;

[0071] Figure 6 It is a structural diagram of a terminal provided in an embodiment of the present application. DETAILED DESCRIPTION

[0072] In order to make the objectives, technical solutions and advantages of the present application clearer, the implementation methods of the present application will be further described in detail below with reference to the accompanying drawings.

[0073] The embodiment of the present application provides a method for generating master audio, which is implemented by a terminal, and the terminal can be a mobile phone, a computer, a tablet computer, a laptop computer, etc. A music application can be installed in the terminal. When the user wants to listen to music, he can search for the song in the music application and click to enter the playback interface of the song. In the playback interface of the song, a function option for selecting the sound quality of the song can be displayed. The user can select a sound quality according to his or her preferences, wherein the selectable sound quality may include master sound quality, lossless sound quality, high-quality sound quality, standard quality sound quality, etc. In the case where the user selects the master sound quality, the terminal can generate the master audio of the song by executing the method for generating master audio provided by the embodiment of the present application, and then play the master audio of the song. The method for generating master audio provided by the embodiment of the present application is described below.

[0074] Figure 1 This is a flow chart of a method for generating a master audio provided by an embodiment of the present application. The method is implemented by a terminal. Figure 1 , the method may include the following processing steps:

[0075] Step 201: Obtain the audio to be processed of the target song.

[0076] In implementation, a music application may be installed on the user's terminal. When the user wants to listen to music, the user may search for the target song in the music application and click to enter the target song's playback interface. In the target song's playback interface, a sound quality option for selecting the song's sound quality may be displayed. After the user clicks on the sound quality option, a sound quality list may be displayed in the playback interface. In the sound quality list, audio playback options of sound quality supported by the target song may be displayed, such as master audio playback, lossless audio playback message, high-quality audio playback message, standard quality audio playback option, etc.

[0077] See also Figure 2 , shows a song playback interface, which displays the song name, lyrics, playback progress bar, start / pause option, comment function option, download function option, previous song option, next song option, sound quality option, etc. Among them, the sound quality option is used to select the sound quality of the currently playing song, such as Figure 2The "SQ" in the sound quality option is "SQ", which means that the sound quality of the currently playing song is SQ lossless sound quality. If the currently playing song is HQ high-quality sound quality, the sound quality option will be displayed as "HQ" in the playback interface. If the user wants to listen to the audio of other sound qualities of the song, he can click the sound quality option. After detecting the user's selection operation on the sound quality option, the terminal displays a sound quality list. In the sound quality list, at least one sound quality playback option supported by the song is displayed. Exemplarily, the sound quality list may include a master audio playback option, an SQ lossless audio playback option, an HQ high-quality audio playback option, a standard quality audio playback option, and the like. The user can click the master audio playback option to trigger the terminal to generate and play the master audio of the song.

[0078] After detecting that the user clicks the master audio playback option, the terminal obtains the cached audio of the target song to be improved locally. The sampling rate of the audio to be improved is lower than the sampling rate of the master audio, and the audio to be improved refers to the audio of the default sound quality, for example, SQ lossless audio, HQ high-quality audio, and standard quality audio.

[0079] After obtaining the audio of the target song to be improved in sound quality, the audio of the target song to be improved in sound quality is upsampled to obtain the audio to be processed at the target sampling rate, wherein the target sampling rate is the sampling rate of the master audio.

[0080] Exemplarily, the audio to be improved in sound quality is SQ lossless audio, the sampling rate of the audio to be improved in sound quality is 44.1kHz, and the sampling rate of the master audio is 192kHz. After obtaining the audio to be improved in sound quality of the target song, the audio to be improved in sound quality with a sampling rate of 44.1kHz is upsampled to obtain the audio to be processed with a sampling rate of 192kHz.

[0081] In addition, the sound quality of the above-mentioned audio to be processed may be better than the sound quality of the audio to be improved, or it may be worse than the sound quality of the audio to be improved. This is related to many factors such as the sound quality of the audio to be improved, the sampling algorithm, etc. The relationship between the sound qualities of the two is not limited in the embodiments of the present application.

[0082] Step 202: Divide the audio to be processed into sub-bands to obtain a plurality of sub-band audios.

[0083] The number of sub-band audios cut can be configured by relevant personnel according to actual needs. Here, N sub-band audios are obtained by cutting for illustration, where N is an integer greater than 1. For example, N is 2.

[0084] In implementation, after obtaining the audio to be processed, the terminal convolves the audio to be processed with N analysis filters respectively, and obtains an analysis filtered signal after convolution with each analysis filter, so that N analysis filtered signals can be obtained. Among them, the passbands of multiple analysis filters are different. The analysis filter allows signals in a specific frequency range to pass through while suppressing signals of other frequencies. Among them, the passband of the analysis filter refers to the frequency range of the signal allowed to pass by the analysis filter. Then, the multiple filtered signals are downsampled to obtain multiple sub-band audios.

[0085] The above processing flow can be expressed by the following formula (1):

[0086]

[0087] Among them, y′ j Indicates the j-th sub-band audio obtained by segmentation, DS N represents N times downsampling, y represents the audio to be processed, which can be a two-channel audio signal. represents the j-th F-order analysis filter, and * represents the convolution operation.

[0088] The calculation process of the above formula (1) can be found in Figure 3 shown.

[0089] In a possible implementation, the processing of step 202 may also be as follows:

[0090] After obtaining the audio to be processed, the terminal downsamples the audio to be processed, and then convolves the downsampled signals with N analysis filters respectively. After convolving with each analysis filter, a sub-band audio is obtained. In this way, N sub-band audios can be obtained.

[0091] Step 203: Input the multiple sub-band audios into the master audio generation model to obtain multiple audios to be synthesized.

[0092] Among them, the master audio generation model is a neural network model, such as an AutoEncoder model.

[0093] In implementation, after obtaining N sub-band audios, the N sub-band audios are stacked and input into a pre-trained master audio generation model, and the master generation model outputs N audios to be synthesized, which can also be called sub-band estimated audios.

[0094] Step 204: perform sub-band synthesis on the multiple audios to be synthesized to obtain the master audio of the target song.

[0095] In the implementation, after obtaining the N audios to be synthesized output by the master audio generation model, the N audios to be synthesized are convolved with N synthesis filters respectively to obtain N synthesis filtered signals. Then, the N synthesis filtered signals are upsampled to obtain N upsampled signals, wherein the synthesis filter can be a low-pass filter for smoothing the reconstructed signal and reducing aliasing and distortion during the reconstruction process. Then, the N upsampled signals are added to obtain the master audio of the target song.

[0096] The above processing flow can be expressed by the following formula (2):

[0097]

[0098] in, Indicates master audio, US N Indicates N-fold upsampling, represents the jth audio to be synthesized, represents the j-th F-order synthesis filter, * represents a convolution operation, and Σ represents a summation operation. Here, the N-fold upsampling may be an N-fold zero interpolation upsampling.

[0099] The calculation process of the above formula (2) can be found in Figure 4 shown.

[0100] Through experimental verification, the method for generating master audio provided by the embodiment of the present application is adopted. The audio to be processed is first divided into sub-bands and then input into the master audio generation model. Compared with directly inputting the audio to be processed into the master audio generation model, the amount of data processed by each layer of the network in the master audio generation model can be effectively reduced, thereby reducing the occupation of computing resources and storage resources, and improving processing efficiency. A set of experimental results are shown below to illustrate the above effects. Referring to Table 1, taking the master audio generation model as the AutoEncoder structure as an example, the master audio generation model includes three encoders, encoder 1, encoder2, and encoder3, and three decoders, decoder 1, decoder2, and decoder3. The data volume of the audio to be processed is 1*2*8640000, which is the input in Table 1.

[0101] When the method for generating master audio provided by the embodiment of the present application is used for experiments, the audio to be processed is divided into two sub-band audios. The data volume of the two sub-band audios can be expressed as 1*4*4320084. The input of encoder1 is two sub-band audios. After encoding processing by encoder1, the output data volume is 1*32*1080020. The output of encoder1 is used as the input of encoder2. After encoder2 encodes the input data, the output data volume is 1*64*270004. The output of encoder2 is used as the input of encoder3, and encoder3 encodes the input data. The output of encoder3 is used as the input of decoder3. After decoder3 decodes the input data, the output data volume is 1*64*270004. The output of decoder3 is used as the input of decoder2. After decoder2 decodes the input data, the output data volume is 1*32*1080020. The output of decoder2 is used as the input of decoder1. After decoder1 decodes the input data, the output data volume is 1*4*4320084. The data output by decoder1 is the audio to be synthesized. The data output by decoder1 is synthesized to obtain the master audio, which is the output in Table 1.

[0102] When the audio to be processed is directly input into the master audio generation model for the experiment, the input of encoder1 is the audio to be processed, and the data volume is 1*2*8640020. After encoding processing by encoder1, the output data volume is 1*32*2160004. The output of encoder1 is used as the input of encoder2. After encoder2 encodes the input data, the output data volume is 1*64*540000. The output of encoder2 is used as the input of encoder3, and encoder3 encodes the input data. The output of encoder3 is used as the input of decoder3. After decoder3 decodes the input data, the output data volume is 1*64*540000. The output of decoder3 is used as the input of decoder2. After decoder2 decodes the input data, the output data volume is 1*32*2160004. The output of decoder2 is used as the input of decoder1. After decoder1 decodes the input data, the output data volume is 1*2*8640020.

[0103] Table 1

[0104] Use this application No sub-band splitting Input 1*2*8640000 1*2*8640000 encoder1 input 1*4*4320084 1*2*8640020 encoder2 input 1*32*1080020 1*32*2160004 encoder3 input 1*64*270004 1*64*540000 decoder3 output 1*64*270004 1*64*540000 decoder2 output 1*32*1080020 1*2*2160004 decoder1 output 1*4*4320084 1*2*8640020 output 1*2*8640000 1*2*8640000

[0105] Through experimental comparison, it can be seen that when the method for master audio generation provided in the embodiment of the present application is adopted, the amount of data processed by encoder2, encoder3, decoder2, and decoder1 is much less than when the sub-band splitting technology is not introduced, thereby reducing the occupation of computing resources and storage resources, and the memory requirement is relatively small, which can better meet the needs of terminal deployment of the master audio generation model.

[0106] The following is an explanation of the training of the master audio generation model:

[0107] First, the original master audio of the sample song is obtained from the music library, and the original master audio is used as the output sample. The sample song can be any song in the music library. Then, the original master audio of the sample song is downsampled to an audio of a first sampling rate, wherein the first sampling rate is lower than the sampling rate of the original master audio, for example, the first sampling rate is 44.1kHz. Then, the audio of the first sampling rate is upsampled to an audio of a second sampling rate, wherein the second sampling rate is the sampling rate of the original master audio. Then, the audio of the second sampling rate is used as an input sample, and the input sample is sub-band segmented to obtain a plurality of sample sub-band audios. Then, the plurality of sample sub-band audios are input into the master audio generation model to be trained to obtain a plurality of sample audios to be synthesized. Then, the plurality of sample audios to be synthesized are sub-band synthesized to obtain a predicted master audio. Then, the loss value is calculated based on the predicted master audio and the original master audio, and then, the parameters in the master audio generation model to be trained are adjusted based on the loss value to complete the first training. The master audio generation model is trained multiple times according to the above method until the training end condition is met and the training is stopped.

[0108] All the above optional technical solutions can be arbitrarily combined to form optional embodiments of the present disclosure, and will not be described in detail here.

[0109] In the technical solution provided by the embodiment of the present application, a master audio generation model is deployed on the terminal side. When the user wants to listen to the master audio of the target song, the user can directly divide the processed audio of the target song into sub-bands through the terminal, and process the divided sub-band audios through the master audio generation model, and then synthesize the multiple processed audios to be synthesized to obtain the master audio of the target song. The entire process is completed by the terminal without the need to interact with the server, saving the bandwidth cost of the terminal.

[0110] Based on the same technical concept, the embodiment of the present application also provides a device for generating a master audio, which can be applied to a terminal, see Figure 5 , the device may include an acquisition module 510, a segmentation module 520, a reasoning module 530, and a synthesis module 540, wherein:

[0111] An acquisition module 510 is used to acquire the audio to be processed of the target song;

[0112] A segmentation module 520, configured to segment the audio to be processed into sub-bands to obtain a plurality of sub-band audios;

[0113] An inference module 530 is used to input the multiple sub-band audios into a master audio generation model to obtain multiple audios to be synthesized;

[0114] The synthesis module 540 is used to perform sub-band synthesis on the multiple audios to be synthesized to obtain the master audio of the target song.

[0115] In a possible implementation, the segmentation module 520 is used to:

[0116] Convolving the audio to be processed with a plurality of analysis filters respectively to obtain a plurality of analysis filtered signals;

[0117] The multiple filtered signals are downsampled to obtain multiple sub-band audios.

[0118] In a possible implementation, the synthesis module 540 is used to:

[0119] Convolving the multiple audios to be synthesized with multiple synthesis filters respectively to obtain multiple synthesis filtered signals;

[0120] Upsampling the multiple synthetic filtered signals to obtain multiple upsampled signals;

[0121] The multiple up-sampled signals are added together to obtain the master audio of the target song.

[0122] In a possible implementation, the sampling rates of the audio to be processed and the master audio are the same.

[0123] In a possible implementation, the acquisition module 510 is used to:

[0124] Acquire the audio to be improved in sound quality of the target song, wherein the sampling rate of the audio to be improved in sound quality is lower than the sampling rate of the master audio;

[0125] The audio to be improved in sound quality is up-sampled to obtain audio to be processed.

[0126] In a possible implementation, the master audio generation model is an autoencoder AutoEncoder model.

[0127] In a possible implementation manner, the device further includes a detection module, which is used to:

[0128] In the playback interface of the target song, a user's selection operation on the master audio playback option of the target song is detected.

[0129] In a possible implementation, the apparatus further includes a training module, configured to:

[0130] Get the original master audio of the sample song;

[0131] Downsampling the original master audio to audio at a first sampling rate, wherein the first sampling rate is lower than the sampling rate of the original master audio;

[0132] Upsampling the audio at the target sampling rate to audio at a second sampling rate, wherein the second sampling rate is the sampling rate of the original master audio;

[0133] Dividing the audio at the second sampling rate into sub-bands to obtain a plurality of sample sub-band audios;

[0134] Inputting the plurality of sample sub-band audios into a master audio generation model to be trained to obtain a plurality of sample audios to be synthesized;

[0135] Performing sub-band synthesis on the plurality of sample audios to be synthesized to obtain a predicted master audio;

[0136] A loss value is calculated based on the predicted master audio and the original master audio, and the master audio generation model to be trained is trained based on the loss value.

[0137] In the technical solution provided by the embodiment of the present application, a master audio generation model is deployed on the terminal side. When the user wants to listen to the master audio of the target song, the user can directly divide the processed audio of the target song into sub-bands through the terminal, and process the divided sub-band audios through the master audio generation model, and then synthesize the multiple processed audios to be synthesized to obtain the master audio of the target song. The entire process is completed by the terminal without the need to interact with the server, saving the bandwidth cost of the terminal.

[0138] It should be noted that: the device for generating master audio provided in the above embodiment only uses the division of the above functional modules as an example when generating master audio. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the terminal is divided into different functional modules to complete all or part of the functions described above. In addition, the device for generating master audio provided in the above embodiment and the method embodiment for generating master audio belong to the same concept. The specific implementation process is detailed in the method embodiment and will not be repeated here.

[0139] Figure 6The structure block diagram of a terminal 600 provided by an exemplary embodiment of the present application is shown. The terminal 600 may be a portable mobile terminal, such as a smart phone, a tablet computer, an MP3 player (Moving Picture Experts Group Audio Layer III), an MP4 player (Moving Picture Experts Group Audio Layer IV), a laptop computer or a desktop computer. The terminal 600 may also be called a user device, a portable terminal, a laptop terminal, a desktop terminal or other names.

[0140] Typically, the terminal 600 includes a processor 601 and a memory 602 .

[0141] The processor 601 may include one or more processing cores, such as a 4-core processor, an 8-core processor, etc. The processor 601 may be implemented in at least one hardware form of DSP (Digital Signal Processing), FPGA (Field-Programmable Gate Array), and PLA (Programmable Logic Array). The processor 601 may also include a main processor and a coprocessor. The main processor is a processor for processing data in the awake state, also known as a CPU (Central Processing Unit); the coprocessor is a low-power processor for processing data in the standby state. In some embodiments, the processor 601 may be integrated with a GPU (Graphics Processing Unit), which is responsible for rendering and drawing the content to be displayed on the display screen. In some embodiments, the processor 601 may also include an AI (Artificial Intelligence) processor, which is used to process computing operations related to machine learning.

[0142] The memory 602 may include one or more computer-readable storage media, which may be non-transitory. The memory 602 may also include a high-speed random access memory, and a non-volatile memory, such as one or more disk storage devices, flash memory storage devices. In some embodiments, the non-transitory computer-readable storage medium in the memory 602 is used to store at least one instruction, which is used to be executed by the processor 601 to implement the method for master audio generation provided in the method embodiment of the present application.

[0143] In some embodiments, the terminal 600 may further optionally include: a peripheral device interface 603 and at least one peripheral device. The processor 601, the memory 602 and the peripheral device interface 603 may be connected via a bus or a signal line. Each peripheral device may be connected to the peripheral device interface 603 via a bus, a signal line or a circuit board. Specifically, the peripheral device includes: at least one of a radio frequency circuit 604, a display screen 605, a camera assembly 606, an audio circuit 607, a positioning assembly 608 and a power supply 609.

[0144] The peripheral device interface 603 may be used to connect at least one peripheral device related to I / O (Input / Output) to the processor 601 and the memory 602. In some embodiments, the processor 601, the memory 602, and the peripheral device interface 603 are integrated on the same chip or circuit board; in some other embodiments, any one or two of the processor 601, the memory 602, and the peripheral device interface 603 may be implemented on a separate chip or circuit board, which is not limited in this embodiment.

[0145] The radio frequency circuit 604 is used to receive and transmit RF (Radio Frequency) signals, also known as electromagnetic signals. The radio frequency circuit 604 communicates with the communication network and other communication devices through electromagnetic signals. The radio frequency circuit 604 converts the electrical signal into an electromagnetic signal for transmission, or converts the received electromagnetic signal into an electrical signal. Optionally, the radio frequency circuit 604 includes: an antenna system, an RF transceiver, one or more amplifiers, a tuner, an oscillator, a digital signal processor, a codec chipset, a user identity module card, etc. The radio frequency circuit 604 can communicate with other terminals through at least one wireless communication protocol. The wireless communication protocol includes but is not limited to: the World Wide Web, a metropolitan area network, an intranet, various generations of mobile communication networks (2G, 3G, 4G and 5G), a wireless local area network and / or a WiFi (Wireless Fidelity) network. In some embodiments, the radio frequency circuit 604 may also include circuits related to NFC (Near Field Communication), which is not limited in this application.

[0146] The display screen 605 is used to display the UI (User Interface). The UI may include graphics, text, icons, videos and any combination thereof. When the display screen 605 is a touch display screen, the display screen 605 also has the ability to collect touch signals on the surface or above the surface of the display screen 605. The touch signal can be input to the processor 601 as a control signal for processing. At this time, the display screen 605 can also be used to provide virtual buttons and / or virtual keyboards, also known as soft buttons and / or soft keyboards. In some embodiments, the display screen 605 can be one, set on the front panel of the terminal 600; in other embodiments, the display screen 605 can be at least two, respectively set on different surfaces of the terminal 600 or in a folding design; in other embodiments, the display screen 605 can be a flexible display screen, set on the curved surface or folding surface of the terminal 600. Even, the display screen 605 can also be set to a non-rectangular irregular shape, that is, a special-shaped screen. The display screen 605 can be made of materials such as LCD (Liquid Crystal Display), OLED (Organic Light-Emitting Diode, organic light-emitting diode).

[0147] The camera assembly 606 is used to capture images or videos. Optionally, the camera assembly 606 includes a front camera and a rear camera. Typically, the front camera is arranged on the front panel of the terminal, and the rear camera is arranged on the back of the terminal. In some embodiments, there are at least two rear cameras, which are any one of a main camera, a depth of field camera, a wide-angle camera, and a telephoto camera, so as to realize the fusion of the main camera and the depth of field camera to realize the background blur function, the fusion of the main camera and the wide-angle camera to realize the panoramic shooting and VR (Virtual Reality) shooting function or other fusion shooting functions. In some embodiments, the camera assembly 606 may also include a flash. The flash can be a monochrome temperature flash or a dual-color temperature flash. A dual-color temperature flash refers to a combination of a warm light flash and a cold light flash, which can be used for light compensation at different color temperatures.

[0148] The audio circuit 607 may include a microphone and a speaker. The microphone is used to collect sound waves from the user and the environment, and convert the sound waves into electrical signals and input them into the processor 601 for processing, or input them into the radio frequency circuit 604 to achieve voice communication. For the purpose of stereo acquisition or noise reduction, there may be multiple microphones, which are respectively arranged at different parts of the terminal 600. The microphone may also be an array microphone or an omnidirectional acquisition microphone. The speaker is used to convert the electrical signal from the processor 601 or the radio frequency circuit 604 into sound waves. The speaker may be a traditional film speaker or a piezoelectric ceramic speaker. When the speaker is a piezoelectric ceramic speaker, it can not only convert the electrical signal into sound waves audible to humans, but also convert the electrical signal into sound waves inaudible to humans for purposes such as ranging. In some embodiments, the audio circuit 607 may also include a headphone jack.

[0149] The positioning component 608 is used to locate the current geographical location of the terminal 600 to implement navigation or LBS (Location Based Service). The positioning component 608 can be a positioning component based on the US GPS (Global Positioning System), China's Beidou system or Russia's Galileo system.

[0150] The power supply 609 is used to power various components in the terminal 600. The power supply 609 can be an alternating current, a direct current, a disposable battery, or a rechargeable battery. When the power supply 609 includes a rechargeable battery, the rechargeable battery can be a wired rechargeable battery or a wireless rechargeable battery. A wired rechargeable battery is a battery charged through a wired line, and a wireless rechargeable battery is a battery charged through a wireless coil. The rechargeable battery can also be used to support fast charging technology.

[0151] In some embodiments, the terminal 600 further includes one or more sensors 610 , including but not limited to: an acceleration sensor 611 , a gyroscope sensor 612 , a pressure sensor 613 , a fingerprint sensor 614 , an optical sensor 615 , and a proximity sensor 616 .

[0152] The acceleration sensor 611 can detect the magnitude of acceleration on the three coordinate axes of the coordinate system established by the terminal 600. For example, the acceleration sensor 611 can be used to detect the components of gravity acceleration on the three coordinate axes. The processor 601 can control the display screen 605 to display the user interface in a horizontal view or a vertical view according to the gravity acceleration signal collected by the acceleration sensor 611. The acceleration sensor 611 can also be used for collecting game or user motion data.

[0153] The gyro sensor 612 can detect the body direction and rotation angle of the terminal 600, and the gyro sensor 612 can cooperate with the acceleration sensor 611 to collect the user's 3D actions on the terminal 600. The processor 601 can implement the following functions based on the data collected by the gyro sensor 612: motion sensing (such as changing the UI according to the user's tilt operation), image stabilization during shooting, game control, and inertial navigation.

[0154] The pressure sensor 613 can be set on the side frame of the terminal 600 and / or the lower layer of the display screen 605. When the pressure sensor 613 is set on the side frame of the terminal 600, it can detect the user's holding signal of the terminal 600, and the processor 601 performs left and right hand recognition or shortcut operation according to the holding signal collected by the pressure sensor 613. When the pressure sensor 613 is set on the lower layer of the display screen 605, the processor 601 controls the operability controls on the UI interface according to the user's pressure operation on the display screen 605. The operability controls include at least one of a button control, a scroll bar control, an icon control, and a menu control.

[0155] The fingerprint sensor 614 is used to collect the user's fingerprint, and the processor 601 identifies the user's identity based on the fingerprint collected by the fingerprint sensor 614, or the fingerprint sensor 614 identifies the user's identity based on the collected fingerprint. When the user's identity is identified as a trusted identity, the processor 601 authorizes the user to perform relevant sensitive operations, including unlocking the screen, viewing encrypted information, downloading software, paying, and changing settings. The fingerprint sensor 614 can be set on the front, back, or side of the terminal 600. When a physical button or a manufacturer logo is set on the terminal 600, the fingerprint sensor 614 can be integrated with the physical button or the manufacturer logo.

[0156] The optical sensor 615 is used to collect the ambient light intensity. In one embodiment, the processor 601 can control the display brightness of the display screen 605 according to the ambient light intensity collected by the optical sensor 615. Specifically, when the ambient light intensity is high, the display brightness of the display screen 605 is increased; when the ambient light intensity is low, the display brightness of the display screen 605 is reduced. In another embodiment, the processor 601 can also dynamically adjust the shooting parameters of the camera assembly 606 according to the ambient light intensity collected by the optical sensor 615.

[0157] The proximity sensor 616, also called a distance sensor, is usually arranged on the front panel of the terminal 600. The proximity sensor 616 is used to collect the distance between the user and the front of the terminal 600. In one embodiment, when the proximity sensor 616 detects that the distance between the user and the front of the terminal 600 is gradually decreasing, the processor 601 controls the display screen 605 to switch from the screen-on state to the screen-off state; when the proximity sensor 616 detects that the distance between the user and the front of the terminal 600 is gradually increasing, the processor 601 controls the display screen 605 to switch from the screen-off state to the screen-on state.

[0158] Those skilled in the art will understand that Figure 6 The structure shown in the figure does not constitute a limitation on the terminal 600, and the terminal 600 may include more or less components than those shown in the figure, or combine some components, or adopt a different component arrangement.

[0159] In an exemplary embodiment, a computer-readable storage medium is also provided, such as a memory including instructions, and the instructions can be executed by a processor in a terminal to complete the method of master audio generation in the above embodiment. The computer-readable storage medium can be non-transitory. For example, the computer-readable storage medium can be a ROM (Read-Only Memory), a RAM (Random Access Memory), a CD-ROM (Compact Disc Read-Only Memory), a magnetic tape, a floppy disk, and an optical data storage device, etc.

[0160] It should be noted that the information (including but not limited to user device information, user personal information, etc.), data (including but not limited to data used for analysis, stored data, displayed data, etc.) and signals (including but not limited to signals transmitted between user terminals and other devices, etc.) involved in this application are all authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant laws, regulations and standards of relevant countries and regions. For example, the original master audio and audio to be processed involved in this application are all obtained with full authorization.

[0161] A person skilled in the art will understand that all or part of the steps to implement the above embodiments may be accomplished by hardware or by instructing related hardware through a program, and the program may be stored in a computer-readable storage medium, and the above-mentioned storage medium may be a read-only memory, a disk or an optical disk, etc.

[0162] The above description is only an optional embodiment of the present application and is not intended to limit the present application. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present application shall be included in the protection scope of the present application.

Claims

1. A method for generating a master audio, characterized in that: The method is applied to a terminal, and the method includes: Get the audio to be processed of the target song; Dividing the audio to be processed into sub-bands to obtain multiple sub-band audios; Inputting the plurality of sub-band audios into a master audio generation model to obtain a plurality of audios to be synthesized; Sub-band synthesis is performed on the multiple audios to be synthesized to obtain the master audio of the target song; the quality of the master audio is higher than the quality of the audio to be processed.

2. The method according to claim 1, characterized in that: The step of dividing the audio to be processed into sub-bands to obtain a plurality of sub-band audios includes: Convolving the audio to be processed with a plurality of analysis filters respectively to obtain a plurality of analysis filtered signals; The multiple analysis and filtered signals are downsampled to obtain multiple sub-band audios.

3. The method according to claim 1 or 2, characterized in that: The step of performing sub-band synthesis on the plurality of audios to be synthesized to obtain the master audio of the target song includes: Convolving the multiple audios to be synthesized with multiple synthesis filters respectively to obtain multiple synthesis filtered signals; Upsampling the multiple synthetic filtered signals to obtain multiple upsampled signals; The multiple up-sampled signals are added together to obtain the master audio of the target song.

4. The method according to claim 3, characterized in that The step of obtaining the audio to be processed of the target song includes: Acquire the audio to be improved in sound quality of the target song, wherein the sampling rate of the audio to be improved in sound quality is lower than the sampling rate of the master audio; The audio to be processed is upsampled to obtain the audio to be processed at a target sampling rate.

5. The method according to any one of claims 1 to 4, characterized in that The master audio generation model is an AutoEncoder model.

6. The method according to any one of claims 1 to 4, characterized in that Before obtaining the to-be-processed audio of the target song, the method further includes: In the playback interface of the target song, a user's selection operation on the master audio playback option of the target song is detected.

7. The method according to claim 1, characterized in that The method further comprises: Get the original master audio of the sample song; Downsampling the original master audio to audio at a first sampling rate, wherein the first sampling rate is lower than the sampling rate of the original master audio; Upsampling the audio at the target sampling rate to audio at a second sampling rate, wherein the second sampling rate is the sampling rate of the original master audio; Dividing the audio at the second sampling rate into sub-bands to obtain a plurality of sample sub-band audios; Inputting the plurality of sample sub-band audios into a master audio generation model to be trained to obtain a plurality of sample audios to be synthesized; Performing sub-band synthesis on the plurality of sample audios to be synthesized to obtain a predicted master audio; A loss value is calculated based on the predicted master audio and the original master audio, and the master audio generation model to be trained is trained based on the loss value.

8. A terminal, characterized in that: The terminal includes a processor and a memory, wherein the memory stores at least one instruction, and the instruction is loaded and executed by the processor to implement the operation performed by the method for generating a master audio according to any one of claims 1 to 7.

9. A computer-readable storage medium, characterized in that: The storage medium stores at least one instruction, which is loaded and executed by the processor to implement the operations performed by the method for generating master audio according to any one of claims 1 to 7.

10. A computer program product, characterized in that The computer program product stores at least one instruction, which is loaded and executed by a processor to implement the operations performed by the method for generating a master audio according to any one of claims 1 to 7.

Images