Method and apparatus for audio processing

Abstract

The application provides a method and device for audio processing, and relates to the technical field of audio. The method comprises the following steps: inputting a positive audio feature vector and a negative audio feature vector into a trained music style transfer model, wherein the positive audio feature vector is used to represent chords and beats of an original song and is also used to represent a music style of a reference song; when the music style transfer model generates an audio feature, the positive audio feature vector is inferred to obtain a first feature probability distribution; the negative audio feature vector is inferred to obtain a second feature probability distribution; based on the two feature probability distributions, a target audio feature vector is determined; when the target audio feature vector does not meet an audio quality condition, the target audio feature vector is taken as the negative audio feature vector, and the step of inputting the positive audio feature vector and the negative audio feature vector into the trained music style transfer model is performed again. The music style transfer can be automatically and efficiently realized by using the application.

Description

Technical Field

[0001] This application relates to the field of audio technology, and in particular to an audio processing method and apparatus. Background Technology

[0002] For a song, users may want to hear versions of it in other musical styles. For example, if a song was originally in the rock style, a user might want to hear a folk version. This change in musical style is called style migration.

[0003] Currently, music style transfer is typically achieved by professionals using specialized music editing software to edit the song audio. However, manual music style transfer is less efficient. Summary of the Invention

[0004] This application provides an audio processing method and apparatus that can automatically achieve music style transfer without manual intervention, resulting in higher efficiency. The technical solution is as follows: Firstly, an audio processing method is provided, the method comprising: The positive and negative audio feature vectors are input into the trained music style transfer model. The positive audio feature vector is used to represent the chords and beats of the original song and also to represent the music style of the reference song. Each time the music style transfer model generates audio features, it infers from the positive audio feature vector to obtain a first feature probability distribution; and it infers from the negative audio feature vector to obtain a second feature probability distribution. Based on the first feature probability distribution and the second feature probability distribution, the target audio feature vector of the output is determined; When the target audio feature vector does not meet the audio quality conditions, the target audio feature vector is used as a negative audio feature vector, and the process is switched to inputting the positive and negative audio feature vectors into the trained music style transfer model until the obtained target audio feature vector meets the audio quality conditions.

[0005] In one possible implementation, the method further includes: When the target audio feature vector meets the audio quality conditions, a target song audio is generated based on the target audio feature vector, wherein the music style of the target song audio matches the music style of the reference song, and the chords and rhythms of the target song audio match the chords and rhythms of the original song.

[0006] In one possible implementation, generating the target song audio based on the target audio feature vector when the target audio feature vector satisfies the audio quality condition includes: Determine the score of the target audio feature vector; if the score is greater than a threshold, determine that the target audio feature vector meets the audio quality conditions; and generate the target song audio based on the target audio feature vector. The step of using the target audio feature vector as a negative audio feature vector when the target audio feature vector does not meet the audio quality conditions includes: If the score is less than the threshold, the target audio feature vector is determined to not meet the audio quality conditions, and the target audio feature vector is used as a negative audio feature vector.

[0007] In one possible implementation, determining the output target audio feature vector based on the first feature probability distribution and the second feature probability distribution includes: Calculate the third feature probability distribution based on the first and second feature probability distributions; Based on the third feature probability distribution, the target audio feature vector for output is determined.

[0008] In one possible implementation, generating the target song audio based on the target audio feature vector includes: The target audio feature vector is decoded to obtain the target accompaniment audio; Obtain the vocal audio of the original song; The target accompaniment audio and the human voice audio are mixed to obtain the target song audio.

[0009] In one possible implementation, the method further includes: Acquire the control audio signal of the original song, wherein the control audio signal is used to characterize the chords and rhythm of the original song; Obtain text information, wherein the text information includes music style indication information; Obtain the musical style features of a reference song, wherein the musical style features are used to characterize the musical style of the reference song; Based on the control audio signal, the text information, and the musical style features of the reference song, a positive audio feature vector is generated.

[0010] In one possible implementation, acquiring the control audio signal of the original song includes: Extract the beat audio signal, chord audio signal, and vocal audio signal from the original song audio; The beat audio signal, chord audio signal, and vocal audio signal are mixed to obtain the control audio signal of the original song.

[0011] Secondly, an audio processing apparatus is provided, the apparatus comprising: The input module is used to input the positive audio feature vector and the negative audio feature vector into the trained music style transfer model. The positive audio feature vector is used to represent the chords and rhythm of the original song and also to represent the music style of the reference song. The inference module is used to infer a first feature probability distribution from the positive audio feature vector each time the music style transfer model generates audio features; and to infer a second feature probability distribution from the negative audio feature vector; and to determine the output target audio feature vector based on the first feature probability distribution and the second feature probability distribution. The generation module is used to, when the target audio feature vector does not meet the audio quality conditions, treat the target audio feature vector as a negative audio feature vector and switch to the process of inputting the positive and negative audio feature vectors into the trained music style transfer model until the obtained target audio feature vector meets the audio quality conditions.

[0012] In one possible implementation, the generation module is further configured to: When the target audio feature vector meets the audio quality conditions, a target song audio is generated based on the target audio feature vector, wherein the music style of the target song audio matches the music style of the reference song, and the chords and rhythms of the target song audio match the chords and rhythms of the original song.

[0013] In one possible implementation, the generation module is configured to: Determine the score of the target audio feature vector; if the score is greater than a threshold, determine that the target audio feature vector meets the audio quality conditions; and generate the target song audio based on the target audio feature vector. If the score is less than the threshold, the target audio feature vector is determined to not meet the audio quality conditions, and the target audio feature vector is used as a negative audio feature vector.

[0014] In one possible implementation, the inference module is used for: Calculate the third feature probability distribution based on the first and second feature probability distributions; Based on the third feature probability distribution, the target audio feature vector for output is determined.

[0015] In one possible implementation, the generation module is configured to: The target audio feature vector is decoded to obtain the target accompaniment audio; Obtain the vocal audio of the original song; The target accompaniment audio and the human voice audio are mixed to obtain the target song audio.

[0016] In one possible implementation, the apparatus further includes an acquisition module for: Acquire the control audio signal of the original song, wherein the control audio signal is used to characterize the chords and rhythm of the original song; Obtain text information, wherein the text information includes music style indication information; Obtain the musical style features of a reference song, wherein the musical style features are used to characterize the musical style of the reference song; Based on the control audio signal, the text information, and the musical style features of the reference song, a positive audio feature vector is generated.

[0017] In one possible implementation, the acquisition module is configured to: Extract the beat audio signal, chord audio signal, and vocal audio signal from the original song audio; The beat audio signal, chord audio signal, and vocal audio signal are mixed to obtain the control audio signal of the original song.

[0018] Thirdly, a computing device is provided, characterized in that the computing device includes a processor and a memory, the memory storing at least one instruction, the instruction being loaded and executed by the processor to perform the operations performed as described in the first aspect and any possible method of audio processing described in the first aspect.

[0019] Fourthly, a computer-readable storage medium is provided, the storage medium storing at least one instruction, the instruction being loaded and executed by a processor to perform the operations performed as described in the first aspect and any possible method of audio processing described in the first aspect.

[0020] Fifthly, a computer program product is provided, the computer program product storing at least one instruction, the instruction being loaded and executed by a processor to perform the operations performed as described in the first aspect and any possible implementation of the audio processing method described in the first aspect.

[0021] The beneficial effects of the technical solution provided in this application are: In the technical solution provided in this application, positive audio feature vectors are used as controls to guide the music style transfer model to generate music of what music style, tempo, and chords. At the same time, a negative feedback mechanism is used to instruct the music style transfer model not to generate music with negative audio feature vectors. The entire music style transfer process does not require human intervention, making it more efficient. Furthermore, the negative feedback mechanism results in higher quality audio of the generated target song. Attached Figure Description

[0022] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0023] Figure 1 This is a schematic diagram of a classifier-free guidance technology provided in an embodiment of this application; Figure 2 This is a flowchart of an audio processing method provided in an embodiment of this application; Figure 3 This is a schematic diagram of an audio processing method provided in an embodiment of this application; Figure 4 This is a schematic diagram of an audio processing method provided in an embodiment of this application; Figure 5 This is a schematic diagram of an audio processing device provided in an embodiment of this application; Figure 6 This is a schematic diagram of the structure of a computing device provided in an embodiment of this application. Detailed Implementation

[0024] To make the objectives, technical solutions, and advantages of this application clearer, the embodiments of this application will be described in further detail below with reference to the accompanying drawings.

[0025] For a song, users may want to hear versions in other musical styles. For example, if a song was originally rock, a user might want to hear a folk version; this change in musical style is called style transfer. Currently, style transfer is usually achieved by professionals using specialized music editing software to edit the song's audio. However, manually performing style transfer is less efficient.

[0026] This application provides an audio processing method that can be implemented by a computing device, such as a mobile phone, tablet computer, laptop computer, desktop computer, or a server or server cluster. In this method, positive and negative audio feature vectors are input into a trained music style transfer model. The positive audio feature vector represents the chords and rhythm of the original song and also represents the musical style of the reference song. The music style transfer model then infers from the positive and negative audio feature vectors to obtain a target audio feature vector. If the target audio feature vector meets the audio quality requirements, a target song audio is generated based on the target audio feature vector. The musical style of the target song audio matches the musical style of the reference song, and the chords and rhythm of the target song audio match those of the original song. If the target audio feature vector does not meet the audio quality requirements, the target audio feature vector is used as a negative audio feature vector, and the process returns to inputting the positive and negative audio feature vectors into the trained music style transfer model. As can be seen, in this method, positive audio feature vectors are used as controls to guide the music style transfer model to generate music of what style, tempo, and chords. At the same time, a negative feedback mechanism is used to instruct the music style transfer model not to generate music with negative audio feature vectors. The entire music style transfer process does not require human intervention, making it more efficient. Furthermore, the negative feedback mechanism results in higher quality audio of the generated target song.

[0027] The following describes the relevant technologies used in the audio processing method provided in the embodiments of this application: The audio processing method provided in this application requires CFG (Classifier-Free Guidance) technology for implementation. The following example illustrates CFG technology.

[0028] See Figure 1If a user wants to know which city attraction A is located in, the user can input the positive suggestion "Where is attraction A?" The system will then generate the corresponding negative suggestion "Attraction A is not in which city?" The positive and negative suggestions are then input into the LLM (Large Language Management) system. The Large Language Model (LLM) infers the probability of each positive output for positive prompts: Shenzhen has a probability of 0.2, Beijing has a probability of 0.6, Shanghai has a probability of 0.1, and Chengdu has a probability of 0.1. The LLM also infers the probability of each negative output for negative prompts: Shenzhen has a probability of 0.3, Beijing has a probability of 0, Shanghai has a probability of 0.4, and Chengdu has a probability of 0.3. Then, the probability of the negative output corresponding to Shenzhen is subtracted from the probability of the positive output corresponding to Shenzhen, resulting in a probability of -0.1 for the answer "Shenzhen". Similarly, the probability of the negative output corresponding to Beijing is subtracted from the probability of the positive output corresponding to Beijing, resulting in a probability of 0.6 for the answer "Beijing". The probability of the negative output corresponding to Shanghai is subtracted from the probability of the positive output corresponding to Shanghai, resulting in a probability of -0.3 for the answer "Shanghai". Finally, the probability of the negative output corresponding to Chengdu is subtracted from the probability of the positive output corresponding to Chengdu, resulting in a probability of -0.2 for the answer "Shanghai". Therefore, the final answer returned to the user is "Beijing".

[0029] The audio processing method provided in the embodiments of this application will be described below with reference to the accompanying drawings. Figure 2 The processing flow of this method may include the following steps: Step 201: Input the positive and negative audio feature vectors into the trained music style transfer model. The positive audio feature vector represents the chords and rhythm of the original song, and also represents the music style of the reference song. The music style can be folk, rock, electronic music, etc. The music style transfer model can be an LLM (Limited Linear Modulation).

[0030] In practice, the terminal device can have a music application installed. If a user wants to change the music style of the original song to that of the reference song, they can select the original song and then select the reference song in the aforementioned music application.

[0031] Furthermore, the terminal device executes the audio processing method provided in the embodiments of this application to achieve music style transfer.

[0032] Alternatively, the terminal device sends a music style migration request to the server, the request carrying the identifiers of the original song and the reference song. Then, upon receiving the music style migration request, the server executes the audio processing method provided in this embodiment to achieve music style migration.

[0033] The audio processing method provided in this application embodiment is the same whether it is executed by a terminal or by a server. The following describes the execution of the audio processing method provided in this application embodiment by a computing device. The computing device can be a terminal device or a server, and this application embodiment does not limit it in this regard.

[0034] The computing device can acquire the audio of the original song and the audio of the reference song. The song audio can be obtained from a music library, provided by the user, or obtained via the Internet. This application embodiment does not limit the method of acquiring the song audio.

[0035] Then, the computing device generates positive audio features based on the audio of the original song and the audio of the reference song. These positive audio feature vectors are used to represent the chords and rhythm of the original song, and also to represent the musical style of the reference song. In addition, the computing device can automatically generate an all-zero vector as the initial negative audio feature vector input to the music style transfer model.

[0036] There may be many implementations for generating positive audio feature vectors, which will be illustrated in subsequent embodiments.

[0037] Step 202: Each time the music style transfer model generates audio features, it infers the positive audio feature vectors to obtain the first feature probability distribution, and infers the negative audio feature vectors to obtain the second feature probability distribution. Based on the first and second feature probability distributions, the target audio feature vector for output is determined.

[0038] In implementation, after inputting positive and negative audio feature vectors into the music style transfer model, the process of generating audio features is essentially the process of outputting tokens. During each output of the next token, the music style transfer model infers the first probability of each token in the vocabulary by reasoning about the positive audio feature vector, and infers the second probability of each token by reasoning about the negative audio feature vector. The first probabilities of each token in the vocabulary form the first feature probability distribution, and the second probabilities form the second feature probability distribution. Then, based on the first and second feature probability distributions, the next token is determined, and based on all determined next tokens, the target audio feature vector is obtained. The process of determining the next token is explained below: In determining the next token, the music style transfer model uses positive audio feature vectors to determine the first probability of each token in the vocabulary, and negative audio feature vectors to determine the second probability. This determination of the first probability based on positive audio feature vectors and the second probability based on negative audio feature vectors is a standard calculation in LLM for determining the next token, and will not be elaborated upon in this embodiment.

[0039] Then, based on the first and second probabilities of each token in the vocabulary, the next token is determined. Specifically, based on the first and second probabilities of each token in the vocabulary, a third probability of each token in the vocabulary can be calculated, and then, based on the third probability of each token in the vocabulary, the next token to be output is determined. This can also be understood as calculating a third feature probability distribution based on the first and second feature probability distributions, where the third feature probability distribution includes the third probability of each token in the vocabulary.

[0040] Here, the next token to be output is determined based on the third probability of each token in the vocabulary. This can be done through methods such as Greedy Sampling, Random Sampling, Nucleus Sampling / Top-p Sampling, and Beam Search. These are all conventional calculations for determining the next token in LLM, and will not be elaborated on in this embodiment.

[0041] The third probability of each token in the vocabulary can be calculated as follows: for each token in the vocabulary, subtract the second probability of the token from the first probability of the token to obtain the third probability of the token.

[0042] After determining all the next tokens, these tokens are combined into a token sequence, and the token sequence is vectorized to obtain the target audio feature vector.

[0043] Step 203: Determine whether the target audio feature vector meets the audio quality conditions.

[0044] In practice, the target audio feature vector can be scored, and the score can be compared with a threshold. Based on the comparison result, it can be determined whether the target audio feature vector meets the audio quality conditions.

[0045] Specifically, the target audio feature vector can be input into the scoring model to obtain a score for the target audio feature vector. The scoring model can be a pre-trained artificial intelligence model, which can be used directly by calling its interface in this embodiment.

[0046] Step 204: If the target audio feature vector meets the audio quality requirements, then generate the target song audio based on the target audio feature vector. Specifically, the musical style of the target song audio matches the musical style of the reference song, and the chords and rhythms of the target song audio match those of the original song.

[0047] In implementation, if the score obtained in step 203 above is greater than the threshold, it is determined that the target audio feature vector meets the audio quality conditions, and the target song audio is generated based on the target audio feature vector. The process of generating the target song audio based on the target audio feature vector can be as follows: The target audio feature vector is decoded to obtain the target accompaniment audio. Then, the original song audio is processed by vocal separation to obtain the original song's vocal audio. Next, the target accompaniment audio and vocal audio are mixed to obtain the target song audio. The resulting target song audio matches the musical style of the reference song, the chords and rhythm of the target song audio match those of the original song, and the lyrics of the target song audio also match those of the original song.

[0048] In one possible implementation, after obtaining the target accompaniment audio, it can be input into a super-resolution model to enhance its audio quality, resulting in an enhanced target accompaniment audio. Then, the target accompaniment audio and the original song's vocal audio signal are mixed to obtain the target song audio. The enhanced target accompaniment audio can be high-definition, stereo, 48kHz sampling rate audio; high-definition refers to high audio quality standards, such as 24-bit depth. The super-resolution model can be a pre-trained artificial intelligence model, which can be directly used by calling its interface in this embodiment.

[0049] Step 205: If the target audio feature vector does not meet the audio quality conditions, then the target audio feature vector is used as a negative audio feature vector, and proceed to step 201.

[0050] In implementation, if the score obtained in step 203 is less than the threshold, it is determined that the target audio feature vector does not meet the audio quality conditions. Therefore, the target audio feature vector is used as a negative audio feature vector, and the process returns to step 201. In step 201, there is no need to generate a positive audio feature vector again; the positive audio feature vector from the previous input music style transfer model can be used directly. The negative audio feature vector uses the target audio feature vector obtained this time. The loop continues until the obtained target audio feature vector meets the audio quality conditions, or the number of iterations reaches the threshold, at which point the loop stops.

[0051] The following is an exemplary description of the method for generating the positive audio feature vector in step 201 above. Several methods are listed below.

[0052] Method 1: Feature extraction is performed on the original song's audio to obtain a first audio feature vector representing the chords and rhythm of the original song. Feature extraction is performed on the reference song's audio to obtain a second audio feature vector representing the reference song's musical style. The first and second audio feature vectors are combined to obtain a positive audio feature vector.

[0053] Method 2: The process involves: acquiring the control audio signal of the original song, where the control audio signal represents the chords and rhythm of the original song; acquiring text information, where the text information includes music style indication information; acquiring the music style feature vector of the reference song, where the music style features represent the music style of the reference song; and then generating a positive audio feature vector based on the control audio signal, text information, and music style features of the reference song.

[0054] The following explains the process of obtaining the control audio signal of the original song: Extract the beat audio signal, chord audio signal, and vocal audio signal from the original song's audio. Mix the beat audio signal, chord audio signal, and vocal audio signal to obtain the control audio signal of the original song.

[0055] Specifically, after acquiring the original song's audio, the computing device can input the audio into a beat recognition model to obtain the beat information of the original song. Then, the beat information is input into an audio synthesis model to obtain the beat audio signal of the original song. The beat audio signal can be in WAV (Waveform Audio File Format). The beat recognition model and audio synthesis model can be pre-trained artificial intelligence models, and in this embodiment, their interfaces can be directly called.

[0056] The computing device inputs the original song's audio into a chord recognition model to obtain the chord information of the original song. Then, it inputs the chord information of the original song into an audio synthesis model to obtain the chord audio signal of the original song. The format of the chord audio signal can be WAV, and the chord audio signal can be a bass WAV that can represent chord information. The chord recognition model can be a pre-trained artificial intelligence model, and in this embodiment, it can be used directly by simply calling its interface.

[0057] The computing device inputs the original song's audio into the vocal separation model to obtain the original song's vocal audio signal. The vocal separation model can be a pre-trained artificial intelligence model, which, in this embodiment, can be used directly by simply calling its interface.

[0058] After obtaining the beat audio signal, chord audio signal, and vocal audio signal of the original song, these signals are mixed to obtain the control audio signal of the original song. Specifically, each of the beat audio signal, chord audio signal, and vocal audio signal has a corresponding weight. The computing device can perform a weighted summation of the beat audio signal, chord audio signal, and vocal audio signal according to their respective weights to obtain the control audio signal of the original song. The weights can be configured by relevant technicians according to actual needs, and this application embodiment does not limit this.

[0059] The control audio signal of the original song has the same duration as the original song's audio.

[0060] The aforementioned text information can be entered by the user through a music application. The text information may include music style indication information, which is used to indicate the music style of the audio that the user wants to obtain (i.e., the music style of the reference song, which is entered in both text and feature form in this application), such as folk, rock, electronic music, etc.

[0061] The following explains the process of obtaining the musical style feature vector of the reference song: After obtaining the audio of the reference song, the computing device can input the audio into a style feature extraction model to obtain the music style feature vector of the reference song. The style feature extraction model can be a pre-trained artificial intelligence model, which can be directly used by calling its interface in this embodiment.

[0062] The following explains how to generate positive audio feature vectors based on control audio signals, text information, and the musical style features of a reference song: The computing device encodes the control audio signal, text information, and musical style feature vector of the original song to obtain a token sequence. Then, it performs feature vectorization on the token sequence to obtain a positive audio feature vector. The first feature vector can be an embedding feature vector.

[0063] Method 3: To enhance the sense of segmentation in the final target song audio, segment information can be introduced.

[0064] Accordingly, the processing of obtaining the control audio signal of the original song in Method 2 above can be as follows: After acquiring the original song's audio, the computing device inputs it into a segment recognition model. This model segments the audio into multiple segments, providing segment information for each segment, indicating whether it's a verse, chorus, pre-chorus, or bridge. The computing device can then generate the target song accompaniment audio for each segment. The following example illustrates generating the target song accompaniment audio for the first segment, which can be any one of the aforementioned segments.

[0065] The computing device inputs the first segment of the song's audio into the beat recognition model to obtain the beat information of the first segment. Then, it inputs the beat information and the segment information of the first segment into the audio synthesis model to obtain the beat audio signal of the first segment. Here, the segment information is input to ensure that the beat audio signals of different segments differ in waveform, so that the final generated accompaniment audio has a better sense of segmentation.

[0066] The computing device inputs the first section of the song's audio into the chord recognition model to obtain the chord information for the first section. Then, it inputs the chord information and the segment information of the first section into the audio synthesis model to obtain the chord audio signal for the first section. Here, the segment information is input to ensure that the chord audio signals of different sections differ in waveform, so that the final generated accompaniment audio has a better sense of segmentation.

[0067] The computing device inputs the audio of the first segment of the song into the vocal separation model to obtain the vocal audio signal of the first segment of the original song.

[0068] After obtaining the beat audio signal, chord audio signal, and vocal audio signal of the first section of the song audio, these signals are mixed to obtain the control audio signal for the first section. Specifically, each of the beat audio signal, chord audio signal, and vocal audio signal has a corresponding weight. The computing device can perform a weighted sum of the beat audio signal, chord audio signal, and vocal audio signal according to their respective weights to obtain the control audio signal for the first section. The control audio signal for the first section has the same duration as the first section of the original song audio.

[0069] Furthermore, because segment information is introduced during the generation of control audio signals, the waveforms of the control audio signals for different segments differ. For example, the control audio signal for the verse can consist of one 750Hz and three 350Hz sine waves, the control audio signal for the chorus can consist of alternating 750Hz and 350Hz sine waves, and the control audio signal for the pre-chorus can consist of alternating 750Hz and 150Hz sine waves. In addition, to further emphasize the sense of segmentation, a designated audio signal can be added after the control audio signal for each segment to indicate the segment change; for example, the designated audio signal could be the audio signal for a rise sound effect.

[0070] The text information processing in Method 2 above can be performed as follows: The computing device can first obtain the lyrics of the original song and extract the lyrics of the first paragraph of the original song. The lyrics of the first paragraph contain the paragraph identifier text of the first paragraph, which is used to indicate which paragraph the first paragraph is.

[0071] Furthermore, the computing device can treat the music style indication information and the lyrics of the first paragraph as text information. With this implementation, when the user inputs the music style indication information, they can input the music style indication information corresponding to each paragraph separately. Therefore, the music style indication information used as text information can refer to the music style indication information corresponding to the first paragraph.

[0072] In addition, the computing device can also process the audio of the reference song as follows: After obtaining the audio of the reference song, the audio can be input into the segment recognition model. The segment recognition model will segment the audio of the reference song into segments, resulting in multiple segments of the audio of the reference song. The model can also obtain the segment information corresponding to each segment of the audio, which can indicate whether the audio is a verse, chorus, pre-chorus, or bridge, etc.

[0073] In addition, the computing device can input the entire audio of the reference song into the style feature extraction model to obtain the music style feature vector of the entire reference song.

[0074] After obtaining the control audio signal corresponding to the first paragraph of the original song, the audio of the second paragraph of the reference song, and the music style feature vector of the entire reference song, the following processing can be performed to obtain the positive audio feature vector: The control audio signal and text information corresponding to the first paragraph of the original song, along with the audio of the second paragraph of the reference song, are segmented into words to obtain the first token sequence. Then, a specified candidate token is inserted at the end of the first token sequence to obtain the second token sequence. The second token sequence is then processed into feature vectors to obtain the first feature vector. Finally, the feature sequence corresponding to the specified candidate token in the first feature vector is replaced with the music style feature vector of the entire reference song to obtain the positive audio feature vector corresponding to the first paragraph.

[0075] Accordingly, the positive audio feature vector input in step 201 above can be the positive audio feature vector corresponding to the first segment obtained here, and then the above steps are performed. Figure 2 The process shown first yields the target accompaniment audio for the first segment. Using the same method, the target accompaniment audio for each segment can be obtained. Then, the accompaniment audio for each target segment is combined according to the playback sequence to obtain the complete target accompaniment audio. Finally, the complete target accompaniment audio is mixed with the original song's vocal audio signal to obtain the complete target song audio.

[0076] The music style transfer model described above can be pre-trained. During training, multiple sample songs can be obtained from the music library. For each sample song, the control audio signal of the sample song is extracted as the first input sample, the music style features of the sample song are extracted as the second input sample, and the accompaniment audio of the sample song is extracted as the output sample. In this way, the first input sample, the second input sample, and the output sample corresponding to each sample song can form a sample pair. Each sample pair is used to optimize and fine-tune the music style transfer model until the training termination condition is met, at which point training stops, and the trained music style transfer model is obtained.

[0077] All of the above-mentioned optional technical solutions can be combined in any way to form the optional embodiments of this application, and will not be described in detail here.

[0078] In the solution provided in this application embodiment, positive audio feature vectors are used as controls to guide the music style transfer model to generate music of what music style, tempo, and chords. At the same time, a negative feedback mechanism is used to instruct the music style transfer model not to generate music with negative audio feature vectors. The entire music style transfer process does not require manual intervention, making it more efficient. Furthermore, the negative feedback mechanism results in higher quality target song audio.

[0079] See Figure 3 In the solution provided in this application embodiment, positive audio feature vectors and negative audio feature vectors are input into the music style transfer model. The music style transfer model performs reasoning on the positive audio feature vectors and negative audio feature vectors respectively. Reasoning on the positive audio feature vectors can obtain the first probability of each token in the vocabulary, and reasoning on the negative audio feature vectors can obtain the second probability of each token in the vocabulary. Then, the first probability of each token is subtracted from the second probability to obtain the third probability of each token. Based on the third probability of each token, the next token to be output can be determined.

[0080] See Figure 4 In the solution provided in this application embodiment, the target audio feature vector obtained by the music style transfer model is scored using a scoring model. If the score is greater than a threshold, a target accompaniment audio is generated based on the target audio feature vector, and then the target song audio is generated. If the score is less than the threshold, the target audio feature vector is used as a negative audio feature vector and input together with the positive audio feature vector into the music style transfer model for inference.

[0081] Based on the same technical concept, embodiments of this application also provide an audio processing apparatus, which can be applied to computing devices, such as... Figure 5 As shown, the device includes an input module 510, an inference module 520, and a generation module 530, wherein: The input module 510 is used to input the positive audio feature vector and the negative audio feature vector into the trained music style transfer model, wherein the positive audio feature vector is used to represent the chords and rhythm of the original song, and also to represent the music style of the reference song. The inference module 520 is used to infer the positive audio feature vector to obtain a first feature probability distribution each time the music style transfer model generates audio features; and to infer the negative audio feature vector to obtain a second feature probability distribution; and to determine the output target audio feature vector based on the first feature probability distribution and the second feature probability distribution. The generation module 530 is used to, when the target audio feature vector does not meet the audio quality conditions, treat the target audio feature vector as a negative audio feature vector and switch to the process of inputting the positive and negative audio feature vectors into the trained music style transfer model until the obtained target audio feature vector meets the audio quality conditions.

[0082] In one possible implementation, the generation module 530 is further configured to: When the target audio feature vector meets the audio quality conditions, a target song audio is generated based on the target audio feature vector, wherein the music style of the target song audio matches the music style of the reference song, and the chords and rhythms of the target song audio match the chords and rhythms of the original song.

[0083] In one possible implementation, the generation module 530 is configured to: Determine the score of the target audio feature vector; if the score is greater than a threshold, determine that the target audio feature vector meets the audio quality conditions; and generate the target song audio based on the target audio feature vector. If the score is less than the threshold, the target audio feature vector is determined to not meet the audio quality conditions, and the target audio feature vector is used as a negative audio feature vector.

[0084] In one possible implementation, the inference module 520 is used for: Calculate the third feature probability distribution based on the first and second feature probability distributions; Based on the third feature probability distribution, the target audio feature vector for output is determined.

[0085] In one possible implementation, the generation module 530 is configured to: The target audio feature vector is decoded to obtain the target accompaniment audio; Obtain the vocal audio of the original song; The target accompaniment audio and the human voice audio are mixed to obtain the target song audio.

[0086] In one possible implementation, the apparatus further includes an acquisition module for: Acquire the control audio signal of the original song, wherein the control audio signal is used to characterize the chords and rhythm of the original song; Obtain text information, wherein the text information includes music style indication information; Obtain the musical style features of a reference song, wherein the musical style features are used to characterize the musical style of the reference song; Based on the control audio signal, the text information, and the musical style features of the reference song, a positive audio feature vector is generated.

[0087] In one possible implementation, the acquisition module is configured to: Extract the beat audio signal, chord audio signal, and vocal audio signal from the original song audio; The beat audio signal, chord audio signal, and vocal audio signal are mixed to obtain the control audio signal of the original song.

[0088] In the solution provided in this application, positive audio feature vectors are used as controls to guide the music style transfer model to generate music of what style, tempo, and chords. At the same time, a negative feedback mechanism is used to instruct the music style transfer model not to generate music with negative audio feature vectors. The entire music style transfer process does not require human intervention, making it more efficient. Furthermore, the negative feedback mechanism results in higher quality audio of the generated target song.

[0089] It should be noted that the audio processing apparatus provided in the above embodiments is only illustrated by the division of the above functional modules. In practical applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the terminal device can be divided into different functional modules to complete all or part of the functions described above. In addition, the audio processing apparatus and the audio processing method embodiments provided in the above embodiments belong to the same concept, and their specific implementation process can be found in the method embodiments, which will not be repeated here.

[0090] Figure 6 This illustration shows a structural block diagram of a computing device 800 provided in an exemplary embodiment of this application. The computing device 800 may be a terminal device, such as a smartphone, tablet computer, MP3 player (Moving Picture Experts Group Audio Layer III), MP4 player (Moving Picture Experts Group Audio Layer IV), laptop computer, or desktop computer. The computing device 800 may also be an audio output device, such as headphones or speakers.

[0091] Typically, computing device 800 includes a processor 801 and a memory 802.

[0092] Processor 801 may include one or more processing cores, such as a quad-core processor or an octa-core processor. Processor 801 may be implemented using at least one hardware form selected from DSP (Digital Signal Processing), FPGA (Field-Programmable Gate Array), and PLA (Programmable Logic Array). Processor 801 may also include a main processor and a coprocessor. The main processor, also known as a CPU (Central Processing Unit), is used to process data in the wake-up state; the coprocessor is a low-power processor used to process data in the standby state. In some embodiments, processor 801 may integrate a GPU (Graphics Processing Unit), which is responsible for rendering and drawing the content to be displayed on the screen. In some embodiments, processor 801 may also include an AI (Artificial Intelligence) processor, which is used to handle computational operations related to machine learning.

[0093] The memory 802 may include one or more computer-readable storage media, which may be non-transitory. The memory 802 may also include high-speed random access memory and non-volatile memory, such as one or more disk storage devices or flash memory devices. In some embodiments, the non-transitory computer-readable storage media in the memory 802 are used to store at least one instruction, which is executed by the processor 801 to implement the audio processing method provided in the method embodiments of this application.

[0094] In some embodiments, the computing device 800 may also optionally include a peripheral device interface 803 and at least one peripheral device. The processor 801, memory 802, and peripheral device interface 803 can be connected via a bus or signal line. Each peripheral device can be connected to the peripheral device interface 803 via a bus, signal line, or circuit board. Specifically, the peripheral device includes at least one of the following: a radio frequency circuit 804, a display screen 805, a camera assembly 806, an audio circuit 807, a positioning assembly 808, and a power supply 809.

[0095] Peripheral device interface 803 can be used to connect at least one I / O (Input / Output) related peripheral device to processor 801 and memory 802. In some embodiments, processor 801, memory 802 and peripheral device interface 803 are integrated on the same chip or circuit board; in some other embodiments, any one or two of processor 801, memory 802 and peripheral device interface 803 can be implemented on separate chips or circuit boards, which is not limited in this embodiment.

[0096] The radio frequency (RF) circuit 804 is used to receive and transmit RF (Radio Frequency) signals, also known as electromagnetic signals. The RF circuit 804 communicates with communication networks and other communication devices via electromagnetic signals. The RF circuit 804 converts electrical signals into electromagnetic signals for transmission, or converts received electromagnetic signals back into electrical signals. Optionally, the RF circuit 804 includes: an antenna system, an RF transceiver, one or more amplifiers, a tuner, an oscillator, a digital signal processor, a codec chipset, a user identity module card, etc. The RF circuit 804 can communicate with other terminals through at least one wireless communication protocol. This wireless communication protocol includes, but is not limited to: the World Wide Web, metropolitan area networks, intranets, various generations of mobile communication networks (2G, 3G, 4G, and 5G), wireless local area networks, and / or WiFi (Wireless Fidelity) networks. In some embodiments, the RF circuit 804 may also include circuitry related to NFC (Near Field Communication), which is not limited in this application.

[0097] Display screen 805 is used to display a UI (User Interface). This UI may include graphics, text, icons, videos, and any combination thereof. When display screen 805 is a touch display screen, it also has the ability to collect touch signals on or above its surface. These touch signals can be input as control signals to processor 801 for processing. In this case, display screen 805 can also be used to provide virtual buttons and / or a virtual keyboard, also known as soft buttons and / or a soft keyboard. In some embodiments, there may be one display screen 805, disposed on the front panel of computing device 800; in other embodiments, there may be at least two display screens, disposed on different surfaces of computing device 800 or in a folded design; in still other embodiments, display screen 805 may be a flexible display screen, disposed on a curved or folded surface of computing device 800. Furthermore, display screen 805 may be configured as a non-rectangular irregular shape, i.e., a non-rectangular screen. Display screen 805 may be made of materials such as LCD (Liquid Crystal Display) or OLED (Organic Light-Emitting Diode).

[0098] The camera assembly 806 is used to acquire images or videos. Optionally, the camera assembly 806 includes a front-facing camera and a rear-facing camera. Typically, the front-facing camera is located on the front panel of the terminal, and the rear-facing camera is located on the back of the terminal. In some embodiments, there are at least two rear-facing cameras, which are any one of a main camera, a depth-sensing camera, a wide-angle camera, and a telephoto camera, to achieve background blurring by fusion of the main camera and the depth-sensing camera, panoramic shooting by fusion of the main camera and the wide-angle camera, VR (Virtual Reality) shooting, or other fusion shooting functions. In some embodiments, the camera assembly 806 may also include a flash. The flash can be a single-color temperature flash or a dual-color temperature flash. A dual-color temperature flash refers to a combination of a warm-light flash and a cool-light flash, which can be used for light compensation at different color temperatures.

[0099] The audio circuit 807 may include a microphone and a speaker. The microphone is used to collect sound waves from the user and the environment, converting the sound waves into electrical signals that are input to the processor 801 for processing, or input to the radio frequency circuit 804 for voice communication. For stereo sound acquisition or noise reduction purposes, multiple microphones may be used, each located at a different part of the computing device 800. The microphone may also be an array microphone or an omnidirectional microphone. The speaker is used to convert electrical signals from the processor 801 or the radio frequency circuit 804 into sound waves. The speaker may be a conventional diaphragm speaker or a piezoelectric ceramic speaker. When the speaker is a piezoelectric ceramic speaker, it can convert electrical signals not only into audible sound waves but also into inaudible sound waves for purposes such as distance measurement. In some embodiments, the audio circuit 807 may also include a headphone jack.

[0100] The positioning component 808 is used to determine the current geographic location of the computing device 800 in order to enable navigation or LBS (Location Based Service). The positioning component 808 can be a positioning component based on GPS (Global Positioning System), BeiDou system, or Galileo system.

[0101] Power supply 809 is used to supply power to the various components in computing device 800. Power supply 809 can be alternating current, direct current, a disposable battery, or a rechargeable battery. When power supply 809 includes a rechargeable battery, the rechargeable battery can be a wired rechargeable battery or a wireless rechargeable battery. A wired rechargeable battery is a battery that is charged via a wired line, and a wireless rechargeable battery is a battery that is charged via a wireless coil. The rechargeable battery can also be used to support fast charging technology.

[0102] In some embodiments, the computing device 800 further includes one or more sensors 810. The one or more sensors 810 include, but are not limited to, an accelerometer 811, a gyroscope 812, a pressure sensor 813, a fingerprint sensor 814, an optical sensor 815, and a proximity sensor 816.

[0103] Accelerometer 811 can detect the magnitude of acceleration along the three coordinate axes of a coordinate system established by computing device 800. For example, accelerometer 811 can be used to detect the components of gravitational acceleration along the three coordinate axes. Processor 801 can control display screen 805 to display the user interface in either a landscape or portrait view based on the gravitational acceleration signal acquired by accelerometer 811. Accelerometer 811 can also be used for games or for acquiring user motion data.

[0104] The gyroscope sensor 812 can detect the orientation and rotation angle of the computing device 800. The gyroscope sensor 812, in conjunction with the accelerometer sensor 811, can collect 3D motion data from the user on the computing device 800. Based on the data collected by the gyroscope sensor 812, the processor 801 can perform the following functions: motion sensing (e.g., changing the UI based on the user's tilt), image stabilization during shooting, game control, and inertial navigation.

[0105] The pressure sensor 813 can be disposed on the side bezel of the computing device 800 and / or on the lower layer of the display screen 805. When the pressure sensor 813 is disposed on the side bezel of the computing device 800, it can detect the user's grip signal on the computing device 800, and the processor 801 can perform left / right hand recognition or quick operation based on the grip signal collected by the pressure sensor 813. When the pressure sensor 813 is disposed on the lower layer of the display screen 805, the processor 801 can control the operable controls on the UI interface based on the user's pressure operation on the display screen 805. The operable controls include at least one of button controls, scroll bar controls, icon controls, and menu controls.

[0106] The fingerprint sensor 814 is used to collect a user's fingerprint. The processor 801 identifies the user based on the fingerprint collected by the fingerprint sensor 814, or vice versa. When the user's identity is verified as trusted, the processor 801 authorizes the user to perform relevant sensitive operations, including unlocking the screen, viewing encrypted information, downloading software, making payments, and changing settings. The fingerprint sensor 814 can be located on the front, back, or side of the computing device 800. When the computing device 800 has physical buttons or a manufacturer's logo, the fingerprint sensor 814 can be integrated with the physical buttons or manufacturer's logo.

[0107] An optical sensor 815 is used to collect ambient light intensity. In one embodiment, the processor 801 can control the display brightness of the display screen 805 based on the ambient light intensity collected by the optical sensor 815. Specifically, when the ambient light intensity is high, the display brightness of the display screen 805 is increased; when the ambient light intensity is low, the display brightness of the display screen 805 is decreased. In another embodiment, the processor 801 can also dynamically adjust the shooting parameters of the camera assembly 806 based on the ambient light intensity collected by the optical sensor 815.

[0108] A proximity sensor 816, also known as a distance sensor, is typically located on the front panel of the computing device 800. The proximity sensor 816 is used to detect the distance between the user and the front of the computing device 800. In one embodiment, when the proximity sensor 816 detects that the distance between the user and the front of the computing device 800 is gradually decreasing, the processor 801 controls the display screen 805 to switch from a screen-on state to a screen-off state; when the proximity sensor 816 detects that the distance between the user and the front of the computing device 800 is gradually increasing, the processor 801 controls the display screen 805 to switch from a screen-off state to a screen-on state.

[0109] Those skilled in the art will understand that Figure 6 The structure shown does not constitute a limitation on the computing device 800, and may include more or fewer components than shown, or combine certain components, or use different component arrangements.

[0110] In an exemplary embodiment, a computer-readable storage medium is also provided, such as a memory including instructions that can be executed by a processor in a terminal to perform the audio processing method described above. This computer-readable storage medium may be non-transitory. For example, the computer-readable storage medium may be ROM (Read-Only Memory), RAM (Random Access Memory), CD-ROM (Compact Disc Read-Only Memory), magnetic tape, floppy disk, and optical data storage devices, etc.

[0111] It should be noted that all information (including but not limited to user device information, user personal information, etc.), data (including but not limited to data used for analysis, stored data, displayed data, etc.), and signals (including but not limited to signals transmitted between user terminals and other devices) involved in this application have been authorized by the user or fully authorized by all parties, and the collection, use, and processing of related data must comply with the relevant laws, regulations, and standards of the relevant countries and regions. For example, all audio and text data involved in this application were obtained with full authorization.

[0112] Those skilled in the art will understand that all or part of the steps of the above embodiments can be implemented by hardware or by a program instructing related hardware. The program can be stored in a computer-readable storage medium, such as a read-only memory, a disk, or an optical disk.

[0113] It should be understood that "multiple" as used herein refers to two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, or B alone. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. Furthermore, the step numbers described herein are merely illustrative of one possible execution order. In some other embodiments, the steps may not be executed in numerical order, such as two steps with different numbers being executed simultaneously, or two steps with different numbers being executed in the reverse order of the illustration. This application does not limit this.

[0114] The above description is merely an optional embodiment of this application and is not intended to limit this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the protection scope of this application.

Claims

1. An audio processing method, characterized in that, The method includes: The positive and negative audio feature vectors are input into the trained music style transfer model. The positive audio feature vector is used to represent the chords and beats of the original song and also to represent the music style of the reference song. Each time the music style transfer model generates audio features, it infers from the positive audio feature vector to obtain a first feature probability distribution; and it infers from the negative audio feature vector to obtain a second feature probability distribution. Based on the first feature probability distribution and the second feature probability distribution, the target audio feature vector of the output is determined; When the target audio feature vector does not meet the audio quality conditions, the target audio feature vector is used as a negative audio feature vector, and the process is switched to inputting the positive and negative audio feature vectors into the trained music style transfer model until the obtained target audio feature vector meets the audio quality conditions.

2. The method according to claim 1, characterized in that, The method further includes: When the target audio feature vector meets the audio quality conditions, a target song audio is generated based on the target audio feature vector, wherein the music style of the target song audio matches the music style of the reference song, and the chords and rhythms of the target song audio match the chords and rhythms of the original song.

3. The method according to claim 2, characterized in that, When the target audio feature vector meets the audio quality conditions, generating the target song audio based on the target audio feature vector includes: Determine the score of the target audio feature vector; if the score is greater than a threshold, determine that the target audio feature vector meets the audio quality conditions; and generate the target song audio based on the target audio feature vector. The step of using the target audio feature vector as a negative audio feature vector when the target audio feature vector does not meet the audio quality conditions includes: If the score is less than the threshold, the target audio feature vector is determined to not meet the audio quality conditions, and the target audio feature vector is used as a negative audio feature vector.

4. The method according to claim 1, characterized in that, The step of determining the output target audio feature vector based on the first feature probability distribution and the second feature probability distribution includes: Calculate the third feature probability distribution based on the first and second feature probability distributions; Based on the third feature probability distribution, the target audio feature vector for output is determined.

5. The method according to claim 2, characterized in that, The process of generating the target song audio based on the target audio feature vector includes: The target audio feature vector is decoded to obtain the target accompaniment audio; Obtain the vocal audio of the original song; The target accompaniment audio and the human voice audio are mixed to obtain the target song audio.

6. The method according to claim 1, characterized in that, The method further includes: Acquire the control audio signal of the original song, wherein the control audio signal is used to characterize the chords and rhythm of the original song; Obtain text information, wherein the text information includes music style indication information; Obtain the musical style features of the reference song, wherein the musical style features are used to characterize the musical style of the reference song; Based on the control audio signal, the text information, and the musical style features of the reference song, a positive audio feature vector is generated.

7. The method according to claim 6, characterized in that, The acquisition of the control audio signal of the original song includes: Extract the beat audio signal, chord audio signal, and vocal audio signal from the original song audio; The beat audio signal, chord audio signal, and vocal audio signal are mixed to obtain the control audio signal of the original song.

8. A computing device, characterized in that, The computing device includes a processor and a memory, the memory storing at least one instruction that is loaded and executed by the processor to perform the operations performed in the audio processing method as described in any one of claims 1-7.

9. A computer-readable storage medium, characterized in that, The storage medium stores at least one instruction, which is loaded and executed by a processor to perform the operation of the audio processing method as described in any one of claims 1-7.

10. A computer program product, characterized in that, The computer program product stores at least one instruction, which is loaded and executed by a processor to perform the operation performed by the audio processing method as described in any one of claims 1-7.

Images