Audio processing methods, devices, media, computing equipment and software products

Abstract

This disclosure provides an audio processing method, apparatus, medium, computing device, and program product, relating to the field of audio processing. The audio processing method includes: acquiring the timbre characteristics of dry audio; determining the timbre similarity between the dry audio and original audio based on the timbre characteristics; selecting M target audios from N original audios based on the timbre similarity; applying the mixing parameters corresponding to each target audio to the dry audio to obtain M wet audios; and then combining the M wet audios based on the timbre similarity to obtain a mixed audio. This method only requires pre-preparing N original audios and recording the mixing parameters of each original audio. It allows the application of mixing parameters from original audios with high timbre similarity to the dry audio to the mixing process of the dry audio, eliminating the need for repeated listening tests to adjust mixing parameters. This improves mixing efficiency, shortens mixing time, and meets more mixing requirements.

Description

Technical Field

[0001] The embodiments of this disclosure relate to the field of computer technology, and more specifically, the embodiments of this disclosure relate to an audio processing method, apparatus, medium, computing device, and program product. Background Technology

[0002] This section is intended to provide background or context for embodiments of this disclosure. The description herein is not intended to imply that it is prior art simply because it is included in this section.

[0003] Audio mixing is a crucial step in the audio production process, involving combining multiple audio tracks such as vocals, instruments, and sound effects to create a harmonious and balanced final audio object.

[0004] Currently, the mixing process requires the use of various audio effects such as compressors, equalizers, reverb units, and push-pull attenuators for adjusting volume to process the audio to be mixed. It mainly relies on manual listening to continuously adjust the parameters of the audio effects in order to improve the audio quality through mixing.

[0005] However, mixing is technically challenging and usually requires experienced mixing engineers to operate it themselves, resulting in low mixing efficiency and difficulty in meeting more mixing needs. Summary of the Invention

[0006] This disclosure provides an audio processing method, apparatus, medium, computing device, and program product to meet more mixing needs.

[0007] In a first aspect of this disclosure, an audio processing method is provided, comprising:

[0008] To obtain the timbre characteristics of dry audio;

[0009] For each of the N preset original audio files, the timbre similarity between the dry audio file and the original audio file is determined based on the timbre characteristics of the original audio file and the timbre characteristics of the dry audio file. Based on the timbre similarity, M original audio files are selected as target audio files from the N original audio files. N and M are both positive integers, and N is greater than or equal to M. The original audio files correspond to preset mixing parameters.

[0010] For each of the M target audios, the dry audio is mixed according to the mixing parameters corresponding to the target audio to obtain the wet audio corresponding to the target audio.

[0011] Based on the timbre similarity between each target audio and the dry audio, the M wet audios are combined to obtain the mixed audio corresponding to the dry audio.

[0012] In another embodiment of this disclosure, based on the timbre similarity between each target audio and the dry audio, M wet audio samples are combined to obtain a mixed audio corresponding to the dry audio, including:

[0013] For each target audio, a weight value corresponding to the target audio is determined based on the timbre similarity between the target audio and the dry audio.

[0014] The M wet audio samples are weighted and combined according to the weight value of each target audio sample to obtain the mixed audio corresponding to the dry audio sample.

[0015] In another embodiment of this disclosure, determining the weight value corresponding to the target audio based on the timbre similarity between the target audio and the dry audio includes:

[0016] The timbre similarity of the target audio is mapped to a weight value within a preset range.

[0017] In another embodiment of this disclosure, mixing the dry audio according to the mixing parameters corresponding to the target audio includes:

[0018] Adjust the speed-related parameters in the mixing parameters according to the audio speed of the dry audio, and / or adjust the loudness-related parameters in the mixing parameters according to the loudness of the dry audio;

[0019] The dry audio is mixed using the adjusted mixing parameters to obtain the corresponding wet audio.

[0020] In another embodiment of this disclosure, the mixing parameters include reverberation time, and adjusting the speed-related parameters in the mixing parameters according to the audio velocity of the dry audio includes:

[0021] If the audio velocity of the dry audio is greater than or equal to a preset velocity threshold, the reverberation time is shortened;

[0022] If the audio velocity of the dry audio is less than the velocity threshold, the reverberation time is increased.

[0023] In another embodiment of this disclosure, the mixing parameters include a compressor threshold, and adjusting the loudness-related parameters in the mixing parameters according to the loudness of the dry audio includes:

[0024] If the loudness of the dry audio is greater than or equal to a preset loudness threshold, the compressor threshold is increased;

[0025] If the loudness of the dry audio is less than the loudness threshold, the compressor threshold is reduced.

[0026] In another embodiment of this disclosure, the mixing parameters include effects parameters, and the mixing parameters corresponding to the original audio are obtained in the following manner:

[0027] The original audio is tuned using preset effects, which include one or more of the following: pre-compressor, parametric equalizer, dynamic equalizer, multi-band exciter, post-compressor, multi-band compressor, stereo enhancer, delay, reverb, and volume adjuster.

[0028] If the adjustment result meets the preset conditions, the effect parameters corresponding to the adjustment result are recorded as the mixing parameters corresponding to the original audio.

[0029] In another embodiment of this disclosure, the original audio is generated by a preset timbre model, which is trained based on pre-labeled dry audio data.

[0030] In another embodiment of this disclosure, obtaining the timbre characteristics of the dry audio signal includes:

[0031] Dry audio is input into a pre-trained deep learning model so that the deep learning model outputs the timbre features corresponding to the dry audio. The deep learning model is trained based on pre-labeled audio samples, which are Mel spectra generated based on preset audio.

[0032] In another embodiment of this disclosure, the audio processing method further includes:

[0033] Obtain the energy characteristics of the mixed audio and the masking characteristics of the corresponding accompaniment audio;

[0034] The volume ratio of the mixed audio to the accompaniment audio is adjusted according to the energy characteristics and the masking characteristics to obtain a volume-balanced mixed audio.

[0035] In another embodiment of this disclosure, the audio processing method further includes:

[0036] Obtain the loudness of the audio track in the mixed audio;

[0037] If the loudness of the audio track does not fall within the preset loudness range, the mixed audio is adjusted using a preset effect to make the loudness of the adjusted mixed audio track fall within the loudness range.

[0038] In a second aspect of this disclosure, an audio processing apparatus is provided, comprising:

[0039] The acquisition module is used to acquire the timbre characteristics of dry audio.

[0040] The similarity determination module is used to determine the timbre similarity between the dry audio and the original audio for each of the preset N original audios, based on the timbre characteristics of the original audio and the timbre characteristics of the dry audio, and select M original audios as target audios from the N original audios based on the timbre similarity, where N and M are both positive integers, and N is greater than or equal to M, and the original audios correspond to preset mixing parameters;

[0041] The wet audio processing module is used to perform mixing processing on the dry audio according to the mixing parameters corresponding to the target audio for each of the M target audios, so as to obtain the wet audio corresponding to the target audio.

[0042] The mixing and combining module is used to combine M wet audio files based on the timbre similarity between each target audio file and the dry audio file to obtain the mixed audio file corresponding to the dry audio file.

[0043] In a third aspect of the present disclosure, a computer-readable storage medium is provided that stores computer-executable instructions, which, when executed by a processor, implement the method as described in any one of the first aspects.

[0044] In a fourth aspect of this disclosure, a computing device is provided, comprising: at least one processor;

[0045] and a memory communicatively connected to the at least one processor;

[0046] The memory stores instructions executable by the at least one processor, which, when executed by the at least one processor, cause the computing device to perform the method as described in any one of the first aspects.

[0047] In a fifth aspect of this disclosure, a computer program product is provided, including a computer program that, when executed by a processor, implements the method as described in any of the first aspects.

[0048] According to the audio processing method, apparatus, medium, computing device, and program product of this disclosure, by acquiring the timbre characteristics of dry audio, the timbre similarity between the dry audio and the original audio can be determined based on the timbre characteristics of the dry audio and the original audio. Based on the timbre similarity, M target audios can be selected from N original audios. The mixing parameters corresponding to each target audio are applied to the dry audio to obtain M wet audios. Then, based on the timbre similarity, the M wet audios are combined to obtain the mixed audio of the dry audio. This method only requires pre-preparing N original audios and recording the mixing parameters of each original audio after mixing. Furthermore, the mixing parameters of the original audio with high timbre similarity to the dry audio can be applied to the mixing process of the dry audio, eliminating the need for repeated listening to the dry audio to adjust the mixing parameters. This improves mixing efficiency, shortens mixing time, and meets more mixing needs. Attached Figure Description

[0049] The above and other objects, features, and advantages of this disclosure will become readily apparent from the following detailed description of exemplary embodiments, taken in conjunction with the accompanying drawings. Several embodiments of this disclosure are illustrated in the drawings by way of example and not limitation, in which:

[0050] Figure 1 The illustration schematically depicts an application scenario of the audio processing method according to an embodiment of the present disclosure;

[0051] Figure 2 A schematic flowchart of an audio processing method according to an embodiment of the present disclosure is shown.

[0052] Figure 3 A schematic flowchart of an audio processing method according to another embodiment of the present disclosure is shown;

[0053] Figure 4 A schematic flowchart of a wet sound audio combination method according to an embodiment of the present disclosure is shown;

[0054] Figure 5 A schematic flowchart of a method for obtaining raw audio mixing parameters according to an embodiment of the present disclosure is shown.

[0055] Figure 6 A schematic diagram of the structure of an effects chain according to an embodiment of the present disclosure is shown.

[0056] Figure 7 A schematic diagram of a storage medium according to an embodiment of the present disclosure is shown.

[0057] Figure 8 A schematic diagram of the structure of an audio processing apparatus according to an embodiment of the present disclosure is shown.

[0058] Figure 9 A schematic diagram of a computing device according to an embodiment of the present disclosure is shown.

[0059] In the accompanying drawings, the same or corresponding reference numerals indicate the same or corresponding parts. Detailed Implementation

[0060] The principles and spirit of this disclosure will now be described with reference to several exemplary embodiments. It should be understood that these embodiments are given merely to enable those skilled in the art to better understand and implement this disclosure, and are not intended to limit the scope of this disclosure in any way. Rather, these embodiments are provided to make this disclosure more thorough and complete, and to fully convey the scope of this disclosure to those skilled in the art.

[0061] Those skilled in the art will recognize that embodiments of this disclosure can be implemented as a system, apparatus, device, method, or computer program product. Therefore, this disclosure can be specifically implemented in the following forms: entirely hardware, entirely software (including firmware, resident software, microcode, etc.), or a combination of hardware and software.

[0062] According to embodiments of this disclosure, an audio processing method, apparatus, medium, computing device, and program product are proposed.

[0063] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this disclosure are all information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, use and processing of the relevant data must comply with relevant laws, regulations and standards, and corresponding operation entry points are provided for users to choose to authorize or refuse.

[0064] Furthermore, the number of any elements in the accompanying drawings is for illustrative purposes only and not for limitation, and any naming is for distinction only and has no limiting meaning.

[0065] The principles and spirit of this disclosure will be explained in detail below with reference to several representative embodiments. Invention Overview

[0067] Audio mixing is a processing technique that involves using audio effects to enhance vocals and accompaniment, making the sound more pleasing, harmonious, and integrated. The mixing process typically utilizes various audio effects such as compressors, equalizers, and reverb units. Manual mixing requires a certain level of music theory knowledge and experience using audio effects, making it a relatively challenging skill.

[0068] However, the number of experienced mixing engineers is relatively small, making it difficult to meet the demand for mixing. Especially with the development of technologies such as AIGC (Artificial Intelligence Generated Content), the number of AI-generated audio works is increasing, and these works also require mixing to improve their quality. Therefore, there is an urgent need for an audio processing method that can achieve automatic mixing to meet these audio mixing requirements.

[0069] Through research and experimentation, the inventors have developed a method to pre-prepare several original audio files with different timbres. Each original audio file can be manually mixed, and the mixing parameters corresponding to the final mixing result can be recorded. For the dry audio file requiring mixing, multiple original audio files with high timbre similarity can be selected based on the timbre similarity between the dry audio file and the original audio files. The mixing parameters of these selected original audio files can then be used to mix the dry audio file separately. The timbre similarity is then mapped as weights, and the results of the dry audio file mixes are weighted and combined to obtain the final mixing result. In this way, only a limited number of pre-prepared original audio files need to be manually mixed, and the mixing parameters used for mixing need to be recorded. Then, the corresponding mixing parameters are selected based on the similarity between the dry audio file and the original audio files and applied to the dry audio file. This eliminates the need for complex manual mixing of the dry audio file, thereby achieving automated mixing of dry audio files, improving mixing efficiency, and meeting more mixing needs. The dry audio file can be unmixed human vocal audio, which can be audio sung by a human or audio sung by AI.

[0070] After introducing the basic principles of this disclosure, various non-limiting embodiments of this disclosure will be described in detail below.

[0071] Application Scenarios Overview

[0072] First refer to Figure 1 , Figure 1 This is a schematic diagram illustrating one application scenario provided in this disclosure. For example... Figure 1 As shown, terminal 102 can communicate with server 101 via a network. A data storage system can store the data that server 101 needs to process. The data storage system can be integrated onto server 101 or located in the cloud or on other network servers. Terminal 102 can be, but is not limited to, various personal computers, laptops, smartphones, tablets, IoT devices, and portable wearable devices. IoT devices can include smart speakers, smart TVs, smart air conditioners, smart in-vehicle devices, etc. Portable wearable devices can include smartwatches, smart bracelets, head-mounted devices, etc. Server 101 can be implemented using a standalone server or a server cluster consisting of multiple servers.

[0073] For scenarios where dry audio needs to be mixed, multiple original audio files can be prepared in advance, and the original audio files can be manually mixed and the mixing parameters recorded. These original audio files and mixing parameters can be stored in server 101, terminal 102 or data storage system. After obtaining the dry audio files to be mixed, server 101 or terminal 102 can mix the dry audio files based on the stored original audio files and mixing parameters.

[0074] The application scenarios mentioned above are only some examples. Those skilled in the art can extend the applications based on the audio processing process. This disclosure does not impose specific limitations on these applications.

[0075] Exemplary methods

[0076] The following is combined Figure 1 Application scenarios, refer to Figures 2 to 6 This document describes an audio processing method according to exemplary embodiments of the present disclosure. It should be noted that the above application scenarios are shown only to facilitate understanding of the spirit and principles of the present disclosure, and the embodiments of the present disclosure are not limited in any way. Rather, the embodiments of the present disclosure can be applied to any applicable scenario.

[0077] Figure 2 This is a schematic flowchart of an audio processing method according to an embodiment of this disclosure. Figure 2 As shown, the method may include:

[0078] Step S201: Obtain the timbre characteristics of the dry audio.

[0079] The audio processing method of this disclosure can be applied to... Figure 1 Server 101 or terminal 102.

[0080] Among them, timbre characteristics can be parameters that characterize the timbre of a human voice. For example, dry audio recordings of bel canto singing usually have a full and rich timbre, while dry audio recordings of pop singing usually have a light and gentle timbre.

[0081] Optionally, the dry audio is input into a pre-trained deep learning model, causing the deep learning model to output the timbre features corresponding to the dry audio. The deep learning model is trained based on pre-labeled audio samples. The audio samples can be Mel spectrograms generated based on preset audio.

[0082] In some possible implementations, the acquisition of timbre features may also include: obtaining the spectrum of the dry audio through Fourier transform or other methods, and selecting some frequency domain features as timbre features.

[0083] Step S202: For each of the preset N original audio files, determine the timbre similarity between the dry audio file and the original audio file based on the timbre characteristics of the original audio file and the dry audio file, and select M original audio files from the N original audio files as target audio files based on the timbre similarity.

[0084] Where N and M are both positive integers, and N is greater than or equal to M, the original audio corresponds to preset mixing parameters. The mixing parameters may include parameters of audio effects such as pre-compressor, parametric equalizer, dynamic equalizer, multi-band exciter, post-compressor, stereo enhancer, delay, reverb, and volume control.

[0085] Optionally, based on N preset original audios, the similarity between the timbre features of each original audio and the timbre features of the dry audio can be calculated to obtain N timbre similarities. Then, M target audios are selected from the N original audios through a preset similarity threshold.

[0086] In one example, such as Figure 3 As shown, N original audio files include audio 1, audio 2, ..., audio N. If the similarity between the timbre features of audio 1 and the timbre features of the dry audio is greater than or equal to a preset first threshold, and the timbre similarity between the timbre features of audio 2 and the timbre features of the dry audio is greater than or equal to the first threshold, and the timbre similarity between the timbre features of other audio files and the timbre features of the dry audio is less than the first threshold, then M is 2, and the M target audio files include audio 1 and audio 2.

[0087] In some possible implementations, after obtaining the similarity between the timbre features of each original audio and the timbre features of the dry audio, the N original audios can be sorted in descending order of similarity, and the first M original audios can be taken as the target audio.

[0088] In some possible implementations, the timbre features of the original audio and the dry audio can be extracted in the same way. For example, a feature extraction model can be pre-trained based on audio samples, and the trained extraction model can be used to extract the timbre features of the original audio and the dry audio respectively.

[0089] In some possible implementations, the original audio can be obtained from a human singing, and different original audio will have different timbres.

[0090] In some possible implementations, the original audio can be generated using a pre-defined timbre model, which is trained based on pre-labeled dry audio data. The timbre model can be an AI model based on artificial intelligence technologies such as AIGC.

[0091] Step S203: For each of the M target audios, perform mixing processing on the dry audio according to the mixing parameters corresponding to the target audio to obtain the wet audio corresponding to the target audio.

[0092] Optionally, for each target audio, the parameters of the target audio when using various audio effects during the mixing process can be obtained, and the dry audio can be processed using the corresponding audio effects according to these parameters to obtain the wet audio.

[0093] In one example, such as Figure 3 As shown, multiple target audios include audio1, audio2, ..., audioM. Taking M=2 as an example, the target audios include audio1 and audio2. During the mixing process, audio1 uses multiple audio effects such as a pre-compressor, parametric equalizer, and dynamic equalizer. The mixing engineer continuously adjusts the parameters of the audio effects by listening to the effect of audio1 under the action of the audio effects. The audio effect parameters that meet the requirements after the mixing engineer's adjustment are recorded as mixing parameter 1. Correspondingly, the audio effect parameters obtained after mixing audio2 are recorded as mixing parameter 2. For dry audio, mixing parameter 1 and mixing parameter 2 can be used as audio effect parameters to apply audio effects such as pre-compressor, parametric equalizer, and dynamic equalizer to the dry audio, resulting in two different wet audios. One of these wet audios is the result of mixing the dry audio using the mixing parameters of audio1 (the wet audio corresponding to audio1), and the other is the result of mixing the dry audio using the mixing parameters of audio2 (the wet audio corresponding to audio2).

[0094] Step S204: Based on the timbre similarity between each target audio and the dry audio, combine the M wet audios to obtain the mixed audio corresponding to the dry audio.

[0095] Optionally, audio processing software is used to align the tracks of the M wet audio files on the time axis. Based on the timbre similarity between the target audio corresponding to the wet audio file and the dry audio file, the volume of the wet audio file tracks is adjusted. All the adjusted tracks are then superimposed to obtain the final mixed audio tracks.

[0096] In some possible implementations, the timbre similarity between the target audio corresponding to the wet audio and the dry audio can be positively correlated with the volume of the wet audio track. For example, if the target audio corresponding to the wet audio is audio 1, the greater the timbre similarity between audio 1 and the dry audio, the greater the volume of the adjusted wet audio.

[0097] In the above embodiments, by acquiring the timbre characteristics of the dry audio, the timbre similarity between the dry audio and the original audio can be determined based on the timbre characteristics of the dry audio and the original audio. Based on the timbre similarity, M target audios can be selected from N original audios. The mixing parameters corresponding to each target audio are then applied to the dry audio to obtain M wet audios. Finally, the M wet audios are combined based on the timbre similarity to obtain the mixed audio of the dry audio. This method only requires preparing N original audios in advance and recording the mixing parameters of each original audio after mixing. Furthermore, the mixing parameters of the original audio with high timbre similarity to the dry audio can be applied to the mixing process of the dry audio, eliminating the need for repeated listening to and adjusting of the mixing parameters on the dry audio. This improves mixing efficiency, shortens mixing time, and meets more mixing needs.

[0098] In one embodiment, such as Figure 4 As shown, based on the timbre similarity between each target audio and the dry audio, M wet audio samples are combined to obtain the mixed audio corresponding to the dry audio, including:

[0099] Step S401: For each target audio, determine the weight value corresponding to the target audio based on the timbre similarity between the target audio and the dry audio.

[0100] The timbre similarity between the target audio and the dry audio can be calculated as a cosine similarity based on the timbre features of the target audio and the dry audio.

[0101] In some possible implementations, the timbre similarity between the target audio and the dry audio can be directly used as the weight value corresponding to the target audio.

[0102] In some possible implementations, the timbre similarity of the target audio can be mapped to a weight value within a preset range. The numerical range of the timbre similarity between the target audio and the dry audio is [-1, 1]. The values ​​in this range can be mapped to values ​​in the range [0, 1], and the mapped timbre similarity values ​​can be used as weight values.

[0103] For example, the mapping relationship between timbre similarity and weight value can be established according to the following formulas (1) and (2):

[0104]

[0105] 1=∑ M weight(k) formula (2)

[0106] In the above formula, weight(k) can represent the weight value corresponding to the kth target audio, and similarity(k) can represent the timbre similarity between the kth target audio and the dry audio.

[0107] Step S402: The M wet audio samples are weighted and combined according to the weight value of each target audio sample to obtain the mixed audio corresponding to the dry audio sample.

[0108] Optionally, M wet audio files are input into audio processing software, and the tracks of all wet audio files are aligned on the time axis. The tracks of the wet audio files corresponding to each target audio file are then superimposed according to the weight value of each target audio file to obtain the audio tracks of the mixed audio.

[0109] In one embodiment, mixing the dry audio according to the mixing parameters corresponding to the target audio includes:

[0110] Adjust the speed-related parameters in the mixing parameters according to the audio speed of the dry audio; then mix the dry audio using the adjusted mixing parameters to obtain the corresponding wet audio.

[0111] The mixing parameters may include reverberation time.

[0112] Through research, the inventors discovered that when using mixing parameters of the target audio to mix dry audio, the impact of audio velocity on the mixing needs to be considered. The audio velocity of the target audio may differ from that of the dry audio. Some mixing parameters of the target audio are determined based on its audio velocity; directly applying these mixing parameters to the dry audio may affect the mixing effect.

[0113] Therefore, the inventors conceived of adjusting parameters in the mixing, such as reverberation time, which are affected by audio speed, based on the speed difference between the target audio and the dry audio, so that the adjusted parameters are more suitable for the audio speed of the dry audio. For example, for fast songs, the reverberation time needs to be appropriately shortened; for slow songs, the reverberation time needs to be appropriately lengthened.

[0114] In some possible implementations, adjusting the speed-related parameters in the mixing parameters based on the audio speed of the dry audio may include: shortening the reverberation time for audio frequencies greater than or equal to a preset speed threshold; and increasing the reverberation time for audio frequencies less than the speed threshold. The speed threshold can be determined based on the audio speed of the target audio.

[0115] In the above embodiments, a speed threshold can be determined based on the audio speed of the target audio, and mixing parameters such as reverberation time can be adjusted according to the relationship between the audio speed of the dry audio and the speed threshold, so that the adjusted parameters are more suitable for the dry audio, thereby enhancing the mixing effect.

[0116] In one embodiment, mixing the dry audio according to the mixing parameters corresponding to the target audio includes:

[0117] Adjust the loudness-related parameters in the mixing parameters according to the loudness of the dry audio; then mix the dry audio using the adjusted mixing parameters to obtain the corresponding wet audio.

[0118] The mixing parameters may include compressor thresholds.

[0119] Through research, the inventors discovered that when using the mixing parameters of the target audio to mix dry audio, the impact of track loudness on the mix must also be considered. The track loudness of the target audio may differ from that of the dry audio. Some mixing parameters of the target audio (such as parameters of audio effects processors like reverb and multi-band compressors) are determined based on the track loudness of the target audio. Directly applying these mixing parameters to the dry audio may affect the mixing effect of the dry audio.

[0120] Therefore, the inventors conceived of first adjusting parameters in the mixing parameters that are affected by the track loudness, such as the compressor threshold, based on the difference in loudness between the target audio and the dry audio tracks, so that the adjusted parameters are more suitable for the track loudness of the dry audio. For example, for low-level tracks, the compressor threshold needs to be appropriately lowered; for high-level tracks, the compressor threshold needs to be appropriately increased.

[0121] In some possible implementations, adjusting the loudness-related parameters in the mixing parameters based on the loudness of the dry audio can include: increasing the compressor threshold if the loudness of some audio samples is greater than or equal to a preset loudness threshold; and decreasing the compressor threshold if the loudness of some audio samples is less than the loudness threshold. The loudness threshold can be determined based on the loudness of the target audio track.

[0122] In the above embodiments, a loudness threshold can be determined based on the loudness of the target audio track, and mixing parameters such as the compressor threshold can be adjusted according to the relationship between the loudness of the dry audio track and the loudness threshold, so that the adjusted parameters are more suitable for the dry audio, thereby enhancing the mixing effect.

[0123] In one embodiment, mixing parameters include effects parameters, such as... Figure 5 As shown, the mixing parameters corresponding to the original audio are obtained in the following way:

[0124] Step S501: Adjust the original audio using preset effects.

[0125] The effects include one or more of the following: pre-compressor, parametric equalizer, dynamic equalizer, multi-band exciter, post-compressor, stereo enhancer, delay unit, reverb unit, and volume control unit.

[0126] Optionally, the original audio is input into audio processing software, and the selected effects are attached to the original audio track. The mixing effect is evaluated through methods such as manual listening, and the parameters of the effects can be continuously adjusted based on the mixing effect.

[0127] In some possible implementations, all the effects needed for mixing and the order in which they are applied can be predefined. For example... Figure 6 As shown, an effects chain can be formed by pre-compressor, parametric equalizer, dynamic equalizer, multi-band exciter, post-compressor, multi-band compressor, stereo enhancer, delay, reverb, and fader. All effects in the effects chain can be attached to the original audio track for adjustment.

[0128] In some possible implementations, it is also possible to determine the accompaniment that is suitable for the original audio, align the original audio and accompaniment tracks using audio processing software, and conduct a listening test based on the simultaneous playback of the original audio and accompaniment, adjusting the parameters of the effects unit according to the listening test results.

[0129] Step S502: If the adjustment result meets the preset conditions, record the effect parameters corresponding to the adjustment result as the mixing parameters corresponding to the original audio.

[0130] Optionally, if the sound quality of the original audio after adjustment meets the requirements, the parameters of the effects used in the adjustment can be recorded. For N original videos, N sets of parameters can be obtained.

[0131] In some possible implementations, if the tuning results meet preset conditions, the audio tempo and track loudness of the original audio under those conditions can also be recorded. This method allows recording the tempo and loudness at which the mixing parameters of the original audio were tuned. Thus, for some effects parameters related to audio tempo and loudness, the effects parameters can be flexibly adjusted based on the recorded tempo and loudness of the original audio and the tempo and loudness of the dry audio to be mixed, making the adjusted parameters more suitable for mixing the dry audio.

[0132] In one embodiment, the audio processing method may further include:

[0133] Obtain the energy characteristics of the mixed audio and the masking characteristics of the corresponding accompaniment audio; adjust the volume ratio of the mixed audio and the accompaniment audio according to the energy characteristics and masking characteristics to obtain a mixed audio with balanced volume.

[0134] The energy characteristics can be calculated using psychoacoustic model two in standard ISO / IEC 11172-3. The masking characteristics can be calculated using the masking threshold curve of each frame of the accompaniment audio, which can also be calculated using psychoacoustic model two in standard ISO / IEC 11172-3. The masking threshold, measured in frequency bands, is a numerical value that measures the ability of sound A to mask sound B, and can be calculated using a psychoacoustic model.

[0135] Optionally, the energy features of each frame of the mixed audio and the masking features of each frame of the accompaniment audio can be obtained. The sum of the energy features of each frame of the mixed audio and the sum of the masking features of each frame of the accompaniment audio can be calculated. Based on the sum of the energy features and the sum of the masking features, combined with a preset mapping relationship, the optimal volume ratio between the mixed audio and the accompaniment audio can be determined. Superimposing the mixed audio and the accompaniment audio according to this ratio yields the final volume-balanced mixed audio. The preset mapping relationship can be based on several audio samples, each with a good volume ratio. By analyzing the sum of the energy features of the human voice and the sum of the masking features of the accompaniment in the audio samples, a mapping relationship between the sum of the energy features and the sum of the masking features and the volume ratio can be established.

[0136] In one embodiment, the audio processing method may further include:

[0137] Obtain the loudness of the audio track in the mixed audio; if the loudness of the audio track does not fall within the preset loudness range, adjust the mixed audio using preset effects to make the loudness of the adjusted mixed audio track fall within the loudness range.

[0138] The loudness range can be determined based on audio distribution standards.

[0139] For example, some song release standards require a track loudness of [-12 LUFS, -8 LUFS]. For the above mixed audio, the Maximizer effect can be used to increase the track loudness to -9 LUFS. The calculation process for the two parameters of Maximizer is as follows:

[0140] Threshold Maximizer =LUFS mixed -(-9); Ceiling Maximizer = -0.2.

[0141] In this way, the loudness of the mixed audio can be increased to meet the requirements of audio release standards without distorting the audio waveform.

[0142] In one embodiment, the audio processing method may further include:

[0143] Add new original audio and mix it by adjusting effects parameters to obtain the corresponding mixing parameters.

[0144] The new original audio can be sung by a human or generated by a new AI model.

[0145] In the above embodiments, by adding the original audio, the range of parameters used in dry audio mixing can be expanded, thereby enhancing the overall mixing effect.

[0146] Exemplary media

[0147] After introducing the methods of exemplary embodiments of this disclosure, the following references are made. Figure 7 The storage medium of the exemplary embodiments of this disclosure will be described.

[0148] refer to Figure 7 As shown, the storage medium 70 stores a program product for implementing the above-described method according to embodiments of the present disclosure. This program product may be a portable compact disc read-only memory (CD-ROM) and includes program code, and can run on a terminal device, such as a personal computer. However, the program product of the present disclosure is not limited thereto.

[0149] The program product may employ any combination of one or more readable media. A readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of readable storage media (a non-exhaustive list) include: an electrical connection having one or more wires, a portable disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0150] A readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, carrying readable program code. This propagated data signal may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A readable signal medium may also be any readable medium other than a readable storage medium.

[0151] Program code for performing the operations disclosed herein can be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java and C++, and conventional procedural programming languages ​​such as C or similar languages. The program code can execute entirely on the user's computing device, partially on the user's computing device, partially on a remote computing device, or entirely on a remote computing device or server. In cases involving remote computing devices, the remote computing devices can be connected to the user's computing device via any type of network, including a local area network (LAN) or a wide area network (WAN).

[0152] Exemplary device

[0153] Having introduced the medium of exemplary embodiments of this disclosure, the following references are made to... Figure 8 The audio processing apparatus of the exemplary embodiments of this disclosure is described to implement the method in any of the above method embodiments. The implementation principle and technical effect are similar, and will not be repeated here.

[0154] like Figure 8 As shown, the audio processing device 800 may include:

[0155] The acquisition module 801 is used to acquire the timbre characteristics of dry audio.

[0156] The similarity determination module 802 is used to determine the timbre similarity between the dry audio and the original audio for each of the preset N original audios based on the timbre characteristics of the original audio and the timbre characteristics of the dry audio, and select M original audios as target audios from the N original audios based on the timbre similarity, where N and M are both positive integers, and N is greater than or equal to M, and the original audios correspond to preset mixing parameters.

[0157] The wet audio processing module 803 is used to perform mixing processing on the dry audio for each of the M target audios according to the mixing parameters corresponding to the target audio, so as to obtain the wet audio corresponding to the target audio.

[0158] The mixing module 804 is used to combine M wet audio files based on the timbre similarity between each target audio file and the dry audio file to obtain the mixed audio file corresponding to the dry audio file.

[0159] In yet another embodiment of this disclosure, the mixing module 804 may include:

[0160] The weight determination unit can be used to determine the weight value corresponding to each target audio based on the timbre similarity between the target audio and the dry audio.

[0161] The wet audio combination unit can be used to weight and combine M wet audios according to the weight value of each target audio to obtain the mixed audio corresponding to the dry audio.

[0162] In yet another embodiment of this disclosure, the weight determination unit is further configured to map the timbre similarity corresponding to the target audio to a weight value within a preset range.

[0163] In another embodiment of this disclosure, the wet audio processing module 803 is further configured to: adjust the speed-related parameters in the mixing parameters according to the audio speed of the dry audio, and / or adjust the loudness-related parameters in the mixing parameters according to the loudness of the dry audio; and perform mixing processing on the dry audio using the adjusted mixing parameters to obtain the corresponding wet audio.

[0164] In another embodiment of this disclosure, the wet sound processing module 803 is further configured to: shorten the reverberation time if the audio velocity of the dry audio is greater than or equal to a preset velocity threshold; and increase the reverberation time if the audio velocity of the dry audio is less than the velocity threshold.

[0165] In another embodiment of this disclosure, the wet sound processing module 803 is further configured to: increase the compressor threshold if the loudness of the dry sound audio is greater than or equal to a preset loudness threshold; and decrease the compressor threshold if the loudness of the dry sound audio is less than the loudness threshold.

[0166] In yet another embodiment of this disclosure, the audio processing apparatus 800 may further include:

[0167] The tuning module is used to tune the original audio using preset effects, which include one or more of the following: pre-compressor, parametric equalizer, dynamic equalizer, multi-band exciter, post-compressor, multi-band compressor, stereo enhancer, delay, reverb, and volume adjuster.

[0168] The recording module is used to record the effect parameters corresponding to the tuning result as the mixing parameters corresponding to the original audio when the tuning result meets the preset conditions.

[0169] In yet another embodiment of this disclosure, the original audio is generated using a preset timbre model, which is trained based on pre-labeled dry audio data.

[0170] In another embodiment of this disclosure, the acquisition module 801 is further configured to input the dry audio into a pre-trained deep learning model, so that the deep learning model outputs the timbre features corresponding to the dry audio, wherein the deep learning model is trained based on pre-labeled audio samples, and the audio samples are Mel spectra generated based on preset audio.

[0171] In yet another embodiment of this disclosure, the audio processing apparatus 800 may further include:

[0172] The feature acquisition module is used to acquire the energy features of the mixed audio and the masking features of the accompaniment audio corresponding to the mixed audio.

[0173] A volume balancing module is used to adjust the volume ratio of the mixed audio and the accompaniment audio according to the energy characteristics and the masking characteristics, so as to obtain a volume-balanced mixed audio.

[0174] In yet another embodiment of this disclosure, the audio processing apparatus 800 may further include:

[0175] The loudness acquisition module is used to acquire the loudness of the audio track of the mixed audio.

[0176] The loudness adjustment module is used to adjust the mixed audio by means of a preset effect when the loudness of the audio track does not fall into the preset loudness range, so that the loudness of the audio track of the adjusted mixed audio falls into the loudness range.

[0177] Exemplary computing device

[0178] Having described the methods, media, and apparatus of exemplary embodiments of this disclosure, the following references... Figure 9 A computing device according to an exemplary embodiment of the present disclosure will be described.

[0179] Figure 9 The computing device 90 shown is merely an example and should not impose any limitation on the functionality and scope of use of the embodiments disclosed herein.

[0180] like Figure 9 As shown, the computing device 90 is presented in the form of a general-purpose computing device. The components of the computing device 90 may include, but are not limited to: at least one processing unit 901, at least one storage unit 902, and a bus 903 connecting different system components (including the processing unit 901 and the storage unit 902). The at least one storage unit 902 stores computer-executable instructions; the at least one processing unit 901 includes a processor that executes the computer-executable instructions to implement the methods described above.

[0181] The 903 bus includes a data bus, a control bus, and an address bus.

[0182] Storage unit 902 may include readable media in the form of volatile memory, such as random access memory (RAM) 9021 and / or cache memory 9022, and may further include readable media in the form of non-volatile memory, such as read-only memory (ROM) 9023.

[0183] Storage unit 902 may also include a program / utility 9025 having a set (at least one) of program modules 9024, such program modules 9024 including but not limited to: operating system, one or more application programs, other program modules and program data, each or some combination of these examples may include an implementation of a network environment.

[0184] The computing device 90 can also communicate with one or more external devices 904 (e.g., keyboard, pointing device, etc.). This communication can be performed via input / output (I / O) interface 905. Furthermore, the computing device 90 can also communicate with one or more networks (e.g., local area network (LAN), wide area network (WAN), and / or public networks, such as the Internet) via network adapter 906. Figure 9 As shown, network adapter 906 communicates with other modules of computing device 90 via bus 903. It should be understood that, although not shown in the figure, other hardware and / or software modules may be used in conjunction with computing device 90, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0185] It should be noted that although several units / modules or sub-units / modules of the audio processing device have been mentioned in the detailed description above, this division is merely exemplary and not mandatory. In fact, according to embodiments of this disclosure, the features and functions of two or more units / modules described above can be embodied in one unit / module. Conversely, the features and functions of one unit / module described above can be further divided and embodied by multiple units / modules.

[0186] Furthermore, although the operations of the methods disclosed herein are described in a specific order in the accompanying drawings, this does not require or imply that these operations must be performed in that specific order, or that all of the operations shown must be performed to achieve the desired result. Additionally or alternatively, certain steps may be omitted, multiple steps may be combined into one step, and / or one step may be broken down into multiple steps.

[0187] While the spirit and principles of this disclosure have been described with reference to several specific embodiments, it should be understood that this disclosure is not limited to the disclosed specific embodiments, and the division of aspects does not imply that features in these aspects cannot be combined for benefit; such division is merely for convenience of expression. This disclosure is intended to cover various modifications and equivalent arrangements included within the spirit and scope of the appended claims.

Claims

1. An audio processing method, characterized in that, include: To obtain the timbre characteristics of dry audio; For each of the N preset original audio files, the timbre similarity between the dry audio file and the original audio file is determined based on the timbre characteristics of the original audio file and the timbre characteristics of the dry audio file. Based on the timbre similarity, M original audio files are selected as target audio files from the N original audio files. N and M are both positive integers, and N is greater than or equal to M. The original audio files correspond to preset mixing parameters. For each of the M target audios, the dry audio is mixed according to the mixing parameters corresponding to the target audio to obtain the wet audio corresponding to the target audio. Based on the timbre similarity between each target audio and the dry audio, the M wet audios are combined to obtain the mixed audio corresponding to the dry audio.

2. The audio processing method according to claim 1, characterized in that, The step of combining M wet audio files based on the timbre similarity between each target audio file and the dry audio file to obtain the mixed audio file corresponding to the dry audio file includes: For each target audio, a weight value corresponding to the target audio is determined based on the timbre similarity between the target audio and the dry audio. The M wet audio samples are weighted and combined according to the weight value of each target audio sample to obtain the mixed audio corresponding to the dry audio sample.

3. The audio processing method according to claim 2, characterized in that, The step of determining the weight value corresponding to the target audio based on the timbre similarity between the target audio and the dry audio includes: The timbre similarity of the target audio is mapped to a weight value within a preset range.

4. The audio processing method according to any one of claims 1 to 3, characterized in that, The mixing process of the dry audio according to the mixing parameters corresponding to the target audio includes: Adjust the speed-related parameters in the mixing parameters according to the audio speed of the dry audio, and / or adjust the loudness-related parameters in the mixing parameters according to the loudness of the dry audio; The dry audio is mixed using the adjusted mixing parameters to obtain the corresponding wet audio.

5. The audio processing method according to claim 4, characterized in that, The mixing parameters include reverberation time, and adjusting the speed-related parameters in the mixing parameters according to the audio velocity of the dry audio includes: If the audio velocity of the dry audio is greater than or equal to a preset velocity threshold, the reverberation time is shortened; If the audio velocity of the dry audio is less than the velocity threshold, the reverberation time is increased.

6. The audio processing method according to claim 4, characterized in that, The mixing parameters include a compressor threshold, and adjusting the loudness-related parameters in the mixing parameters according to the loudness of the dry audio includes: If the loudness of the dry audio is greater than or equal to a preset loudness threshold, the compressor threshold is increased; If the loudness of the dry audio is less than the loudness threshold, the compressor threshold is reduced.

7. The audio processing method according to any one of claims 1 to 3, characterized in that, The mixing parameters include effects parameters, and the mixing parameters corresponding to the original audio are obtained in the following way: The original audio is tuned using preset effects, which include one or more of the following: pre-compressor, parametric equalizer, dynamic equalizer, multi-band exciter, post-compressor, multi-band compressor, stereo enhancer, delay, reverb, and volume adjuster. If the adjustment result meets the preset conditions, the effect parameters corresponding to the adjustment result are recorded as the mixing parameters corresponding to the original audio.

8. The audio processing method according to any one of claims 1 to 3, characterized in that, The original audio is generated through a preset timbre model, which is trained based on pre-labeled dry audio data.

9. The audio processing method according to any one of claims 1 to 3, characterized in that, The acquisition of the timbre features of the dry audio includes: Dry audio is input into a pre-trained deep learning model so that the deep learning model outputs the timbre features corresponding to the dry audio. The deep learning model is trained based on pre-labeled audio samples, which are Mel spectra generated based on preset audio.

10. The audio processing method according to any one of claims 1 to 3, characterized in that, Also includes: Obtain the energy characteristics of the mixed audio and the masking characteristics of the corresponding accompaniment audio; The volume ratio of the mixed audio to the accompaniment audio is adjusted according to the energy characteristics and the masking characteristics to obtain a volume-balanced mixed audio.

11. The audio processing method according to claim 10, characterized in that, Also includes: Obtain the loudness of the audio track in the mixed audio; If the loudness of the audio track does not fall within the preset loudness range, the mixed audio is adjusted using a preset effect to make the loudness of the adjusted mixed audio track fall within the loudness range.

12. An audio processing device, characterized in that, include: The acquisition module is used to acquire the timbre characteristics of dry audio. The similarity determination module is used to determine the timbre similarity between the dry audio and the original audio for each of the preset N original audios, based on the timbre characteristics of the original audio and the timbre characteristics of the dry audio, and select M original audios as target audios from the N original audios based on the timbre similarity, where N and M are both positive integers, and N is greater than or equal to M, and the original audios correspond to preset mixing parameters; The wet audio processing module is used to perform mixing processing on the dry audio according to the mixing parameters corresponding to the target audio for each of the M target audios, so as to obtain the wet audio corresponding to the target audio. The mixing and combining module is used to combine M wet audio files based on the timbre similarity between each target audio file and the dry audio file to obtain the mixed audio file corresponding to the dry audio file.

13. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions, which, when executed by a processor, implement the method as described in any one of claims 1 to 11.

14. A computing device, characterized in that, include: At least one processor; and a memory communicatively connected to the at least one processor; The memory stores instructions executable by the at least one processor, which, when executed by the at least one processor, cause the computing device to perform the method as described in any one of claims 1 to 11.

15. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the method as described in any one of claims 1 to 11.

Images