Audio processing method and device, medium, computing equipment and program product

By obtaining the tone characteristics and similarity of the dry sound audio, selecting the parameters of the target audio for automatic mixing processing, the problem of low mixing efficiency is solved and efficient audio mixing is achieved.

CN119068891AActive Publication Date: 2024-12-03HANGZHOU NETEASE CLOUD MUSIC TECH CO LTD

Patent Information

Application Number
CN202411067965.2
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-08-06
Publication Date
2024-12-03
Estimated Expiration
2044-08-06

AI Technical Summary

Technical Problem

During the mixing process, experienced mixers need to manually tune the audio effects, which leads to inefficiency and difficulty in meeting a large number of mixing needs, especially when artificial intelligence generates content.

Method used

By obtaining the tone characteristics of the dry sound audio, select the target audio based on the tone similarity to the original audio, and automatically mix the mixing parameters, and weighted combinations with the tone similarity to generate the final mixing audio.

Benefits of technology

Improves mixing efficiency and shortens mixing time, and does not need to repeatedly listen to debug parameters for dry sound audio, meeting more mixing needs.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119068891A_ABST
    Figure CN119068891A_ABST
Patent Text Reader

Abstract

The embodiment of the invention provides an audio processing method and device, a medium, computing equipment and a program product, and relates to the field of audio processing. The audio processing method comprises the following steps: obtaining timbre characteristics of dry sound audio, determining timbre similarity between the dry sound audio and original audio according to the timbre characteristics, and selecting M target audio from N original audio according to the timbre similarity, and applying the sound mixing parameters corresponding to the target audios to the dry sound audios to obtain M wet sound audios, and combining the M wet sound audios based on the timbre similarity to obtain a sound mixing audio. According to the mode, only the N original audios need to be prepared in advance, the audio mixing parameters of each original audio are recorded, then the audio mixing parameters of the original audios with high timbre similarity with the dry audio can be applied to the audio mixing process of the dry audio, repeated audition on the dry audio is not needed to debug the audio mixing parameters, the audio mixing efficiency can be improved, and the audio quality is improved. The sound mixing time is shortened, and more sound mixing requirements are met.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The embodiments of the present disclosure relate to the field of computer technology, and more specifically, the embodiments of the present disclosure relate to an audio processing method, apparatus, medium, computing device, and program product. Background Art

[0002] This section is intended to provide a background or context to the embodiments of the present disclosure. No description herein is admitted to be prior art by virtue of its inclusion in this section.

[0003] Audio Mixing is a key step in the audio production process, which involves combining multiple audio tracks such as vocals, instruments, and effects to form a harmonious and balanced final audio object.

[0004] Currently, the mixing process requires the use of multiple audio effects such as compressors, equalizers, reverberators, and push-pull attenuators for adjusting the volume to mix the audio to be mixed. It mainly relies on manual listening to continuously adjust the parameters of the audio effects to improve the audio quality through mixing.

[0005] However, the technical difficulty of mixing is relatively high, and usually requires experienced mixers to operate it personally, resulting in low mixing efficiency and difficulty in meeting more mixing needs. Summary of the invention

[0006] The present disclosure provides an audio processing method, apparatus, medium, computing device and program product to meet more mixing requirements.

[0007] In a first aspect of the embodiments of the present disclosure, there is provided an audio processing method, comprising:

[0008] Obtaining the timbre characteristics of dry audio;

[0009] For each of the preset N original audios, determine the timbre similarity between the dry sound audio and the original audio according to the timbre characteristics of the original audio and the timbre characteristics of the dry sound audio, and select M original audios from the N original audios as target audios according to the timbre similarity, where N and M are both positive integers, and N is greater than or equal to M, and the original audios correspond to preset mixing parameters;

[0010] For each target audio among the M target audios, mixing the dry audio according to the mixing parameter corresponding to the target audio to obtain a wet audio corresponding to the target audio;

[0011] According to the timbre similarity between each target audio and the dry audio, M wet audios are combined to obtain a mixed audio corresponding to the dry audio.

[0012] In another embodiment of the present disclosure, according to the timbre similarity between each target audio and the dry audio, M wet audios are combined to obtain a mixed audio corresponding to the dry audio, including:

[0013] For each target audio, determine a weight value corresponding to the target audio according to the timbre similarity between the target audio and the dry audio;

[0014] The M wet audios are weightedly combined according to the weight value of each target audio to obtain a mixed audio corresponding to the dry audio.

[0015] In another embodiment of the present disclosure, determining a weight value corresponding to the target audio according to the timbre similarity between the target audio and the dry audio includes:

[0016] The timbre similarity corresponding to the target audio is mapped to a weight value within a preset interval.

[0017] In another embodiment of the present disclosure, mixing the dry audio according to the mixing parameter corresponding to the target audio includes:

[0018] Adjusting speed-related parameters in the mixing parameters according to the audio speed of the dry audio, and / or adjusting loudness-related parameters in the mixing parameters according to the loudness of the dry audio;

[0019] The dry audio is mixed using the adjusted mixing parameters to obtain the corresponding wet audio.

[0020] In another embodiment of the present disclosure, the mixing parameters include reverberation time, and adjusting the parameters related to speed in the mixing parameters according to the audio speed of the dry audio includes:

[0021] If the audio speed of the dry audio is greater than or equal to a preset speed threshold, shortening the reverberation time;

[0022] If the audio speed of the dry audio is less than the speed threshold, the reverberation time is increased.

[0023] In another embodiment of the present disclosure, the mixing parameters include a compressor threshold, and adjusting the parameters related to the loudness in the mixing parameters according to the loudness of the dry audio includes:

[0024] If the loudness of the dry audio is greater than or equal to a preset loudness threshold, increasing the compressor threshold;

[0025] If the loudness of the dry audio is less than the loudness threshold, the compressor threshold is reduced.

[0026] In another embodiment of the present disclosure, the mixing parameters include effector parameters, and the mixing parameters corresponding to the original audio are obtained in the following manner:

[0027] The original audio is adjusted by a preset effector, wherein the effector includes one or more of a pre-compressor, a parametric equalizer, a dynamic equalizer, a multi-band exciter, a post-compressor, a multi-band compressor, a stereo enhancer, a delay, a reverberator and a volume adjuster;

[0028] If the adjustment result meets the preset conditions, the effector parameters corresponding to the adjustment result are recorded as the mixing parameters corresponding to the original audio.

[0029] In another embodiment of the present disclosure, the original audio is generated by a preset timbre model, and the timbre model is trained based on pre-labeled dry sound data.

[0030] In another embodiment of the present disclosure, obtaining the timbre characteristics of dry audio includes:

[0031] The dry audio is input into a pre-trained deep learning model so that the deep learning model outputs the timbre features corresponding to the dry audio, wherein the deep learning model is trained based on pre-labeled audio samples, and the audio samples are mel spectrograms generated based on preset audio.

[0032] In another embodiment of the present disclosure, the audio processing method further includes:

[0033] Acquire energy characteristics of the mixed audio and masking characteristics of the accompaniment audio corresponding to the mixed audio;

[0034] The volume ratio of the mixed audio and the accompaniment audio is adjusted according to the energy feature and the masking feature to obtain a mixed audio with balanced volume.

[0035] In another embodiment of the present disclosure, the audio processing method further includes:

[0036] Obtaining the loudness of the audio track of the mixed audio;

[0037] If the loudness of the audio track does not fall within the preset loudness interval, the mixed audio is adjusted by a preset effector so that the loudness of the audio track of the adjusted mixed audio falls within the loudness interval.

[0038] In a second aspect of the embodiments of the present disclosure, there is provided an audio processing device, comprising:

[0039] An acquisition module, used to acquire the timbre characteristics of dry audio;

[0040] A similarity determination module is used to determine, for each of the preset N original audios, the timbre similarity between the dry sound audio and the original audio according to the timbre characteristics of the original audio and the timbre characteristics of the dry sound audio, and select M original audios from the N original audios as target audios according to the timbre similarity, where N and M are both positive integers, and N is greater than or equal to M, and the original audios correspond to preset mixing parameters;

[0041] A wet sound processing module, configured to perform mixing processing on the dry sound audio according to a mixing parameter corresponding to each target audio among the M target audios, to obtain a wet sound audio corresponding to the target audio;

[0042] The mixing and combining module is used to combine M wet sound audios according to the timbre similarity between each target audio and the dry sound audio to obtain a mixed audio corresponding to the dry sound audio.

[0043] In a third aspect of the embodiments of the present disclosure, a computer-readable storage medium is provided, in which computer-executable instructions are stored. When a processor executes the computer-executable instructions, the method as described in any one of the first aspects is implemented.

[0044] In a fourth aspect of the embodiments of the present disclosure, there is provided a computing device, comprising: at least one processor;

[0045] and a memory communicatively coupled to the at least one processor;

[0046] The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the computing device to perform the method as described in any one of the first aspects.

[0047] In a fifth aspect of the embodiments of the present disclosure, a computer program product is provided, including a computer program, wherein when the computer program is executed by a processor, the method according to any one of the first aspects is implemented.

[0048] According to the audio processing method, apparatus, medium, computing device and program product of the embodiments of the present disclosure, by acquiring the timbre characteristics of dry audio, the timbre similarity between the dry audio and the original audio can be determined based on the timbre characteristics of the dry audio and the timbre characteristics of the original audio, and M target audios can be selected from N original audios based on the timbre similarity, and the mixing parameters corresponding to each target audio are respectively applied to the dry audio to obtain M wet audios, and then the M wet audios are combined based on the timbre similarity to obtain the mixed audio of the dry audio. This method only requires preparing N original audios in advance and recording the mixing parameters of each original audio after mixing, and then the mixing parameters of the original audio with high timbre similarity to the dry audio can be applied to the mixing process of the dry audio, without the need to repeatedly audition the dry audio to debug the mixing parameters, thereby improving the mixing efficiency, shortening the mixing time, and meeting more mixing needs. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] The above and other objects, features and advantages of the exemplary embodiments of the present disclosure will become readily understood by reading the detailed description below with reference to the accompanying drawings. In the accompanying drawings, several embodiments of the present disclosure are shown in an exemplary and non-limiting manner, in which:

[0050] Figure 1 A schematic diagram schematically shows an application scenario of the audio processing method according to an embodiment of the present disclosure;

[0051] Figure 2 The following schematically shows a flow chart of an audio processing method according to an embodiment of the present disclosure;

[0052] Figure 3 The following is a schematic diagram showing a flow chart of an audio processing method according to another embodiment of the present disclosure;

[0053] Figure 4 A schematic diagram of a wet sound audio combination method according to an embodiment of the present disclosure is schematically shown;

[0054] Figure 5 A schematic diagram of a process of obtaining original audio mixing parameters according to an embodiment of the present disclosure is schematically shown;

[0055] Figure 6 The structure diagram of an effector chain according to an embodiment of the present disclosure is schematically shown;

[0056] Figure 7 A schematic diagram schematically shows a storage medium according to an embodiment of the present disclosure;

[0057] Figure 8 The structure diagram of an audio processing device according to an embodiment of the present disclosure is schematically shown;

[0058] Fig. 9 A schematic diagram of a computing device according to an embodiment of the present disclosure is schematically shown.

[0059] In the drawings, the same or corresponding reference numerals represent the same or corresponding parts. DETAILED DESCRIPTION

[0060] The principles and spirit of the present disclosure will be described below with reference to several exemplary embodiments. It should be understood that these embodiments are provided only to enable those skilled in the art to better understand and implement the present disclosure, and are not intended to limit the scope of the present disclosure in any way. On the contrary, these embodiments are provided to make the present disclosure more thorough and complete, and to fully convey the scope of the present disclosure to those skilled in the art.

[0061] Those skilled in the art will appreciate that the embodiments of the present disclosure may be implemented as a system, device, apparatus, method or computer program product. Therefore, the present disclosure may be specifically implemented in the following forms, namely: complete hardware, complete software (including firmware, resident software, microcode, etc.), or a combination of hardware and software.

[0062] According to an embodiment of the present disclosure, an audio processing method, apparatus, medium, computing device, and program product are proposed.

[0063] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this disclosure are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant laws, regulations and standards, and corresponding operation entrances shall be provided for users to choose to authorize or refuse.

[0064] In addition, any number of elements in the drawings is for illustration and not limitation, and any naming is only for distinction and does not have any limiting meaning.

[0065] The principle and spirit of the present disclosure are explained in detail below with reference to several representative embodiments of the present disclosure. SUMMARY OF THE INVENTION

[0067] Mixing is a method of processing audio, which refers to using audio effects to process vocals and accompaniment to make the sound better, more harmonious, and more integrated. The mixing process usually uses a variety of audio effects such as compressors, equalizers, and reverberators. Manual mixing requires certain music theory knowledge and experience in using audio effects, which is quite difficult.

[0068] However, there are few experienced mixers, which makes it difficult to meet people's demand for mixing. Especially with the development of technologies such as AIGC (Artificial Intelligence Generated Content), the number of audio works sung by AI is increasing, and these audio works also need to be mixed to improve the quality of the works. Therefore, there is an urgent need for an audio processing method that can achieve automatic mixing to meet the needs of audio mixing.

[0069] After research and experiments, the inventors of the present invention can prepare several original audios in advance, and these original audios can have different timbres. Each original audio can be manually mixed and the mixing parameters corresponding to the final mixing result can be recorded. For the dry audio that needs to be mixed, multiple original audios with high timbre similarity can be selected according to the timbre similarity between the dry audio and the original audio, and the mixing parameters of these selected original audios can be used to mix the dry audio respectively, and then the timbre similarity is mapped to a weighted combination of the results of the dry audio mixing to obtain the final mixing result. In this way, it is only necessary to manually mix the limited number of original audios prepared in advance and record the mixing parameters used for mixing, and then select the corresponding mixing parameters based on the similarity between the dry audio and the original audio and apply them to the dry audio. There is no need to perform complex manual mixing on the dry audio, so that the automatic mixing of the dry audio can be achieved, the mixing efficiency is improved, and more mixing needs can be met. Among them, the dry audio can be unmixed human voice audio, and the human voice audio can be audio sung by a person or audio sung by AI.

[0070] After introducing the basic principles of the present disclosure, various non-limiting embodiments of the present disclosure are described in detail below.

[0071] Application Scenario Overview

[0072] First reference Figure 1 , Figure 1 This is a schematic diagram of an application scenario provided by the present disclosure. Figure 1 As shown, the terminal 102 can communicate with the server 101 through the network. The data storage system can store the data that the server 101 needs to process. The data storage system can be integrated on the server 101, or it can be placed on the cloud or other network servers. Among them, the terminal 102 can be, but is not limited to, various personal computers, laptops, smart phones, tablet computers, Internet of Things devices and portable wearable devices. The Internet of Things devices can be smart speakers, smart TVs, smart air conditioners, smart car-mounted devices, etc. Portable wearable devices can be smart watches, smart bracelets, head-mounted devices, etc. The server 101 can be implemented with an independent server or a server cluster consisting of multiple servers.

[0073] For scenarios where dry audio needs to be mixed, multiple original audios can be prepared in advance and the original audios can be manually mixed and the mixing parameters can be recorded, and these original audios and mixing parameters can be stored in the server 101, terminal 102 or data storage system. After obtaining the dry audio to be mixed, the server 101 or terminal 102 can mix the dry audio based on the stored original audio and mixing parameters.

[0074] The application scenarios mentioned above are only some examples. Those skilled in the art may expand the application based on the audio processing process, and the embodiments of the present disclosure do not impose any specific limitations on this.

[0075] Exemplary Methods

[0076] Combine the following Figure 1 For application scenarios, refer to Figures 2 to 6 To describe the audio processing method according to the exemplary embodiment of the present disclosure. It should be noted that the above application scenarios are only shown to facilitate understanding of the spirit and principle of the present disclosure, and the embodiments of the present disclosure are not limited in this respect. On the contrary, the embodiments of the present disclosure can be applied to any applicable scenario.

[0077] Figure 2 FIG. 1 is a flow chart of an audio processing method according to an embodiment of the present disclosure. Figure 2 As shown, the method may include:

[0078] Step S201, obtaining the timbre characteristics of dry audio.

[0079] The audio processing method of the present disclosure can be applied to Figure 1 The server 101 or the terminal 102 in .

[0080] The timbre feature may be a parameter that characterizes the timbre characteristics of human voice. For example, dry voice audio using bel canto singing usually has a full and rich timbre, and dry voice audio using pop singing usually has a light and gentle timbre.

[0081] Optionally, the dry audio is input into a pre-trained deep learning model so that the deep learning model outputs the timbre features corresponding to the dry audio. The deep learning model is trained based on pre-labeled audio samples. The audio sample can be a mel spectrum generated based on preset audio.

[0082] In some possible implementations, the method for acquiring the timbre features may also include: acquiring the frequency spectrum of the dry audio through Fourier transform and other methods, and selecting some frequency domain features as the timbre features.

[0083] Step S202, for each of the preset N original audios, determine the timbre similarity between the dry sound audio and the original audio according to the timbre characteristics of the original audio and the timbre characteristics of the dry sound audio, and select M original audios from the N original audios as target audios according to the timbre similarity.

[0084] Wherein, N and M are both positive integers, and N is greater than or equal to M, and the original audio corresponds to preset mixing parameters. The mixing parameters may include parameters of audio effectors such as a pre-compressor, a parametric equalizer, a dynamic equalizer, a multi-band exciter, a post-compressor, a multi-band compressor, a stereo enhancer, a delay, a reverberator, and a volume regulator.

[0085] Optionally, based on the preset N original audios, the similarity between the timbre characteristics of each original audio and the timbre characteristics of the dry sound audio is calculated to obtain N timbre similarities, and then M target audios are selected from the N original audios through a preset similarity threshold.

[0086] In one example, if Figure 3 As shown, the N original audios include audio 1, audio 2, ..., audio N. If the similarity between the timbre characteristics of audio 1 and the timbre characteristics of the dry audio is greater than or equal to a preset first threshold, the timbre similarity between the timbre characteristics of audio 2 and the timbre characteristics of the dry audio is greater than or equal to the first threshold, and the timbre similarities between the timbre characteristics of other audios and the timbre characteristics of the dry audio are all less than the first threshold, then M is 2, and the M target audios include audio 1 and audio 2.

[0087] In some possible implementations, after obtaining the similarity between the timbre features of each original audio and the timbre features of the dry audio, the N original audios can also be sorted in order of similarity from large to small, and the first M original audios are taken as target audios.

[0088] In some possible implementations, the timbre features of the original audio and the timbre features of the dry audio can be extracted in the same way. For example, a feature extraction model can be trained in advance based on audio samples, and the timbre features of the original audio and the dry audio can be extracted respectively using the trained extraction model.

[0089] In some possible implementations, the original audio may be obtained by human singing, and different original audios have different timbres.

[0090] In some possible implementations, the original audio may be generated by a preset timbre model, which is trained based on pre-labeled dry sound data. The timbre model may be an AI model based on artificial intelligence technologies such as AIGC.

[0091] Step S203: for each target audio among the M target audios, the dry audio is mixed according to the mixing parameters corresponding to the target audio to obtain the wet audio corresponding to the target audio.

[0092] Optionally, for each target audio, parameters of the target audio when using various audio effectors during the mixing process may be obtained, and the dry audio may be processed using corresponding audio effectors according to these parameters to obtain the wet audio.

[0093] In one example, if Figure 3 As shown, the multiple target audios include audio 1, audio 2, ..., audio M. Taking M=2 as an example, the target audio includes audio 1 and audio 2. In the mixing process, multiple audio effectors such as a pre-compressor, a parameter equalizer, and a dynamic equalizer are applied to audio 1. The mixer continuously adjusts the parameters of the audio effectors by auditioning the effect of audio 1 under the action of the audio effectors. The audio effector parameters that meet the audio effect standards after the mixer's adjustment are recorded as mixing parameters 1. Correspondingly, the audio effector parameters obtained after the mixing of audio 2 are recorded as mixing parameters 2. For dry audio, the pre-compressor, the parameter equalizer, the dynamic equalizer, and other audio effectors can be used to act on the dry audio with mixing parameters 1 and mixing parameters 2 as audio effector parameters, respectively, to obtain two different wet audios. One of the two wet audios is the result of mixing the dry audio with the mixing parameters of audio 1 (the wet audio corresponding to audio 1), and the other is the result of mixing the dry audio with the mixing parameters of audio 2 (the wet audio corresponding to audio 2).

[0094] Step S204: Combine the M wet audios according to the timbre similarity between each target audio and the dry audio to obtain a mixed audio corresponding to the dry audio.

[0095] Optionally, use audio processing software to align the audio tracks of M wet audios on the timeline, adjust the volume of the audio track of the wet audio based on the timbre similarity between the target audio corresponding to the wet audio and the dry audio, and superimpose all the adjusted audio tracks together to obtain the audio track of the final mixed audio.

[0096] In some possible implementations, the timbre similarity between the target audio corresponding to the wet audio and the dry audio may be positively correlated with the volume of the track of the wet audio. For example, the target audio corresponding to the wet audio is audio 1, and the greater the timbre similarity between audio 1 and the dry audio, the greater the volume of the wet audio after adjustment.

[0097] In the above embodiment, by obtaining the timbre characteristics of the dry audio, the timbre similarity between the dry audio and the original audio can be determined based on the timbre characteristics of the dry audio and the timbre characteristics of the original audio, and M target audios can be selected from the N original audios based on the timbre similarity, and the mixing parameters corresponding to each target audio are applied to the dry audio to obtain M wet audios, and then the M wet audios are combined based on the timbre similarity to obtain the mixed audio of the dry audio. This method only requires preparing N original audios in advance and recording the mixing parameters of each original audio after mixing, and then the mixing parameters of the original audio with high timbre similarity to the dry audio can be applied to the mixing process of the dry audio, without the need to repeatedly listen to the dry audio to debug the mixing parameters, thereby improving the mixing efficiency, shortening the mixing time, and meeting more mixing needs.

[0098] In one embodiment, Figure 4 As shown, according to the timbre similarity between each target audio and the dry audio, M wet audios are combined to obtain a mixed audio corresponding to the dry audio, including:

[0099] Step S401: for each target audio, determine a weight value corresponding to the target audio according to the timbre similarity between the target audio and the dry audio.

[0100] The timbre similarity between the target audio and the dry audio may be a cosine similarity calculated based on the timbre features of the target audio and the timbre features of the dry audio.

[0101] In some possible implementations, the timbre similarity between the target audio and the dry audio may be directly used as a weight value corresponding to the target audio.

[0102] In some possible implementations, the timbre similarity corresponding to the target audio may also be mapped to a weight value within a preset interval. The numerical range of the timbre similarity between the target audio and the dry audio is [-1,1], and the value of this interval may be mapped to a value in the interval [0,1], and the value after the timbre similarity mapping is used as the weight value.

[0103] Exemplarily, the mapping relationship between the timbre similarity and the weight value can be established according to the following formula (1) and formula (2):

[0104]

[0105] 1=∑ M weight(k) formula (2)

[0106] In the above formula, weight(k) may represent the weight value corresponding to the k-th target audio, and similarity(k) may represent the timbre similarity between the k-th target audio and the dry audio.

[0107] Step S402: weighted combination is performed on the M wet audios according to the weight value of each target audio to obtain a mixed audio corresponding to the dry audio.

[0108] Optionally, M wet audios are input into audio processing software, and the audio tracks of all wet audios are aligned on the timeline, and the audio tracks of the wet audios corresponding to the target audios are superimposed together according to the weight value of each target audio to obtain the audio track of the mixed audio.

[0109] In one embodiment, mixing the dry audio according to the mixing parameters corresponding to the target audio includes:

[0110] The speed-related parameters in the mixing parameters are adjusted according to the audio speed of the dry audio; the dry audio is mixed using the adjusted mixing parameters to obtain the corresponding wet audio.

[0111] The mixing parameters may include reverberation time.

[0112] The inventor has found through research that if the mixing parameters of the target audio are used to mix the dry audio, the effect of the audio speed on the mixing needs to be considered. The audio speed of the target audio may be different from the audio speed of the dry audio. Some mixing parameters of the target audio are determined based on the audio speed of the target audio. If the mixing parameters are directly used for the dry audio, the mixing effect of the dry audio may be affected.

[0113] Therefore, the inventors of the present invention think that the parameters affected by the audio speed, such as the reverberation time, in the mixing parameters can be adjusted according to the speed difference between the target audio and the dry audio, so that the adjusted parameters are more suitable for the audio speed of the dry audio. For example, for fast songs, the reverberation time needs to be appropriately shortened; for slow songs, the reverberation time needs to be appropriately lengthened.

[0114] In some possible implementations, adjusting the speed-related parameters in the mixing parameters according to the audio speed of the dry audio may include: if the audio speed of some dry audio is greater than or equal to a preset speed threshold, shortening the reverberation time; if the audio speed of some dry audio is less than the speed threshold, increasing the reverberation time. The speed threshold may be determined according to the audio speed of the target audio.

[0115] In the above embodiment, the speed threshold can be determined according to the audio speed of the target audio, and the mixing parameters such as the reverberation time can be adjusted according to the relationship between the audio speed of the dry audio and the speed threshold, so that the adjusted parameters are more suitable for the dry audio, thereby enhancing the mixing effect.

[0116] In one embodiment, mixing the dry audio according to the mixing parameters corresponding to the target audio includes:

[0117] The parameters related to the loudness in the mixing parameters are adjusted according to the loudness of the dry audio; the dry audio is mixed using the adjusted mixing parameters to obtain the corresponding wet audio.

[0118] Among other things, the mixing parameters may include a compressor threshold.

[0119] The inventor has found through research that if the mixing parameters of the target audio are used to mix the dry audio, the influence of the track loudness on the mixing needs to be considered. The track loudness of the target audio may be different from the track loudness of the dry audio. Some mixing parameters of the target audio (such as the parameters of audio effectors such as reverberators and multi-band compressors) are determined based on the track loudness of the target audio. If the mixing parameters are directly used for the dry audio, the mixing effect of the dry audio may be affected.

[0120] Therefore, the inventors of the present invention think that the parameters affected by the track loudness, such as the compressor threshold in the mixing parameters, can be adjusted according to the difference in track loudness between the target audio and the dry audio, so that the adjusted parameters are more suitable for the track loudness of the dry audio. For example, for low-level tracks, the compressor threshold needs to be appropriately reduced; for high-level tracks, the compressor threshold needs to be appropriately increased.

[0121] In some possible implementations, adjusting the loudness-related parameters in the mixing parameters according to the loudness of the dry audio may include: if the loudness of some dry audio is greater than or equal to a preset loudness threshold, increasing the compressor threshold; if the loudness of some dry audio is less than the loudness threshold, reducing the compressor threshold. The loudness threshold may be determined according to the loudness of the track of the target audio.

[0122] In the above embodiment, the loudness threshold can be determined according to the track loudness of the target audio, and the mixing parameters such as the compressor threshold can be adjusted according to the relationship between the track loudness of the dry audio and the loudness threshold, so that the adjusted parameters are more suitable for the dry audio, thereby enhancing the mixing effect.

[0123] In one embodiment, the mixing parameters include effector parameters, such as Figure 5 As shown, the mixing parameters corresponding to the original audio are obtained in the following way:

[0124] Step S501, adjusting the original audio through a preset effector.

[0125] Among them, the effector includes one or more of a pre-compressor, a parametric equalizer, a dynamic equalizer, a multi-band exciter, a post-compressor, a multi-band compressor, a stereo enhancer, a delay, a reverberator and a volume adjuster.

[0126] Optionally, the original audio is input into audio processing software, and the selected effects are attached to the audio track of the original audio. The mixing effect is evaluated by manual audition, and the parameters of the effects can be continuously adjusted according to the mixing effect.

[0127] In some possible implementations, all effects required for mixing and the order in which different effects are used for processing can be predefined. Figure 6 As shown, an effects chain can be composed of a pre-compressor, a parametric equalizer, a dynamic equalizer, a multi-band exciter, a post-compressor, a multi-band compressor, a stereo enhancer, a delay, a reverberator and a fader, and all the effects in the effects chain can be hung on the audio track of the original audio for adjustment.

[0128] In some possible implementations, it is also possible to determine an accompaniment that is suitable for the original audio, align the original audio and the accompaniment with the audio tracks through audio processing software, and audition the original audio and the accompaniment based on the situation where they are played together, and adjust the parameters of the effector according to the audition effect.

[0129] Step S502: If the adjustment result meets the preset conditions, the effector parameters corresponding to the adjustment result are recorded as the mixing parameters corresponding to the original audio.

[0130] Optionally, if the audition effect of the original audio after adjustment meets the requirements, the effector parameters used for adjustment can be recorded. For N original videos, N groups of parameters can be obtained.

[0131] In some possible implementations, if the adjustment result meets the preset conditions, the audio speed and track loudness of the original audio under the conditions can also be recorded. In this way, it is possible to record at what speed and loudness the mixing parameters of the original audio are adjusted, so that for some effect parameters related to audio speed and loudness, the effect parameters can be flexibly adjusted based on the recorded speed and loudness of the original audio and the speed and loudness of the dry audio to be mixed, so that the adjusted parameters are more suitable for the mixing of the dry audio.

[0132] In one embodiment, the audio processing method may further include:

[0133] The energy characteristics of the mixed audio and the masking characteristics of the accompaniment audio corresponding to the mixed audio are obtained; and the volume ratio of the mixed audio and the accompaniment audio is adjusted according to the energy characteristics and the masking characteristics to obtain the mixed audio with balanced volume.

[0134] The energy feature can be calculated according to the psychoacoustic model 2 in the standard ISO / IEC 11172-3. The masking feature can be calculated according to the masking threshold curve of each frame of audio in the accompaniment audio, and the masking threshold curve can also be calculated according to the psychoacoustic model 2 in the standard ISO / IEC 11172-3. The masking threshold can refer to a value that measures the masking ability of sound A to sound B in units of frequency band, which can be calculated by the psychoacoustic model.

[0135] Optionally, the energy characteristics of each frame of audio in the mixed audio and the masking characteristics of each frame of audio in the accompaniment audio are obtained, and the sum of the energy characteristics of each frame of the mixed audio and the sum of the masking characteristics of each frame of the accompaniment audio can be calculated; according to the sum of the energy characteristics and the sum of the masking characteristics, combined with a preset mapping relationship, the optimal volume ratio of the mixed audio and the accompaniment audio can be determined, and the mixed audio and the accompaniment audio are superimposed according to the ratio to obtain the final mixed audio with balanced volume. The preset mapping relationship can be obtained based on several audio samples, each audio sample has a good volume ratio, and by analyzing the sum of the energy characteristics of the human voice in the audio sample and the sum of the masking characteristics of the accompaniment, a mapping relationship between the two parameters of the sum of the energy characteristics and the sum of the masking characteristics and the volume ratio can be established.

[0136] In one embodiment, the audio processing method may further include:

[0137] The loudness of the audio track of the mixed audio is obtained; if the loudness of the audio track does not fall within the preset loudness range, the mixed audio is adjusted through a preset effector so that the loudness of the audio track of the mixed audio after adjustment falls within the loudness range.

[0138] The loudness interval may be determined based on an audio distribution standard.

[0139] For example, some song release standards require the track loudness to be [-12LUFS, -8LUFS]. For the above mixed audio, you can use the Maximizer effect to increase the track loudness to -9LUFS. The calculation process of the two parameter values ​​of the Maximizer is as follows:

[0140] Threshold Maximizer =LUFS mixed -(-9); Ceiling Maximizer =-0.2.

[0141] In this way, the loudness of the mixed audio can be increased to meet the requirements of audio distribution standards without distorting the audio waveform.

[0142] In one embodiment, the audio processing method may further include:

[0143] New original audio is added, and the new original audio is mixed by adjusting effector parameters and the like to obtain mixing parameters corresponding to the new original audio.

[0144] The new original audio can be sung by a human or generated by a new AI model.

[0145] In the above embodiment, by adding the original audio, the range of parameters used for dry audio mixing can be expanded, thereby enhancing the overall mixing effect.

[0146] Exemplary Media

[0147] After introducing the method of the exemplary embodiment of the present disclosure, next, refer to Figure 7 A storage medium according to an exemplary embodiment of the present disclosure is described.

[0148] refer to Figure 7 As shown, the storage medium 70 stores a program product for implementing the above method according to an embodiment of the present disclosure, which can be a portable compact disk read-only memory (CD-ROM) and includes program code, and can be run on a terminal device, such as a personal computer. However, the program product of the present disclosure is not limited thereto.

[0149] The program product may use any combination of one or more readable media. The readable medium may be a readable signal medium or a readable storage medium. The readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples (non-exhaustive list) of readable storage media include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0150] The readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, wherein the readable program code is carried. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. The readable signal medium may also be any readable medium other than a readable storage medium.

[0151] Program code for performing the disclosed operations of the present disclosure may be written in any combination of one or more programming languages, including object-oriented programming languages, such as Java, C++, etc., and conventional procedural programming languages, such as "C" or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user device, partially on a remote computing device, or entirely on a remote computing device or server. In the case of a remote computing device, the remote computing device may be connected to the user computing device through any type of network, including a local area network (LAN) or a wide area network (WAN).

[0152] Exemplary Devices

[0153] After introducing the medium of the exemplary embodiment of the present disclosure, next, reference is made to Figure 8 An audio processing device according to an exemplary embodiment of the present disclosure is described, which is used to implement a method in any of the above method embodiments. Its implementation principle and technical effects are similar and will not be repeated here.

[0154] like Figure 8 As shown, the audio processing device 800 may include:

[0155] The acquisition module 801 is used to acquire the timbre characteristics of the dry audio.

[0156] The similarity determination module 802 is used to determine the timbre similarity between the dry sound audio and the original audio for each of the preset N original audios according to the timbre characteristics of the original audio and the timbre characteristics of the dry sound audio, and select M original audios from the N original audios as target audios according to the timbre similarity, where N and M are both positive integers, and N is greater than or equal to M, and the original audios correspond to preset mixing parameters.

[0157] The wet sound processing module 803 is used to perform mixing processing on the dry sound audio according to the mixing parameters corresponding to the target audio for each target audio among the M target audios, so as to obtain the wet sound audio corresponding to the target audio.

[0158] The mixing and combining module 804 is used to combine the M wet audios according to the timbre similarity between each target audio and the dry audio to obtain a mixed audio corresponding to the dry audio.

[0159] In yet another embodiment of the present disclosure, the mixing and combining module 804 may include:

[0160] The weight determination unit may be configured to determine, for each target audio, a weight value corresponding to the target audio according to a timbre similarity between the target audio and the dry audio.

[0161] The wet sound combination unit can be used to perform weighted combination on M wet sound audios according to the weight value of each target audio to obtain a mixed audio corresponding to the dry sound audio.

[0162] In yet another embodiment of the present disclosure, the weight determination unit is further configured to map the timbre similarity corresponding to the target audio to a weight value within a preset interval.

[0163] In another embodiment of the present disclosure, the wet sound processing module 803 is further used to: adjust the speed-related parameters in the mixing parameters according to the audio speed of the dry sound audio, and / or adjust the loudness-related parameters in the mixing parameters according to the loudness of the dry sound audio; and perform mixing processing on the dry sound audio using the adjusted mixing parameters to obtain corresponding wet sound audio.

[0164] In another embodiment of the present disclosure, the wet sound processing module 803 is further used to: if the audio speed of the dry sound audio is greater than or equal to a preset speed threshold, shorten the reverberation time; if the audio speed of the dry sound audio is less than the speed threshold, increase the reverberation time.

[0165] In another embodiment of the present disclosure, the wet sound processing module 803 is further used to: if the loudness of the dry sound audio is greater than or equal to a preset loudness threshold, increase the compressor threshold; if the loudness of the dry sound audio is less than the loudness threshold, reduce the compressor threshold.

[0166] In yet another embodiment of the present disclosure, the audio processing device 800 may further include:

[0167] A tuning module is used to tune the original audio through a preset effector, wherein the effector includes one or more of a pre-compressor, a parametric equalizer, a dynamic equalizer, a multi-band exciter, a post-compressor, a multi-band compressor, a stereo enhancer, a delay, a reverberator and a volume adjuster.

[0168] The recording module is used to record the effector parameters corresponding to the adjustment result as the mixing parameters corresponding to the original audio when the adjustment result meets the preset conditions.

[0169] In yet another embodiment of the present disclosure, the original audio is generated by a preset timbre model, and the timbre model is trained based on pre-labeled dry sound data.

[0170] In another embodiment of the present disclosure, the acquisition module 801 is also used to input the dry sound audio into a pre-trained deep learning model so that the deep learning model outputs the timbre features corresponding to the dry sound audio. The deep learning model is trained based on pre-labeled audio samples, and the audio samples are mel-spectroscopy generated based on preset audio.

[0171] In yet another embodiment of the present disclosure, the audio processing device 800 may further include:

[0172] The feature acquisition module is used to acquire the energy feature of the mixed audio and the masking feature of the accompaniment audio corresponding to the mixed audio.

[0173] The volume balancing module is used to adjust the volume ratio of the mixed audio and the accompaniment audio according to the energy feature and the masking feature to obtain the mixed audio after volume balance.

[0174] In yet another embodiment of the present disclosure, the audio processing device 800 may further include:

[0175] The loudness acquisition module is used to obtain the loudness of the audio track of the mixed audio.

[0176] The loudness adjustment module is used to adjust the mixed audio through a preset effector when the loudness of the audio track does not fall within the preset loudness interval, so that the loudness of the audio track of the mixed audio after adjustment falls within the loudness interval.

[0177] Exemplary Computing Devices

[0178] After introducing the method, medium and apparatus of the exemplary embodiments of the present disclosure, next, reference is made to Fig. 9 A computing device according to an exemplary embodiment of the present disclosure is described.

[0179] Fig. 9 The computing device 90 shown is only an example and should not bring any limitation to the functionality and scope of use of the embodiments of the present disclosure.

[0180] like Fig. 9 As shown, the computing device 90 is in the form of a general computing device. The components of the computing device 90 may include, but are not limited to: at least one processing unit 901, at least one storage unit 902, and a bus 903 connecting different system components (including the processing unit 901 and the storage unit 902). Among them, at least one storage unit 902 stores computer-executable instructions; at least one processing unit 901 includes a processor, and the processor executes the computer-executable instructions to implement the method described above.

[0181] The bus 903 includes a data bus, a control bus, and an address bus.

[0182] The storage unit 902 may include a readable medium in the form of a volatile memory, such as a random access memory (RAM) 9021 and / or a cache memory 9022 , and may further include a readable medium in the form of a non-volatile memory, such as a read-only memory (ROM) 9023 .

[0183] The storage unit 902 may also include a program / utility 9025 having a set (at least one) of program modules 9024, such program modules 9024 including but not limited to: an operating system, one or more application programs, other program modules, and program data, each of which or some combination may include an implementation of a network environment.

[0184] The computing device 90 may also communicate with one or more external devices 904 (e.g., keyboards, pointing devices, etc.). Such communication may be performed via an input / output (I / O) interface 905. Furthermore, the computing device 90 may also communicate with one or more networks (e.g., a local area network (LAN), a wide area network (WAN), and / or a public network, such as the Internet) via a network adapter 906. Fig. 9 As shown, the network adapter 906 communicates with other modules of the computing device 90 via the bus 903. It should be understood that, although not shown in the figure, other hardware and / or software modules can be used in conjunction with the computing device 90, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0185] It should be noted that although several units / modules or sub-units / modules of the audio processing device are mentioned in the above detailed description, this division is merely exemplary and not mandatory. In fact, according to an embodiment of the present disclosure, the features and functions of two or more units / modules described above may be embodied in one unit / module. Conversely, the features and functions of one unit / module described above may be further divided to be embodied by multiple units / modules.

[0186] In addition, although the operations of the disclosed method are described in a specific order in the drawings, this does not require or imply that the operations must be performed in this specific order, or that all the operations shown must be performed to achieve the desired results. Additionally or alternatively, some steps may be omitted, multiple steps may be combined into one step, and / or one step may be decomposed into multiple steps.

[0187] Although the spirit and principle of the present disclosure have been described with reference to several specific embodiments, it should be understood that the present disclosure is not limited to the disclosed specific embodiments, and the division of various aspects does not mean that the features in these aspects cannot be combined to benefit, and such division is only for the convenience of expression. The present disclosure is intended to cover various modifications and equivalent arrangements included in the spirit and scope of the attached claims.

Claims

1. An audio processing method, characterized in that: include: Obtaining the timbre characteristics of dry audio; For each of the preset N original audios, determine the timbre similarity between the dry sound audio and the original audio according to the timbre characteristics of the original audio and the timbre characteristics of the dry sound audio, and select M original audios from the N original audios as target audios according to the timbre similarity, where N and M are both positive integers, and N is greater than or equal to M, and the original audios correspond to preset mixing parameters; For each target audio among the M target audios, the dry audio is mixed according to the mixing parameter corresponding to the target audio to obtain a wet audio corresponding to the target audio; According to the timbre similarity between each target audio and the dry audio, M wet audios are combined to obtain a mixed audio corresponding to the dry audio.

2. The audio processing method according to claim 1, characterized in that: The combining of the M wet audios according to the timbre similarity between each target audio and the dry audio to obtain a mixed audio corresponding to the dry audio includes: For each target audio, determine a weight value corresponding to the target audio according to the timbre similarity between the target audio and the dry audio; The M wet audios are weightedly combined according to the weight value of each target audio to obtain a mixed audio corresponding to the dry audio.

3. The audio processing method according to claim 2, characterized in that: The determining, according to the timbre similarity between the target audio and the dry audio, a weight value corresponding to the target audio includes: The timbre similarity corresponding to the target audio is mapped to a weight value within a preset interval.

4. The audio processing method according to any one of claims 1 to 3, characterized in that: The mixing process of the dry audio according to the mixing parameter corresponding to the target audio includes: Adjusting speed-related parameters in the mixing parameters according to the audio speed of the dry audio, and / or adjusting loudness-related parameters in the mixing parameters according to the loudness of the dry audio; The dry audio is mixed using the adjusted mixing parameters to obtain the corresponding wet audio.

5. The audio processing method according to claim 4, characterized in that: The mixing parameters include reverberation time, and adjusting the speed-related parameters in the mixing parameters according to the audio speed of the dry audio includes: If the audio speed of the dry audio is greater than or equal to a preset speed threshold, shortening the reverberation time; If the audio speed of the dry audio is less than the speed threshold, the reverberation time is increased.

6. The audio processing method according to claim 4, characterized in that: The mixing parameters include a compressor threshold, and adjusting parameters related to loudness in the mixing parameters according to the loudness of the dry audio includes: If the loudness of the dry audio is greater than or equal to a preset loudness threshold, increasing the compressor threshold; If the loudness of the dry audio is less than the loudness threshold, the compressor threshold is lowered.

7. An audio processing device, characterized in that: include: An acquisition module, used to acquire the timbre characteristics of dry audio; A similarity determination module is used to determine, for each of the preset N original audios, the timbre similarity between the dry sound audio and the original audio according to the timbre characteristics of the original audio and the timbre characteristics of the dry sound audio, and select M original audios from the N original audios as target audios according to the timbre similarity, where N and M are both positive integers, and N is greater than or equal to M, and the original audios correspond to preset mixing parameters; A wet sound processing module, configured to perform mixing processing on the dry sound audio according to a mixing parameter corresponding to each target audio among the M target audios, to obtain a wet sound audio corresponding to the target audio; The mixing and combining module is used to combine M wet sound audios according to the timbre similarity between each target audio and the dry sound audio to obtain a mixed audio corresponding to the dry sound audio.

8. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-executable instructions, and when a processor executes the computer-executable instructions, the method according to any one of claims 1 to 6 is implemented.

9. A computing device, characterized in that include: at least one processor; and a memory communicatively coupled to the at least one processor; The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the computing device to perform the method according to any one of claims 1 to 6.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the method according to any one of claims 1 to 6 is implemented.

Citation Information

Patent Citations

  • Audio data processing method and device and computer storage medium

    CN109920446A

  • Intelligent tone tuning method and device based on timbre, medium and computing equipment

    CN113870873A

  • Sound mixing method and device, equipment and storage medium

    CN117153131A

  • Encoding method for sound signal

    JP2003255930A

  • Loudness adjustment for downmixed audio content

    WO2015038522A1

Cited By

  • Song recording method, electronic equipment and computer readable storage medium

    CN119626186A

  • Audio processing method, program product and electronic device

    CN121034339A