Audio processing method, device, computing equipment and medium

Abstract

The embodiments of the present disclosure provide an audio processing method, apparatus, computing device and medium. The main component audio and surround component audio corresponding to the target multi-channel audio are determined based on the left channel audio and right channel audio corresponding to the audio to be processed, and the frequency band weight parameters corresponding to the main component audio and surround component audio of the target multi-channel audio obtained through the first audio network training, thereby obtaining the rendered audio based on the surround component audio and the target mapping parameters obtained through the second audio network training, and then obtaining the target multi-channel audio corresponding to the audio to be processed based on the main component audio and the rendered audio corresponding to the target multi-channel audio. By using the parameters obtained through the audio processing network training to perform audio processing, the problem of reduced separation caused by the imbalance of weights used when separating signals when there are multiple main sound sources in the audio can be avoided, thereby improving the processing effect of the audio processing method.

Description

Technical Field

[0001] The embodiments of the present disclosure relate to the field of multimedia technology. More specifically, the embodiments of the present disclosure relate to an audio processing method, apparatus, computing device and medium. Background Art

[0002] This section is intended to provide a background or context to the embodiments of the present disclosure. No description herein is admitted to be prior art by virtue of its inclusion in this section.

[0003] In order to render ordinary stereo audio into a surround immersive effect, it is usually necessary to upmix the two-channel audio to multi-channel audio.

[0004] In the related art, when upmixing two-channel audio, the principal components of the two-channel audio are often first determined through principal component analysis (PCA), and then the orthogonal surround components of the principal components are determined. Then, a linear combination of the determined principal components and surround components is used as multi-channel audio to achieve upmixing of the two-channel audio.

[0005] In the above implementation process, it is assumed that the main component and the surround component are completely orthogonal. Therefore, it is only suitable for processing audio with only one main sound source. When there are multiple main sound sources in the audio to be processed, the processing effect of the above audio processing method will be greatly reduced. Summary of the invention

[0006] In view of the problem that the audio processing methods in the related art have poor processing effect when processing the to-be-processed audio with multiple main sound sources, the embodiments of the present disclosure at least provide an audio processing method, apparatus, computing device and medium.

[0007] In a first aspect of the embodiments of the present disclosure, there is provided an audio processing method, the method comprising:

[0008] Determine the main component audio and the surround component audio corresponding to the target multi-channel audio based on the left channel audio and the right channel audio corresponding to the audio to be processed, and the frequency band weight parameters of the frequency bands to which the main component audio and the surround component audio of the target multi-channel audio belong, the frequency band weight parameters are obtained by training the first audio network, and the target multi-channel audio includes a target left channel audio part and a target right channel audio part;

[0009] Obtaining rendered audio based on the surround component audio and the target mapping parameters, where the target mapping parameters are obtained by training the second audio network;

[0010] Based on the main component audio and the rendered audio corresponding to the target multi-channel audio, the target multi-channel audio corresponding to the audio to be processed is obtained.

[0011] In one embodiment of the present disclosure, based on the left channel audio and the right channel audio corresponding to the audio to be processed, and the frequency band weight parameters of the frequency bands to which the main component audio and the surround component audio of the target multi-channel audio belong, determining the main component audio and the surround component audio corresponding to the target multi-channel audio includes:

[0012] Determine the main component audio based on the left channel audio, the right channel audio and the frequency band weight parameter corresponding to the frequency band to which the main component audio belongs;

[0013] The surround component audio is determined based on the left channel audio, the right channel audio, and the frequency band weight parameter corresponding to the frequency band to which the surround component audio belongs.

[0014] In one embodiment of the present disclosure, the main component audio includes a main component left channel audio and a main component right channel audio;

[0015] Determining the main component audio based on the left channel audio, the right channel audio, and a frequency band weight parameter corresponding to the frequency band to which the main component audio belongs, including:

[0016] Based on a first frequency band weight parameter for processing the left channel audio corresponding to the frequency band to which the main component audio belongs, and a second frequency band weight parameter for processing the right channel audio, weighted summing the left channel audio and the right channel audio is performed to obtain the main component left channel audio;

[0017] Based on the third frequency band weight parameter for processing the left channel audio corresponding to the frequency band to which the main component audio belongs, and the fourth frequency band weight parameter for processing the right channel audio, the left channel audio and the right channel audio are weightedly summed to obtain the main component right channel audio.

[0018] In one embodiment of the present disclosure, the surround component audio includes a surround component left channel audio and a surround component right channel audio;

[0019] Determining the surround component audio based on the left channel audio, the right channel audio, and a frequency band weight parameter corresponding to the frequency band to which the surround component audio belongs, including:

[0020] Based on a fifth frequency band weight parameter for processing the left channel audio corresponding to the frequency band to which the surround component audio belongs, and a sixth frequency band weight parameter for processing the right channel audio, weighted summing the left channel audio and the right channel audio is performed to obtain the surround component left channel audio;

[0021] Based on the seventh frequency band weight parameter for processing the left channel audio corresponding to the frequency band to which the surround component audio belongs, and the eighth frequency band weight parameter for processing the right channel audio, the left channel audio and the right channel audio are weightedly summed to obtain the surround component left channel audio.

[0022] In one embodiment of the present disclosure, the surround component audio includes a surround component left channel audio and a surround component right channel audio;

[0023] Get rendered audio based on the surround component audio and target mapping parameters, including:

[0024] Based on the first target mapping parameter in the target mapping parameter for processing the left channel audio of the surround component, and the second target mapping parameter in the target mapping parameter for processing the right channel audio of the surround component, the left channel audio of the surround component and the right channel audio of the surround component are weightedly summed to obtain rendered audio.

[0025] In one embodiment of the present disclosure, obtaining target multi-channel audio corresponding to the audio to be processed based on the main component audio and the rendered audio includes:

[0026] The main component audio and the rendered audio are superimposed to obtain the target multi-channel audio corresponding to the audio to be processed.

[0027] In one embodiment of the present disclosure, the training process of the frequency band weight parameters and the target mapping parameters includes:

[0028] Based on the sample multi-channel audio, obtain the sample left channel audio and the sample right channel audio;

[0029] Based on the sample left channel audio and the sample right channel audio, determine a first sample audio feature, a second sample audio feature, and a third sample audio feature, the first sample audio feature is used to indicate the power sum of the sample left channel audio and the sample right channel audio, the second sample audio feature is used to indicate the power difference between the sample left channel audio and the sample right channel audio, and the third sample audio feature is used to indicate the real cross-correlation power of the sample left channel audio and the sample right channel audio;

[0030] Determine the first sample audio and the second sample audio based on the sample left channel audio, the sample right channel audio, and a plurality of predicted frequency band weight parameters obtained by processing the first sample audio feature, the second sample audio feature, and the third sample audio feature through the first audio processing network;

[0031] Obtain predicted rendered audio based on the second sample audio and the predicted mapping parameters obtained by processing the second sample audio through the second audio processing network;

[0032] Based on the first sample audio and the predicted rendered audio, obtaining predicted multi-channel audio;

[0033] Based on a target loss function indicating the difference between the predicted multi-channel audio and the sample multi-channel audio, the first audio processing network and the second audio processing network are trained to obtain frequency band weight parameters and target mapping parameters.

[0034] In one embodiment of the present disclosure, the sample multi-channel audio includes sample left front channel audio, sample right front channel audio, sample center channel audio, sample left surround channel audio, and sample right surround channel audio;

[0035] Based on the sample multi-channel audio, obtain the sample left channel audio and the sample right channel audio, including:

[0036] The sample center channel audio and the sample left surround channel audio are weighted respectively by preset weight parameters, and the sample left channel audio is determined based on the weighted result and the sample left front channel audio;

[0037] The sample center channel audio and the sample right surround channel audio are weighted respectively by preset weight parameters, and the sample right channel audio is determined based on the weighted results and the sample right front channel audio.

[0038] In one embodiment of the present disclosure, determining a first sample audio feature, a second sample audio feature, and a third sample audio feature based on a sample left channel audio and a sample right channel audio includes:

[0039] Determine a first sample audio feature and a second sample audio feature based on the sample left channel audio, the complex conjugate audio corresponding to the sample left channel audio, the sample right channel audio, the complex conjugate audio corresponding to the sample right channel audio, and a target smoothing parameter;

[0040] A third sample audio feature is determined based on a real part of a multiplication result of the sample left channel audio and a complex conjugate audio corresponding to the sample right channel audio, and a target smoothing factor.

[0041] In one embodiment of the present disclosure, based on the sample left channel audio, the sample right channel audio, and a plurality of predicted frequency band weight parameters obtained by processing the first sample audio feature, the second sample audio feature, and the third sample audio feature through the first audio processing network, before determining the first sample audio and the second sample audio, the method further includes:

[0042] The first sample audio feature, the second sample feature and the third sample audio feature are input into the first audio processing network, and the first sample audio feature, the second sample feature and the third sample audio feature are processed respectively through the first linear transformation layer and the second linear transformation layer included in the first audio processing network to obtain multiple predicted frequency band weight parameters for obtaining the first sample audio and multiple predicted frequency band weight parameters for obtaining the second sample audio.

[0043] In one embodiment of the present disclosure, before obtaining the predicted rendered audio based on the second sample audio and the predicted mapping parameters obtained by processing the second sample audio through the second audio processing network, the method further includes:

[0044] The second sample audio is input into the second audio processing network, and the second sample audio is processed through the linear transformation layer included in the second audio processing network to obtain the prediction mapping parameter.

[0045] In one embodiment of the present disclosure, based on a target loss function indicating a difference between a predicted multi-channel audio and a sample multi-channel audio, training a first audio processing network and a second audio processing network to obtain a frequency band weight parameter and a target mapping parameter includes:

[0046] Determining a first loss function based on the predicted multi-channel audio and the sample multi-channel audio;

[0047] Determining a plurality of second loss functions based on the predicted amplitude differences between the channels of the multi-channel audio and the amplitude differences between the channels of the sample multi-channel audio;

[0048] Determine a target loss function based on the first loss function and the plurality of second loss functions;

[0049] The first audio processing network and the second audio processing network are trained based on the target loss function until a training cutoff condition is met, thereby obtaining frequency band weight parameters and target mapping parameters.

[0050] In a second aspect of the embodiments of the present disclosure, there is provided an audio processing device, the device comprising:

[0051] A determination module, configured to determine the main component audio and the surround component audio corresponding to the target multi-channel audio based on the left channel audio and the right channel audio corresponding to the audio to be processed, and the frequency band weight parameters of the frequency bands to which the main component audio and the surround component audio of the target multi-channel audio belong, respectively, wherein the frequency band weight parameters are obtained by training the first audio network, and the target multi-channel audio includes a target left channel audio part and a target right channel audio part;

[0052] A first acquisition module is used to acquire rendered audio based on surround component audio and target mapping parameters, where the target mapping parameters are obtained through training of a second audio network;

[0053] The second acquisition module is used to acquire the target multi-channel audio corresponding to the audio to be processed based on the main component audio and the rendering audio corresponding to the target multi-channel audio.

[0054] In one embodiment of the present disclosure, the determination module, when used to determine the main component audio and the surround component audio corresponding to the target multi-channel audio based on the left channel audio and the right channel audio corresponding to the audio to be processed, and the frequency band weight parameters of the frequency bands to which the main component audio and the surround component audio of the target multi-channel audio belong, respectively, includes:

[0055] A first determining unit, configured to determine the main component audio based on the left channel audio, the right channel audio, and a frequency band weight parameter corresponding to the frequency band to which the main component audio belongs;

[0056] The second determination unit is used to determine the surround component audio based on the left channel audio, the right channel audio and the frequency band weight parameter corresponding to the frequency band to which the surround component audio belongs.

[0057] In one embodiment of the present disclosure, the main component audio includes a main component left channel audio and a main component right channel audio;

[0058] The first determination unit, when used to determine the main component audio based on the left channel audio, the right channel audio and the frequency band weight parameter corresponding to the frequency band to which the main component audio belongs, is used to:

[0059] Based on a first frequency band weight parameter for processing the left channel audio corresponding to the frequency band to which the main component audio belongs, and a second frequency band weight parameter for processing the right channel audio, weighted summing the left channel audio and the right channel audio is performed to obtain the main component left channel audio;

[0060] Based on the third frequency band weight parameter for processing the left channel audio corresponding to the frequency band to which the main component audio belongs, and the fourth frequency band weight parameter for processing the right channel audio, the left channel audio and the right channel audio are weightedly summed to obtain the main component right channel audio.

[0061] In one embodiment of the present disclosure, the surround component audio includes a surround component left channel audio and a surround component right channel audio;

[0062] The second determination unit, when used to determine the surround component audio based on the left channel audio, the right channel audio and the frequency band weight parameter corresponding to the frequency band to which the surround component audio belongs, is used to:

[0063] Based on a fifth frequency band weight parameter for processing the left channel audio corresponding to the frequency band to which the surround component audio belongs, and a sixth frequency band weight parameter for processing the right channel audio, weighted summing the left channel audio and the right channel audio is performed to obtain the surround component left channel audio;

[0064] Based on the seventh frequency band weight parameter for processing the left channel audio corresponding to the frequency band to which the surround component audio belongs, and the eighth frequency band weight parameter for processing the right channel audio, the left channel audio and the right channel audio are weightedly summed to obtain the surround component left channel audio.

[0065] In one embodiment of the present disclosure, the surround component audio includes a surround component left channel audio and a surround component right channel audio;

[0066] The first acquisition module, when used to acquire rendered audio based on the surround component audio and the target mapping parameters, is used to:

[0067] Based on the first target mapping parameter in the target mapping parameter for processing the left channel audio of the surround component, and the second target mapping parameter in the target mapping parameter for processing the right channel audio of the surround component, the left channel audio of the surround component and the right channel audio of the surround component are weightedly summed to obtain rendered audio.

[0068] In one embodiment of the present disclosure, the second acquisition module, when used to acquire the target multi-channel audio corresponding to the audio to be processed based on the main component audio and the rendered audio, is used to:

[0069] The main component audio and the rendered audio are superimposed to obtain the target multi-channel audio corresponding to the audio to be processed.

[0070] In one embodiment of the present disclosure, the device further includes a training module, which includes:

[0071] A first acquisition unit, configured to acquire a sample left channel audio and a sample right channel audio based on the sample multi-channel audio;

[0072] A first determining unit is used to determine a first sample audio feature, a second sample audio feature, and a third sample audio feature based on the sample left channel audio and the sample right channel audio, the first sample audio feature is used to indicate the power sum of the sample left channel audio and the sample right channel audio, the second sample audio feature is used to indicate the power difference between the sample left channel audio and the sample right channel audio, and the third sample audio feature is used to indicate the real cross-correlation power of the sample left channel audio and the sample right channel audio;

[0073] A second determination unit is used to determine the first sample audio and the second sample audio based on the sample left channel audio, the sample right channel audio, and a plurality of predicted frequency band weight parameters obtained by processing the first sample audio feature, the second sample audio feature, and the third sample audio feature through the first audio processing network;

[0074] A second acquisition unit, configured to acquire predicted rendered audio based on the second sample audio and the predicted mapping parameters obtained by processing the second sample audio through the second audio processing network;

[0075] A third acquisition unit, configured to acquire predicted multi-channel audio based on the first sample audio and the predicted rendered audio;

[0076] A training unit is used to train the first audio processing network and the second audio processing network based on a target loss function indicating a difference between the predicted multi-channel audio and the sample multi-channel audio to obtain a frequency band weight parameter and a target mapping parameter.

[0077] In one embodiment of the present disclosure, the sample multi-channel audio includes sample left front channel audio, sample right front channel audio, sample center channel audio, sample left surround channel audio, and sample right surround channel audio;

[0078] The first acquisition unit, when used to acquire the sample left channel audio and the sample right channel audio based on the sample multi-channel audio, is used to:

[0079] The sample center channel audio and the sample left surround channel audio are weighted respectively by preset weight parameters, and the sample left channel audio is determined based on the weighted result and the sample left front channel audio;

[0080] The sample center channel audio and the sample right surround channel audio are weighted respectively by preset weight parameters, and the sample right channel audio is determined based on the weighted results and the sample right front channel audio.

[0081] In one embodiment of the present disclosure, the first determining unit, when used to determine the first sample audio feature, the second sample audio feature, and the third sample audio feature based on the sample left channel audio and the sample right channel audio, is used to:

[0082] Determine a first sample audio feature and a second sample audio feature based on the sample left channel audio, the complex conjugate audio corresponding to the sample left channel audio, the sample right channel audio, the complex conjugate audio corresponding to the sample right channel audio, and a target smoothing parameter;

[0083] A third sample audio feature is determined based on a real part of a multiplication result of the sample left channel audio and a complex conjugate audio corresponding to the sample right channel audio, and a target smoothing factor.

[0084] In one embodiment of the present disclosure, the training module further includes:

[0085] The first processing unit is used to input the first sample audio feature, the second sample feature and the third sample audio feature into the first audio processing network, and process the first sample audio feature, the second sample feature and the third sample audio feature through the first linear transformation layer and the second linear transformation layer included in the first audio processing network respectively to obtain multiple predicted frequency band weight parameters for obtaining the first sample audio and multiple predicted frequency band weight parameters for obtaining the second sample audio.

[0086] In one embodiment of the present disclosure, the device further includes:

[0087] The second processing unit is used to input the second sample audio into the second audio processing network, and process the second sample audio through the linear transformation layer included in the second audio processing network to obtain the prediction mapping parameter.

[0088] In one embodiment of the present disclosure, the training unit, when training the first audio processing network and the second audio processing network based on the target loss function indicating the difference between the predicted multi-channel audio and the sample multi-channel audio to obtain the frequency band weight parameter and the target mapping parameter, is used to:

[0089] Determining a first loss function based on the predicted multi-channel audio and the sample multi-channel audio;

[0090] Determining a plurality of second loss functions based on the predicted amplitude differences between the channels of the multi-channel audio and the amplitude differences between the channels of the sample multi-channel audio;

[0091] Determine a target loss function based on the first loss function and the plurality of second loss functions;

[0092] The first audio processing network and the second audio processing network are trained based on the target loss function until a training cutoff condition is met, thereby obtaining frequency band weight parameters and target mapping parameters.

[0093] In a third aspect of the embodiments of the present disclosure, a computing device is provided, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the operations performed by the audio processing method provided in the first aspect and any embodiment of the first aspect are implemented.

[0094] In a fourth aspect of the embodiments of the present disclosure, a computer-readable storage medium is provided, on which a program is stored, and the program is used by a processor to execute the operations performed by the audio processing method provided in the first aspect and any embodiment of the first aspect.

[0095] In a fifth aspect of the embodiments of the present disclosure, a computer program product is provided, including a computer program, which, when executed by a processor, implements the operations performed by the audio processing method provided in the first aspect and any embodiment of the first aspect.

[0096] According to the audio processing method, apparatus, computing device, medium, etc. of the embodiments of the present disclosure, the main component audio and surround component audio corresponding to the target multi-channel audio are determined based on the left channel audio and right channel audio corresponding to the audio to be processed, and the frequency band weight parameters corresponding to the frequency bands to which the main component audio and surround component audio of the target multi-channel audio belong, respectively, obtained through the first audio network training, thereby obtaining the rendered audio based on the surround component audio and the target mapping parameters obtained through the second audio network training, and then obtaining the target multi-channel audio corresponding to the audio to be processed based on the main component audio and the rendered audio corresponding to the target multi-channel audio. By using the parameters obtained through the audio processing network training to perform audio processing, the problem of reduced separation caused by the imbalance of weights used when separating signals when there are multiple main sound sources in the audio can be avoided, so that the audio processing process does not need to consider the number of main sound sources in the audio to be processed, thereby improving the accuracy of the processed target multi-channel audio and improving the processing effect of the audio processing method. BRIEF DESCRIPTION OF THE DRAWINGS

[0097] The above and other objects, features and advantages of the exemplary embodiments of the present disclosure will become readily understood by reading the detailed description below with reference to the accompanying drawings. In the accompanying drawings, several embodiments of the present disclosure are shown in an exemplary and non-limiting manner, in which:

[0098] Figure 1 is a flowchart of an audio processing method according to an exemplary embodiment of the present disclosure;

[0099] Figure 2 is a schematic diagram of obtaining left and right channel audio based on audio to be processed according to an exemplary embodiment of the present disclosure;

[0100] Figure 3 is a flow chart of a parameter training process according to an exemplary embodiment of the present disclosure;

[0101] Figure 4 is a schematic diagram of a network structure of a first audio processing network according to an exemplary embodiment of the present disclosure;

[0102] Figure 5 is a schematic diagram of a network structure of a second audio processing network according to an exemplary embodiment of the present disclosure;

[0103] Figure 6is a flow chart of a training process according to an exemplary embodiment of the present disclosure;

[0104] Figure 7 is a block diagram of an audio processing device according to an exemplary embodiment of the present disclosure;

[0105] Figure 8 is a schematic diagram of a computer-readable storage medium according to an exemplary embodiment of the present disclosure;

[0106] Fig. 9 is a structural schematic diagram of a computing device according to an exemplary embodiment of the present disclosure;

[0107] In the drawings, the same or corresponding reference numerals represent the same or corresponding parts. DETAILED DESCRIPTION

[0108] The principles and spirit of the present disclosure will be described below with reference to several exemplary embodiments. It should be understood that these embodiments are provided only to enable those skilled in the art to better understand and implement the present disclosure, and are not intended to limit the scope of the present disclosure in any way. On the contrary, these embodiments are provided to make the present disclosure more thorough and complete, and to fully convey the scope of the present disclosure to those skilled in the art.

[0109] Those skilled in the art will appreciate that the embodiments of the present disclosure may be implemented as a system, device, apparatus, method or computer program product. Therefore, the present disclosure may be specifically implemented in the following forms, namely: complete hardware, complete software (including firmware, resident software, microcode, etc.), or a combination of hardware and software.

[0110] It should be understood herein that any number of elements in the drawings is for illustration rather than limitation, and any naming is only for distinction rather than having any limiting meaning.

[0111] According to the embodiments of the present disclosure, an audio processing method, apparatus, computing device and medium are proposed. The above method can be executed by a computing device to process the audio to be processed to obtain the target multi-channel audio. The computing device can be a server, such as a server, multiple servers, a server cluster, a cloud computing platform, etc. Optionally, the computing device can also be a terminal device, such as a smart phone, a tablet computer, a desktop computer, a portable computer, a smart speaker, etc. The present disclosure does not limit the device type and device quantity of the computing device.

[0112] After introducing the application environment of the present disclosure, the specific implementation method of the present disclosure is introduced below.

[0113] See also Figure 1 , Figure 1 is a flowchart of an audio processing method according to an exemplary embodiment of the present disclosure, the method comprising:

[0114] Step 101: determine the main component audio and surround component audio corresponding to the target multi-channel audio based on the left channel audio and the right channel audio corresponding to the audio to be processed, and the frequency band weight parameters corresponding to the frequency bands to which the main component audio and the surround component audio of the target multi-channel audio belong, the frequency band weight parameters are obtained by training the first audio network, and the target multi-channel audio includes a target left channel audio part and a target right channel audio part.

[0115] The audio to be processed may be a two-channel audio. Optionally, the audio to be processed may also be a multi-channel audio, and the number of channels included in the audio to be processed is less than the number of channels included in the target multi-channel audio.

[0116] Step 102: Obtain rendered audio based on the surround component audio and the target mapping parameters, where the target mapping parameters are obtained through training of the second audio network.

[0117] Step 103: Based on the main component audio and the rendered audio corresponding to the target multi-channel audio, obtain the target multi-channel audio corresponding to the audio to be processed.

[0118] The present disclosure determines the main component audio and surround component audio corresponding to the target multi-channel audio based on the left channel audio and the right channel audio corresponding to the audio to be processed, and the frequency band weight parameters obtained by training the first audio network and corresponding to the frequency bands to which the main component audio and the surround component audio of the target multi-channel audio belong, thereby obtaining the rendered audio based on the surround component audio and the target mapping parameters obtained by training the second audio network, and then obtaining the target multi-channel audio corresponding to the audio to be processed based on the main component audio and the rendered audio corresponding to the target multi-channel audio. By using the parameters obtained by training the audio processing network to perform audio processing, the problem of reduced separation caused by an imbalance in weights used when separating signals when there are multiple main sound sources in the audio can be avoided, so that the audio processing process does not need to consider the number of main sound sources in the audio to be processed, thereby improving the accuracy of the processed target multi-channel audio and improving the processing effect of the audio processing method.

[0119] After introducing the basic process of the present disclosure, various non-limiting embodiments of the present disclosure are described in detail below.

[0120] In some embodiments, before step 101, the left channel audio and the right channel audio may be acquired in advance based on the audio to be processed, so that the target multi-channel audio may be acquired based on the left channel audio and the right channel audio. Optionally, the left channel audio and the right channel audio may be acquired through the following process.

[0121] When the audio to be processed is dual-channel audio, the left channel audio and the right channel audio included in the audio to be processed can be directly acquired.

[0122] In the case where the audio to be processed is multi-channel audio, the audio to be processed may be processed in advance to obtain left channel audio and right channel audio corresponding to the audio to be processed.

[0123] Taking the case where the audio to be processed is a multi-channel audio including 5 channels, the audio to be processed may include a left front channel audio, a right front channel audio, a center channel audio, a left surround channel audio, and a right surround channel audio, the left channel audio and the right channel audio corresponding to the audio to be processed may be obtained by the following processing method:

[0124] The center channel audio and the left surround channel audio are weighted respectively by preset weight parameters, and the left channel audio is determined based on the weighted result and the left front channel audio; and the center channel audio and the right surround channel audio are weighted respectively by preset weight parameters, and the right channel audio is determined based on the weighted result and the right front channel audio.

[0125] For example, the left channel audio and the right channel audio can be determined by the following formula (1) and formula (2) respectively:

[0126] L=FL+a×C+a×RL (1)

[0127] R=FR+a×C+a×RR (2)

[0128] Among them, L represents left channel audio, R represents right channel audio, FL represents left front channel audio, FR represents right front channel audio, C represents center channel audio, RL represents left surround channel audio, RR represents right surround channel audio, and a represents a preset weight parameter.

[0129] Optionally, the value of the preset weight parameter may be 0.71. Taking the preset weight parameter of 0.71 as an example, using 0.71 as the preset weight parameter to weight the audio is equivalent to attenuating the audio by 3 decibels (dB). When the preset parameter is 0.71, the principle of obtaining the left channel audio and the right channel audio can be seen in Figure 2 , Figure 2 is a schematic diagram of obtaining left and right channel audio based on audio to be processed according to an exemplary embodiment of the present disclosure, such as Figure 2As shown, when obtaining the left channel audio, the center channel audio can be attenuated by 3dB and then summed with the left front channel audio, and the summed result can be summed with the result of the left surround channel audio attenuated by 3dB to obtain the left channel audio; when obtaining the right channel audio, the center channel audio can be attenuated by 3dB and then summed with the right front channel audio, and the summed result can be summed with the result of the right surround channel audio attenuated by 3dB to obtain the right channel audio.

[0130] The above is only an exemplary method for obtaining left and right channel audio based on the audio to be processed. In more possible implementations, other methods can also be used to obtain left and right channel audio. The present disclosure does not limit which specific method is used.

[0131] After the left and right channel audios corresponding to the audio to be processed are obtained, the target multi-channel audio can be obtained based on the obtained left and right channel audios.

[0132] In some embodiments, for step 101, when determining the main component audio and surround component audio corresponding to the target multi-channel audio based on the left channel audio and the right channel audio corresponding to the audio to be processed, and the frequency band weight parameters corresponding to the frequency bands to which the main component audio and the surround component audio of the target multi-channel audio belong, the following steps may be included:

[0133] Step 1011: Determine the main component audio based on the left channel audio, the right channel audio, and the frequency band weight parameter corresponding to the frequency band to which the main component audio belongs.

[0134] The main component audio may include a main component left channel audio and a main component right channel audio. Based on this, for the above step 1011, when determining the main component audio, the following steps may be included:

[0135] Step 1011-1: Based on a first frequency band weight parameter for processing the left channel audio corresponding to the frequency band to which the main component audio belongs, and a second frequency band weight parameter for processing the right channel audio, weighted summation is performed on the left channel audio and the right channel audio to obtain the main component left channel audio.

[0136] For example, the main component left channel audio can be obtained by the following formula (3):

[0137]

[0138] in, represents the main component left channel audio, w1 represents the weight parameter of the first frequency band, X L represents the left channel audio, w2 represents the second frequency band weight parameter, X Rrepresents the right channel audio, b represents the frequency band number, k represents the frequency point number, and n represents the time series number.

[0139] Step 1011-2: Based on the third frequency band weight parameter for processing the left channel audio corresponding to the frequency band to which the main component audio belongs, and the fourth frequency band weight parameter for processing the right channel audio, weighted summation is performed on the left channel audio and the right channel audio to obtain the main component right channel audio.

[0140] For example, the main component right channel audio can be obtained by the following formula (4):

[0141]

[0142] in, represents the right channel audio of the main component, w3 represents the weight parameter of the third frequency band, X L represents the left channel audio, w4 represents the fourth frequency band weight parameter, X R represents the right channel audio, b represents the frequency band number, k represents the frequency point number, and n represents the time series number.

[0143] Step 1012: Determine the surround component audio based on the left channel audio, the right channel audio, and the frequency band weight parameter corresponding to the frequency band to which the surround component audio belongs.

[0144] The surround component audio may include a surround component left channel audio and a surround component right channel audio. Based on this, for the above step 1012, when determining the surround component audio, the following steps may be included:

[0145] Step 1012-1: Based on the fifth frequency band weight parameter for processing the left channel audio corresponding to the frequency band to which the surround component audio belongs, and the sixth frequency band weight parameter for processing the right channel audio, weighted summation is performed on the left channel audio and the right channel audio to obtain the surround component left channel audio.

[0146] For example, the surround component left channel audio can be obtained by the following formula (5):

[0147]

[0148] in, represents the left channel audio of the surround component, w5 represents the weight parameter of the fifth frequency band, X L represents the left channel audio, w6 represents the sixth frequency band weight parameter, X R represents the right channel audio, b represents the frequency band number, k represents the frequency point number, and n represents the time series number.

[0149] Step 1012-2: Based on the seventh frequency band weight parameter for processing the left channel audio corresponding to the frequency band to which the surround component audio belongs, and the eighth frequency band weight parameter for processing the right channel audio, weighted summation is performed on the left channel audio and the right channel audio to obtain the surround component left channel audio.

[0150] For example, the surround component right channel audio can be obtained by the following formula (6):

[0151]

[0152] in, represents the left channel audio of the surround component, w7 represents the weight parameter of the seventh frequency band, X L represents the left channel audio, w8 represents the eighth frequency band weight parameter, X R represents the right channel audio, b represents the frequency band number, k represents the frequency point number, and n represents the time series number.

[0153] It should be noted that the numbers of the above-mentioned steps do not constitute a limitation on the execution order of the above-mentioned two steps. Taking step 1011 and step 1012 as an example, optionally, step 1011 can be executed first and then step 1012, or step 1012 can be executed first and then step 1011, or step 1011 and step 1012 can be executed at the same time. The present disclosure does not limit the specific execution order.

[0154] Through the above process, the main component audio and the surround component audio can be determined according to the linear estimation method, and the main component audio and the surround component audio can be expressed as a linear combination of stereo signals (that is, multi-channel signals), so that the target multi-channel audio can be determined based on the determined main component audio and surround component audio.

[0155] The present invention separates the main component audio and the surround component audio by utilizing the frequency band weight parameters obtained by training the first audio processing network. The main component audio and the surround component audio with a high degree of separation can be obtained without obtaining the split track file. In addition, the separation of the main component audio and the surround component audio is achieved based on the audio processing network, thereby avoiding the problem of reduced separation due to imbalance in the weight of the separation signal when there are multiple main sound sources in the audio, thereby improving the audio separation effect.

[0156] In some embodiments, for step 102, when obtaining rendered audio based on the surround component audio and the target mapping parameters, it can be implemented in the following manner:

[0157] Based on the first target mapping parameter in the target mapping parameter for processing the left channel audio of the surround component, and the second target mapping parameter in the target mapping parameter for processing the right channel audio of the surround component, the left channel audio of the surround component and the right channel audio of the surround component are weightedly summed to obtain rendered audio.

[0158] Surround component audio can enhance the diversity of diffuse signals and create an immersive sound effect. When the surround component signal is rendered into multi-channel audio, a decorrelator can be used to generate decomposed diffuse signals that are independent of each other in each channel. For example, the surround component signal can be rendered into multi-channel audio by the following formula (7) to obtain rendered audio:

[0159]

[0160] in, Indicates rendering audio, o c，1 represents the first target mapping parameter, Indicates the left channel audio of the surround component, o c，2 represents the second target mapping parameter, Represents the right channel audio of the surround component, b represents the frequency band number, k represents the frequency point number, and n represents the time series number.

[0161] The target mapping parameters obtained by training the second audio processing network are used to render the surround component audio, thereby improving the rendering effect of the immersive audio.

[0162] In some embodiments, for step 103, when obtaining the target multi-channel audio corresponding to the audio to be processed based on the main component audio and the rendered audio, it can be implemented in the following manner:

[0163] The main component audio and the rendered audio are superimposed to obtain the target multi-channel audio corresponding to the audio to be processed.

[0164] For example, the target multi-channel audio can be obtained by the following formula (8):

[0165]

[0166] Among them, U c Indicates the target multi-channel audio, Represents the main component audio (including the main component left channel audio and the main component right channel audio), Represents the rendered audio obtained based on the surround component audio, b represents the frequency band number, k represents the frequency point number, and n represents the time series number.

[0167] Since the main component audio is a directional audio signal, when the main component audio is mapped to multi-channel audio, the spatial sound image orientation of the main component audio needs to be maintained. Therefore, it is necessary to estimate the direction of the main component audio and then map it to the left and right channels according to the angle, while other channels other than the left and right channels do not need to retain the main component audio. Therefore, the main component audio in the above formula (8) includes the main component left channel audio and the main component right channel audio, and the main component audio of other channels is 0 after mapping and can be ignored, which can ensure that the main component audio used to obtain the target multi-channel audio can truly restore the spatial position distribution of the original sound field.

[0168] The above process is an introduction on how to use the parameters obtained by pre-training to obtain the target multi-channel audio based on the audio to be processed. The following is an introduction to the process of obtaining parameters through pre-training.

[0169] In some embodiments, the training process of the frequency band weight parameters and the target mapping parameters can be found in Figure 3 , Figure 3 is a flow chart of a parameter training process according to an exemplary embodiment of the present disclosure. Figure 3 As shown, the parameter training process includes:

[0170] Step 301: Based on the sample multi-channel audio, obtain the sample left channel audio and the sample right channel audio.

[0171] The sample multi-channel audio may be audio in a multi-channel data set, and the sample multi-channel audio may include sample left front channel audio, sample right front channel audio, sample center channel audio, sample left surround channel audio, and sample right surround channel audio.

[0172] Taking the sample multi-channel audio including the above five channels as an example, the sample left channel audio and the sample right channel audio can be obtained by the following steps:

[0173] Step 3011: weight the sample center channel audio and the sample left surround channel audio respectively by using preset weight parameters, and determine the sample left channel audio based on the weighted result and the sample left front channel audio.

[0174] Step 3012: weight the sample center channel audio and the sample right surround channel audio respectively by using preset weight parameters, and determine the sample right channel audio based on the weighted result and the sample right front channel audio.

[0175] The process of obtaining the sample left and right channel audio through the above steps 3011 and 3012 can be referred to formula (1) and formula (2) and related introductions, which will not be repeated here.

[0176] Step 302: Based on the sample left channel audio and the sample right channel audio, determine a first sample audio feature, a second sample audio feature, and a third sample audio feature, wherein the first sample audio feature is used to indicate the power sum of the sample left channel audio and the sample right channel audio, the second sample audio feature is used to indicate the power difference between the sample left channel audio and the sample right channel audio, and the third sample audio feature is used to indicate the real cross-correlation power of the sample left channel audio and the sample right channel audio.

[0177] In a possible implementation, the first sample audio feature, the second sample audio feature, and the third sample audio feature may be determined by the following steps:

[0178] Step 3021: Determine a first sample audio feature and a second sample audio feature based on the sample left channel audio, the complex conjugate audio corresponding to the sample left channel audio, the sample right channel audio, the complex conjugate audio corresponding to the sample right channel audio, and a target smoothing parameter.

[0179] For example, the first sample audio feature and the second sample audio feature may be determined by the following formula (9) and formula (10), respectively:

[0180]

[0181]

[0182] Among them, Φ S represents the first sample audio feature (that is, the power sum of the sample left channel audio and the sample right channel audio), Φ D represents the second sample audio feature (that is, the power difference between the sample left channel audio and the sample right channel audio), X L Represents the sample left channel audio, Represents the complex conjugate audio of the left channel audio of the sample, X R Represents the sample right channel audio, represents the complex conjugate audio of the sample right channel audio, b represents the frequency band number, k represents the frequency point number, n represents the time series number, α represents the target smoothing parameter, optionally, α can be any integer value, and the present disclosure does not limit the specific value of α.

[0183] Step 3022: Determine a third sample audio feature based on the real part of the multiplication result of the sample left channel audio and the complex conjugate audio corresponding to the sample right channel audio, and the target smoothing factor.

[0184] For example, the third sample audio feature can be determined by the following formula (11):

[0185]

[0186] Among them, ΦXC represents the third sample audio feature (that is, the real part of the cross-correlation power of the sample left channel audio feature and the sample right channel), R represents the real part of the result, X L Represents the sample left channel audio, represents the complex conjugate audio of the sample right channel audio, b represents the frequency band number, k represents the frequency point number, n represents the time series number, α represents the target smoothing parameter, optionally, α can be any integer value, and the present disclosure does not limit the specific value of α.

[0187] Step 303: Determine the first sample audio and the second sample audio based on the sample left channel audio, the sample right channel audio, and multiple predicted frequency band weight parameters obtained by processing the first sample audio feature, the second sample audio feature, and the third sample audio feature through the first audio processing network.

[0188] The first sample audio is a sample main component audio, and the second sample audio is a sample surround component audio.

[0189] It should be noted that, before step 303, the first sample audio feature, the second sample audio feature and the third sample audio feature may be input into the first audio processing network, so as to obtain a plurality of predicted frequency band parameters through the first audio processing network.

[0190] In a possible implementation, the first audio processing network may be a deep neural network (DNN). The first audio processing network may include a first linear transformation layer and a second linear transformation layer, and the first linear transformation layer and the second linear transformation layer may both be composed of a linear transformation layer using a linear rectification (Rectified Linear Unit, ReLU) function as an activation function and a linear transformation layer using a normalization (Sigmoid) function as an activation function.

[0191] Taking the first audio processing network including the first linear transformation layer and the second linear transformation layer as an example, after the first sample audio feature, the second sample feature and the third sample audio feature are input into the first audio processing network, the first sample audio feature, the second sample feature and the third sample audio feature can be processed respectively by the first linear transformation layer and the second linear transformation layer included in the first audio processing network to obtain multiple predicted frequency band weight parameters for obtaining the first sample audio and multiple predicted frequency band weight parameters for obtaining the second sample audio, so that in step 303, the first sample audio and the second sample audio can be determined based on the multiple predicted frequency band weight parameters.

[0192] Optionally, when the first sample audio feature, the second sample audio feature and the third sample audio feature are input into the first audio processing network, a target feature can be obtained for the first sample audio feature, the second sample audio feature and the third sample audio feature, and the spliced ​​target feature can be input into the first audio processing network.

[0193] See also Figure 4 , Figure 4 is a schematic diagram of a network structure of a first audio processing network according to an exemplary embodiment of the present disclosure. Figure 4 As shown, the first linear transformation layer and the second linear transformation layer can be composed of three linear transformation layers using the ReLU function as the activation function and a linear transformation layer using the Sigmoid function as the activation function. Figure 4 When the first audio processing network shown in the figure is used to obtain multiple predicted frequency band weight parameters, the first sample audio feature, the second sample audio feature and the third sample audio feature can be respectively input into the first linear change layer and the second linear transformation layer, so as to obtain multiple predicted frequency band weight parameters for weighted processing of the sample left channel audio and multiple predicted frequency band weight parameters for weighted processing of the sample right channel audio through the first linear change layer and the second linear transformation layer, respectively. The internal processing processes of the first linear change layer and the second linear transformation layer are respectively described below.

[0194] For the first linear change layer, three linear transformation layers including the first linear change layer and using the ReLU function as the activation function can be used to perform linear rectification processing on the features of each input layer in turn to obtain rectified features, and then the rectified features can be normalized by using the linear transformation layer using the Sigmoid function as the activation function, and based on the features obtained by the normalization processing, multiple predicted frequency band weight parameters (including the first predicted frequency band weight parameter w1, the second predicted frequency band weight parameter w2, the third predicted frequency band weight parameter w3, and the fourth predicted frequency band weight parameter w4) for obtaining the first sample audio are determined.

[0195] For the second linear change layer, three linear transformation layers including the second linear change layer and using the ReLU function as the activation function can be used to perform linear rectification processing on the features of each input layer in turn to obtain rectified features, and then the rectified features can be normalized by using the linear transformation layer using the Sigmoid function as the activation function, and based on the features obtained by the normalization processing, multiple predicted frequency band weight parameters (including the fifth predicted frequency band weight parameter w5, the sixth predicted frequency band weight parameter w6, the seventh predicted frequency band weight parameter w7, and the eighth predicted frequency band weight parameter w8) for obtaining the second sample audio are determined.

[0196] After determining multiple predicted frequency band weight parameters for obtaining the first sample audio and multiple predicted frequency band weight parameters for obtaining the second sample audio, the first sample audio and the second sample audio can be determined based on the determined predicted frequency band parameters. The specific determination method can be found in the above step 101 and the introduction of formulas (3) to (6), which will not be repeated here.

[0197] Step 304: Obtain predicted rendering audio based on the second sample audio and the predicted mapping parameters obtained by processing the second sample audio through the second audio processing network.

[0198] It should be noted that, before step 304, the second sample audio may be input into the second audio processing network, so as to obtain the prediction mapping parameters through the second audio processing network.

[0199] In a possible implementation, the second audio processing network may be a DNN. The second audio processing network may include a linear transformation layer, which may be composed of a linear transformation layer using a ReLU function as an activation function and a linear transformation layer using a Sigmoid function as an activation function.

[0200] When obtaining the prediction mapping parameters through the second audio processing network, the second sample audio can be processed through the linear transformation layer included in the second audio processing network to obtain the prediction mapping parameters, so that in step 304, the rendered audio can be obtained based on the prediction mapping parameters.

[0201] See also Figure 5 , Figure 5 is a schematic diagram of a network structure of a second audio processing network according to an exemplary embodiment of the present disclosure. Figure 5 As shown, the linear transformation layer of the second audio processing network is composed of two linear transformation layers using the ReLU function as the activation function and a linear transformation layer using the Sigmoid function as the activation function. Figure 5 When the second audio processing network shown is used to obtain the prediction mapping parameters, the second sample audio represented in the form of a vector can be input into the linear transformation layer to obtain the prediction mapping parameters through the linear transformation layer. The internal processing process of the linear transformation layer is described below.

[0202] For example, the features of each layer can be linearly rectified in turn by two linear transformation layers including a linear transformation layer and using a ReLU function as an activation function to obtain rectified features, and then the rectified features can be normalized by a linear transformation layer using a Sigmoid function as an activation function, and the prediction mapping parameters (including the first prediction mapping parameter o) can be determined based on the features obtained by the normalization.c,1 and the second prediction mapping parameter o c,2 ).

[0203] After the prediction mapping parameters are determined, the rendered audio can be obtained based on the determined prediction mapping parameters. The specific acquisition method can refer to the above step 102 and the introduction of the relevant formula (7), which will not be repeated here.

[0204] Step 305: Obtain predicted multi-channel audio based on the first sample audio and the predicted rendering audio.

[0205] It should be noted that the implementation process of the above step 305 can refer to the introduction of step 103, which will not be repeated here.

[0206] Step 306: Based on the target loss function indicating the difference between the predicted multi-channel audio and the sample multi-channel audio, train the first audio processing network and the second audio processing network to obtain frequency band weight parameters and target mapping parameters.

[0207] The above step 306 can be implemented by the following steps:

[0208] Step 3061: Determine a first loss function based on the predicted multi-channel audio and the sample multi-channel audio.

[0209] The first loss function may be a minimum mean square error (MSE) loss function. For example, the first loss function may be represented by the following formula (12):

[0210]

[0211] Among them, L C represents the first loss function, Y C Represents sample multi-channel audio, U C Indicates the prediction of multi-channel audio, K represents the number of frequency bands, k represents the frequency point number, and n represents the time series number.

[0212] Step 3062: Determine a plurality of second loss functions based on the predicted amplitude differences between the various channels of the multi-channel audio and the amplitude differences between the various channels of the sample multi-channel audio.

[0213] The second loss function may be an Inter Channel Level Differences (ICLD) loss function. For example, the second loss function may refer to the following formula (13):

[0214]

[0215] Among them, L ICLDrepresents the second loss function, Represents the amplitude difference between any two channels of sampled multi-channel audio. It indicates the predicted amplitude difference between any two channels of multi-channel audio. K indicates the number of frequency bands, k indicates the frequency point number, and n indicates the time series number.

[0216] Taking the sample multi-channel audio and the predicted multi-channel audio as audio including 7 channels as an example, there are 21 second loss functions between the sample multi-channel audio and the predicted multi-channel audio.

[0217] Step 3063: Determine a target loss function based on the first loss function and the multiple second loss functions.

[0218] In a possible implementation, the sum of the first loss function and the plurality of second loss functions may be determined as the target loss function. For example, the target loss function may be determined by the following formula (14):

[0219]

[0220] Among them, L represents the target loss function, L C represents the first loss function, L ICLD represents the second loss function, and C represents multiple channels.

[0221] Step 3064: Train the first audio processing network and the second audio processing network based on the target loss function until a training cutoff condition is met, and obtain frequency band weight parameters and target mapping parameters.

[0222] It should be noted that, when the training cutoff condition is met, the trained first audio processing network and the second audio processing network can be obtained, and the predicted frequency band weight parameters output by the trained first audio processing network are the frequency band weight parameters to be obtained, and the predicted mapping parameters output by the trained second audio processing network are the target mapping parameters to be obtained.

[0223] The training cutoff condition may be that the function value of the target loss function satisfies the set condition, or that the number of iterations reaches the number of devices. Optionally, the training cutoff condition may also be other conditions, which are not limited in the present disclosure.

[0224] It should be noted that Figure 3The training process shown is an iterative process, that is, each time a sample left channel audio and a sample right channel audio are obtained through step 301, the obtained sample left channel audio and sample right channel audio can be processed through steps 302 to 305 to obtain a predicted multi-channel audio, and then through step 306, the first audio processing network and the second audio processing network are trained based on the target loss function, and then the next sample left channel audio and sample right channel audio obtained are processed through steps 302 to 305 to obtain a predicted multi-channel audio, and then through step 306, the trained first audio processing network and the second audio processing network are continued to be trained based on the target loss function, and so on, until the training cutoff condition is met, the trained first audio processing network and the second audio processing network can be obtained, and the frequency band weight parameters and the target mapping parameters can be obtained.

[0225] The present disclosure uses a deep neural network in the separation link of the main component audio and the surround component audio, so that the deep neural network can be used for channel separation, and the power sum of the input signal, the power difference of the input signal and the cross-correlation function of the input signal are defined as the input features of the deep neural network to train the deep neural network. The deep neural network includes multiple layers of linear transformation layers, and the linear transformation layers included in the deep neural network can be used to perform linear rectification processing on the input features of each layer. After iterative linear rectification processing is achieved through multiple layers of linear change layers, the audio output of different channels can be obtained through the linear transformation layer using the Sigmoid function as the activation function, so that channel separation can be achieved through the deep neural network, and then the advantages of the deep neural network can be utilized to improve the separation of audio of different channels through the training of the deep neural network, and at the same time, the occurrence of audio distortion and sound quality damage caused by time domain, frequency domain transformation and nonlinear processing of the audio can be reduced.

[0226] In addition, the present disclosure also introduces a deep neural network in the rendering link of the surround component audio. The introduction of the deep neural network can avoid the situation where the learning factors of some channels are empty when rendering the surround component audio when there are multiple main components in the audio. Therefore, the present disclosure can adapt to the situation where there are multiple main sound sources in the audio, thereby improving the audio quality of the generated multi-channel audio.

[0227] Based on the various optional embodiments involved in the above training process, the flow of the training process of the present disclosure can be found in Figure 6 , Figure 6 is a flowchart of a training process according to an exemplary embodiment of the present disclosure. Figure 6As shown, after obtaining the sample multi-channel audio from the multi-channel data set, step 301 can be used to down-mix based on the sample multi-channel audio to obtain sample left channel audio and sample right channel audio, so that the main component and the surround component can be separated through steps 302 to 303 to obtain a first sample audio as the main component audio and a second sample audio as the surround component, and then mapping is performed based on the first sample audio, and rendering is performed based on the second sample audio through step 304, so that through step 305, based on the result of mapping the first sample audio and the result of rendering the second sample audio, the predicted multi-channel audio is obtained, and then through step 306, based on the loss function indicating the difference between the predicted multi-channel audio and the sample multi-channel audio, training can be performed through back propagation. The specific implementation process can be referred to the above steps 301 to 306, which will not be repeated here.

[0228] After introducing the audio processing method according to the exemplary embodiment of the present disclosure, the structure of the audio processing apparatus according to the exemplary embodiment of the present disclosure and the computing device for implementing the audio processing method will be described next.

[0229] See also Figure 7 , Figure 7 is a block diagram of an audio processing device according to an exemplary embodiment of the present disclosure, the device comprising:

[0230] A determination module 701 is used to determine the main component audio and the surround component audio corresponding to the target multi-channel audio based on the left channel audio and the right channel audio corresponding to the audio to be processed, and the frequency band weight parameters corresponding to the frequency bands to which the main component audio and the surround component audio of the target multi-channel audio belong, respectively, wherein the frequency band weight parameters are obtained by training the first audio network, and the target multi-channel audio includes a target left channel audio part and a target right channel audio part;

[0231] A first acquisition module 702, configured to acquire rendered audio based on surround component audio and target mapping parameters, wherein the target mapping parameters are obtained by training a second audio network;

[0232] The second acquisition module 703 is used to acquire the target multi-channel audio corresponding to the audio to be processed based on the main component audio and the rendering audio corresponding to the target multi-channel audio.

[0233] In one embodiment of the present disclosure, the determination module 701, when used to determine the main component audio and the surround component audio corresponding to the target multi-channel audio based on the left channel audio and the right channel audio corresponding to the audio to be processed, and the frequency band weight parameters of the frequency bands to which the main component audio and the surround component audio of the target multi-channel audio belong, respectively, includes:

[0234] A first determining unit, configured to determine the main component audio based on the left channel audio, the right channel audio, and a frequency band weight parameter corresponding to the frequency band to which the main component audio belongs;

[0235] The second determination unit is used to determine the surround component audio based on the left channel audio, the right channel audio and the frequency band weight parameter corresponding to the frequency band to which the surround component audio belongs.

[0236] In one embodiment of the present disclosure, the main component audio includes a main component left channel audio and a main component right channel audio;

[0237] The first determination unit, when used to determine the main component audio based on the left channel audio, the right channel audio and the frequency band weight parameter corresponding to the frequency band to which the main component audio belongs, is used to:

[0238] Based on a first frequency band weight parameter for processing the left channel audio corresponding to the frequency band to which the main component audio belongs, and a second frequency band weight parameter for processing the right channel audio, weighted summing the left channel audio and the right channel audio is performed to obtain the main component left channel audio;

[0239] Based on the third frequency band weight parameter for processing the left channel audio corresponding to the frequency band to which the main component audio belongs, and the fourth frequency band weight parameter for processing the right channel audio, the left channel audio and the right channel audio are weightedly summed to obtain the main component right channel audio.

[0240] In one embodiment of the present disclosure, the surround component audio includes a surround component left channel audio and a surround component right channel audio;

[0241] The second determination unit, when used to determine the surround component audio based on the left channel audio, the right channel audio and the frequency band weight parameter corresponding to the frequency band to which the surround component audio belongs, is used to:

[0242] Based on a fifth frequency band weight parameter for processing the left channel audio corresponding to the frequency band to which the surround component audio belongs, and a sixth frequency band weight parameter for processing the right channel audio, weighted summing the left channel audio and the right channel audio is performed to obtain the surround component left channel audio;

[0243] Based on the seventh frequency band weight parameter for processing the left channel audio corresponding to the frequency band to which the surround component audio belongs, and the eighth frequency band weight parameter for processing the right channel audio, the left channel audio and the right channel audio are weightedly summed to obtain the surround component left channel audio.

[0244] In one embodiment of the present disclosure, the surround component audio includes a surround component left channel audio and a surround component right channel audio;

[0245] The first acquisition module 702, when used to acquire the rendered audio based on the surround component audio and the target mapping parameters, is used to:

[0246] Based on the first target mapping parameter in the target mapping parameter for processing the left channel audio of the surround component, and the second target mapping parameter in the target mapping parameter for processing the right channel audio of the surround component, the left channel audio of the surround component and the right channel audio of the surround component are weightedly summed to obtain rendered audio.

[0247] In one embodiment of the present disclosure, the second acquisition module 703, when used to acquire the target multi-channel audio corresponding to the audio to be processed based on the main component audio and the rendered audio, is used to:

[0248] The main component audio and the rendered audio are superimposed to obtain the target multi-channel audio corresponding to the audio to be processed.

[0249] In one embodiment of the present disclosure, the device further includes a training module, which includes:

[0250] A first acquisition unit, configured to acquire a sample left channel audio and a sample right channel audio based on the sample multi-channel audio;

[0251] A first determining unit is used to determine a first sample audio feature, a second sample audio feature, and a third sample audio feature based on the sample left channel audio and the sample right channel audio, the first sample audio feature is used to indicate the power sum of the sample left channel audio and the sample right channel audio, the second sample audio feature is used to indicate the power difference between the sample left channel audio and the sample right channel audio, and the third sample audio feature is used to indicate the real cross-correlation power of the sample left channel audio and the sample right channel audio;

[0252] A second determination unit is used to determine the first sample audio and the second sample audio based on the sample left channel audio, the sample right channel audio, and a plurality of predicted frequency band weight parameters obtained by processing the first sample audio feature, the second sample audio feature, and the third sample audio feature through the first audio processing network;

[0253] A second acquisition unit, configured to acquire predicted rendered audio based on the second sample audio and the predicted mapping parameters obtained by processing the second sample audio through the second audio processing network;

[0254] A third acquisition unit, configured to acquire predicted multi-channel audio based on the first sample audio and the predicted rendered audio;

[0255] A training unit is used to train the first audio processing network and the second audio processing network based on a target loss function indicating a difference between the predicted multi-channel audio and the sample multi-channel audio to obtain a frequency band weight parameter and a target mapping parameter.

[0256] In one embodiment of the present disclosure, the sample multi-channel audio includes sample left front channel audio, sample right front channel audio, sample center channel audio, sample left surround channel audio, and sample right surround channel audio;

[0257] The first acquisition unit, when used to acquire the sample left channel audio and the sample right channel audio based on the sample multi-channel audio, is used to:

[0258] The sample center channel audio and the sample left surround channel audio are weighted respectively by preset weight parameters, and the sample left channel audio is determined based on the weighted result and the sample left front channel audio;

[0259] The sample center channel audio and the sample right surround channel audio are weighted respectively by preset weight parameters, and the sample right channel audio is determined based on the weighted results and the sample right front channel audio.

[0260] In one embodiment of the present disclosure, the first determining unit, when used to determine the first sample audio feature, the second sample audio feature, and the third sample audio feature based on the sample left channel audio and the sample right channel audio, is used to:

[0261] Determine a first sample audio feature and a second sample audio feature based on the sample left channel audio, the complex conjugate audio corresponding to the sample left channel audio, the sample right channel audio, the complex conjugate audio corresponding to the sample right channel audio, and a target smoothing parameter;

[0262] A third sample audio feature is determined based on a real part of a multiplication result of the sample left channel audio and a complex conjugate audio corresponding to the sample right channel audio, and a target smoothing factor.

[0263] In one embodiment of the present disclosure, the training module further includes:

[0264] The first processing unit is used to input the first sample audio feature, the second sample feature and the third sample audio feature into the first audio processing network, and process the first sample audio feature, the second sample feature and the third sample audio feature through the first linear transformation layer and the second linear transformation layer included in the first audio processing network respectively to obtain multiple predicted frequency band weight parameters for obtaining the first sample audio and multiple predicted frequency band weight parameters for obtaining the second sample audio.

[0265] In one embodiment of the present disclosure, the device further includes:

[0266] The second processing unit is used to input the second sample audio into the second audio processing network, and process the second sample audio through the linear transformation layer included in the second audio processing network to obtain the prediction mapping parameter.

[0267] In one embodiment of the present disclosure, the training unit, when training the first audio processing network and the second audio processing network based on the target loss function indicating the difference between the predicted multi-channel audio and the sample multi-channel audio to obtain the frequency band weight parameter and the target mapping parameter, is used to:

[0268] Determining a first loss function based on the predicted multi-channel audio and the sample multi-channel audio;

[0269] Determining a plurality of second loss functions based on the predicted amplitude differences between the channels of the multi-channel audio and the amplitude differences between the channels of the sample multi-channel audio;

[0270] Determine a target loss function based on the first loss function and the plurality of second loss functions;

[0271] The first audio processing network and the second audio processing network are trained based on the target loss function until a training cutoff condition is met, thereby obtaining frequency band weight parameters and target mapping parameters.

[0272] It should be noted that although several modules / units of the audio processing device are mentioned in the above detailed description, this division is merely exemplary and not mandatory. In fact, according to an embodiment of the present disclosure, the features and functions of two or more modules / units described above may be embodied in one module / unit. Conversely, the features and functions of one module / unit described above may be further divided to be embodied by multiple modules / units.

[0273] The embodiment of the present disclosure also provides a computer-readable storage medium. Figure 8 is a schematic diagram of a computer-readable storage medium according to an exemplary embodiment of the present disclosure, such as Figure 8 As shown, a computer program 801 is stored on the storage medium, and when the computer program 801 is executed by a processor, the audio processing method provided by any embodiment of the present disclosure can be executed.

[0274] The present disclosure also provides a computing device, which may include a memory and a processor, wherein the memory is used to store computer instructions that can be run on the processor, and the processor is used to implement the audio processing method provided by any embodiment of the present disclosure when executing the computer instructions. Fig. 9 , Fig. 9 It is a structural diagram of a computing device according to an exemplary embodiment of the present disclosure. The computing device 900 may include but is not limited to: a processor 910, a memory 920, and a bus 930 connecting different system components (including the memory 920 and the processor 910).

[0275] The memory 920 stores computer instructions, which can be executed by the processor 910, so that the processor 910 can execute the audio processing method provided by any embodiment of the present disclosure. The memory 920 may include a random access memory unit RAM 921, a cache memory unit 922 and / or a read-only memory unit ROM 923. The memory 920 may also include: a program tool 925 having a set of program modules 924, and the program modules 924 include but are not limited to: an operating system, one or more application programs, other program modules and program data, and one or more combinations of these program modules may include the implementation of a network environment.

[0276] The bus 930 may include, for example, a data bus, an address bus, and a control bus. The computing device 900 may also communicate with an external device 950 through an I / O interface 940. The external device 950 may be, for example, a keyboard, a Bluetooth device, etc. The computing device 900 may also communicate with one or more networks through a network adapter 960. For example, the network may be a local area network, a wide area network, a public network, etc. Fig. 9 As shown, the network adapter 960 can also communicate with other modules of the computing device 900 via the bus 930 .

[0277] The embodiments of the present disclosure further provide a computer program product, which includes a computer program. When the program is executed by the processor 910 of the computing device 900, the audio processing method provided by any embodiment of the present disclosure can be implemented.

[0278] In addition, although the operations of the disclosed method are described in a specific order in the drawings, this does not require or imply that the operations must be performed in this specific order, or that all the operations shown must be performed to achieve the desired results. Additionally or alternatively, some steps may be omitted, multiple steps may be combined into one step, and / or one step may be decomposed into multiple steps.

[0279] Although the spirit and principle of the present disclosure have been described with reference to several specific embodiments, it should be understood that the present disclosure is not limited to the disclosed specific embodiments, and the division of various aspects does not mean that the features in these aspects cannot be combined to benefit, and such division is only for the convenience of expression. The present disclosure is intended to cover various modifications and equivalent arrangements included in the spirit and scope of the attached claims.

Claims

1. An audio processing method, characterized in that: The method comprises: Determine the main component audio and the surround component audio corresponding to the target multi-channel audio based on the left channel audio and the right channel audio corresponding to the audio to be processed, and the frequency band weight parameters corresponding to the frequency bands to which the main component audio and the surround component audio of the target multi-channel audio belong, wherein the frequency band weight parameters are obtained by training the first audio network, and the target multi-channel audio includes a target left channel audio part and a target right channel audio part; Obtaining rendered audio based on the surround component audio and target mapping parameters, wherein the target mapping parameters are obtained by training a second audio network; Based on the main component audio corresponding to the target multi-channel audio and the rendered audio, obtaining the target multi-channel audio corresponding to the audio to be processed; The step of obtaining target multi-channel audio corresponding to the audio to be processed based on the main component audio and the rendered audio includes: The main component audio and the rendered audio are superimposed to obtain a target multi-channel audio corresponding to the audio to be processed.

2. The method according to claim 1, characterized in that The method determines the main component audio and the surround component audio corresponding to the target multi-channel audio based on the left channel audio and the right channel audio corresponding to the audio to be processed, and the frequency band weight parameters corresponding to the frequency bands to which the main component audio and the surround component audio of the target multi-channel audio belong, including: Determine the main component audio based on the left channel audio, the right channel audio and a frequency band weight parameter corresponding to the frequency band to which the main component audio belongs; The surround component audio is determined based on the left channel audio, the right channel audio, and a frequency band weight parameter corresponding to the frequency band to which the surround component audio belongs.

3. The method according to claim 2, characterized in that The main component audio includes a main component left channel audio and a main component right channel audio; The determining the main component audio based on the left channel audio, the right channel audio, and a frequency band weight parameter corresponding to the frequency band to which the main component audio belongs includes: Based on a first frequency band weight parameter for processing the left channel audio corresponding to the frequency band to which the main component audio belongs, and a second frequency band weight parameter for processing the right channel audio, weighted summing the left channel audio and the right channel audio is performed to obtain the main component left channel audio; Based on the third frequency band weight parameter for processing the left channel audio corresponding to the frequency band to which the main component audio belongs, and the fourth frequency band weight parameter for processing the right channel audio, the left channel audio and the right channel audio are weightedly summed to obtain the main component right channel audio.

4. The method according to claim 2, characterized in that: The surround component audio includes a surround component left channel audio and a surround component right channel audio; The determining the surround component audio based on the left channel audio, the right channel audio, and a frequency band weight parameter corresponding to the frequency band to which the surround component audio belongs includes: Based on a fifth frequency band weight parameter for processing the left channel audio corresponding to the frequency band to which the surround component audio belongs, and a sixth frequency band weight parameter for processing the right channel audio, weighted summing the left channel audio and the right channel audio is performed to obtain the surround component left channel audio; Based on the seventh frequency band weight parameter for processing the left channel audio corresponding to the frequency band to which the surround component audio belongs, and the eighth frequency band weight parameter for processing the right channel audio, the left channel audio and the right channel audio are weightedly summed to obtain the surround component left channel audio.

5. The method according to claim 1, characterized in that The surround component audio includes a surround component left channel audio and a surround component right channel audio; The step of obtaining rendered audio based on the surround component audio and the target mapping parameters includes: Based on a first target mapping parameter in the target mapping parameters for processing the left channel audio of the surround component, and a second target mapping parameter in the target mapping parameters for processing the right channel audio of the surround component, a weighted sum is performed on the left channel audio of the surround component and the right channel audio of the surround component to obtain the rendered audio.

6. The method according to claim 1, characterized in that The training process of the frequency band weight parameter and the target mapping parameter includes: Based on the sample multi-channel audio, obtain the sample left channel audio and the sample right channel audio; Based on the sample left channel audio and the sample right channel audio, determine a first sample audio feature, a second sample audio feature, and a third sample audio feature, wherein the first sample audio feature is used to indicate the power sum of the sample left channel audio and the sample right channel audio, the second sample audio feature is used to indicate the power difference between the sample left channel audio and the sample right channel audio, and the third sample audio feature is used to indicate the real cross-correlation power of the sample left channel audio and the sample right channel audio; Determine a first sample audio and a second sample audio based on the sample left channel audio, the sample right channel audio, and a plurality of predicted frequency band weight parameters obtained by processing the first sample audio feature, the second sample audio feature, and the third sample audio feature through a first audio processing network; Obtaining predicted rendered audio based on the second sample audio and the predicted mapping parameters obtained by processing the second sample audio through a second audio processing network; Based on the first sample audio and the predicted rendered audio, obtaining predicted multi-channel audio; Based on a target loss function indicating a difference between the predicted multi-channel audio and the sample multi-channel audio, the first audio processing network and the second audio processing network are trained to obtain frequency band weight parameters and target mapping parameters.

7. The method according to claim 6, characterized in that The sample multi-channel audio includes sample left front channel audio, sample right front channel audio, sample center channel audio, sample left surround channel audio and sample right surround channel audio; The method of obtaining a sample left channel audio and a sample right channel audio based on the sample multi-channel audio comprises: The sample center channel audio and the sample left surround channel audio are weighted respectively by preset weight parameters, and the sample left channel audio is determined based on the weighted result and the sample left front channel audio; The sample center channel audio and the sample right surround channel audio are weighted respectively by the preset weight parameters, and the sample right channel audio is determined based on the weighted result and the sample right front channel audio.

8. The method according to claim 6, characterized in that The determining, based on the sample left channel audio and the sample right channel audio, a first sample audio feature, a second sample audio feature, and a third sample audio feature comprises: Determine the first sample audio feature and the second sample audio feature based on the sample left channel audio, the complex conjugate audio corresponding to the sample left channel audio, the sample right channel audio, the complex conjugate audio corresponding to the sample right channel audio, and a target smoothing parameter; The third sample audio feature is determined based on the real part of the multiplication result of the sample left channel audio and the complex conjugate audio corresponding to the sample right channel audio, and a target smoothing factor.

9. The method according to claim 6, characterized in that Before determining the first sample audio and the second sample audio based on the sample left channel audio, the sample right channel audio, and a plurality of predicted frequency band weight parameters obtained by processing the first sample audio feature, the second sample audio feature, and the third sample audio feature through a first audio processing network, the method further includes: The first sample audio feature, the second sample audio feature and the third sample audio feature are input into a first audio processing network, and the first sample audio feature, the second sample audio feature and the third sample audio feature are processed respectively through a first linear transformation layer and a second linear transformation layer included in the first audio processing network to obtain multiple predicted frequency band weight parameters for obtaining the first sample audio and multiple predicted frequency band weight parameters for obtaining the second sample audio.

10. The method according to claim 6, characterized in that Before obtaining the predicted rendered audio based on the second sample audio and the predicted mapping parameters obtained by processing the second sample audio through the second audio processing network, the method further includes: The second sample audio is input into the second audio processing network, and the second sample audio is processed through a linear transformation layer included in the second audio processing network to obtain the prediction mapping parameter.

11. The method according to claim 6, characterized in that The step of training the first audio processing network and the second audio processing network based on a target loss function indicating a difference between the predicted multi-channel audio and the sample multi-channel audio to obtain frequency band weight parameters and target mapping parameters comprises: Determining a first loss function based on the predicted multi-channel audio and the sample multi-channel audio; Determining a plurality of second loss functions based on the amplitude differences between the predicted multi-channel audio and the amplitude differences between the sample multi-channel audio; Determining the target loss function based on the first loss function and the plurality of second loss functions; The first audio processing network and the second audio processing network are trained based on the target loss function until a training cutoff condition is met, thereby obtaining the frequency band weight parameters and the target mapping parameters.

12. An audio processing device, characterized in that: The device comprises: A determination module, configured to determine the main component audio and the surround component audio corresponding to the target multi-channel audio based on the left channel audio and the right channel audio corresponding to the audio to be processed, and the frequency band weight parameters corresponding to the frequency bands to which the main component audio and the surround component audio of the target multi-channel audio belong, respectively, wherein the frequency band weight parameters are obtained by training the first audio network, and the target multi-channel audio includes a target left channel audio part and a target right channel audio part; A first acquisition module, configured to acquire rendered audio based on the surround component audio and target mapping parameters, wherein the target mapping parameters are obtained through training of a second audio network; A second acquisition module, configured to acquire a target multi-channel audio corresponding to the audio to be processed based on a main component audio corresponding to the target multi-channel audio and the rendered audio; Wherein, the second acquisition module, when used to acquire the target multi-channel audio corresponding to the audio to be processed based on the main component audio and the rendered audio, is used to: The main component audio and the rendered audio are superimposed to obtain a target multi-channel audio corresponding to the audio to be processed.

13. The device according to claim 12, characterized in that The determination module, when used to determine the main component audio and the surround component audio corresponding to the target multi-channel audio based on the left channel audio and the right channel audio corresponding to the audio to be processed, and the frequency band weight parameters of the frequency bands to which the main component audio and the surround component audio of the target multi-channel audio belong, respectively, includes: A first determining unit, configured to determine the main component audio based on the left channel audio, the right channel audio, and a frequency band weight parameter corresponding to the frequency band to which the main component audio belongs; The second determining unit is used to determine the surround component audio based on the left channel audio, the right channel audio and a frequency band weight parameter corresponding to the frequency band to which the surround component audio belongs.

14. The device according to claim 13, characterized in that The main component audio includes a main component left channel audio and a main component right channel audio; The first determination unit, when used to determine the main component audio based on the left channel audio, the right channel audio, and a frequency band weight parameter corresponding to the frequency band to which the main component audio belongs, is used to: Based on a first frequency band weight parameter for processing the left channel audio corresponding to the frequency band to which the main component audio belongs, and a second frequency band weight parameter for processing the right channel audio, weighted summing the left channel audio and the right channel audio is performed to obtain the main component left channel audio; Based on the third frequency band weight parameter for processing the left channel audio corresponding to the frequency band to which the main component audio belongs, and the fourth frequency band weight parameter for processing the right channel audio, the left channel audio and the right channel audio are weightedly summed to obtain the main component right channel audio.

15. The device according to claim 13, characterized in that The surround component audio includes a surround component left channel audio and a surround component right channel audio; The second determination unit, when used to determine the surround component audio based on the left channel audio, the right channel audio, and a frequency band weight parameter corresponding to the frequency band to which the surround component audio belongs, is used to: Based on a fifth frequency band weight parameter for processing the left channel audio corresponding to the frequency band to which the surround component audio belongs, and a sixth frequency band weight parameter for processing the right channel audio, weighted summing the left channel audio and the right channel audio is performed to obtain the surround component left channel audio; Based on the seventh frequency band weight parameter for processing the left channel audio corresponding to the frequency band to which the surround component audio belongs, and the eighth frequency band weight parameter for processing the right channel audio, the left channel audio and the right channel audio are weightedly summed to obtain the surround component left channel audio.

16. The device according to claim 12, characterized in that The surround component audio includes a surround component left channel audio and a surround component right channel audio; The first acquisition module, when used to acquire the rendered audio based on the surround component audio and the target mapping parameters, is used to: Based on a first target mapping parameter in the target mapping parameters for processing the left channel audio of the surround component, and a second target mapping parameter in the target mapping parameters for processing the right channel audio of the surround component, a weighted sum is performed on the left channel audio of the surround component and the right channel audio of the surround component to obtain the rendered audio.

17. The device according to claim 12, characterized in that The device also includes a training module, which includes: A first acquisition unit, configured to acquire a sample left channel audio and a sample right channel audio based on the sample multi-channel audio; a first determining unit, configured to determine a first sample audio feature, a second sample audio feature, and a third sample audio feature based on the sample left channel audio and the sample right channel audio, wherein the first sample audio feature is used to indicate a power sum of the sample left channel audio and the sample right channel audio, the second sample audio feature is used to indicate a power difference between the sample left channel audio and the sample right channel audio, and the third sample audio feature is used to indicate a real cross-correlation power of the sample left channel audio and the sample right channel audio; a second determining unit, configured to determine a first sample audio and a second sample audio based on the sample left channel audio, the sample right channel audio, and a plurality of predicted frequency band weight parameters obtained by processing the first sample audio feature, the second sample audio feature, and the third sample audio feature through a first audio processing network; A second acquisition unit, configured to acquire a predicted rendered audio based on the second sample audio and a predicted mapping parameter obtained by processing the second sample audio through a second audio processing network; A third acquisition unit, configured to acquire predicted multi-channel audio based on the first sample audio and the predicted rendered audio; A training unit is used to train the first audio processing network and the second audio processing network based on a target loss function indicating a difference between the predicted multi-channel audio and the sample multi-channel audio to obtain frequency band weight parameters and target mapping parameters.

18. The device according to claim 17, characterized in that The sample multi-channel audio includes sample left front channel audio, sample right front channel audio, sample center channel audio, sample left surround channel audio and sample right surround channel audio; The first acquisition unit, when used to acquire the sample left channel audio and the sample right channel audio based on the sample multi-channel audio, is used to: The sample center channel audio and the sample left surround channel audio are weighted respectively by preset weight parameters, and the sample left channel audio is determined based on the weighted result and the sample left front channel audio; The sample center channel audio and the sample right surround channel audio are weighted respectively by the preset weight parameters, and the sample right channel audio is determined based on the weighted result and the sample right front channel audio.

19. The device according to claim 17, characterized in that The first determining unit, when used to determine the first sample audio feature, the second sample audio feature, and the third sample audio feature based on the sample left channel audio and the sample right channel audio, is used to: Determine the first sample audio feature and the second sample audio feature based on the sample left channel audio, the complex conjugate audio corresponding to the sample left channel audio, the sample right channel audio, the complex conjugate audio corresponding to the sample right channel audio, and a target smoothing parameter; The third sample audio feature is determined based on the real part of the multiplication result of the sample left channel audio and the complex conjugate audio corresponding to the sample right channel audio, and a target smoothing factor.

20. The device according to claim 17, characterized in that The training module also includes: The first processing unit is used to input the first sample audio feature, the second sample audio feature and the third sample audio feature into a first audio processing network, and process the first sample audio feature, the second sample audio feature and the third sample audio feature through a first linear transformation layer and a second linear transformation layer included in the first audio processing network, respectively, to obtain multiple predicted frequency band weight parameters for obtaining the first sample audio and multiple predicted frequency band weight parameters for obtaining the second sample audio.

21. The device according to claim 17, characterized in that The device also includes: The second processing unit is used to input the second sample audio into the second audio processing network, and process the second sample audio through the linear transformation layer included in the second audio processing network to obtain the prediction mapping parameter.

22. The device according to claim 17, characterized in that The training unit, when used to train the first audio processing network and the second audio processing network based on a target loss function indicating a difference between the predicted multi-channel audio and the sample multi-channel audio to obtain a frequency band weight parameter and a target mapping parameter, is used to: Determining a first loss function based on the predicted multi-channel audio and the sample multi-channel audio; Determining a plurality of second loss functions based on the amplitude differences between the predicted multi-channel audio and the amplitude differences between the sample multi-channel audio; Determining the target loss function based on the first loss function and the plurality of second loss functions; The first audio processing network and the second audio processing network are trained based on the target loss function until a training cutoff condition is met, thereby obtaining the frequency band weight parameters and the target mapping parameters.

23. A computing device, characterized in that The computing device includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the operations performed by the audio processing method according to any one of claims 1 to 11 when executing the program.

24. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a program, and the program is used by a processor to execute the operations performed by the audio processing method according to any one of claims 1 to 11.

Images