Multi-channel upmixing method and system based on deep learning, storage medium and equipment

By using deep learning technology to process the dual-channel LC3 code stream on the Bluetooth receiver, it generates a multi-channel PCM signal, which solves the problems of high 5.1 channel code rate and insufficient user experience in Bluetooth audio technology, and realizes the use of Bluetooth devices Efficient multi-channel playback and immersion enhancement.

CN119007744APending Publication Date: 2024-11-22SHENZHEN BARROT TECH LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202410681255.2
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-05-29
Publication Date
2024-11-22

AI Technical Summary

Technical Problem

When the existing Bluetooth audio technology plays 5.1 channel surround sound, the high bit rate during the data flow process leads to a great challenge in the front end of the RF, which is prone to lag and interference. At the same time, the 5.1 channel format sound source is insufficient, and the user experience is insufficient.

Method used

Using a multi-channel upmixing method based on deep learning, decoding, feature extraction and neural network processing of the dual-channel LC3 code stream at the Bluetooth receiver end, generate PCM signals of multiple channels, and use time-frequency transformation and overlapping phases. Add technology to save bit rate and improve immersion.

Benefits of technology

Achieve support for 5.1 channel playback on low-power Bluetooth and classic Bluetooth devices, reducing over-the-air transmission bitrate, reducing interference in wireless environments, and providing better immersion than dual-channel stereo, suitable for multiple The sound channel is like the 7.1 channel processing.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119007744A_ABST
    Figure CN119007744A_ABST
Patent Text Reader

Abstract

The invention discloses a multi-channel up-mixing method and system based on deep learning, a storage medium and equipment, and belongs to the technical field of Bluetooth audio, and the method comprises the steps: decoding a dual-channel LC3 code stream to transform domain noise shaping, and obtaining left and right channel spectral coefficients; performing feature extraction on the dual-track LC3 code stream to obtain a sub-band energy sum, a sub-band energy difference and sub-band cross correlation; inputting the combined features including the sub-band energy sum, the sub-band energy difference, the sub-band cross-correlation, the LTPF enabling mark and the fundamental tone delay into a pre-training neural network model, and outputting sub-band gains of a plurality of sound channels; interpolating the sub-band gain of each sound channel to obtain frequency bin gains of a plurality of sound channels; generating spectral coefficients of a plurality of sound channels according to the frequency bin gain of each sound channel and the spectral coefficients of the left and right sound channels; and performing low-delay improved inverse discrete cosine transform on the spectral coefficient of each sound channel to obtain PCM signals of a plurality of sound channels. According to the invention, multi-channel upmixing is realized through a deep learning technology, and the immersion of the user is enhanced.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application belongs to the field of Bluetooth audio technology, and in particular, relates to a multi-channel upmixing method, system, storage medium and device based on deep learning. Background Art

[0002] The current mainstream Bluetooth audio encoders are as follows: SBC: mandatory requirement of the A2DP protocol, the most widely used, and must be supported by all Bluetooth audio devices, but the sound quality is average; AAC-LC: better sound quality and more widely used, many mainstream mobile phones support it, but compared with SBC, it occupies a larger memory and has a higher computational complexity. Many Bluetooth devices are based on embedded platforms with limited battery capacity, poor processor computing power and limited memory, and its patent fees are high; aptX series: better sound quality, but very high bit rate, aptX requires a bit rate of 384kbps, while aptX -HD has a bit rate of 576kbps, and it is a unique technology of Qualcomm and is relatively closed; LDAC: The sound quality is good, but the bit rate is also very high, which are 330kbps, 660kbps and 990kbps respectively. Since the wireless environment of Bluetooth devices is particularly complex, it is difficult to stably support such a high bit rate, and it is a unique technology of Sony and is also very closed; LHDC: The sound quality is good, but the bit rate is also very high, typically including 400kbps, 600kbps and 900kbps. Such a high bit rate puts high demands on the baseband / RF design of Bluetooth. Based on the above reasons, the Bluetooth International Alliance Bluetooth Sig and many manufacturers launched LC3, which is mainly for low-power Bluetooth and can also be used for classic Bluetooth. It has the advantages of lower latency, higher sound quality and coding gain, and no patent fees in the Bluetooth field, and has attracted the attention of many manufacturers.

[0003] Bluetooth speakers are now quite popular. In addition to enjoying stereo sound, people also want to improve their sense of immersion. 5.1-channel surround sound is a widely used audio format that can provide a good user experience. However, there are two major challenges in playing 5.1-channel audio sources on Bluetooth speakers. Challenge 1: If the audio source is in 5.1 format, the data flow used in Bluetooth speakers is as follows: (1) Bluetooth transmitter: 5.1-channel audio source -> decoded into 6-channel PCM signal -> audio encoding for each channel (such as LC3 encoding) -> transmit the encoded audio signal; (2) Bluetooth receiver: receive 6-channel audio signal -> decode -> play with Bluetooth speakers. The disadvantage of the data flow process in Bluetooth speakers is that the bit rate of air transmission is high. If each channel is compressed with the recommended bit rate of 124kbps, the total bit rate in the air is 744kbps. Although this is not a problem for LEAudio, the high bit rate poses a greater challenge to the Bluetooth RF front end, and the audio is prone to jamming and will interfere with other devices around. Challenge 2: There are relatively few audio sources in 5.1-channel format. The more popular audio sources on the Internet are mainly dual-channel, which provides a poor experience, especially the lack of immersion.

[0004] In the existing two-channel upmixing technology to multi-channel, PCA (principal component analysis) is used more frequently and is also used in Dolby surround sound decoders. Due to the time-varying characteristics of the correlation and nonlinear relationship between multi-channel signals, the surround sound generated is quite different from the original surround sound, and the upmixed audio lacks spatial sense and immersion. Summary of the invention

[0005] In response to the above-mentioned technical problems existing in the prior art, the present application provides a multi-channel upmixing method, system, storage medium and device based on deep learning. For signals with 5.1 channels or dual channels as the sound source, end-to-end deep learning technology is adopted to support the playback of 5.1 channel signals at the Bluetooth receiving end, thereby achieving a better user experience.

[0006] In order to achieve the above-mentioned purpose, the first technical solution adopted in the present application is: to provide a multi-channel upmixing method based on deep learning, including: at the Bluetooth receiving end, decoding the dual-channel LC3 code stream into transform domain noise shaping to obtain left channel spectral coefficients and right channel spectral coefficients; performing feature extraction on the dual-channel LC3 code stream according to the left channel spectral coefficients and the right channel spectral coefficients to obtain sub-band energy sum, sub-band energy difference and sub-band cross-correlation; inputting the combined features including sub-band energy sum, sub-band energy difference, sub-band cross-correlation, LTPF enable flag and pitch delay into a pre-trained neural network model, and outputting sub-band gains of multiple channels, wherein the LTPF enable flag and pitch delay are obtained in the decoding process; interpolating the sub-band gain of each channel to obtain frequency bin gains of multiple channels; generating spectral coefficients of multiple channels according to the frequency bin gain of each channel, the left channel spectral coefficient and the right channel spectral coefficient; performing a low-delay improved inverse discrete cosine transform on the spectral coefficient of each channel to obtain PCM signals of multiple channels.

[0007] The second technical solution adopted in the present application is to provide a multi-channel upmixing system based on deep learning, including: a module for decoding a dual-channel LC3 code stream into transform domain noise shaping at a Bluetooth receiving end to obtain a left channel spectral coefficient and a right channel spectral coefficient; a module for performing feature extraction on the dual-channel LC3 code stream according to the left channel spectral coefficient and the right channel spectral coefficient to obtain a sub-band energy sum, a sub-band energy difference, and a sub-band cross-correlation; a module for converting the sub-band energy sum, the sub-band energy difference, the sub-band cross-correlation, the LTPF enable flag, and the pitch delay The combined features of are input into a pre-trained neural network model, and the sub-band gains of multiple channels are output, wherein the LTPF enable flag and the pitch delay are obtained in the decoding process; a module for interpolating the sub-band gains of each channel to obtain frequency bin gains of multiple channels; a module for generating spectral coefficients of multiple channels according to the frequency bin gain of each channel, the spectral coefficient of the left channel and the spectral coefficient of the right channel; a module for performing a low-delay improved inverse discrete cosine transform on the spectral coefficient of each channel to obtain PCM signals of multiple channels.

[0008] The third technical solution adopted in the present application is: providing a computer-readable storage medium storing computer instructions, wherein the computer instructions are operated to execute the multi-channel upmixing method based on deep learning in solution one.

[0009] The fourth technical solution adopted in the present application is: providing a computer device, which includes a processor and a memory, the memory stores computer instructions, wherein the processor operates the computer instructions to execute the deep learning-based multi-channel upmixing method in solution one.

[0010] The beneficial effects that can be achieved by the technical solution of the present application are: the technical solution of the present application can be applied to both low-power Bluetooth and classic Bluetooth, and by utilizing the existing time-frequency transformation and overlap-addition technology, the increase in algorithm delay is avoided; by adopting deep learning technology, for the original sound source of 5.1 channels, the bit rate of air transmission can be saved, and only the dual-channel bit rate needs to be transmitted, reducing interference in the wireless environment. For the original sound source of dual channels, it can provide the audience with a better sense of immersion than dual-channel stereo, and it can not only process 5.1 channels, but also other multi-channels, such as 7.1 channels. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] Figure 1 It is a flowchart of a specific implementation method of the multi-channel upmixing method based on deep learning of the present application;

[0012] Figure 2 is a schematic diagram of an example of a training process of a pre-trained neural network model in the present application;

[0013] Figure 3 is a schematic diagram of an example of the online reasoning process of the present application at the Bluetooth receiving end;

[0014] Figure 4 It is a schematic diagram of a specific implementation of the multi-channel upmixing system based on deep learning of the present application. DETAILED DESCRIPTION

[0015] The preferred embodiments of the present application are described in detail below in conjunction with the accompanying drawings so that the advantages and features of the present application can be more easily understood by those skilled in the art, thereby making a clearer and more definite definition of the protection scope of the present application.

[0016] A Bluetooth transmitter generally refers to a device with Bluetooth transmission function, such as a Bluetooth transceiver, mobile phone, tablet computer, laptop computer, etc.

[0017] The Bluetooth receiver generally refers to a Bluetooth speaker, or a device with a Bluetooth speaker, such as a car Bluetooth playback system.

[0018] This application takes 5.1 channels as an example, but the principle can also be applied to a larger or smaller number of channels, such as 7.1 channels.

[0019] A typical 5.1 channel includes the front left channel (Front Left Channel), the front center channel (Front Center Channel), the front right channel (Front Right Channel), the back left surround channel (Back Left Surround), the back right surround channel (Back Right Surround) and the subwoofer channel (Low Frequency Effect).

[0020] The idea of ​​this application is as follows:

[0021] 1. In the training phase (usually based on PC or server training), the neural network is trained based on the 5.1 surround channel signal and its downmixed two-channel stereo signal to achieve the mapping of two-channel signal to multi-channel signal;

[0022] 2. In the inference phase (usually on a device with Bluetooth reception):

[0023] a) Firstly, feature extraction is performed based on the binaural signal;

[0024] b) then using a neural network to generate multi-channel sub-band gains;

[0025] c) Sub-band gain interpolation obtains the gain of all frequency bins;

[0026] d) generating multi-channel spectral coefficients using frequency bin gains and binaural spectral coefficients;

[0027] e) performing a low-delay improved inverse discrete cosine transform on the multi-channel spectral coefficients to obtain a time-domain PCM signal;

[0028] f) Finally, the front center channel is low-pass filtered to generate a subwoofer channel, thereby achieving upmixing from two-channel stereo to 5.1-channel surround sound.

[0029] Figure 1 It is a flowchart of a specific implementation of the multi-channel upmixing method based on deep learning of the present application.

[0030] exist Figure 1 In a specific embodiment shown, the deep learning-based multi-channel upmixing method of the present application includes process S101, at the Bluetooth receiving end, decoding the dual-channel LC3 code stream into transform domain noise shaping to obtain left channel spectral coefficients and right channel spectral coefficients.

[0031] In this specific implementation, a dual-channel LC3 code stream, i.e., a left channel code stream and a right channel code stream, is obtained at the Bluetooth receiving end through a Bluetooth communication module, and then the dual-channel LC3 code stream is partially decoded to transform domain noise shaping to obtain left channel spectral coefficients and right channel spectral coefficients:

[0032] X left (k), X right (k), k=0…N F -1

[0033] exist Figure 1 In a specific embodiment shown, the deep learning-based multi-channel upmixing method of the present application includes process S102, performing feature extraction on the dual-channel LC3 code stream according to the left channel spectral coefficients and the right channel spectral coefficients to obtain the sub-band energy sum, sub-band energy difference and sub-band cross-correlation.

[0034] In a specific embodiment of the present application, feature extraction is performed on a dual-channel LC3 code stream according to left channel spectral coefficients and right channel spectral coefficients to obtain subband energy sums, subband energy differences and subband cross-correlations, including: according to the Bark frequency, the left channel spectral coefficients and the right channel spectral coefficients are divided into a predetermined number of subbands; according to the left channel spectral coefficients, the right channel spectral coefficients and each subband, the left channel subband energy, the right channel subband energy and the subband cross-correlation are obtained; the left channel subband energy and the right channel subband energy are subjected to corresponding predetermined operations to obtain the subband energy sums and subband energy differences.

[0035] Specifically, firstly, the left channel spectral coefficients and the right channel spectral coefficients are divided into a predetermined number of sub-bands according to the Bark frequency.

[0036] In a specific example of the present application, taking the sampling rate as 48 kHz and the frame length as 10 ms as an example, 480 spectral coefficients are output per frame, that is, N F =480, and is divided into 21 subbands according to the Bark frequency (the subband numbers are b=0, 1, 2, ..., 20). The number of spectral coefficients in each subband is: 4, 4, 4, 4, 4, 4, 4, 8, 8, 8, 16, 16, 16, 24, 24, 32, 48, 64, 96.

[0037] Then, the subband energy is calculated based on the left channel spectral coefficient, the right channel spectral coefficient and each subband. Take the left channel subband energy as an example:

[0038]

[0039] Where b is the subband number, N b is the number of subbands.

[0040] Right channel subband energy E B,right The formula of (b) is similar to the left channel subband energy and is omitted here.

[0041] Then, the energy sum of the left and right channels is calculated, that is, the sub-band energy sum:

[0042] Esum (m,b)=(1-α)E sum (m-1,b)+α(E B,left (b)+E B,right (b))

[0043] Where m is the frame number, m represents the current frame, m-1 represents the previous frame, and α is the smoothing coefficient, which can be 0.1 to 0.2.

[0044] Calculate the energy difference between the left and right channels, that is, the sub-band energy difference:

[0045] E diff (m,b)=(1-α)E diff (m-1,b)+α(E B,left (b)-E B,right (b))

[0046] According to the left channel spectral coefficient, the right channel spectral coefficient and each sub-band, the cross-correlation of the left and right channels, that is, the sub-band cross-correlation, is calculated:

[0047] First, the cross-correlation between the left and right channels is calculated in each subband:

[0048]

[0049] Then smoothing is performed to obtain the smoothed subband cross-correlation:

[0050] XCORR(m,b)=(1-α)XCORR(m-1,b)+αXCORR B (b)

[0051] Among them, performing smoothing can avoid noise caused by sudden changes.

[0052] exist Figure 1 In a specific embodiment shown, the deep learning-based multi-channel upmixing method of the present application includes process S103, which inputs the combined features including sub-band energy sum, sub-band energy difference, sub-band cross-correlation, LTPF enable flag and pitch delay into a pre-trained neural network model, and outputs sub-band gains of multiple channels, wherein the LTPF enable flag and pitch delay are obtained during the decoding process.

[0053] In this specific embodiment, the combined features input into the pre-trained neural network model include the sub-band energy sum, sub-band energy difference and sub-band cross-correlation obtained above, as well as the LTPF enable flag ltpf_active and pitch delay p which can be obtained during the decoding process without separate calculation. fr , these features are combined together to form a combined feature, including sub-band energy sum, sub-band energy difference, sub-band cross-correlation, LTPF enable flag and pitch delay.

[0054] In a specific embodiment of the present application, the multiple channels include: a front left channel, a front right channel, a front center channel, a rear left surround channel, and a rear right surround channel.

[0055] The above combined features are then input into the pre-trained neural network model to output the gain corresponding to each sub-band, that is, the sub-band gains of multiple channels.

[0056] Front left channel: g FL,L,B (b),g FL,R,B (b), b=0…N b -1

[0057] Front right channel: g FR,L,B (b),g FR,R,B (b), b=0…N b -1

[0058] Front center channel: g FC,L,B (b),g FC,R,B (b), b=0…N b -1

[0059] Rear left surround: g BL,L,B (b),g BL,R,B (b), b=0…N b -1

[0060] Rear right surround: g BR,L,B (b),g BR,R,B (b), b=0…N b -1

[0061] Among them, taking the front left channel as an example, FL means Front Left, that is, this gain is used to generate the front left channel, L means Left, that is, the left channel in the dual-channel signal needed to generate the front left channel, B means this is the sub-band gain, and b means the sub-band number.

[0062] In a specific embodiment of the present application, the training process of the pre-trained neural network model includes: obtaining a multi-channel sound source, and decoding the multi-channel sound source to obtain time-domain PCM signals of multiple channels; performing feature extraction on the time-domain PCM signal of each channel to obtain a multi-channel feature; performing downmixing on the time-domain PCM signals of multiple channels to obtain a two-channel stereo signal; performing feature extraction on the two-channel stereo signal to obtain a two-channel feature; inputting the multi-channel feature and the two-channel feature into the neural network for training, and calculating the loss of the neural network based on the output of the neural network; and adjusting the weight and bias of the neural network based on the loss, and when the loss meets the training termination condition, stopping the training to obtain the pre-trained neural network model.

[0063] Figure 2It is a schematic diagram of an example of the training process of the pre-trained neural network model in this application.

[0064] like Figure 2 As shown, firstly, a multi-channel sound source is obtained, and down-mixing is performed on the multi-channel sound source to obtain a dual-channel signal, and then feature extraction 2 is performed on the multi-channel sound source to obtain a multi-channel feature, and feature extraction 3 is performed on the dual-channel signal to obtain a dual-channel feature, and then the multi-channel feature and the dual-channel feature are input into a neural network for training, and a sub-band gain is output, and the sub-band gain is interpolated, and a new spectral coefficient is generated according to the dual-channel spectral coefficient, and LD-IMDCT (low delay improved inverse discrete cosine transform) is performed on the new spectral coefficient to obtain a time domain PCM, and then the frequency domain loss and the time domain loss are calculated according to the multi-channel feature, the new spectral coefficient and the time domain PCM, and then the total loss of the neural network is obtained, and then the weight and bias of the neural network are adjusted according to the total loss, and when the loss reaches the training termination condition, the training is stopped to obtain a pre-trained neural network model.

[0065] Combine the following Figure 2 A detailed description of the training process of the pre-trained neural network model:

[0066] Specifically, first obtain a certain number of multi-channel sound sources as training materials, such as 5.1-channel sound sources. Then perform feature extraction, that is, Figure 2 Feature extraction in 2 includes two parts:

[0067] (1) Decode the multi-channel sound source to obtain the time domain PCM signal: FL (n), x FR (n), x FC (n), x BL (n), x BR (n), which is the time domain data of the multi-channel feature, and is subsequently used to calculate the time domain loss.

[0068] (2) Encode and decode the above time domain signal into transform domain noise shaping output spectrum coefficients, X FL (n), X FR (n), X FC (n), X BL (n), X BR (n), which is the spectral coefficient of the multi-channel feature, and is subsequently used to calculate the frequency domain loss.

[0069] The result of feature extraction 2 is a multi-channel feature, including time domain data of the multi-channel feature and spectral coefficients of the multi-channel feature.

[0070] Then, the time-domain PCM signals of the above multiple channels are down-mixed into two-channel stereo signals. This is a mature technology. One method of down-mixing 5.1 channels into two channels is briefly described as follows:

[0071] Left channel x left (n) = x FL (n)*0.4142+x FR (n)*0.0000+x FC (n)*0.2929+lfe(n)*

[0072] 0.0000+x BL (n)*0.2929+x BR (n)*0.0000

[0073] Right channel x right (n) = x FL (n)*0.0000+x FR (n)*0.4142+x FC (n)*0.2929+lfe(n)*

[0074] 0.0000+x BL (n)*0.0000+x BR (n)*0.2929

[0075] Then, feature extraction is performed on the obtained two-channel stereo signal, namely Figure 2 Feature extraction 3 in , firstly, LC3 encoding is performed on each channel of the two-channel stereo signal, and the bit stream is output. Then, partial LC3 decoding is performed on the bit stream to transform domain noise shaping, and the two-channel spectrum coefficients are output:

[0076]

[0077] Then, the binaural spectrum coefficients are divided into subbands, and the subband energy sum, subband energy difference and subband cross-correlation are calculated, wherein the calculation method of the subband energy sum, subband energy difference and subband cross-correlation is the same as the calculation method described above, which is omitted here and will not be repeated. Then, the subband energy sum, subband energy difference, subband cross-correlation, LTPF enable flag and pitch delay are combined into binaural features, wherein the LTPF enable flag and pitch delay are obtained during the decoding process.

[0078] The multi-channel features and binaural features obtained above are then input into the neural network for training, and the weights and biases of the neural network are adjusted using the back propagation algorithm. When the loss is less than the preset value or reaches the preset number of training times, the training is stopped to obtain a pre-trained neural network model.

[0079] The input of the neural network is binaural features, that is, Figure 2The result of feature extraction 3 is output as the estimated multi-channel sub-band gain. The multi-channel spectral coefficients are reconstructed based on the estimated multi-channel sub-band gain and the decoded dual-channel spectral coefficients, which are subsequently used to calculate the frequency domain loss. The multi-channel spectral coefficients are subjected to a low-latency improved inverse discrete cosine transform to generate a time domain PCM signal, which is subsequently used to calculate the time domain loss. Considering the front-to-back correlation of the audio, in addition, surround sound generally contains multiple reflections of the wall. Each input feature is a certain number of continuous frames. The following are two typical configurations: if the frame length is 10ms, it can generally be 10 frames; if the frame length is 7.5ms, it can generally be 15 frames. The actual application scenario can also be selected according to requirements (such as the complexity requirements of the system).

[0080] In a specific embodiment of the present application, the loss includes a total loss of multiple channels in the time domain and a total loss of multiple channels in the frequency domain.

[0081] The loss function (i.e. error) used during back propagation uses the sum of frequency domain and time domain losses, which is defined as:

[0082] Frequency domain loss of the front left channel:

[0083]

[0084] Where X FL (m,k) is the kth spectral coefficient of the mth frame, Figure 2 The spectral coefficient of the front left channel in the multi-channel spectral coefficient feature obtained by feature extraction 2, To reconstruct the multi-channel spectral coefficients, the frequency domain loss calculation method of other channels is similar and is omitted here.

[0085] Total loss in frequency domain:

[0086] L Freq,tot =L Freq,FL +L Freq,FR +L Freq,FC +L Freq,BL +L Freq,BR

[0087] Time domain loss of the front left channel:

[0088]

[0089] Total loss in time domain:

[0090] L Time,tot =L Time,FL +L Time,FR +L Time,FC +L Time,BL +L Time,BR

[0091] Finally, the total loss is:

[0092] Loss = L Freq,tot +βL Time,tot

[0093] β is used to adjust the weight of the time domain loss. Generally speaking, whether the time domain waveform completely matches the original waveform has no sensitivity to the hearing experience as the frequency domain. The typical value can be 0.2 to 0.3.

[0094] In addition, the application does not limit the choice of neural network. Considering the front-to-back correlation characteristics of audio frames, recurrent neural network (RNN) or its improved versions LSTM, GRU, etc. are preferably selected.

[0095] exist Figure 1 In a specific implementation shown, the multi-channel upmixing method based on deep learning of the present application includes a process S104 of interpolating the sub-band gain of each channel to obtain frequency bin gains of multiple channels.

[0096] In this specific implementation, the above sub-band gains are interpolated to obtain the gain of each frequency bin. Taking the left channel as an example, there are 21x2 sub-band gains in total, and 400x2 frequency bin gains need to be obtained after interpolation. The interpolation method is mature and is omitted here.

[0097] g FL,L,B (b),g FL,R,B (b), b=0…N b -1

[0098] Interpolation yields:

[0099] g FL,L,Freq (k),g FL,R,Freq (k), k=0…N F -1

[0100] Among them, Freq means that this gain corresponds to the frequency bin, that is, Frequency Bin.

[0101] exist Figure 1 In a specific embodiment shown, the deep learning-based multi-channel upmixing method of the present application includes a process S105 of generating spectral coefficients of multiple channels according to the frequency bin gain, left channel spectral coefficient and right channel spectral coefficient of each channel.

[0102] In a specific embodiment of the present application, spectral coefficients of multiple channels are generated according to the frequency bin gain, left channel spectral coefficient and right channel spectral coefficient of each channel, including: adding the product of the left gain of the frequency bin gain of each channel and the left channel spectral coefficient and the product of the right gain of the frequency bin gain of each channel and the right channel spectral coefficient to obtain the spectral coefficients of multiple channels.

[0103] Specifically, the spectral coefficients of the multi-channel signal are reconstructed according to the frequency bin gain of each channel and the binaural spectral coefficients (ie, the left channel spectral coefficients and the right channel spectral coefficients):

[0104]

[0105] exist Figure 1 In a specific embodiment shown, the deep learning-based multi-channel upmixing method of the present application includes process S106, performing a low-delay improved inverse discrete cosine transform on the spectral coefficients of each channel to obtain PCM signals of multiple channels.

[0106] In this specific implementation, based on the spectral coefficients of the above-mentioned multiple channels, an inverse time-frequency transform, namely a low-delay improved discrete cosine transform (LD-IMDCT), is performed to obtain a multi-channel audio PCM signal for output by an audio device (such as a Bluetooth speaker).

[0107]

[0108] In a specific embodiment of the present application, the multi-channel upmixing method based on deep learning of the present application also includes: performing low-pass filtering on the PCM signal of the front center channel to generate a PCM signal of the subwoofer channel.

[0109] In this specific embodiment, the PCM signal of the front center channel is low-pass filtered to generate the PCM signal of the subwoofer channel, thereby achieving upmixing from two-channel stereo to 5.1-channel surround sound.

[0110]

[0111] Figure 3 It is a schematic diagram of an example of the online reasoning process of the present application at the Bluetooth receiving end.

[0112] The generation process of the PCM signal of the left channel is used as an example. The other channels are similar and will not be described in detail. Figure 3As shown, first, at the Bluetooth receiving end, the left channel bitstream and the right channel bitstream are partially decoded to transform domain noise shaping respectively, so as to obtain the left channel MDCT spectral coefficient and the right channel MDCT spectral coefficient, and then feature extraction 1 is performed, and the extracted features (including subband energy sum, subband energy difference, subband cross-correlation, LTPF enable flag and pitch delay) are input into the pre-trained neural network model to obtain subband gain, and then the subband gain is interpolated to obtain frequency bin gain, and according to the left channel MDCT spectral coefficient and the right channel MDCT spectral coefficient, the front left channel spectral coefficient is generated, and the low-delay improved inverse discrete cosine transform is performed on the front left channel spectral coefficient, so as to output the PCM signal of the front left channel.

[0113] In the multi-channel upmixing method based on deep learning of the present application, the increase of algorithm delay is avoided by utilizing the existing time-frequency transformation and overlap-addition technology; by adopting deep learning technology, for the original sound source of 5.1 channels, the bit rate of air transmission can be saved, and only the dual-channel bit rate needs to be transmitted, thereby reducing interference in the wireless environment. For the original sound source of the dual-channel, it can provide the audience with a better sense of immersion than dual-channel stereo, and it can not only process 5.1 channels, but also other multi-channels, such as 7.1 channels.

[0114] Figure 4 It is a schematic diagram of a specific implementation of the multi-channel upmixing system based on deep learning of the present application.

[0115] exist Figure 4 In a specific embodiment shown in the figure, the multi-channel upmixing system based on deep learning of the present application includes: a module 401 for decoding a dual-channel LC3 code stream into transform domain noise shaping at a Bluetooth receiving end to obtain a left channel spectral coefficient and a right channel spectral coefficient; a module 402 for performing feature extraction on the dual-channel LC3 code stream according to the left channel spectral coefficient and the right channel spectral coefficient to obtain a sub-band energy sum, a sub-band energy difference, and a sub-band cross-correlation; a module 403 for extracting a combined feature including the sub-band energy sum, the sub-band energy difference, the sub-band cross-correlation, the LTPF enable flag, and the pitch delay. A module 403 is used to input a pre-trained neural network model and output sub-band gains of multiple channels, wherein the LTPF enable flag and the pitch delay are obtained in the decoding process; a module 404 is used to interpolate the sub-band gains of each channel to obtain frequency bin gains of multiple channels; a module 405 is used to generate spectral coefficients of multiple channels according to the frequency bin gains of each channel, the spectral coefficients of the left channel and the spectral coefficients of the right channel; and a module 406 is used to perform a low-delay improved inverse discrete cosine transform on the spectral coefficients of each channel to obtain PCM signals of multiple channels.

[0116] In a specific embodiment of the present application, feature extraction is performed on a dual-channel LC3 code stream according to left channel spectral coefficients and right channel spectral coefficients to obtain subband energy sums, subband energy differences and subband cross-correlations, including: according to the Bark frequency, the left channel spectral coefficients and the right channel spectral coefficients are divided into a predetermined number of subbands; according to the left channel spectral coefficients, the right channel spectral coefficients and each subband, the left channel subband energy, the right channel subband energy and the subband cross-correlation are obtained; the left channel subband energy and the right channel subband energy are subjected to corresponding predetermined operations to obtain the subband energy sums and subband energy differences.

[0117] In a specific embodiment of the present application, the plurality of channels include: a front left channel, a front right channel, a front center channel, a rear left surround channel, and a rear right surround channel.

[0118] In a specific embodiment of the present application, the training process of the pre-trained neural network model includes: obtaining a multi-channel sound source, and decoding the multi-channel sound source to obtain time-domain PCM signals of multiple channels; performing feature extraction on the time-domain PCM signal of each channel to obtain a multi-channel feature; performing downmixing on the time-domain PCM signals of multiple channels to obtain a two-channel stereo signal; performing feature extraction on the two-channel stereo signal to obtain a two-channel feature; inputting the multi-channel feature and the two-channel feature into the neural network for training, and calculating the loss of the neural network based on the output of the neural network; and adjusting the weight and bias of the neural network based on the loss, and when the loss meets the training termination condition, stopping the training to obtain the pre-trained neural network model.

[0119] In a specific embodiment of the present application, the loss includes a total loss of multiple channels in the time domain and a total loss of multiple channels in the frequency domain.

[0120] In a specific embodiment of the present application, spectral coefficients of multiple channels are generated according to the frequency bin gain, left channel spectral coefficient and right channel spectral coefficient of each channel, including: adding the product of the left gain of the frequency bin gain of each channel and the left channel spectral coefficient and the product of the right gain of the frequency bin gain of each channel and the right channel spectral coefficient to obtain the spectral coefficients of multiple channels.

[0121] In a specific embodiment of the present application, the deep learning-based multi-channel upmixing system of the present application also includes: a module for performing low-pass filtering on the PCM signal of the front center channel to generate a PCM signal of the subwoofer channel.

[0122] In the multi-channel upmixing system based on deep learning of the present application, the increase of algorithm delay is avoided by utilizing the existing time-frequency transformation and overlap-addition technology; by adopting deep learning technology, for the original sound source of 5.1 channels, the bit rate of air transmission can be saved, and only the dual-channel bit rate needs to be transmitted, thereby reducing interference in the wireless environment. For the original sound source of the dual-channel, it can provide the audience with a better sense of immersion than dual-channel stereo, and it can not only process 5.1 channels, but also other multi-channels, such as 7.1 channels.

[0123] In a specific embodiment of the present application, a computer-readable storage medium stores computer instructions, which are operated to execute the multi-channel upmixing method based on deep learning described in any embodiment. The storage medium can be directly in hardware, in a software module executed by a processor, or in a combination of the two.

[0124] The software modules may reside in RAM memory, flash memory, ROM memory, EPROM memory, EEPROM memory, registers, hard disk, removable disk, CD-ROM or any other form of storage medium known in the art. An exemplary storage medium is coupled to the processor so that the processor can read information from and write information to the storage medium.

[0125] The processor may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field programmable gate arrays (FPGA), or other programmable logic devices, discrete gates or transistor logic, discrete hardware components, or any combination thereof. A general-purpose processor may be a microprocessor, but in an alternative, the processor may be any conventional processor, controller, microcontroller, or state machine. The processor may also be implemented as a combination of computing devices, such as a combination of a DSP and a microprocessor, a plurality of microprocessors, one or more microprocessors combined with a DSP core, or any other such configuration. In an alternative, the storage medium may be integrated with the processor. The processor and the storage medium may reside in an ASIC. The ASIC may reside in a user terminal. In an alternative, the processor and the storage medium may reside in a user terminal as discrete components.

[0126] In a specific embodiment of the present application, a computer device includes a processor and a memory, wherein the memory stores computer instructions, wherein the processor operates the computer instructions to execute the deep learning-based multi-channel upmixing method described in any embodiment.

[0127] In the embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0128] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0129] The above description is only an embodiment of the present application and does not limit the patent scope of the present application. Any equivalent structural transformation made using the contents of the present application specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present application.

Claims

1. A multi-channel upmixing method based on deep learning, characterized in that: include: At the Bluetooth receiver, the dual-channel LC3 code stream is decoded into transform domain noise shaping to obtain the left channel spectral coefficient and the right channel spectral coefficient; Perform feature extraction on the dual-channel LC3 code stream according to the left channel spectral coefficient and the right channel spectral coefficient to obtain sub-band energy sum, sub-band energy difference and sub-band cross-correlation; Inputting a combined feature including the subband energy sum, the subband energy difference, the subband cross-correlation, the LTPF enable flag and the pitch delay into a pre-trained neural network model, and outputting subband gains of multiple channels, wherein the LTPF enable flag and the pitch delay are obtained in a decoding process; Interpolating the sub-band gain of each channel to obtain frequency bin gains of the multiple channels; Generate spectral coefficients of the multiple channels according to the frequency bin gain of each channel, the left channel spectral coefficient and the right channel spectral coefficient; A low-delay inverse discrete cosine transform is performed on the spectral coefficient of each channel to obtain PCM signals of the multiple channels.

2. The multi-channel upmixing method based on deep learning according to claim 1, characterized in that: The plurality of channels include a front left channel, a front right channel, a front center channel, a rear left surround channel, and a rear right surround channel.

3. The multi-channel upmixing method based on deep learning according to claim 2, characterized in that: Also includes: The PCM signal of the front center channel is subjected to low-pass filtering to generate a PCM signal of a subwoofer channel.

4. The multi-channel upmixing method based on deep learning according to claim 1, characterized in that: The performing feature extraction on the dual-channel LC3 code stream according to the left channel spectral coefficient and the right channel spectral coefficient to obtain sub-band energy sum, sub-band energy difference and sub-band cross-correlation includes: According to the Bark frequency, the left channel spectral coefficient and the right channel spectral coefficient are divided into a predetermined number of sub-bands; Obtaining left channel subband energy, right channel subband energy and subband cross-correlation according to the left channel spectral coefficient, the right channel spectral coefficient and each of the subbands; The left channel sub-band energy and the right channel sub-band energy are subjected to corresponding predetermined operations to obtain the sub-band energy sum and the sub-band energy difference.

5. The multi-channel upmixing method based on deep learning according to claim 1, characterized in that: The step of generating spectral coefficients of the plurality of channels according to the frequency bin gain of each channel, the left channel spectral coefficient, and the right channel spectral coefficient comprises: The spectral coefficients of the multiple channels are obtained by adding a product of a left gain of the frequency bin gain of each channel and the left channel spectral coefficient and a product of a right gain of the frequency bin gain of each channel and the right channel spectral coefficient.

6. The multi-channel upmixing method based on deep learning according to claim 1, characterized in that: The training process of the pre-trained neural network model includes: Acquire a multi-channel sound source, and decode the multi-channel sound source to obtain time-domain PCM signals of the multiple channels; Extracting features from the time-domain PCM signal of each channel to obtain multi-channel features; Performing downmixing on the time-domain PCM signals of the multiple channels to obtain a two-channel stereo signal; Extracting features from the two-channel stereo signal to obtain two-channel features; Inputting the multi-channel features and the binaural features into a neural network for training, and calculating the loss of the neural network according to the output of the neural network; and The weights and biases of the neural network are adjusted according to the loss, and when the loss meets the training termination condition, the training is stopped to obtain the pre-trained neural network model.

7. The multi-channel upmixing method based on deep learning according to claim 6, characterized in that: The loss includes a total loss of the plurality of channels in a time domain and a total loss of the plurality of channels in a frequency domain.

8. A multi-channel upmixing system based on deep learning, characterized in that: include: A module for decoding the dual-channel LC3 code stream into transform domain noise shaping at the Bluetooth receiver to obtain the left channel spectral coefficient and the right channel spectral coefficient; A module for performing feature extraction on the dual-channel LC3 code stream according to the left channel spectral coefficient and the right channel spectral coefficient to obtain sub-band energy sum, sub-band energy difference and sub-band cross-correlation; A module for inputting a combined feature including the sub-band energy sum, the sub-band energy difference, the sub-band cross-correlation, the LTPF enable flag and the pitch delay into a pre-trained neural network model, and outputting sub-band gains of multiple channels, wherein the LTPF enable flag and the pitch delay are obtained during the decoding process; A module for interpolating the sub-band gain of each channel to obtain frequency bin gains of the multiple channels; A module for generating spectral coefficients of the plurality of channels according to the frequency bin gain of each channel, the left channel spectral coefficient and the right channel spectral coefficient; A module for performing low-delay improved inverse discrete cosine transform on the spectral coefficients of each channel to obtain PCM signals of the multiple channels.

9. A computer-readable storage medium storing computer instructions, wherein the computer instructions are operated to execute the deep learning-based multi-channel upmixing method according to any one of claims 1 to 7.

10. A computer device comprising a processor and a memory, wherein the memory stores computer instructions, wherein the processor operates the computer instructions to execute the deep learning-based multi-channel upmixing method according to any one of claims 1 to 7.