A method, system and encoding method for audio bandwidth extension based on deep learning

Abstract

The present application discloses an audio bandwidth extension method, system and encoding method based on deep learning, which belongs to the field of audio coding technology. The method includes: encoding audio data using an SBC encoder or an mSBC encoder, analyzing and filtering the current frame audio data, and obtaining multiple subbands corresponding to the current frame audio data; calculating the subband energy of each subband, and detecting and judging the bandwidth of the current frame audio data according to the subband energy; under the condition that the bandwidth of the current frame audio data is a narrow bandwidth, the bandwidth of the current frame audio data is extended by a pre-trained bandwidth extension model. In the encoding process of the SBC or mSBC encoder, the present application utilizes the encoding characteristics of the SBC or mSBC encoder itself to perform frequency band extension, thereby eliminating the process of time-frequency conversion, and at the same time utilizes the existing parameters of the encoding process to perform bandwidth extension, saving computing power and effectively improving audio quality.

Description

Technical Field

[0001] The present application relates to the field of audio coding technology, and in particular to an audio bandwidth expansion method, system and coding method based on deep learning. Background Art

[0002] As a mandatory requirement of the A2DP protocol, the SBC codec is the most widely used audio codec that all classic Bluetooth audio devices must support. Among them, the SBC codec has two typical applications in Bluetooth audio, including SBC: Subband Codec, which is mainly used for Bluetooth music playback and encodes and decodes music signals, with a typical sampling rate of 44.1kHz and 48kHz; mSBC: Modified Subband Codec, which is mainly used for encoding and decoding broadband voice during Bluetooth calls, with a sampling rate of 16kHz. At present, there are many narrowband audios. During Bluetooth playback, the audio is encoded and decoded by the codec, resulting in low audio quality and reduced user listening experience. At present, the audio bandwidth expansion method involves the conversion of time domain and frequency domain in the process of audio bandwidth expansion, which consumes a lot of computing power. In addition, it has a large delay and cannot meet the delay requirements of Bluetooth audio. In addition, the existing technology knows the real bandwidth information of the audio before the audio bandwidth is expanded, while the sampling rate of the audio input to SBC and mSBC is known, but the real bandwidth is unknown, so the existing audio bandwidth expansion method is not suitable for the SBC codec. Summary of the invention

[0003] In view of the problems in the prior art that when frequency band expansion is performed to obtain wide-bandwidth audio, the operation process is complicated, the power consumption is high, and SBC encoding and decoding cannot be applied, the present application proposes an audio bandwidth expansion method, system and encoding method based on deep learning.

[0004] In the first aspect, the present application proposes an audio bandwidth extension method based on deep learning, including: encoding audio data using an SBC encoder or an mSBC encoder, analyzing and filtering the current frame audio data, and obtaining multiple sub-bands corresponding to the current frame audio data; calculating the sub-band energy of each sub-band, and detecting and judging the bandwidth of the current frame audio data based on the sub-band energy; under the condition that the bandwidth of the current frame audio data is a narrow bandwidth, expanding the bandwidth of the current frame audio data through a pre-trained bandwidth extension model.

[0005] Optionally, the audio data is encoded using an SBC encoder or an mSBC encoder, including: the audio transmitting end where the SBC encoder or the mSBC encoder is located negotiates audio data encoding parameters with the audio receiving end, and under the condition that the audio receiving end supports the corresponding sampling rate, encodes the audio data with the sampling rate using the SBC encoder or the mSBC encoder.

[0006] Optionally, the pre-training process of the bandwidth extension model includes: obtaining broadband speech and filtering the broadband speech to obtain narrowband speech, wherein the broadband speech has a wide bandwidth and the narrowband speech has only a narrow bandwidth; performing feature extraction on the broadband speech and the narrowband speech respectively to obtain true broadband spectrum coefficients corresponding to the broadband speech and narrowband spectrum coefficients corresponding to the narrowband speech; training the generation network according to the low-frequency spectrum coefficients extracted from the true broadband spectrum coefficients and the narrowband spectrum coefficients, so that the generation network can generate high-frequency spectrum coefficients according to the narrowband spectrum coefficients.

[0007] Optionally, the pre-training process of the bandwidth extension model also includes: combining low-frequency spectrum coefficients and high-frequency spectrum coefficients to obtain pseudo-wideband spectrum coefficients; judging true wideband spectrum coefficients and pseudo-wideband spectrum coefficients through a discriminant network, and optimizing the generation network.

[0008] In the second aspect, the present application proposes an audio bandwidth extension system based on deep learning, including: an analysis and filtering module, which uses an SBC encoder or an mSBC encoder to encode audio data, analyzes and filters the current frame audio data, and obtains multiple sub-bands corresponding to the current frame audio data; a bandwidth detection module, which calculates the sub-band energy of each sub-band, and detects and judges the bandwidth of the current frame audio data based on the sub-band energy; a bandwidth extension module, which, under the condition that the bandwidth of the current frame audio data is a narrow bandwidth, expands the bandwidth of the current frame audio data through a pre-trained bandwidth extension model.

[0009] On the third aspect, the present application proposes an audio encoding method, including encoding audio data using an SBC encoder or an mSBC encoder, analyzing and filtering the current frame audio data to obtain multiple sub-bands corresponding to the current frame audio data; calculating the sub-band energy of each sub-band, and detecting and judging the bandwidth of the current frame audio data based on the sub-band energy; under the condition that the bandwidth of the current frame audio data is a narrow bandwidth, expanding the bandwidth of the current frame audio data through a pre-trained bandwidth extension model; the SBC encoder or mSBC encoder encodes the current frame audio data according to the new bandwidth spectrum coefficients after the bandwidth expansion, and outputs the corresponding bit stream.

[0010] In one technical solution of the present application, a computer-readable storage medium is provided, which stores a computer program, wherein the computer program is operated to execute the method in Solution 1 or Solution 2.

[0011] In one technical solution of the present application, a computer device is provided, which includes a processor and a memory, wherein the memory stores a computer program, wherein the processor operates the computer program to execute the method in Solution 1 or Solution 2.

[0012] The beneficial effect of the present application is as follows: during the encoding process of the SBC or mSBC encoder, the present application utilizes the encoding characteristics of the SBC or mSBC encoder itself to perform frequency band expansion, thereby eliminating the process of time-frequency conversion, and at the same time utilizes the existing parameters of the encoding process to expand the bandwidth, thereby saving computing power and effectively improving the audio quality. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, a brief introduction will be given below to the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0014] Figure 1 It is a flowchart of an implementation method of the audio bandwidth extension method based on deep learning of the present application;

[0015] Figure 2 is a schematic diagram of an example of the bandwidth extension model training and reasoning process of the present application;

[0016] Figure 3 is a schematic diagram of an example of the audio bandwidth extension method of the present application;

[0017] Figure 4 is a schematic diagram of an implementation of an audio bandwidth extension system based on deep learning of the present application;

[0018] Figure 5 It is a schematic diagram of an implementation of the audio encoding method of the present application.

[0019] The above drawings have shown clear embodiments of the present application, which will be described in more detail later. These drawings and text descriptions are not intended to limit the scope of the present application in any way, but to illustrate the concept of the present application to those skilled in the art by referring to specific embodiments. DETAILED DESCRIPTION

[0020] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0021] The terms "first", "second", "third", "fourth", etc. (if any) in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a product or device that includes a series of steps or units is not necessarily limited to those units that are clearly listed, but may include other units that are not clearly listed or inherent to these products or devices.

[0022] As a mandatory requirement of the A2DP protocol, the SBC codec is the most widely used audio codec that all classic Bluetooth audio devices must support. Among them, the SBC codec has two typical applications in Bluetooth audio, including SBC: Subband Codec, which is mainly used for Bluetooth music playback and encodes and decodes music signals, with a typical sampling rate of 44.1kHz and 48kHz; mSBC: Modified Subband Codec, which is mainly used for encoding and decoding broadband voice during Bluetooth calls, with a sampling rate of 16kHz. At present, there are many narrowband audios. During Bluetooth playback, the audio is encoded and decoded by the codec, resulting in low audio quality and reduced user listening experience. At present, the audio bandwidth expansion method involves the conversion of time domain and frequency domain in the process of audio bandwidth expansion, which consumes a lot of computing power. In addition, it has a large delay and cannot meet the delay requirements of Bluetooth audio. In addition, the existing technology knows the real bandwidth information of the audio before the audio bandwidth is expanded, while the sampling rate of the audio input to SBC and mSBC is known, but the real bandwidth is unknown, so the existing audio bandwidth expansion method is not suitable for the SBC codec.

[0023] In response to the above problems, the present application proposes an audio bandwidth expansion method, system and encoding method based on deep learning. The method includes: encoding audio data using an SBC encoder or an mSBC encoder, analyzing and filtering the current frame audio data to obtain multiple subbands corresponding to the current frame audio data; calculating the subband energy of each subband, and detecting and judging the bandwidth of the current frame audio data based on the subband energy; under the condition that the bandwidth of the current frame audio data is a narrow bandwidth, the bandwidth of the current frame audio data is expanded through a pre-trained bandwidth expansion model.

[0024] In the process of encoding audio data by the SBC encoder or mSBC encoder, the present application omits the time-frequency conversion process required for bandwidth expansion in the prior art by utilizing the characteristics of the encoding process, thereby reducing the computing power required for the entire bandwidth expansion process. In addition, bandwidth expansion is performed by deep learning to ensure the accuracy of bandwidth expansion.

[0025] The technical solution of the present application and how the technical solution of the present application solves the above-mentioned technical problems are described in detail below with specific embodiments. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. The embodiments of the present application will be described below in conjunction with the accompanying drawings.

[0026] Figure 1 A flowchart of an implementation of an audio bandwidth extension method based on deep learning in the present application is shown.

[0027] exist Figure 1 In the manner shown, the deep learning-based audio bandwidth extension method of the present application includes process S101, encoding the audio data using an SBC encoder or an mSBC encoder, analyzing and filtering the current frame audio data, and obtaining multiple sub-bands corresponding to the current frame audio data.

[0028] exist Figure 1 In the illustrated implementation manner, the bandwidth extension process of the present application is performed during the encoding process of the audio data by the SBC encoder or the mSBC encoder, and the audio bandwidth is directly extended using the parameters obtained during the encoding process. During the encoding process, the encoded audio data is analyzed and filtered by the analysis filter in the encoder to obtain multiple sub-bands corresponding to the audio data.

[0029] Specifically, taking the mSBC encoder as an example, in the process of encoding audio data, a frame of input PCM audio data is encoded, where the length of each frame is 120 samples, and each frame of data is passed through an analysis filter to obtain 8 sub-band signals, where each sub-band signal has 15 sub-band samples corresponding to it, which can be specifically expressed as:

[0030] a)X N (0,1),X N (0,2),,…,X N (0,14)

[0031] b)X N (1,1),X N (1,2),,…,X N (1,14)

[0032] c)……

[0033] d)X N (7,1),X N (7,2),,…,X N (7,14)

[0034] 8 sub-bands, a total of 120 sub-band sample data. It should be noted that this application is configured with a sampling rate of 16KHz and 8 sub-bands of the mSBC encoder. If other sampling rates and sub-band number configurations are sampled, the processing principle is the same as above and will not be repeated here.

[0035] Optionally, the audio data is encoded using an SBC encoder or an mSBC encoder, including: the audio transmitting end where the SBC encoder or the mSBC encoder is located negotiates audio data encoding parameters with the audio receiving end, and under the condition that the audio receiving end supports the corresponding sampling rate, encodes the audio data with the sampling rate using the SBC encoder or the mSBC encoder.

[0036] In this optional embodiment, when the audio data is encoded using the SBC encoder or the mSBC encoder and the bandwidth is expanded, in order to ensure that after the bandwidth of the audio data is expanded, the audio decoding end can decode the code stream after the bandwidth expansion, it is necessary to negotiate the encoding parameters with the audio receiving end, that is, the audio decoding end, that is, the encoder end will expand the audio bandwidth only when the audio receiving end is able to decode the audio after the bandwidth expansion. In addition, the audio bandwidth expansion method of the present application has basic requirements for the audio data to be encoded when the SBC encoder or the mSBC encoder performs bandwidth expansion during the audio encoding process.

[0037] Specifically, the SBC encoder is mainly used for the encoding process when the Bluetooth device is playing music. For example, in the combination of a mobile phone and a Bluetooth headset, the mobile phone is the transmitter and the Bluetooth headset is the receiver. First, when the transmitter and the receiver negotiate the parameters, if the receiver can support the playback of SBC audio with a sampling rate of 48kHz, the encoder of the transmitter outputs a 48kHz sampling rate, corresponding to an SBC code stream with a bandwidth of 24kHz. Specifically as follows: If the audio source sampling rate of the transmitter is 48kHz and the effective bandwidth is 24kHz, the normal encoding mode is used and no bandwidth expansion is required; if the audio source of the transmitter is 48kHz, but the bandwidth detection result during the encoding process is less than 24kHz, the bandwidth extension mode is started to expand the audio bandwidth.

[0038] Specifically, the mSBC encoder is used for the encoding process of Bluetooth devices when making calls. For example, in the case of a mobile phone and a Bluetooth headset, the mobile phone is the transmitter and the Bluetooth headset is the receiver. When encoding, the transmitter and the receiver first negotiate the parameters. If the receiver can support the playback of mSBC audio with a sampling rate of 16kHz, the encoder of the transmitter outputs a 16kHz sampling rate, corresponding to an mSBC code stream with a bandwidth of 8kHz. Specifically: If the audio source sampling rate of the transmitter is 16kHz and the effective bandwidth is 8Hz, the normal encoding mode is used and no bandwidth expansion is required; if the audio source of the transmitter is 16kHz, but the bandwidth detection result during the encoding process is less than 8kHz, the bandwidth expansion mode is started to expand the audio bandwidth.

[0039] exist Figure 1 In the illustrated implementation manner, the deep learning-based audio bandwidth extension method of the present application includes process S102, calculating the sub-band energy of each sub-band, and detecting and judging the bandwidth of the current frame audio data according to the sub-band energy.

[0040] In this implementation, after a frame of audio data is divided into sub-bands by an analysis filter, the sub-band energy of each sub-band is calculated, and then the bandwidth of the current frame of audio data is detected by the sub-band energy to determine whether the current frame of audio data is broadband data or narrowband data. If it is narrowband data, a subsequent bandwidth expansion process is required.

[0041] Specifically, the following takes a frame of audio data with 8 sub-bands as an example to illustrate the process of detecting the bandwidth of the current frame of audio data.

[0042] First calculate the subband energy: Where i = 0 to 7 represents the subband number. Bandwidth detection is performed based on subband energy. One detection method is as follows: the following three conditions are judged respectively:

[0043] Condition 1: If When i = 0 to 3;

[0044] Condition 2: If

[0045] Condition 3:

[0046] Among them, condition 1 is to determine whether the current signal has enough energy in the narrowband part, condition 2 is to ensure that the energy of the high-frequency sub-band is small enough, and condition 3 is to ensure that the ratio of the total energy of the low-frequency sub-band to the total energy of the high-frequency sub-band is large enough. If conditions 1, 2 and 3 are all met, it means that the current frame audio data is a narrowband signal. It should be noted that the three thresholds of Threshold1, Threshold2 and Threshold3 can be determined based on empirical values, and the thresholds can also be appropriately adjusted according to actual conditions. This application does not impose specific restrictions on the specific values ​​of the thresholds.

[0047] exist Figure 1 In the illustrated implementation manner, the deep learning-based audio bandwidth extension method of the present application includes process S103, in which, under the condition that the bandwidth of the current frame audio data is a narrow bandwidth, the bandwidth of the current frame audio data is extended by a pre-trained bandwidth extension model.

[0048] In this implementation, after determining that the current frame audio data is narrowband data and needs to be bandwidth expanded, the narrowband data is processed by a pre-trained bandwidth expansion model to obtain corresponding broadband data.

[0049] Optionally, the pre-training process of the bandwidth extension model includes: obtaining broadband audio and filtering the broadband audio to obtain narrowband audio, wherein the broadband audio has a wide bandwidth and the narrowband audio has only a narrow bandwidth; performing feature extraction on the broadband audio and the narrowband audio respectively to obtain true broadband spectrum coefficients corresponding to the broadband audio and narrowband spectrum coefficients corresponding to the narrowband audio; and training the bandwidth extension model according to the true broadband spectrum coefficients and the narrowband spectrum coefficients, so that the bandwidth extension model can generate broadband spectrum coefficients corresponding to the narrowband audio according to the narrowband spectrum coefficients.

[0050] In this optional embodiment, when training the bandwidth extension model, first obtain the data for model training, and obtain broadband audio, such as audio with a sampling rate of 16KHz and an effective bandwidth of 8KHz. The broadband audio can be obtained by publicly available audio materials or self-recorded audio materials. Among them, the broadband audio has spectral energy information in the bandwidth range of 0-8KHz. Then obtain narrowband audio, corresponding to the above-mentioned broadband audio, the sampling rate of the narrowband audio is 16KHZ, and the effective bandwidth is 4KHz. In actual situations, during a call, the mobile phone may receive a narrowband signal from the base station with a sampling rate of 8kHz and an effective bandwidth of 4kHz. The mobile phone will resample this narrowband signal to 16kHz and then transmit it to the mSBC encoder. The resampling will not change the effective bandwidth, thereby obtaining narrowband audio. In the process of training the bandwidth model in this application, by low-pass filtering the obtained broadband audio to obtain narrowband audio, you can get narrowband audio with a sampling rate of 16KHz and an effective bandwidth of 4KHz. The narrowband audio is obtained by low-pass filtering the broadband audio, which ensures the consistency of the two audios in other aspects except bandwidth, and helps to ensure the accuracy of training.

[0051] In this optional embodiment, after obtaining broadband audio and narrowband audio respectively, feature extraction of broadband audio and narrowband audio is performed to obtain broadband spectral coefficients corresponding to broadband audio and narrowband spectral coefficients corresponding to narrowband audio. The broadband feature extraction method is the same as the narrowband feature extraction method, and both inputs are 16kHz audio signals. The difference is that a low-pass filter is used to filter out high-frequency signals before narrowband feature extraction, and then a standard SBC analysis filter is used to generate sub-band signals. During training, broadband features and narrowband features are simultaneously input into the neural network to train the neural network. When the training error reaches or exceeds the expected target, the weights and related parameters are frozen. It should be noted that the above-mentioned audio distance with a sampling rate of 16KHz is similar in principle to other audio.

[0052] Optionally, the pre-training process of the bandwidth extension model also includes: determining the error between the true wideband spectrum coefficient and the wideband spectrum coefficient through a discriminant network, and optimizing the bandwidth extension model according to the error until the error is less than a preset threshold.

[0053] In this optional embodiment, after obtaining the predicted wideband spectral coefficients corresponding to the narrowband audio through the bandwidth extension model, the error between the true wideband spectral coefficients and the wideband spectral coefficients is judged through the discriminant network. If the error between the two is large, the relevant parameters of the bandwidth extension model are adjusted to optimize the bandwidth extension model, so that the wideband spectral coefficients obtained by the bandwidth extension model are closer to the true wideband spectral coefficients, thereby improving the training effect of the bandwidth extension model.

[0054] Specifically, Figure 2An example of the bandwidth extension model training and reasoning process of the present application is shown.

[0055] like Figure 2 As shown, in the offline training process, first obtain broadband audio, for example, the sampling rate is 16KHz and the effective bandwidth is 8KHz. Then the broadband audio is low-pass filtered to obtain narrowband audio. Feature extraction is performed on broadband audio and narrowband audio respectively to obtain corresponding broadband features and narrowband features, and deep learning neural network training is performed to finally determine the bandwidth extension model. In the online reasoning process of the bandwidth extension model, narrowband audio, for example, with a sampling rate of 16KHz and an effective bandwidth of 4KHz, is subjected to feature extraction and bandwidth detection of narrowband audio, and the bandwidth of the narrowband audio is extended through the bandwidth extension model, and the encoding parameters of the encoding are updated according to the broadband spectrum coefficients obtained after the extension, and the encoding of the remaining encoding modules is performed, and the encoding results are sent to the audio receiving end via Bluetooth communication.

[0056] Specifically, Figure 3 An example of the audio bandwidth extension method of the present application is shown.

[0057] like Figure 3 As shown, Figure 3 The flowchart shows the process of bandwidth expansion for narrowband audio during the encoding process of audio data by the SBC encoder and the mSBC encoder. First, PCM audio data is received in the encoder, and the audio data is analyzed and filtered through an analysis filter to obtain multiple corresponding sub-bands. Then, the bandwidth detection of the current audio is completed by calculating the sub-band energy of the obtained sub-band samples. If the audio data is wideband audio, that is, the effective bandwidth is equal to the Nyquist bandwidth, there is no need to expand the bandwidth, and the subsequent encoding process is directly carried out, and bits are allocated according to the wideband audio to complete the subsequent encoding. If it is detected that the current audio data is narrowband data, that is, the effective bandwidth is lower than the Nyquist bandwidth, the bandwidth is expanded by the bandwidth extension method of the present application to obtain wideband audio corresponding to the narrowband audio, and then bits are allocated according to the expanded wideband audio to complete the subsequent encoding.

[0058] In the encoding process of the SBC or mSBC encoder, the present application utilizes the encoding characteristics of the SBC or mSBC encoder itself to perform frequency band expansion, thereby eliminating the process of time-frequency conversion, and utilizing the existing parameters of the encoding process to perform bandwidth expansion, thereby saving computing power and effectively improving the audio quality. During the bandwidth expansion process, bandwidth detection is performed. Before the bandwidth expansion, it is not necessary to know the bandwidth of the audio in advance. In addition, the bandwidth can be detected based on the existing information in the encoding process with the aid of a small amount of calculation. Therefore, the bandwidth expansion process of the present application can be applied to the scenario of Bluetooth calls where the audio data may be wide bandwidth or narrow bandwidth. By performing bandwidth expansion during the encoding process, the present invention does not increase the algorithm delay except for adding some operation delays, thereby improving the user experience.

[0059] Figure 4 An embodiment of the deep learning-based audio bandwidth extension system of the present application is shown.

[0060] exist Figure 4 In the illustrated implementation manner, the audio bandwidth extension system of the present application based on deep learning includes: an analysis and filtering module 401, which encodes the audio data using an SBC encoder or an mSBC encoder, analyzes and filters the current frame audio data, and obtains multiple sub-bands corresponding to the current frame audio data; a bandwidth detection module 402, which calculates the sub-band energy of each sub-band, and detects and determines the bandwidth of the current frame audio data based on the sub-band energy; a bandwidth extension module 403, which, under the condition that the bandwidth of the current frame audio data is a narrow bandwidth, performs bandwidth extension on the current frame audio data through a pre-trained bandwidth extension model.

[0061] Optionally, the audio data is encoded using an SBC encoder or an mSBC encoder, including: the audio transmitting end where the SBC encoder or the mSBC encoder is located negotiates audio data encoding parameters with the audio receiving end, and under the condition that the audio receiving end supports the corresponding sampling rate, encodes the audio data with the sampling rate using the SBC encoder or the mSBC encoder.

[0062] Optionally, the pre-training process of the bandwidth extension model includes: obtaining broadband audio and filtering the broadband audio to obtain narrowband audio, wherein the broadband audio has a wide bandwidth and the narrowband audio has only a narrow bandwidth; performing feature extraction on the broadband audio and the narrowband audio respectively to obtain true broadband spectrum coefficients corresponding to the broadband audio and narrowband spectrum coefficients corresponding to the narrowband audio; and training the bandwidth extension model according to the true broadband spectrum coefficients and the narrowband spectrum coefficients, so that the bandwidth extension model can generate broadband spectrum coefficients corresponding to the narrowband audio according to the narrowband spectrum coefficients.

[0063] Optionally, the pre-training process of the bandwidth extension model also includes: determining the error between the true wideband spectrum coefficient and the wideband spectrum coefficient through a discriminant network, and optimizing the bandwidth extension model according to the error until the error is less than a preset threshold.

[0064] The audio bandwidth extension system based on deep learning in this application uses the characteristics of the SBC or mSBC encoder itself to perform frequency band extension during the encoding process of the SBC or mSBC encoder, eliminating the process of time-frequency conversion, and using the existing parameters of the encoding process to expand the bandwidth, saving computing power, and effectively improving the audio quality. During the bandwidth extension process, bandwidth detection is performed. Before the bandwidth extension, it is not necessary to know the bandwidth of the audio in advance. In addition, based on the existing information in the encoding process and supplemented by a small amount of calculation, the bandwidth can be detected. Therefore, the bandwidth extension process of this application can be applied to the scene of Bluetooth calls where the audio data may be wide bandwidth or narrow bandwidth. By performing bandwidth extension during the encoding process, the present invention does not increase the algorithm delay except for adding some operation delays, thereby ensuring the user experience.

[0065] Figure 5 An implementation of the audio encoding method of the present application is shown.

[0066] exist Figure 5 In the illustrated implementation manner, the audio encoding method of the present application includes: process S501, encoding the audio data using an SBC encoder or an mSBC encoder, analyzing and filtering the current frame audio data, and obtaining multiple sub-bands corresponding to the current frame audio data; process S502, calculating the sub-band energy of each sub-band, and detecting and judging the bandwidth of the current frame audio data based on the sub-band energy; process S503, under the condition that the bandwidth of the current frame audio data is a narrow bandwidth, bandwidth expansion of the current frame audio data is performed through a pre-trained bandwidth expansion model; process S504, the SBC encoder or the mSBC encoder encodes the current frame audio data according to the new bandwidth spectrum coefficient after the bandwidth expansion, and outputs the corresponding bit stream.

[0067] Optionally, the audio data is encoded using an SBC encoder or an mSBC encoder, including: the audio transmitting end where the SBC encoder or the mSBC encoder is located negotiates audio data encoding parameters with the audio receiving end, and under the condition that the audio receiving end supports the corresponding sampling rate, encodes the audio data with the sampling rate using the SBC encoder or the mSBC encoder.

[0068] Optionally, the pre-training process of the bandwidth extension model includes: obtaining broadband audio and filtering the broadband audio to obtain narrowband audio, wherein the broadband audio has a wide bandwidth and the narrowband audio has only a narrow bandwidth; performing feature extraction on the broadband audio and the narrowband audio respectively to obtain true broadband spectrum coefficients corresponding to the broadband audio and narrowband spectrum coefficients corresponding to the narrowband audio; and training the bandwidth extension model according to the true broadband spectrum coefficients and the narrowband spectrum coefficients, so that the bandwidth extension model can generate broadband spectrum coefficients corresponding to the narrowband audio according to the narrowband spectrum coefficients.

[0069] The audio coding method of the present application uses the characteristics of the SBC or mSBC encoder itself to perform frequency band expansion during the encoding process of the SBC or mSBC encoder, eliminating the process of time-frequency conversion, and using the existing parameters of the encoding process to perform bandwidth expansion, saving computing power, and effectively improving the audio quality. During the bandwidth expansion process, bandwidth detection is performed. Before the bandwidth expansion, it is not necessary to know the bandwidth of the audio in advance. In addition, based on the existing information in the encoding process with a small amount of calculation, the bandwidth can be detected. Therefore, the bandwidth expansion process of the present application can be applied to the scene of Bluetooth calls where the audio data may be wide bandwidth or narrow bandwidth. The present invention performs bandwidth expansion during the encoding process, except for adding some operation delays, does not increase the algorithm delay, and the user experience is better. The principle of each step in the audio coding method of the present application is consistent with the process of the above-mentioned audio expansion method in China, and will not be repeated here.

[0070] In a specific embodiment of the present application, a computer-readable storage medium stores a computer program, wherein the computer program is operated to execute the audio bandwidth extension method or audio encoding method based on deep learning described in any embodiment. The storage medium may be directly in hardware, in a software module executed by a processor, or in a combination of the two.

[0071] The software modules may reside in RAM memory, flash memory, ROM memory, EPROM memory, EEPROM memory, registers, hard disk, removable disk, CD-ROM or any other form of storage medium known in the art. An exemplary storage medium is coupled to the processor so that the processor can read information from and write information to the storage medium.

[0072] The processor may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field programmable gate arrays (FPGA), or other programmable logic devices, discrete gates or transistor logic, discrete hardware components, or any combination thereof. A general-purpose processor may be a microprocessor, but in an alternative, the processor may be any conventional processor, controller, microcontroller, or state machine. The processor may also be implemented as a combination of computing devices, such as a combination of a DSP and a microprocessor, a plurality of microprocessors, one or more microprocessors combined with a DSP core, or any other such configuration. In an alternative, the storage medium may be integrated with the processor. The processor and the storage medium may reside in an ASIC. The ASIC may reside in a user terminal. In an alternative, the processor and the storage medium may reside in a user terminal as discrete components.

[0073] In a specific embodiment of the present application, a computer device includes a processor and a memory, the memory storing a computer program, wherein: the processor operates the computer program to execute the deep learning-based audio bandwidth extension method or audio encoding method described in any embodiment.

[0074] In the embodiments provided in the present application, it should be understood that the disclosed device can be implemented in other ways. For example, the device embodiments described above are only schematic, for example, the division of units is only a logical function division, and there may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0075] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0076] The above are merely embodiments of the present application, and are not intended to limit the patent scope of the present application. Any equivalent structural transformations made using the contents of the present application specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present application.

Claims

1. A method for extending audio bandwidth based on deep learning, characterized in that: include: Encode the audio data using an SBC encoder or an mSBC encoder, analyze and filter the current frame audio data, and obtain multiple sub-bands corresponding to the current frame audio data; Calculate the subband energy of each subband, and detect and judge the bandwidth of the current frame audio data according to the subband energy, wherein: If all the sub-band energies of the low-frequency sub-bands are greater than a first threshold, and the sum of the sub-band energies of the high-frequency sub-bands is greater than a second threshold, and the ratio of the sum of the sub-band energies of the low-frequency sub-bands to the sum of the sub-band energies of the high-frequency sub-bands is greater than a third threshold, then the bandwidth of the current frame audio data is a narrow bandwidth; Under the condition that the bandwidth of the current frame audio data is a narrow bandwidth, the bandwidth of the current frame audio data is expanded by using a pre-trained bandwidth expansion model.

2. The method for extending audio bandwidth based on deep learning according to claim 1, characterized in that: The method of encoding the audio data by using an SBC encoder or an mSBC encoder includes: The audio transmitting end where the SBC encoder or the mSBC encoder is located negotiates audio data encoding parameters with the audio receiving end, and encodes the audio data with the sampling rate through the SBC encoder or the mSBC encoder under the condition that the audio receiving end supports the corresponding sampling rate.

3. The method for extending audio bandwidth based on deep learning according to claim 1, characterized in that: The pre-training process of the bandwidth extension model includes: Acquire broadband audio, and filter the broadband audio to obtain narrowband audio, wherein the broadband audio has a wide bandwidth and the narrowband audio has only a narrow bandwidth; Performing feature extraction on the wideband audio and the narrowband audio respectively to obtain true wideband spectrum coefficients corresponding to the wideband audio and narrowband spectrum coefficients corresponding to the narrowband audio; The bandwidth extension model is trained according to the true wideband spectrum coefficients and the narrowband spectrum coefficients, so that the bandwidth extension model can generate wideband spectrum coefficients corresponding to the narrowband audio according to the narrowband spectrum coefficients.

4. The method for extending audio bandwidth based on deep learning according to claim 3, characterized in that: The pre-training process of the bandwidth extension model further includes: The error between the true wideband spectrum coefficient and the wideband spectrum coefficient is determined by a discriminant network, and the bandwidth extension model is optimized according to the error until the error is less than a preset threshold.

5. An audio bandwidth extension system based on deep learning, characterized in that: include: An analysis and filtering module, which encodes the audio data using an SBC encoder or an mSBC encoder, performs analysis and filtering on the current frame audio data, and obtains a plurality of sub-bands corresponding to the current frame audio data; A bandwidth detection module calculates the sub-band energy of each sub-band, and detects and determines the bandwidth of the current frame audio data according to the sub-band energy, wherein: If all the sub-band energies of the low-frequency sub-bands are greater than a first threshold, and the sum of the sub-band energies of the high-frequency sub-bands is greater than a second threshold, and the ratio of the sum of the sub-band energies of the low-frequency sub-bands to the sum of the sub-band energies of the high-frequency sub-bands is greater than a third threshold, then the bandwidth of the current frame audio data is a narrow bandwidth; A bandwidth extension module is used to extend the bandwidth of the current frame audio data through a pre-trained bandwidth extension model under the condition that the bandwidth of the current frame audio data is a narrow bandwidth.

6. An audio encoding method, characterized in that: include: Encode the audio data using an SBC encoder or an mSBC encoder, analyze and filter the current frame audio data, and obtain multiple sub-bands corresponding to the current frame audio data; Calculate the subband energy of each subband, and detect and judge the bandwidth of the current frame audio data according to the subband energy, wherein: If all the sub-band energies of the low-frequency sub-bands are greater than a first threshold, and the sum of the sub-band energies of the high-frequency sub-bands is greater than a second threshold, and the ratio of the sum of the sub-band energies of the low-frequency sub-bands to the sum of the sub-band energies of the high-frequency sub-bands is greater than a third threshold, then the bandwidth of the current frame audio data is a narrow bandwidth; Under the condition that the bandwidth of the current frame audio data is a narrow bandwidth, bandwidth expansion is performed on the current frame audio data by using a pre-trained bandwidth expansion model; The SBC encoder or the mSBC encoder encodes the current frame audio data according to the new bandwidth spectrum coefficient after bandwidth expansion, and outputs a corresponding code stream.

7. The audio encoding method according to claim 6, characterized in that: The method of encoding the audio data by using an SBC encoder or an mSBC encoder comprises: The audio transmitting end where the SBC encoder or the mSBC encoder is located negotiates audio data encoding parameters with the audio receiving end, and encodes the audio data with the sampling rate through the SBC encoder or the mSBC encoder under the condition that the audio receiving end supports the corresponding sampling rate.

8. The audio encoding method according to claim 6, characterized in that: The pre-training process of the bandwidth extension model includes: Acquire broadband audio, and filter the broadband audio to obtain narrowband audio, wherein the broadband audio has a wide bandwidth and the narrowband audio has only a narrow bandwidth; Performing feature extraction on the wideband audio and the narrowband audio respectively to obtain true wideband spectrum coefficients corresponding to the wideband audio and narrowband spectrum coefficients corresponding to the narrowband audio; The bandwidth extension model is trained according to the true wideband spectrum coefficients and the narrowband spectrum coefficients, so that the bandwidth extension model can generate wideband spectrum coefficients corresponding to the narrowband audio according to the narrowband spectrum coefficients.

9. A computer-readable storage medium storing a computer program, wherein the computer program is operated to execute the audio bandwidth extension method based on deep learning according to any one of claims 1 to 4 or the audio encoding method according to any one of claims 6 to 8.

10. A computer device comprising a processor and a memory, wherein the memory stores a computer program, wherein: The processor operates the computer program to execute the deep learning-based audio bandwidth extension method described in any one of claims 1 to 4 or the audio encoding method described in any one of claims 6 to 8.

Images