Frequency spectrum shaping parameter derivation method in LC3 coding framework based on neural network

Abstract

The invention relates to the technical field of voice signal processing and audio coding, and discloses a frequency spectrum shaping parameter derivation method in an LC3 coding framework based on a neural network. The method comprises the following steps: performing frequency band energy analysis on an input audio frame, calculating energy of each frequency band according to a preset frequency band division rule, and obtaining multi-frequency band energy information representing spectrum distribution characteristics of the current frame; based on the multi-band energy information, combining a frame-level coding parameter and a transient detection mark, constructing an input feature vector for neural network reasoning, and predicting a spectrum shaping parameter consistent with an LC3 standard by a neural network; and the prediction parameters are further quantized and coded according to the LC3 standard and are used for subsequent spectrum shaping processing, so that on the premise of not changing an LC3 standard bit stream structure and decoder end behaviors, the improvement of a spectrum shaping parameter derivation mode is realized, and the stability and adaptability of coded speech spectrum features are improved.

Description

Technical Field

[0001] This invention relates to the field of speech signal processing and audio coding technology, specifically to a method for deriving spectral shaping parameters in the LC3 coding framework based on neural networks. Background Technology

[0002] Low Complexity Communication Codec (LC3), the low-latency audio coding format adopted by the Bluetooth LEAudio standard, has its technical specifications developed and released by the Bluetooth Special Interest Group (SIG). Leveraging its efficient audio signal compression and transmission capabilities, it has been widely used in various voice communication scenarios. An LC3 encoder typically completes core steps sequentially, including frame processing, time-frequency transformation, spectrum shaping, quantization, and entropy coding. The Spectrum Noise Shaping (SNS) module is a crucial component, primarily deriving spectrum shaping parameters based on the frequency band energy distribution of the input audio frame. Through spectrum shaping, it guides the distribution of quantization noise in different frequency regions, balancing both coding efficiency and subjective listening quality. Existing LC3 SNS modules generally employ analytical algorithms based on energy statistics and transient detection rules, combined with a pre-defined codebook to represent and quantize the spectrum shaping parameters. The number of output parameters, parameter formats, and bitstream structure strictly adhere to the LC3 standard, forming a mature engineering practice in low-latency communication and meeting basic subjective listening requirements. However, as voice communication systems expand into diverse scenarios such as intelligent voice interaction and voice control, the ability of encoded speech to preserve features in subsequent speech processing or recognition tasks has gradually become a key requirement. The existing LC3 SNS module spectrum shaping strategy is mainly designed around the characteristics of human hearing and has not been specifically optimized for the sensitivity of speech recognition models to spectrum structure, resulting in the encoded speech spectrum features being difficult to adapt to the needs of subsequent speech recognition and other processing.

[0003] Meanwhile, existing SNS modules rely on fixed codebooks and predefined rules to derive parameters. The parameter derivation process is mainly based on single-frame statistical features, lacking the ability to adaptively adjust to different speech content, environmental conditions, and continuous features between frames. In some application scenarios, this can easily lead to drastic changes in spectral shaping parameters between frames, thus affecting the stability of subsequent speech feature processing. Existing research has confirmed that, with the quantization structure remaining unchanged, the distribution of spectral noise across different frequency bands directly affects the speech feature extraction and recognition processing results. This indicates that, while maintaining the LC3 coding framework and bitstream structure, there is feasible technical space to improve speech feature preservation capabilities by optimizing the spectral shaping parameter derivation method. Therefore, how to provide a more flexible spectral shaping parameter derivation method without changing the LC3 encoder bitstream structure and decoder behavior to adapt to various speech application needs, including speech recognition, has become a key problem that urgently needs to be solved in the current LC3 SNS technology field. Summary of the Invention

[0004] To address the shortcomings of existing technologies, this invention provides a method for deriving spectral shaping parameters in the LC3 coding framework based on neural networks, thus solving the problems mentioned in the background section.

[0005] To achieve the above objectives, the present invention provides the following technical solution: a method for deriving spectral shaping parameters in an LC3 coding framework based on neural networks, applied to an LC3 audio coding system, the method comprising the following steps: Perform time-frequency transformation processing on the input audio frame, calculate the energy value of each frequency band according to the preset frequency band division rules, and obtain multi-frequency band energy information that characterizes the spectral distribution of the current frame. Based on the multi-band energy information, an input feature vector is constructed by combining frame duration, sampling rate, number of available bits in the current frame, transient detection flag, and energy statistical features. The transient detection flag is used to indicate whether there is a transient energy change in the current audio frame, and the energy statistical features are used to describe the overall spectrum distribution characteristics. The input feature vector is input to the lightweight neural network prediction module deployed in the LC3 encoder. The neural network prediction module outputs a number of spectral shaping parameters consistent with the number specified in the LC3 standard. The neural network adopts a low-precision numerical representation, and the number of parameters and computational complexity are adapted to the real-time operation requirements of the embedded digital signal processor. The spectrum shaping parameters are input into the parameter quantization and encoding module defined by the LC3 standard, and the corresponding spectrum shaping side information is generated according to the quantization rules and encoding methods specified by the LC3 standard. Interpolation is performed on the quantized spectrum shaping parameters, and then the spectrum coefficients are shaped based on the interpolated spectrum shaping parameters to indirectly control the distribution characteristics of quantization noise in different frequency bands. Subsequently, the subsequent encoding process defined by the LC3 standard is executed to generate an encoded bitstream that conforms to the LC3 standard. The neural network prediction module functionally replaces the traditional spectrum shaping parameter derivation module in the LC3 encoder, and does not change the bitstream syntax structure, parameter organization method, or decoder processing behavior specified by the LC3 standard.

[0006] Preferably, the input feature vector also includes the spectrum shaping parameter corresponding to the previous audio frame. By introducing this parameter, the neural network's ability to model the temporal continuity between adjacent audio frames is enhanced, thereby improving the stability of the predicted spectrum shaping parameter between frames.

[0007] Preferably, the lightweight neural network is a feedforward neural network structure with a simplified network depth, which can be quantized into low-precision integer form. The inference process can be completed in sub-milliseconds on an embedded digital signal processor, meeting the real-time requirements of LC3 encoding.

[0008] Preferably, the neural network is trained using a training objective that includes multiple constraints. The training objective includes at least structural consistency constraints, speech recognition feature preservation constraints, and parameter smoothing constraints. The structural consistency constraints are used to limit the predicted spectral shaping parameters to maintain consistency with the traditional LC3 spectral shaping parameters in terms of statistical distribution and structural characteristics. The speech recognition feature preservation constraints are used to reduce the difference between speech before and after encoding in the speech recognition feature space. The parameter smoothing constraints are used to suppress drastic changes between adjacent spectral shaping parameters.

[0009] Preferably, the structural consistency constraint is achieved by comparing the spectrum shaping parameters predicted by the neural network with the reference spectrum shaping parameters output by the traditional LC3 spectrum shaping parameter derivation module, ensuring consistency between the two in statistical distribution and structural characteristics; the speech recognition feature preservation constraint is achieved by comparing the feature differences of speech before and after encoding, the features including Mel frequency scale features or speech model embedding space features; the parameter smoothing constraint is achieved by limiting the variation amplitude between adjacent spectrum shaping parameters.

[0010] Preferably, when the neural network is deployed on an embedded digital signal processor, it adopts fixed-point quantization, chip memory loading, and a dedicated neural network library to accelerate execution. The embedded digital signal processor is based on the Cadence HiFi4 architecture, and the dedicated neural network library is Cadence HiFiNNLib. This deployment method meets the requirements for low power consumption and real-time operation.

[0011] Preferably, the network weights of the neural network are 8-bit fixed-point numbers, the biases are 32-bit fixed-point numbers, the intermediate activation values ​​are 16-bit fixed-point numbers, and the corresponding quantization ratios and zero-points are stored in a constant table. During the deployment phase, the model weights and quantization parameters are loaded into the on-chip memory of the embedded digital signal processor to reduce the latency caused by external memory access.

[0012] Preferably, the time-frequency transformation adopts the modified discrete cosine transform, and the preset frequency band division rule is executed in accordance with the frequency band division method specified in the LC3 standard to ensure that the acquisition of multi-frequency band energy information meets the requirements of the LC3 coding system.

[0013] An audio encoding system, comprising: The frequency band energy calculation module is used to perform time-frequency transformation processing on the input audio frame, calculate the energy value of each frequency band according to the preset frequency band division rules, and obtain multi-frequency band energy information; The input feature construction module is used to construct an input feature vector based on the multi-band energy information, combined with frame duration, sampling rate, number of available bits in the current frame, transient detection flag, energy statistical features, and optional spectrum shaping parameters of the previous frame. The neural network prediction module is a lightweight feedforward neural network structure used to predict the same number of spectrum shaping parameters as specified in the LC3 standard based on the input feature vector. The neural network adopts a low-precision numerical representation and functionally replaces the traditional spectrum shaping parameter derivation module. The parameter quantization and encoding module is used to quantize and encode the spectrum shaping parameters according to the LC3 standard to generate corresponding spectrum shaping side information; The spectrum shaping module performs interpolation on the quantized spectrum shaping parameters, and then shapes the spectrum coefficients based on the interpolated spectrum shaping parameters. Its implementation method is consistent with the LC3 standard. The subsequent encoding module is used to execute the subsequent processes defined by the LC3 standard, such as quantization, entropy encoding, and bitstream encapsulation, to generate an encoded bitstream that conforms to the LC3 standard.

[0014] A computer-readable storage medium having program instructions stored thereon.

[0015] This invention provides a method for deriving spectral shaping parameters in the LC3 coding framework based on neural networks. It has the following beneficial effects: 1. This invention constructs an input feature vector that includes multi-band energy information, frame-level parameters, transient detection flags, and optional previous frame spectrum shaping parameters. It combines a multi-constraint training mechanism that ensures structural consistency, preserves speech recognition features, and smooths parameters to achieve accurate prediction of spectrum shaping parameters. This effectively suppresses drastic changes in inter-frame parameters, reduces interference from background noise on speech spectrum features, and avoids a significant decline in subjective listening quality.

[0016] 2. This invention replaces the traditional spectrum shaping parameter derivation module with a lightweight neural network in the LC3 coding framework, without changing the bitstream structure, number of parameters, and decoder behavior specified by the LC3 standard. This achieves full compatibility with the LC3 standard, ensuring that the encoded bitstream can be seamlessly parsed by all standard LC3 decoders. At the same time, it makes the spectral characteristics of the decoded speech more stable in subsequent speech processing applications, improving the adaptability of speech-related applications to encoded speech.

[0017] 3. This invention employs a lightweight neural network structure with low-precision quantization, combined with an embedded digital signal processor-adapted deployment strategy, to enable real-time inference operation of the model on a low-power hardware platform. It can be directly integrated into existing LC3 encoders without additional hardware modifications. While meeting the low-latency requirements of LC3 encoding, it provides flexible adaptability for various application scenarios such as speech recognition and intelligent voice interaction, and has good technical scalability. Attached Figure Description

[0018] Figure 1 This is the overall flowchart of the present invention; Figure 2 This is a flowchart illustrating the module composition of the audio encoding system of the present invention. Detailed Implementation

[0019] The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0020] Please refer to Appendix 1 - Appendix Figure 2 This invention provides a method for deriving spectral shaping parameters in an LC3 coding framework based on neural networks, applied to an LC3 audio coding system. The method includes the following steps: Perform time-frequency transformation processing on the input audio frame, calculate the energy value of each frequency band according to the preset frequency band division rules, and obtain multi-frequency band energy information that characterizes the spectral distribution of the current frame. Based on multi-band energy information, an input feature vector is constructed by combining frame duration, sampling rate, number of available bits in the current frame, transient detection flag, and energy statistical features. The transient detection flag is used to indicate whether there is transient energy change in the current audio frame, and the energy statistical features are used to describe the overall spectrum distribution characteristics. The input feature vector is fed into a lightweight neural network prediction module deployed in the LC3 encoder. The neural network prediction module outputs a number of spectrum shaping parameters consistent with the LC3 standard. The neural network uses a low-precision numerical representation, and the number of parameters and computational complexity are adapted to the real-time operation requirements of the embedded digital signal processor. The input feature vector also includes spectrum shaping parameters corresponding to the previous audio frame. By introducing these parameters, the neural network's ability to model the temporal continuity between adjacent audio frames is enhanced, thereby improving the stability of the predicted spectrum shaping parameters between frames.

[0021] Specifically, when deriving the spectrum shaping parameters for the input audio frame, the audio signal is first converted from the time domain to the frequency domain through time-frequency transformation. Then, according to the preset frequency band division rules, the frequency domain signal is divided into bands and the energy value of each band is calculated. This obtains multi-band energy information that can accurately characterize the spectrum distribution characteristics of the current audio frame, providing basic frequency domain feature support for subsequent parameter prediction.

[0022] To enable the neural network to fully capture the key information and coding constraints of the audio frame, when constructing a comprehensive input feature vector, multi-band energy information is used as the core, integrating frame-level coding parameters such as frame duration, sampling rate, and the number of available bits in the current frame. Transient detection flags and energy statistical features are also included. The transient detection flags are used to accurately indicate whether there are transient energy changes in the current audio frame, while the energy statistical features are used to comprehensively describe the overall distribution characteristics of the spectrum, helping the neural network to understand the global characteristics of the spectrum more deeply.

[0023] Building upon this, a spectrum shaping parameter corresponding to the previous audio frame is further introduced. This parameter conveys the correlation information between adjacent audio frames, enhancing the neural network's ability to model the temporal continuity between frames. This effectively avoids large fluctuations in the spectrum shaping parameter between frames, thereby improving the inter-frame stability of the prediction parameters.

[0024] The constructed input feature vector is fed into the lightweight neural network prediction module deployed in the LC3 encoder. This neural network uses a low-precision numerical representation, and its parameter count and computational complexity have been specially optimized to fully adapt to the real-time operation requirements of the embedded digital signal processor, ensuring stable operation under the constraints of low power consumption and low latency. At the same time, it outputs spectral shaping parameters that are completely consistent with the number specified in the LC3 standard, ensuring the compatibility of subsequent encoding processes with the LC3 standard.

[0025] Input the spectrum shaping parameters into the parameter quantization and encoding module defined by the LC3 standard, and generate the corresponding spectrum shaping side information according to the quantization rules and encoding methods specified by the LC3 standard. Interpolation is performed on the quantized spectrum shaping parameters, and then the spectrum coefficients are shaped based on the interpolated spectrum shaping parameters to indirectly control the distribution characteristics of quantization noise in different frequency bands. Subsequently, the subsequent encoding process defined by the LC3 standard is executed to generate an encoded bitstream that conforms to the LC3 standard. The neural network prediction module functionally replaces the traditional spectrum shaping parameter derivation module in the LC3 encoder, and does not change the bitstream syntax structure, parameter organization method, or decoder processing behavior specified by the LC3 standard.

[0026] Specifically, after predicting and quantizing the spectrum shaping parameters, interpolation is performed based on the quantized spectrum shaping parameters, and the spectral coefficients in the frequency domain are then shaped. Interpolation fills the gaps in parameter distribution in the frequency domain, making parameter transitions more seamless. Shaping, on the other hand, adjusts the energy distribution of the spectral coefficients according to the constraints set for the parameters, thereby indirectly ensuring that the distribution of quantization noise in different frequency regions conforms to the preset optimization target. This optimizes the spectrum shaping parameter derivation process while meeting the requirements of LC3 coding efficiency and standard compatibility.

[0027] After processing, the subsequent encoding process defined in the LC3 standard is continuously executed to finally generate an encoded bitstream that fully conforms to the LC3 standard, ensuring that the encoding result can be parsed normally by the standard LC3 decoder.

[0028] The lightweight neural network used to predict spectral shaping parameters employs a feedforward neural network structure. This structure features a simplified network depth, enabling it to be quantized into low-precision integer form, effectively reducing resource consumption during computation. Furthermore, its inference process has been specifically optimized, allowing for rapid completion on an embedded digital signal processor, achieving low-latency operation and fully meeting the stringent real-time requirements of LC3 encoding.

[0029] It is worth noting that this neural network prediction module only replaces the traditional spectrum shaping parameter derivation module in the LC3 encoder at the functional level. In terms of bitstream syntax structure, parameter organization method and decoder processing behavior, it strictly follows the LC3 standard without any changes, thus ensuring full compatibility with the LC3 standard system.

[0030] The neural network is trained by training objectives that include multiple constraints. The training objectives include at least structural consistency constraints, speech recognition feature preservation constraints, and parameter smoothing constraints. The structural consistency constraints are used to ensure that the predicted spectral shaping parameters are consistent with the traditional LC3 spectral shaping parameters in terms of statistical distribution and structural characteristics. The speech recognition feature preservation constraints are used to reduce the difference between speech before and after encoding in the speech recognition feature space. The parameter smoothing constraints are used to suppress drastic changes between adjacent spectral shaping parameters.

[0031] Specifically, to ensure that the spectral shaping parameters predicted by the neural network meet both the basic requirements of the LC3 coding system and are adaptable to the needs of subsequent speech-related applications, the neural network training process introduces a training objective with multiple constraints. These constraints collaboratively guide the model to output better spectral shaping parameters. Specifically, the structural consistency constraint aligns the predicted parameters with the statistical distribution and structural characteristics of traditional LC3 spectral shaping parameters, ensuring that the predicted parameters do not deviate from the basic behavioral logic of LC3 coding and guaranteeing compatibility with the LC3 standard. The speech recognition feature preservation constraint focuses on controlling the differences in speech recognition feature space before and after encoding. This constraint allows the encoded speech spectral features to retain as much key information as possible from the original speech, reducing the adverse impact of the encoding process on speech recognition performance and improving the adaptability of the encoded speech to the speech recognition system. The parameter smoothing constraint regulates inter-frame parameter changes, effectively suppressing drastic fluctuations in the spectral shaping parameters corresponding to adjacent audio frames, avoiding instability in spectral processing due to parameter mutations, and thus improving the overall stability of parameter quantization and spectral shaping processes. These multiple constraints work together to enable the trained neural network to output spectral shaping parameters that are more conducive to the stable preservation of speech features while meeting the constraints of the LC3 standard.

[0032] Structural consistency constraints are achieved by comparing the spectral shaping parameters predicted by the neural network with the reference spectral shaping parameters output by the traditional LC3 spectral shaping parameter derivation module, ensuring consistency between the two in terms of statistical distribution and structural characteristics; speech recognition feature preservation constraints are achieved by comparing the feature differences of speech before and after encoding, including Mel frequency scale features or speech model embedding space features; parameter smoothness constraints are achieved by limiting the variation amplitude between adjacent spectral shaping parameters.

[0033] When the neural network is deployed on the embedded digital signal processor, fixed-point quantization, chip memory loading, and a dedicated neural network library are used to accelerate execution. The embedded digital signal processor is based on the Cadence HiFi4 architecture, and the dedicated neural network library is Cadence HiFiNNLib. This deployment method meets the requirements for low power consumption and real-time operation.

[0034] The network weights of the neural network use 8-bit fixed-point numbers, the biases use 32-bit fixed-point numbers, and the intermediate activation values ​​use 16-bit fixed-point numbers. The corresponding quantization ratio and zero-point storage are stored in a constant table. During the deployment phase, the model weights and quantization parameters are loaded into the on-chip memory of the embedded digital signal processor to reduce the latency caused by external memory access.

[0035] The time-frequency transformation adopts the modified discrete cosine transform, and the preset frequency band division rules are implemented in accordance with the frequency band division method specified in the LC3 standard to ensure that the acquisition of multi-frequency band energy information meets the requirements of the LC3 coding system.

[0036] Specifically, in the deployment optimization of neural networks, targeted fixed-point quantization design is implemented for network parameters. This allows network weights, biases, and intermediate activation values ​​to adopt fixed-point numbers with specific precision. Low-precision fixed-point numbers are used for network weights to reduce storage footprint and computational complexity, while higher-precision fixed-point numbers are used for biases to ensure computational accuracy. Intermediate activation values ​​are chosen to balance accuracy and efficiency, adapting to the inference process. Simultaneously, the quantization ratio and zero-point information required during quantization are pre-stored as a constant table for direct access during deployment, reducing real-time computational overhead. During deployment, the solidified model weights and quantization parameters are loaded into the embedded digital signal processor's memory. This ensures that data reading and processing during inference are performed within the processor's local high-speed storage area, avoiding frequent access to external memory and significantly reducing latency caused by data transmission, thereby improving inference efficiency.

[0037] In the acquisition of multi-band energy information, the time-frequency transformation employs modified discrete cosine transform. This transformation method effectively adapts to the time-frequency conversion requirements of LC3 encoding, achieving efficient conversion of audio signals from the time domain to the frequency domain. The preset frequency band division rules strictly follow the frequency band division method specified in the LC3 standard, ensuring that the frequency band division logic and range are consistent with the LC3 encoding system when performing band division processing on the converted frequency domain signal. This guarantees that the calculated multi-band energy information accurately meets the requirements of subsequent LC3 encoding processing stages, providing standard-compliant basic feature support for the accurate prediction of spectrum shaping parameters.

[0038] An audio encoding system, comprising: The frequency band energy calculation module is used to perform time-frequency transformation processing on the input audio frame, calculate the energy value of each frequency band according to the preset frequency band division rules, and obtain multi-frequency band energy information; The input feature construction module is used to construct an input feature vector based on multi-band energy information, combined with frame duration, sampling rate, number of available bits in the current frame, transient detection flag, energy statistical features, and optional spectrum shaping parameters of the previous frame. The neural network prediction module is a lightweight feedforward neural network structure used to predict the same number of spectrum shaping parameters as specified in the LC3 standard based on the input feature vector. The neural network uses a low-precision numerical representation and functionally replaces the traditional spectrum shaping parameter derivation module. The parameter quantization and encoding module is used to quantize and encode the spectrum shaping parameters according to the LC3 standard to generate the corresponding spectrum shaping side information. The spectrum shaping module performs interpolation on the quantized spectrum shaping parameters, and then shapes the spectrum coefficients based on the interpolated spectrum shaping parameters. Its implementation method is consistent with the LC3 standard. The subsequent encoding module is used to execute the subsequent processes defined by the LC3 standard, such as quantization, entropy encoding, and bitstream encapsulation, to generate an encoded bitstream that conforms to the LC3 standard.

[0039] Specifically, the audio coding system is built around the LC3 standard architecture. Through the coordinated operation of various functional modules, it optimizes the derivation process of spectrum shaping parameters while maintaining standard compatibility, thereby achieving efficient audio coding that is adaptable to multiple scenarios.

[0040] The frequency band energy calculation module, as the core of the front-end processing, is responsible for converting the input audio frame from the time domain to the frequency domain, and then calculating the energy value of each frequency band according to the preset frequency band division rules. Finally, it outputs multi-frequency band energy information that can accurately reflect the spectral distribution characteristics of the current audio frame, providing basic frequency domain data support for subsequent parameter prediction.

[0041] The input feature construction module integrates key frame-level parameters such as frame duration, sampling rate, and available bits in the current frame, based on the multi-band energy information output by the frequency band energy calculation module. It also incorporates transient detection flags and energy statistical features. The transient detection flags capture transient energy changes in the audio frame, while the energy statistical features present the overall distribution pattern of the spectrum. Furthermore, this module can selectively introduce spectral shaping parameters from the previous frame. By mining inter-frame correlation information, it constructs a comprehensive and temporally coherent input feature vector, providing ample information for neural network prediction.

[0042] The neural network prediction module employs a lightweight feedforward neural network structure, reducing computational complexity and resource consumption through low-precision numerical representation, thus adapting it to the operational requirements of embedded scenarios. Functionally, this module completely replaces the traditional rule-based spectrum shaping parameter derivation module. After receiving the feature vector generated by the input feature construction module, it accurately predicts the number of spectrum shaping parameters consistent with the LC3 standard, ensuring that the parameters are consistent with the standard requirements in terms of quantity and meaning, thus guaranteeing compatibility for subsequent encoding processes.

[0043] The parameter quantization and encoding module strictly follows the relevant provisions of the LC3 standard, performs quantization processing on the spectrum shaping parameters output by the neural network prediction module, converts the parameters into a standard-compliant representation, and then generates the corresponding spectrum shaping side information through encoding, providing the necessary parameter data for the construction of the bit stream.

[0044] The spectrum shaping module follows the same implementation method as the LC3 standard, performing interpolation based on the quantized spectrum shaping parameters and shaping the spectral coefficients in the frequency domain. Interpolation ensures a smooth transition of parameters, while shaping optimizes the energy distribution of the spectral coefficients, indirectly making the distribution of quantization noise in the frequency domain more reasonable. This optimizes the derivation process of the spectrum shaping parameters while meeting the LC3 coding efficiency and standard compatibility requirements.

[0045] The subsequent encoding module takes over the processing results from the spectrum shaping module and strictly executes the subsequent processes defined by the LC3 standard, such as quantization, entropy coding, and bitstream encapsulation, to convert the pre-processed audio data into an LC3-compliant encoded bitstream. This ensures that the bitstream can be parsed normally by all standard LC3 decoders, guaranteeing the system's complete compatibility with the LC3 standard system.

[0046] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A method for deriving spectral shaping parameters in an LC3 coding framework based on neural networks, characterized in that, Applied to the LC3 audio encoding system, the method includes the following steps: Perform time-frequency transformation processing on the input audio frame, calculate the energy value of each frequency band according to the preset frequency band division rules, and obtain multi-frequency band energy information that characterizes the spectral distribution of the current frame. Based on the multi-band energy information, an input feature vector is constructed by combining frame duration, sampling rate, number of available bits in the current frame, transient detection flag, and energy statistical features. The transient detection flag is used to indicate whether there is a transient energy change in the current audio frame, and the energy statistical features are used to describe the overall spectrum distribution characteristics. The input feature vector is input to the lightweight neural network prediction module deployed in the LC3 encoder. The neural network prediction module outputs a number of spectral shaping parameters consistent with the number specified in the LC3 standard. The neural network adopts a low-precision numerical representation, and the number of parameters and computational complexity are adapted to the real-time operation requirements of the embedded digital signal processor. The spectrum shaping parameters are input into the parameter quantization and encoding module defined by the LC3 standard, and the corresponding spectrum shaping side information is generated according to the quantization rules and encoding methods specified by the LC3 standard. Interpolation is performed on the quantized spectrum shaping parameters, and then the spectrum coefficients are shaped based on the interpolated spectrum shaping parameters to indirectly control the distribution characteristics of quantization noise in different frequency bands. Subsequently, the subsequent encoding process defined by the LC3 standard is executed to generate an encoded bitstream that conforms to the LC3 standard. The neural network prediction module functionally replaces the traditional spectrum shaping parameter derivation module in the LC3 encoder, and does not change the bitstream syntax structure, parameter organization method, or decoder processing behavior specified by the LC3 standard.

2. The method according to claim 1, characterized in that, The input feature vector also includes the spectrum shaping parameter corresponding to the previous audio frame. By introducing this parameter, the neural network's ability to model the temporal continuity between adjacent audio frames is enhanced, thereby improving the stability of the predicted spectrum shaping parameter between frames.

3. The method according to claim 1, characterized in that, The lightweight neural network is a feedforward neural network structure with a simplified network depth. It can be quantized into low-precision integer form, and the inference process can be completed in sub-milliseconds on an embedded digital signal processor, meeting the real-time requirements of LC3 encoding.

4. The method according to claim 1, characterized in that, The neural network is trained using a training objective that includes multiple constraints. The training objective includes at least structural consistency constraints, speech recognition feature preservation constraints, and parameter smoothing constraints. The structural consistency constraints are used to limit the predicted spectral shaping parameters to maintain consistency with the traditional LC3 spectral shaping parameters in terms of statistical distribution and structural characteristics. The speech recognition feature preservation constraints are used to reduce the difference between speech before and after encoding in the speech recognition feature space. The parameter smoothing constraints are used to suppress drastic changes between adjacent spectral shaping parameters.

5. The method according to claim 4, characterized in that, The structural consistency constraint is achieved by comparing the spectrum shaping parameters predicted by the neural network with the reference spectrum shaping parameters output by the traditional LC3 spectrum shaping parameter derivation module, ensuring consistency between the two in statistical distribution and structural characteristics; the speech recognition feature preservation constraint is achieved by comparing the feature differences of speech before and after encoding, the features including Mel frequency scale features or speech model embedding space features; the parameter smoothing constraint is achieved by limiting the variation amplitude between adjacent spectrum shaping parameters.

6. The method according to claim 1, characterized in that, When the neural network is deployed on an embedded digital signal processor, it adopts fixed-point quantization, chip memory loading, and a dedicated neural network library to accelerate execution. The embedded digital signal processor is based on the Cadence HiFi4 architecture, and the dedicated neural network library is Cadence HiFiNNLib. This deployment method meets the requirements for low power consumption and real-time operation.

7. The method according to claim 6, characterized in that, The network weights of the neural network are 8-bit fixed-point numbers, the biases are 32-bit fixed-point numbers, and the intermediate activation values ​​are 16-bit fixed-point numbers. The corresponding quantization ratios and zero-points are stored in a constant table. During the deployment phase, the model weights and quantization parameters are loaded into the on-chip memory of the embedded digital signal processor to reduce the latency caused by external memory access.

8. The method according to claim 1, characterized in that, The time-frequency transformation adopts the modified discrete cosine transform, and the preset frequency band division rule is executed in accordance with the frequency band division method specified in the LC3 standard to ensure that the acquisition of multi-frequency band energy information meets the requirements of the LC3 coding system.

9. An audio encoding system, comprising the method according to any one of claims 1-8, characterized in that, include: The frequency band energy calculation module is used to perform time-frequency transformation processing on the input audio frame, calculate the energy value of each frequency band according to the preset frequency band division rules, and obtain multi-frequency band energy information; The input feature construction module is used to construct an input feature vector based on the multi-band energy information, combined with frame duration, sampling rate, number of available bits in the current frame, transient detection flag, energy statistical features, and optional spectrum shaping parameters of the previous frame. The neural network prediction module is a lightweight feedforward neural network structure used to predict the same number of spectrum shaping parameters as specified in the LC3 standard based on the input feature vector. The neural network adopts a low-precision numerical representation and functionally replaces the traditional spectrum shaping parameter derivation module. The parameter quantization and encoding module is used to quantize and encode the spectrum shaping parameters according to the LC3 standard to generate corresponding spectrum shaping side information; The spectrum shaping module performs interpolation on the quantized spectrum shaping parameters, and then shapes the spectrum coefficients based on the interpolated spectrum shaping parameters. Its implementation method is consistent with the LC3 standard. The subsequent encoding module is used to execute the subsequent processes defined by the LC3 standard, such as quantization, entropy encoding, and bitstream encapsulation, to generate an encoded bitstream that conforms to the LC3 standard.

10. A computer-readable storage medium having program instructions stored thereon, characterized in that, When the program instructions are executed by the processor, the processor performs the spectral shaping parameter derivation method in the LC3 coding framework based on neural networks as described in any one of claims 1 to 8.

Images