Audio codec model construction method, apparatus and device

Abstract

The application discloses an audio coding model construction method, device and equipment. The method includes the following steps: constructing a network structure of an audio coding model adopting a dynamic frame frequency mechanism; learning network parameters of the model from audio training data; dividing a multi-level residual vector quantizer related to acoustic characteristics in the model into multiple quantizer groups; obtaining audio training data of different frequency bands corresponding to different quantizer groups; and adjusting network parameters of the quantizer groups according to the audio training data of the frequency bands corresponding to the quantizer groups. By using this processing method, different frequency band-specific quantization optimization strategies are adopted for acoustic characteristics of different frequency bands in the case of dynamic frame rate, dynamic frame rate and frequency band reconstruction are coordinated, and low-frequency energy of an audio signal is prevented from covering high-frequency details when the dynamic frame rate is low; therefore, the full-band reconstruction quality is effectively ensured to be uniform, and the audio reconstruction quality is improved.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, specifically to methods and apparatus for constructing audio codec models, as well as electronic devices. Background Technology

[0002] In recent years, advancements in artificial intelligence (AI) technology have dramatically changed the way humans interact with machines, particularly in the field of speech processing. A typical audio processing model is a neural audio encoding / decoding model based on a dynamic frame rate mechanism, used to efficiently compress audio to reduce storage or network bandwidth requirements.

[0003] The neural audio codec model based on dynamic frame rate mechanism consists of an encoder, a feature aggregation module, a residual vector quantization (RVQ) module, a feature deaggregation module, and a decoder. The encoder converts the input audio stream into a coded signal (acoustic features represented by continuous values); the feature aggregation module merges the acoustic features of adjacent similar frames; the residual vector quantization replaces the merged acoustic features with approximate vectors (vector elements are integers) from the codebook, obtaining acoustic features represented by discrete values ​​(acoustic feature tokens). This process is called vector quantization, and its purpose is to efficiently compress the amount of acoustic feature data; the feature deaggregation module deaggregates the acoustic tokens; and the decoder converts the deaggregated acoustic tokens back into audio, completing audio reconstruction. Currently, the RVQ architecture in this neural audio codec model based on dynamic frame rate mechanism adopts a "one-size-fits-all" quantization method across the entire frequency band of the audio signal.

[0004] Ideally, audio codecs should be transparent to end users, making it impossible to distinguish the decoded audio from the original audio at an auditory level, and avoiding perceptible latency introduced during the encoding / decoding process. However, in the process of implementing this application, the applicant discovered that the prior art has at least the following problems: when the dynamic frame rate is low, the low-frequency energy of the audio signal will mask the high-frequency details of the audio signal, and a large amount of high-frequency semantic information (such as sibilance and instrument transients) will be lost, resulting in uneven spectral reconstruction quality in the 0–24 kHz range. For example, the MUSHRA (Multi-Stimuli with Hidden Reference and Anchor) score in the 16–24 kHz high-frequency band is often below 70, and users can clearly distinguish the reconstructed audio from the original audio at an auditory level. Summary of the Invention

[0005] This application provides a method for constructing an audio codec model to address the problem in existing technologies where low-frequency energy in audio signals masks high-frequency details at low dynamic frame rates. This application also provides an audio codec model construction apparatus and system, as well as an electronic device.

[0006] This application provides a method for constructing an audio codec model, including: Construct a network structure for an audio codec model that employs a dynamic frame rate mechanism; The network parameters of the model are learned from the audio training data; The multi-level residual vector quantizer related to acoustic features in the model is divided into multiple quantizer groups; Acquire audio training data for different frequency bands corresponding to different quantizer groups; The network parameters of the quantizer group are adjusted based on the audio training data of the frequency band corresponding to the quantizer group.

[0007] Optionally, the multi-level residual vector quantizer is divided into multiple quantizer groups, including: The multi-level residual vector quantizer is divided into three groups of quantizers; The step of adjusting the network parameters of the quantizer group based on the audio training data of the frequency band corresponding to the quantizer group includes: Based on the low-frequency audio training data, adjust the network parameters of the first set of quantizers; Based on the audio training data in the mid-to-low frequency band, adjust the network parameters of the second set of quantizers; Based on the high-frequency audio training data, adjust the network parameters of the third quantizer.

[0008] Optionally, the audio codec model includes a dual-stream audio codec model, which includes an acoustic feature quantization module and a semantic feature quantization module; The step of dividing the multi-level residual vector quantizer related to acoustic features in the model into multiple quantizer groups includes: The multi-level residual vector quantizer in the acoustic feature quantization module is divided into multiple quantizer groups.

[0009] Optionally, the model includes an acoustic feature de-aggregation module, used to de-aggregate the acoustic feature tokens output by the acoustic feature quantization module based on the semantic feature tokens output by the semantic feature quantization module after frame merging.

[0010] Optionally, the model includes a feature aggregation module; The feature aggregation module is used to obtain the feature similarity between adjacent frames; if the feature similarity is greater than or equal to the feature similarity threshold, the features of adjacent frames are merged; and the aggregated features are adjusted according to the features of the original frames before merging.

[0011] Optionally, the model includes a feature de-aggregation module; The feature de-aggregation module is used to obtain the number of frames in the merged frame; restore the aggregated features to the original number of frames based on the number of frames; and adjust the features of the restored frame based on the context features.

[0012] Optionally, the model includes a dual-stream audio codec model, and the feature aggregation module includes an acoustic feature aggregation module and a semantic feature aggregation module; The acoustic feature aggregation module is used to obtain the acoustic feature similarity between adjacent frames; if the acoustic feature similarity is greater than or equal to the acoustic feature similarity threshold, the acoustic features of adjacent frames are merged; and the acoustic aggregation features are adjusted according to the acoustic features of the original frames before merging. And / or, the semantic feature aggregation module is used to merge semantic feature adjacent frames corresponding to the merged acoustic feature adjacent frames; and adjust the semantic aggregation features according to the semantic features of the original frames before merging.

[0013] Optionally, the model includes a feature de-aggregation module, which includes an acoustic feature de-aggregation module and a semantic feature de-aggregation module; The acoustic feature de-aggregation module is used to obtain the number of frames in the acoustic feature merging frame; restore the acoustic aggregated features to the original number of frames based on the number of frames; and adjust the acoustic features of the restored frame based on the context acoustic features. And / or, the semantic feature de-aggregation module is used to obtain the number of frames; restore the semantic aggregated features to the original number of frames based on the number of frames; and adjust the semantic features of the restored frames based on the context semantic features.

[0014] Optionally, the feature aggregation module is also used to obtain a feature similarity threshold corresponding to the audio type.

[0015] Optionally, the aggregated features can be adjusted based on the features of the original frames before merging using a local window attention module.

[0016] Optionally, the training data includes labeled data of merged frames.

[0017] Optionally, the encoder and decoder employ a transformer architecture that introduces a self-attention mechanism.

[0018] This application provides an audio codec model construction apparatus, comprising: Network building units are used to construct network structures for audio codec models that employ dynamic frame rate mechanisms. A full network training unit is used to learn the network parameters of the model from audio training data; A quantizer grouping unit is used to divide the multi-level residual vector quantizer related to acoustic features in the model into multiple quantizer groups; A multi-band audio acquisition unit is used to acquire audio training data of different frequency bands corresponding to different quantizer groups; The frequency band training unit is used to adjust the network parameters of the quantizer group based on the audio training data of the frequency band corresponding to the quantizer group.

[0019] This application provides an electronic device, including: Processor; and A memory for storing a program for implementing the method described in any of the preceding methods, wherein the device is powered on and the program of the method is executed by the processor.

[0020] This application also provides a computer-readable storage medium storing instructions that, when executed on a computer, cause the computer to perform the various methods described above.

[0021] This application also provides a computer program product including instructions that, when run on a computer, cause the computer to perform the various methods described above.

[0022] Compared with the prior art, this application has the following advantages: The audio codec model construction method provided in this application constructs a network structure for an audio codec model employing a dynamic frame rate mechanism; learns the network parameters of the model from audio training data; divides the multi-level residual vector quantizer related to acoustic features in the model into multiple quantizer groups; acquires audio training data for different frequency bands corresponding to different quantizer groups; and adjusts the network parameters of the quantizer groups based on the audio training data for the frequency bands corresponding to the quantizer groups. This approach enables the use of different frequency band-specific quantization optimization strategies for acoustic features in different frequency bands when using a dynamic frame rate, achieving synergy between dynamic frame rate and frequency band reconstruction. This avoids low-frequency energy masking high-frequency details in the audio signal when the dynamic frame rate is low; therefore, it effectively ensures uniform reconstruction quality across the entire frequency band (e.g., 0–24 kHz spectrum) and improves audio reconstruction quality. Attached Figure Description

[0023] Figure 1 This is a flowchart illustrating an embodiment of the audio codec model construction method provided in this application; Figure 2 This is a schematic diagram of an embodiment of the audio codec model construction method provided in this application. Detailed Implementation

[0024] Many specific details are set forth in the following description to provide a full understanding of this application. However, this application can be implemented in many other ways different from those described herein, and those skilled in the art can make similar extensions without departing from the spirit of this application; therefore, this application is not limited to the specific embodiments disclosed below.

[0025] This application provides a method and apparatus for constructing audio codec models, as well as an electronic device. The various solutions are described in detail below in each embodiment.

[0026] First Embodiment Please refer to Figure 1 This is a flowchart of the audio codec model construction method of this application. In this embodiment, the method may include the following steps: Step S101: Construct the network structure of the audio codec model using a dynamic frame rate mechanism.

[0027] The method provided in this application embodiment is used to construct an audio codec model employing a dynamic frame rate mechanism. The audio codec model is an end-to-end model learned from audio training data. Model construction includes two stages: 1. Constructing the network structure of the model; 2. Training the model parameters based on the training data. This step S101 involves constructing the network structure of the model.

[0028] In one example, the audio codec model adopts a five-segment architecture: Encoder-Aggregation Module-Quantizer-Disaggregation Module-Decoder. The audio signal is first processed by the encoder to extract audio features; then, the feature aggregation module merges the audio features of adjacent similar frames to achieve a dynamic frame rate mechanism; next, the residual vector quantizer converts the signal into discrete codes (audio features represented by discrete values, audio feature tokens); then, the feature disaggregation module restores the merged audio feature tokens to the original frame rate; finally, the decoder reconstructs the audio. This single-stream fusion approach mixes and quantizes the acoustic and semantic features of the audio signal, thus effectively improving processing efficiency and saving system resources.

[0029] In practice, the audio signal is processed by an encoder to extract features, which can be achieved as follows: the audio signal is processed by an acoustic encoder to extract acoustic features, and by a semantic encoder to extract semantic features. The features of adjacent similar frames are merged through a feature aggregation module, which can be achieved as follows: the acoustic feature aggregation module obtains the acoustic feature similarity between adjacent frames; if the acoustic feature similarity is greater than or equal to the acoustic feature similarity threshold, the acoustic features of adjacent frames are merged; the semantic feature aggregation module merges the semantic feature adjacent frames corresponding to the merged acoustic feature adjacent frames. The acoustic and semantic features of the merged frames are fused, and the quantizer converts the fused features, represented by continuous values, into fused feature tokens, represented by discrete values; the feature de-aggregation module restores the merged frame fused feature tokens to the original frame rate, and finally, the decoder reconstructs the audio.

[0030] In another example, the audio codec model is a two-stream neural audio codec model. This model employs a two-stream coding structure, including an acoustic codec module and a semantic codec module. The acoustic codec module handles acoustic features and includes an acoustic encoder, an acoustic feature aggregation module, an acoustic feature quantization module, an acoustic feature token deaggregation module, and an acoustic decoder. The semantic codec module handles semantic features and includes a semantic encoder, a semantic feature aggregation module, a semantic feature quantization module, a semantic feature token deaggregation module, and a semantic decoder. This two-stream architecture allows the acoustic branch to focus on waveform detail reconstruction, while the semantic branch preserves high-level semantics. It retains rich linguistic content while compressing high-dimensional features using the quantizer RVQ, avoiding low-level detail noise interfering with high-level semantics.

[0031] Figure 2 An example of a dual-stream neural audio codec model employing a dynamic frame rate mechanism is presented. Figure 2As can be seen, the model includes two encoding and decoding processes: one is the acoustic encoding and decoding module, which includes an acoustic encoder, an acoustic feature aggregation module (AAM), an acoustic feature quantization module (RVQ), an acoustic feature token deaggregation module (ADM), a feature fusion module (C), and an acoustic decoder; the other is the semantic encoding and decoding module, which includes a semantic encoder, a semantic feature aggregation module (SAM), a semantic feature quantization module (Semantic RVQ), a semantic feature token deaggregation module (SDM), and a semantic decoder. The acoustic encoder extracts acoustic features from the audio signal and outputs a sequence of acoustic feature vectors represented by continuous values, such as 50 frames per second (20ms frame length, 50 frames per second) and 50 acoustic feature vectors per second. The semantic encoder extracts semantic features from the audio signal. The acoustic feature aggregation module obtains the acoustic feature similarity between adjacent frames; if the acoustic feature similarity is greater than or equal to an acoustic feature similarity threshold, the acoustic features of adjacent frames are merged. The semantic feature aggregation module merges the semantic feature adjacent frames corresponding to the merged acoustic feature adjacent frames. The acoustic feature quantization module converts the aggregated acoustic features into acoustic feature tokens. Acoustic feature tokens are acoustic feature vectors represented by discrete values ​​(integers), such as 50 acoustic feature tokens per second; the acoustic feature de-aggregation module is used to de-aggregate acoustic feature tokens based on the number of frames in the merged frames; the semantic feature quantization module is used to convert the aggregated semantic features into semantic feature tokens; the semantic feature de-aggregation module is used to de-aggregate semantic feature tokens based on the number of frames in the merged frames; the feature fusion module is used to fuse the de-aggregated acoustic feature tokens and the de-aggregated semantic feature tokens; the acoustic decoder is used to reconstruct audio based on the fused features; and the semantic decoder is used to reconstruct semantic features based on the de-aggregated semantic feature tokens.

[0032] In one example, the Acoustic Feature De-aggregation (ADM) module is used to de-aggregate the frame-merged acoustic feature tokens output by the Semantic Feature Quantization (SDM) module based on the frame-merged semantic feature tokens output by the Semantic Feature Quantization (SDM) module. This processing method allows the quantized discrete semantic features (decoding information from the SDM module) output by the Semantic Feature Quantization (RVQ) module to be fed back to the ADM module for guidance, thereby optimizing dynamic compression decoding performance.

[0033] In one example, the encoder uses SEANet. The SEANet encoder employs a design combining convolutional neural networks and residual blocks. An initial convolutional layer converts the audio waveform into a feature map, and multiple convolutional blocks progressively reduce the feature map resolution. Cross-layer information transfer is achieved through SEANetResnetBlock. The SEANet decoder is structurally symmetrical to the encoder and is responsible for reconstructing the audio waveform from the quantized feature vectors, using transposed convolutions for upsampling.

[0034] In another example, both the encoder and decoder employ a transformer architecture that incorporates a self-attention mechanism. This approach enhances the modeling capability for long sequences, thereby improving model performance.

[0035] Step S103: Learn the network parameters of the model from the audio training data.

[0036] The method provided in this application embodiment trains the model multiple times. Step S103 is the first training. The audio training data used in the first training is not limited to the frequency range of the audio data and can include audio training data of the entire frequency band. Through machine learning algorithms, the model is trained for the first time based on the audio training data of the entire frequency band, and the network parameters of the model are learned from the audio training data.

[0037] Step S105: Divide the multi-level residual vector quantizer related to acoustic features in the model into multiple quantizer groups.

[0038] Step S107: Obtain audio training data for different frequency bands corresponding to different quantizer groups.

[0039] Step S109: Adjust the network parameters of the quantizer group based on the audio training data of the frequency band corresponding to the quantizer group.

[0040] The method provided in this application embodiment divides the multi-level quantizer related to acoustic features in the model into multiple groups in step S105, and performs multi-stage frequency band separation progressive training in step S109 to achieve codebook professional optimization.

[0041] In one example, step S105 can be implemented as follows: the multi-level residual vector quantizer is divided into three groups of quantizers. For example, the multi-level residual vector quantizer is a 10-level quantizer, with layers 1-4 in the first group, layers 5-8 in the second group, and layers 8-10 in the third group. Step S109 can be implemented as follows: adjust the network parameters of the first group of quantizers based on the low-frequency audio training data; adjust the network parameters of the second group of quantizers based on the mid-low frequency audio training data; and adjust the network parameters of the third group of quantizers based on the high-frequency audio training data. Using this "encoder downsampling guidance - quantizer grouping freeze" training method, the dedicated RVQ codebook is optimized in stages for the low-frequency (0–8kHz), mid-low frequency (8–16kHz), and high-frequency (16–24kHz) bands, realizing a frequency band-specific three-stage progressive training mechanism. This avoids multi-objective gradient conflicts, overcomes the problem of full-band reconstruction imbalance, and achieves transparent reconstruction of high-frequency details in 48kHz audio.

[0042] In one example, the audio codec model is a dual-stream neural audio codec model employing a dynamic frame rate mechanism. The dual-stream neural audio codec model includes an acoustic feature quantization module and a semantic feature quantization module. Step S105 can be implemented as follows: the multi-level residual vector quantizer included in the acoustic feature quantization module of the dual-stream neural audio codec model is divided into multiple quantizer groups.

[0043] In one example, the acoustic feature quantization module includes a ten-level residual vector quantizer, and the semantic feature quantization module includes a two-level residual vector quantizer. This improves both audio detail processing and semantic processing efficiency. The first set of quantizers includes levels 1-4, corresponding to audio with a sampling rate of 16kHz (preserving the "main frequency band" of human voice, 0-8kHz). The first stage trains the codebook for the first set of quantizers in the acoustic branch, focusing on fundamental frequency reconstruction; the semantic branch (semantic feature quantization module) simultaneously processes the downsampled input of the semantic encoder, establishing a low-frequency semantic mapping (converting semantic features into semantic feature tokens). The second set of quantizers includes levels 4-8, corresponding to 32kHz audio (preserving 0-16kHz). The second stage freezes the first set of quantizers and trains the codebook for the second set of quantizers, enhancing consonant details in the 8–16 kHz range; the semantic branch adapts the mid-frequency semantic representation. The third quantizer group includes quantizers at levels 8-10. The third quantizer group corresponds to 48kHz audio (preserving 0-24kHz). In the third stage, the first and second quantizer groups are frozen, and the codebook of the third quantizer group is trained to focus on high-frequency harmonic completion of 16–24 kHz. Semantic branches process the full-frequency input to ensure the integrity of high-level semantics.

[0044] In specific implementation, Figure 2The dual-stream neural audio codec model demonstrated implements a three-stage progressive training with frequency band separation. Figure 2 In the acoustic branch, the original waveform (audio data) is input into a convolutional acoustic encoder to extract multi-scale time-frequency features (acoustic features). These acoustic features flow through a content-aware acoustic feature aggregation module (lightweight Transformer + attention), dynamically compressing the frame rate based on real-time calculated Mel spectral entropy and zero-crossing rate (frame rate variation range: 5–25 Hz). The compressed acoustic features are then fed into three sets of residual vector quantizers (RVQs), generating acoustic feature tokens focused on low frequencies (0–8 kHz), mid-low frequencies (8–16 kHz), and high frequencies (16–24 kHz). In the semantic branch, the original waveform is synchronously input into a pre-trained WavLM-large model to extract high-level semantic features. After processing by a lightweight semantic encoder and a synchronous frame rate adjustment module (sharing content analysis results with the acoustic branch), the semantic features are independently generated by RVQs to produce a semantic feature token stream, preserving key semantics such as phonemes, prosody, and speaker identity. After quantization, the acoustic feature token and semantic feature token are reconstructed using the de-aggregation module (transposed convolution + interpolation) to restore the original temporal resolution. The resulting concatenation is then input into the acoustic decoder to reconstruct the waveform.

[0045] like Figure 2 As shown, in specific implementation, a joint reconstruction loss can be used when training the dual-stream audio codec model to constrain the dual-stream collaborative optimization. The joint reconstruction mechanism refers to the following: during the decoding stage, the quantized acoustic feature token and semantic feature token are concatenated in the hidden layer dimension and input to the acoustic decoder. The original waveform is reconstructed through the ISTFT header, and the corresponding loss is the waveform reconstruction loss. The reconstruction loss is responsible for ensuring that the reconstructed waveform is as close as possible to the original waveform. The semantic feature token is simultaneously input to the semantic decoder to reconstruct the SSL features and calculate the auxiliary MSE loss (Mean Squared Error Loss). This ensures that semantic information is not lost due to quantization and constrains the semantic information to be lossless during the quantization process. The corresponding loss is the semantic auxiliary loss.

[0046] In one example, during model training, the following loss functions are fused: waveform reconstruction loss (Lmel, Mel spectrogram loss), multi-discriminator adversarial loss (loss of least-squares generative adversarial network (GAN) based on multi-period discriminator (MPD) / multi-scale discriminator (MSD) / subband STFT), feature matching loss (LFM), quantizer commit loss (Lcommit), and semantic auxiliary loss (Laux). The total loss function is calculated as: Ltotal = λ1Lmel + λ2Ladv + λ3Lfm + λ4Lcommit + λ5Laux. A perceptual loss is then introduced later to further improve the perceived sound quality. This approach achieves a dual-path reconstruction and multi-objective joint optimization mechanism.

[0047] In practice, large-scale training and multi-domain generalization validation can be employed. For example, training can be conducted based on 5 million hours of speech, 800,000 hours of music, and 200,000 hours of general audio.

[0048] In one example, a multi-stage training strategy is employed to gradually bridge the gap between "low bitrate compression" and "high audio quality". The first stage trains the encoder and decoder simultaneously, rapidly converging the basic representation; the second stage freezes the encoder and fine-tunes the decoder, further improving the convergence performance.

[0049] As can be seen, the dual-stream audio codec model in this embodiment is a dual-stream neural audio codec system that deeply integrates multi-band separation and dynamic frame rate mechanisms. Through acoustic-semantic decoupling, dynamic bit rate control, and frequency band separation training, it efficiently discretizes and reconstructs the original waveform signal with high fidelity, achieving accurate reconstruction of the entire frequency band and adaptive allocation of computing resources. It achieves a breakthrough balance in the triangular constraints of "compression efficiency-reconstruction quality-semantic preservation", and can provide high-fidelity, low-latency, and semantically rich discrete audio representations for large audio-language models (ALMs).

[0050] In one example, the audio codec model includes a feature aggregation module, which is used to obtain the feature similarity between adjacent frames. If the feature similarity is greater than or equal to a feature similarity threshold, the features of adjacent frames are merged. The aggregated features are then adjusted based on the features of the original frames before merging. This processing method enhances temporal continuity and compensates for the loss of high-frequency details caused by frame merging by adjusting the aggregated features based on the features of the original frames before merging.

[0051] In practice, the similarity of features between adjacent frames is calculated, such as cosine similarity. Features of highly similar adjacent frames are merged to generate aggregated features. For example, highly similar frame groups are pooled to generate aggregated features, and a frame number attribute (length attribute, Length Token) is added. The aggregated features are adjusted based on the features of the original frames before merging using a Local Attention module. For example, the original frames and aggregated features are interleaved, with the original frames as the query term and the aggregated features as key / value pairs. Feature enhancement is performed using a Transformer with a Local Attention mechanism (e.g., window size = 3). The final merged frame (adjusted aggregated features) is retrieved from the Transformer output, and the compressed feature sequence (frame rate 5–25 Hz) and frame number are output.

[0052] In practice, the training data may include labeled data of merged frames, which can be used as learnable embedding vectors to guide frame decomposition at the decoding end, thereby effectively improving the accuracy of frame decomposition.

[0053] In one example, the audio codec model includes a feature de-aggregation module, which obtains the frame number of the merged frames; restores the aggregated features to the original frame number based on the frame number; and adjusts the features of the restored frames based on contextual features. This processing method involves repeating each merged frame based on the received frame number. For example, merging frames t2 and t3 involves copying the aggregated features of these two frames twice to restore the original sequence length. While the length of the repeated frame sequence is restored, it contains step-like redundancy. Contextual information is used to eliminate the hard boundaries caused by repetition, smoothing the features. For example, it can be input into a Transformer with a local attention mechanism, using contextual information to eliminate hard boundaries caused by repetition and smooth the features. Finally, the restored original rate frame sequence is output for use by subsequent decoders or synthesizers.

[0054] In one example, the feature aggregation module in the dual-stream audio codec model includes an acoustic feature aggregation module and a semantic feature aggregation module. The acoustic feature aggregation module is used to obtain the acoustic feature similarity between adjacent frames. If the acoustic feature similarity is greater than or equal to the acoustic feature similarity threshold, the acoustic features of adjacent frames are merged. The acoustic aggregated features are adjusted according to the acoustic features of the original frames before merging. This processing method enables the calculation of the similarity of acoustic features between adjacent frames. The acoustic features of adjacent frames with high similarity are merged to generate acoustic aggregated features. The acoustic aggregated features are adjusted according to the acoustic features of the original frames before merging to enhance temporal continuity.

[0055] In one example, the semantic feature aggregation module is used to merge semantic feature-adjacent frames corresponding to the merged acoustic feature-adjacent frames; and adjusts the semantic aggregated features based on the semantic features of the original frames before merging. This processing method allows the semantic features of adjacent frames corresponding to the merged acoustic feature to be merged, generating semantic aggregated features; and the adjustment of the semantic aggregated features based on the semantic features of the original frames before merging enhances temporal continuity.

[0056] In one example, the feature de-aggregation module in the dual-stream audio codec model includes an acoustic feature de-aggregation module and a semantic feature de-aggregation module. The acoustic feature de-aggregation module is used to obtain the frame number of the acoustic feature merging frame; based on the frame number, the acoustic aggregated features are restored to the original frame number; and the acoustic features of the restored frame are adjusted based on the contextual acoustic features. This processing method involves repeating each merged frame according to the received frame number. For example, merging frames t2 and t3 involves copying the aggregated features of these two frames twice to restore the original sequence length. Although the length of the repeated frame sequence is restored, there is a step-like redundancy; contextual information is used to eliminate the hard boundaries caused by repetition and smooth the features. For example, it can be input into a Transformer with a local attention mechanism to eliminate the hard boundaries caused by repetition and smooth the features using contextual information. Finally, the restored original rate frame sequence is output for use by subsequent decoders or synthesizers.

[0057] In one example, the semantic feature de-aggregation module is used to obtain the frame number; restore the semantic aggregated features to the original frame number based on the frame number; and adjust the semantic features of the restored frame based on the contextual semantic features. This processing method enables dual-stream shared merged frame numbers, ensuring zero acoustic-semantic timing misalignment under variable frame rates and avoiding cross-modal misalignment. It effectively reduces the complexity of encoding / decoding synchronization logic and significantly improves system robustness and engineering maintainability. In specific implementation, through synchronous frame rate adjustment and joint reconstruction loss, the naturalness of sound quality and the comprehensibility of content are synchronously guaranteed under a dynamic frame rate of 5–25 Hz.

[0058] In one example, the feature aggregation module in the audio codec model is also used to obtain a feature similarity threshold corresponding to the audio type. If the feature similarity of adjacent frames is greater than or equal to the feature similarity threshold, the features of adjacent frames are merged. For example, the feature similarity threshold for speech segments (sampling rate 5–15 Hz) is 0.85, and the feature similarity threshold for music segments (sampling rate 15–25 Hz) is 0.75. This content-aware threshold adaptive processing method allows the model inference stage to dynamically switch the frame merging threshold according to the real-time audio type, achieving adaptive high-quality reconstruction of speech, music, and ambient sound.

[0059] As can be seen from the above embodiments, the audio codec model construction method provided in this application constructs a network structure for an audio codec model employing a dynamic frame rate mechanism; learns the network parameters of the model from audio training data; divides the multi-level residual vector quantizer related to acoustic features in the model into multiple quantizer groups; acquires audio training data for different frequency bands corresponding to different quantizer groups; and adjusts the network parameters of the quantizer groups based on the audio training data for the frequency bands corresponding to the quantizer groups. This processing method enables the use of different frequency band-specific quantization optimization strategies for acoustic features in different frequency bands when using a dynamic frame rate, achieving synergy between dynamic frame rate and frequency band reconstruction. This avoids low-frequency energy masking high-frequency details in the audio signal when the dynamic frame rate is low; therefore, it can effectively ensure uniform reconstruction quality across the entire frequency band (e.g., 0–24 kHz spectrum) and improve audio reconstruction quality.

[0060] Second Embodiment In the above embodiments, a method is provided; correspondingly, this application also provides an apparatus. This apparatus corresponds to the embodiments of the method described above. Since the apparatus embodiments are basically similar to the method embodiments, they are described simply; relevant details can be found in the descriptions of the method embodiments. The apparatus embodiments described below are merely illustrative.

[0061] This application also provides an apparatus comprising: a network construction unit, a full network training unit, a quantizer grouping unit, a multi-band audio acquisition unit, and a frequency-band training unit.

[0062] The network construction unit is used to construct the network structure of the audio codec model using a dynamic frame rate mechanism; the full network training unit is used to learn the network parameters of the model from the audio training data; the quantizer grouping unit is used to divide the multi-level residual vector quantizers related to acoustic features in the model into multiple quantizer groups; the multi-band audio acquisition unit is used to acquire audio training data of different frequency bands corresponding to different quantizer groups; and the frequency band training unit is used to adjust the network parameters of the quantizer group according to the audio training data of the frequency bands corresponding to the quantizer group.

[0063] In one example, the quantizer grouping unit is specifically used to divide the multi-level residual vector quantizer into three groups of quantizers; the frequency band training unit is specifically used to adjust the network parameters of the first group of quantizers according to the low-frequency audio training data; adjust the network parameters of the second group of quantizers according to the mid-low frequency audio training data; and adjust the network parameters of the third group of quantizers according to the high-frequency audio training data.

[0064] In one example, the audio codec model includes a dual-stream audio codec model, which includes an acoustic feature quantization module and a semantic feature quantization module; a quantizer grouping unit, specifically used to divide the multi-level residual vector quantizer in the acoustic feature quantization module into multiple quantizer groups.

[0065] In one example, the model includes an acoustic feature de-aggregation module, which is used to de-aggregate the frame-merged acoustic feature tokens output by the semantic feature quantization module based on the frame-merged semantic feature tokens output by the semantic feature quantization module.

[0066] In one example, the model includes a feature aggregation module; the feature aggregation module is used to obtain the feature similarity of adjacent frames; if the feature similarity is greater than or equal to the feature similarity threshold, the features of adjacent frames are merged; and the aggregated features are adjusted according to the features of the original frames before merging.

[0067] In one example, the model includes a feature de-aggregation module; the feature de-aggregation module is used to obtain the number of frames in the merged frame; restore the aggregated features to the original number of frames based on the number of frames; and adjust the features of the restored frame based on the context features.

[0068] In one example, the model includes a dual-stream audio codec model, and the feature aggregation module includes an acoustic feature aggregation module and a semantic feature aggregation module. The acoustic feature aggregation module is used to obtain the acoustic feature similarity of adjacent frames; if the acoustic feature similarity is greater than or equal to the acoustic feature similarity threshold, the acoustic features of adjacent frames are merged; the acoustic aggregation features are adjusted according to the acoustic features of the original frames before merging; and / or, the semantic feature aggregation module is used to merge semantic feature adjacent frames corresponding to the merged acoustic feature adjacent frames; the semantic aggregation features are adjusted according to the semantic features of the original frames before merging.

[0069] In one example, the model includes a feature de-aggregation module, which comprises an acoustic feature de-aggregation module and a semantic feature de-aggregation module. The acoustic feature de-aggregation module is used to obtain the number of frames in the acoustic feature merging frame; restore the acoustic aggregated features to the original number of frames based on the number of frames; adjust the acoustic features of the restored frame based on the context acoustic features; and / or, the semantic feature de-aggregation module is used to obtain the number of frames; restore the semantic aggregated features to the original number of frames based on the number of frames; adjust the semantic features of the restored frame based on the context semantic features.

[0070] In one example, the feature aggregation module is also used to obtain a feature similarity threshold corresponding to the audio type.

[0071] In one example, the aggregated features are adjusted based on the features of the original frames before merging using a local window attention module.

[0072] In one example, the training data includes labeled data with merged frame numbers.

[0073] In one example, the encoder and decoder employ a transformer architecture that introduces a self-attention mechanism.

[0074] Third Embodiment In the above embodiments, an audio codec model construction method is provided. Correspondingly, this application also provides an electronic device. This device corresponds to the embodiments of the above method. Since the device embodiments are basically similar to the method embodiments, the description is relatively simple, and relevant parts can be referred to in the description of the method embodiments. The device embodiments described below are merely illustrative.

[0075] The electronic device of this embodiment includes: a memory and a processor; the memory is used to store a program for implementing the audio codec model construction method, and the device is powered on and runs the program of the audio codec model construction method through the processor.

[0076] Memory can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk or optical disk.

[0077] In specific implementations, the electronic device may further include one or more of the following components: a power supply component, an input / output (I / O) interface, and a communication component. The power supply component provides power to various components of the electronic device. The power supply component may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to the electronic device. The I / O interface provides an interface between the processor 503 and peripheral interface modules, which may be a keyboard, click wheel, buttons, etc. The communication component is configured to facilitate wired or wireless communication between the electronic device and user devices (such as smartphones, tablets, etc.).

[0078] Fourth embodiment This application also provides a computer-readable storage medium. Since the embodiments of the computer-readable storage medium are substantially similar to the method embodiments, the description is relatively simple; relevant details can be found in the description of the method embodiments. The computer-readable storage medium embodiments described below are merely illustrative.

[0079] In this embodiment, a non-transitory computer-readable storage medium including instructions is provided, such as a memory including instructions, which can be executed by a processor of an electronic device to complete the audio codec model construction method provided in this disclosure. For example, the non-transitory computer-readable storage medium may be a ROM, random access memory (RAM), CD-ROM, magnetic tape, floppy disk, and optical data storage device, etc.

[0080] It should be noted that the embodiments of this application may involve the use of user data. In practical applications, user-specific personal data may be used in the scheme described herein within the scope permitted by applicable laws and regulations, provided that it complies with the applicable laws and regulations of the country (e.g., with the user's explicit consent, with the user being properly notified, etc.).

[0081] Although this application discloses preferred embodiments as described above, it is not intended to limit this application. Any person skilled in the art can make possible changes and modifications without departing from the spirit and scope of this application. Therefore, the scope of protection of this application should be determined by the scope defined in the claims of this application.

[0082] In a typical configuration, a computing device includes one or more processors (CPUs), input / output interfaces, network interfaces, and memory.

[0083] Memory may include non-persistent storage in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0084] 1. Computer-readable media includes both permanent and non-permanent, removable and non-removable media that can store information by any method or technology. Information can be computer-readable instructions, data structures, program modules, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include non-transitory computer-readable media, such as modulated data signals and carrier waves.

[0085] 2. Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

Claims

1. A method for constructing an audio codec model, characterized in that, include: Construct a network structure for an audio codec model that employs a dynamic frame rate mechanism; The network parameters of the model are learned from the audio training data; The multi-level residual vector quantizer related to acoustic features in the model is divided into multiple quantizer groups; Acquire audio training data for different frequency bands corresponding to different quantizer groups; The network parameters of the quantizer group are adjusted based on the audio training data of the frequency band corresponding to the quantizer group.

2. The method according to claim 1, characterized in that, The multi-level residual vector quantizer is divided into multiple quantizer groups, including: The multi-level residual vector quantizer is divided into three groups of quantizers; The step of adjusting the network parameters of the quantizer group based on the audio training data of the frequency band corresponding to the quantizer group includes: Based on the low-frequency audio training data, adjust the network parameters of the first set of quantizers; Based on the audio training data in the mid-to-low frequency band, adjust the network parameters of the second set of quantizers; Based on the high-frequency audio training data, adjust the network parameters of the third quantizer.

3. The method according to claim 1, characterized in that, The audio codec model includes a dual-stream audio codec model, which includes an acoustic feature quantization module and a semantic feature quantization module; The step of dividing the multi-level residual vector quantizer related to acoustic features in the model into multiple quantizer groups includes: The multi-level residual vector quantizer in the acoustic feature quantization module is divided into multiple quantizer groups.

4. The method according to claim 3, characterized in that, The model includes an acoustic feature de-aggregation module, which is used to de-aggregate the acoustic feature tokens output by the acoustic feature quantization module based on the semantic feature tokens output by the semantic feature quantization module after frame merging.

5. The method according to claim 1, characterized in that, The model includes a feature aggregation module; The feature aggregation module is used to obtain the feature similarity between adjacent frames; if the feature similarity is greater than or equal to the feature similarity threshold, the features of adjacent frames are merged; and the aggregated features are adjusted according to the features of the original frames before merging.

6. The method according to claim 5, characterized in that, The model includes a feature de-aggregation module; The feature de-aggregation module is used to obtain the number of frames in the merged frame; restore the aggregated features to the original number of frames based on the number of frames; and adjust the features of the restored frame based on the context features.

7. The method according to claim 5, characterized in that, The model includes a dual-stream audio codec model, and the feature aggregation module includes an acoustic feature aggregation module and a semantic feature aggregation module. The acoustic feature aggregation module is used to obtain the acoustic feature similarity between adjacent frames; if the acoustic feature similarity is greater than or equal to the acoustic feature similarity threshold, the acoustic features of adjacent frames are merged; and the acoustic aggregation features are adjusted according to the acoustic features of the original frames before merging. And / or, the semantic feature aggregation module is used to merge semantic feature adjacent frames corresponding to the merged acoustic feature adjacent frames; Based on the semantic features of the original frames before merging, adjust the semantic aggregation features.

8. The method according to claim 7, characterized in that, The model includes a feature de-aggregation module, which includes an acoustic feature de-aggregation module and a semantic feature de-aggregation module. The acoustic feature de-aggregation module is used to obtain the number of frames in the acoustic feature merging frame; restore the acoustic aggregated features to the original number of frames based on the number of frames; and adjust the acoustic features of the restored frame based on the context acoustic features. And / or, the semantic feature de-aggregation module is used to obtain the number of frames; restore the semantic aggregated features to the original number of frames based on the number of frames; and adjust the semantic features of the restored frames based on the context semantic features.

9. The method according to claim 5, characterized in that, The feature aggregation module is also used to obtain the feature similarity threshold corresponding to the audio type.

10. The method according to claim 5, characterized in that, The local window attention module adjusts the aggregated features based on the features of the original frames before merging.

11. The method according to claim 6, characterized in that, The training data includes labeled data of merged frames.

12. The method according to claim 1, characterized in that, The encoder and decoder employ a transformer architecture that incorporates a self-attention mechanism.

13. An audio codec model construction device, characterized in that, include: Network building units are used to construct network structures for audio codec models that employ dynamic frame rate mechanisms. A full network training unit is used to learn the network parameters of the model from audio training data; A quantizer grouping unit is used to divide the multi-level residual vector quantizer related to acoustic features in the model into multiple quantizer groups; A multi-band audio acquisition unit is used to acquire audio training data of different frequency bands corresponding to different quantizer groups; The frequency band training unit is used to adjust the network parameters of the quantizer group based on the audio training data of the frequency band corresponding to the quantizer group.

14. An electronic device, characterized in that, include: processor; as well as A memory for storing a program for implementing the method according to any one of claims 1 to 12, wherein the device is powered on and the program for running the method is executed by the processor.

Images