Audio encoding method, audio decoding method, and audio codec system

Abstract

The application discloses an audio encoding method, an audio decoding method and an audio encoding and decoding system. The method comprises the following steps: obtaining audio to be transmitted and a transmission bit rate; extracting a semantic feature sequence of the audio, and performing vector quantization on the semantic feature sequence based on a semantic codebook to obtain a discrete semantic token sequence encoded based on a first bit rate; extracting a first acoustic feature of the audio, determining a second bit rate which satisfies a transmission bit rate constraint together with the first bit rate, performing downsampling processing on the first acoustic feature based on the second bit rate to obtain a second acoustic feature, and performing vector quantization on the second acoustic feature based on an acoustic codebook to obtain a discrete acoustic token sequence encoded based on the second bit rate; and transmitting the discrete semantic token sequence and the discrete acoustic token sequence to an audio decoding device to reconstruct the audio. The application solves the technical problem that a traditional audio encoding and decoding scheme is difficult to simultaneously consider strong semantic representation and high-fidelity reconstruction in an ultralow bit rate scene.

Description

Technical Field

[0001] This application relates to the field of audio encoding and decoding technology, and more specifically, to an audio encoding method, an audio decoding method, and an audio encoding and decoding system. Background Technology

[0002] Audio coding technology is the core of audio storage, transmission, and downstream processing. Traditional audio encoders, such as MP3 encoders, mainly rely on hand-designed signal processing pipelines and psychoacoustic models. They are highly effective in waveform compression and transmission, but because their primary optimization goal is audio reconstruction quality, they typically require medium to high bit rates of hundreds of thousands of bits per second in actual operation. With the development of large-scale storage, low-bandwidth transmission, and emerging applications such as audio generation and understanding, the industry's demand for more compact audio representation is increasingly urgent. In such ultra-low bit rate application scenarios, traditional encoders, lacking mechanisms for explicitly modeling high-level semantic information, are not only inefficient but also suffer from severe perceptual degradation, making it difficult to meet practical needs.

[0003] Advances in neural network technology have propelled audio coding into a new paradigm. Neural audio encoders, through end-to-end learning, automatically acquire optimized, compact representations, achieving lower bit rates and greater flexibility, thus becoming a crucial direction for ultra-low bit rate audio coding. However, existing neural audio encoders are mainly divided into acoustic encoders and semantic encoders. The former requires a high token rate to preserve details, while the latter suffers from poor reconstruction quality, insufficient generalization, and misalignment of the text's semantic space. In ultra-low bit rate scenarios, balancing strong semantic representation with high-fidelity reconstruction remains a pressing technical challenge. Summary of the Invention

[0004] This application provides an audio encoding method, an audio decoding method, and an audio encoding / decoding system to at least solve the technical problem that traditional audio encoding / decoding schemes struggle to simultaneously achieve strong semantic representation and high-fidelity reconstruction in ultra-low bit rate scenarios.

[0005] According to one aspect of the embodiments of this application, an audio encoding method is provided, comprising: acquiring a first audio to be transmitted and a target transmission bit rate; extracting a first semantic feature sequence of the first audio, and performing vector quantization on the first semantic feature sequence based on a target semantic codebook to obtain a first discrete semantic token sequence, wherein the first discrete semantic token sequence is encoded according to a preset first bit rate; extracting a first acoustic feature of the first audio, determining a second bit rate based on the target transmission bit rate and the first bit rate, performing downsampling processing on the first acoustic feature based on the second bit rate to obtain a second acoustic feature, and performing vector quantization on the second acoustic feature based on the target acoustic codebook to obtain a first discrete acoustic token sequence, wherein the sum of the first bit rate and the second bit rate satisfies the constraint of the target transmission bit rate, and the first discrete acoustic token sequence is encoded according to the second bit rate; and transmitting the first discrete semantic token sequence and the first discrete acoustic token sequence to an audio decoding device to reconstruct the first audio.

[0006] Optionally, extracting a first semantic feature sequence from the first audio and performing vector quantization on the first semantic feature sequence based on a target semantic codebook to obtain a first discrete semantic token sequence includes: extracting features from the first audio using a feature extraction module in a pre-trained semantic coding model to obtain a first semantic feature sequence; performing mapping transformation on a preset basic semantic codebook using a codebook transformation network in a vector quantization module of the semantic coding model to obtain a target semantic codebook; and performing nearest neighbor matching on the first semantic feature sequence according to the target semantic codebook to obtain a first discrete semantic token sequence.

[0007] Optionally, the training process of the semantic coding model includes: constructing an initial coding model, wherein the initial coding model includes a feature extraction module and a vector quantization module; acquiring multiple second audio samples as training samples; for each second audio sample, inputting the second audio sample into the initial coding model to obtain a second semantic feature sequence output by the feature extraction module and a second discrete semantic token sequence output by the vector quantization module; constructing a first loss function for constraining semantic consistency and a second loss function for constraining temporal structure consistency based on the second discrete semantic token sequence; constructing a third loss function based on the quantization error between the second semantic feature sequence and the second discrete semantic token sequence; and adjusting the model parameters of the initial coding model based on the first loss function, the second loss function, and the third loss function to obtain the semantic coding model.

[0008] Optionally, a first loss function for constraining semantic consistency is constructed based on the second discrete semantic token sequence, including: analyzing the second discrete semantic token sequence using a pre-trained text generation model to obtain generated text content; obtaining the text content annotation corresponding to the second audio; and constructing the first loss function based on the difference between the generated text content and the text content annotation.

[0009] Optionally, a second loss function for constraining temporal structure consistency is constructed based on the second discrete semantic token sequence, including: extracting frame-level features of the second audio using a pre-trained audio self-supervised model; mapping the second discrete semantic token sequence to the same representation space as the frame-level features to obtain mapped features; determining the cosine distance between the frame-level features and the mapped features, and constructing the second loss function based on the cosine distance.

[0010] Optionally, the process involves extracting a first acoustic feature from the first audio, downsampling the first acoustic feature based on a second bit rate to obtain a second acoustic feature, and vector quantizing the second acoustic feature based on a target acoustic codebook to obtain a first discrete acoustic token sequence. This includes: extracting features from the first audio using a feature extraction module in a pre-trained acoustic coding model to obtain the first acoustic feature, wherein the dimensions of the first acoustic feature include at least one of the following: timbre, energy variation, prosodic structure, and transient information; determining a target pooling ratio corresponding to the second bit rate using an adaptive pooling module in the acoustic coding model based on a preset mapping relationship table, and pooling the first acoustic feature based on the target pooling ratio to obtain the second acoustic feature, wherein the mapping relationship table stores mapping relationships between multiple bit rates and multiple pooling ratios; performing a mapping transformation on a preset basic acoustic codebook using a codebook transformation network in the vector quantization module of the acoustic coding model to obtain a target acoustic codebook, and performing nearest neighbor matching on the second acoustic feature according to the target acoustic codebook to obtain the first discrete acoustic token sequence.

[0011] Optionally, the training process of the acoustic coding model includes: constructing an initial coding model, wherein the initial coding model includes a feature extraction module, an adaptive pooling module, and a vector quantization module; acquiring multiple second audio samples as training samples; for each second audio sample, inputting the second audio sample into the initial coding model to obtain a third acoustic feature output by the feature extraction module, a fourth acoustic feature output by the adaptive pooling module, and a second discrete acoustic token sequence output by the vector quantization module; constructing a fourth loss function based on the second discrete acoustic token sequence to constrain the consistency of the acoustic features; constructing a fifth loss function based on the quantization error between the fourth acoustic feature and the second discrete acoustic token sequence; and adjusting the model parameters of the initial coding model based on the fourth and fifth loss functions to obtain the acoustic coding model.

[0012] Optionally, transmitting the first discrete semantic token sequence and the first discrete acoustic token sequence to an audio decoding device to reconstruct the first audio includes: performing time alignment on the first discrete semantic token sequence and the first discrete acoustic token sequence; concatenating the corresponding discrete semantic tokens and discrete acoustic tokens in the first discrete semantic token sequence and the first discrete acoustic token sequence step by step to obtain a joint encoded representation; and transmitting the joint encoded representation to the audio decoding device to reconstruct the first audio.

[0013] According to another aspect of the embodiments of this application, an audio decoding method is also provided, comprising: receiving a first discrete semantic token sequence reflecting semantic features of a first audio and a first discrete acoustic token sequence reflecting acoustic features of a first audio transmitted by an audio encoding device, wherein the first discrete semantic token sequence is encoded according to a first bit rate, the first discrete acoustic token sequence is encoded according to a second bit rate, and the sum of the first bit rate and the second bit rate satisfies the constraint of a target transmission bit rate; performing time alignment of the first discrete semantic token sequence and the first discrete acoustic token sequence based on the time steps of the latent space of a pre-trained audio reconstruction model; and using the audio reconstruction model to perform audio reconstruction in the latent space based on the time-aligned first discrete semantic token sequence and the first discrete acoustic token sequence to obtain the restored first audio, wherein the audio reconstruction model is a stream matching model based on a diffusion Transformer architecture.

[0014] Optionally, an audio reconstruction model is used to reconstruct audio in the latent space based on a time-aligned first discrete semantic token sequence and a first discrete acoustic token sequence to obtain the reconstructed first audio. This includes: obtaining a preset initial latent representation; inputting the initial latent representation and the time-aligned first discrete semantic token sequence and the first discrete acoustic token sequence into the audio reconstruction model to obtain the first audio output by the audio reconstruction model. The audio reconstruction model starts with the initial latent representation in the latent space, generates a target latent representation step by step with the discrete semantic tokens and discrete acoustic tokens of the corresponding time step as condition variables, maps the target latent representation to a time-domain audio signal, and composes the time-domain audio signals of all time steps into the first audio.

[0015] According to another aspect of the embodiments of this application, an audio encoding and decoding system is also provided, including: an audio encoding device and an audio decoding device, wherein the audio encoding device is configured to: acquire a first audio to be transmitted and a target transmission bit rate; extract a first semantic feature sequence of the first audio, and perform vector quantization on the first semantic feature sequence based on a target semantic codebook to obtain a first discrete semantic token sequence, wherein the first discrete semantic token sequence is encoded according to a preset first bit rate; extract a first acoustic feature of the first audio, determine a second bit rate based on the target transmission bit rate and the first bit rate, perform downsampling processing on the first acoustic feature based on the second bit rate to obtain a second acoustic feature, and perform vector quantization on the second acoustic feature based on the target acoustic codebook. A first discrete acoustic token sequence is obtained, wherein the sum of the first bit rate and the second bit rate satisfies the constraint of the target transmission bit rate, and the first discrete acoustic token sequence is encoded according to the second bit rate; the first discrete semantic token sequence and the first discrete acoustic token sequence are transmitted to an audio decoding device; the audio decoding device is used to time-align the first discrete semantic token sequence and the first discrete acoustic token sequence based on the time step of the latent space of the pre-trained audio reconstruction model; the audio reconstruction model is used to perform audio reconstruction in the latent space based on the time-aligned first discrete semantic token sequence and the first discrete acoustic token sequence to obtain the restored first audio, wherein the audio reconstruction model is a stream matching model based on the diffusion Transformer architecture.

[0016] According to another aspect of the embodiments of this application, a computer program product is also provided, the computer program product comprising: a computer program, wherein the computer program, when executed by a processor, implements the above-described audio encoding method.

[0017] According to another aspect of the embodiments of this application, an electronic device is also provided, the electronic device including: a memory and a processor, wherein the memory stores a computer program, and the processor is configured to execute the above-described audio encoding method through the computer program.

[0018] In this embodiment, audio encoding follows a semantic-first approach. First, a semantic coding model is used to extract features and perform vector quantization on the audio, resulting in a semantic discrete token with an explicit semantic basis. Then, acoustic features are extracted based on an acoustic coding model, and adaptive time compression is applied to these features based on transmission bit rate constraints. The compressed features are then vector-quantized to obtain an acoustic discrete token that reflects fine-grained acoustic details. Finally, both are sent to the decoding end for reconstruction. This entire scheme, through a semantic-acoustic complementary architecture, enables high-quality audio reconstruction at ultra-low bit rates, effectively solving the technical problem of traditional audio encoding and decoding schemes struggling to simultaneously achieve strong semantic representation and high-fidelity reconstruction in ultra-low bit rate scenarios. Attached Figure Description

[0019] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments of this application and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings:

[0020] Figure 1 This is a schematic diagram of the structure of an optional audio codec system according to an embodiment of this application;

[0021] Figure 2 This is a schematic diagram illustrating the training of an optional semantic coding model according to an embodiment of this application;

[0022] Figure 3 This is a schematic diagram illustrating the training of an optional acoustic coding model according to an embodiment of this application;

[0023] Figure 4 This is a schematic diagram of the structure of an optional audio reconstruction model according to an embodiment of this application;

[0024] Figure 5 This is a schematic diagram of an optional audio reconstruction process according to an embodiment of this application;

[0025] Figure 6 This is a flowchart illustrating an optional audio encoding method according to an embodiment of this application;

[0026] Figure 7 This is a flowchart illustrating an optional audio decoding method according to an embodiment of this application;

[0027] Figure 8 This is a schematic diagram of the structure of an optional electronic device according to an embodiment of this application. Detailed Implementation

[0028] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort should fall within the scope of protection of the present application.

[0029] It should be noted that the terms "first," "second," etc., used in the specification, claims, and drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0030] To better understand the embodiments of this application, the following is a translation and explanation of some nouns or terms that appear in the description of the embodiments of this application:

[0031] Token Rate: In audio coding, the number of discrete tokens transmitted or processed per unit time, measured in Hz, is an important indicator of the compactness of audio representation. The higher the token rate, the more fine-grained acoustic details need to be preserved, and the higher the corresponding bit rate.

[0032] Bit rate (bps): also known as bit rate, is the number of bits of audio data transmitted or stored per unit of time. It directly determines the degree of audio compression and transmission efficiency.

[0033] Vector Quantization (VQ) is a technique used in audio coding for data compression. It maps continuous audio signal vectors to discrete codebook vectors, thereby reducing data redundancy and achieving compression.

[0034] Codebook: A set of discrete vectors used for data compression in vector quantization, and a core component of the vector quantization process. The codebook pre-stores a series of standard discrete vectors. During encoding, continuous signal vectors are mapped to the nearest discrete vector in the codebook, and data compression is achieved through this mapping.

[0035] Flow matching: a technique in the field of audio coding and generation that learns a continuous-time velocity field to transmit a simple distribution to a complex audio distribution in a latent space. It is often used for efficient and stable audio generation and reconstruction.

[0036] High-fidelity reconstruction: One of the core goals of audio coding, it refers to the audio signal restored after encoding and compression being highly consistent with the original audio signal in terms of auditory perception, preserving as much fine-grained acoustic detail, timbre and sound quality as possible, and reducing distortion.

[0037] Example 1

[0038] To address the technical challenge of traditional audio codec schemes simultaneously achieving strong semantic representation and high-fidelity reconstruction in ultra-low bitrate scenarios, embodiments of this application provide an audio codec system based on a novel architecture, such as... Figure 1 As shown, the audio encoding and decoding system includes: an audio encoding device 11 and an audio decoding device 12, wherein:

[0039] Audio encoding device 11 is used to acquire a first audio signal to be transmitted and a target transmission bit rate; extract a first semantic feature sequence of the first audio signal, and perform vector quantization on the first semantic feature sequence based on the target semantic codebook to obtain a first discrete semantic token sequence, wherein the first discrete semantic token sequence is encoded according to a preset first bit rate; extract a first acoustic feature of the first audio signal, determine a second bit rate based on the target transmission bit rate and the first bit rate, perform downsampling processing on the first acoustic feature based on the second bit rate to obtain a second acoustic feature, and perform vector quantization on the second acoustic feature based on the target acoustic codebook to obtain a first discrete acoustic token sequence, wherein the sum of the first bit rate and the second bit rate satisfies the constraint of the target transmission bit rate, and the first discrete acoustic token sequence is encoded according to the second bit rate; and transmit the first discrete semantic token sequence and the first discrete acoustic token sequence to audio decoding device 12.

[0040] The audio decoding device 12 is used to time-align the first discrete semantic token sequence and the first discrete acoustic token sequence based on the time step of the latent space of the pre-trained audio reconstruction model; and to use the audio reconstruction model to perform audio reconstruction in the latent space based on the time-aligned first discrete semantic token sequence and the first discrete acoustic token sequence to obtain the restored first audio, wherein the audio reconstruction model is a stream matching model based on the diffusion Transformer architecture.

[0041] The structure and function of the audio encoding device and the audio decoding device are explained below with reference to the specific implementation process.

[0042] The first audio signal to be transmitted can be a voice signal in a real-time voice communication scenario, or music data to be encoded and stored in a music production scenario. The target transmission bit rate is usually determined by the network bandwidth, storage constraints, and other factors in the corresponding application scenario.

[0043] As an optional implementation, the audio encoding device can generate the first discrete semantic token sequence based on the first audio through the following steps S11-S12:

[0044] S11, use the feature extraction module in the pre-trained semantic coding model to extract features from the first audio to obtain the first semantic feature sequence.

[0045] Optionally, the first audio signal can be preprocessed by resampling, normalization, and framing to meet the input requirements of the feature extraction module. Then, the feature extraction module can be used to extract features from the preprocessed audio signal to obtain a continuous semantic feature sequence.

[0046] S12, the codebook transformation network in the vector quantization module of the semantic coding model is used to perform mapping transformation on the preset basic semantic codebook to obtain the target semantic codebook. The first semantic feature sequence is matched with the nearest neighbor according to the target semantic codebook to obtain the first discrete semantic token sequence.

[0047] The basic semantic codebook is usually obtained by clustering intermediate layers during the training of the semantic coding model. The codebook transformation network can be a lightweight neural network (such as a 1-2 layer MLP (Multi-Layer Perceptron), a linear transformation network, a small attention network, etc.). Its input is the basic codebook and its output is the effective codebook. Through codebook mapping, the quantization process can maintain stability while having learnable capabilities.

[0048] As an optional implementation, to accurately generate the discrete semantic token sequence described above, a semantic encoding model can be trained through the following steps S21-S26:

[0049] S21, Construct the initial encoding model, which includes a feature extraction module and a vector quantization module.

[0050] Optionally, the encoder part of an existing speech understanding model can be selected and initialized as a feature extraction module; the vector quantization module includes a learnable codebook transformation network and a matching module for performing nearest neighbor matching.

[0051] S22, acquire multiple second audio samples as training samples.

[0052] To improve the model's versatility, the second audio file can be in various formats and with different content types.

[0053] S23, for each second audio, input the second audio into the initial encoding model to obtain the second semantic feature sequence output by the feature extraction module and the second discrete semantic token sequence output by the vector quantization module.

[0054] S24, construct a first loss function for constraining semantic consistency and a second loss function for constraining temporal structure consistency based on the second discrete semantic token sequence.

[0055] To ensure that discrete semantic tokens retain linguistically discriminative semantic information even at extremely low bit rates, this application introduces a text generation model as a semantic consistency supervisor to explicitly align the semantic tokens to the text semantic space. Optionally, the first loss function can be constructed through the following steps S2411-S2413:

[0056] S2411, using a pre-trained text generation model to analyze the second discrete semantic token sequence to obtain the generated text content.

[0057] This text generation model can use a generative large language model. To reduce training overhead and maintain model stability, an efficient parameter fine-tuning strategy can be adopted when training the text generation model. Only a small number of adaptation parameters of the text generation model are updated, while the main parameters of the text generation model are kept frozen.

[0058] When inputting the second semantic feature sequence, the second discrete semantic token sequence can be projected onto the word vector embedding space of the text generation model through a learnable adaptation layer to form a continuous learnable embedding representation. Then, it is input into the text generation model along with the corresponding task prompt word to obtain the generated text content.

[0059] S2412, obtain the text content annotation corresponding to the second audio.

[0060] It can be understood that the semantic consistency supervision in this application is supervised learning, which requires obtaining text content annotations as corresponding sample labels.

[0061] S2413, construct the first loss function based on the difference between the generated text content and the text content annotation.

[0062] The first loss function can be the cross-entropy loss function, the autoregressive generation loss function, etc.

[0063] Optionally, considering the potential semantic token collapse or temporal structure loss issues that may arise from using only semantic consistency supervision, this application embodiment also introduces a temporal consistency constraint based on an audio self-supervised representation model to maintain the temporal continuity and structural expressiveness of semantic tokens. Optionally, a second loss function can be constructed through the following steps S2421-S2423:

[0064] S2421 utilizes a pre-trained audio self-supervised model to extract frame-level features of the second audio.

[0065] Among them, the audio self-supervised model can be a feature extraction network based on temporal extraction of audio features. Since the time step in audio coding is usually reflected in the number of frames, the frame-level features of the audio can be extracted as a reference for the temporal structure.

[0066] S2422, the second discrete semantic token sequence is mapped to the same representation space as the frame-level features to obtain the mapped features.

[0067] Alternatively, a lightweight projection network can be constructed to map the second discrete semantic token sequence to a representation space that is the same as or comparable to the frame-level features.

[0068] S2423, determine the cosine distance between frame-level features and mapped features, and construct a second loss function based on the cosine distance.

[0069] Among them, the cosine distance can be used to measure whether the temporal variation trend of semantic tokens is consistent with the frame-level features.

[0070] S25, construct a third loss function based on the quantization error of the second semantic feature sequence and the second discrete semantic token sequence.

[0071] Optionally, the third loss function can be the commitment loss function, which forces the continuous latent vector of the audio signal to maintain a close relationship with the discrete vector after quantization, thereby reducing information loss and bias during the quantization process and avoiding redundant or invalid mapping of the quantized vector.

[0072] S26. Based on the first loss function, the second loss function, and the third loss function, adjust the model parameters of the initial encoding model to obtain the semantic encoding model.

[0073] The third loss function is primarily used to adjust the model parameters of the vector quantization module, while the first and second loss functions are used to adjust the overall model parameters of the feature extraction and vector quantization modules. Specifically, backpropagation can be used for model parameter adjustment.

[0074] Figure 2 A schematic diagram of training an optional semantic coding model is shown. Its main process corresponds to steps S21-S26 above, and will not be elaborated further here.

[0075] The training process described above, by introducing language consistency supervision, can explicitly align discrete semantic tokens to the text semantic space, preserving high-level semantic information related to the language task at extremely low bitrates, reducing the risk of semantic drift, and thus improving the usability and consistency of semantic representations in tasks such as speech transcription and audio semantic description. By introducing temporal consistency constraints, discrete semantic tokens can have stronger temporal continuity and structural expressive power, especially in non-speech or weakly labeled audio scenarios, which can suppress semantic token sequence collapse, improve the robustness of the training process and the interpretability of the output representation. By freezing the basic semantic codebook and introducing a learnable codebook transformation network to construct an effective codebook, combined with quantization error constraints, it helps to reduce discretization errors and stabilize the codebook usage distribution, thereby improving the stability and reusability of discrete semantic tokens.

[0076] Ultimately, the first discrete semantic token sequence generated by the semantic coding model can serve as a stable conditional input for subsequent acoustic detail modeling and generative decoding reconstruction, providing a controllable semantic skeleton for high-fidelity audio reconstruction under extremely low bit rate conditions, and at the same time providing a unified semantic intermediate representation for multi-task audio understanding and generation applications.

[0077] As an optional implementation, the audio encoding device can generate the first discrete acoustic token sequence based on the first audio by the following steps S31-S34:

[0078] S31, the feature extraction module in the pre-trained acoustic coding model is used to extract features from the first audio to obtain the first acoustic features.

[0079] The feature extraction module can employ a multi-layer temporal modeling structure to jointly model the short-term and long-term dependencies of audio, thereby enhancing the ability to represent the acoustic details of different audio types.

[0080] When performing feature extraction, the audio can be resampled first to convert it into high-sampling-rate audio data as input to the feature extraction module. The dimensions of the first acoustic feature extracted can include at least one of the following: timbre, energy variation, prosodic structure, and transient information.

[0081] S32, determine the second bit rate based on the target transmission bit rate and the first bit rate.

[0082] The target transmission bit rate is limited by hardware and cannot be adjusted. In low bit rate scenarios, it is usually 275bps, 375bps, 575bps, or 975bps. The encoding token rate of the discrete semantic token sequence is also usually fixed, that is, the corresponding first bit rate is also fixed, such as 175bps. In order to preserve the fine-grained acoustic information of the audio as much as possible, the difference between the target transmission bit rate and the first bit rate can be directly used as the second bit rate, such as 100bps, 200bps, 400bps, or 800bps. Of course, it is also possible to set it to a value smaller than this difference.

[0083] S33, using the adaptive pooling module in the acoustic coding model to determine the target pooling ratio corresponding to the second bit rate based on a preset mapping table, and pooling the first acoustic feature based on the target pooling ratio to obtain the second acoustic feature.

[0084] The mapping table stores the mapping relationships between multiple bit rates and multiple pooling ratios. For example, 100bps corresponds to a pooling ratio of 1 / 8, 200bps corresponds to a pooling ratio of 1 / 4, and so on. It should be noted that this is only an illustrative example and does not constitute a specific limitation.

[0085] By setting an adaptive pooling module with a variable pooling ratio, acoustic features can be downsampled in an adaptive manner in the time dimension. Different compression parameters can be dynamically selected according to a preset strategy, so that acoustic features can be represented at different time resolutions, and flexible control of acoustic bit rate can be achieved.

[0086] S34, the codebook transformation network in the vector quantization module of the acoustic coding model is used to perform mapping transformation on the preset basic acoustic codebook to obtain the target acoustic codebook. The second acoustic feature is matched with the nearest neighbor according to the target acoustic codebook to obtain the first discrete acoustic token sequence.

[0087] The basic acoustic codebook is typically obtained by clustering intermediate layers during the training of the acoustic coding model. The codebook transformation network can be a lightweight neural network, with the basic codebook as input and the effective codebook as output. Through codebook mapping, the quantization process can maintain stability while possessing learning capabilities.

[0088] As an optional implementation, to accurately generate the discrete acoustic token sequence described above, the acoustic coding model can be trained through the following steps S41-S46:

[0089] S41, Construct the initial encoding model, which includes: a feature extraction module, an adaptive pooling module, and a vector quantization module.

[0090] The feature extraction module can adopt a multi-layer temporal modeling structure, the adaptive pooling module includes a preset mapping table between bit rate and pooling ratio and pooling layers corresponding to different pooling ratios, and the vector quantization module includes a learnable codebook transformation network and a matching module for performing nearest neighbor matching.

[0091] S42, acquire multiple second audio samples as training samples.

[0092] To improve the model's versatility, the second audio file can be in various formats and with different content types.

[0093] S43, for each second audio, input the second audio into the initial encoding model to obtain the third acoustic feature output by the feature extraction module, the fourth acoustic feature output by the adaptive pooling module, and the second discrete acoustic token sequence output by the vector quantization module.

[0094] S44, a fourth loss function is constructed based on the second discrete acoustic token sequence to constrain the consistency of acoustic features.

[0095] Optionally, to ensure the accuracy of the extracted acoustic features, a supervised learning training method can be adopted. First, a second discrete semantic token sequence corresponding to the second discrete acoustic token sequence is obtained. Then, an audio decoder is used to decode both the second discrete acoustic token sequence and the second discrete semantic token sequence to generate reconstructed audio. A fourth loss function is constructed based on the difference between the reconstructed audio and the second audio, achieving acoustic feature consistency supervision. The fourth loss function can be the mean squared error loss function.

[0096] S45, a fifth loss function is constructed based on the quantization error of the fourth acoustic feature and the second discrete acoustic token sequence.

[0097] Optionally, the fifth loss function can be the commitment loss function, which forces the audio acoustic features to maintain a close correlation with the quantized discrete vector, reducing information loss and bias during the quantization process, while avoiding redundancy or invalid mapping of the quantized vector.

[0098] S46. Based on the fourth and fifth loss functions, the model parameters of the initial coding model are adjusted to obtain the acoustic coding model.

[0099] The fifth loss function is primarily used to adjust the model parameters of the vector quantization module, while the fourth loss function is used to adjust the model parameters of the feature extraction module, adaptive pooling module, and vector quantization module as a whole. Specifically, backpropagation can be used to adjust the model parameters.

[0100] Figure 3 A schematic diagram of the training of an optional acoustic coding model is shown. Its main process corresponds to steps S41-S46 above, and will not be described in detail here.

[0101] The training process described above extracts acoustic features from the audio at a high sampling rate, preserving key acoustic information such as timbre and rhythm even at a low coding bit rate, thus providing a high-quality acoustic foundation for subsequent audio reconstruction. By performing structured discrete coding of acoustic features, the uncertainty of acoustic information during transmission and decoding can be reduced, lowering the risk of distortion or loss of detail during audio reconstruction and improving the temporal and auditory stability of the reconstructed audio. By introducing an adaptive compression mechanism with a variable pooling ratio in the acoustic feature compression stage, the generation rate of discrete acoustic tokens can be dynamically adjusted according to different application scenarios and network bandwidth, effectively reducing the coding bit rate while ensuring audio quality, and achieving flexible adaptive control of the acoustic bit rate.

[0102] It should be noted that the semantic coding model and the acoustic model in this application can be trained simultaneously.

[0103] As an optional implementation, to improve the efficiency of subsequent audio reconstruction, when transmitting the first discrete semantic token sequence and the first discrete acoustic token sequence, the audio encoding device can first perform time alignment on the first discrete semantic token sequence and the first discrete acoustic token sequence, and then concatenate the corresponding discrete semantic tokens and discrete acoustic tokens in the first discrete semantic token sequence and the first discrete acoustic token sequence step by step to obtain a joint encoded representation, and then transmit the joint encoded representation to the audio decoding device to reconstruct the first audio.

[0104] Of course, the time alignment operation can also be performed by the audio decoding device, and the audio encoding device can directly transmit the first discrete semantic token sequence and the first discrete acoustic token sequence to the audio decoding device.

[0105] As an optional implementation, the audio decoding device can perform audio reconstruction using the audio reconstruction model through the following steps S51-S52:

[0106] S51, based on the time step of the latent space of the audio reconstruction model, the first discrete semantic token sequence and the first discrete acoustic token sequence are time-aligned.

[0107] S52, using an audio reconstruction model, audio reconstruction is performed in the latent space based on the time-aligned first discrete semantic token sequence and the first discrete acoustic token sequence to obtain the restored first audio.

[0108] Among them, the audio reconstruction model adopts a decoding structure based on the generative model, such as the stream matching model based on the diffusion Transformer architecture, which can model and synthesize audio signals in the latent space. Figure 4 The structure of an optional audio reconstruction model is shown, which includes:

[0109] 1) Input layer, which receives two core inputs: the main inputs are discrete semantic tokens and discrete acoustic tokens, which provide core data for the model; the conditional input is time step t, which is converted into an embedding of the same dimension as the features by the multilayer perceptron on the right, and is used to control the generation process.

[0110] 2) Attention sub-layer, including: Root Mean Square Normalization (RMSNorm) layer, Scale and Shift layer, Rotary Position Embedding (RoPE) layer, Multi-Head Self-Attention (MHSA) layer, and Scale layer. The attention output is added to the original input residual to prevent gradient vanishing. The features of the temporal embedding t map are used to inject temporal conditions into the attention layer.

[0111] 3) Feedforward sub-layer, including: Root Mean Square Normalization (RMSNorm) layer, scaling and shifting layer (Scale, Shift), LLaMA multilayer perceptron (LLaMA MLP), and scaling layer (Scale). Among them, the LLaMA multilayer perceptron is used for feedforward calculation to perform nonlinear feature transformation. Its output is added to the residual of the attention layer output. The features of the temporal embedding t map are used to inject temporal conditions into the feedforward sub-layer.

[0112] The combination of the attention sub-layer and the feedforward sub-layer constitutes a Transformer block. The ×N marked in the figure indicates that the Transformer block is repeatedly stacked with N layers to extract features at depth.

[0113] 4) Output layer, used to output the reconstructed audio.

[0114] This application embodiment maintains consistency between discrete semantic tokens, discrete acoustic tokens, and the latent representation in the audio reconstruction model in the time dimension. This allows for the joint introduction of semantic and acoustic conditions at the frame level, enabling the audio reconstruction model to obtain corresponding semantic and acoustic constraints at each time step. This ensures the continuity and consistency of the reconstructed audio in terms of overall structure and local changes, avoiding problems such as semantic misalignment or missing acoustic details.

[0115] Alternatively, audio reconstruction can be achieved through the following steps S521-S522:

[0116] S521, Obtain the preset initial potential representation.

[0117] The initial latent representation can be random noise collected from a pre-defined prior distribution.

[0118] S522, input the initial latent representation and the time-aligned first discrete semantic token sequence and first discrete acoustic token sequence into the audio reconstruction model to obtain the first audio output by the audio reconstruction model.

[0119] In this process, the audio reconstruction model starts with an initial latent representation in the latent space, generates a target latent representation step by step using the discrete semantic tokens and discrete acoustic tokens of the corresponding time step as condition variables, maps the target latent representation to a time-domain audio signal, and finally combines the time-domain audio signals of all time steps into the first audio.

[0120] Compared to traditional deterministic decoding methods, this generative reconstruction scheme can effectively compensate for the unavoidable information loss in low bit-rate discrete representations, and improve the naturalness, realism and overall listening quality of the reconstructed audio.

[0121] Figure 5 A complete audio encoding and decoding flowchart is shown. In the audio encoding device, the semantic encoding model processes the audio to obtain a discrete semantic token sequence, and the acoustic encoding model processes the audio to obtain a discrete acoustic token sequence. In the audio decoding device, the audio reconstruction model generates reconstructed audio by starting with random noise in the latent space and taking the discrete semantic token sequence and the discrete acoustic token sequence as conditional inputs.

[0122] In this embodiment, audio encoding follows a semantic-first approach. First, a semantic coding model is used to extract features and perform vector quantization on the audio, resulting in a semantic discrete token with an explicit semantic basis. Then, acoustic features are extracted based on an acoustic coding model, and adaptive time compression is applied to these features based on transmission bit rate constraints. The compressed features are then vector-quantized to obtain an acoustic discrete token that reflects fine-grained acoustic details. Finally, both are sent to the decoding end for reconstruction. This entire scheme, through a semantic-acoustic complementary architecture, enables high-quality audio reconstruction at ultra-low bit rates, effectively solving the technical problem of traditional audio encoding and decoding schemes struggling to simultaneously achieve strong semantic representation and high-fidelity reconstruction in ultra-low bit rate scenarios.

[0123] Example 2

[0124] Based on the audio encoding and decoding system of Embodiment 1, this application provides an audio encoding method implemented by an audio encoding device. It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. Although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in a different order than that shown here.

[0125] Figure 6 This is a flowchart illustrating an audio encoding method according to an embodiment of this application, such as... Figure 6 As shown, the method includes the following steps S602-S608:

[0126] Step S602: Obtain the first audio to be transmitted and the target transmission bit rate;

[0127] Step S604: Extract the first semantic feature sequence of the first audio, and perform vector quantization on the first semantic feature sequence based on the target semantic codebook to obtain the first discrete semantic token sequence, wherein the first discrete semantic token sequence is encoded according to a preset first bit rate;

[0128] Step S606: Extract the first acoustic feature of the first audio, determine the second bit rate based on the target transmission bit rate and the first bit rate, downsample the first acoustic feature based on the second bit rate to obtain the second acoustic feature, and perform vector quantization on the second acoustic feature based on the target acoustic codebook to obtain the first discrete acoustic token sequence. The sum of the first bit rate and the second bit rate satisfies the constraint of the target transmission bit rate, and the first discrete acoustic token sequence is encoded according to the second bit rate.

[0129] Step S608: The first discrete semantic token sequence and the first discrete acoustic token sequence are transmitted to the audio decoding device to reconstruct the first audio.

[0130] The following section explains each step of the audio encoding method in conjunction with a specific implementation process.

[0131] Optionally, when extracting the first semantic feature sequence of the first audio and performing vector quantization on the first semantic feature sequence based on the target semantic codebook to obtain the first discrete semantic token sequence, it can be achieved as follows: using the feature extraction module in the pre-trained semantic coding model to extract features from the first audio to obtain the first semantic feature sequence; using the codebook transformation network in the vector quantization module of the semantic coding model to perform mapping transformation processing on the preset basic semantic codebook to obtain the target semantic codebook; and performing nearest neighbor matching on the first semantic feature sequence according to the target semantic codebook to obtain the first discrete semantic token sequence.

[0132] Optionally, the semantic coding model can be trained as follows: Construct an initial coding model, which includes a feature extraction module and a vector quantization module; acquire multiple second audio samples as training samples; for each second audio sample, input the second audio sample into the initial coding model to obtain a second semantic feature sequence output by the feature extraction module and a second discrete semantic token sequence output by the vector quantization module; construct a first loss function based on the second semantic feature sequence to constrain semantic consistency; construct a first loss function based on the second discrete semantic token sequence to constrain semantic consistency and a second loss function based on constraining temporal structure consistency; adjust the model parameters of the initial coding model based on the first loss function, the second loss function, and the third loss function to obtain the semantic coding model.

[0133] Optionally, when constructing the first loss function for constraining semantic consistency based on the second discrete semantic token sequence, it can be implemented as follows: analyze the second discrete semantic token sequence using a pre-trained text generation model to obtain the generated text content; obtain the text content annotation corresponding to the second audio; and construct the first loss function based on the difference between the generated text content and the text content annotation.

[0134] Optionally, when constructing a second loss function for constraining temporal structure consistency based on the second discrete semantic token sequence, it can be implemented as follows: extract frame-level features of the second audio using a pre-trained audio self-supervised model; map the second discrete semantic token sequence to the same representation space as the frame-level features to obtain the mapped features; determine the cosine distance between the frame-level features and the mapped features, and construct the second loss function based on the cosine distance.

[0135] Optionally, when extracting the first acoustic features of the first audio, downsampling the first acoustic features based on the second bit rate to obtain the second acoustic features, and performing vector quantization on the second acoustic features based on the target acoustic codebook to obtain the first discrete acoustic token sequence, this can be achieved as follows: The feature extraction module in the pre-trained acoustic coding model extracts features from the first audio to obtain the first acoustic features, wherein the dimensions of the first acoustic features include at least one of the following: timbre, energy variation, prosodic structure, and transient information; the adaptive pooling module in the acoustic coding model determines the target pooling ratio corresponding to the second bit rate based on a preset mapping table, and performs pooling processing on the first acoustic features based on the target pooling ratio to obtain the second acoustic features, wherein the mapping table stores mapping relationships between multiple bit rates and multiple pooling ratios; the codebook transformation network in the vector quantization module of the acoustic coding model performs mapping transformation processing on the preset basic acoustic codebook to obtain the target acoustic codebook, and performs nearest neighbor matching on the second acoustic features according to the target acoustic codebook to obtain the first discrete acoustic token sequence.

[0136] Optionally, the acoustic coding model can be trained as follows: Construct an initial coding model, which includes a feature extraction module, an adaptive pooling module, and a vector quantization module; acquire multiple second audio samples as training samples; for each second audio sample, input the second audio sample into the initial coding model to obtain a third acoustic feature output by the feature extraction module, a fourth acoustic feature output by the adaptive pooling module, and a second discrete acoustic token sequence output by the vector quantization module; construct a fourth loss function based on the second discrete acoustic token sequence to constrain the consistency of the acoustic features; construct a fifth loss function based on the quantization error between the fourth acoustic feature and the second discrete acoustic token sequence; adjust the model parameters of the initial coding model based on the fourth and fifth loss functions to obtain the acoustic coding model.

[0137] Optionally, when transmitting the first discrete semantic token sequence and the first discrete acoustic token sequence to the audio decoding device to reconstruct the first audio, it can be achieved by: time-aligning the first discrete semantic token sequence and the first discrete acoustic token sequence; concatenating the corresponding discrete semantic tokens and discrete acoustic tokens in the first discrete semantic token sequence and the first discrete acoustic token sequence step by step to obtain a joint encoded representation; and transmitting the joint encoded representation to the audio decoding device to reconstruct the first audio.

[0138] It should be noted that each step in the audio encoding method in this application corresponds one-to-one with the function of the audio encoding device in the audio encoding and decoding system in embodiment 1. Since embodiment 1 has been described in detail, some details not shown in this embodiment can be referred to embodiment 1, and will not be elaborated further here.

[0139] Example 3

[0140] Based on the audio codec system of Embodiment 1, this application embodiment also provides an audio decoding method implemented by an audio decoding device. It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. Furthermore, although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in a different order than that shown here.

[0141] Figure 7 This is a flowchart illustrating an audio decoding method according to an embodiment of this application, as shown below. Figure 7 As shown, the method includes the following steps S702-S706:

[0142] Step S702: Receive a first discrete semantic token sequence reflecting the semantic features of the first audio and a first discrete acoustic token sequence reflecting the acoustic features of the first audio transmitted by an audio encoding device, wherein the first discrete semantic token sequence is encoded according to a first bit rate, the first discrete acoustic token sequence is encoded according to a second bit rate, and the sum of the first bit rate and the second bit rate satisfies the constraint of the target transmission bit rate.

[0143] Step S704: Time-align the first discrete semantic token sequence and the first discrete acoustic token sequence based on the time step of the latent space of the pre-trained audio reconstruction model.

[0144] Step S706: Using an audio reconstruction model, audio reconstruction is performed in the latent space based on the time-aligned first discrete semantic token sequence and the first discrete acoustic token sequence to obtain the restored first audio. The audio reconstruction model is a stream matching model based on the diffusion Transformer architecture.

[0145] The following section explains each step of the audio encoding method in conjunction with a specific implementation process.

[0146] Optionally, when using an audio reconstruction model to perform audio reconstruction in the latent space based on a time-aligned first discrete semantic token sequence and a first discrete acoustic token sequence to obtain the reconstructed first audio, this can be achieved as follows: obtain a preset initial latent representation; input the initial latent representation and the time-aligned first discrete semantic token sequence and the first discrete acoustic token sequence into the audio reconstruction model to obtain the first audio output by the audio reconstruction model, wherein the audio reconstruction model starts with the initial latent representation in the latent space, generates a target latent representation step by step with the discrete semantic tokens and discrete acoustic tokens of the corresponding time step as condition variables, maps the target latent representation to a time-domain audio signal, and composes the first audio from the time-domain audio signals of all time steps.

[0147] It should be noted that each step in the audio decoding method in this application corresponds one-to-one with the function of the audio decoding device in the audio encoding and decoding system in embodiment 1. Since embodiment 1 has been described in detail, some details not shown in this embodiment can be referred to embodiment 1, and will not be elaborated further here.

[0148] Example 4

[0149] According to an embodiment of this application, a computer program product is also provided, which includes a computer program, wherein when the computer program is executed by a processor, it implements the audio encoding method in embodiment 2 or the audio decoding method in embodiment 3.

[0150] According to an embodiment of this application, a non-volatile storage medium is also provided, which includes a stored computer program, wherein the device containing the non-volatile storage medium executes the audio encoding method in embodiment 2 or the audio decoding method in embodiment 3 by running the computer program.

[0151] According to an embodiment of this application, a processor is also provided for running a computer program, wherein the computer program executes the audio encoding method in embodiment 2 or the audio decoding method in embodiment 3 during runtime.

[0152] According to an embodiment of this application, an electronic device is also provided, comprising: a memory and a processor, wherein the memory stores a computer program, and the processor is configured to execute the audio encoding method of embodiment 2 or the audio decoding method of embodiment 3 through the computer program.

[0153] Optionally, the computer program may execute the following steps during runtime: acquiring a first audio signal to be transmitted and a target transmission bit rate; extracting a first semantic feature sequence from the first audio signal, and performing vector quantization on the first semantic feature sequence based on the target semantic codebook to obtain a first discrete semantic token sequence, wherein the first discrete semantic token sequence is encoded according to a preset first bit rate; extracting a first acoustic feature from the first audio signal, determining a second bit rate based on the target transmission bit rate and the first bit rate, performing downsampling on the first acoustic feature based on the second bit rate to obtain a second acoustic feature, and performing vector quantization on the second acoustic feature based on the target acoustic codebook to obtain a first discrete acoustic token sequence, wherein the sum of the first bit rate and the second bit rate satisfies the constraint of the target transmission bit rate, and the first discrete acoustic token sequence is encoded according to the second bit rate; and transmitting the first discrete semantic token sequence and the first discrete acoustic token sequence to an audio decoding device to reconstruct the first audio signal.

[0154] Optionally, the computer program may also execute the following steps during runtime: receiving a first discrete semantic token sequence reflecting the semantic features of the first audio and a first discrete acoustic token sequence reflecting the acoustic features of the first audio transmitted by an audio encoding device, wherein the first discrete semantic token sequence is encoded according to a first bit rate, the first discrete acoustic token sequence is encoded according to a second bit rate, and the sum of the first bit rate and the second bit rate satisfies the constraint of the target transmission bit rate; performing time alignment of the first discrete semantic token sequence and the first discrete acoustic token sequence based on the time step of the latent space of a pre-trained audio reconstruction model; and using the audio reconstruction model to perform audio reconstruction in the latent space based on the time-aligned first discrete semantic token sequence and the first discrete acoustic token sequence to obtain the restored first audio, wherein the audio reconstruction model is a stream matching model based on the diffusion Transformer architecture.

[0155] As an alternative implementation, the above-mentioned electronic device may exist in the form of a mobile terminal, a computer terminal, or a similar computing device. Figure 8 A hardware block diagram of an electronic device for implementing an audio encoding or decoding method is shown. Figure 8 As shown, the electronic device 80 may include one or more (shown as 802a, 802b, ..., 802n) processors 802 (processors 802 may include, but are not limited to, processing devices such as microprocessors or programmable logic devices), a memory 804 for storing data, and a transmission device 806 for communication functions. In addition, it may also include: a display, an input / output interface, a universal serial bus port (which may be included as one of the ports of a BUS bus), a network interface, a power supply, and / or a camera. Those skilled in the art will understand that... Figure 8The structure shown is for illustrative purposes only and does not limit the structure of the electronic device described above. For example, electronic device 80 may also include components that are more... Figure 8 The more or fewer components shown, or having the same Figure 8 The different configurations shown.

[0156] It should be noted that the aforementioned one or more processors 802 and / or other data processing circuits are generally referred to herein as "data processing circuits". These data processing circuits may be embodied, in whole or in part, in software, hardware, firmware, or any other combination thereof. Furthermore, the data processing circuits may be a single, independent processing module, or may be integrated, in whole or in part, into any other element of the electronic device 80. As involved in the embodiments of this application, the data processing circuit serves as a processor control mechanism (e.g., selection of a variable resistor termination path connected to an interface).

[0157] The memory 804 can be used to store software programs and modules of application software, such as the program instructions / data storage device corresponding to the audio encoding method or audio decoding method in the embodiments of this application. The processor 802 executes various functional applications and data processing by running the software programs and modules stored in the memory 804, thereby implementing the aforementioned audio encoding method or audio decoding method. The memory 804 may include high-speed random access memory, and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 804 may further include memory remotely located relative to the processor 802, and these remote memories can be connected to the electronic device 80 via a network. Examples of the aforementioned networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0158] The transmission device 806 is used to receive or send data via a network. Specific examples of the network described above may include a wireless network provided by the communication provider of the electronic device 80. In one example, the transmission device 806 includes a network adapter that can connect to other network devices via a base station to communicate with the Internet. In another example, the transmission device 806 may be a radio frequency module used for wireless communication with the Internet.

[0159] The display can be, for example, a touchscreen LCD display that allows the user to interact with the user interface of the electronic device 80.

[0160] The sequence numbers of the above embodiments are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0161] In the above embodiments of this application, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0162] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. The device embodiments described above are merely illustrative; for example, the division of units can be a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the displayed or discussed mutual couplings, direct couplings, or communication connections may be through some interfaces; indirect couplings or communication connections between units or modules may be electrical or other forms.

[0163] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0164] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0165] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as a USB flash drive, read-only memory, random access memory, portable hard drive, magnetic disk, or optical disk.

[0166] The above are merely preferred embodiments of this application. It should be noted that those skilled in the art can make various improvements and modifications without departing from the principles of this application, and these improvements and modifications should also be considered within the scope of protection of this application.

Claims

1. An audio encoding method, characterized in that, include: Obtain the first audio file to be transmitted and the target transmission bit rate; Extract the first semantic feature sequence of the first audio, and perform vector quantization on the first semantic feature sequence based on the target semantic codebook to obtain the first discrete semantic token sequence, wherein the first discrete semantic token sequence is encoded according to a preset first bit rate; The first acoustic feature of the first audio is extracted, the difference between the target transmission bit rate and the first bit rate is determined as the second bit rate, the first acoustic feature is downsampled based on the second bit rate to obtain the second acoustic feature, and the second acoustic feature is vector quantized based on the target acoustic codebook to obtain the first discrete acoustic token sequence, wherein the sum of the first bit rate and the second bit rate satisfies the constraint of the target transmission bit rate, and the first discrete acoustic token sequence is encoded according to the second bit rate; The first discrete semantic token sequence and the first discrete acoustic token sequence are transmitted to the audio decoding device to reconstruct the first audio; The process involves extracting a first acoustic feature from the first audio audio, downsampling the first acoustic feature based on the second bit rate to obtain a second acoustic feature, and performing vector quantization on the second acoustic feature based on a target acoustic codebook to obtain a first discrete acoustic token sequence. This includes: extracting features from the first audio audio using a feature extraction module in a pre-trained acoustic coding model to obtain a first acoustic feature, wherein the dimensions of the first acoustic feature include at least one of the following: timbre, energy variation, prosodic structure, and transient information; determining a target pooling ratio corresponding to the second bit rate using an adaptive pooling module in the acoustic coding model based on a preset mapping table, and pooling the first acoustic feature based on the target pooling ratio to obtain the second acoustic feature, wherein the mapping table stores mapping relationships between multiple bit rates and multiple pooling ratios; performing a mapping transformation on a preset basic acoustic codebook using a codebook transformation network in the vector quantization module of the acoustic coding model to obtain the target acoustic codebook; and performing nearest neighbor matching on the second acoustic feature according to the target acoustic codebook to obtain the first discrete acoustic token sequence.

2. The method according to claim 1, characterized in that, Extract the first semantic feature sequence from the first audio, and perform vector quantization on the first semantic feature sequence based on the target semantic codebook to obtain the first discrete semantic token sequence, including: The first audio is used to extract features using the feature extraction module in the pre-trained semantic coding model to obtain the first semantic feature sequence; The codebook transformation network in the vector quantization module of the semantic coding model is used to perform mapping transformation on the preset basic semantic codebook to obtain the target semantic codebook. The first semantic feature sequence is then matched with the nearest neighbor according to the target semantic codebook to obtain the first discrete semantic token sequence.

3. The method according to claim 2, characterized in that, The training process of the semantic coding model includes: Construct an initial encoding model, wherein the initial encoding model includes: a feature extraction module and a vector quantization module; Multiple second audio samples were obtained as training samples; For each second audio, the second audio is input into the initial encoding model to obtain the second semantic feature sequence output by the feature extraction module and the second discrete semantic token sequence output by the vector quantization module; Based on the second discrete semantic token sequence, a first loss function for constraining semantic consistency and a second loss function for constraining temporal structure consistency are constructed. A third loss function is constructed based on the quantization error between the second semantic feature sequence and the second discrete semantic token sequence; The semantic coding model is obtained by adjusting the model parameters of the initial coding model based on the first loss function, the second loss function, and the third loss function.

4. The method according to claim 3, characterized in that, A first loss function for constraining semantic consistency is constructed based on the second discrete semantic token sequence, including: The second discrete semantic token sequence is analyzed using a pre-trained text generation model to obtain the generated text content; Obtain the text content annotation corresponding to the second audio; The first loss function is constructed based on the difference between the generated text content and the text content annotation.

5. The method according to claim 3, characterized in that, A second loss function is constructed based on the second discrete semantic token sequence to constrain temporal structure consistency, including: Frame-level features of the second audio are extracted using a pre-trained audio self-supervised model; The second discrete semantic token sequence is mapped to the same representation space as the frame-level features to obtain the mapped features; Determine the cosine distance between the frame-level features and the mapped features, and construct the second loss function based on the cosine distance.

6. The method according to claim 1, characterized in that, The training process of the acoustic coding model includes: Construct an initial encoding model, wherein the initial encoding model includes: a feature extraction module, an adaptive pooling module, and a vector quantization module; Multiple second audio samples were obtained as training samples; For each second audio, the second audio is input into the initial encoding model to obtain the third acoustic feature output by the feature extraction module, the fourth acoustic feature output by the adaptive pooling module, and the second discrete acoustic token sequence output by the vector quantization module; A fourth loss function is constructed based on the second discrete acoustic token sequence to constrain the consistency of acoustic features; A fifth loss function is constructed based on the quantization error between the fourth acoustic feature and the second discrete acoustic token sequence; The acoustic coding model is obtained by adjusting the model parameters of the initial coding model based on the fourth loss function and the fifth loss function.

7. The method according to claim 1, characterized in that, Transmitting the first discrete semantic token sequence and the first discrete acoustic token sequence to an audio decoding device to reconstruct the first audio includes: Time alignment is performed on the first discrete semantic token sequence and the first discrete acoustic token sequence; By concatenating the first discrete semantic token sequence and the corresponding discrete semantic token and discrete acoustic token in the first discrete acoustic token sequence step by step, a joint encoded representation is obtained; The joint encoded representation is transmitted to an audio decoding device to reconstruct the first audio.

8. An audio decoding method, characterized in that, include: The system receives a first discrete semantic token sequence reflecting the semantic features of a first audio and a first discrete acoustic token sequence reflecting the acoustic features of the first audio transmitted by an audio encoding device, wherein the first discrete semantic token sequence is encoded at a first bit rate and the first discrete acoustic token sequence is encoded at a second bit rate, and the sum of the first bit rate and the second bit rate satisfies the constraint of the target transmission bit rate. The first discrete semantic token sequence and the first discrete acoustic token sequence are time-aligned based on the time steps of the latent space of the pre-trained audio reconstruction model. The audio reconstruction model is used to reconstruct the audio in the latent space based on the time-aligned first discrete semantic token sequence and the first discrete acoustic token sequence to obtain the restored first audio. The audio reconstruction model is a stream matching model based on the diffusion Transformer architecture.

9. The method according to claim 8, characterized in that, The audio reconstruction model is used to reconstruct audio in the latent space based on the time-aligned first discrete semantic token sequence and the first discrete acoustic token sequence, resulting in the reconstructed first audio, including: Obtain the preset initial latent representation; The initial latent representation, along with the time-aligned first discrete semantic token sequence and first discrete acoustic token sequence, are input into the audio reconstruction model to obtain the first audio output by the audio reconstruction model. The audio reconstruction model starts with the initial latent representation in the latent space, generates a target latent representation step by step using the discrete semantic tokens and discrete acoustic tokens of the corresponding time step as condition variables, maps the target latent representation to a time-domain audio signal, and assembles the time-domain audio signals of all time steps into the first audio.

10. An audio encoding / decoding system, characterized in that, include: Audio encoding device and audio decoding device, wherein, The audio encoding device is configured to: acquire a first audio signal to be transmitted and a target transmission bit rate; extract a first semantic feature sequence from the first audio signal and perform vector quantization on the first semantic feature sequence based on a target semantic codebook to obtain a first discrete semantic token sequence, wherein the first discrete semantic token sequence is encoded according to a preset first bit rate; extract a first acoustic feature from the first audio signal, determine the difference between the target transmission bit rate and the first bit rate as a second bit rate, perform downsampling on the first acoustic feature based on the second bit rate to obtain a second acoustic feature, and perform vector quantization on the second acoustic feature based on a target acoustic codebook to obtain a first discrete acoustic token sequence, wherein the sum of the first bit rate and the second bit rate satisfies the constraint of the target transmission bit rate, and the first discrete acoustic token sequence is encoded according to the second bit rate; transmit the first discrete semantic token sequence and the first discrete acoustic token sequence to the audio decoding device; wherein the first acoustic feature of the first audio signal is extracted, and the first semantic feature sequence is vector quantized based on the second bit rate... The process involves downsampling a first acoustic feature to obtain a second acoustic feature, and then performing vector quantization on the second acoustic feature based on a target acoustic codebook to obtain a first discrete acoustic token sequence. This includes: extracting features from the first audio using a feature extraction module in a pre-trained acoustic coding model to obtain a first acoustic feature, wherein the dimensions of the first acoustic feature include at least one of the following: timbre, energy variation, prosodic structure, and transient information; determining a target pooling ratio corresponding to the second bit rate using an adaptive pooling module in the acoustic coding model based on a preset mapping table, and pooling the first acoustic feature based on the target pooling ratio to obtain the second acoustic feature, wherein the mapping table stores mapping relationships between multiple bit rates and multiple pooling ratios; performing a mapping transformation on a preset basic acoustic codebook using a codebook transformation network in the vector quantization module of the acoustic coding model to obtain the target acoustic codebook; and performing nearest neighbor matching on the second acoustic feature according to the target acoustic codebook to obtain the first discrete acoustic token sequence. The audio decoding device is used to time-align the first discrete semantic token sequence and the first discrete acoustic token sequence based on the time steps of the latent space of the pre-trained audio reconstruction model; and to use the audio reconstruction model to perform audio reconstruction in the latent space based on the time-aligned first discrete semantic token sequence and the first discrete acoustic token sequence to obtain the restored first audio, wherein the audio reconstruction model is a stream matching model based on the diffusion Transformer architecture.

11. A computer program product, characterized in that, include: A computer program, wherein when executed by a processor, the computer program implements the audio encoding method according to any one of claims 1 to 7 or the audio decoding method according to any one of claims 8 to 9.

12. An electronic device, characterized in that, include: A memory and a processor, wherein the memory stores a computer program, and the processor is configured to execute, via the computer program, the audio encoding method of any one of claims 1 to 7 or the audio decoding method of any one of claims 8 to 9.

Images