Audio processing method and apparatus based on dual-stream discrete representation, device, and medium

Abstract

This invention relates to the field of speech and semantic technology, and discloses an audio processing method, apparatus, device, and medium based on dual-stream discrete representation. The method includes: acquiring target audio, a task identifier, and at least one input condition; generating a conditional prefix sequence; performing dual-stream discrete encoding on the target audio to obtain a semantic discrete label sequence and an acoustic discrete label sequence, and then generating a target label sequence; training an autoregressive generative network based on the conditional prefix sequence and the target label sequence, combining label lookup, splicing fusion, and continuous signal reconstruction to obtain reconstructed audio, and updating the dual-stream discrete encoding / decoding network to form a temporal audio processing model; and obtaining the task processing result through the model. This invention can be applied to business scenarios such as fintech and healthcare. By unifying the representation of multiple input conditions and performing discrete modeling and generation processing on semantic information and acoustic details respectively, it improves model reusability while balancing semantic consistency and acoustic details.

Description

Technical Field

[0001] This invention relates to the field of speech semantics technology, and in particular to an audio processing method, apparatus, device and medium based on dual-stream discrete representation. Background Technology

[0002] As the application of autoregressive and self-attention models in sequence modeling continues to expand, audio processing is gradually shifting from single-task modeling to a unified modeling framework. However, existing technologies generally suffer from insufficient representational power and fragmented modeling methods when transforming audio representations into discrete structures that can be processed by sequence models. On the one hand, traditional discrete coding methods struggle to simultaneously consider semantic information and acoustic details, leading to a significant conflict between semantic consistency and sound quality in reconstructed audio. On the other hand, different tasks often rely on independent modeling processes, lacking a unified sequence representation and generation mechanism, making it difficult for models to share capabilities and resulting in poor overall system scalability.

[0003] In the fintech sector, audio data is widely used in scenarios such as intelligent customer service, voice-based risk control verification, and transaction voice recording analysis. Existing technologies typically build independent models for speech separation, speech enhancement, or speaker recognition, making it difficult to process multiple time-aligned tasks within a unified framework. When dealing with multi-source speech mixing, target speaker extraction, and the collaborative processing of speech content and text conditions, traditional methods exhibit inconsistencies in conditional information fusion and sequence representation, leading to unstable mapping between speech content and business semantics, which in turn affects the continuity of subsequent risk control analysis and business processing.

[0004] In the healthcare field, audio data is used to assist in medical records, voice consultations, and rehabilitation assessments, placing high demands on semantic fidelity and sound detail. Current technologies struggle to simultaneously preserve both semantic information and physiologically relevant acoustic details in patient speech using a single acoustic modeling approach. In multi-task scenarios (such as combined applications of speech restoration and speech conversion), multiple models often need to be processed in series, increasing the complexity of the processing chain. Furthermore, the lack of a unified representation and generation mechanism across different tasks makes it difficult to transfer and reuse models across different medical scenarios, limiting the system's application capabilities in complex medical environments. Summary of the Invention

[0005] The main objective of this invention is to provide an audio processing method, apparatus, device, and storage medium based on dual-stream discrete representation. This invention aims to address the technical problem that existing technologies lack an audio processing mechanism capable of consistently representing different input conditions within a unified sequence modeling framework, while also considering both semantic information and acoustic details for discrete modeling and generation. This results in poor model reusability and insufficient consistency of reconstruction results in multi-task scenarios.

[0006] To achieve the above objectives, the present invention provides an audio processing method based on dual-stream discrete representation, comprising: Obtain the target audio, the task identifier, and at least one input condition corresponding to the target audio; perform condition feature extraction processing on the at least one input condition in the corresponding modality; construct a condition feature sequence based on the extracted condition features; and perform task feature generation processing on the task identifier to obtain the task features. Perform feature mapping processing on the conditional feature sequence, and concatenate the mapped conditional feature sequence with the task features in chronological order to obtain a conditional prefix sequence; The target audio is subjected to dual-stream discrete coding processing using a dual-stream discrete coding and decoding network to obtain a semantic discrete tag sequence and an acoustic discrete tag sequence. The semantic discrete tag sequence and the acoustic discrete tag sequence are subjected to hierarchical interleaving and delayed arrangement to obtain the target tag sequence; The conditional prefix sequence is input into the autoregressive generator network, and the target label sequence is used as the supervision sequence to train the autoregressive generator network to obtain the label generator network. Perform a tag lookup process on the semantic discrete tag sequence and the acoustic discrete tag sequence respectively to obtain the semantic vector sequence and the acoustic vector sequence; The semantic vector sequence and the acoustic vector sequence are spliced ​​and fused, and the splicing and fusion result is input into the decoding end of the dual-stream discrete codec network for continuous signal reconstruction to obtain the reconstructed audio; Based on the difference between the reconstructed audio and the target audio, the dual-stream discrete codec network is updated to obtain the updated dual-stream discrete codec network. The tag generation network is combined with the decoding end of the updated dual-stream discrete codec network to obtain a temporal audio processing model; A condition prefix sequence is constructed based on the input conditions to be processed and the task identifier to be processed, and the condition prefix sequence is input into the temporal audio processing model to obtain the task processing result.

[0007] Furthermore, to achieve the above objectives, the present invention provides an audio processing apparatus based on dual-stream discrete representation, comprising: The conditional input processing module is used to acquire target audio, task identifier, and at least one input condition corresponding to the target audio; perform conditional feature extraction processing on the at least one input condition in the corresponding modality; construct a conditional feature sequence based on the extracted conditional features; and perform task feature generation processing on the task identifier to obtain task features. A prefix construction module is used to perform feature mapping processing on the conditional feature sequence and concatenate the mapped conditional feature sequence with the task features in chronological order to obtain a conditional prefix sequence. A dual-stream discrete coding module is used to perform dual-stream discrete coding processing on the target audio using a dual-stream discrete coding and decoding network to obtain a semantic discrete tag sequence and an acoustic discrete tag sequence. The tag arrangement module is used to perform hierarchical interleaving and delayed arrangement of the semantic discrete tag sequence and the acoustic discrete tag sequence to obtain the target tag sequence; A training module is used to input the conditional prefix sequence into the autoregressive generator network and train the autoregressive generator network using the target label sequence as a supervision sequence to obtain a label generator network. The vector recovery module is used to perform a tag lookup table process on the semantic discrete tag sequence and the acoustic discrete tag sequence respectively to obtain the semantic vector sequence and the acoustic vector sequence; The audio reconstruction module is used to splice and fuse the semantic vector sequence and the acoustic vector sequence, and input the splicing and fusion result into the decoding end of the dual-stream discrete codec network to reconstruct the continuous signal and obtain the reconstructed audio. The network update module is used to update the dual-stream discrete codec network based on the difference between the reconstructed audio and the target audio, so as to obtain the updated dual-stream discrete codec network. The model assembly module is used to combine the tag generation network with the decoding end of the updated dual-stream discrete codec network to obtain a temporal audio processing model; The task reasoning module is used to construct a condition prefix sequence based on the input conditions to be processed and the task identifier to be processed, and input the condition prefix sequence into the temporal audio processing model to obtain the task processing result.

[0008] Furthermore, to achieve the above objectives, the present invention also provides a computer device, the computer device including a memory, a processor, and an audio processing program based on dual-stream discrete representation stored in the memory and executable on the processor, wherein the audio processing program based on dual-stream discrete representation, when executed by the processor, implements the steps of the audio processing method based on dual-stream discrete representation as described above.

[0009] Furthermore, to achieve the above objectives, the present invention also provides a computer-readable storage medium storing an audio processing program based on a two-stream discrete representation, wherein the audio processing program based on the two-stream discrete representation, when executed by a processor, implements the steps of the audio processing method based on the two-stream discrete representation as described above.

[0010] Beneficial Effects: This invention relates to the field of speech and semantic technology, and discloses an audio processing method, apparatus, device, and medium based on dual-stream discrete representation. The method includes: acquiring target audio, task identifier, and at least one input condition, generating a conditional prefix sequence; using a dual-stream discrete encoding / decoding network to perform dual-stream discrete encoding on the target audio, obtaining a semantic discrete label sequence and an acoustic discrete label sequence, generating a target label sequence; training an autoregressive generative network based on the conditional prefix sequence and the target label sequence, combining label lookup, splicing fusion, and continuous signal reconstruction to obtain reconstructed audio, and updating the dual-stream discrete encoding / decoding network to form a temporal audio processing model, through which the task processing result is obtained. This invention can be applied to business scenarios such as fintech and healthcare. By uniformly representing multiple input conditions as conditional prefix sequences and discretely encoding semantic information and acoustic details respectively, and then combining autoregressive generation and reconstruction update processing, a unified temporal audio processing model is formed, thereby improving model reusability in multi-task scenarios and balancing semantic consistency and acoustic details. Attached Figure Description

[0011] The present invention will be further described below with reference to the accompanying drawings and embodiments. In the accompanying drawings: Figure 1 This is a schematic diagram of an application environment for an audio processing method based on dual-stream discrete representation in one embodiment of the present invention; Figure 2 This is a flowchart illustrating an embodiment of the audio processing method based on dual-stream discrete representation of the present invention. Figure 3 This is a schematic diagram of the functional modules of a preferred embodiment of the audio processing device based on dual-stream discrete representation of the present invention; Figure 4 This is a schematic diagram of the structure of a computer device according to an embodiment of the present invention; Figure 5 This is another structural schematic diagram of a computer device according to one embodiment of the present invention. Detailed Implementation

[0012] It should be understood that the specific embodiments described herein are for illustrative purposes only and are not intended to limit the scope of the invention.

[0013] The audio processing method based on dual-stream discrete representation provided in this invention can be applied to, for example... Figure 1In this application environment, the client communicates with the server via a network. The server can obtain the target audio, task identifier, and at least one input condition from the client to generate a conditional prefix sequence. It then uses a dual-stream discrete codec network to perform dual-stream discrete encoding on the target audio, obtaining a semantic discrete label sequence and an acoustic discrete label sequence, generating a target label sequence. Based on the conditional prefix sequence and the target label sequence, an autoregressive generative network is trained. Combined with label lookup, splicing fusion, and continuous signal reconstruction, the reconstructed audio is obtained, and the dual-stream discrete codec network is updated. This results in a temporal audio processing model, through which the task processing result is obtained. This invention can be applied to business scenarios such as fintech and healthcare. By uniformly representing multiple input conditions as conditional prefix sequences and discretely encoding semantic information and acoustic details respectively, and then combining autoregressive generation and reconstruction update processing, a unified temporal audio processing model is formed, thereby improving model reusability in multi-task scenarios while balancing semantic consistency and acoustic details. The client can be, but is not limited to, various personal computers, laptops, smartphones, tablets, and portable wearable devices. The server can be implemented using a standalone server or a server cluster consisting of multiple servers. The present invention will now be described in detail through specific embodiments.

[0014] Please see Figure 2 , Figure 2 This is a flowchart illustrating an embodiment of the audio processing method based on dual-stream discrete representation provided by the present invention. It should be noted that although the logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than that shown here.

[0015] like Figure 2 As shown, the audio processing method based on dual-stream discrete representation proposed in this invention includes the following steps: S10: Obtain the target audio, the task identifier, and at least one input condition corresponding to the target audio; perform condition feature extraction processing on the at least one input condition in the corresponding modality; construct a condition feature sequence based on the extracted condition features; and perform task feature generation processing on the task identifier to obtain task features. In this embodiment, the target audio corresponds to a continuous time signal that needs to participate in subsequent processing. Its data structure is typically represented as a one-dimensional or two-dimensional sequence, containing amplitude variation information in the time dimension. It can be acquired through a real-time voice acquisition interface or a stored audio reading interface. During acquisition, a unified sampling rate conversion and amplitude range normalization process ensure that the target audio has a consistent data scale, facilitating the formation of an aligned representation structure with the input conditions. The task identifier describes the category of the current processing task. Its source can be a task code defined in the business system or a task tag provided by an external scheduling module. The task identifier is a discrete symbol in its original form and needs to be converted into a vector representation through mapping to participate in subsequent calculations.

[0016] At least one input condition includes at least one of conditional audio, reference audio, or text conditions, the source of which varies depending on the business scenario. Conditional audio corresponds to speech data with interference or mixed signals, reference audio corresponds to speech segments or feature samples of the target speaker, and text conditions correspond to semantic constraints or descriptions of target content. Addressing the differences in data structures across different modalities, conditional feature extraction processing corresponding to the modality is performed on at least one input condition. Specifically, conditional audio is used to extract semantically relevant representations through a time-series feature extraction structure, reference audio is used to generate speaker vectors through a voiceprint feature extraction structure, and text conditions are used to generate semantic vectors through a text encoding structure. The outputs of all feature extraction processes are in vector sequence form, thus achieving cross-modal unification at the representation level.

[0017] After obtaining the features of each modality, a conditional feature sequence is constructed based on the extracted conditional features. This process is achieved by integrating features from different modalities. When the input conditions contain multiple modalities, the features of each modality are concatenated or fused according to the input order or a preset order, so that features from different sources are arranged in the same sequence structure. When the input conditions contain only a single modality, the corresponding features are directly converted into a sequence structure while preserving their temporal or semantic order. The conditional feature sequence maintains a uniform dimensional form in structure, enabling it to be further combined with task features.

[0018] The task feature generation process converts task identifiers into vector forms that can participate in sequence representation. This process maps discrete task identifiers to a vector space through an identifier mapping operation, and then uses dimensionality transformation to ensure that the task features and conditional feature sequences are consistent in representation dimension. This allows task information to constrain and guide the conditional features in subsequent processing. Task features, as global control information, participate in sequence representation and together with the conditional feature sequences, constitute a unified input representation.

[0019] In practical implementation, conditional feature extraction can be adjusted based on the combination of input conditions. When the input conditions only contain conditional audio, semantic features can be directly extracted using a time-series encoding structure to form a conditional feature sequence. When the input conditions also contain reference audio, voiceprint features can be extracted and concatenated with semantic features to form a joint representation. When the input conditions include text conditions, semantic vectors can be generated using a text encoding structure and fused with audio features. In task feature generation, task identifiers can be converted into initial vectors using lookup table mapping or embedding vector mapping, and then the dimensions can be adjusted using linear transformations to ensure consistency between task features and conditional feature sequences. In fintech, transaction voice can be used as conditional audio and combined with speaker reference audio to generate a conditional feature sequence for subsequent voice verification processing. In healthcare, user voice and text descriptions can be used as input conditions simultaneously, and a unified sequence representation can be formed through multimodal feature extraction and fusion to adapt to different voice content expression formats.

[0020] This embodiment integrates information from multiple sources in a unified representation space by uniformly processing the target audio, task identifier, and at least one input condition, and performing corresponding feature extraction and sequence construction on different modal inputs. At the same time, task constraint information is introduced through task feature generation processing, thereby improving the collaborative expression ability between different input conditions and enhancing the representation consistency in multi-task scenarios.

[0021] S20, Perform feature mapping processing on the conditional feature sequence, and concatenate the mapped conditional feature sequence with the task features in chronological order to obtain a conditional prefix sequence; In this embodiment, the conditional feature sequence is represented as a sequence structure formed by arranging multiple feature vectors in a certain order, originating from a unified representation of inputs from different modalities. Since different modal features differ in dimension, distribution range, and semantic expression, feature mapping processing is required to transform the conditional feature sequence into a unified representation space. Feature mapping is implemented through a vector transformation structure, inputting each feature vector in the conditional feature sequence to a mapping unit for linear or nonlinear transformation. This ensures that the output features maintain dimensionality consistent with the subsequent sequence modeling structure, while simultaneously normalizing the feature value range to enhance the comparability and fusionability between features from different sources. Position-related information can be introduced during the mapping process to preserve the original arrangement relationship of each feature vector in the sequence, thereby maintaining the sequential structure between input conditions.

[0022] Task features are represented as vector structures generated from task identifiers, maintaining the same dimensionality as the mapped conditional feature sequence. Concatenating the mapped conditional feature sequence with the task features in chronological order means arranging and combining the two types of features along the sequence dimension, allowing the task features to participate as one or more starting elements in the overall representation. The concatenation operation is completed by inserting task feature vectors at the beginning of the sequence or at a specified position, enabling task information to act as a global constraint throughout the sequence. The concatenated result forms a conditional prefix sequence, which structurally contains both the task features and the mapped conditional feature sequence, maintaining a consistent vector dimension and sequence order, providing the input foundation for subsequent sequence modeling.

[0023] In this process, feature mapping and splicing operations together complete the unified expression of multimodal conditions and task information, transforming the originally scattered feature data into a unified sequence representation with an ordered structure, thereby enabling different types of information to be jointly modeled in the same representation space.

[0024] In practical implementation, feature mapping can employ a single-layer or multi-layer mapping structure. Each feature vector in the conditional feature sequence is input into a mapping unit for transformation, outputting a mapped feature sequence with a unified dimension. During the mapping process, different dimensions can be weighted using parameterized weights to adapt to the distribution differences of different modal features. During the concatenation process, task features can be inserted at the beginning of the mapped conditional feature sequence, placing them at the front of the sequence. Alternatively, task features can be expanded to multiple positions and distributed throughout the sequence to enhance the expressive strength of task information within the sequence. In fintech, task features corresponding to risk control tasks can be inserted at the front of the conditional feature sequence, enabling subsequent processing to differentiate speech content based on task type. In healthcare, task features corresponding to different processing needs can be concatenated with the speech feature sequence, allowing speech data to carry task information within a unified representation.

[0025] This embodiment performs feature mapping processing on the conditional feature sequence to represent different modal features in a unified dimensional space. By concatenating the conditional feature sequence with the task features in chronological order to form a conditional prefix sequence, the task information and conditional information are integrated in the same sequence structure, thereby improving the consistent expression capability of multimodal information and enhancing the guiding role of task constraints on sequence representation.

[0026] S30, the target audio is subjected to dual-stream discrete coding processing using a dual-stream discrete coding and decoding network to obtain a semantic discrete tag sequence and an acoustic discrete tag sequence; In this embodiment, the target audio is a continuous time-domain signal, which needs to be decomposed by an encoding structure before entering discrete representation. The dual-stream discrete codec network includes parallel semantic processing and acoustic processing structures. Both parts receive the same target audio at the input, but perform differentiated processing on their internal processing paths. The semantic processing structure is used to extract abstract representations related to speech content. It performs frame-by-frame processing on the target audio through a time-series coding unit and extracts context-related semantic feature vectors within each time slice. These semantic feature vectors form a continuous semantic feature sequence in the time dimension. The acoustic processing structure is used to extract features related to timbre, spectral morphology, and articulation details. It obtains an acoustic feature vector sequence through frequency domain transformation and multi-scale feature extraction units, thus preserving fine-grained variations in speech.

[0027] After obtaining the semantic and acoustic feature sequences, the continuous features need to be mapped to discrete representations. Discrete encoding is implemented through quantization units, which contain vector sets used to map the input feature vectors to a discrete index space. The semantic feature vectors output by the semantic processing structure are matched by the semantic quantization unit, replacing each semantic feature vector with its corresponding discrete index, forming a semantic discrete label sequence. The acoustic feature vectors output by the acoustic processing structure are matched by the acoustic quantization unit, converting each acoustic feature vector into its corresponding discrete index, forming an acoustic discrete label sequence. The discrete label sequence maintains the same sequential structure as the original audio frames in the time dimension, allowing semantic information and acoustic details to be expressed in discrete form.

[0028] Dual-stream discrete coding achieves the separation of semantic and acoustic information through parallel paths, enabling independent modeling of the two types of information in discrete space while preserving temporal order. Semantic discrete tag sequences are used to describe the structural information of speech content, while acoustic discrete tag sequences are used to describe the timbre and detail information of speech. Both can participate in different levels of sequence modeling in subsequent processing.

[0029] In practical implementation, the semantic processing structure can adopt a multi-layer time-series coding structure to encode the target audio frame by frame and extract context-related features; the acoustic processing structure can adopt a multi-scale frequency domain processing structure to decompose information in different frequency bands and extract fine-grained acoustic features. The quantization unit can adopt a hierarchical quantization structure to map continuous features to discrete indices step by step, so that high-level semantic information and low-level acoustic details correspond to different discrete spaces.

[0030] This embodiment inputs the target audio into a dual-stream discrete codec network and extracts semantic and acoustic features respectively. Then, it generates a semantic discrete tag sequence and an acoustic discrete tag sequence through quantization processing, so that the speech content and timbre details are independently represented in discrete space, thereby improving the expressive ability of semantic information and acoustic details while maintaining the temporal order structure.

[0031] S40, perform hierarchical interleaving and delayed arrangement on the semantic discrete tag sequence and the acoustic discrete tag sequence to obtain the target tag sequence; In this embodiment, the semantic discrete tag sequence represents the set of semantic discrete indices extracted from the target audio, and the acoustic discrete tag sequence represents the set of acoustic discrete indices extracted from the target audio. Although both sequences are arranged along the time dimension, they express different contents. The former focuses on speech content and contextual semantics, while the latter focuses on timbre, spectral morphology, and pronunciation details. If the two types of sequences are directly input into the subsequent sequence processing structure, problems such as scattered arrangement relationships between different categories of tags, unclear hierarchical information, and unstable correspondence at the same time position can easily occur. Therefore, it is necessary to perform hierarchical interleaving and delayed arrangement of the semantic discrete tag sequence and the acoustic discrete tag sequence. The hierarchy in the hierarchical interleaving arrangement processing comes from the hierarchical belonging relationship formed by the discrete tags in the encoding stage, or it can come from the grouping relationship of the tags in the discrete representation. The meaning of interleaving is to rearrange the originally separately arranged semantic and acoustic tags into the same tag sequence according to a unified rule, so that different categories of tags form a jointly processed arrangement structure in the same sequence space. The delay in delayed permutation is not simply a matter of time postponement, but rather assigning different sequence position offsets to different levels and categories of markers. This ensures that the semantic and acoustic information associated with the same time position has a clear relative order when the sequence is unfolded, thus avoiding overlap of multiple types of markers at the same position.

[0032] The implementation of hierarchical interleaving is based on first determining the hierarchical position of each discrete marker in the semantic and acoustic discrete marker sequences, as well as the order information corresponding to the original time positions. After determination, semantic and acoustic discrete markers from the same time position or the same local time interval are written into a unified sequence according to a preset interleaving order. This preset interleaving order can be set to semantic-first writing, acoustic-first writing, or writing sequentially according to hierarchical level. The result is not a simple concatenation, but rather a structure in which multiple types of markers appear alternately in the unified sequence. Delayed arrangement further applies positional offset control on top of interleaving, setting different offsets for markers at different levels, causing some markers to shift their writing positions in the sequence relative to their original time positions, forming a staggered arrangement. This staggered arrangement prevents multiple layers of markers corresponding to the same time from stacking in the same sequence position, instead distributing them in adjacent or intermittent positions, allowing subsequent sequence processing structures to distinguish the hierarchical and category origins of the markers during reading. After hierarchical interleaving and delayed permutation, the semantic discrete tag sequence and the acoustic discrete tag sequence are rearranged into the target tag sequence. The target tag sequence retains the original temporal order constraint and contains hierarchical, categorical and misalignment relationships.

[0033] The target label sequence is structurally represented as a one-dimensional discrete index sequence, but it already carries multiple types of information. Each position in the target label sequence is no longer a natural continuation of a single category label, but rather the result of hierarchical interleaving and delayed permutation processing. The construction of the target label sequence enables subsequent sequence modeling structures to simultaneously read semantic and acoustic information from a single input sequence and identify the relative relationships between labels at different levels based on positional differences. The direct significance of this processing is that it integrates information originally scattered across two discrete sequences into a uniformly schedulable sequence input form. For customer service recordings, multi-speaker call logs, and risk control voice verification audio in fintech businesses, the target label sequence can uniformly map changes in speech content and speaker timbre to a single sequence representation. For consultation voice, rehabilitation training voice, and continuous spoken voice in healthcare businesses, the target label sequence can simultaneously incorporate semantic expression and vocal details into a unified sequence structure, facilitating subsequent processing structures to distinguish between content information and vocal performance information.

[0034] In practical implementation, semantic discrete tag sequences and acoustic discrete tag sequences can be processed by first grouping and then interleaving according to hierarchy. Discrete tags in the semantic discrete tag sequence are first grouped into multiple semantic hierarchy sets based on their hierarchical affiliation, and discrete tags in the acoustic discrete tag sequence are grouped into multiple acoustic hierarchy sets. Then, the discrete tags in each set are alternately arranged according to their time position to form an initial interleaving sequence. Finally, positional offsets are applied to the discrete tags in the initial interleaving sequence based on hierarchical differences to form the target tag sequence. Alternatively, they can be processed by first aligning and then allocating according to their time position. At each time position, the corresponding semantic and acoustic discrete tags are extracted, written into the sequence according to a preset category order, and then different delays are set according to the hierarchy level to distribute the higher-level and lower-level tags in a dispersed manner. The preset interleaving order and delay settings can also be adjusted for different business environments. In fintech applications, for transaction voice verification scenarios, increasing the resolution of acoustic tags related to speaker timbre in the arrangement allows the target tag sequence to enhance the sequential expression of speaker features while preserving speech content. For intelligent customer service quality inspection scenarios, increasing the prominence of semantic tags related to semantic content in the arrangement makes the target tag sequence more suitable for subsequent processing of business semantics. In healthcare applications, for rehabilitation assessment voice scenarios, increasing the delay differences of acoustic tags related to vocalization states makes different vocal details easier to distinguish in the target tag sequence. For health consultation voice scenarios, maintaining a relatively balanced interleaving density between semantic and acoustic tags balances content expression and sound performance.

[0035] This embodiment rearranges the two types of discrete information represented separately into a unified target label sequence by hierarchically interleaving and delaying the semantic discrete label sequence and the acoustic discrete label sequence. This allows the target label sequence to carry hierarchical and category information while maintaining the temporal order, and distinguishes the relative relationship of different levels of labels through positional offset, thereby enhancing the collaborative expression ability of semantic and acoustic information in the unified sequence representation.

[0036] S50, the conditional prefix sequence is input into the autoregressive generator network, and the target label sequence is used as the supervision sequence to train the autoregressive generator network to obtain the label generator network; In this embodiment, the conditional prefix sequence corresponds to the input sequence participating in the discrete label generation constraints. It contains sequential representations of task features and conditional features, used to load task categories, speech content constraints, speaker reference information, or text semantic information into the autoregressive generative network. The autoregressive generative network includes an input embedding layer, a positional encoding layer, multi-layer self-attention units, a feedforward unit, and an output projection layer. The input embedding layer receives the conditional prefix sequence and the target label sequence after positional offset processing, mapping the sequence elements to a unified latent space. The positional encoding layer writes sequential information to each input position. The multi-layer self-attention unit establishes dependencies on all information prior to the current input position based on causal masking. The feedforward unit expands and compresses the dimensionality of the hidden representation. The output projection layer maps the hidden representation to a discrete label distribution. Residual connections and normalization structures maintain representation stability between layers. The number of layers in a multi-layer self-attention unit can be set to 12, 18, or 24; the hidden dimension can be set to 512, 768, or 1024; the number of attention heads can be set to 8, 12, or 16; and the feedforward unit dimension can be set to 2048, 3072, or 4096.

[0037] The target label sequence is used to provide position-wise supervision information. During training, the target label sequence is first offset by one position, ensuring that the current position prediction relies only on the preceding label and the conditional prefix sequence, thus preserving the autoregressive constraint. The offset target label sequence and the conditional prefix sequence are concatenated along the sequence dimension and then input into the autoregressive generative network. The network outputs a discrete label probability distribution at each prediction position. The supervision sequence consists of the unoffset target label sequence, and the supervision method employs position-wise classification constraints to align the network output distribution with the corresponding ground truth labels in the supervision sequence. Parameter updates are applied to the input embedding layer, multi-layer self-attention units, feedforward units, and output projection layers. The number of training epochs can be set to 20 to 80, the batch size can be set to 8, 16, or 32, the learning rate can be set to 0.0001, 0.00005, or 0.00001, the optimizer can use an adaptive moment estimation structure, the gradient clipping threshold can be set to 1 or 5, and the maximum sequence length can be set to 512, 1024, or 2048. During training, a termination condition is set. The termination condition can be that the validation set loss value decreases by less than 0.001 for five consecutive rounds, the validation set label prediction accuracy no longer improves for five consecutive rounds, or the training rounds reach a preset upper limit of 40, 60, or 80. When any termination condition is met, parameter updates stop, and the network parameters corresponding to the minimum validation set loss value are determined as the label generation network.

[0038] The input data consists of a conditional prefix sequence and a shifted target label sequence, while the output data is a discrete label prediction distribution of the same length as the target label sequence. The input embedding layer and the positional encoding layer are sequentially connected, the positional encoding layer is connected to the first layer's self-attention unit, and their respective attention units and feedforward units are stacked sequentially. The final hidden representation layer is connected to the output projection layer. In fintech applications, the input data can be a conditional prefix sequence formed from transaction voice, customer service voice, or risk control verification voice, while the supervision data is a target label sequence consistent with the corresponding business voice. In healthcare applications, the input data can be a conditional prefix sequence formed from health consultation voice, rehabilitation training voice, or long-term follow-up voice, while the supervision data is a target label sequence consistent with the corresponding medical voice. Through a unified input and supervision organization, the autoregressive generative network can learn the mapping relationship between conditional information and target labels in different business voice environments.

[0039] In one implementation, the autoregressive generative network uses a 12-layer decode-only structure with a hidden dimension of 768, 12 attention heads, a feedforward unit dimension of 3072, a batch size of 16, a learning rate of 0.0001, a maximum sequence length of 1024, a gradient pruning threshold of 1, an upper limit of 40 training epochs, and a termination condition where the validation set loss value decreases by less than 0.001 for five consecutive epochs or the number of training epochs reaches 40. In another implementation, the autoregressive generative network uses a 24-layer decode-only structure with a hidden dimension of 1024, 16 attention heads, a feedforward unit dimension of 4096, a batch size of 8, a learning rate of 0.00005, an upper limit of 80 training epochs, and a termination condition where the validation set label prediction accuracy no longer improves for five consecutive epochs or the number of training epochs reaches 80. A lighter structure can also be used, with 8 layers, 512 hidden dimensions, 8 attention heads, 2048 feedforward unit dimensions, 512 maximum sequence lengths, 32 batch sizes, 0.0001 learning rate, and termination conditions set as the validation set loss value decreasing by less than 0.001 for 3 consecutive rounds or the number of training rounds reaching 30.

[0040] In fintech applications, conditional prefix sequences can consist of customer transaction call features, target speaker reference information, and task identifiers. Supervision data uses target labeled sequences corresponding to the transaction calls. After training, the label generation network can generate discrete label distributions matching the business speech based on the conditional prefix sequences, under business tasks such as risk control verification, target speaker extraction, or speech enhancement. In healthcare applications, conditional prefix sequences can consist of consultation speech, supplementary text information, and task identifiers. Supervision data uses target labeled sequences corresponding to the consultation speech. After training, the label generation network can learn the semantic expressions and conditional constraints in medical speech, making it suitable for processing categories such as speech restoration, semantic preservation processing, or multi-source speech separation. In different business environments, the maximum sequence length can be adjusted based on the average speech duration, the number of layers and hidden dimensions can be adjusted based on memory resources, and the batch size and termination conditions can be adjusted based on the training sample size.

[0041] For example, the autoregressive training loss function for the target labeled sequence is:

[0042] in, This represents the training loss of the autoregressive generative network. N represents the total number of labels involved in training. n represents the index of the current label position. () indicates that the parameter is determined by the parameter A defined conditional probability distribution. This represents the network parameters of the autoregressive generative network. This represents the nth target label. This represents all historical tags preceding the nth target tag. C represents the conditional prefix.

[0043] This implementation inputs the conditional prefix sequence into the autoregressive generative network and trains it using the target labeled sequence as the supervised sequence. This allows task information and conditional information to participate in discrete label prediction constraints within a unified sequence representation. During position-by-position learning, the network establishes a correspondence between the conditional prefix sequence and the target labeled sequence. Simultaneously, the training convergence process is controlled by the termination condition, thereby forming a label generation network capable of responding to various input conditions and different task identifiers, enhancing the conditional constraint capability and prediction stability of sequence generation.

[0044] S60, perform a tag lookup table process on the semantic discrete tag sequence and the acoustic discrete tag sequence respectively to obtain the semantic vector sequence and the acoustic vector sequence; In this embodiment, the semantic discrete tag sequence and the acoustic discrete tag sequence are each composed of discrete indices, with each discrete tag corresponding to a position in the discrete representation space. Tag lookup table processing is used to map the discrete indices to continuous vector representations, enabling the discrete representations to participate in subsequent continuous space processing. Each semantic discrete tag in the semantic discrete tag sequence undergoes an index reading operation through a semantic mapping structure. The semantic mapping structure contains a set of semantic vectors arranged by discrete tag number, with each semantic discrete tag corresponding to one semantic vector, thus forming a semantic vector sequence. Similarly, each acoustic discrete tag in the acoustic discrete tag sequence undergoes an index reading operation through an acoustic mapping structure. The acoustic mapping structure contains a set of acoustic vectors arranged by discrete tag number, with each acoustic discrete tag corresponding to one acoustic vector, thus forming an acoustic vector sequence.

[0045] The semantic and acoustic mapping structures can be constructed from either the quantization representation set used in the discrete encoding stage or from a set of vectors that are independently initialized and continuously updated during training. The semantic vector set focuses on expressing semantic information in speech, such as speech content structure, pronunciation patterns, or contextual relationships. The acoustic vector set focuses on expressing acoustic information in speech, such as spectral details, timbre features, or vocal differences. The index position of each discrete semantic tag in the semantic mapping structure is consistent with its discrete number, and the index position of each discrete acoustic tag in the acoustic mapping structure is consistent with its discrete number, thus ensuring a one-to-one correspondence between discrete indices and continuous vectors during the tag lookup process.

[0046] During the label lookup process, the order of the semantic discrete label sequence remains unchanged, and the semantic vectors obtained from the lookup form a semantic vector sequence in the original order. Similarly, the order of the acoustic discrete label sequence remains unchanged, and the acoustic vectors obtained from the lookup form an acoustic vector sequence in the original order. The dimension of the semantic vectors can be set to 128, 256, or 512, and the dimension of the acoustic vectors can also be set to 128, 256, or 512, or the same dimension can be set according to the requirements of subsequent fusion processing. The vectors in the semantic mapping structure and the acoustic mapping structure can be adjusted through gradient updates during training, so that the continuous representation gradually approximates the semantic and acoustic feature distributions corresponding to the discrete encoding stage.

[0047] In one implementation, both the semantic mapping structure and the acoustic mapping structure are stored in matrix form. The number of rows in the matrix corresponds to the total number of discrete tags, and the number of columns corresponds to the vector dimension. Each semantic discrete tag in the semantic discrete tag sequence serves as an index to directly access the corresponding row vector in the semantic mapping structure, and each acoustic discrete tag in the acoustic discrete tag sequence serves as an index to directly access the corresponding row vector in the acoustic mapping structure, thereby constructing semantic vector sequences and acoustic vector sequences respectively. The semantic vector dimension is set to 256, and the acoustic vector dimension is set to 256, ensuring that the two types of vectors have a consistent representation space in subsequent processing stages.

[0048] In another implementation, the semantic and acoustic mapping structures are organized hierarchically, with discrete markers at different levels corresponding to different sub-mapping regions. Each semantic discrete marker in the semantic discrete marker sequence selects its corresponding sub-mapping region for lookup based on hierarchical information, and each acoustic discrete marker in the acoustic discrete marker sequence selects its corresponding sub-mapping region for lookup based on hierarchical information, thus forming hierarchically distinct semantic and acoustic vector sequences. The semantic vector dimension is set to 512, and the acoustic vector dimension is set to 512 to enhance high-level semantic expression and acoustic detail expression capabilities.

[0049] In fintech, semantic vector sequences can be used to express semantic information in transaction voice messages, such as transaction instructions, customer intent, or risk warnings, while acoustic vector sequences can be used to express speaker characteristics, voice environment, or call quality information. In healthcare, semantic vector sequences can be used to express symptom descriptions, behavioral expressions, or health status information in consultation voice messages, while acoustic vector sequences can be used to express vocal features, emotional changes, or audio quality information. The size and vector dimensions of the semantic and acoustic mapping structures can be adjusted according to the complexity of the voice message in different business environments.

[0050] This embodiment performs a tag lookup process on the semantic discrete tag sequence and the acoustic discrete tag sequence, mapping the discrete index to a continuous vector representation. This enables semantic and acoustic information to be expressed in a unified continuous space while maintaining the original arrangement order and correspondence, thus providing a stable continuous representation basis for subsequent fusion processing.

[0051] S70, the semantic vector sequence and the acoustic vector sequence are spliced ​​and fused, and the splicing and fusion result is input into the decoding end of the dual-stream discrete codec network for continuous signal reconstruction to obtain the reconstructed audio; In this embodiment, the semantic vector sequence and the acoustic vector sequence carry different types of information in a continuous representation space. The semantic vector sequence is used to express the structure and context of speech content, while the acoustic vector sequence is used to express spectral details, timbre features, and vocal differences. The splicing and fusion processing is used to integrate the two types of vectors in a unified representation space, enabling semantic and acoustic information to form a joint representation in the same temporal structure. The splicing and fusion processing can perform position-by-position alignment based on the correspondence between positions in the sequence, combining each semantic vector in the semantic vector sequence with the corresponding acoustic vector in the acoustic vector sequence. The combination method can be vector splicing, weighted superposition, or mapping fusion. Vector splicing connects the semantic and acoustic vectors along the feature dimension to form a fused vector. Weighted superposition applies weights to the semantic and acoustic vectors and then sums them to form a fused vector. Mapping fusion uses a fusion mapping structure to uniformly transform the two types of vectors to form a fused vector.

[0052] In the vector concatenation method, the semantic vector dimension and the acoustic vector dimension can be set to 128, 256, or 512, respectively. The dimension of the concatenated fused vector is the sum of the two dimensions; for example, concatenating 256 vectors results in a 512-dimensional fused vector. In the weighted superposition method, the semantic weight and acoustic weight can be set to 0.5 and 0.5, 0.6 and 0.4, or 0.7 and 0.3, respectively, adjusting the proportion of semantic and acoustic information in the fused representation. In the mapping fusion method, a fusion mapping structure can be constructed. The semantic and acoustic vectors are input into a unified mapping space, and the fused vector is obtained through linear transformation or a multi-layer mapping structure. The dimension of the fused vector can be set to 256, 512, or 768. The fused vectors are arranged in their original order to form the concatenated fusion result, ensuring the temporal consistency of the sequence.

[0053] The spliced ​​and fused result serves as the continuous representation input to the decoder of the dual-stream discrete codec network. The decoder converts the continuous representation into a continuous audio signal representation. Internally, the decoder may include an upsampling module, a feature reconstruction module, and a signal generation module. The upsampling module expands the fused vector sequence in the time dimension to the target audio sampling resolution; the feature reconstruction module converts the fused vector sequence into a continuous spectral representation; and the signal generation module restores the spectral representation to a time-domain audio signal. The upsampling factor can be set to 2, 4, or 8; the spectral representation dimension can be set to 80 or 128; and the sampling rate can be set to 16000, 22050, or 44100. During the continuous signal reconstruction process, the fused vector sequence is gradually restored to a continuous audio waveform through layer-by-layer mapping, resulting in the reconstructed audio.

[0054] The input vector sequence is fed into an upsampling module. The upsampling module outputs an expanded feature sequence, which is then input into a feature reconstruction module. The feature reconstruction module outputs a continuous spectrum representation, which is input into a signal generation module. The signal generation module outputs a time-domain audio signal as the reconstructed audio. In fintech applications, the input data can be semantic and acoustic vector sequences corresponding to transaction voice, customer service voice, or risk verification voice. The output data is the reconstructed audio, used to recover the voice content and speaker features. In healthcare applications, the input data can be semantic and acoustic vector sequences corresponding to health consultation voice, rehabilitation training voice, or follow-up voice. The output data is the reconstructed audio, used to recover the voice expression content and audio details.

[0055] In one implementation, the semantic vector sequence and the acoustic vector sequence are fused using a positional vector concatenation method. The semantic vector dimension is set to 256, the acoustic vector dimension is set to 256, and the fused vector dimension is 512. The fused vector sequence is input to the decoder, with an upsampling factor of 4, a spectral representation dimension of 80, and a sampling rate of 22050. A continuous audio signal is then reconstructed through layer-by-layer mapping. In another implementation, the semantic vector sequence and the acoustic vector sequence are fused using a weighted superposition method. The semantic weight is set to 0.6, the acoustic weight to 0.4, and the fused vector dimension is set to 256. The decoder upsampling factor is set to 2, and the sampling rate is set to 16000. This method is suitable for short speech lengths or real-time processing scenarios. Alternatively, a mapping fusion method can be used. The semantic vector sequence and the acoustic vector sequence are input to a fusion mapping structure containing 2 or 3 layers of mapping units, with an output dimension of 512 or 768, and then input to the decoder for continuous signal reconstruction.

[0056] This embodiment combines semantic vector sequences and acoustic vector sequences to form a joint representation of semantic and acoustic information in a unified representation space. The fused representation is then restored to a continuous audio signal through the decoding end, so that the reconstructed audio simultaneously contains speech content structure and audio detail information, thereby improving the integrity and consistency of continuous signal reconstruction.

[0057] S80, based on the difference between the reconstructed audio and the target audio, the dual-stream discrete codec network is updated to obtain the updated dual-stream discrete codec network; In this embodiment, the difference between the reconstructed audio and the target audio is used to characterize the deviation of the two-stream discrete codec network during the continuous signal recovery process. The extraction objects of the difference include time-domain waveform differences, spectral distribution differences, and high-level representation differences. Time-domain differences reflect the deviations between the reconstructed audio and the target audio in amplitude variation, periodic structure, and local transients, which are suitable for constraining audio continuity and waveform trends. Spectral distribution differences reflect the deviations between the reconstructed audio and the target audio in short-time energy distribution, formant structure, and harmonic details, which are suitable for constraining timbre, clarity, and fine-grained acoustic features. High-level representation differences reflect the distance between the reconstructed audio and the target audio in the coding representation space, which are suitable for constraining semantic relevance and overall representation consistency. The essence of the update process is to adjust the internal parameters of the two-stream discrete codec network under the difference constraints, so that the semantic coding branch, acoustic coding branch, quantization representation structure, and decoding end converge towards a smaller difference. The semantic coding branch is responsible for mapping the target audio to a semantically continuous representation, the acoustic coding branch is responsible for mapping the target audio to an acoustically continuous representation, the quantization representation structure is responsible for mapping the continuous representation to the discrete representation space, and the decoding end is responsible for recovering the continuous signal from the discrete representation. During the update process, the reconstructed audio and the target audio are simultaneously input into the difference evaluation unit. The difference evaluation unit outputs time-domain difference values, spectral difference values, and representation difference values. Then, the parameter update unit adjusts the trainable parameters in the dual-stream discrete codec network in reverse based on these difference values. The parameter update unit can connect the semantic coding branch, acoustic coding branch, quantization representation structure, and decoding end, enabling different parts to be updated synchronously according to a unified difference constraint. If the reconstructed audio is close to the target audio in the time domain but has a large spectral difference, the parameter update unit increases the influence of the spectral-related difference terms; if the spectral distribution is similar but the high-level representation deviation is large, the parameter update unit increases the influence of the representation difference terms. Time-domain difference extraction can use sample-by-sample difference, frame-level energy difference, or in-window waveform deviation statistics. Spectral difference extraction can use amplitude spectrum difference, logarithmic spectrum difference, or Mel spectrum difference. High-level representation difference extraction can use distance statistics between coded representations. Training parameters can be set to a learning rate of 0.0001, 0.00005, or 0.00001; batch size can be set to 8, 16, or 32; the optimizer can use an adaptive moment estimation structure; the gradient pruning threshold can be set to 1 or 5; and the parameter update period can be set to update once per batch, or once after accumulating 2 or 4 batches. Termination conditions need to be written into the update process. Termination conditions can be set as follows: the validation set spectral difference value decreases by less than 0.001 for 5 consecutive rounds; the validation set comprehensive difference value no longer decreases for 5 consecutive rounds; or the number of update rounds reaches 40, 60, or 80. When any termination condition is met, parameter updates stop, and the dual-stream discrete codec network corresponding to the minimum comprehensive difference value is determined as the updated dual-stream discrete codec network.

[0058] In one implementation, the reconstructed audio and the target audio are framed within a time window of 20 ms to 40 ms. Temporal and metronome differences are extracted from each frame, and high-level representation differences are extracted from the encoded representation. The three types of differences are weighted and used as the update criterion. The temporal difference weight can be set to 0.3, the metronome difference weight to 0.5, and the representation difference weight to 0.2. The learning rate is set to 0.0001, the batch size to 16, the gradient pruning threshold to 1, and the termination condition is set to a decrease in the overall difference value of the validation set of less than 0.001 for five consecutive rounds or a total of 60 update rounds. In another implementation, focusing on improving semantic preservation, the representation difference weight is increased to 0.4, the metronome difference weight is set to 0.4, the temporal difference weight is set to 0.2, the learning rate is set to 0.00005, and the batch size is set to 8. This approach is suitable for environments with high semantic preservation requirements. Alternatively, a setting that emphasizes acoustic detail recovery can be adopted, with the metronome difference weight set to 0.6, the temporal difference weight set to 0.25, and the representation difference weight set to 0.15. In fintech applications, input data can be set as the reconstructed audio of a transaction call and the target audio of the transaction call, with difference extraction constraining the consistency between the speaker's timbre and the business semantics. In healthcare applications, input data can be set as the reconstructed audio of a health consultation and the target audio of a health consultation, with difference extraction constraining the consistency of speech content, emotional fluctuations, and vocal details. For environments with limited computing resources, the batch size can be reduced and the upper limit of the number of update rounds can be lowered; for environments with higher requirements for audio detail, the proportion of spectral difference terms can be increased and the sampling rate can be improved.

[0059] For example, the update loss function for a two-stream discrete codec network is:

[0060] in, This represents the overall loss used when updating a two-stream discrete codec network. This represents the spectral reconstruction loss, used to constrain the difference in spectral representation between the reconstructed audio and the target audio. This indicates adversarial loss, used to constrain the realism of the reconstructed audio. The feature matching loss is used to constrain the difference in intermediate representations between the reconstructed audio and the target audio. This represents the quantization constraint loss, used to constrain the consistency between the quantized representation and the encoded representation. This represents the self-supervised representation constraint loss, used to constrain the consistency of semantically related representations. The weighting coefficients represent the spectral reconstruction loss. This represents the weighting coefficient against losses. This represents the weight coefficients of the feature matching loss. The weighting coefficients represent the quantization constraint loss. This represents the weighting coefficient of the self-supervised representation constraint loss.

[0061] This embodiment updates the dual-stream discrete codec network based on the differences between the reconstructed audio and the target audio, allowing time-domain bias, spectral bias, and representation bias to participate in parameter adjustment. The semantic coding branch, acoustic coding branch, quantization representation structure, and decoding end are simultaneously optimized under the same update constraints, thereby enabling the updated dual-stream discrete codec network to more stably preserve semantic information and acoustic details, and improve the consistency between the reconstructed audio and the target audio.

[0062] S90, the marker generation network is combined with the decoding end of the updated dual-stream discrete codec network to obtain a temporal audio processing model; In this embodiment, the marker generation network outputs the probability distribution or discrete marker result of the discrete marker sequence. The decoder of the updated dual-stream discrete codec network receives the continuous representation corresponding to the discrete markers and reconstructs the continuous signal. Combining the marker generation network with the decoder of the updated dual-stream discrete codec network connects the discrete marker generation capability and the continuous signal recovery capability in the same structure, allowing the input conditions to be directly converted into continuous audio results by the decoder after the discrete representation is obtained by the marker generation network. The combination method can adopt a sequential connection structure, establishing a mapping relationship between the output interface of the marker generation network and the input interface of the decoder of the updated dual-stream discrete codec network. The discrete marker sequence output by the marker generation network needs to be converted into a vector representation before being input to the decoder. The vector representation conversion process can reuse the semantic vector sequence and acoustic vector sequence obtained by marker lookup table processing, or it can convert the discrete marker sequence into a continuous vector representation through a unified mapping structure. The combined structure maintains consistency in the time dimension, and the arrangement order in the discrete marker sequence corresponds one-to-one with the arrangement order in the input sequence of the decoder, enabling the decoder to gradually recover the continuous signal according to the marker generation order.

[0063] The updated dual-stream discrete codec network's decoder internally includes an upsampling module, a feature recovery module, and a signal generation module. The continuous vector sequence output by the tag generation network is input to the upsampling module, which performs temporal resolution expansion on the sequence. The feature recovery module maps the expanded vector sequence into a spectral representation, and the signal generation module converts the spectral representation into a time-domain audio signal. The interface between the tag generation network and the decoder needs to ensure consistent vector dimensions, which can be achieved by setting a unified latent space dimension, such as 256, 512, or 768. The combined overall structure forms an end-to-end processing structure from input conditions to continuous audio output, with the input being a conditional prefix sequence and the output being the reconstructed audio.

[0064] In fintech applications, input data can be set as a conditional prefix sequence and task identifier corresponding to customer call voice. After the labeling generation network outputs a discrete label sequence, it is input to the decoder through a combinatorial structure to generate continuous audio, used to recover transaction voice content and speaker features. In healthcare applications, input data can be set as a conditional prefix sequence and task identifier corresponding to health consultation or rehabilitation training voice. After the labeling generation network outputs a discrete label sequence, it is input to the decoder through a combinatorial structure to generate continuous audio, used to recover voice content expression and vocal details. Different audio quality requirements can be adapted by adjusting the latent space dimension and upsampling factor in different business environments.

[0065] In one implementation, after the tag generation network outputs a discrete tag sequence, a continuous vector sequence is obtained through a lookup table mapping. The dimension of the continuous vector is set to 512, consistent with the input dimension at the decoding end. The upsampling factor at the decoding end is set to 4, and the sampling rate is set to 22050. These are combined to form a complete temporal audio processing model. In another implementation, after the tag generation network outputs a discrete tag sequence, it is directly converted into a continuous vector representation through a mapping structure. The vector dimension is set to 256, the upsampling factor at the decoding end is set to 2, and the sampling rate is set to 16000, suitable for real-time processing environments. A higher-dimensional setting can also be used, setting the vector dimension to 768, the upsampling factor to 8, and the sampling rate to 44100, for processing environments with high requirements for audio detail. In fintech businesses, different dimension and sampling rate settings can be selected based on the complexity of the transaction voice; in healthcare businesses, different upsampling factors and vector dimensions can be selected based on the required detail in voice expression.

[0066] This embodiment combines the tag generation network with the decoding end of the updated dual-stream discrete codec network, enabling the discrete tag generation result to be directly converted into a continuous audio signal. This achieves a unified processing structure from input conditions to audio output, thereby improving the consistency of audio generation and processing efficiency under multi-task conditions.

[0067] S100: Construct a condition prefix sequence based on the input conditions to be processed and the task identifier to be processed, and input the condition prefix sequence into the temporal audio processing model to obtain the task processing result.

[0068] In this embodiment, the input conditions to be processed correspond to the input data used to constrain the direction of audio processing during the model application phase. The data format may include at least one of the following: audio conditions to be processed, reference audio conditions to be processed, or text conditions to be processed. The task identifier to be processed corresponds to the task category information of the current business objective, used to indicate the processing direction corresponding to the output result, such as speech enhancement, target speaker extraction, speech conversion, or sound source separation. The process of constructing the prefix sequence of the conditions to be processed is essentially to perform consistent representation processing on the input conditions to be processed and the task identifier to be processed according to the representation rules established in the training phase during the inference phase, so that the model input maintains the same data organization method as the training input. After the input conditions to be processed enter the feature extraction structure, the features to be processed are generated according to the modality type. After the task identifier to be processed enters the identifier mapping structure, the features to be processed are generated. The features to be processed are transformed into a unified representation space through feature mapping processing, and then concatenated with the features to be processed in chronological order to form the prefix sequence of the conditions to be processed. The condition prefix sequence to be processed is represented in the data structure as a set of sequence vectors of uniform dimension. The front of the vector carries task constraint information, and the back of the vector carries input condition information, so that the task category and the condition content can participate in subsequent reasoning.

[0069] The temporal audio processing model is a pre-combined inference structure, internally containing a label generation network and a decoder end of an updated two-stream discrete codec network. After the conditional prefix sequence to be processed is input into the temporal audio processing model, the label generation network outputs discrete label generation results based on the conditional prefix sequence. These discrete label generation results are then converted by the decoder into continuous audio output or task-related processed output. The form of the task processing result is determined by the task identifier. In the speech enhancement category, the task processing result can be represented as denoised speech output; in the target speaker extraction category, the task processing result can be represented as extracted target speech; in the speech conversion category, the task processing result can be represented as converted speech; and in the source separation category, the task processing result can be represented as separated target audio. The inference phase does not involve parameter updates, but it is necessary to ensure that the dimension, arrangement order, and task encoding method of the conditional prefix sequence to be processed remain consistent with those in the model training phase. Otherwise, the conditional constraints read by the label generation network will shift, causing the task processing result to deviate from the input requirements.

[0070] The model input receives the input conditions to be processed and the task identifier to be processed, and the output outputs the task processing result. In fintech business, the input conditions to be processed can be set as customer call voice, speaker reference voice, or business text prompts, and the task identifier to be processed can be set as transaction voice enhancement, speaker extraction, or voice verification related categories. The output data is the processing result corresponding to the task. In healthcare business, the input conditions to be processed can be set as consultation voice, rehabilitation training voice, or text description information, and the task identifier to be processed can be set as voice restoration, voice conversion, or multi-source voice separation categories. The output data is the audio processing result corresponding to the task. To adapt to inference efficiency requirements, the length of the prefix sequence of the conditions to be processed can be limited to 512, 1024, or 2048, the inference batch size can be set to 1, 4, or 8, and the output audio sampling rate can be set to 16000, 22050, or 44100.

[0071] In one implementation, the input conditions to be processed include audio to be processed and reference audio to be processed. The audio to be processed is processed through a conditional feature extraction structure to generate semantic features to be processed. The reference audio to be processed is processed through a reference feature extraction structure to generate speaker features to be processed. The task identifier to be processed is processed through an identifier mapping structure to generate task features to be processed. The semantic features to be processed and the speaker features to be processed are processed through feature mapping to form a sequence of conditional features to be processed. This sequence is then concatenated with the task features to be processed to form a prefix sequence of conditional features to be processed. The prefix sequence of conditional features to be processed is input into a temporal audio processing model, and the output is the target speaker extraction result. In another implementation, the input conditions to be processed include audio to be processed and text conditions to be processed. The text conditions to be processed are processed through a text encoding structure to generate text features to be processed. The text features to be processed and the semantic features corresponding to the audio to be processed jointly construct the sequence of conditional features to be processed. The task identifier to be processed corresponds to the speech conversion category, and the output is the converted audio. Alternatively, an input format containing only audio to be processed can be used. The audio to be processed is mapped to form a sequence of conditional features to be processed, and then combined with the task identifier to form a prefix sequence of conditional features to be processed. This is used for speech enhancement or sound source separation scenarios. In fintech applications, the input conditions can be set as customer transaction call audio and target speaker reference audio, the task identifier can be set as speaker extraction category, and the output is the cleaned-up voice of the target speaker. In healthcare applications, the input conditions can be set as user consultation voice and text description information, the task identifier can be set as voice repair category, and the output is the repaired voice content.

[0072] This embodiment constructs a condition prefix sequence based on the input conditions and the task identifier, and inputs the condition prefix sequence into the temporal audio processing model. This ensures that the input organization method in the inference stage is consistent with the representation rules in the training stage. Task category information and condition information can work together in a unified sequence to affect the model output, thereby making the task processing results more accurately respond to different input conditions and different task categories.

[0073] In one embodiment, step S10 above includes: S101, Obtain the target audio, the task identifier, and at least one input condition corresponding to the target audio; S102, perform self-supervised semantic extraction on the conditional audio contained in the at least one input condition to obtain semantic audio features; S103, Perform voiceprint feature extraction on the reference audio contained in the at least one input condition to obtain voiceprint reference features; S104, perform text encoding extraction on the text conditions contained in the at least one input condition to obtain text encoding features; S105, at least one of the semantic audio features, the voiceprint reference features, and the text encoding features is determined as a conditional feature; S106, Perform a dimension alignment operation on the conditional features to obtain aligned conditional features; S107, when there are multiple conditional features, the alignment conditional features are concatenated according to the input order to obtain a conditional feature sequence; S108, when there is only one condition feature, perform serialization processing on the alignment condition feature to obtain a condition feature sequence; S109, Perform an identifier mapping operation on the task identifier to obtain an initial task vector, and perform a dimension projection operation on the initial task vector to obtain task features.

[0074] In this embodiment, the target audio corresponds to the raw audio data that needs to enter the unified processing flow. The data source can be a real-time acquired speech stream, an offline stored recording file, or a historical speech segment from a business system. The task identifier corresponds to the category information of the current processing target and is used to distinguish the input organization method and subsequent constraint method under different processing directions. At least one input condition corresponding to the target audio is used to supplement the constraint information outside the target audio. The data form can be at least one of conditional audio, reference audio, or text conditions. There is a correspondence between the target audio and at least one input condition. This correspondence can be expressed as a temporal synchronization relationship, a business association relationship, or a target object pointing relationship. The acquisition stage not only completes data reading but also needs to complete format unification processing, including sampling rate unification, channel number unification, amplitude range normalization, and duration pruning or padding, so that data from different sources can enter the subsequent feature extraction process. The task identifier remains in discrete symbol form after acquisition and does not participate in audio modality calculation until the task feature generation process begins.

[0075] The conditional feature extraction process for each modality employs different feature extraction methods for different input conditions. The feature extraction process for conditional audio focuses on speech content-related representations. The self-supervised semantic extraction operation processes the conditional audio frame-by-frame using a time-series encoding structure to extract semantic audio features reflecting speech content, pronunciation patterns, and contextual dependencies. In the self-supervised semantic extraction operation, the input unit receives the conditional audio frame sequence, the encoding unit performs contextual aggregation on the frame-level representation, and the output unit forms a set of semantic audio features arranged along the time dimension. The source of semantic audio features is content information in the conditional audio, rather than speaker identity or spectral texture; therefore, semantic audio features serve as content constraints in subsequent processing. The feature extraction process for reference audio focuses on speaker attributes and timbre differences. The voiceprint feature extraction operation encodes the reference audio using a speaker representation structure to generate voiceprint reference features. Voiceprint reference features reflect stable speaker-related attributes, such as timbre contour, vocal style, and physiological differences, making them suitable as constraint information related to the target object in subsequent processing. The feature extraction process corresponding to text conditions focuses on the semantic representation of the text. The text encoding extraction operation converts discrete text fragments into continuous vectors through word vector mapping and context encoding structures, resulting in text encoding features. Text encoding features represent the semantic content, syntactic relations, and contextual combination information in the text conditions, and are used to supplement representation capabilities in scenarios where audio conditions are lacking or additional semantic constraints are required.

[0076] At least one of the semantic audio features, speaker reference features, and text-encoded features is identified as a conditional feature. This process is not a simple aggregation, but rather a selection and retention of effective features based on the actual composition of the current input conditions. When the input conditions contain only conditional audio, the conditional features consist solely of semantic audio features; when the input conditions also contain speaker reference audio, the conditional features consist of both semantic audio features and speaker reference features; when the input conditions include text conditions, text-encoded features participate in the conditional feature set. The process of determining conditional features clarifies the scope of effective information that needs to participate in the unified representation, avoiding features without an input source occupying the representation space.

[0077] Dimension alignment operations address the differences in vector dimension, temporal resolution, and representation distribution among different modal features. Semantic audio features, speaker reference features, and text-encoded features typically originate from different extraction structures, resulting in inconsistent vector lengths and permutation densities. Without dimensional alignment, these different modal features cannot be stably combined in a unified sequence space. Dimension alignment operations map conditional features to a unified representation dimension through linear transformations, nonlinear mappings, resampling, or length expansion. For semantic audio features arranged continuously along the temporal dimension, dimensional alignment maintains the temporal order and adjusts the vector length. For speaker reference features that are short or in single-vector form, dimensional alignment can generate representations compatible with other features through copy expansion, positional padding, or projection mapping. For text-encoded features, dimensional alignment uses vector projection and positional encoding to allow textual semantic information to enter a representation space consistent with the audio features. The result of dimensional alignment is aligned conditional features, which are unified in dimension, composable in structure, and can be incorporated into the same sequence representation.

[0078] The construction of conditional feature sequences is processed differently depending on the number of conditional features. When there are multiple conditional features, they are concatenated in the input order. The input order can be determined based on a predefined order of input sources or by prioritizing based on business meaning. Combination processing can place multiple aligned conditional features into the same sequence structure through vector-level concatenation, segmented concatenation, or sequential arrangement, preserving the source differences and arrangement relationships of information from different sources in a unified sequence. For example, semantic audio features can be located at the beginning, voiceprint reference features in the middle, and text encoding features at the end; the arrangement order can also be adjusted according to task requirements. The result of combination processing is a conditional feature sequence, which not only preserves the information of each modality but also the relative arrangement relationships between modalities. When there is only one conditional feature, the aligned conditional feature undergoes serialization processing. The role of serialization processing is to convert a single vector or a single feature set into a sequence structure that meets the requirements of subsequent inputs. If the aligned conditional feature itself is already a multi-vector set arranged in time, the serialization processing maintains the original order; if the aligned conditional feature is in single-vector form, it is formed into the smallest sequence unit through position expansion, repetition mapping, or length regularization. Regardless of whether there is one or more conditional features, the output is uniformly a sequence of conditional features, so that subsequent processing no longer depends on the difference in the number of input modalities.

[0079] After performing an identifier mapping operation, the task identifier yields an initial task vector. This initial task vector, after undergoing a dimensionality projection operation, yields task features. The identifier mapping operation converts discrete task categories into continuous vector representations. Mapping methods can include task dictionary lookup, embedded matrix indexing, or predefined vector table matching. The initial task vector reflects information about the task category itself, but its dimensionality may not match the conditional feature sequence; therefore, a dimensionality projection operation is necessary. This dimensionality projection operation uses a linear mapping structure or a multi-layer transformation structure to convert the initial task vector into a representation dimension compatible with the conditional feature sequence, forming the task features. The task features serve to express global constraint information; their significance lies not in providing audio content or speaker attributes, but in clarifying the task direction that subsequent processing should follow. The task feature generation process transforms the task identifier from a discrete category signal into a continuous representation that can participate in a unified sequence expression, thus forming the foundation for subsequent inputs together with the conditional feature sequence.

[0080] This embodiment acquires the target audio, task identifier, and at least one input condition in a unified manner. It then performs conditional feature extraction processing on the conditional audio, reference audio, and text conditions according to their respective modalities. Based on the extraction results, it completes conditional feature determination, dimension alignment, and conditional feature sequence construction. Simultaneously, it performs identifier mapping and dimension projection on the task identifier to obtain task features. This allows input information from different sources and with different structures to be converted into a unified, continuous representation. The conditional feature sequence retains the content information, speaker information, and textual semantic information of the input conditions, while the task features retain the task category information. Both types of results remain compatible in the representation space, thereby enhancing the consistent expressive ability of multiple input conditions within the same processing system and improving the uniformity of input organization under different task categories.

[0081] In one embodiment, step S20 above includes: S201, Perform a linear projection transformation operation on the conditional feature sequence to obtain an initial mapped feature sequence; S202, the initial mapped feature sequence is mapped to a preset embedding space, and feature dimension alignment is performed to obtain the mapped conditional feature sequence; S203, assign sequence positions to the task features and the mapped condition feature sequences respectively to obtain the corresponding arrangement positions; S204, based on the task arrangement position and the condition feature arrangement position, the mapped condition feature sequence and the task features are concatenated in chronological order to obtain a condition prefix sequence.

[0082] In this embodiment, the conditional feature sequence has already undergone cross-modal feature extraction and unified serialization representation in the previous stage. However, this sequence still retains the statistical distribution, dimensional structure, and modal origin differences of the original conditional features, and cannot be directly used as input for subsequent sequence generation structures. Therefore, feature mapping processing is required. The goal of feature mapping processing is to transform each feature unit in the conditional feature sequence into an embedding form consistent with the subsequent input representation space, and to achieve dimensional unification and distribution regularization without disrupting the original order relationship. The linear projection transformation operation undertakes the vector mapping function in this processing. The linear projection transformation operation receives each feature vector in the conditional feature sequence, performs matrix multiplication and bias superposition on each feature vector, projects the original features from the extraction space to the intermediate representation space, and outputs the initial mapped feature sequence. The initial mapped feature sequence still maintains the same arrangement order and sequence length as the conditional feature sequence, but the feature vector at each position has been transformed from the original modal representation into an intermediate result more suitable for unified embedding representation. The necessity of linear projection transformation is that even if the features formed after the extraction of different modal conditions have achieved preliminary dimensional unification, there may still be differences in numerical distribution, feature amplitude and semantic aggregation. Directly splicing them before and after the task features will cause an imbalance in the representation scale, affecting the subsequent sequence position encoding and context modeling effect. Therefore, it is necessary to compress, stretch or redistribute the conditional features into a controllable space through parameterized transformation.

[0083] After the initial mapped feature sequence is generated, it needs to be transformed into a pre-constructed input embedding space to perform feature dimension alignment. The pre-constructed input embedding space represents the vector space that receives input uniformly in the subsequent sequence generation structure. Each dimension in the space has a fixed length, fixed order, and fixed parameter constraints. The feature dimension alignment operation is not a simple length pruning, but rather a further regularization of the initial mapped feature sequence into a unified embedding interface, so that conditional features from different sources are expressed under the same vector dimension arrangement rules. This process can include dimension padding, dimension compression, normalization mapping, and embedding boundary constraints. For feature vectors with a length shorter than the dimension of the input embedding space, the representation dimension can be padded through projection expansion; for feature vectors with a length longer than the dimension of the input embedding space, the main expression components can be preserved through dimension reduction mapping; for cases where the distribution ranges of different features are inconsistent, modal differences can be flattened through normalization and scaling. After the feature dimension alignment operation, the mapped conditional feature sequence is obtained. The mapped conditional feature sequence is consistent with the initial mapped feature sequence in terms of temporal structure, but the feature units at each position are fully compatible with the subsequent input interface and can participate in the unified sequence construction without introducing an additional modality-specific transformation module.

[0084] Before participating in the unified sequence construction, task features and mapped conditional feature sequences need to be assigned their respective sequence positions. Task features express task category, task direction, or task constraint information; their role is not to provide local modal content but rather global control information. Therefore, their arrangement position in the conditional prefix sequence needs to be clearly defined. The sequence position allocation operation determines the task arrangement position for the task features and simultaneously determines the conditional feature arrangement position for each feature unit in the mapped conditional feature sequence. The task arrangement position can be set at the beginning of the conditional prefix sequence or at several fixed positions within the prefix local interval to ensure that the task information has a stable and identifiable relative position during subsequent sequence reading. The conditional feature arrangement positions are then unfolded sequentially according to the original order of the mapped conditional feature sequence and arranged either after or around the task arrangement position. The significance of the sequence position allocation operation is to transform vectors that originally only had feature value meaning into sequence units that simultaneously possess positional and content semantics. If the task arrangement position and condition feature arrangement position are missing, although the mapped condition feature sequence and task features can be numerically concatenated, the concatenated sequence cannot stably express the relative order between task information and condition information. When the subsequent model reads the data, it cannot distinguish the sequence roles of task control information and condition content information, thus weakening the constraint capability of the unified representation.

[0085] Based on the task and condition feature positions, the mapped condition feature sequence and task features are concatenated in chronological order to obtain the condition prefix sequence. This chronological order is not a direct copy of the original audio timeline, but rather an order organization rule within a unified sequence space. This rule ensures that the task features and the mapped condition feature sequence are arranged in the same sequence structure according to a predetermined pattern. The concatenation operation can employ a pre-insertion method, placing the task features at the beginning of the sequence; a segmented insertion method, where task features occupy several designated positions, with the mapped condition feature sequence distributed in subsequent positions; or, when stronger task control is required, repeated placement of task features, distributing multiple task feature units in the early region of the condition prefix sequence. Regardless of the arrangement method, the task and condition feature positions jointly determine the final condition prefix sequence structure. After the condition prefix sequence is formed, each position in the sequence simultaneously possesses three types of meaning: positional meaning, feature source meaning, and task constraint meaning. The introduction of task features gives the conditional prefix sequence a clear task control direction as a whole. The mapped conditional feature sequence ensures that the information of the conditional input is completely preserved, while the temporal sequence concatenation ensures that the two types of information can enter the subsequent processing structure in a unified order.

[0086] From a data structure perspective, a conditional prefix sequence is a collection of vector sequences with a unified dimension, order, and embedding interface. The beginning or designated positions of this sequence carry task features, while subsequent positions carry the mapped conditional feature sequence. Under different business conditions, the length of the conditional feature sequence and the arrangement of task features can vary, but after feature mapping, feature dimension alignment, and sequence position allocation, the final output conditional prefix sequence maintains consistent representation rules. For fintech businesses, the conditional feature sequence can originate from transaction voice, customer service voice, speaker reference voice, or business text prompts. After linear projection transformation and feature dimension alignment, a mapped conditional feature sequence is formed, which is then concatenated with task features representing task categories in chronological order to form a conditional prefix sequence with constraints for risk verification, target speaker extraction, or voice enhancement. For healthcare businesses, the conditional feature sequence can originate from consultation voice, rehabilitation training voice, target reference voice, or supplementary text descriptions. After the same processing, a mapped conditional feature sequence is formed, which is then concatenated with task features to form a conditional prefix sequence with constraints for voice restoration, semantic preservation, or multi-source speech separation. It is evident that the conditional prefix sequence is not a simple stacking of the original conditional features, but rather an ordered prefix input structure formed after unified representation space mapping, positional semantic allocation, and temporal order organization.

[0087] This embodiment transforms conditional information from different sources and with different statistical distributions into a unified input embedding space by performing linear projection transformation and feature dimension alignment operations on the conditional feature sequences. By assigning sequence positions to the task features and the mapped conditional feature sequences respectively, and concatenating them in chronological order to form a conditional prefix sequence, the task constraint information and conditional content information maintain a clear relative positional relationship and unified vector representation rules within the same sequence structure. The resulting conditional prefix sequence can simultaneously carry task category information and multimodal conditional information, improving the standardization of the unified input representation and enhancing the subsequent processing structure's ability to collaboratively perceive task direction and conditional content.

[0088] In one embodiment, step S30 above includes: S301, the target audio is input into the semantic coding branch of the dual-stream discrete codec network for semantic coding to obtain semantic continuous features; S302, Perform multi-level residual quantization on the semantic continuous features to obtain a semantic discrete label sequence; S303, The target audio is input into the acoustic coding branch contained in the dual-stream discrete codec network for acoustic coding to obtain acoustic continuous features; S304, Perform multi-layer residual quantization on the acoustic continuous features to obtain an acoustic discrete label sequence.

[0089] In this embodiment, after the target audio enters the dual-stream discrete codec network, it does not use a single path to complete representation compression, but instead enters the semantic coding branch and the acoustic coding branch separately. The semantic coding branch in the dual-stream discrete codec network is used to extract continuous representations related to speech content, focusing on the combination of articulatory units, contextual semantic relationships, sentence structure information, and content change trends in the audio. During semantic coding processing, the target audio is first converted into a temporally ordered frame-level representation, and then context aggregation is completed through multi-layer temporal coding units, so that the content association between adjacent time positions is absorbed into the continuous representation, forming semantically continuous features. The semantically continuous features are not a simple downsampling result of the original waveform, but an abstracted and compressed content representation. The temporal order is still preserved, but the feature values ​​at individual positions emphasize semantic adjacency and content separability. The semantic coding branch can be composed of convolutional downsampling units, sequence coding units, and feature compression units. The convolutional downsampling unit is responsible for reducing temporal resolution and expanding the local receptive range, the sequence coding unit is responsible for establishing content dependencies over a longer time span, and the feature compression unit is responsible for normalizing the high-dimensional hidden representation into a continuous vector suitable for quantization input.

[0090] After semantically continuous features are generated, multi-level residual quantization is performed. Multi-level residual quantization does not replace the continuous vector with a single discrete index all at once; instead, it decomposes the quantization process into multiple levels, each level processing the residual information not fully expressed in the previous level. The first quantization layer performs nearest-neighbor matching on the semantically continuous features, outputting the first-level discrete index and the first-level reconstructed vector. The residual calculation unit uses the difference between the semantically continuous features and the first-level reconstructed vector as the input to the next quantization layer. Subsequent quantization layers continue to discretely map the residuals until a predetermined number of layers is reached. The resulting set of discrete indices is expanded according to hierarchy and time position, forming a semantic discrete label sequence. Each discrete label in the semantic discrete label sequence corresponds to the semantic residual expression result at a certain level and time position; therefore, the sequence retains not only the temporal order but also hierarchical information. The significance of multi-level residual quantization is to avoid the problem of coarse semantic expression that occurs when using a single quantization unit, allowing content information to gradually approximate the original continuous features through hierarchical discrete representation.

[0091] The target audio is simultaneously input into the acoustic coding branch. The acoustic coding branch focuses on processing spectral envelope, harmonic structure, timbre differences, vocal quality, energy distribution, and fine-grained transient changes. During acoustic coding, local frequency components and short-time structures in the target audio are encoded using a multi-scale acoustic extraction unit. This unit can extract acoustic features at different time windows and frequency resolutions, preserving both short-time details and long-time stable components. After acoustic coding, continuous acoustic features are formed. These differ from semantic continuous features in their source; the former focuses on sound representation, while the latter focuses on speech content. Continuous acoustic features correspond to the original audio in the time dimension and emphasize timbre, intelligibility, noise patterns, and formant distribution in the feature dimension. The acoustic coding branch can consist of a short-time analysis unit, a spectral modeling unit, and a continuous representation compression unit. The short-time analysis unit separates local acoustic components, the spectral modeling unit integrates multi-band information, and the continuous representation compression unit generates acoustic continuous vectors suitable for quantization.

[0092] Acoustic continuous features also undergo multi-level residual quantization. This process is similar in logic to the quantization of semantic continuous features, but the quantization objective differs, focusing on expressing fine-grained acoustic differences using hierarchical discrete indexes. The first quantization layer performs discrete mapping on the acoustic continuous features, forming the first-level acoustic discrete index and providing the reconstruction result; the remaining differences enter the next quantization layer for further discrete mapping; after multi-level quantization, the acoustic discrete indexes at different levels are organized according to temporal and hierarchical order, forming an acoustic discrete label sequence. The discrete labels in the acoustic discrete label sequence correspond to a hierarchical compressed representation of acoustic details, with different levels of labels collectively describing the timbre, spectrum, and local acoustic details at the same time location. Using multi-level residual quantization instead of single-level discrete mapping improves the ability to restore acoustic details while maintaining the compactness of the discrete representation and reduces the information loss caused by single-level quantization.

[0093] Dual-stream discrete coding-decoding networks process the same target audio in parallel through semantic coding and acoustic coding branches, achieving separate representations of content and acoustic information. Semantic discrete tag sequences and acoustic discrete tag sequences are not two sets of mutually exclusive results, but rather two sets of discrete representations obtained from different representation dimensions for the same target audio. Both semantic and acoustic coding processes rely on the target audio, but their extraction of key points and subsequent quantization objectives differ, thus enabling the formation of content-oriented semantic discrete tag sequences and detail-oriented acoustic discrete tag sequences, respectively. For customer calls, agent scripts, transaction instruction voice, and risk verification voice in fintech businesses, semantic discrete tag sequences can carry transaction semantics, script structure, and question-and-answer content, while acoustic discrete tag sequences can carry speaker timbre, environmental noise, and call acoustic state. For health consultation voice, rehabilitation training voice, and follow-up oral voice in healthcare businesses, semantic discrete tag sequences can carry symptom descriptions, behavioral expressions, and consultation content, while acoustic discrete tag sequences can carry vocal intensity, breath variations, and sound quality details. Dual-stream discrete coding thus splits the same target audio into two types of discrete representations that can be used in parallel, providing input for subsequent unified sequence organization and reconstruction processing.

[0094] For example, the semantic discrete tag sequence representation is:

[0095] in, This represents a sequence of semantically discrete tokens. s represents a single semantically discrete token. t represents the time position index. l represents the quantization level index. This represents the semantic discrete label at the l-th quantization level and time position t.

[0096] The acoustic discrete label sequence representation is:

[0097] in, This represents a sequence of discrete acoustic markers. 'a' represents a single discrete acoustic marker. 't' represents the time position index. 'l' represents the quantization level index. This represents the acoustic discrete marker at the l-th quantization level and time position t.

[0098] This embodiment inputs the target audio into both a semantic coding branch and an acoustic coding branch, and performs multi-level residual quantization on both semantic and acoustic continuous features to form a semantic discrete label sequence and an acoustic discrete label sequence. This allows the content information and acoustic detail information in the audio to be separately entered into the discrete representation space. Semantic coding improves the stability of speech content representation, acoustic coding improves the integrity of timbre and spectral detail representation, and multi-level residual quantization enables the discrete mapping of continuous features to have hierarchical compensation capabilities, thereby enhancing the discrete representation's ability to simultaneously carry semantic and acoustic information.

[0099] In one embodiment, step S40 above includes: S401, based on the semantic discrete tag sequence and the acoustic discrete tag sequence, determine the hierarchical attribution relationship and time correspondence relationship corresponding to each semantic discrete tag and each acoustic discrete tag; S403, based on the hierarchical affiliation relationship, the semantic discrete tag sequence and the acoustic discrete tag sequence are respectively subjected to hierarchical splitting processing to obtain multiple semantic hierarchical tag subsequences and multiple acoustic hierarchical tag subsequences; S404, based on the time correspondence and the preset interleaving order, multiple semantic level marker subsequences and multiple acoustic level marker subsequences are hierarchically interleaved to obtain an interleaved marker sequence; S405, perform delayed arrangement processing on the discrete markers at different levels in the interleaved marker sequence, so that the discrete markers at different levels correspond to different arrangement positions, to obtain a delayed interleaved marker sequence; S406, fill the empty positions in the delayed interleaving marker sequence with placeholders, and perform sequential expansion processing on the delayed interleaving marker sequence after filling with placeholders to obtain the target marker sequence.

[0100] In this embodiment, the semantic discrete tag sequence and the acoustic discrete tag sequence correspond to the output results of the target audio in two types of discrete representation spaces, respectively. The former is used to express content-related information, while the latter is used to express timbre, spectrum, and fine-grained vocal information. Although both types of sequences maintain the original temporal order, each discrete tag already carries hierarchical information at the time of generation. The hierarchical attribution relationship is used to identify the attribution position of each semantic discrete tag and each acoustic discrete tag in the hierarchical discrete representation, and the temporal correspondence relationship is used to identify the corresponding position of each semantic discrete tag and each acoustic discrete tag on the original time axis. The hierarchical attribution relationship solves the problem of difficulty in distinguishing different hierarchical tags mixed in the same sequence, and the temporal correspondence relationship solves the problem of difficulty in aligning semantic tags and acoustic tags at the same temporal position. After both relationships are determined, the subsequent arrangement no longer depends on the original output order, but uses the hierarchical information and temporal information as a unified rearrangement basis.

[0101] Hierarchical decomposition is applied to both semantic and acoustic discrete token sequences, breaking down the two types of sequences, originally arranged continuously in time, into multiple semantic hierarchical token subsequences and multiple acoustic hierarchical token subsequences, respectively. During decomposition, each semantic discrete token is assigned to its corresponding semantic hierarchical token subsequence based on its hierarchical affiliation, and each acoustic discrete token is assigned to its corresponding acoustic hierarchical token subsequence based on its hierarchical affiliation. After this processing, the multi-level discrete tokens originally distributed in a single sequence are separated into their respective hierarchical sets, the mixed state between levels is broken up, and the order within a level is preserved. The formation of multiple semantic and acoustic hierarchical token subsequences allows information from different levels to be organized separately, and also ensures that discrete tokens within the same level have a consistent processing unit. This decomposition is not intended to increase the number of sequences, but rather to establish subsequent interleaving and delayed permutations on clear hierarchical boundaries, avoiding disordered overlap of tokens from different levels during subsequent permutations.

[0102] The preset interleaving order specifies the writing order of multiple semantic-level and acoustic-level marker subsequences within a unified sequence. This order can be set according to rules such as semantic priority, acoustic priority, alternation of semantic and acoustic elements, high-to-low hierarchy, or low-to-high hierarchy. Regardless of the rule used, temporal correspondence is the constraint. The hierarchical interleaving process does not simply concatenate the two types of subsequences end-to-end. Instead, it alternately writes discrete markers from different categories and levels at the same time position, creating a simultaneously readable arrangement of multiple semantic-level and acoustic-level marker subsequences within a unified sequence space. The result is an interleaved marker sequence. The positions in the interleaved marker sequence no longer correspond to a natural continuation of a single category of marker, but are instead a hybrid sequence reorganized according to temporal correspondence and the preset interleaving order. This sequence retains the original temporal correlation while explicitly carrying the alternation between semantic and acoustic markers within a single sequence.

[0103] Delayed permutation processing operates on discrete markers at different levels within an interleaved marker sequence. Discrete markers at different levels differ in their representational accuracy and informational value. If written synchronously, multi-level markers at the same time location are prone to positional conflicts, making it difficult to distinguish the origin of the level during subsequent readings. Delayed permutation processing applies different positional offsets to discrete markers at different levels, assigning them different permutation positions. Higher-level markers can maintain smaller offsets, while lower-level markers can be assigned larger offsets, or they can be reassigned according to preset rules. After this processing, multi-level markers that were originally overlapping at the same time location are expanded into multiple adjacent or spaced positions, forming a delayed interleaved marker sequence. Each discrete marker in the delayed interleaved marker sequence remains traceable to its original time location and level, but its position within the unified sequence has been redistributed according to level differences. This allows subsequent reading processes to simultaneously identify both temporal order and level differences.

[0104] After delayed permutation, empty permutation positions will appear in the delayed interleaving marker sequence. These empty positions do not carry valid discrete markers, but they play a necessary role in sequence length and positional relationships. If these empty positions are directly deleted, the relative positional relationships between markers at different time periods and levels will be compressed, and the hierarchical expansion effect brought by delayed permutation will be lost. Therefore, placeholder markers need to be filled into these empty positions. Placeholder markers are used to maintain the integrity of the sequence position; they do not carry semantic content expression or acoustic detail expression, but only to indicate that the position has a structural occupancy function. The delayed interleaving marker sequence after filling in placeholder markers is complete in length, continuous in position, and structurally directly readable by subsequent processing units. Sequential expansion processing is performed on this basis, expanding the delayed interleaving marker sequence after filling in placeholder markers according to the permutation positions to obtain the target marker sequence. The target marker sequence maintains a single sequence structure, but internally it has already encoded time correspondence, hierarchical affiliation, the interleaving relationship between semantic and acoustic categories, and the delay relationship corresponding to different levels. Therefore, the target marker sequence is no longer a simple concatenation of discrete markers, but a unified sequence expression after hierarchical splitting, interleaving and rearrangement, delay and misalignment, and placeholder padding.

[0105] From a sequence organization perspective, the target labeled sequence possesses three types of information. Temporal information determines the order of the labels, hierarchical information determines their relative unfolding positions, and category information determines whether the labels originate from a semantic discrete label sequence or an acoustic discrete label sequence. These three types of information are uniformly expressed within the same sequence, eliminating the need for subsequent sequence processing structures to separately parse the semantic and acoustic discrete label sequences at the input end. Instead, they directly read the target labeled sequence position by position. This processing compresses the discrete representations, originally scattered across two types of sequences and multiple levels, into a unified and schedulable sequence form, while preserving the original correspondences between labels and the rearranged structural relationships. For audio data with complex speech content, significant timbre variations, or high noise levels, this processing method enables discrete information from different sources to form more stable organizational rules within a unified sequence.

[0106] This embodiment determines the hierarchical and temporal relationships corresponding to each semantic and acoustic discrete tag, and then performs hierarchical splitting, hierarchical interleaving, delayed arrangement, filling of missing positions, and sequential expansion processing to rearrange the multi-level discrete information scattered in the semantic and acoustic discrete tag sequences into a unified target tag sequence. The target tag sequence simultaneously preserves temporal order, hierarchical differences, and category distinctions, reducing conflicts between tags of different levels at the same position and enhancing the collaborative organization capability of semantic and acoustic information in the unified sequence representation.

[0107] In one embodiment, step S60 above includes: S601, determine the semantic codebook partition corresponding to each semantic discrete tag according to the level to which each semantic discrete tag belongs in the semantic discrete tag sequence; S602, based on each semantic codebook partition, performs a tag lookup process on each semantic discrete tag to obtain multiple semantic tag vectors; S603, based on the arrangement position in the semantic discrete marker sequence, perform sequence recombination processing on multiple semantic marker vectors to obtain multiple semantic hierarchical marker vector sequences; S604, perform hierarchical combination processing on the multiple semantic hierarchical marker vector sequences to obtain a semantic vector sequence; S605, determine the acoustic codebook partition corresponding to each acoustic discrete symbol according to the level to which each acoustic discrete symbol belongs in the acoustic discrete symbol sequence; S606, based on each acoustic codebook partition, performs a tag lookup process on each acoustic discrete tag to obtain multiple acoustic tag vectors; S607, based on the arrangement position in the acoustic discrete marker sequence, perform sequence recombination processing on multiple acoustic marker vectors to obtain multiple acoustic level marker vector sequences; S608, perform hierarchical combination processing on the multiple acoustic hierarchical marker vector sequences to obtain an acoustic vector sequence.

[0108] In this embodiment, the semantic discrete tag sequence and the acoustic discrete tag sequence respectively carry content-related discrete representation and acoustic-related discrete representation. The two types of discrete tags do not exist solely as single numbers, but also possess hierarchical and positional attributes. The hierarchy to which each semantic discrete tag belongs reflects its position within the hierarchical quantization structure, with different levels corresponding to different granularities of semantic expression depth. Similarly, the hierarchy to which each acoustic discrete tag belongs reflects its position within the hierarchical quantization structure, with different levels corresponding to different granularities of timbre, spectrum, and local acoustic detail expression. Determining the semantic codebook partition corresponding to each semantic discrete tag based on its hierarchy in the semantic discrete tag sequence essentially remaps the semantic discrete tags from the original unified numbering space to a lookup table space with hierarchical constraints. The semantic codebook partition is used to limit the vector subspace in which each semantic discrete tag is indexed and matched, preventing mixed retrieval of semantic discrete tags from different levels within the same codebook region. Without distinguishing semantic codebook partitions, discrete tags from different levels may share the same vector region, causing high-level semantic generalization information and low-level semantic detail information to overlap in continuous representations. The determination of acoustic codebook partitions corresponds to this, with the aim of confining each discrete acoustic tag to a continuous representation subspace that conforms to its hierarchical attributes, so that the high-level acoustic contours and low-level acoustic details enter the corresponding vector retrieval areas respectively, thereby maintaining the independence of hierarchical acoustic information.

[0109] Based on each semantic codebook partition, a tag lookup process is performed on each semantic discrete tag to obtain multiple semantic tag vectors. This process is not a single mapping of the entire semantic discrete tag sequence, but rather an index lookup of each semantic discrete tag individually. Each semantic discrete tag establishes a one-to-one correspondence with a vector entity in its corresponding semantic codebook partition. During the lookup, the semantic discrete tag number is used as the index entry, and the vector unit in the semantic codebook partition is used as the output result. The multiple output semantic tag vectors retain the hierarchical and positional attributes of the original semantic discrete tags. The generation of multiple semantic tag vectors means that the discrete index is restored to a continuous vector representation, but at this point, the continuous vector is still in a discrete, scattered state and has not yet been restored to a semantic vector sequence organized hierarchically and positionally. The acoustic side processing is the same. Based on each acoustic codebook partition, a tag lookup process is performed on each acoustic discrete tag to obtain multiple acoustic tag vectors. These multiple acoustic tag vectors correspond to the continuous representation of each acoustic discrete tag at its respective hierarchy, maintaining the hierarchical and positional attributes unchanged. The purpose of this process is to remap discrete tags to vector representations that can participate in continuous spatial operations, while avoiding the loss of hierarchical distinction due to the sharing of a unified codebook for acoustic discrete tags at different levels.

[0110] Based on the arrangement positions in the semantic discrete tag sequence, multiple semantic tag vectors undergo sequence recombination to obtain multiple semantic hierarchical tag vector sequences. This process serves as a structure recovery function. Although the multiple semantic tag vectors have been obtained through table lookup, the lookup result is still a set of vectors output in the order of index reading, not yet restored to a sequence representation with complete hierarchical and temporal order. The arrangement positions in the semantic discrete tag sequence indicate the sequential position of each semantic discrete tag in the original semantic discrete tag sequence. Based on this arrangement position, the sequence recombination process rewrites the multiple semantic tag vectors into their corresponding levels and positions, forming multiple semantic hierarchical tag vector sequences. These multiple semantic hierarchical tag vector sequences are not simply sets of split vectors, but rather multiple sets of continuous representations that simultaneously satisfy the requirements of hierarchical affiliation and temporal order. Each set of semantic hierarchical tag vector sequences contains only semantic tag vectors of the same level, but the temporal arrangement relationship within the group is still preserved. The same sequence recombination process is performed on the acoustic side, restoring multiple acoustic tag vectors into multiple acoustic hierarchical tag vector sequences based on their arrangement positions in the acoustic discrete tag sequence. Through this process, the continuous vector representation originally scattered by discrete indices regains its temporal and hierarchical organization capabilities, providing the necessary structure for subsequent synthesis into a complete sequence.

[0111] A hierarchical combination process is performed on multiple semantic-level labeled vector sequences to obtain a semantic vector sequence. This step performs hierarchical aggregation. Multiple semantic-level labeled vector sequences represent semantic representations at different levels, with higher levels closer to overall content abstraction and lower levels closer to local semantic supplementation. The hierarchical combination process integrates the vector sequences at each level on a unified time coordinate or a unified sequence coordinate, transforming the originally dispersed semantic continuous representations at different levels into a single semantic vector sequence. The hierarchical combination process can include position-by-position concatenation, position-by-position weighted superposition, layer-by-layer mapping fusion, or sequential merging according to hierarchical order. Regardless of the implementation, the output is a unified semantic vector sequence. The vector at each position in the semantic vector sequence no longer corresponds to a single representation at a particular level, but rather integrates the available multi-level semantic expressions at that position, thus preserving both high-level semantic generalization and low-level semantic details. When performing hierarchical combination processing on multiple acoustic-level labeled vector sequences to obtain an acoustic vector sequence, the processing goal shifts to the unified recovery of acoustic representations at different levels. High-level acoustic representations are more focused on overall timbre and long-term contours, while low-level acoustic representations are more focused on detailed spectrum and local differences in sound production. Hierarchical combination processing restores multiple acoustic level marker vector sequences into a single acoustic vector sequence, enabling the acoustic vector sequence to have both overall acoustic consistency and retain local details.

[0112] This embodiment determines the semantic codebook partition corresponding to each semantic discrete tag based on its hierarchical level in the semantic discrete tag sequence, and determines the acoustic codebook partition corresponding to each acoustic discrete tag based on its hierarchical level in the acoustic discrete tag sequence. This allows discrete tags from different levels to enter their corresponding vector retrieval areas, reducing representation interference caused by hierarchical mixing. By performing tag lookup, sequence recombination, and hierarchical combination processing on each semantic and acoustic discrete tag respectively, the discrete index retains both hierarchical and arrangement information when recovering the continuous representation, ultimately forming a structurally complete semantic vector sequence and acoustic vector sequence. This processed continuous representation takes into account both high-level general information and low-level detailed information, improving the completeness of semantic and acoustic information recovery in continuous space.

[0113] In one embodiment, step S70 above includes: S701, establish a vector position correspondence based on the arrangement positions in the semantic vector sequence and the acoustic vector sequence; S702, based on the vector position correspondence, each semantic vector in the semantic vector sequence is aligned position by position with each acoustic vector in the acoustic vector sequence to obtain multiple aligned vector groups; S703, perform vector concatenation on each of the alignment vector groups to obtain multiple fused vectors, and obtain multiple fused vector units; S704, according to the arrangement order of the multiple alignment vector groups, perform sequential combination processing on the multiple fusion vector units to obtain the splicing and fusion result; S705, the splicing and fusion result is input into the decoding end of the dual-stream discrete codec network to reconstruct the continuous signal and obtain the reconstructed continuous signal representation; S706, Perform audio restoration processing on the reconstructed continuous signal representation to obtain the reconstructed audio.

[0114] In this embodiment, the semantic vector sequence and the acoustic vector sequence are derived from continuous recovery results of different discrete representations. The former is used to express speech content-related information, while the latter is used to express timbre, spectral morphology, vocalization state, and fine-grained acoustic differences. Although both types of sequences are represented as continuous vectors, they differ in their formation method, expressive emphasis, and internal density. Directly concatenating the two types of sequences can easily lead to misalignment of content and acoustic information at the same time point, resulting in a lack of correspondence in the fused input received by the subsequent decoding end at local positions. Establishing a vector position correspondence based on the arrangement positions in the semantic and acoustic vector sequences serves to determine the corresponding units of the two types of vectors in a unified temporal coordinate system. The arrangement positions are derived from the order information retained in the previous processing of the two types of sequences, and each semantic vector and each acoustic vector has a traceable position index. The vector position correspondence does not change the original arrangement of the two types of sequences, but rather clarifies which semantic vectors and which acoustic vectors should be classified into the same processing unit through position mapping, thereby providing a clear basis for subsequent positional alignment processing.

[0115] Based on the vector positional correspondence, each semantic vector in the semantic vector sequence is aligned position-by-position with each acoustic vector in the acoustic vector sequence, resulting in multiple aligned vector groups. Positional alignment does not mean compressing all vectors to the same length, but rather organizing them into multiple local units according to a one-to-one correspondence. Each aligned vector group contains at least one semantic vector and one acoustic vector, sharing the same or corresponding sequence positions. If the two types of sequences differ in the number of local positions, positional alignment can be achieved through position remapping, local padding, or neighboring position absorption. However, the output must still maintain that the semantic and acoustic expressions within each aligned vector group originate from the same or adjacent time regions. The formation of multiple aligned vector groups transforms the vector expressions originally scattered across two independent sequences into local fusion objects based on position. After this processing, subsequent splicing and fusion are no longer performed on two complete sequences, but rather on local units with clear positional correspondences, which helps control the fusion granularity and maintain temporal order stability.

[0116] Vector concatenation is performed on each of the multiple alignment vector groups to obtain multiple fused vector units. The concatenation occurs along the feature dimension, not the sequence dimension. For each alignment vector group, the content representation dimension of the semantic vector and the acoustic detail dimension of the acoustic vector are preserved and synthesized into a single fused vector unit through dimensional concatenation. If both the semantic and acoustic vector dimensions are set to 256, the dimension of each fused vector unit can be 512; if both are 512 and 256, the dimension can be 768. The purpose of vector concatenation is not simply to increase dimensions, but to allow semantic and acoustic information at the same location to coexist in a single continuous unit. Each fused vector unit contains both semantically relevant sub-vectors and acoustically relevant sub-vectors, preserving the differences between the two types of representations while establishing a common representation of both types at the same location. These multiple fused vector units thus form a unified, continuous input basis that can be directly read by the decoder.

[0117] Following the arrangement order of multiple alignment vector groups, a sequential combination process is performed on multiple fusion vector units to obtain the spliced ​​fusion result. Although the multiple fusion vector units have already completed the merging of semantic and acoustic information at local positions, they are still just a collection of scattered units. The sequential combination process reorganizes the multiple fusion vector units into a continuous sequence structure according to the arrangement order of the multiple alignment vector groups. The arrangement order maintains the original temporal progression relationship of the semantic vector sequence and the acoustic vector sequence, so that the spliced ​​fusion result maintains complete temporal continuity at the sequence level. Each sequence position in the spliced ​​fusion result corresponds to a fusion vector unit, and the preceding and following positions within the sequence still reflect the temporal order in the original audio. Through this process, the vector fusion result at local positions is expanded into a global sequence input result, and the subsequent decoding end can gradually recover the continuous signal under a unified temporal order.

[0118] The concatenated and fused result is input into the decoder of the two-stream discrete codec network for continuous signal reconstruction, resulting in a reconstructed continuous signal representation. The decoder no longer receives discrete indices or separated semantic and acoustic vectors; instead, it receives the concatenated and fused result that has already undergone position alignment and dimension concatenation. The decoder may contain a time-spreading unit, a continuous feature recovery unit, and a spectrum generation unit. The time-spreading unit expands the concatenated and fused result to a higher sampling density at a higher temporal resolution. The continuous feature recovery unit restores the fused vector units to a smoother, more continuous intermediate representation. The spectrum generation unit maps the continuous intermediate representation to a reconstructable audio spectrum representation. The continuous signal reconstruction process does not directly output the final audio; instead, it first obtains the reconstructed continuous signal representation. This reconstructed continuous signal representation is a continuous intermediate representation between the fused vector sequence and the final audio, containing recovered time-frequency structure information, energy distribution information, and local detail information. This representation serves as a transition from the fused features to the audio waveform, allowing the decoding process to unfold gradually in a continuous space.

[0119] Audio restoration processing is performed on the reconstructed continuous signal representation to obtain the reconstructed audio. Audio restoration transforms the reconstructed continuous signal representation into a playable, storable, and analyzable time-domain audio signal. If the reconstructed continuous signal representation exhibits a spectral structure, the audio restoration process includes spectrum-to-waveform transformation; if the reconstructed continuous signal representation exhibits high-dimensional time-domain features, the audio restoration process includes waveform restoration and amplitude shaping. After audio restoration, the reconstructed audio is output. The reconstructed audio is consistent with the target audio in terms of temporal continuity, reflects the semantic information preserved in the semantic vector sequence in terms of content expression, and reflects the timbre, spectrum, and local detail information preserved in the acoustic vector sequence in terms of sound performance. Thus, the semantic vector sequence and the acoustic vector sequence are transformed into a single continuous output result through positional correspondence, local splicing, sequential combination, continuous signal reconstruction, and audio restoration.

[0120] This embodiment establishes a vector position correspondence based on the arrangement positions in the semantic vector sequence and the acoustic vector sequence, and aligns each semantic vector in the semantic vector sequence with each acoustic vector in the acoustic vector sequence position by position, so that the continuous representations from two different sources form a clear correspondence at the same time position. By performing vector concatenation and sequential combination processing on multiple aligned vector groups, a concatenated and fused result containing both semantic and acoustic information is formed. The concatenated and fused result is then input into the decoding end of a dual-stream discrete codec network for continuous signal reconstruction and audio restoration processing, so that the output reconstructed audio retains both content expression and sound details. This improves the joint expression capability of semantic and acoustic information during continuous signal restoration and enhances the integrity of the reconstructed audio in terms of temporal order, content consistency, and timbre details.

[0121] In one embodiment, an audio processing apparatus based on dual-stream discrete representation is provided, which corresponds one-to-one with the audio processing method based on dual-stream discrete representation in the above embodiments. (Refer to...) Figure 3 , Figure 3 This is a schematic diagram of the functional modules of a preferred embodiment of the audio processing device based on two-stream discrete representation of the present invention. The modules include a conditional input processing module 10, a prefix construction module 20, a two-stream discrete encoding module 30, a tag arrangement module 40, a generation and training module 50, a vector recovery module 60, an audio reconstruction module 70, a network update module 80, a model assembly module 90, and a task inference module 100. Detailed descriptions of each functional module are as follows: The conditional input processing module 10 is used to acquire target audio, task identifier and at least one input condition corresponding to the target audio, perform conditional feature extraction processing on the at least one input condition in the corresponding modality, construct a conditional feature sequence based on the extracted conditional features, and perform task feature generation processing on the task identifier to obtain task features. Prefix construction module 20 is used to perform feature mapping processing on the conditional feature sequence and concatenate the mapped conditional feature sequence with the task features in chronological order to obtain a conditional prefix sequence. The dual-stream discrete coding module 30 is used to perform dual-stream discrete coding processing on the target audio using a dual-stream discrete coding and decoding network to obtain a semantic discrete tag sequence and an acoustic discrete tag sequence. The tag arrangement module 40 is used to perform hierarchical interleaving and delayed arrangement on the semantic discrete tag sequence and the acoustic discrete tag sequence to obtain the target tag sequence; The training module 50 is used to input the conditional prefix sequence into the autoregressive generator network and train the autoregressive generator network with the target label sequence as the supervision sequence to obtain the label generator network. The vector recovery module 60 is used to perform a tag lookup table process on the semantic discrete tag sequence and the acoustic discrete tag sequence respectively to obtain the semantic vector sequence and the acoustic vector sequence; The audio reconstruction module 70 is used to splice and fuse the semantic vector sequence and the acoustic vector sequence, and input the splicing and fusion result into the decoding end of the dual-stream discrete codec network to reconstruct the continuous signal and obtain the reconstructed audio. The network update module 80 is used to update the dual-stream discrete codec network based on the difference between the reconstructed audio and the target audio, so as to obtain the updated dual-stream discrete codec network. The model assembly module 90 is used to combine the tag generation network with the decoding end of the updated dual-stream discrete codec network to obtain a temporal audio processing model. The task reasoning module 100 is used to construct a condition prefix sequence based on the input conditions to be processed and the task identifier to be processed, and input the condition prefix sequence to be processed into the temporal audio processing model to obtain the task processing result.

[0122] Specific limitations regarding the audio processing device based on dual-stream discrete representation can be found in the foregoing limitations of the audio processing method based on dual-stream discrete representation, and will not be repeated here. Each module in the aforementioned audio processing device based on dual-stream discrete representation can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in hardware or independently of the processor in a computer device, or stored in software in the memory of a computer device, so that the processor can call and execute the operations corresponding to each module.

[0123] In one embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be as follows: Figure 4 As shown, the computer device includes a processor, memory, network interface, and database connected via a system bus. The processor provides deterministic and control capabilities. The memory includes non-volatile and / or volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and database. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage media. The network interface is used to communicate with external clients via a network connection. When executed by the processor, the computer program implements the functions or steps of a server-side audio processing method based on dual-stream discrete representation.

[0124] In one embodiment, a computer device is provided, which may be a client, and its internal structure diagram may be as follows: Figure 5 As shown, the computer device includes a processor, memory, network interface, display screen, and input devices connected via a system bus. The processor provides determination and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage media. The network interface is used to communicate with an external server via a network connection. When executed by the processor, the computer program implements client-side functions or steps of an audio processing method based on dual-stream discrete representation.

[0125] In one embodiment, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to perform the following steps: Obtain the target audio, the task identifier, and at least one input condition corresponding to the target audio; perform condition feature extraction processing on the at least one input condition in the corresponding modality; construct a condition feature sequence based on the extracted condition features; and perform task feature generation processing on the task identifier to obtain the task features. Perform feature mapping processing on the conditional feature sequence, and concatenate the mapped conditional feature sequence with the task features in chronological order to obtain a conditional prefix sequence; The target audio is subjected to dual-stream discrete coding processing using a dual-stream discrete coding and decoding network to obtain a semantic discrete tag sequence and an acoustic discrete tag sequence. The semantic discrete tag sequence and the acoustic discrete tag sequence are subjected to hierarchical interleaving and delayed arrangement to obtain the target tag sequence; The conditional prefix sequence is input into the autoregressive generator network, and the target label sequence is used as the supervision sequence to train the autoregressive generator network to obtain the label generator network. Perform a tag lookup process on the semantic discrete tag sequence and the acoustic discrete tag sequence respectively to obtain the semantic vector sequence and the acoustic vector sequence; The semantic vector sequence and the acoustic vector sequence are spliced ​​and fused, and the splicing and fusion result is input into the decoding end of the dual-stream discrete codec network for continuous signal reconstruction to obtain the reconstructed audio; Based on the difference between the reconstructed audio and the target audio, the dual-stream discrete codec network is updated to obtain the updated dual-stream discrete codec network. The tag generation network is combined with the decoding end of the updated dual-stream discrete codec network to obtain a temporal audio processing model; A condition prefix sequence is constructed based on the input conditions to be processed and the task identifier to be processed, and the condition prefix sequence is input into the temporal audio processing model to obtain the task processing result.

[0126] In one embodiment, a computer-readable storage medium is provided, which may be non-volatile or volatile, and a computer program is stored thereon, which, when executed by a processor, performs the following steps: Obtain the target audio, the task identifier, and at least one input condition corresponding to the target audio; perform condition feature extraction processing on the at least one input condition in the corresponding modality; construct a condition feature sequence based on the extracted condition features; and perform task feature generation processing on the task identifier to obtain the task features. Perform feature mapping processing on the conditional feature sequence, and concatenate the mapped conditional feature sequence with the task features in chronological order to obtain a conditional prefix sequence; The target audio is subjected to dual-stream discrete coding processing using a dual-stream discrete coding and decoding network to obtain a semantic discrete tag sequence and an acoustic discrete tag sequence. The semantic discrete tag sequence and the acoustic discrete tag sequence are subjected to hierarchical interleaving and delayed arrangement to obtain the target tag sequence; The conditional prefix sequence is input into the autoregressive generator network, and the target label sequence is used as the supervision sequence to train the autoregressive generator network to obtain the label generator network. Perform a tag lookup process on the semantic discrete tag sequence and the acoustic discrete tag sequence respectively to obtain the semantic vector sequence and the acoustic vector sequence; The semantic vector sequence and the acoustic vector sequence are spliced ​​and fused, and the splicing and fusion result is input into the decoding end of the dual-stream discrete codec network for continuous signal reconstruction to obtain the reconstructed audio; Based on the difference between the reconstructed audio and the target audio, the dual-stream discrete codec network is updated to obtain the updated dual-stream discrete codec network. The tag generation network is combined with the decoding end of the updated dual-stream discrete codec network to obtain a temporal audio processing model; A condition prefix sequence is constructed based on the input conditions to be processed and the task identifier to be processed, and the condition prefix sequence is input into the temporal audio processing model to obtain the task processing result.

[0127] It should be noted that the functions or steps that can be implemented by the computer-readable storage medium or computer device described above can be referred to the relevant descriptions on the server side and client side in the foregoing method embodiments. To avoid repetition, they will not be described one by one here.

[0128] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), Rambus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0129] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is used as an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above.

[0130] It should be noted that any software tools or components not belonging to this company appearing in the embodiments of this application are merely illustrative examples and do not represent actual use. All user personal information involved in the embodiments of this application has been authorized (with knowledge and consent) by the relevant parties or has been fully authorized by all parties, and the executing entity may obtain it through various legal and compliant means. The collection, storage, use, processing, transmission, provision, and disclosure of the information, data, and signals involved all comply with relevant laws and regulations and do not violate public order and good morals.

[0131] The above-described embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be included within the protection scope of the present invention.

Claims

1. A method of audio processing based on a dual-stream discrete representation, characterized in that, Includes the following steps: Obtain the target audio, the task identifier, and at least one input condition corresponding to the target audio; perform condition feature extraction processing on the at least one input condition in the corresponding modality; construct a condition feature sequence based on the extracted condition features; and perform task feature generation processing on the task identifier to obtain the task features. Perform feature mapping processing on the conditional feature sequence, and concatenate the mapped conditional feature sequence with the task features in chronological order to obtain a conditional prefix sequence; The target audio is subjected to dual-stream discrete coding processing using a dual-stream discrete coding and decoding network to obtain a semantic discrete tag sequence and an acoustic discrete tag sequence. The semantic discrete tag sequence and the acoustic discrete tag sequence are subjected to hierarchical interleaving and delayed arrangement to obtain the target tag sequence; The conditional prefix sequence is input into the autoregressive generator network, and the target label sequence is used as the supervision sequence to train the autoregressive generator network to obtain the label generator network. Perform a tag lookup process on the semantic discrete tag sequence and the acoustic discrete tag sequence respectively to obtain the semantic vector sequence and the acoustic vector sequence; The semantic vector sequence and the acoustic vector sequence are spliced ​​and fused, and the splicing and fusion result is input into the decoding end of the dual-stream discrete codec network for continuous signal reconstruction to obtain the reconstructed audio; Based on the difference between the reconstructed audio and the target audio, the dual-stream discrete codec network is updated to obtain the updated dual-stream discrete codec network. The tag generation network is combined with the decoding end of the updated dual-stream discrete codec network to obtain a temporal audio processing model; A condition prefix sequence is constructed based on the input conditions to be processed and the task identifier to be processed, and the condition prefix sequence is input into the temporal audio processing model to obtain the task processing result.

2. The method of audio processing based on dual-stream discrete representation as claimed in claim 1, wherein, The process involves acquiring target audio, a task identifier, and at least one input condition corresponding to the target audio; performing conditional feature extraction processing on the at least one input condition for the corresponding modality; constructing a conditional feature sequence based on the extracted conditional features; and performing task feature generation processing on the task identifier to obtain task features, including: Obtain the target audio, the task identifier, and at least one input condition corresponding to the target audio; Self-supervised semantic extraction is performed on the conditional audio contained in the at least one input condition to obtain semantic audio features; Voiceprint feature extraction is performed on the reference audio contained in the at least one input condition to obtain voiceprint reference features; Text encoding extraction is performed on the text conditions contained in the at least one input condition to obtain text encoding features; At least one of the semantic audio features, the voiceprint reference features, and the text encoding features is determined as a conditional feature; Perform a dimension alignment operation on the conditional features to obtain aligned conditional features; When there are multiple conditional features, the alignment conditional features are concatenated according to the input order to obtain a conditional feature sequence. When there is only one condition feature, the alignment condition feature is serialized to obtain a condition feature sequence; Perform an identifier mapping operation on the task identifier to obtain an initial task vector, and perform a dimension projection operation on the initial task vector to obtain task features.

3. The method of audio processing based on dual-stream discrete representation as claimed in claim 1, wherein, Perform feature mapping processing on the conditional feature sequence, and concatenate the mapped conditional feature sequence with the task features in chronological order to obtain a conditional prefix sequence, including: Perform a linear projection transformation operation on the conditional feature sequence to obtain an initial mapped feature sequence; The initial mapped feature sequence is mapped to a preset embedding space, and feature dimension alignment is performed to obtain the mapped conditional feature sequence; Assign sequence positions to the task features and the mapped condition feature sequences respectively to obtain the corresponding arrangement positions; Based on the task arrangement position and the condition feature arrangement position, the mapped condition feature sequence and the task features are concatenated in chronological order to obtain the condition prefix sequence.

4. The method of audio processing based on dual-stream discrete representation of claim 1, wherein, The target audio is subjected to dual-stream discrete coding processing using a dual-stream discrete codec network to obtain a semantic discrete tag sequence and an acoustic discrete tag sequence, including: The target audio is input into the semantic coding branch of the dual-stream discrete codec network for semantic coding to obtain semantically continuous features; Perform multi-level residual quantization on the semantically continuous features to obtain a semantically discrete label sequence; The target audio is input into the acoustic coding branch of the dual-stream discrete codec network for acoustic coding to obtain continuous acoustic features; Multi-level residual quantization is performed on the acoustic continuous features to obtain an acoustic discrete label sequence.

5. The method of audio processing based on dual-stream discrete representation of claim 1, wherein, The semantic discrete tag sequence and the acoustic discrete tag sequence are subjected to hierarchical interleaving and delayed permutation to obtain the target tag sequence, including: Based on the semantic discrete tag sequence and the acoustic discrete tag sequence, determine the hierarchical and temporal correspondences of each semantic discrete tag and each acoustic discrete tag; Based on the hierarchical affiliation, the semantic discrete tag sequence and the acoustic discrete tag sequence are respectively subjected to hierarchical splitting to obtain multiple semantic hierarchical tag subsequences and multiple acoustic hierarchical tag subsequences; Based on the time correspondence and the preset interleaving order, multiple semantic level marker subsequences and multiple acoustic level marker subsequences are hierarchically interleaved to obtain an interleaved marker sequence; The discrete markers at different levels in the interleaved marker sequence are subjected to delayed permutation processing so that the discrete markers at different levels correspond to different permutation positions, thus obtaining a delayed interleaved marker sequence; Placeholders are filled into the empty positions in the delayed interleaving marker sequence, and the delayed interleaving marker sequence after filling in the placeholders is subjected to sequential expansion processing to obtain the target marker sequence.

6. The method of audio processing based on dual-stream discrete representation of claim 1, wherein, Perform a tag lookup table process on the semantic discrete tag sequence and the acoustic discrete tag sequence respectively to obtain a semantic vector sequence and an acoustic vector sequence, including: Based on the level to which each semantic discrete symbol belongs in the semantic discrete symbol sequence, determine the semantic codebook partition corresponding to each semantic discrete symbol; Based on each semantic codebook partition, a tag lookup table is performed on each semantic discrete tag to obtain multiple semantic tag vectors; Based on the arrangement positions in the semantic discrete tag sequence, a sequence recombination process is performed on multiple semantic tag vectors to obtain multiple semantic hierarchical tag vector sequences; A hierarchical combination process is performed on multiple semantic hierarchical marker vector sequences to obtain a semantic vector sequence; Based on the level to which each acoustic discrete symbol belongs in the acoustic discrete symbol sequence, determine the acoustic codebook partition corresponding to each acoustic discrete symbol; Based on each acoustic codebook partition, a tag lookup table is performed on each acoustic discrete tag to obtain multiple acoustic tag vectors; Based on the arrangement positions in the acoustic discrete marker sequence, a sequence recombination process is performed on multiple acoustic marker vectors to obtain multiple acoustic hierarchical marker vector sequences; A hierarchical combination process is performed on multiple acoustic hierarchical marker vector sequences to obtain an acoustic vector sequence.

7. The dual-stream discrete representation based audio processing method of claim 1, wherein, The semantic vector sequence and the acoustic vector sequence are concatenated and fused, and the concatenation and fusion result is input into the decoding end of the dual-stream discrete codec network for continuous signal reconstruction to obtain reconstructed audio, including: Establish a vector position correspondence based on the arrangement positions in the semantic vector sequence and the acoustic vector sequence; Based on the vector position correspondence, each semantic vector in the semantic vector sequence is aligned position by position with each acoustic vector in the acoustic vector sequence to obtain multiple aligned vector groups; Multiple fused vectors are obtained by concatenating the alignment vector groups, resulting in multiple fused vector units. According to the arrangement order of the multiple alignment vector groups, the multiple fusion vector units are subjected to sequential combination processing to obtain the splicing and fusion result; The splicing and fusion result is input into the decoding end of the dual-stream discrete codec network to reconstruct the continuous signal and obtain the reconstructed continuous signal representation. The reconstructed continuous signal is subjected to audio restoration processing to obtain the reconstructed audio.

8. An apparatus for audio processing based on dual-stream discrete representation, characterized in that, The audio processing device based on dual-stream discrete representation includes: The conditional input processing module is used to acquire target audio, task identifier, and at least one input condition corresponding to the target audio; perform conditional feature extraction processing on the at least one input condition in the corresponding modality; construct a conditional feature sequence based on the extracted conditional features; and perform task feature generation processing on the task identifier to obtain task features. A prefix construction module is used to perform feature mapping processing on the conditional feature sequence and concatenate the mapped conditional feature sequence with the task features in chronological order to obtain a conditional prefix sequence. A dual-stream discrete coding module is used to perform dual-stream discrete coding processing on the target audio using a dual-stream discrete coding and decoding network to obtain a semantic discrete tag sequence and an acoustic discrete tag sequence. The tag arrangement module is used to perform hierarchical interleaving and delayed arrangement of the semantic discrete tag sequence and the acoustic discrete tag sequence to obtain the target tag sequence; A training module is used to input the conditional prefix sequence into the autoregressive generator network and train the autoregressive generator network using the target label sequence as a supervision sequence to obtain a label generator network. The vector recovery module is used to perform a tag lookup table process on the semantic discrete tag sequence and the acoustic discrete tag sequence respectively to obtain the semantic vector sequence and the acoustic vector sequence; The audio reconstruction module is used to splice and fuse the semantic vector sequence and the acoustic vector sequence, and input the splicing and fusion result into the decoding end of the dual-stream discrete codec network to reconstruct the continuous signal and obtain the reconstructed audio. The network update module is used to update the dual-stream discrete codec network based on the difference between the reconstructed audio and the target audio, so as to obtain the updated dual-stream discrete codec network. The model assembly module is used to combine the tag generation network with the decoding end of the updated dual-stream discrete codec network to obtain a temporal audio processing model; The task reasoning module is used to construct a condition prefix sequence based on the input conditions to be processed and the task identifier to be processed, and input the condition prefix sequence into the temporal audio processing model to obtain the task processing result.

9. A computer device, comprising: The computer device includes a memory, a processor, and an audio processing program based on a two-stream discrete representation stored in the memory and executable on the processor, wherein the audio processing program based on the two-stream discrete representation, when executed by the processor, implements the steps of the audio processing method based on a two-stream discrete representation as described in any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, The storage medium stores an audio processing program based on dual-stream discrete representation, which, when executed by a processor, implements the steps of the audio processing method based on dual-stream discrete representation as described in any one of claims 1-7.

Images