A method, system, terminal device, and medium for collaborative speech and music generation based on dynamic hybrid attention and expert architecture.

Abstract

This invention discloses a method, system, terminal device, and medium for collaborative speech and music generation based on a dynamic hybrid attention and expert architecture, relating to the field of audio generation technology. The method includes: acquiring multimodal input containing at least one of audio, text, and visual data; embedding the input and mapping it to a unified space to obtain a fusion sequence; setting several attention heads; dynamically selecting attention heads to activate experts through attention head routing gating; weighted aggregating the calculation results to obtain hybrid attention features; generating expert probability distributions through expert set routing gating; dynamically selecting expert sets to activate experts and calculating task-adaptive features; and performing audio generation processing based on an audio language modeling head to obtain speech or music results. This invention achieves collaborative speech and music generation, dynamically allocates computational resources, balances domain-specific and cross-domain general knowledge learning, resolves task conflicts and data imbalance problems, and improves the quality and efficiency of audio generation.

Description

Technical Field

[0001] This invention relates to the field of audio generation technology, and in particular to a method, system, terminal device, and medium for the collaborative generation of speech and music based on a dynamic hybrid attention and expert architecture. Background Technology

[0002] In recent years, generative artificial intelligence, represented by large-scale language models, has made revolutionary progress, driving the development of unified content generation models across multiple modalities such as text, images, and audio. In the auditory field, although speech synthesis and music generation have achieved significant results, they have mostly been developed in isolation within independent, dedicated models, hindering the advancement of general audio synthesis technology and limiting the model's ability to seamlessly perceive, reason, and create diverse audio content.

[0003] In existing technologies, speech synthesis models such as Step-audio treat speech modeling as a language modeling task, demonstrating outstanding performance in zero-shot speech cloning and controllable generation; music generation models such as MusicGen have verified the feasibility of generating high-fidelity music using autoregressive Transformers. However, unifying these two models faces two major challenges: firstly, task conflict—speech generation emphasizes semantic clarity and speaker identity restoration, while music generation focuses on artistic structures such as harmony and rhythm, leading to a trade-off in performance due to the inherent differences in optimization objectives; secondly, data imbalance—high-quality speech corpora far outnumber music corpora, making speech data prone to dominate learning in naive joint training, reducing the quality of music generation, as early attempts like UniAudio have been limited by this. Furthermore, the fixed Top-K routing strategy of traditional hybrid expert models ignores differences in lexical complexity, resulting in poor computational efficiency.

[0004] Therefore, there is an urgent need for a method that can dynamically allocate computing resources, decouple domain and cross-domain knowledge learning, and adapt to imbalanced data training, in order to fill the gaps in existing technologies. Summary of the Invention

[0005] The technical problem this invention aims to solve lies in the fact that, in the field of audio generation technology, existing technologies develop speech generation and music generation separately. When modeling them in a unified manner, they face the following challenges: speech generation emphasizes semantic clarity and identity restoration, while music generation focuses on artistic structure. This difference in optimization objectives leads to a performance conflict, as does the fact that speech corpora are far more numerous than music corpora, making it easier for dominant learning to reduce the quality of music generation. Furthermore, traditional hybrid expert fixed routing strategies suffer from data imbalance due to computational inefficiency. Therefore, an effective solution is urgently needed to address these technical problems.

[0006] To solve the above-mentioned technical problems, the technical solution adopted by the present invention is as follows: In a first aspect, the present invention provides a method for co-generating speech and music based on a dynamically hybrid attention and expert architecture, the method comprising: Acquire multimodal input, wherein the multimodal input includes at least one of audio input, text input, and visual input; The multimodal input is processed by embedding representation, converting audio input into discrete acoustic word sequences, text input into text embedding vectors, and visual input into soft visual tags, which are then mapped to a unified embedding space to obtain a fused sequence. Several attention heads are set up. Each attention head is dynamically selected as an attention head activation expert for the fusion sequence through attention head routing gating. Based on the attention calculation results of the attention head activation experts and the gating probability of the attention head routing gating, the hybrid attention features are calculated. By using a pre-built expert set routing gating, an expert probability distribution of the expert set is generated. Based on the expert probability distribution of the expert set, a dynamic routing strategy is used to select experts to activate and calculate task adaptation features. The expert set contains several experts with different functions, and the expert set includes at least experts who learn domain-specific knowledge and experts who learn cross-domain general knowledge. Based on the audio language modeling head, the task adaptation features are processed to generate audio, resulting in speech or music.

[0007] In one implementation, the embedding representation processing of the multimodal input, which converts audio input into a discrete acoustic lexical sequence, text input into a text embedding vector, and visual input into soft visual tags and maps them to a unified embedding space to obtain a fused sequence, includes: The audio waveform of the audio input is encoded using a neural audio codec, and cascaded prediction is performed through a delay mechanism to obtain a multi-channel discrete acoustic word sequence. The text input is segmented using a word segmenter and then converted into text embedding vectors in a unified embedding space through an embedding mapping layer. Visual input is encoded using a visual Transformer to obtain block features, which are then mapped to soft visual labels in a unified embedding space using a projector. The multi-channel discrete acoustic lexical sequence, the text embedding vector, and the soft visual markers are fused to obtain a fused sequence.

[0008] In one implementation, a plurality of attention heads are set, each attention head dynamically selects an attention head activation expert for the fused sequence through an attention head routing gating, and calculates hybrid attention features based on the attention calculation results of the attention head activation experts and the gating probability of the attention head routing gating, including: Several attention heads are set up, and each attention head is configured with an independent pool of attention head candidate experts. The attention head routing gate corresponds one-to-one with the attention head candidate expert pool. For each attention head, based on the fusion sequence, the expert probability distribution of the candidate expert pool of the attention head is calculated using attention head routing gating, and a TopN routing strategy is adopted to select a preset number of attention heads to activate experts; For each attention head, the parameter matrix of the selected attention head activation expert is invoked to perform attention calculation. The calculation result is multiplied element by element by the gating probability of the corresponding attention head route gating and then summed to obtain the output of a single attention head. The outputs of all attention heads are aggregated to obtain the hybrid attention features.

[0009] In one implementation, the step of generating an expert probability distribution for an expert set through a pre-built expert set routing gating, and selecting activated experts using a dynamic routing strategy based on the expert probability distribution of the expert set, and calculating task adaptation features, includes: The hybrid attention features are processed word-by-word using expert set routing gating, and the expert probability distribution of each word in the expert set is output. Using the Top-P dynamic routing strategy, the probability distribution of experts in the expert set is sorted and accumulated from high to low probability value until the accumulated probability reaches a preset threshold. Experts whose accumulated probability reaches the preset threshold are selected as active experts in the expert set. Based on the expert set activation experts, the hybrid attention features are calculated to obtain the output results of all the expert set activation experts; The outputs of all activated experts in the expert set are weighted and summed using normalized gating probabilities to obtain task adaptation features.

[0010] In one implementation, the expert set includes several functionally differentiated experts, including: Routing experts are used to capture domain-specific knowledge specific to voice or music; Shared experts are used to learn cross-domain general knowledge across speech and music; Empty expert, used to perform skip computation on simple lexical units.

[0011] In one implementation, the training steps for the attention head, attention head routing gating, expert set, and expert set routing gating in the audio model used to implement the generation method include: Dense models are trained on both speech-specific and music-specific datasets to obtain domain prototype experts who possess knowledge of the corresponding domains. The speech-specific dataset and the music-specific dataset are subjected to balanced sampling to obtain a balanced sampled dataset. The domain prototype experts are integrated into the expert set, and a parameter adjustment strategy is used on the balanced sampling dataset to train the expert set routing gate and the experts in the expert set who are used to learn cross-domain general knowledge. Joint fine-tuning training is performed on the parameters of the attention head, the attention head routing gating, the expert set, and the expert set routing gating.

[0012] In one implementation, the parameter adjustment strategy and joint fine-tuning training include: Freeze the parameters of the domain prototype experts and update only the parameters of the expert set routing gate and the experts in the expert set used to learn cross-domain general knowledge; Unfreeze the frozen parameters, and on the balanced sampling dataset, guide experts to make balanced use of the combined optimization objective of load balancing loss and main task loss, and jointly fine-tune and optimize the model training effect.

[0013] Secondly, embodiments of the present invention also provide a speech and music co-generation system based on a dynamically hybrid attention and expert architecture, the system comprising: An input acquisition module is used to acquire multimodal input, wherein the multimodal input includes at least one of audio input, text input, and visual input; The fusion sequence acquisition module is used to perform embedding representation processing on the multimodal input, converting audio input into discrete acoustic word sequences, text input into text embedding vectors, and visual input into soft visual tags, and mapping them to a unified embedding space to obtain a fusion sequence; The hybrid attention feature acquisition module is used to set up several attention heads, each attention head dynamically selects an attention head activation expert for the fusion sequence through attention head routing gating, and calculates hybrid attention features based on the attention calculation results of the attention head activation experts and the gating probability of the attention head routing gating. The task adaptation feature acquisition module is used to generate the expert probability distribution of the expert set through a pre-built expert set routing gating, and to select experts to activate the expert set using a dynamic routing strategy based on the expert probability distribution of the expert set, and to calculate the task adaptation features. The expert set includes several experts with different functions, and the expert set includes at least experts who have learned domain-specific knowledge and experts who have learned cross-domain general knowledge. The result audio acquisition module is used to perform audio generation processing on the task adaptation features to obtain the result audio of speech or music.

[0014] Thirdly, embodiments of the present invention also provide a terminal device, the terminal device including a memory, a processor, and a speech and music co-generation program based on dynamic hybrid attention and expert architecture stored in the memory and executable on the processor, wherein when the processor executes the speech and music co-generation program based on dynamic hybrid attention and expert architecture, it implements the steps of the speech and music co-generation method based on dynamic hybrid attention and expert architecture as described in any of the above schemes.

[0015] Fourthly, embodiments of the present invention also provide a computer-readable storage medium storing a speech and music co-generation program based on a dynamically hybrid attention and expert architecture. When the speech and music co-generation program based on a dynamically hybrid attention and expert architecture is executed by a processor, it implements the steps of the speech and music co-generation method based on a dynamically hybrid attention and expert architecture as described in any of the above schemes.

[0016] Beneficial Effects: A method, system, terminal device, and medium for collaborative speech and music generation based on a dynamic hybrid attention and expert architecture, relating to the field of audio generation technology, are presented. The method first acquires multimodal input, including at least one of audio input, text input, and visual input. Subsequently, the multimodal input undergoes embedding representation processing, converting audio input into a discrete acoustic lexical sequence, text input into a text embedding vector, and visual input into soft visual tags, which are then mapped to a unified embedding space to obtain a fused sequence. Subsequently, several attention heads are set, each dynamically selecting an attention head activation expert for the fused sequence through attention head routing gating. Based on the attention calculation results of the attention head activation experts and the gating probability of the attention head routing gating, hybrid attention features are calculated. Next, an expert probability distribution is generated using a pre-constructed expert set routing gating system. Based on this distribution, a dynamic routing strategy is used to select and activate experts within the expert set, calculating task adaptation features. The expert set includes several functionally differentiated experts, at least those learning domain-specific knowledge and those learning cross-domain general knowledge. Finally, based on an audio language modeling head, the task adaptation features are processed to generate audio output for speech or music. This invention effectively decouples domain-specific knowledge and cross-domain general knowledge in speech and music by using a unified embedding mapping for multimodal inputs, a hybrid attention mechanism combining dynamic expert selection via an attention head routing gating system, and a design that dynamically selects functionally differentiated experts using an expert set routing gating system. This alleviates task conflicts in the unified generation of both. Furthermore, the dynamic routing strategy allocates computational resources based on lexical complexity, improving resource utilization and efficiency, ultimately achieving high-quality collaborative generation of speech and music. Attached Figure Description

[0017] Figure 1 A flowchart illustrating a specific implementation of the speech and music co-generation method based on dynamic hybrid attention and expert architecture provided in this invention.

[0018] Figure 2 This is a flowchart and architecture diagram of the speech and music co-generation method based on dynamic hybrid attention and expert architecture provided in the embodiments of the present invention.

[0019] Figure 3 This is a schematic diagram of the training process in the speech and music co-generation method based on dynamic hybrid attention and expert architecture provided in the embodiments of the present invention.

[0020] Figure 4 This is a schematic diagram of the speech and music co-generation device based on dynamic hybrid attention and expert architecture provided in an embodiment of the present invention.

[0021] Figure 5 This is a block diagram illustrating the internal structure of the terminal device provided in an embodiment of the present invention. Detailed Implementation

[0022] To make the objectives, technical solutions, and effects of this invention clearer and more explicit, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.

[0023] The flowchart shown in the attached diagram is for illustrative purposes only and does not necessarily include all content, operations, or steps, nor does it require execution in the described order. For example, some operations or steps can be broken down, combined, or partially merged, so the actual execution order may change depending on the actual situation.

[0024] It should be understood that the terminology used in this specification is for the purpose of describing particular embodiments only and is not intended to limit the invention. As used in this specification and the appended claims, the singular forms “a,” “an,” and “the” are intended to include the plural forms unless the context clearly indicates otherwise.

[0025] It should be understood that, in order to clearly describe the technical solutions of the embodiments of the present invention, the terms "first" and "second" are used in the embodiments of the present invention to distinguish identical or similar items with essentially the same function and effect. For example, "first control information" and "second control information" are only used to distinguish different control information and do not limit their order.

[0026] Those skilled in the art will understand that the words "first" and "second" do not limit the quantity or the order of execution, and that the words "first" and "second" do not necessarily imply that they are different.

[0027] It should also be understood that the term “and / or” as used in this specification and the appended claims refers to any combination of one or more of the associated listed items and all possible combinations, and includes such combinations.

[0028] In recent years, generative artificial intelligence, represented by large-scale language models, has made revolutionary progress, driving the development of unified content generation models across multiple modalities such as text, images, and audio. In the auditory domain, while significant achievements have been made in speech synthesis and music generation, these two tasks are typically developed in isolation within dedicated models, hindering the advancement of general-purpose audio synthesis techniques. This independent development limits the models' ability to seamlessly perceive, reason, and create diverse audio content like humans.

[0029] Most existing audio generation technologies focus on a single domain. For example, in speech synthesis, models like Step-audio have demonstrated unprecedented capabilities in zero-shot speech cloning and controlled speech generation by modeling speech as a language modeling task; while in music generation, models like MusicGen have verified the feasibility of generating high-fidelity music using autoregressive Transformers. However, unifying speech and music generation into a single model still faces two major challenges that current technologies have not yet effectively addressed. The first is the task conflict between speech and music generation. Speech generation emphasizes semantic clarity and speaker identification, while music generation focuses on complex artistic structures such as harmony and rhythm. These two optimization objectives are inherently different, and when forced into the same model, conflicting optimization goals often arise, causing the model to sacrifice the performance of one task while improving the performance of the other. The second is data imbalance. Currently, high-quality, large-scale speech corpora are far more abundant than music corpora. Under a naive joint training framework, speech data, due to its quantitative advantage, dominates the model's learning process, leading to a significant decline in the quality of music generation. For example, early attempts like UniAudio were limited in performance on data-scarce tasks such as music due to data imbalance. Furthermore, while traditional hybrid expert models offer potential architectural solutions for mitigating multimodal conflicts, their fixed Top-K routing strategy is computationally inefficient. This method assigns a fixed number of experts to each token, ignoring the differences in complexity among different tokens and thus limiting the model's computational efficiency.

[0030] Therefore, this invention addresses the problems of task conflict and data imbalance in existing technologies for collaborative speech and music generation by proposing a speech and music collaborative generation method based on dynamic hybrid attention and expert architecture.

[0031] At the model structure level, this invention proposes a unified hybrid expert audio model, such as... Figure 1 As shown, the model takes multimodal inputs such as text, sound, and video as inputs, and outputs audio outputs such as music generation, speech synthesis, and timbre cloning.

[0032] Specifically, a dynamic capacity hybrid expert architecture is introduced into this model, such as... Figure 2 The hybrid expert module is shown in the diagram, and this invention applies this hybrid expert architecture to a feedforward network. Unlike traditional hybrid expert architectures with a fixed number of activated experts, this architecture can dynamically adjust the number of activated experts based on the complexity of different labels, thereby improving computational efficiency while ensuring generation quality. Furthermore, by introducing multiple types of experts, the model can effectively decouple domain-specific feature learning computation from cross-domain general feature learning, and supports adaptive skipping of redundant labels, further improving resource utilization.

[0033] Secondly, at the feature modeling level, this invention proposes a hybrid attention mechanism, such as... Figure 2 As shown in the hybrid attention module, the idea of ​​dynamically activating the parameters of hybrid experts is transferred to the attention module. By dynamically assigning activation parameters to each attention head and keeping the candidate expert pool of each attention head independent, the mixing of experts from multiple attention heads is avoided. Through the hybrid attention module, the dynamic activation of all major modules of the entire model is achieved, improving the model's state space and thus further enhancing the model's potential.

[0034] Finally, at the training level, this invention designs a phased training strategy, such as... Figure 3 As shown in the training process diagram, this strategy uses a process of gradually imparting domain knowledge, integration and warm-up, and joint collaborative optimization to enable the model to focus on domain feature learning, cross-domain fusion and overall performance improvement at different stages, thereby achieving stable training and effective collaborative learning even under imbalanced data conditions.

[0035] This embodiment provides a method for co-generating speech and music based on a dynamic hybrid attention and expert architecture, such as... Figure 1 As shown, the specific steps include the following: Step S100: Obtain multimodal input, wherein the multimodal input includes at least one of audio input, text input and visual input.

[0036] In this embodiment, the multimodal input covers typical scenarios of speech and music generation. Specifically, audio input can be raw audio signals such as speech waveforms to be continued or music clips; text input can be instructional text such as speech synthesis scripts or style descriptions for music generation; and visual input can be visual information such as video frames containing the speaker's lip movements or scene images corresponding to the music. These multimodal inputs will be converted into... Figure 2 The audio tokens, text tokens, and visual tokens shown provide the raw input data for subsequent unified embedding processing. The purpose of using multimodal input is to broaden the adaptability of the generation scenario through multi-source information collection, and to provide a data foundation for cross-modal fusion and unified generation.

[0037] Step S200: Perform embedding representation processing on the multimodal input, convert the audio input into a discrete acoustic lexical sequence, convert the text input into a text embedding vector, convert the visual input into soft visual tags, and map them to a unified embedding space to obtain a fused sequence.

[0038] In this embodiment, heterogeneous input unification processing is performed on multimodal inputs. For audio, text, and visual inputs, embedding methods adapted to their data characteristics are employed respectively. Specifically, continuous audio waveforms are converted into discrete acoustic word sequences, text is converted into semantic embedding vectors, and visual information is converted into soft visual tags. Finally, these are converged into the same embedding space through dimensional mapping to form... Figure 2 The unified representation before fusion, i.e., the fusion sequence, shown in the diagram, is used for subsequent data processing. By unifying the processing of multimodal inputs, the problem of multimodal data format differences is solved, providing a common input basis for subsequent hybrid attention and hybrid expert architecture processing, and realizing the synergy of cross-modal information.

[0039] In one implementation, the embedding representation processing of the multimodal input, which converts audio input into a discrete acoustic word sequence, text input into a text embedding vector, and visual input into soft visual tags and maps them to a unified embedding space to obtain a fused sequence, specifically includes the following steps: Step S210: Encode the audio waveform of the audio input using a neural audio codec, and perform cascaded prediction through a delay mechanism to obtain a multi-channel discrete acoustic word sequence; Step S220: Use a word segmenter to segment the text input and convert it into a text embedding vector in a unified embedding space through an embedding mapping layer; Step S230: Encode the visual input using a visual Transformer to obtain block features, and map them to soft visual labels in a unified embedding space using a projector. Step S240: Fuse the multi-channel discrete acoustic lexical sequence, the text embedding vector, and the soft visual marker to obtain a fused sequence.

[0040] In this embodiment, for audio input, a neural audio codec is used to convert the continuous audio waveform into a discrete sequence of acoustic tokens across multiple channels. Specifically, a DAC (High-Fidelity Audio Compression with Improved RVQGAN) is used to encode 16kHz audio into 12 parallel channels at an 8kHz ratio.

[0041] For an audio segment lasting T time steps, the structure of its output acoustic lexical sequence is as follows: ,in It is the first The time step to the first Discrete lexical indices for each channel. Unlike channel-by-channel prediction and fully synchronous independent prediction methods, this embodiment uses a delay mechanism for cascaded prediction to better model the complex dependencies between channels and improve audio quality. This method introduces a specific time-step delay for each channel, and its pattern can be represented by a delay vector. To define precisely, where Representing the The generation delay of each channel relative to the initial time step.

[0042] In actual generation, the model does not output all 12 channels of tokens at once, but follows the order defined by the delay vectors. First, it generates tokens for channels with a delay of 0, then uses this new information to generate tokens for channels with a delay of 1, and so on, forming a cascaded prediction stream. In this example, a sequential delay pattern is used. This approach allows the prediction of each channel to be conditioned on the newly generated lexical units of the preceding channels at the same time step, thereby significantly improving the coherence and fidelity of the generated audio.

[0043] For text input, this embodiment first uses the Qwen2.5vl word segmenter to convert it into a text word sequence, and then maps it into a 2048-dimensional vector through an embedding mapping layer.

[0044] For visual inputs such as video, the same visual Transformer as the Qwen-VL method is used to segment the image into 14x14 pixel blocks and encode them into block features. Then, the visual feature dimensions are mapped from 1280 dimensions to an embedding space consistent with the 2048 dimensions of the model text embedding through a projector, generating soft visual tags that can be fused with text and audio representations.

[0045] Finally, the generated multi-channel discrete acoustic lexical sequence, text embedding vector, and soft visual markers are fused to obtain a fused sequence for subsequent use.

[0046] Step S300: Set up several attention heads. Each attention head dynamically selects an attention head activation expert for the fusion sequence through attention head routing gating. Based on the attention calculation results of the attention head activation experts and the gating probability of the attention head routing gating, calculate the hybrid attention features.

[0047] In this embodiment, a traditional attention layer with fixed parameters is replaced by configuring dedicated routing gating and expert resources for several independent attention heads. Each attention head dynamically selects and activates experts from a dedicated pool of candidate experts through attention head routing gating, and then aggregates the calculation results using gating probability weighting. Figure 2 The hybrid attention module addresses the limitations of attention head parameter sharing, enhances the model's ability to differentiate features across speech and music modalities, and provides a foundation for resolving cross-task feature conflicts.

[0048] In one implementation, a plurality of attention heads are set, and each attention head dynamically selects an attention head activation expert for the fused sequence through an attention head routing gating. Based on the attention calculation results of the attention head activation experts and the gating probability of the attention head routing gating, a hybrid attention feature is calculated. Specifically, this includes the following steps: Step S310: Set up several attention heads, each attention head is configured with an independent attention head candidate expert pool, wherein the attention head routing gate corresponds one-to-one with the attention head candidate expert pool; Step S320: For each attention head, based on the fusion sequence, use attention head routing gating to calculate the expert probability distribution of the attention head candidate expert pool, and adopt the TopN routing strategy to select a preset number of attention heads to activate experts. Step S330: For each attention head, call the parameter matrix of the selected attention head activation expert to perform attention calculation, multiply the calculation result element by element with the gating probability of the corresponding attention head route gating and sum them to obtain the output of a single attention head; Step S340: Aggregate the outputs of all attention heads to obtain hybrid attention features.

[0049] In this embodiment, a hybrid attention layer is used to replace the traditional standard attention layer in the model. The core of this replacement step is that a dynamic activation strategy will be adopted for the parameters of each attention head.

[0050] First, assume there are a total of The first attention point, for the first Each attention point will be selected and activated by the expert predicted through the Top2 routing gating network. ,in It is a gating matrix. The input tensor is used. In this embodiment, the model has a total of 36 layers, with 16 attention heads per layer and a total hidden layer dimension of 2048, therefore each head has a dimension of 128.

[0051] Subsequently, the two expert indices with the highest probabilities are assigned computation matrices to each attention head. Standard attention calculations are performed independently within each attention head, and the probabilities calculated by the gating network are used as weights. The specific expression is as follows:

[0052] in, , Indicates the first The output of each attention head, Indicates input The first one calculated by the gating network The weight of each expert, Indicates the first The parameter matrices of the nth expert are respectively the nth expert's parameter matrix. The weight matrix of Q, K, and V corresponding to each expert. , This indicates element-wise multiplication of matrices.

[0053] This method achieves dynamic activation of attention module parameters while ensuring that each attention head does not interfere with others, increasing the number of activated states in the model. Ultimately, this allows the model to simultaneously and dynamically activate the feed-forward network (FFN) and attention layers for different tasks in speech and music generation, thus resolving the joint training conflict caused by speech and music generation tasks and improving the model's efficiency and accuracy. In other words, as... Figure 2 The hybrid attention mechanism shown improves the flexibility of model parameter activation and enhances the overall multi-task feature modeling capability by adapting the attention parameters to the dynamic activation method.

[0054] Step S400: Generate the expert probability distribution of the expert set through the pre-built expert set routing gating, and select experts to activate the expert set using a dynamic routing strategy based on the expert probability distribution of the expert set, and calculate the task adaptation features. The expert set contains several experts with different functions, and the expert set includes at least experts who learn domain-specific knowledge and experts who learn cross-domain general knowledge.

[0055] In this embodiment, as Figure 1As shown in the hybrid expert module, hybrid attention features are processed through a pre-built expert set routing gating system. A dynamic routing strategy is then used to select and activate experts from the expert set, which includes both domain-specific and cross-domain general experts, ultimately calculating task-adaptive features. This step differs from the traditional fixed expert activation mode, achieving decoupled modeling of domain-specific and general knowledge. It dynamically allocates computational resources while adapting to the task differences between speech and music, effectively mitigating task conflicts in unified generation.

[0056] In one implementation, the process of generating an expert probability distribution for an expert set through a pre-built expert set routing gating, and then using a dynamic routing strategy to select and activate experts based on the expert probability distribution, and calculating task adaptation features, specifically includes the following steps: Step S410: Use expert set routing gating to process the hybrid attention features word by word, and output the expert probability distribution of each word in the expert set; Step S420: Using the Top-P dynamic routing strategy, sort the expert probability distribution of the expert set from high to low probability values ​​and accumulate them until the accumulated probability reaches a preset threshold. Select the experts whose accumulated probability reaches the preset threshold as the active experts of the expert set. Step S430: Based on the expert set activation experts, calculate the hybrid attention features to obtain the output results of all the expert set activation experts; Step S440: The output results of all activated experts in the expert set are weighted and summed with normalized gating probabilities to obtain task adaptation features.

[0057] In this embodiment, the expert set routing gating consists of linear layers. For an input tensor, the expert set routing gating calculates the expert probability distribution for each label. Then, using a Top-P routing strategy, the smallest expert set whose cumulative probability exceeds a preset threshold p is dynamically selected. In this embodiment, the preset threshold p can be chosen as 0.7. With this setting, the model can adaptively determine the number of activated experts based on the label complexity, efficiently allocating computational resources.

[0058] After determining the expert set and activating the experts, the final output of the hybrid expert layer is a weighted sum of the selected expert outputs, with the corresponding weights being normalized gating probabilities. This is formally expressed as:

[0059] For an input tensor , For sequence length, To hide the dimension, the first Expert probability distribution of input labels , It is the first The output of an expert.

[0060] like Figure 2 As shown in the hybrid expert module, this dynamic adaptive computing architecture uses a Top-P routing mechanism instead of a fixed Top-K routing mechanism, enabling the model to dynamically allocate the number of experts according to the complexity of the input labeling, thereby achieving adaptive allocation of computing resources and improving efficiency and flexibility.

[0061] Besides using the Top-P routing mechanism, other mechanisms for dynamically determining the number of experts can be designed to replace Top-P sampling. For example, a lightweight complexity prediction network can be introduced, which directly predicts an optimal number of experts K' for each token, where K' is a variable, thus achieving dynamic Top-K' routing. Alternatively, the entropy of the gated probability distribution can be directly used as an indicator of complexity, with high entropy corresponding to more experts and low entropy corresponding to fewer experts.

[0062] Regarding the technical solution of this embodiment, the current solution is to rigidly select a subset of experts. A soft combination approach can also be used as an alternative, where all experts participate in the computation. Specifically, the gating system dynamically adjusts the weight gradient assigned to each expert based on the complexity of the token. The gradient for complex tokens will be more concentrated on a few key experts, while the gradient for simple tokens will be more dispersed or smaller overall, thus achieving expert selection.

[0063] In one implementation, the expert set includes several functionally differentiated experts, including: a routing expert for capturing domain-specific knowledge of speech or music; a sharing expert for learning cross-domain general knowledge across speech and music; and an empty expert for performing skip computation on simple lexical units.

[0064] In this embodiment, the expert pool comprises three types of experts: routing experts responsible for capturing domain-specific knowledge, shared experts learning cross-domain general knowledge, and empty experts who allow computation to be skipped for simple labels. In this hybrid expert structure, the number of routing experts, shared experts, and empty experts can be set to 8, 2, and 1, respectively, to decouple domain-specific features from cross-domain general features, and to enhance the model's adaptability by allowing computation to be skipped through empty experts.

[0065] In this embodiment, a shared expert is not mandatory. A separate, always-active shared expert can be replaced by a mechanism that forces partial parameter sharing among different routing experts. For example, it can be stipulated that the first few layers or specific modules of all experts, such as the attention projection layer, share the same set of parameters. Similarly, an empty expert can also be replaced. Specifically, instead of setting an explicit empty expert, a skip option can be added to the gating mechanism. When the gating selects skip, the input token will bypass the entire hybrid expert module directly through a residual connection. This also functionally achieves computation skipping.

[0066] Step S500: Based on the audio language modeling head, perform audio generation processing on the task adaptation features to obtain the resulting audio of speech or music.

[0067] In this embodiment, the audio language modeling head can adopt an autoregressive prediction architecture to predict discrete acoustic lexical sequences step-by-step based on task-adaptive features. This sequence can be in the same multi-channel format as the output of the aforementioned neural audio codec, i.e., 12 parallel channels, and the prediction process reuses previously generated lexical information to ensure temporal coherence.

[0068] Once the prediction is complete, the discrete acoustic word sequence is input into a neural audio codec of the same type to perform the decoding operation. The discrete words are restored to continuous audio waveforms through inverse transformation, and finally the audio result of speech or music is obtained.

[0069] Through the adaptation design of the audio language modeling head and codec, the transformation from task adaptation features to high-fidelity audio is realized. Furthermore, based on the domain signals contained in the task adaptation features, such as speech semantics or music structure, the corresponding type of audio can be adaptively generated to ensure the semantic clarity of speech and the integrity of music, thus completing the transformation from multimodal input to unified audio output.

[0070] In one implementation, the training steps for the attention head, attention head routing gating, expert set, and expert set routing gating in the audio model used to implement the generation method specifically include the following steps: Step S610: Train dense models on the speech-specific dataset and the music-specific dataset respectively to obtain domain prototype experts who have mastered the knowledge of the corresponding domains; Step S620: Perform balanced sampling on the speech-specific dataset and the music-specific dataset to obtain a balanced sampled dataset; Step S630: Integrate the domain prototype experts into the expert set, and use a parameter adjustment strategy on the balanced sampling dataset to train the expert set routing gate and the experts in the expert set who are used to learn cross-domain general knowledge; Step S640: Perform joint fine-tuning training on the parameters of the attention head, the attention head routing gating, the expert set, and the expert set routing gating.

[0071] In this embodiment, to resolve data imbalance and task conflicts, the following design is implemented: Figure 3 The three-stage training process is shown.

[0072] The first stage involves independent expert training, where dense models are trained on imbalanced datasets in their respective domains to form prototype experts. Structurally, the core computational unit of each prototype expert, the feedforward network layer (FFN), employs a multilayer perceptron (MLP) architecture with gated activation. In this embodiment, this stage of training uses the AdamW optimizer and a cosine learning rate scheduler, with a global batch size of 48 and a base learning rate of... .

[0073] Subsequently, balanced sampling was performed on the speech-specific dataset and the music-specific dataset to obtain a balanced sampled dataset with a relatively balanced amount of speech-specific data and music-specific data, which was used for subsequent training.

[0074] Finally, the second and third stages specifically include integrating the domain prototype experts into the expert set, using parameter tuning strategies on the balanced sampling dataset to train the expert set routing gating and the experts in the expert set used to learn cross-domain general knowledge, and performing joint fine-tuning training on the parameters of the attention head, the attention head routing gating, the expert set, and the expert set routing gating to obtain the trained consensus hybrid expert audio model.

[0075] In this final step of parameter tuning and joint fine-tuning training, a differentiated learning rate strategy can be used for end-to-end training. For example, in the initial stages of joint training, a very low learning rate can be set for the pre-trained experts, while a higher learning rate can be set for the randomly initialized gating and shared experts, and these learning rates can be dynamically adjusted as training progresses. Alternatively, a strategy such as... Figure 3 The training process shown is freezing, preheating, and thawing.

[0076] In one implementation, the parameter adjustment strategy and joint fine-tuning training specifically include the following steps: Step S631: Freeze the parameters of the domain prototype experts, and only update the parameters of the expert set routing gate and the experts in the expert set used to learn cross-domain general knowledge; Step S641: Unfreeze the frozen parameters. On the balanced sampling dataset, guide experts to make balanced use of the combined optimization objective of load balancing loss and main task loss, and jointly fine-tune and optimize the model training effect.

[0077] This embodiment illustrates the second and third stages of the training process of freezing, preheating, and thawing.

[0078] The second stage is the integration and warm-up of hybrid experts, which involves merging prototype experts into the hybrid expert architecture and freezing their weights, training the gating module and shared experts on the sampled balanced dataset, and stabilizing the routing and shared features.

[0079] Specifically, the feedforward network layers of each prototype expert pre-trained in the first stage are first extracted and integrated into a new hybrid expert layer as routing experts with specific knowledge, and their weights are temporarily frozen. Subsequently, on a small, sampled and balanced dataset, the model will be specifically trained on the newly introduced, randomly initialized components, namely the gating module responsible for expert selection and the shared expert responsible for learning general knowledge.

[0080] These experts, whether routing experts or sharing experts, all employ a multilayer perceptron architecture with gated activation, and their computational flow is represented as follows:

[0081] Wherein, input vector The dimension is 2048.

[0082] This calculation process involves three core weight matrices and a gated projection matrix. Upward projection matrix and downlink projection matrix The dimensions of these matrices vary depending on the type of expert.

[0083] For routing experts and All dimensions are ,and The dimension is .

[0084] For shared experts, the dimensions of these matrices are correspondingly [dimensions to be filled in]. and That is, the intermediate layer dimensions correspond to 2752 and 1376 respectively.

[0085] This warm-up step can stabilize the routing mechanism and pre-train shared features without interfering with the already learned expert knowledge, thereby effectively avoiding the catastrophic forgetting problem that may be caused by direct joint training.

[0086] The third stage involves unfreezing all parameters, fine-tuning them end-to-end on a large-scale balanced dataset, and using load balancing loss to achieve final collaborative joint training.

[0087] To calculate load balancing loss, we first define two key metrics.

[0088] The first key indicator is experts lexical allocation ratio This metric measures what percentage of tokens in a training batch are routed to the expert. The expression is:

[0089] in, Experts The word allocation ratio represents the experts The percentage of processed lexical units out of the total number of lexical units in the batch. This represents the total number of lexical units in a training batch. The index representing a word element, that is, the first word in the batch. Each word element, Indicates that the gating network is the first The original score vector output by each word element. This indicates that the function returns... The expert index corresponding to the element with the largest value in the vector, i.e., the term in the gating network. The chosen expert The index representing the expert, that is, the index representing the first expert in the expert pool. One expert, This indicates an indicator function; if the condition within the square brackets is true, i.e., a token... Assigned to experts If the expression is true, the function value is 1; otherwise, it is 0.

[0090] The second key indicator is experts Average routing probability This metric measures the efficiency of gating the network in a training batch by assigning it to experts. The average probability value is expressed as:

[0091] in, Indicates that the gating network is the first In the probability distribution of the output of the nth word, the nth word... The probability value corresponding to each expert It has passed deal with).

[0092] Based on the above two metrics, the final expression for the auxiliary load balancing loss is:

[0093] in, This represents the final value of the auxiliary load balancing loss. This indicates the total number of experts.

[0094] This formula penalizes unbalanced routing behavior by multiplying each expert's allocation proportion by the average probability and summing the results.

[0095] Finally, the auxiliary loss is combined with the main task loss to form the total loss function, which is expressed as:

[0096] in, , This represents the total loss, which is the final optimization objective during model training. This represents the main task loss, used to measure the model's prediction accuracy on the core task, i.e., the prediction accuracy of speech or music generation. It is the cross-entropy loss. The weighting coefficient represents the load balancing loss and is used to balance the two objectives of the main task and load balancing. This coefficient adopts a linear annealing strategy, with an initial value of 0.025, and decays linearly to a minimum value of 0.001 after 10,000 steps during training.

[0097] Ultimately, this loss function guides experts to leverage imbalanced data to gradually achieve cross-task collaborative learning and knowledge transfer. Through a systematic training paradigm, a three-stage training process—independent training, ensemble warm-up, and collaborative training—is proposed to avoid catastrophic forgetting and instability, fully utilize imbalanced data, and achieve smooth fusion and collaborative enhancement of cross-domain knowledge.

[0098] In summary, the technical solutions described in the above embodiments can simultaneously address the task conflict and data imbalance issues in speech and music generation. On the one hand, the combination of dynamic capacity hybrid expert architecture and hybrid attention allows the model to focus on domain features while maintaining cross-domain generality, effectively mitigating optimization objective conflicts. On the other hand, phased training ensures knowledge absorption and transfer under imbalanced data conditions, preventing speech data from excessively dominating music generation. This invention enhances the model's expressive power while maintaining computational efficiency, achieving an overall performance improvement and synergistic gain in the unified generation of speech and music.

[0099] like Figure 4As shown in the figure, this embodiment of the invention provides a speech and music collaborative generation system based on dynamic hybrid attention and expert architecture. The system includes: an input acquisition module 10, a fusion sequence acquisition module 20, a hybrid attention feature acquisition module 30, a task adaptation feature acquisition module 40, and a result audio acquisition module 50.

[0100] Specifically, the input acquisition module 10 is used to acquire multimodal input, which includes at least one of audio input, text input, and visual input; the fusion sequence acquisition module 20 is used to perform embedding representation processing on the multimodal input, converting the audio input into a discrete acoustic word sequence, the text input into a text embedding vector, and the visual input into soft visual tags, and mapping them to a unified embedding space to obtain a fusion sequence; the hybrid attention feature acquisition module 30 is used to set up several attention heads, each attention head dynamically selecting an attention head activation expert for the fusion sequence through attention head routing gating, and activating the attention head based on the attention head activation expert. The attention calculation result of the home and the gating probability of the attention head route gating are used to calculate the hybrid attention feature; the task adaptation feature acquisition module 40 is used to generate the expert probability distribution of the expert set through the pre-constructed expert set route gating, and based on the expert probability distribution of the expert set, use a dynamic routing strategy to select experts in the expert set to activate experts and calculate the task adaptation feature, wherein the expert set contains several experts with different functions, and the expert set includes at least experts who learn domain-specific knowledge and experts who learn cross-domain general knowledge; the result audio acquisition module 50 is used to perform audio generation processing on the task adaptation feature to obtain the result audio of speech or music.

[0101] Based on the above embodiments, the present invention also provides a terminal device, the principle block diagram of which can be as follows: Figure 5 As shown, the terminal device includes a processor, memory, network interface, display screen, and temperature sensor connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides the environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The network interface is used to communicate with external terminals via a network connection. When executed by the processor, the computer program implements a speech and music co-generation method based on dynamic hybrid attention and expert architecture. The display screen can be an LCD screen or an e-ink screen. The temperature sensor is pre-installed inside the terminal device to detect the operating temperature of the internal components.

[0102] Those skilled in the art will understand that Figure 5The schematic diagram shown is only a partial structural diagram related to the present invention and does not constitute a limitation on the terminal device to which the present invention is applied. The specific terminal device may include more or fewer components than shown in the figure, or combine certain components, or have different component arrangements.

[0103] In one embodiment, a terminal device is provided, including a memory and one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by one or more processors. The one or more programs include instructions for performing the following operations: Acquire multimodal input, wherein the multimodal input includes at least one of audio input, text input, and visual input; The multimodal input is processed by embedding representation, converting audio input into discrete acoustic word sequences, text input into text embedding vectors, and visual input into soft visual tags, which are then mapped to a unified embedding space to obtain a fused sequence. Several attention heads are set up. Each attention head is dynamically selected as an attention head activation expert for the fusion sequence through attention head routing gating. Based on the attention calculation results of the attention head activation experts and the gating probability of the attention head routing gating, the hybrid attention features are calculated. By using a pre-built expert set routing gating, an expert probability distribution of the expert set is generated. Based on the expert probability distribution of the expert set, a dynamic routing strategy is used to select experts to activate and calculate task adaptation features. The expert set contains several experts with different functions, and the expert set includes at least experts who learn domain-specific knowledge and experts who learn cross-domain general knowledge. Based on the audio language modeling head, the task adaptation features are processed to generate audio, resulting in speech or music.

[0104] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided by this invention can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), Rambus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0105] In summary, this invention discloses a method, system, terminal device, and medium for collaborative speech and music generation based on a dynamic hybrid attention and expert architecture, relating to the field of audio generation technology. The method first acquires multimodal input, including at least one of audio input, text input, and visual input. Then, it performs embedding representation processing on the multimodal input, converting the audio input into a discrete acoustic word sequence, the text input into a text embedding vector, and the visual input into soft visual tags, mapping them to a unified embedding space to obtain a fused sequence. Subsequently, several attention heads are set, each dynamically selecting an attention head activation expert for the fused sequence through attention head routing gating. Based on the attention calculation results of the attention head activation experts and the gating probability of the attention head routing gating, hybrid attention features are calculated. Next, an expert probability distribution is generated using a pre-constructed expert set routing gating system. Based on this distribution, a dynamic routing strategy is used to select and activate experts within the expert set, calculating task adaptation features. The expert set includes several functionally differentiated experts, at least those learning domain-specific knowledge and those learning cross-domain general knowledge. Finally, based on an audio language modeling head, the task adaptation features are processed to generate audio output for speech or music. This invention effectively decouples domain-specific knowledge and cross-domain general knowledge in speech and music by using a unified embedding mapping for multimodal inputs, a hybrid attention mechanism combining dynamic expert selection via an attention head routing gating system, and a design that dynamically selects functionally differentiated experts using an expert set routing gating system. This alleviates task conflicts in the unified generation of both. Furthermore, the dynamic routing strategy allocates computational resources based on lexical complexity, improving resource utilization and efficiency, ultimately achieving high-quality collaborative generation of speech and music.

[0106] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0107] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A method for co-generating speech and music based on dynamic hybrid attention and expert architecture, characterized in that, The method includes: Acquire multimodal input, which includes audio input, text input, and visual input; The multimodal input is processed by embedding representation, converting audio input into discrete acoustic word sequences, text input into text embedding vectors, and visual input into soft visual tags, which are then mapped to a unified embedding space to obtain a fused sequence. Several attention heads are configured, each dynamically selecting an attention head activation expert for the fusion sequence through an attention head routing gating system. Based on the attention calculation results of the attention head activation experts and the gating probability of the attention head routing gating system, a hybrid attention feature is calculated. Specifically, this includes: setting up several attention heads, each configured with an independent pool of attention head candidate experts, wherein the attention head routing gating system corresponds one-to-one with the attention head candidate expert pool; for each attention head, based on the fusion sequence, calculating the expert probability distribution of the attention head candidate expert pool using the attention head routing gating system, and selecting a preset number of attention head activation experts using a TopN routing strategy; for each attention head, calling the parameter matrix of the selected attention head activation expert to perform attention calculation, multiplying the calculation result element-wise with the gating probability of the corresponding attention head routing gating system, and summing the results to obtain the output of a single attention head; aggregating the outputs of all attention heads to obtain the hybrid attention feature. A pre-constructed expert set routing gating system is used to generate an expert probability distribution for the expert set. Based on this distribution, a dynamic routing strategy is used to select active experts and calculate task-adaptive features. The expert set includes several functionally differentiated experts, at least those learning domain-specific knowledge and those learning cross-domain general knowledge. Specifically, this involves: processing the hybrid attention features word-by-word using the expert set routing gating system to output the expert probability distribution for each word in the expert set; using a Top-P dynamic routing strategy, sorting and accumulating the expert probability distributions in the expert set from highest to lowest probability value until the accumulated probability reaches a preset threshold, and selecting experts whose accumulated probability reaches the preset threshold as active experts; calculating the hybrid attention features based on the active experts to obtain the output results of all active experts; and weighting and summing the output results of all active experts using normalized gating probabilities to obtain the task-adaptive features. Based on the audio language modeling head, the task adaptation features are processed to generate audio, resulting in speech or music.

2. The speech and music co-generation method based on dynamic hybrid attention and expert architecture according to claim 1, characterized in that, The embedding representation processing of the multimodal input involves converting audio input into a discrete acoustic word sequence, text input into a text embedding vector, and visual input into soft visual tags, which are then mapped to a unified embedding space to obtain a fused sequence, including: The audio waveform of the audio input is encoded using a neural audio codec, and cascaded prediction is performed through a delay mechanism to obtain a multi-channel discrete acoustic word sequence. The text input is segmented using a word segmenter and then converted into text embedding vectors in a unified embedding space through an embedding mapping layer. Visual input is encoded using a visual Transformer to obtain block features, which are then mapped to soft visual labels in a unified embedding space using a projector. The multi-channel discrete acoustic lexical sequence, the text embedding vector, and the soft visual markers are fused to obtain a fused sequence.

3. The speech and music co-generation method based on dynamic hybrid attention and expert architecture according to claim 1, characterized in that, The expert set includes several functionally differentiated experts, including: Routing experts are used to capture domain-specific knowledge specific to voice or music; Shared experts are used to learn cross-domain general knowledge across speech and music; Empty expert, used to perform skip computation on simple lexical units.

4. The speech and music co-generation method based on dynamic hybrid attention and expert architecture according to any one of claims 1-3, characterized in that, The training steps for the attention head, attention head routing gating, expert set, and expert set routing gating in the audio model used to implement the generation method include: Dense models are trained on both speech-specific and music-specific datasets to obtain domain prototype experts who possess knowledge of the corresponding domains. The speech-specific dataset and the music-specific dataset are subjected to balanced sampling to obtain a balanced sampled dataset. The domain prototype experts are integrated into the expert set, and a parameter adjustment strategy is used on the balanced sampling dataset to train the expert set routing gate and the experts in the expert set who are used to learn cross-domain general knowledge. Joint fine-tuning training is performed on the parameters of the attention head, the attention head routing gating, the expert set, and the expert set routing gating.

5. The speech and music co-generation method based on dynamic hybrid attention and expert architecture according to claim 4, characterized in that, The parameter adjustment strategy and joint fine-tuning training include: Freeze the parameters of the domain prototype experts and update only the parameters of the expert set routing gate and the experts in the expert set used to learn cross-domain general knowledge; Unfreeze the frozen parameters, and on the balanced sampling dataset, guide experts to make balanced use of the combined optimization objective of load balancing loss and main task loss, and jointly fine-tune and optimize the model training effect.

6. A speech and music co-generation system based on dynamic hybrid attention and expert architecture, characterized in that, The system, used to implement the speech and music co-generation method based on dynamic hybrid attention and expert architecture as described in any one of claims 1-5, comprises: The input acquisition module is used to acquire multimodal input, which includes audio input, text input and visual input; The fusion sequence acquisition module is used to perform embedding representation processing on the multimodal input, converting audio input into discrete acoustic word sequences, text input into text embedding vectors, and visual input into soft visual tags, and mapping them to a unified embedding space to obtain a fusion sequence; The hybrid attention feature acquisition module is used to set up several attention heads, each attention head dynamically selects an attention head activation expert for the fusion sequence through attention head routing gating, and calculates hybrid attention features based on the attention calculation results of the attention head activation experts and the gating probability of the attention head routing gating. The task adaptation feature acquisition module is used to generate the expert probability distribution of the expert set through a pre-built expert set routing gating, and to select experts to activate the expert set using a dynamic routing strategy based on the expert probability distribution of the expert set, and to calculate the task adaptation features. The expert set includes several experts with different functions, and the expert set includes at least experts who have learned domain-specific knowledge and experts who have learned cross-domain general knowledge. The result audio acquisition module is used to perform audio generation processing on the task adaptation features to obtain the result audio of speech or music.

7. A terminal device, characterized in that, The terminal device includes a memory, a processor, and a speech and music co-generation program based on dynamic hybrid attention and expert architecture stored in the memory and executable on the processor. When the processor executes the speech and music co-generation program based on dynamic hybrid attention and expert architecture, it implements the steps of the speech and music co-generation method based on dynamic hybrid attention and expert architecture as described in any one of claims 1-5.

8. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a speech and music co-generation program based on a dynamic hybrid attention and expert architecture. When the speech and music co-generation program based on a dynamic hybrid attention and expert architecture is executed by a processor, it implements the steps of the speech and music co-generation method based on a dynamic hybrid attention and expert architecture as described in any one of claims 1-5.

Images