Voice and music collaborative generation method and system based on dynamic mixed attention and expert architecture, terminal equipment and medium

Abstract

The invention discloses a voice and music collaborative generation method and system based on a dynamic mixed attention and expert architecture, terminal equipment and a medium, and relates to the technical field of audio generation. The method comprises the following steps: acquiring multi-modal input containing at least one of audio, text and vision, performing embedding processing on the multi-modal input, and mapping the multi-modal input to a unified space to obtain a fusion sequence; setting a plurality of attention heads, dynamically selecting the attention heads to activate experts through attention head routing gating, and weighting and aggregating calculation results to obtain mixed attention features; generating expert probability distribution through expert set routing gating, dynamically selecting an expert set to activate experts, and calculating task adaptation features; and performing audio generation processing based on the audio language modeling head to obtain a voice or music result. According to the method, collaborative generation of voice and music, dynamic allocation of computing resources, balance of domain-specific and cross-domain general knowledge learning are realized, the problems of task conflict and data imbalance are solved, and the audio generation quality and efficiency are improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of audio generation, in particular to a speech and music collaborative generation method and system based on dynamic mixing attention and expert architecture, a terminal device and a medium. BACKGROUND

[0002] In recent years, generative artificial intelligence represented by large language models has made revolutionary progress, promoting the development of unified content generation models across text, images, audio and other multi-modalities. In the field of hearing, although speech synthesis and music generation have made remarkable achievements, they are often developed in isolation in independent and dedicated models, hindering the progress of general audio synthesis technology and limiting the ability of models to seamlessly perceive, reason and create diverse audio content.

[0003] In the prior art, speech synthesis models such as Step-audio model model speech as a language modeling task, and perform outstandingly in zero-shot voice cloning and controllable generation; music generation models such as MusicGen model verify the feasibility of autoregressive Transformer generating high-fidelity music. However, unified modeling of the two faces two major challenges: first, task conflict, speech generation emphasizes semantic clarity and speaker identity restoration, music generation focuses on artistic structures such as harmony and rhythm, and the natural differences in optimization goals lead to performance trade-offs; second, data imbalance, high-quality speech corpus is much more than music corpus, and speech data is easy to dominate learning in naive joint training, reducing music generation quality, such as UniAudio and other early attempts are limited by this. In addition, the fixed Top-K routing strategy of traditional hybrid expert models ignores the complexity differences of word units, and the computational efficiency is poor.

[0004] Therefore, there is an urgent need for a method that can dynamically allocate computing resources, decouple domain and cross-domain knowledge learning, and adapt to imbalanced data training to fill the gap in the prior art. SUMMARY

[0005] The technical problem to be solved by the present application is that in the field of audio generation, the prior art separates speech generation and music generation for development, and when unified modeling, it faces the task conflict problem that speech generation emphasizes semantic clarity and identity restoration, music generation focuses on artistic structure, and the difference in optimization goals leads to performance trade-offs, and the data imbalance problem that speech corpus is much more than music corpus, which is easy to dominate learning and reduces music generation quality, and the computational efficiency of the traditional hybrid expert fixed routing strategy is poor. Therefore, there is an urgent need for an effective solution to solve the above technical problems.

[0006] To solve the above technical problems, the technical solution adopted by the present application is as follows: In a first aspect, the present application provides a voice and music collaborative generation method based on dynamic mixed attention and expert architecture, comprising: obtaining a multi-modal input, the multi-modal input comprising at least one of an audio input, a text input, and a visual input; performing embedding representation processing on the multi-modal input, converting the audio input into a discrete acoustic token sequence, converting the text input into a text embedding vector, converting the visual input into a soft visual token, and mapping to a unified embedding space to obtain a fusion sequence; setting a plurality of attention heads, each attention head being dynamically selected by an attention head routing gate to activate an expert on the fusion sequence, and calculating a mixed attention feature based on the attention calculation result of the attention head activated expert and the gating probability of the attention head routing gate; generating an expert probability distribution of an expert set through pre-constructed routing gate of the expert set, and selecting an expert set activated expert using a dynamic routing strategy based on the expert probability distribution of the expert set to calculate a task adaptation feature, wherein the expert set comprises a plurality of functionally differentiated experts, and the expert set at least includes an expert learning domain-specific knowledge and an expert learning cross-domain general knowledge; performing audio generation processing on the task adaptation feature based on an audio language modeling head to obtain a result audio of voice or music.

[0007] In an implementation manner, the embedding representation processing on the multi-modal input, the audio input converted into a discrete acoustic token sequence, the text input converted into a text embedding vector, the visual input converted into a soft visual token, and mapped to a unified embedding space to obtain a fusion sequence, comprises: encoding the audio waveform of the audio input using a neural audio codec, and performing cascading prediction through a delay mechanism to obtain a multi-channel discrete acoustic token sequence; performing word segmentation on the text input using a word segmenter, and converting to a text embedding vector in a unified embedding space through an embedding mapping layer; encoding the visual input using a visual Transformer to obtain block features, and mapping to a soft visual token in a unified embedding space through a projector; fusing the multi-channel discrete acoustic token sequence, the text embedding vector, and the soft visual token to obtain a fusion sequence.

[0008] In an implementation manner, the setting a plurality of attention heads, each attention head being dynamically selected by an attention head routing gate to activate an expert on the fusion sequence, and calculating a mixed attention feature based on the attention calculation result of the attention head activated expert and the gating probability of the attention head routing gate, comprises: A plurality of attention heads are set, and each attention head is configured with an independent attention head candidate expert pool, wherein the attention head routing gate is one-to-one corresponding to the attention head candidate expert pool; For each attention head, based on the fusion sequence, the expert probability distribution of the attention head candidate expert pool is calculated using the attention head routing gate, and a TopN routing strategy is adopted to select a preset number of attention head activation experts; For each attention head, the parameter matrix of the selected attention head activation expert is called to perform attention calculation, and the calculation result is multiplied element by element with the gating probability of the corresponding attention head routing gate and then summed to obtain the output of a single attention head. The outputs of all attention heads are aggregated to obtain a hybrid attention feature.

[0009] In an implementation manner, the expert set routing gate is used to generate the expert probability distribution of the expert set, and the expert set activation expert is selected based on the expert probability distribution of the expert set using a dynamic routing strategy, and the task adaptation feature is calculated, including: The hybrid attention feature is processed word by word using the expert set routing gate, and the expert probability distribution of each word in the expert set is output; The Top-P dynamic routing strategy is used to sort and accumulate the expert probability distribution of the expert set in descending order of probability value until the accumulated probability reaches a preset threshold, and the expert with the accumulated probability reaching the preset threshold is selected as the expert set activation expert; Based on the expert set activation expert, the hybrid attention feature is calculated to obtain the output results of all the expert set activation experts; The output results of all the expert set activation experts are weighted and summed with normalized gating probabilities to obtain a task adaptation feature.

[0010] In an implementation manner, the plurality of functionally differentiated experts included in the expert set include: A routing expert is used to capture domain-specific knowledge specific to speech or music; A shared expert is used to learn cross-domain general knowledge across speech and music; An empty expert is used to perform skip calculation on simple word units.

[0011] In an implementation manner, the training steps of the attention head, the attention head routing gate, the expert set, and the expert set routing gate in the audio model used to implement the generation method include: A dense model is trained on a speech-specific dataset and a music-specific dataset respectively to obtain a domain prototype expert that masters the corresponding domain knowledge; balance sampling on the voice-specific dataset and the music-specific dataset to obtain a balanced sampling dataset; integrate the domain prototype expert into the expert set, and train the expert set routing gate and the experts in the expert set for learning cross-domain general knowledge using a parameter adjustment strategy on the balanced sampling dataset; perform joint fine-tuning training on parameters of the attention head, the attention head routing gate, the expert set, and the expert set routing gate.

[0012] In an implementation manner, the parameter adjustment strategy and the joint fine-tuning training include: freeze parameters of the domain prototype expert, and update only parameters of the expert set routing gate and the experts in the expert set for learning cross-domain general knowledge; unfreeze the frozen parameters, and jointly fine-tune and optimize model training effects by guiding experts to balance utilization through a combination optimization target of a load balancing loss and a main task loss on the balanced sampling dataset.

[0013] In a second aspect, the embodiments of the present application also provide a voice and music collaborative generation system based on a dynamic mixed attention and expert architecture, the system comprising: an input acquisition module configured to acquire a multi-modal input, the multi-modal input comprising at least one of an audio input, a text input, and a visual input; a fusion sequence acquisition module configured to perform embedding representation processing on the multi-modal input, convert the audio input into a discrete acoustic token sequence, convert the text input into a text embedding vector, convert the visual input into soft visual labels, and map the soft visual labels to a unified embedding space to obtain a fusion sequence; a mixed attention feature acquisition module configured to set a plurality of attention heads, each attention head dynamically selecting an attention head activated expert for the fusion sequence through an attention head routing gate, and calculating a mixed attention feature based on an attention calculation result of the attention head activated expert and a gating probability of the attention head routing gate; a task adaptation feature acquisition module configured to generate an expert probability distribution of an expert set through a pre-constructed expert set routing gate, and select an expert set activated expert using a dynamic routing strategy based on the expert probability distribution of the expert set to calculate a task adaptation feature, wherein the expert set comprises a plurality of functionally differentiated experts, and the expert set at least includes an expert for learning domain-specific knowledge and an expert for learning cross-domain general knowledge; a result audio acquisition module configured to perform audio generation processing on the task adaptation feature to obtain a result audio of voice or music.

[0014] In a third aspect, the embodiments of the present application further provide a terminal device, which comprises a memory, a processor, and a voice and music collaborative generation program based on a dynamic hybrid attention and expert architecture stored in the memory and executable on the processor. When the processor executes the voice and music collaborative generation program based on the dynamic hybrid attention and expert architecture, the steps of the voice and music collaborative generation method based on the dynamic hybrid attention and expert architecture in any of the above solutions are implemented.

[0015] In a fourth aspect, the embodiments of the present application further provide a computer readable storage medium, which stores a voice and music collaborative generation program based on a dynamic hybrid attention and expert architecture. When the voice and music collaborative generation program based on the dynamic hybrid attention and expert architecture is executed by a processor, the steps of the voice and music collaborative generation method based on the dynamic hybrid attention and expert architecture in any of the above solutions are implemented.

[0016] Advantages: The voice and music collaborative generation method, system, terminal device and medium based on a dynamic hybrid attention and expert architecture relate to the technical field of audio generation. The method first acquires a multi-modal input, which includes at least one of an audio input, a text input and a visual input. Then, the multi-modal input is subjected to embedding representation processing, the audio input is converted into a discrete acoustic token sequence, the text input is converted into a text embedding vector, the visual input is converted into a soft visual label and mapped to a unified embedding space to obtain a fusion sequence. Subsequently, a plurality of attention heads are set, each attention head dynamically selects an attention head activated expert for the fusion sequence through an attention head routing gate, and calculates a hybrid attention feature based on an attention calculation result of the attention head activated expert and a gating probability of the attention head routing gate. Then, an expert set routing gate is used to generate an expert probability distribution of the expert set, and an expert set activated expert is selected using a dynamic routing strategy based on the expert probability distribution of the expert set, and a task adaptation feature is calculated, wherein the expert set comprises a plurality of functionally differentiated experts, and the expert set at least includes an expert learning domain-specific knowledge and an expert learning cross-domain general knowledge. Finally, an audio generation processing is performed on the task adaptation feature based on an audio language modeling head to obtain a result audio of voice or music. The present application effectively decouples the domain-specific knowledge and the cross-domain general knowledge of voice and music by using the unified embedding mapping of the multi-modal input, the hybrid attention mechanism of dynamically selecting and activating the expert through the attention head routing gate, and the design of dynamically selecting the functionally differentiated expert through the expert set routing gate, thereby relieving the task conflict of unified generation of voice and music, and dynamically allocating the computational resources according to the token complexity through the dynamic routing strategy, thereby improving the resource utilization and efficiency, and finally realizing the high-quality collaborative generation of voice and music. BRIEF DESCRIPTION OF DRAWINGS

[0017] Figure 1 A flowchart illustrating a specific implementation of the speech and music co-generation method based on dynamic hybrid attention and expert architecture provided in this invention.

[0018] Figure 2 This is a flowchart and architecture diagram of the speech and music co-generation method based on dynamic hybrid attention and expert architecture provided in the embodiments of the present invention.

[0019] Figure 3 This is a schematic diagram of the training process in the speech and music co-generation method based on dynamic hybrid attention and expert architecture provided in the embodiments of the present invention.

[0020] Figure 4 This is a schematic diagram of the speech and music co-generation device based on dynamic hybrid attention and expert architecture provided in an embodiment of the present invention.

[0021] Figure 5 This is a block diagram illustrating the internal structure of the terminal device provided in an embodiment of the present invention. Detailed Implementation

[0022] To make the objectives, technical solutions, and effects of this invention clearer and more explicit, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.

[0023] The flowchart shown in the attached diagram is for illustrative purposes only and does not necessarily include all content, operations, or steps, nor does it require execution in the described order. For example, some operations or steps can be broken down, combined, or partially merged, so the actual execution order may change depending on the actual situation.

[0024] It should be understood that the terminology used in this specification is for the purpose of describing particular embodiments only and is not intended to limit the invention. As used in this specification and the appended claims, the singular forms “a,” “an,” and “the” are intended to include the plural forms unless the context clearly indicates otherwise.

[0025] It should be understood that, in order to clearly describe the technical solutions of the embodiments of the present invention, the terms "first" and "second" are used in the embodiments of the present invention to distinguish identical or similar items with essentially the same function and effect. For example, "first control information" and "second control information" are only used to distinguish different control information and do not limit their order.

[0026] Those skilled in the art will understand that the words "first" and "second" do not limit the quantity or the order of execution, and that the words "first" and "second" do not necessarily imply that they are different.

[0027] It should also be understood that the term “and / or” as used in this specification and the appended claims refers to any combination of one or more of the associated listed items and all possible combinations, and includes such combinations.

[0028] In recent years, generative artificial intelligence, represented by large-scale language models, has made revolutionary progress, driving the development of unified content generation models across multiple modalities such as text, images, and audio. In the auditory domain, while significant achievements have been made in speech synthesis and music generation, these two tasks are typically developed in isolation within dedicated models, hindering the advancement of general-purpose audio synthesis techniques. This independent development limits the models' ability to seamlessly perceive, reason, and create diverse audio content like humans.

[0029] Most existing audio generation technologies focus on a single domain. For example, in speech synthesis, models like Step-audio have demonstrated unprecedented capabilities in zero-shot speech cloning and controlled speech generation by modeling speech as a language modeling task; while in music generation, models like MusicGen have verified the feasibility of generating high-fidelity music using autoregressive Transformers. However, unifying speech and music generation into a single model still faces two major challenges that current technologies have not yet effectively addressed. The first is the task conflict between speech and music generation. Speech generation emphasizes semantic clarity and speaker identification, while music generation focuses on complex artistic structures such as harmony and rhythm. These two optimization objectives are inherently different, and when forced into the same model, conflicting optimization goals often arise, causing the model to sacrifice the performance of one task while improving the performance of the other. The second is data imbalance. Currently, high-quality, large-scale speech corpora are far more abundant than music corpora. Under a naive joint training framework, speech data, due to its quantitative advantage, dominates the model's learning process, leading to a significant decline in the quality of music generation. For example, early attempts like UniAudio were limited in performance on data-scarce tasks such as music due to data imbalance. Furthermore, while traditional hybrid expert models offer potential architectural solutions for mitigating multimodal conflicts, their fixed Top-K routing strategy is computationally inefficient. This method assigns a fixed number of experts to each token, ignoring the differences in complexity among different tokens and thus limiting the model's computational efficiency.

[0030] Therefore, this invention addresses the problems of task conflict and data imbalance in existing technologies for collaborative speech and music generation by proposing a speech and music collaborative generation method based on dynamic hybrid attention and expert architecture.

[0031] At the model structure level, this invention proposes a unified hybrid expert audio model, such as... Figure 1 As shown, the model takes multimodal inputs such as text, sound, and video as inputs, and outputs audio outputs such as music generation, speech synthesis, and timbre cloning.

[0032] Specifically, a dynamic capacity hybrid expert architecture is introduced into this model, such as... Figure 2 The hybrid expert module is shown in the diagram, and this invention applies this hybrid expert architecture to a feedforward network. Unlike traditional hybrid expert architectures with a fixed number of activated experts, this architecture can dynamically adjust the number of activated experts based on the complexity of different labels, thereby improving computational efficiency while ensuring generation quality. Furthermore, by introducing multiple types of experts, the model can effectively decouple domain-specific feature learning computation from cross-domain general feature learning, and supports adaptive skipping of redundant labels, further improving resource utilization.

[0033] Secondly, at the feature modeling level, this invention proposes a hybrid attention mechanism, such as... Figure 2 As shown in the hybrid attention module, the idea of ​​dynamically activating the parameters of hybrid experts is transferred to the attention module. By dynamically assigning activation parameters to each attention head and keeping the candidate expert pool of each attention head independent, the mixing of experts from multiple attention heads is avoided. Through the hybrid attention module, the dynamic activation of all major modules of the entire model is achieved, improving the model's state space and thus further enhancing the model's potential.

[0034] Finally, at the training level, this invention designs a phased training strategy, such as... Figure 3 As shown in the training process diagram, this strategy uses a process of gradually imparting domain knowledge, integration and warm-up, and joint collaborative optimization to enable the model to focus on domain feature learning, cross-domain fusion and overall performance improvement at different stages, thereby achieving stable training and effective collaborative learning even under imbalanced data conditions.

[0035] This embodiment provides a method for co-generating speech and music based on a dynamic hybrid attention and expert architecture, such as... Figure 1 As shown, the specific steps include the following: Step S100: Obtain multimodal input, wherein the multimodal input includes at least one of audio input, text input and visual input.

[0036] In this embodiment, the multimodal input covers typical scenarios of speech and music generation. Specifically, audio input can be raw audio signals such as speech waveforms to be continued or music clips; text input can be instructional text such as speech synthesis scripts or style descriptions for music generation; and visual input can be visual information such as video frames containing the speaker's lip movements or scene images corresponding to the music. These multimodal inputs will be converted into... Figure 2 The audio tokens, text tokens, and visual tokens shown provide the raw input data for subsequent unified embedding processing. The purpose of using multimodal input is to broaden the adaptability of the generation scenario through multi-source information collection, and to provide a data foundation for cross-modal fusion and unified generation.

[0037] Step S200: Perform embedding representation processing on the multimodal input, convert the audio input into a discrete acoustic lexical sequence, convert the text input into a text embedding vector, convert the visual input into soft visual tags, and map them to a unified embedding space to obtain a fused sequence.

[0038] In this embodiment, heterogeneous input unification processing is performed on multimodal inputs. For audio, text, and visual inputs, embedding methods adapted to their data characteristics are employed respectively. Specifically, continuous audio waveforms are converted into discrete acoustic word sequences, text is converted into semantic embedding vectors, and visual information is converted into soft visual tags. Finally, these are converged into the same embedding space through dimensional mapping to form... Figure 2 The unified representation before fusion, i.e., the fusion sequence, shown in the diagram, is used for subsequent data processing. By unifying the processing of multimodal inputs, the problem of multimodal data format differences is solved, providing a common input basis for subsequent hybrid attention and hybrid expert architecture processing, and realizing the synergy of cross-modal information.

[0039] In one implementation, the embedding representation processing of the multimodal input, which converts audio input into a discrete acoustic word sequence, text input into a text embedding vector, and visual input into soft visual tags and maps them to a unified embedding space to obtain a fused sequence, specifically includes the following steps: Step S210: Encode the audio waveform of the audio input using a neural audio codec, and perform cascaded prediction through a delay mechanism to obtain a multi-channel discrete acoustic word sequence; Step S220: Use a word segmenter to segment the text input and convert it into a text embedding vector in a unified embedding space through an embedding mapping layer; Step S230: Encode the visual input using a visual Transformer to obtain block features, and map them to soft visual labels in a unified embedding space using a projector. Step S240: Fuse the multi-channel discrete acoustic lexical sequence, the text embedding vector, and the soft visual marker to obtain a fused sequence.

[0040] In this embodiment, for audio input, a neural audio codec is used to convert the continuous audio waveform into a discrete sequence of acoustic tokens across multiple channels. Specifically, a DAC (High-Fidelity Audio Compression with Improved RVQGAN) is used to encode 16kHz audio into 12 parallel channels at an 8kHz ratio.

[0041] For an audio segment lasting T time steps, the structure of its output acoustic lexical sequence is as follows: ,in It is the first The time step to the first Discrete lexical indices for each channel. Unlike channel-by-channel prediction and fully synchronous independent prediction methods, this embodiment uses a delay mechanism for cascaded prediction to better model the complex dependencies between channels and improve audio quality. This method introduces a specific time-step delay for each channel, and its pattern can be represented by a delay vector. To define precisely, where Representing the The generation delay of each channel relative to the initial time step.

[0042] In actual generation, the model does not output all 12 channels of tokens at once, but follows the order defined by the delay vectors. First, it generates tokens for channels with a delay of 0, then uses this new information to generate tokens for channels with a delay of 1, and so on, forming a cascaded prediction stream. In this example, a sequential delay pattern is used. This approach allows predictions for each channel to be conditioned on newly generated lexical units from the preceding channels at the same time step, thereby significantly improving the coherence and fidelity of the generated audio.

[0043] For text input, this embodiment first uses the Qwen2.5vl word segmenter to convert it into a text word sequence, and then maps it into a 2048-dimensional vector through an embedding mapping layer.

[0044] For visual inputs such as video, the same visual Transformer as the Qwen-VL method is used to segment the image into 14x14 pixel blocks and encode them into block features. Then, the visual feature dimensions are mapped from 1280 dimensions to an embedding space consistent with the 2048 dimensions of the model text embedding through a projector, generating soft visual tags that can be fused with text and audio representations.

[0045] Finally, the generated multi-channel discrete acoustic lexical sequence, text embedding vector, and soft visual markers are fused to obtain a fused sequence for subsequent use.

[0046] Step S300: Set up several attention heads. Each attention head dynamically selects an attention head activation expert for the fusion sequence through attention head routing gating. Based on the attention calculation results of the attention head activation experts and the gating probability of the attention head routing gating, calculate the hybrid attention features.

[0047] In this embodiment, a traditional attention layer with fixed parameters is replaced by configuring dedicated routing gating and expert resources for several independent attention heads. Each attention head dynamically selects and activates experts from a dedicated pool of candidate experts through attention head routing gating, and then aggregates the calculation results using gating probability weighting. Figure 2 The hybrid attention module addresses the limitations of attention head parameter sharing, enhances the model's ability to differentiate features across speech and music modalities, and provides a foundation for resolving cross-task feature conflicts.

[0048] In one implementation, a plurality of attention heads are set, and each attention head dynamically selects an attention head activation expert for the fused sequence through an attention head routing gating. Based on the attention calculation results of the attention head activation experts and the gating probability of the attention head routing gating, a hybrid attention feature is calculated. Specifically, this includes the following steps: Step S310: Set up several attention heads, each attention head is configured with an independent attention head candidate expert pool, wherein the attention head routing gate corresponds one-to-one with the attention head candidate expert pool; Step S320: For each attention head, based on the fusion sequence, use attention head routing gating to calculate the expert probability distribution of the attention head candidate expert pool, and adopt the TopN routing strategy to select a preset number of attention heads to activate experts. Step S330: For each attention head, call the parameter matrix of the selected attention head activation expert to perform attention calculation, multiply the calculation result element by element with the gating probability of the corresponding attention head route gating and sum them to obtain the output of a single attention head; Step S340: Aggregate the outputs of all attention heads to obtain hybrid attention features.

[0049] In this embodiment, a hybrid attention layer is used to replace the traditional standard attention layer in the model. The core of this replacement step is that a dynamic activation strategy will be adopted for the parameters of each attention head.

[0050] First, assume there are a total of The first attention point, for the first Each attention point will be selected and activated by the expert predicted through the Top2 routing gating network. ,in It is a gating matrix. The input tensor is used. In this embodiment, the model has a total of 36 layers, with 16 attention heads per layer and a total hidden layer dimension of 2048, therefore each head has a dimension of 128.

[0051] Subsequently, the two expert indices with the highest probabilities are assigned computation matrices to each attention head. Standard attention calculations are performed independently within each attention head, and the probabilities calculated by the gating network are used as weights. The specific expression is as follows:

[0052] in, , Indicates the first The output of each attention head, Indicates input The first one calculated by the gating network The weight of each expert, Indicates the first The parameter matrices of the nth expert are respectively the nth expert's parameter matrix. The weight matrix of Q, K, and V corresponding to each expert , This indicates element-wise multiplication of matrices.

[0053] This method achieves dynamic activation of attention module parameters while ensuring that each attention head does not interfere with others, increasing the number of activated states in the model. Ultimately, this allows the model to simultaneously and dynamically activate the feed-forward network (FFN) and attention layers for different tasks in speech and music generation, thus resolving the joint training conflict caused by speech and music generation tasks and improving the model's efficiency and accuracy. In other words, as... Figure 2 The hybrid attention mechanism shown improves the flexibility of model parameter activation and enhances the overall multi-task feature modeling capability by adapting the attention parameters to the dynamic activation method.

[0054] Step S400: Generate the expert probability distribution of the expert set through the pre-built expert set routing gating, and select experts to activate the expert set using a dynamic routing strategy based on the expert probability distribution of the expert set, and calculate the task adaptation features. The expert set contains several experts with different functions, and the expert set includes at least experts who learn domain-specific knowledge and experts who learn cross-domain general knowledge.

[0055] In this embodiment, as Figure 1As shown in the hybrid expert module, hybrid attention features are processed through a pre-built expert set routing gating system. A dynamic routing strategy is then used to select and activate experts from the expert set, which includes both domain-specific and cross-domain general experts, ultimately calculating task-adaptive features. This step differs from the traditional fixed expert activation mode, achieving decoupled modeling of domain-specific and general knowledge. It dynamically allocates computational resources while adapting to the task differences between speech and music, effectively mitigating task conflicts in unified generation.

[0056] In one implementation, the process of generating an expert probability distribution for an expert set through a pre-built expert set routing gating, and then using a dynamic routing strategy to select and activate experts based on the expert probability distribution, and calculating task adaptation features, specifically includes the following steps: Step S410: Use expert set routing gating to process the hybrid attention features word by word, and output the expert probability distribution of each word in the expert set; Step S420: Using the Top-P dynamic routing strategy, sort the expert probability distribution of the expert set from high to low probability values ​​and accumulate them until the accumulated probability reaches a preset threshold. Select the experts whose accumulated probability reaches the preset threshold as the active experts of the expert set. Step S430: Based on the expert set activation experts, calculate the hybrid attention features to obtain the output results of all the expert set activation experts; Step S440: The output results of all activated experts in the expert set are weighted and summed with normalized gating probabilities to obtain task adaptation features.

[0057] In this embodiment, the expert set routing gating consists of linear layers. For an input tensor, the expert set routing gating calculates the expert probability distribution for each label. Then, using a Top-P routing strategy, the smallest expert set whose cumulative probability exceeds a preset threshold p is dynamically selected. In this embodiment, the preset threshold p can be chosen as 0.7. With this setting, the model can adaptively determine the number of activated experts based on the label complexity, efficiently allocating computational resources.

[0058] After determining the expert set and activating the experts, the final output of the hybrid expert layer is a weighted sum of the selected expert outputs, with the corresponding weights being normalized gating probabilities. This is formally expressed as:

[0059] For an input tensor , For sequence length, To hide the dimension, the first Expert probability distribution of input labels , It is the first The output of an expert.

[0060] like Figure 2 As shown in the hybrid expert module, this dynamic adaptive computing architecture uses a Top-P routing mechanism instead of a fixed Top-K routing mechanism, enabling the model to dynamically allocate the number of experts according to the complexity of the input labeling, thereby achieving adaptive allocation of computing resources and improving efficiency and flexibility.

[0061] Besides using the Top-P routing mechanism, other mechanisms for dynamically determining the number of experts can be designed to replace Top-P sampling. For example, a lightweight complexity prediction network can be introduced, which directly predicts an optimal number of experts K' for each token, where K' is a variable, thus achieving dynamic Top-K' routing. Alternatively, the entropy of the gated probability distribution can be directly used as an indicator of complexity, with high entropy corresponding to more experts and low entropy corresponding to fewer experts.

[0062] Regarding the technical solution of this embodiment, the current solution is to rigidly select a subset of experts. A soft combination approach can also be used as an alternative, where all experts participate in the computation. Specifically, the gating system dynamically adjusts the weight gradient assigned to each expert based on the complexity of the token. The gradient for complex tokens will be more concentrated on a few key experts, while the gradient for simple tokens will be more dispersed or smaller overall, thus achieving expert selection.

[0063] In one implementation, the expert set includes several functionally differentiated experts, including: a routing expert for capturing domain-specific knowledge of speech or music; a sharing expert for learning cross-domain general knowledge across speech and music; and an empty expert for performing skip computation on simple lexical units.

[0064] In this embodiment, the expert pool comprises three types of experts: routing experts responsible for capturing domain-specific knowledge, shared experts learning cross-domain general knowledge, and empty experts who allow computation to be skipped for simple labels. In this hybrid expert structure, the number of routing experts, shared experts, and empty experts can be set to 8, 2, and 1, respectively, to decouple domain-specific features from cross-domain general features, and to enhance the model's adaptability by allowing computation to be skipped through empty experts.

[0065] In this embodiment, a shared expert is not mandatory. A separate, always-active shared expert can be replaced by a mechanism that forces partial parameter sharing among different routing experts. For example, it can be stipulated that the first few layers or specific modules of all experts, such as the attention projection layer, share the same set of parameters. Similarly, an empty expert can also be replaced. Specifically, instead of setting an explicit empty expert, a skip option can be added to the gating mechanism. When the gating selects skip, the input token will bypass the entire hybrid expert module directly through a residual connection. This also functionally achieves computation skipping.

[0066] Step S500: Based on the audio language modeling head, perform audio generation processing on the task adaptation features to obtain the resulting audio of speech or music.

[0067] In this embodiment, the audio language modeling head can adopt an autoregressive prediction architecture to predict discrete acoustic lexical sequences step-by-step based on task-adaptive features. This sequence can be in the same multi-channel format as the output of the aforementioned neural audio codec, i.e., 12 parallel channels, and the prediction process reuses previously generated lexical information to ensure temporal coherence.

[0068] Once the prediction is complete, the discrete acoustic word sequence is input into a neural audio codec of the same type to perform the decoding operation. The discrete words are restored to continuous audio waveforms through inverse transformation, and finally the audio result of speech or music is obtained.

[0069] Through the adaptation design of the audio language modeling head and codec, the transformation from task adaptation features to high-fidelity audio is realized. Furthermore, based on the domain signals contained in the task adaptation features, such as speech semantics or music structure, the corresponding type of audio can be adaptively generated to ensure the semantic clarity of speech and the integrity of music, thus completing the transformation from multimodal input to unified audio output.

[0070] In one implementation, the training steps for the attention head, attention head routing gating, expert set, and expert set routing gating in the audio model used to implement the generation method specifically include the following steps: Step S610: Train dense models on the speech-specific dataset and the music-specific dataset respectively to obtain domain prototype experts who have mastered the knowledge of the corresponding domains; Step S620: Perform balanced sampling on the speech-specific dataset and the music-specific dataset to obtain a balanced sampled dataset; Step S630: Integrate the domain prototype experts into the expert set, and use a parameter adjustment strategy on the balanced sampling dataset to train the expert set routing gate and the experts in the expert set who are used to learn cross-domain general knowledge; Step S640: Perform joint fine-tuning training on the parameters of the attention head, the attention head routing gating, the expert set, and the expert set routing gating.

[0071] In this embodiment, to resolve data imbalance and task conflicts, the following design is implemented: Figure 3 The three-stage training process is shown.

[0072] The first stage involves independent expert training, where dense models are trained on imbalanced datasets in their respective domains to form prototype experts. Structurally, the core computational unit of each prototype expert, the feedforward network layer (FFN), employs a multilayer perceptron (MLP) architecture with gated activation. In this embodiment, this stage of training uses the AdamW optimizer and a cosine learning rate scheduler, with a global batch size of 48 and a base learning rate of... .

[0073] Subsequently, balanced sampling was performed on the speech-specific dataset and the music-specific dataset to obtain a balanced sampled dataset with a relatively balanced amount of speech-specific data and music-specific data, which was used for subsequent training.

[0074] Finally, the second and third stages specifically include integrating the domain prototype experts into the expert set, using parameter tuning strategies on the balanced sampling dataset to train the expert set routing gating and the experts in the expert set used to learn cross-domain general knowledge, and performing joint fine-tuning training on the parameters of the attention head, the attention head routing gating, the expert set, and the expert set routing gating to obtain the trained consensus hybrid expert audio model.

[0075] In this final step of parameter tuning and joint fine-tuning training, a differentiated learning rate strategy can be used for end-to-end training. For example, in the initial stages of joint training, a very low learning rate can be set for the pre-trained experts, while a higher learning rate can be set for the randomly initialized gating and shared experts, and these learning rates can be dynamically adjusted as training progresses. Alternatively, a strategy such as... Figure 3 The training process shown is freezing, preheating, and thawing.

[0076] In one implementation, the parameter adjustment strategy and joint fine-tuning training specifically include the following steps: Step S631: Freeze the parameters of the domain prototype experts, and only update the parameters of the expert set routing gate and the experts in the expert set used to learn cross-domain general knowledge; Step S641: Unfreeze the frozen parameters. On the balanced sampling dataset, guide experts to make balanced use of the combined optimization objective of load balancing loss and main task loss, and jointly fine-tune and optimize the model training effect.

[0077] This embodiment illustrates the second and third stages of the training process of freezing, preheating, and thawing.

[0078] The second stage is the integration and warm-up of hybrid experts, which involves merging prototype experts into the hybrid expert architecture and freezing their weights, training the gating module and shared experts on the sampled balanced dataset, and stabilizing the routing and shared features.

[0079] Specifically, the feedforward network layers of each prototype expert pre-trained in the first stage are first extracted and integrated into a new hybrid expert layer as routing experts with specific knowledge, and their weights are temporarily frozen. Subsequently, on a small, sampled and balanced dataset, the model will be specifically trained on the newly introduced, randomly initialized components, namely the gating module responsible for expert selection and the shared expert responsible for learning general knowledge.

[0080] These experts, whether routing experts or sharing experts, all employ a multilayer perceptron architecture with gated activation, and their computational flow is represented as follows:

[0081] Wherein, input vector The dimension is 2048.

[0082] This calculation process involves three core weight matrices and a gated projection matrix. Upward projection matrix and downlink projection matrix The dimensions of these matrices vary depending on the type of expert.

[0083] For routing experts and All dimensions are ,and The dimension is .

[0084] For shared experts, the dimensions of these matrices are correspondingly [dimensions to be filled in]. and That is, the intermediate layer dimensions correspond to 2752 and 1376 respectively.

[0085] This warm-up step can stabilize the routing mechanism and pre-train shared features without interfering with the already learned expert knowledge, thereby effectively avoiding the catastrophic forgetting problem that may be caused by direct joint training.

[0086] The third stage involves unfreezing all parameters, fine-tuning them end-to-end on a large-scale balanced dataset, and using load balancing loss to achieve final collaborative joint training.

[0087] To calculate load balancing loss, we first define two key metrics.

[0088] The first key indicator is experts lexical allocation ratio This metric measures what percentage of tokens in a training batch are routed to the expert. The expression is:

[0089] in, Experts The word allocation ratio represents the experts The percentage of processed lexical units out of the total number of lexical units in the batch. This represents the total number of lexical units in a training batch. The index representing a word element, that is, the first word in the batch. Each word element, Indicates that the gating network is the first The original score vector output by each word element. This indicates that the function returns... The expert index corresponding to the element with the largest value in the vector, i.e., the term in the gating network. The chosen expert The index representing the expert, that is, the index representing the first expert in the expert pool. One expert, This indicates an indicator function; if the condition within the square brackets is true, i.e., a token... Assigned to experts If the expression is true, the function value is 1; otherwise, it is 0.

[0090] The second key indicator is experts Average routing probability This metric measures the efficiency of gating the network in a training batch by assigning it to experts. The average probability value is expressed as:

[0091] in, Indicates that the gating network is the first In the probability distribution of the output of the nth word, the nth word... The probability value corresponding to each expert It has passed deal with).

[0092] Based on the above two metrics, the final expression for the auxiliary load balancing loss is:

[0093] in, This represents the final value of the auxiliary load balancing loss. This indicates the total number of experts.

[0094] This formula penalizes unbalanced routing behavior by multiplying each expert's allocation proportion by the average probability and summing the results.

[0095] Finally, the auxiliary loss is combined with the main task loss to form the total loss function, which is expressed as:

[0096] in, , This represents the total loss, which is the final optimization goal during model training. This represents the main task loss, used to measure the model's prediction accuracy on the core task, i.e., the prediction accuracy of speech or music generation. It is the cross-entropy loss. The weighting coefficient represents the load balancing loss and is used to balance the two objectives of the main task and load balancing. This coefficient adopts a linear annealing strategy, with an initial value of 0.025, and decays linearly to a minimum value of 0.001 after 10,000 steps during training.

[0097] Ultimately, this loss function guides experts to leverage imbalanced data to gradually achieve cross-task collaborative learning and knowledge transfer. Through a systematic training paradigm, a three-stage training process—independent training, ensemble warm-up, and collaborative training—is proposed to avoid catastrophic forgetting and instability, fully utilize imbalanced data, and achieve smooth fusion and collaborative enhancement of cross-domain knowledge.

[0098] In summary, the technical solutions described in the above embodiments can simultaneously address the task conflict and data imbalance issues in speech and music generation. On the one hand, the combination of dynamic capacity hybrid expert architecture and hybrid attention allows the model to focus on domain features while maintaining cross-domain generality, effectively mitigating optimization objective conflicts. On the other hand, phased training ensures knowledge absorption and transfer under imbalanced data conditions, preventing speech data from excessively dominating music generation. This invention enhances the model's expressive power while maintaining computational efficiency, achieving an overall performance improvement and synergistic gain in the unified generation of speech and music.

[0099] like Figure 4As shown in the figure, this embodiment of the invention provides a speech and music collaborative generation system based on dynamic hybrid attention and expert architecture. The system includes: an input acquisition module 10, a fusion sequence acquisition module 20, a hybrid attention feature acquisition module 30, a task adaptation feature acquisition module 40, and a result audio acquisition module 50.

[0100] Specifically, the input acquisition module 10 is used to acquire multimodal input, which includes at least one of audio input, text input, and visual input; the fusion sequence acquisition module 20 is used to perform embedding representation processing on the multimodal input, converting the audio input into a discrete acoustic word sequence, the text input into a text embedding vector, and the visual input into soft visual tags, and mapping them to a unified embedding space to obtain a fusion sequence; the hybrid attention feature acquisition module 30 is used to set up several attention heads, each attention head dynamically selecting an attention head activation expert for the fusion sequence through attention head routing gating, and activating the attention head based on the attention head activation expert. The attention calculation result of the home and the gating probability of the attention head route gating are used to calculate the hybrid attention feature; the task adaptation feature acquisition module 40 is used to generate the expert probability distribution of the expert set through the pre-constructed expert set route gating, and based on the expert probability distribution of the expert set, use a dynamic routing strategy to select experts in the expert set to activate experts and calculate the task adaptation feature, wherein the expert set contains several experts with different functions, and the expert set includes at least experts who learn domain-specific knowledge and experts who learn cross-domain general knowledge; the result audio acquisition module 50 is used to perform audio generation processing on the task adaptation feature to obtain the result audio of speech or music.

[0101] Based on the above embodiments, the present invention also provides a terminal device, the principle block diagram of which can be as follows: Figure 5 As shown, the terminal device includes a processor, memory, network interface, display screen, and temperature sensor connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides the environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The network interface is used to communicate with external terminals via a network connection. When executed by the processor, the computer program implements a speech and music co-generation method based on dynamic hybrid attention and expert architecture. The display screen can be an LCD screen or an e-ink screen. The temperature sensor is pre-installed inside the terminal device to detect the operating temperature of the internal components.

[0102] Those skilled in the art will understand that Figure 5The schematic diagram shown is only a partial structural diagram related to the present invention and does not constitute a limitation on the terminal device to which the present invention is applied. The specific terminal device may include more or fewer components than shown in the figure, or combine certain components, or have different component arrangements.

[0103] In one embodiment, a terminal device is provided, including a memory and one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by one or more processors. The one or more programs include instructions for performing the following operations: Acquire multimodal input, wherein the multimodal input includes at least one of audio input, text input, and visual input; The multimodal input is processed by embedding representation, converting audio input into discrete acoustic word sequences, text input into text embedding vectors, and visual input into soft visual tags, which are then mapped to a unified embedding space to obtain a fused sequence. Several attention heads are set up. Each attention head is dynamically selected as an attention head activation expert for the fusion sequence through attention head routing gating. Based on the attention calculation results of the attention head activation experts and the gating probability of the attention head routing gating, the hybrid attention features are calculated. By using a pre-built expert set routing gating, an expert probability distribution of the expert set is generated. Based on the expert probability distribution of the expert set, a dynamic routing strategy is used to select experts to activate and calculate task adaptation features. The expert set contains several experts with different functions, and the expert set includes at least experts who learn domain-specific knowledge and experts who learn cross-domain general knowledge. Based on the audio language modeling head, the task adaptation features are processed to generate audio, resulting in speech or music.

[0104] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided by this invention can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), Rambus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0105] In summary, this invention discloses a method, system, terminal device, and medium for collaborative speech and music generation based on a dynamic hybrid attention and expert architecture, relating to the field of audio generation technology. The method first acquires multimodal input, including at least one of audio input, text input, and visual input. Then, it performs embedding representation processing on the multimodal input, converting the audio input into a discrete acoustic word sequence, the text input into a text embedding vector, and the visual input into soft visual tags, mapping them to a unified embedding space to obtain a fused sequence. Subsequently, several attention heads are set, each dynamically selecting an attention head activation expert for the fused sequence through attention head routing gating. Based on the attention calculation results of the attention head activation experts and the gating probability of the attention head routing gating, hybrid attention features are calculated. Next, an expert probability distribution is generated using a pre-constructed expert set routing gating system. Based on this distribution, a dynamic routing strategy is used to select and activate experts within the expert set, calculating task adaptation features. The expert set includes several functionally differentiated experts, at least those learning domain-specific knowledge and those learning cross-domain general knowledge. Finally, based on an audio language modeling head, the task adaptation features are processed to generate audio output for speech or music. This invention effectively decouples domain-specific knowledge and cross-domain general knowledge in speech and music by using a unified embedding mapping for multimodal inputs, a hybrid attention mechanism combining dynamic expert selection via an attention head routing gating system, and a design that dynamically selects functionally differentiated experts using an expert set routing gating system. This alleviates task conflicts in the unified generation of both. Furthermore, the dynamic routing strategy allocates computational resources based on lexical complexity, improving resource utilization and efficiency, ultimately achieving high-quality collaborative generation of speech and music.

[0106] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0107] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A method for co-generating speech and music based on dynamic hybrid attention and expert architecture, characterized in that, The method includes: Acquire multimodal input, wherein the multimodal input includes at least one of audio input, text input, and visual input; The multimodal input is processed by embedding representation, converting audio input into discrete acoustic word sequences, text input into text embedding vectors, and visual input into soft visual tags, which are then mapped to a unified embedding space to obtain a fused sequence. Several attention heads are set up. Each attention head is dynamically selected as an attention head activation expert for the fusion sequence through attention head routing gating. Based on the attention calculation results of the attention head activation experts and the gating probability of the attention head routing gating, the hybrid attention features are calculated. By using a pre-built expert set routing gating, an expert probability distribution of the expert set is generated. Based on the expert probability distribution of the expert set, a dynamic routing strategy is used to select experts to activate and calculate task adaptation features. The expert set contains several experts with different functions, and the expert set includes at least experts who learn domain-specific knowledge and experts who learn cross-domain general knowledge. Based on the audio language modeling head, the task adaptation features are processed to generate audio, resulting in speech or music.

2. The speech and music co-generation method based on dynamic hybrid attention and expert architecture according to claim 1, characterized in that, The embedding representation processing of the multimodal input involves converting audio input into a discrete acoustic word sequence, text input into a text embedding vector, and visual input into soft visual tags, which are then mapped to a unified embedding space to obtain a fused sequence, including: The audio waveform of the audio input is encoded using a neural audio codec, and cascaded prediction is performed through a delay mechanism to obtain a multi-channel discrete acoustic word sequence. The text input is segmented using a word segmenter and then converted into text embedding vectors in a unified embedding space through an embedding mapping layer. Visual input is encoded using a visual Transformer to obtain block features, which are then mapped to soft visual labels in a unified embedding space using a projector. The multi-channel discrete acoustic lexical sequence, the text embedding vector, and the soft visual markers are fused to obtain a fused sequence.

3. The speech and music co-generation method based on dynamic hybrid attention and expert architecture according to claim 1, characterized in that, The process involves setting up several attention heads, each of which uses an attention head routing gating system to dynamically select an attention head activation expert for the fused sequence. Based on the attention calculation results of the attention head activation experts and the gating probability of the attention head routing gating system, a hybrid attention feature is calculated, including: Several attention heads are set up, and each attention head is configured with an independent pool of attention head candidate experts. The attention head routing gate corresponds one-to-one with the attention head candidate expert pool. For each attention head, based on the fusion sequence, the expert probability distribution of the candidate expert pool of the attention head is calculated using attention head routing gating, and a TopN routing strategy is adopted to select a preset number of attention heads to activate experts; For each attention head, the parameter matrix of the selected attention head activation expert is invoked to perform attention calculation. The calculation result is multiplied element by element by the gating probability of the corresponding attention head route gating and then summed to obtain the output of a single attention head. The outputs of all attention heads are aggregated to obtain the hybrid attention features.

4. The speech and music co-generation method based on dynamic hybrid attention and expert architecture according to claim 1, characterized in that, The process involves generating an expert probability distribution for the expert set through a pre-built expert set routing gating system, and then using a dynamic routing strategy to select and activate experts based on this distribution, and calculating task adaptation features. This includes: The hybrid attention features are processed word-by-word using expert set routing gating, and the expert probability distribution of each word in the expert set is output. Using the Top-P dynamic routing strategy, the probability distribution of experts in the expert set is sorted and accumulated from high to low probability value until the accumulated probability reaches a preset threshold. Experts whose accumulated probability reaches the preset threshold are selected as active experts in the expert set. Based on the expert set activation experts, the hybrid attention features are calculated to obtain the output results of all the expert set activation experts; The outputs of all activated experts in the expert set are weighted and summed using normalized gating probabilities to obtain task adaptation features.

5. The speech and music co-generation method based on dynamic hybrid attention and expert architecture according to claim 1, characterized in that, The expert set includes several functionally differentiated experts, including: Routing experts are used to capture domain-specific knowledge specific to voice or music; Shared experts are used to learn cross-domain general knowledge across speech and music; Empty expert, used to perform skip computation on simple lexical units.

6. The speech and music co-generation method based on dynamic hybrid attention and expert architecture according to any one of claims 1-5, characterized in that, The training steps for the attention head, attention head routing gating, expert set, and expert set routing gating in the audio model used to implement the generation method include: Dense models are trained on both speech-specific and music-specific datasets to obtain domain prototype experts who possess knowledge of the corresponding domains. The speech-specific dataset and the music-specific dataset are subjected to balanced sampling to obtain a balanced sampled dataset. The domain prototype experts are integrated into the expert set, and a parameter adjustment strategy is used on the balanced sampling dataset to train the expert set routing gate and the experts in the expert set who are used to learn cross-domain general knowledge. Joint fine-tuning training is performed on the parameters of the attention head, the attention head routing gating, the expert set, and the expert set routing gating.

7. The speech and music co-generation method based on dynamic hybrid attention and expert architecture according to claim 6, characterized in that, The parameter adjustment strategy and joint fine-tuning training include: Freeze the parameters of the domain prototype experts and update only the parameters of the expert set routing gate and the experts in the expert set used to learn cross-domain general knowledge; Unfreeze the frozen parameters, and on the balanced sampling dataset, guide experts to make balanced use of the combined optimization objective of load balancing loss and main task loss, and jointly fine-tune and optimize the model training effect.

8. A speech and music co-generation system based on dynamic hybrid attention and expert architecture, characterized in that, The system includes: An input acquisition module is used to acquire multimodal input, wherein the multimodal input includes at least one of audio input, text input, and visual input; The fusion sequence acquisition module is used to perform embedding representation processing on the multimodal input, converting audio input into discrete acoustic word sequences, text input into text embedding vectors, and visual input into soft visual tags, and mapping them to a unified embedding space to obtain a fusion sequence; The hybrid attention feature acquisition module is used to set up several attention heads, each attention head dynamically selects an attention head activation expert for the fusion sequence through attention head routing gating, and calculates hybrid attention features based on the attention calculation results of the attention head activation experts and the gating probability of the attention head routing gating. The task adaptation feature acquisition module is used to generate the expert probability distribution of the expert set through a pre-built expert set routing gating, and to select experts to activate the expert set using a dynamic routing strategy based on the expert probability distribution of the expert set, and to calculate the task adaptation features. The expert set includes several experts with different functions, and the expert set includes at least experts who have learned domain-specific knowledge and experts who have learned cross-domain general knowledge. The result audio acquisition module is used to perform audio generation processing on the task adaptation features to obtain the result audio of speech or music.

9. A terminal device, characterized in that, The terminal device includes a memory, a processor, and a speech and music co-generation program based on dynamic hybrid attention and expert architecture stored in the memory and executable on the processor. When the processor executes the speech and music co-generation program based on dynamic hybrid attention and expert architecture, it implements the steps of the speech and music co-generation method based on dynamic hybrid attention and expert architecture as described in any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a speech and music co-generation program based on a dynamic hybrid attention and expert architecture. When the speech and music co-generation program based on a dynamic hybrid attention and expert architecture is executed by a processor, it implements the steps of the speech and music co-generation method based on a dynamic hybrid attention and expert architecture as described in any one of claims 1-7.

Images