A personalized song creation method based on multi-model cooperation

Abstract

The application discloses a kind of individualized song creation methods based on multi-model cooperation, it is related to audio creation technical field, including the following steps: S1, the natural language creation demand of user input, reference voice and music control parameter are obtained;S2, based on the natural language creation demand, constructs three-section Prompt guide structure and guides large language model to generate structured lyrics with paragraph mark, prosody mark and emotion orientation;S3, based on the structured lyrics, long-time audio of independent vocal track and accompaniment track is generated through two-stage Transformer architecture;The application constructs the whole-process automation collaborative architecture of "demand input-lyrics generation-music synthesis-voice color conversion-finished product output", solves the pain points of existing technology process fragmentation, human interface through standardized data interface and cross-module collaborative logic design, realizes from user natural language demand+3 second reference voice to the end-to-end output of individualized song finished product, greatly reduces the creation threshold.

Description

Technical Field

[0001] This invention relates to the field of audio creation technology, and in particular to a personalized song creation method based on multi-model collaboration. Background Technology

[0002] With the development of AI audio technology, the demand for automated song creation is becoming increasingly prominent. Currently, most mainstream song generation models focus on creating short audio clips of less than 30 seconds. To generate songs longer than 3 minutes, users need to manually splice multiple short audio clips, which is cumbersome. Traditional vocal presentation methods rely heavily on individual singing abilities, and most people's singing level is limited by their professional skills, making it difficult to achieve the desired effect. Relying on professional singers for recording is also costly. At the same time, the lyrics creation, melody generation, and vocal synthesis processes are independent of each other and need to be manually connected with the help of professional software, which requires a high level of professional skills from users.

[0003] The technical root cause of the above defects is: (1) Lack of long-term generation capability: The music generation adopts a single-track token modeling scheme, which makes it easy for the accompaniment signal to mask the semantic features of the human voice. Furthermore, it lacks a long-term structural association mechanism, which cannot guarantee the continuity between segments. As a result, songs longer than 3 minutes need to be spliced ​​manually, making it difficult to achieve the automatic generation of complete songs. (2) Zero-sample personalization bottleneck: The singing conversion technology lacks a targeted zero-sample architecture design, fails to achieve full-link optimization of "timbre stripping-feature extraction-precise generation", makes it difficult to capture fine-grained timbre features, requires a large amount of training data from target personnel, and cannot meet the needs of ordinary users for "3-second voice custom timbre". (3) Fragmented creation process: The creation process has not formed an end-to-end closed loop. The data formats of lyrics, melody, vocals and other links are incompatible and there is no unified collaborative optimization mechanism. Users need to master multiple professional software and manually connect each link, resulting in a very high creation threshold and making it difficult for ordinary users to operate. (4) Long-term audio quality defects: The generation of long-term audio does not adopt a layered modeling strategy, which cannot take into account both the overall structural coherence and the fidelity of local details, resulting in "one thing is not enough" and often causes problems such as structural chaos and sound quality loss, which cannot meet the high fidelity requirements. In summary, a personalized song creation method based on multi-model collaboration is proposed to solve the above problems. Summary of the Invention

[0004] The purpose of this invention is to solve the problems in the prior art by proposing a personalized song creation method based on multi-model collaboration.

[0005] A personalized song creation method based on multi-model collaboration includes the following steps: S1. Obtain the user's input of natural language creation requirements, and reference speech and music control parameters; S2. Based on the natural language creation requirements, a three-stage Prompt structure is constructed to guide the large language model to generate structured lyrics with paragraph markers, prosody annotations, and emotional guidance. S3. Based on the structured lyrics, generate long-duration audio of independent vocal tracks and accompaniment tracks through a two-stage Transformer architecture; S4. Based on the reference speech, the vocal track is timbre-converted using a zero-sample singing conversion model to obtain a vocal track with the target timbre. S5. Blend the converted target vocal track with the accompaniment track to output a personalized song.

[0006] Preferably, in step S1, the natural language creation requirements include style, emotion, theme, and language; the reference speech duration is greater than or equal to 3 seconds; and the music control parameters include genre selection, instrument combination, and vocal part type.

[0007] Preferably, in step S2, the three-segment Prompt structure includes: Requirements analysis: This is used to extract style, emotion, theme, and language elements from the stated natural language creation requirements; Format constraints: Used to specify the paragraph structure of the lyrics, the number of sentences in each paragraph, and the rhythmic requirements; Style example: Used to embed classic lyric fragments of the corresponding style as a reference.

[0008] Preferably, in step S3, based on the structured lyrics, a long-duration audio track containing independent vocal tracks and accompaniment tracks is generated through a two-stage Transformer architecture, specifically including: S31. The first stage, employing a two-stage Transformer architecture, involves music language modeling, including: Perform track decoupling and predict the next token, generating independent feature-encoded tokens for the vocal track and the accompaniment track respectively; Perform progressive conditional modeling by binding paragraph markers in the structured lyrics with musical structure markers and gradually inputting them into the model as contextual information to achieve temporal alignment between musical structure and lyric rhythm. Based on the input reference style fragments, and through music context learning, melodic features matching the reference style are collaboratively generated for the vocal track and the accompaniment track. The first stage outputs a semantically coarse-grained token; S32. The second stage, employing the two-stage Transformer architecture, performs residual modeling, including: Based on the semantic-level coarse-grained tokens, multiple sets of residual layer tokens are generated, which are used to characterize timbre texture, frequency dynamics, and overtone details. The semantic-level coarse-grained tokens are fused with multiple sets of residual layer tokens and reconstructed by a decoder into high-fidelity audio containing independent vocal tracks and accompaniment tracks.

[0009] Preferably, in step S4, based on the reference speech, the vocal track is subjected to timbre conversion using a zero-sample singing conversion model to obtain a vocal track with the target timbre, specifically including: (1) External timbre shift: Based on the pre-trained OpenVoiceV2 model, the source vocal track output in step S4 is processed by the spectrum perturbation algorithm to completely strip away the original timbre information and generate pure semantic features that retain only semantic and rhythmic features.

[0010] (2) Accurate extraction of timbre features: The pre-trained speaker verification model of the CAM++ architecture is called to extract features from the user's reference speech and generate fine-grained timbre vectors.

[0011] (3) Diffusion-based timbre generation: The Transformer with U-Net-style jump connection structure is used as the core generation network. The time embedding is used as the prefix token and the normalization input of the adaptive layer. Pure semantic features and target timbre vectors are fused. The target timbre vocal track is generated step by step through the flow matching diffusion algorithm to balance timbre similarity and audio naturalness.

[0012] (4) F0 adaptive adjustment: The F0 profile of the source vocal track is extracted in real time by the RMVPE algorithm, and the fundamental frequency is adaptively adjusted in combination with the music control parameters selected by the user.

[0013] Preferably, in step S5, the converted target vocal track is blended with the accompaniment track to output a personalized song, specifically including: Provides a visual interface to adjust the volume ratio between the target vocal track and the accompaniment track; The adjusted dual-track audio is combined into a final audio file, and can be exported in WAV or MP3 format. Perform noise reduction and loudness normalization on the exported audio files.

[0014] Preferably, the output loudness target of the loudness normalization process is -14 LUFS.

[0015] A personalized song creation system based on multi-model collaboration includes: The input module is used to obtain the user's input of natural language creation requirements, reference speech, and music control parameters; The lyrics generation module is built on a large language model and has a built-in Prompt template library. It is used to generate structured lyrics with paragraph marks, rhythm annotations and emotional guidance based on the needs of natural language creation. The long-duration music generation module is built on a two-stage Transformer architecture and includes four sub-modules: lyrics parsing, dual-track modeling, residual optimization, and lightweight upsampling. It is used to convert structured lyrics into long-duration audio containing independent vocal tracks and accompaniment tracks. The zero-sample singing conversion module is built on the diffusion Transformer architecture and integrates four sub-modules: external timbre shifting, speaker timbre extraction, diffusion conversion, and F0 adaptive adjustment. It is used to convert the timbre of the source vocal track based on the reference speech and output the target timbre vocal track. The audio fusion output module provides volume adjustment for vocals and accompaniment, supports exporting to the two mainstream formats of WAV and MP3, and is used to blend the target vocal track with the accompaniment track to output a personalized song. Preferably, the large language model in the lyrics generation module is DeepSeek.

[0016] Compared with existing technologies, the advantages of this invention are: 1. This invention constructs a fully automated collaborative architecture that integrates "demand input - lyrics generation - music synthesis - timbre conversion - finished product output". Through standardized data interfaces and cross-module collaborative logic design, it solves the pain points of existing technology processes being fragmented and requiring manual intervention, and realizes end-to-end output from user natural language demand + 3-second reference speech to personalized song finished product, greatly reducing the creative threshold.

[0017] 2. This invention constructs a full-link zero-sample singing conversion module consisting of "external timbre shifting - fine-grained timbre feature extraction - diffusion-type timbre generation - F0 adaptive adjustment". This invention only requires the user to provide the reference speech of the target speaker, and can complete the timbre conversion of the source human voice track without any pre-training data. It fundamentally breaks through the dependence of traditional singing conversion technology on large-scale target speaker training data, and significantly reduces the data threshold and economic cost of personalized song creation.

[0018] 3. This invention effectively solves the problems of chaotic song structure, disconnect between lyrics and music, and loss of sound quality caused by the fragmentation of the process and the limitation of short-term generation in the existing technology by adopting a two-stage Transformer architecture that integrates "track decoupling next token prediction" and "structural progressive conditional modeling". Attached Figure Description

[0019] Figure 1 This is a flowchart of the personalized song creation method of the present invention.

[0020] Figure 2 This is a structural diagram of the long-duration music generation module in this invention.

[0021] Figure 3 This is the flowchart of the zero-sample singing conversion part in this invention. Detailed Implementation

[0022] To facilitate understanding of this application and to make the aforementioned objectives, features, and advantages of this application more apparent, a detailed description of specific embodiments of this application is provided below in conjunction with the accompanying drawings. Numerous specific details are set forth in the following description to provide a thorough understanding of this application, and preferred embodiments are shown in the accompanying drawings. However, this application can be implemented in many different forms and is not limited to the embodiments described herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of the disclosure of this application. This application can be implemented in many other ways different from those described herein, and those skilled in the art can make similar modifications without departing from the spirit of this application; therefore, this application is not limited to the specific embodiments disclosed below. Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this application, "a plurality of" means at least two, such as two, three, etc., unless otherwise explicitly specified. In the description of this application, "several" means at least one, such as one, two, etc., unless otherwise explicitly specified. It should be noted that when an element is referred to as being "fixed to" another element, it can be directly attached to the other element or there may be an intervening element. When an element is referred to as being "connected to" another element, it can be directly connected to the other element or there may be an intervening element. The terms "vertical," "horizontal," "left," "right," and similar expressions used herein are for illustrative purposes only and do not represent the only possible implementations. Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is only for describing particular implementations and is not intended to limit the scope of this application. The term "and / or" as used herein includes any and all combinations of one or more of the associated listed items.

[0023] Reference Figure 1 As shown, a personalized song creation method based on multi-model collaboration includes the following steps: Step 1: Parameterized Input of Requirements - Personalized Basic Data Collection.

[0024] Users complete three core operations through a visual interface, providing complete parameter support for subsequent creation: ① Input natural language creation requirements (including style, such as pop / nursery rhyme; emotion, such as cheerful / lyrical; theme, such as birthday wishes / dream chasing; language, such as Chinese / English); ② Upload reference voice of the target speaker for more than 3 seconds (natural voice such as daily conversation, reading aloud, and advertising scripts are all acceptable, supporting MP3 / WAV format. The system automatically detects the audio duration and clarity, and provides a prompt if it is less than 3 seconds or the signal-to-noise ratio is insufficient); ③ Configure music control parameters (genre selection: supports switching between folk / rock / electronic / traditional Chinese style, etc.; instrument combination: supports free selection of 50+ instruments such as guitar / piano / drums; voice type: male / female / children's voices are selectable).

[0025] Step 2: Intelligent generation of structured lyrics – the core prerequisite for lyrics and music adaptation.

[0026] The lyrics generation module is user-demand oriented, precisely guiding the large language model through a three-stage Prompt process: ① Requirement analysis: Extracting core elements such as style and emotion from user input; ② Format constraints: Defining the lyric paragraph structure (intro / verse / chorus / bridge / outro) and the number of sentences in each paragraph, such as "2 sentences intro + 2 verse paragraphs (8 sentences per paragraph) + 2 chorus paragraphs (4 sentences per paragraph) + 2 sentences outro"; ③ Style examples: Embedding classic lyric fragments of the corresponding style as references. Ultimately, this guides DeepSeek to output structured lyrics with paragraph markers, rhythmic annotations (such as "Sentence 1: rhymes with 'ang', suitable for a 4-second melody"), and emotional guidance (such as "This needs to reflect a warm tone, the words should be gentle"), ensuring that the lyric length, rhythm, and melody are highly compatible with the rhythm of the subsequently generated music, avoiding "lyric-music mismatch".

[0027] Step 3: Long-duration dual-track music generation – the core technology for long-duration high-fidelity music.

[0028] This step utilizes a two-stage Transformer architecture to achieve layered modeling, balancing long-term structural coherence with detail fidelity, generating independent vocal and accompaniment tracks. This lays the foundation for subsequent personalized conversions. Figure 2 As shown: (1) First stage (Music Language Modeling): As the foundational stage for long-term music generation, it implements three core functions to ensure the clarity and structural coherence of the audio tracks: ① Track-decoupled Next-Token Prediction: Innovatively adopts a dual-output independent modeling structure of "voice token + accompaniment token", which completely separates the feature encoding of the voice and accompaniment, avoiding signal ambiguity caused by mixed encoding, and fundamentally improving the independent clarity of the two audio tracks; ② Structural Progressive Conditioning: Contextual information is gradually added at the lyric paragraph level (such as intro / verse / chorus, etc.), and the lyric paragraph markers are precisely bound to the music structure markers and embedded into the model input, ensuring that the music structure of the long song is highly aligned with the rhythm of the lyrics, effectively avoiding "lyrics and music out of sync"; ③ In-Context Learning (ICL) for Music: Supports dual-track reference input of voice and accompaniment. Based on the reference style fragments of the input, "style co-transformation" can be achieved (such as "voice and accompaniment synchronously from Japanese city style"). The system features both pop-to-English rap conversion and bidirectional generation capabilities (generating matching melodies for lyrics and adapting lyrics to melodies). The core output of this stage is a coarse-grained semantic token (codebook-0) at layer 0 of the audio, laying the foundation for subsequent processes.

[0029] (2) Second stage (Residual Modeling): Based on the semantic tokens output in the first stage, the focus is on improving sound quality and detail, while incorporating audio hierarchical representation and decoding logic: ① Residual layer generation: Based on the audio semantic token (codebook-0) output in Stage-1, seven sets of residual layer tokens, codebook 1 to codebook 7, are generated. These residual layers carry key information such as timbre texture, frequency dynamics, and overtone details, achieving a significant improvement in audio detail performance; ② High-fidelity audio reconstruction: Combining the audio discrete code hierarchical representation mechanism (layer 0 is semantic coarse-grained, used to align lyrics and music structure; layers 1-7 are detail-granular, responsible for optimizing dimensions such as timbre, frequency, and sound quality), the layer 0 token generated in Stage-1 is fully integrated with the layer 1-7 tokens supplemented in Stage-2, and a 44.1kHz / 16bit high-fidelity audio is reconstructed through a dedicated decoder to meet professional-grade listening requirements.

[0030] To address the issue of "disconnect between lyrics and music," the two-stage Transformer architecture avoids signal interference through parallel modeling of "voice + accompaniment." The progressive structural conditions bind lyric segments to musical structure markers, and the long-term attention mechanism ensures temporal alignment, with WER ≤ 12% and character error rate (CER) ≤ 3%.

[0031] Step 4: Zero-sample high-similarity vocal conversion – the key to achieving personalized timbre.

[0032] Using user-uploaded reference speech as the core, the voice conversion is completed through four collaborative steps, without requiring pre-training data from the target speaker. Figure 3 As shown: (1) External timbre shifting: Based on the pre-trained OpenVoiceV2 model, the source vocal track output by the long-time music generation module is processed by the spectrum perturbation algorithm to completely strip away the original timbre information and generate pure semantic features (S) that retain only semantic and rhythmic features. hi f te d) To avoid the "source timbre leakage" problem after conversion.

[0033] (2) Accurate extraction of timbre features: The pre-trained speaker verification model of the CAM++ architecture is called to extract features from the 3-second reference speech uploaded by the user and generate a 512-dimensional fixed-dimensional fine-grained timbre vector. This vector fully preserves the unique timbre features of the target speaker, such as breath, resonance position, and pronunciation habits, providing the core basis for high similarity conversion.

[0034] (3) Diffusion-style timbre generation: The Transformer (U-DiT) with U-Net-style jump connection structure is used as the core generation network. The time embedding is used as the prefix token and the normalization input of the adaptive layer. Pure semantic features and target timbre vectors are fused. The target timbre vocal track is generated step by step through the flow matching diffusion algorithm (the number of iteration steps is set to 100) to balance timbre similarity and audio naturalness.

[0035] (4) F0 adaptive adjustment: The F0 profile of the source vocal track is extracted in real time through the RMVPE algorithm. Combined with the vocal part type selected by the user, the fundamental frequency is automatically shifted when switching between genders (male to female +12 semitones, female to male -12 semitones). When switching between genders, the F0 fluctuation range is finely adjusted (±3 semitones) to ensure that the vocal melody and accompaniment are highly matched after the conversion. The root mean square error of F0 (F0RMSE) is controlled within 35 to avoid the problem of "out of tune".

[0036] To address the issue of "personalization requiring a large amount of data," an innovative end-to-end solution of "timbre stripping - precise extraction - diffusion generation" is adopted. The external timbre shifting submodule completely eliminates the residual timbre of the source human voice, the CAM++ model extracts fine-grained features from a 3-second reference speech, and the diffusion Transformer generates the target timbre based on the features. No pre-training data of the target speaker is required, enabling "any voice can sing."

[0037] Step 5: Audio Fusion and Multi-Format Output – The final step in delivering the finished product.

[0038] The system provides a visual audio editing interface, with core functions including: ① Volume balance adjustment: By default, the dual tracks are blended at a ratio of "40% vocals + 60% accompaniment". Users can customize the adjustment via a slider, and the blending effect can be previewed in real time; ② Format selection and export: Supports two mainstream formats: WAV (44.1kHz / 16bit, suitable for professional post-production) and MP3 (320kbps, suitable for daily distribution). During export, audio noise reduction and loudness normalization are automatically performed (loudness is set to -14LUFS); ③ Work management: Automatically saves user creation records, supports secondary editing (such as readjusting timbre and modifying lyrics) and sharing functions, meeting the needs of multiple usage scenarios.

[0039] A personalized song creation system based on multi-model collaboration includes: 1. Input module, used to obtain the user's input of natural language creation requirements, reference speech, and music control parameters. 2. Lyrics generation module: Built on the DeepSeek large language model, it has a built-in Prompt template library covering 10+ music styles and 8+ emotional dimensions. Its core function is to convert users' natural language needs (such as "Chinese nursery rhymes, birthday themes") into structured lyrics with paragraph markers (intro / verse / chorus / bridge / outro) and rhythm annotations, ensuring that the lyrics are highly compatible with the subsequent music generation process. 3. Long-duration music generation module: Based on a two-stage Transformer architecture, it includes four sub-modules: lyrics parsing, dual-track modeling, residual optimization, and lightweight upsampling. Its core function is to convert structured lyrics into a 44.1kHz high-fidelity complete song containing independent vocal tracks and accompaniment tracks. 4. Zero-sample singing conversion module: Based on the diffusion Transformer architecture, it integrates four sub-modules: external timbre shifting, speaker timbre extraction, diffusion conversion, and F0 adaptive adjustment. Its core function is to convert the vocal track of the generated song into the target timbre with only 3 seconds of target speaker reference speech, while keeping the lyrics and melody unchanged. 5. Audio Merging and Output Module: Provides adjustable volume ratio for vocals and accompaniment (default 40% vocals + 60% accompaniment), supports exporting to WAV / MP3 formats, and its core function is to achieve precise and personalized blending of vocals and accompaniment to output the final song. Inter-module collaboration and data flow: Each module achieves seamless connection through standardized data interfaces, forming a closed-loop collaboration: The structured lyrics output by the lyrics generation module are encapsulated in XML format and then sent to the long-duration music generation module; The long-duration music generation module outputs a dual-track audio file, in which the vocal track, combined with the user-input target timbre reference audio (natural speech of more than 3 seconds), is sent to the zero-sample vocal conversion module, and the accompaniment track is directly sent to the audio merging and output module; After the zero-sample vocal conversion module completes the conversion based on the characteristics of the target timbre reference audio, the output target timbre vocal track is merged with the accompaniment track in the audio merging module according to a preset volume ratio, and finally a personalized song product is generated.

[0040] Example To verify the practicality of the technical solution, the specific implementation effects are illustrated below in three core scenarios: Scenario 1: Customized Birthday Nursery Rhymes for Parents and Children - Parents create a birthday song exclusively for their 3-year-old child.

[0041] Operation Process: ① Input requirements: "Chinese language, nursery rhyme style, birthday blessing theme, cheerful mood"; ② Upload a 3-second audio clip of your child's daily conversation (content: "Mom, I want to eat cake", signal-to-noise ratio SNR=28dB); ③ Configure parameters: "BPM=100, xylophone + tambourine accompaniment, children's voice". System Processing Results: Generates a 4-minute 10-second personalized nursery rhyme with lyrics including playful content such as "Little baby, grow up quickly, candles light up little wishes". After conversion, the similarity between the voice and the child's original voice is SECS=0.89, the word error rate (WER) is 9.5%, there is no mechanical sound, and it can be directly used for birthday party playback. The entire operation by parents takes ≤8 minutes.

[0042] Scenario 2: Production of idol voice support songs - Fans create exclusive offline support songs for their idols.

[0043] Operation Process: ① Input requirements: "Chinese language, pop-rock style, dream-chasing theme, passionate emotion"; ② Upload a 10-second public interview audio clip of the idol (no singing component, SNR=25dB); ③ Configure parameters: "BPM=120, electric guitar + drums + bass accompaniment, male vocals". System Processing Effect: Generates a 3-minute 45-second support song with lyrics that fit the idol's growth experience. The converted vocals accurately reproduce the idol's vocal characteristics (SECS=0.87), and the singing and accompaniment blend naturally. It can be directly used for offline support activities or online platform distribution without the need for professional recording studio recording.

[0044] Scenario 3: Customized corporate brand advertising jingle - A milk tea brand created an advertising jingle featuring the voice of its spokesperson to promote its new product.

[0045] Operation Process: ① Input requirements: "Chinese, upbeat and trendy style, youthful and sweet theme, energetic and emotional"; ② Upload a 3-second audio clip of the brand ambassador's advertising script (SNR=30dB); ③ Configure parameters: "BPM=95, guitar + keyboard accompaniment, female vocals". System Processing Results: Generates a 2-minute 30-second advertising jingle, with lyrics incorporating brand selling points such as "silky smooth taste, refreshing aftertaste". The ambassador's voice similarity (SECS=0.86) and DNSMOS OVRL=3.15 are significantly improved. Compared to traditional advertising jingle recording (requiring the ambassador to spend over 2 hours in the studio, costing over 100,000 RMB), this solution takes ≤10 minutes, reduces costs by over 99%, and allows for rapid iteration and modification.

[0046] 1. Zero-sample timbre conversion innovation: Addressing the problem of "personalization requiring a large amount of data", an innovative full-link solution of "timbre stripping - precise extraction - diffusion generation" is adopted. The external timbre shifting submodule completely eliminates the residual timbre of the source human voice. The CAM++ model extracts fine-grained features from a 3-second reference speech. The diffusion Transformer generates the target timbre based on the features. No pre-training data of the target speaker is required, realizing "any voice can sing".

[0047] 2. Innovation in dual-track modeling and structural binding:

[0048] 3. End-to-end collaborative architecture innovation: To address the issue of "high barriers to creation", a fully automated architecture is built, which eliminates the need for manual intervention from user input to the output of the finished song. Ordinary users only need to know how to type and upload files to complete the creation.

[0049] 4. Innovative Layered Modeling and Lossless Upsampling: Addressing the issue of "poor quality over long periods," layered modeling employs "semantic-level tokens + residual optimization" to balance structure and detail. Vocos upsampling achieves lossless upsampling from 16kHz to 44.1kHz, with DNSMOSOVRL ≥ 3.1, ensuring a high-fidelity listening experience. This invention deeply couples three technologies: "lyrics generation and adaptation technology, long-duration music generation technology, and zero-sample vocal conversion technology," constructing a fully automated collaborative architecture that integrates "demand input - lyrics generation - music synthesis - timbre conversion - finished product output." Through standardized data interfaces and cross-module collaborative logic design, it solves the pain points of fragmented processes and the need for manual intervention in existing technologies, achieving end-to-end output from user natural language requirements plus 3 seconds of reference speech to a personalized song.

[0050] As is known from common technical knowledge, this invention can be implemented through other embodiments that do not depart from its spirit or essential characteristics. Therefore, the disclosed embodiments described above are merely illustrative in all respects and are not the only ones. All modifications within the scope of this invention or its equivalents are included in this invention.

Claims

1. A personalized song creation method based on multi-model collaboration, characterized in that: Includes the following steps: S1. Obtain the user's input of natural language creation requirements, and reference speech and music control parameters; S2. Based on the natural language creation requirements, a three-stage Prompt structure is constructed to guide the large language model to generate structured lyrics with paragraph markers, prosody annotations, and emotional guidance. S3. Based on the structured lyrics, generate long-duration audio of independent vocal tracks and accompaniment tracks through a two-stage Transformer architecture; S4. Based on the reference speech, the vocal track is timbre-converted using a zero-sample singing conversion model to obtain a vocal track with the target timbre. S5. Blend the converted target vocal track with the accompaniment track to output a personalized song.

2. The personalized song creation method based on multi-model collaboration according to claim 1, characterized in that: In step S1, the natural language creation requirements include style, emotion, theme, and language; the reference speech duration is greater than or equal to 3 seconds; and the music control parameters include genre selection, instrument combination, and voice type.

3. The personalized song creation method based on multi-model collaboration according to claim 1, characterized in that: In step S2, the three-stage Prompt boot structure includes: Requirements analysis: This is used to extract style, emotion, theme, and language elements from the stated natural language creation requirements; Format constraints: Used to specify the paragraph structure of the lyrics, the number of sentences in each paragraph, and the rhythmic requirements; Style example: Used to embed classic lyric fragments of the corresponding style as a reference.

4. The personalized song creation method based on multi-model collaboration according to claim 3, characterized in that: In step S3, based on the structured lyrics, a long-duration audio track containing independent vocal tracks and accompaniment tracks is generated using a two-stage Transformer architecture, specifically including: S31. The first stage, employing a two-stage Transformer architecture, involves music language modeling, including: Perform track decoupling and predict the next token, generating independent feature-encoded tokens for the vocal track and the accompaniment track respectively; Perform progressive conditional modeling by binding paragraph markers in the structured lyrics with musical structure markers and gradually inputting them into the model as contextual information to achieve temporal alignment between musical structure and lyric rhythm. Based on the input reference style fragments, and through music context learning, melodic features matching the reference style are collaboratively generated for the vocal track and the accompaniment track. The first stage outputs a semantically coarse-grained token; S32. The second stage, employing the two-stage Transformer architecture, performs residual modeling, including: Based on the semantic-level coarse-grained tokens, multiple sets of residual layer tokens are generated, which are used to characterize timbre texture, frequency dynamics, and overtone details. The semantic-level coarse-grained tokens are fused with multiple sets of residual layer tokens and reconstructed by a decoder into high-fidelity audio containing independent vocal tracks and accompaniment tracks.

5. The personalized song creation method based on multi-model collaboration according to claim 1, characterized in that: In step S4, based on the reference speech, the vocal track is timbre-converted using a zero-sample singing conversion model to obtain a vocal track with the target timbre, specifically including: (1) External timbre shift: Based on the pre-trained OpenVoiceV2 model, the source vocal track output in step S4 is processed by the spectrum perturbation algorithm to completely strip away the original timbre information and generate pure semantic features that retain only semantic and rhythmic features. (2) Accurate extraction of timbre features: The pre-trained speaker verification model of the CAM++ architecture is called to extract features from the user's reference speech and generate fine-grained timbre vectors. (3) Diffusion-based timbre generation: The Transformer with U-Net-style jump connection structure is used as the core generation network. The time embedding is used as the prefix token and the normalization input of the adaptive layer. Pure semantic features and target timbre vectors are fused. The target timbre vocal track is generated step by step through the flow matching diffusion algorithm to balance timbre similarity and audio naturalness. (4) F0 adaptive adjustment: The F0 profile of the source vocal track is extracted in real time by the RMVPE algorithm, and the fundamental frequency is adaptively adjusted in combination with the music control parameters selected by the user.

6. The personalized song creation method based on multi-model collaboration according to claim 1, characterized in that: In step S5, the converted target vocal track is blended with the accompaniment track to output a personalized song, specifically including: Provides a visual interface to adjust the volume ratio between the target vocal track and the accompaniment track; The adjusted dual-track audio is combined into a final audio file, and can be exported in WAV or MP3 format. Perform noise reduction and loudness normalization on the exported audio files.

7. The personalized song creation method based on multi-model collaboration according to claim 6, characterized in that: The loudness target of the loudness normalization process is -14 LUFS.

8. A personalized song creation system based on multi-model collaboration, characterized in that, include: The input module is used to obtain the user's input of natural language creation requirements, reference speech, and music control parameters; The lyrics generation module is built on a large language model and has a built-in Prompt template library. It is used to generate structured lyrics with paragraph marks, rhythm annotations and emotional guidance based on the needs of natural language creation. The long-duration music generation module is built on a two-stage Transformer architecture and includes four sub-modules: lyrics parsing, dual-track modeling, residual optimization, and lightweight upsampling. It is used to convert structured lyrics into long-duration audio containing independent vocal tracks and accompaniment tracks. The zero-sample singing conversion module is built on the diffusion Transformer architecture and integrates four sub-modules: external timbre shifting, speaker timbre extraction, diffusion conversion, and F0 adaptive adjustment. It is used to convert the timbre of the source vocal track based on the reference speech and output the target timbre vocal track. The audio fusion output module provides the function of adjusting the volume ratio of vocals and accompaniment, and supports exporting in two mainstream formats, WAV and MP3. It is used to merge the target vocal track with the accompaniment track to output a personalized song.

9. A personalized song creation system based on multi-model collaboration according to claim 8, characterized in that: The large language model used in the lyrics generation module is DeepSeek.

Images