Personalized chorus method based on sound ray analysis and terminal

Abstract

The invention provides a personalized chorus generation method based on sound ray analysis and a terminal. Comprising the following steps: receiving a user audio sample and modeling to generate sound parameters of a user; performing structure segmentation on the target song to obtain song fragments, and calculating the fragment difficulty of each song fragment; based on the sound parameters of the user and the fragment difficulty of the target song, matching and selecting a reference singer; and performing role allocation optimization on each song fragment, and determining singing states of the user and the reference singer in each song fragment. According to the method, personalized chorus generation based on three-party joint optimization of user sound rays, reference singers and song difficulty is realized, the problems of stiff adaptation, insufficient real-time performance, weak compliance and the like are solved, and chorus naturalness, interactive experience and commercial feasibility are remarkably improved.

Description

Technical Field

[0001] This invention relates to the field of digital content generation technology, and in particular to a personalized chorus method and terminal based on vocal analysis. Background Technology

[0002] With the development of intelligent music, human-computer interaction, and digital content generation technologies, the demand for personalized music experiences is growing. However, existing methods for generating and interacting with choruses have shortcomings in terms of user adaptability, real-time performance, allocation optimization, and compliance, impacting user interaction and commercial viability. Summary of the Invention

[0003] The technical problem to be solved by the present invention is to provide a personalized chorus method and terminal based on vocal analysis, which realizes adaptive segment-level role allocation and voice part generation of user vocal lines, and achieves low latency, natural alignment and traceability of personalized chorus.

[0004] A personalized chorus generation method based on vocal ray analysis, the method comprising: Receive user audio samples and model them to generate the user's voice parameters; The target song is structurally segmented to obtain song fragments, and the fragment difficulty of each song fragment is calculated; Based on the user's voice parameters and the difficulty of the segment of the target song, a reference singer is matched and selected; For each song segment, role allocation optimization is performed to determine the singing status of the user and the reference singer in each song segment.

[0005] To solve the above-mentioned technical problems, another technical solution adopted by the present invention is as follows: A personalized choral terminal based on vocal analysis includes a memory, a processor, and a computer program stored in the memory and running on the processor. When the processor executes the computer program, it performs the following steps: Receive user audio samples and model them to generate the user's voice parameters; The target song is structurally segmented to obtain song fragments, and the fragment difficulty of each song fragment is calculated; Based on the user's vocal parameters and the difficulty of the segment of the target song, a reference singer is matched and selected; For each song segment, role allocation optimization is performed to determine the singing status of the user and the reference singer in each song segment.

[0006] The beneficial effects of this invention are as follows: By constructing a full-link personalized chorus generation system consisting of user voice modeling, song difficulty quantification, reference singer matching, and segment role allocation, and by achieving adaptive decision-making based on a joint optimization mechanism of user voice parameters, song segment difficulty, and reference singer characteristics, the system maps user singing ability to song segment difficulty and dynamically allocates the singing role of each segment through optimization objectives and constraints. This process ensures that chorus generation is always dynamically adapted to the user's voice characteristics and real-time singing status, effectively avoiding problems such as abrupt singing, awkward transitions, and high failure rates caused by fixed templates or one-way matching. This guarantees that the final output chorus audio meets preset standards in terms of personalization, naturalness, and synchronization. The entire process is driven by data-driven joint optimization, ensuring that the chorus output quality meets musical art and engineering indicators while improving the intelligence level, real-time response capability, and commercial compliance of the generation process. Attached Figure Description

[0007] Figure 1 A flowchart illustrating the steps of a personalized choral singing method based on vocal ray analysis, provided in an embodiment of the present invention; Figure 2 A flowchart illustrating a personalized chorus method based on vocal timbre analysis applied in a real-time personalized chorus scenario on an online karaoke platform, as provided in an embodiment of the present invention. Figure 3 A flowchart illustrating a personalized chorus method based on vocal analysis applied to a live interactive scenario, as provided in an embodiment of the present invention; Figure 4 A flowchart of dynamic programming state transition for fragment-level allocation optimization provided in an embodiment of the present invention; Figure 5 A flowchart illustrating the real-time audio processing link and latency budget provided in this embodiment of the invention; Figure 6 A schematic diagram of the structure of a personalized choral terminal based on vocal ray analysis provided in an embodiment of the present invention; Label Explanation: 1. A personalized choral terminal based on vocal ray analysis; 2. Processor; 3. Memory. Detailed Implementation

[0008] To explain in detail the technical content, objectives, and effects of the present invention, the following description is provided in conjunction with the embodiments and accompanying drawings.

[0009] Please refer to Figure 1 A personalized chorus generation method based on vocal ray analysis includes steps 110 to 140.

[0010] Step 110: Receive user audio samples and model them to generate the user's voice parameters. The user's voice parameters are generated by building a model from short audio samples submitted by the user (e.g., 20-40 seconds of a cappella singing or reading).

[0011] Step 120: Structurally segment the target song to obtain song fragments, and calculate the fragment difficulty of each song fragment. For example, the system first divides the song into structural partitions, segments the target song into song fragments, and calculates the fragment difficulty of each song fragment.

[0012] Step 130: Match and select a reference singer based on the user's voice parameters and the difficulty of a segment of the target song. For example, select the most suitable reference singer from the singer reference library based on the user's voice parameters and the calculated difficulty of a segment of the target song.

[0013] Step 140: Optimize role assignment for each song segment to determine the performance status of the user and the reference singer in each song segment. For example, the system optimizes role assignment for each song segment, assigning each segment to the optimal performance status of a user solo, a reference singer solo, or a duet between the two.

[0014] As described above, the beneficial effects of this invention are as follows: By constructing a full-link personalized chorus generation system consisting of user voice modeling, song difficulty quantification, reference singer matching, and segment role allocation, it achieves joint optimization and adaptive decision-making based on user voice parameters, song segment difficulty, and reference singer characteristics. This system precisely maps user singing ability to song structure and dynamically allocates the singing role of each segment under multiple constraints through optimization algorithms. This ensures that the generation logic is always dynamically adjusted around user voice characteristics and real-time status, effectively solving problems such as abrupt singing, disjointed switching, and adaptation failures in traditional fixed template or one-way matching modes. It significantly improves the performance of choral audio in terms of personalized expression, auditory naturalness, and singing synchronization. This process, with data-driven and joint optimization as its core mechanism, ensures that the output results meet musical artistry and engineering indicators while comprehensively enhancing the system's intelligence, real-time interactive capabilities, and commercial compliance.

[0015] Further, step 110 includes steps 111 to 115.

[0016] Step 111: Preprocess user audio samples. The system processes short a cappella or reading audio samples from users to effectively suppress the influence of environmental noise and room acoustic reflections, extracting pure direct user audio and providing a clean signal source for subsequent high-precision acoustic feature analysis.

[0017] Step 112: Extract the fundamental frequency and energy profile from the preprocessed user audio samples to determine the user's comfortable vocal range. The system extracts the fundamental frequency trajectory and energy profile from the preprocessed user audio samples to determine the user's comfortable vocal range.

[0018] Specifically, the system performs the following process to extract the fundamental frequency and quantize the frequency range: 1. F0 fundamental frequency extraction: Analyze the 20-40 second a cappella sample submitted by the user frame by frame, extract the fundamental frequency value (F0) of each frame, and obtain a series of continuous frequency values ​​(such as 220Hz, 247Hz, 294Hz, etc.).

[0019] 2. Mapping to semitone bins: Discretizes continuous F0 frequency values ​​into MIDI note numbers in semitones according to international standard pitch. For example, 220Hz is mapped to A3 (MIDI number 57), 247Hz is mapped to B3 (number 59), 294Hz is mapped to D4 (number 62), etc., where middle C (C4) corresponds to number 60.

[0020] 3. Statistical Analysis and Screening of Stable Pitches: Statistical analysis is performed on all semitone bins appearing in the sample, including their frequency of occurrence, cumulative duration, and average energy level. To remove unstable or invalid vocalizations, the system applies screening rules: pitches with too short a duration (e.g., less than 0.1 seconds) (which may be glissando or noise), pitches with too low energy (which may be breathy or weak vocalizations), and pitches that appear too infrequently (e.g., <3 times) (which may be accidental).

[0021] 4. Determine the comfortable vocal range set and interval: After the above filtering, the set of semitones that the user can sing stably and clearly is obtained, denoted as the comfortable vocal range set S(U). For example, S(U) = {58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 69, 70}, corresponding to pitches A#3, B3, C4, C#4, D4, D#4, E4, F4, F#4, G4, A4, A#4, etc., respectively. Then, the minimum value L_U and the maximum value H_U are extracted from S(U) to form the user's comfortable vocal range interval [L_U, H_U]. In the example above, L_U = 58 (lowest comfortable pitch A#3), H_U = 70 (highest comfortable pitch A#4).

[0022] The practical significance of this quantized pitch range is that, in subsequent processing, if the pitch of a melody segment in a song exceeds the user's [H_U] or falls below [L_U], the segment is marked as high difficulty; if the pitch falls within the comfortable pitch range, it is considered a segment that the user can handle; for pitches near the boundaries, the difficulty can be reduced by assigning a duet. For example, if a melody segment in a song contains F5 (MIDI number 77), which far exceeds the user's example H_U=70, the system will automatically increase the difficulty score H_j of that segment, and during role allocation optimization, it will tend to assign that segment to a reference singer for solo singing, or arrange for the user to sing a duet with a reference singer with the latter handling the high notes, thereby achieving personalized singing adaptation based on a precisely quantized pitch range.

[0023] Step 113: Extract the user's timbre embedding vector and style embedding vector from the preprocessed user audio samples. This step extracts embedding vectors representing the user's timbre characteristics and singing style from the preprocessed user audio samples.

[0024] Step 114: Evaluate the user's rhythm and vocal ability parameters based on the preprocessed user audio samples. For example, analyze the user's maximum syllable rate, the proportion of large intervals they can tolerate, and beat stability index based on the preprocessed user audio samples to quantitatively evaluate the user's rhythm control ability and vocal function parameters.

[0025] Step 115: Generate structured sound parameters by integrating the comfortable vocal range, timbre embedding vector, style embedding vector, and the user's rhythm and vocal ability parameters. For example, by integrating the extracted comfortable vocal range, timbre embedding vector, style embedding vector, rhythm parameters, and vocal ability parameters, a structured set of user sound parameters is formed, providing a complete feature description for subsequent matching and optimization.

[0026] As described above, this invention, through multi-layer feature extraction and quantitative analysis, not only characterizes the user's static vocal range and timbre characteristics but also dynamically evaluates their singing rhythm and vocal ability, thereby constructing a comprehensive and fine-grained user voice model. This model overcomes the limitations of traditional methods that rely solely on single features such as pitch or timbre, laying a reliable data foundation for subsequent accurate song difficulty matching, singer matching, and role allocation, thus ensuring the personalization and adaptability of the chorus generation system from the source.

[0027] Further, step 120 includes steps 121 to 124.

[0028] Step 121: Detect the beats, strong beats, and measure positions of the target song, and divide the target song into song segments with measures as the smallest scheduling unit based on measure boundaries. For example, the system performs beat tracking and strong beat detection on the target song, identifies the complete beat grid and paragraph structure of the song, and logically divides the entire song into independent segment units based on the identified measure boundary positions, providing a structured basis for subsequent difficulty assessment and role assignment.

[0029] Step 122: Mark the switchable boundaries of each song segment. For example, based on the segmentation, the system combines pause markers in the lyrics, obvious silences or breathing gaps in the audio energy, and common musical phrase division rules in music theory to identify and mark the potential boundary positions in each segment where singing roles can be switched, ensuring that subsequent role allocation optimization can be adjusted in reasonable positions that conform to musical expression habits.

[0030] Step 123: Extract features for each song segment to represent singing difficulty. For example, for each song segment, the system analyzes its melody line to extract the range, the highest and lowest notes within the segment; calculates the proportion of major interval leaps; calculates the note density and syllable density per unit time; and evaluates its rhythmic complexity. These features together constitute the characteristics representing singing difficulty.

[0031] Step 124: For the extracted features, the system calculates the quantized difficulty value of each song segment by normalization and weighted summation. For example, the system first performs min-max normalization on each difficulty feature across the entire song to eliminate differences. Then, it performs linear weighted summation on the normalized features using pre-trained weight coefficients, and maps the weighted sum to a scalar value between 0 and 1 using a non-linear activation function (such as the sigmoid function). This value is the quantized difficulty score of the segment.

[0032] As described above, this invention achieves an accurate mapping from music signals to quantified difficulty indicators through meticulous structured analysis and feature extraction of songs. Unlike traditional methods that assess the difficulty of the entire song, this invention's segment-level difficulty quantification accurately reflects the dynamic changes in the singer's ability requirements across different sections of the song. This fine-grained difficulty characterization provides crucial data support for subsequent precise and adaptive role allocation optimization based on the user's actual vocal abilities, effectively avoiding problems such as disjointed performances or insufficient expressiveness caused by inappropriate difficulty matching, and improving the rationality and naturalness of personalized choral generation.

[0033] Further, step 130 includes steps 131 to 134.

[0034] Step 131: Calculate the overlap of vocal range, timbre similarity, and style similarity between the user and each preset singer. For example, the system compares the user's comfortable vocal range with the standard vocal range of the reference singer, and calculates the overlap of vocal range by the ratio of the intersection and union of sets; at the same time, based on the timbre embedding vector and style embedding vector of the user and the reference singer, it calculates their cosine similarity in high-dimensional space to obtain quantified timbre similarity and style matching degree.

[0035] Step 132: Based on the user's vocal parameters and the characteristics of the target song, evaluate the overall singability of the target song for the user. For example, the system comprehensively considers the distribution of rhythmic ability, vocal limits, and song segment difficulty features in the user's vocal parameters. By analyzing the degree of matching between the user's ability and the song requirements, it calculates the overall probability or suitability score of the user's ability to successfully perform the song, forming an overall singability index at the song level.

[0036] Step 133: Based on vocal range overlap, timbre similarity, style similarity, and overall singability, obtain a comprehensive score for each preset singer. For example, the system uses a preset weighted fusion strategy to fuse the scores of the above four dimensions, generating a single comprehensive matching score for each reference singer. The calculation formula reflects the relative importance of different dimensional features in singer selection.

[0037] Step 134: Select the preset singer with the highest overall score as the reference singer. For example, the system sorts all candidate reference singers by their overall scores and selects the singer with the highest score as the reference singer suitable for the current user and the target song.

[0038] As described above, the reference singer matching mechanism of this invention employs a multi-dimensional joint evaluation and optimization strategy. Unlike traditional techniques that simply screen based on vocal range or a single timbre label, this invention incorporates four key factors—vocal range matching, timbre adaptation, style fit, and overall song singability—into a unified evaluation framework, achieving precise selection through a quantitative comprehensive scoring mechanism. This multi-level, multi-dimensional matching method not only enhances the acoustic compatibility between the user and the reference singer but also ensures that the selected reference singer's style and ability simultaneously match the user's characteristics and the song's requirements. This provides a highly compatible reference benchmark for subsequent role allocation and chorus generation, significantly enhancing the harmony and artistic expression of the choral work.

[0039] Further, step 140 includes steps 141 to 143.

[0040] Step 141: Define a first decision variable and a second decision variable for each song segment. The first decision variable indicates whether the user is singing in the song segment, and the second decision variable indicates whether the reference singer is singing in the song segment. For example, the system uses binary variables. and These represent the user and the reference singer at the 1st rank, respectively. j The performance state in each segment (with a value of 0 or 1), and the combination conditions The final singing mode for each segment is determined: 1 indicates a solo performance, and 2 indicates a chorus. This representation method breaks free from the limitations of traditional "pseudo-roles" definitions, allowing the optimization model to more flexibly handle parallel decisions between solo and chorus performances.

[0041] Step 142: Based on the first and second decision variables of all song segments, construct an objective function. This objective function maximizes the overall singing adaptation benefit and minimizes the cost introduced by role switching and the penalty introduced by segments exceeding the user's singing ability. For example, the objective function can be expressed as:

[0042] This function achieves joint optimization of performance quality and performance feasibility by balancing the maximization of benefits and the minimization of penalties.

[0043] Step 143: Under preset optimization constraints, solve for the optimal allocation scheme using dynamic programming or integer linear programming. For example, the system imposes the following constraints during the solution process: role switching is only allowed at marked switchable boundaries; the number of consecutive solo segments performed by a user or reference singer must not be less than the preset minimum consecutive length. The proportion of choral segments to the total number of segments must fall within the range. Within; the total number of character switching throughout the entire song does not exceed [number missing]. Based on these constraints, the system can use a dynamic programming algorithm with state extension (states include the current singing role, continuous length, and number of switching times) or establish an integer linear programming model for efficient solution, and fall back to a heuristic algorithm in case of timeout to ensure real-time performance.

[0044] As described above, the role allocation optimization process of this invention constructs a rigorous and well-constrained mathematical optimization model. This model overcomes the rigidity of role definitions in traditional allocation methods by introducing parallel binary decision variables. Through the design of a multi-objective function integrating economic benefits, switching costs, and difficulty penalties, it achieves a multi-dimensional balance between artistic expression, performance feasibility, and auditory coherence. By integrating constraints such as musical structure boundaries, performance continuity, and chorus ratios, it ensures that the generated allocation scheme conforms to the laws of musical expression and adapts to the user's actual abilities. This systematic optimization framework allows chorus generation to move beyond fixed templates or empirical rules, instead relying on quantitative indicators and optimization algorithms for adaptive decision-making, thereby fundamentally improving the rationality, naturalness, and success rate of personalized chorus performances.

[0045] Furthermore, step 140 also includes steps 144 to 146.

[0046] Step 144: If a singing synthesis path is adopted, the corresponding melody is synthesized based on the acoustic features of the reference singer, and a lookback window mechanism is used to pre-render and cache the audio for the subsequent preset number of measures. For example, according to the role allocation scheme, in the segment that requires the reference singer to sing, the system uses a singing synthesis model trained based on the acoustic parameters of the selected singer to synthesize a reference voice that perfectly matches the corresponding melody and lyrics. To ensure real-time performance, the system adopts a scrolling window mechanism: while processing the current segment, the synthesized audio for the next 2-4 measures is pre-rendered and cached synchronously; when the user experiences temporary tempo changes during singing (such as dragging or rushing the beat), the system will trigger a low-overhead re-render at the next measure or phrase boundary to realign the pre-cached content and ensure choral synchronization.

[0047] Step 145: If the vocal conversion path is adopted, the authorized dry vocal material of the song is called, and the timbre of the dry vocal material is converted into the timbre of the reference singer through a timbre conversion model to generate the reference vocal part. For example, for songs with official copyrights, the system can directly call its officially released dry audio as the source signal. Through a timbre conversion model trained based on RVC, So-VITS, etc., the timbre characteristics of the source dry vocal material are converted into the timbre characteristics of the reference singer, while keeping the melody, rhythm and lyrics of the source audio unchanged. If there is no official dry vocal material, vocal separation technology (such as Demucs / HT-Demucs) can be used to extract the dry vocal material from the complete mix, and the extraction result is used for online conversion to improve sound quality fidelity and real-time performance.

[0048] Step 146: If harmonic parts need to be generated, a multi-part accompaniment matching the main melody of the target song is generated based on the song's chord information or through a model. For example, when the role allocation scheme includes harmonic accompaniment requirements, the system can generate a secondary melody line that forms a specific interval relationship with the original melody based on the song's chord progression and harmonic rules. Alternatively, a neural harmony generation model based on attention mechanisms or diffusion models can be used to directly generate a richer and more artistically expressive multi-part accompaniment based on the main melody audio or symbolic representation.

[0049] As described above, this invention integrates three technical paths—voice synthesis, vocal conversion, and harmony generation—to construct a unified, flexible, and efficient reference voice generation architecture. This architecture can automatically select the optimal generation path based on song copyright status, real-time requirements, sound quality requirements, and artistic expression needs. A scrolling pre-rendering and caching mechanism ensures a low-latency, real-time interactive experience; timbre conversion of licensed dry vocals guarantees sound quality and copyright compliance; and a hybrid harmony generation method combining rules and neural networks enhances the artistic expressiveness of the chorus. This multi-path fusion design allows the system to adapt to a wide range of application scenarios, achieving a multi-objective balance between generation efficiency, sound fidelity, and artistic expression while ensuring output quality.

[0050] Furthermore, it also includes steps 150 to 152.

[0051] Step 150: Employ a dynamic time warping algorithm with a limited window length to fine-tune and align the user's vocal parts with the reference vocal parts of the reference singer. For example, the system uses a dynamic time warping algorithm with a limited window length to continuously compare the user's real-time audio clips with pre-rendered reference vocal parts within a limited review window of no more than 40 milliseconds while the user is streaming. This algorithm achieves dynamic time scaling and alignment by performing local path search based on the similarity of the two audio clips in the short-time spectrum, thereby correcting minor asynchronys caused by subtle rhythmic differences or instantaneous fluctuations between the user and the reference vocal parts.

[0052] Step 151: Apply audio processing to the user's vocal part and the reference part in parallel, and perform sidechain energy control. For example, the system sends the user's vocal stream and the aligned reference part into a multi-channel audio processing link in parallel. The system uses a sidechain triggering mechanism, in which the reference part modulates the user's compressor threshold. When the reference part enters, the gain of the user channel is automatically reduced, and vice versa, thereby intelligently adjusting the relative loudness and prominence of the two in the mix to ensure the clarity and balance of the chorus.

[0053] Step 152: Control the total latency of the fine-tuning alignment process and the audio processing process within a preset threshold. For example, the system performs fine-grained latency management across the entire process: input / output and buffer latency is controlled within 5-6 milliseconds, pitch analysis and dynamic time warping latency is limited to 10-20 milliseconds, dynamic equalization and effects processing latency is less than 2 milliseconds, and spatial effects latency such as reverb is adjustable from 0-10 milliseconds. Through pipelined and parallel computing design, the system ensures that the end-to-end total latency from user vocal input to final stereo mix output strictly does not exceed 40 milliseconds. When computing resources are strained, the system initiates degradation strategies, including disabling computationally expensive dynamic time warping fine-tuning, shortening the pre-rendered buffer length, or using simpler alignment algorithms (such as beat-locked loops), to maintain the baseline performance of real-time interaction.

[0054] As described above, the real-time alignment and mixing control module of this invention, through dynamic time warping with a limited window length, multimodal parallel audio processing, and refined latency budget management, enables sub-frame-level dynamic tracking and correction of reference voices while allowing users to retain subtle changes in natural singing rhythm. Furthermore, through professional mixing techniques such as sidechain energy control, it intelligently handles the acoustic interaction between the user and the reference voices. This solves the fundamental problem that window rendering or offline alignment methods cannot meet the needs of real-time interaction. Moreover, while ensuring extremely low latency, it effectively improves the performance of choral works in terms of time synchronization, sound quality integration, and naturalness of sound, providing a technically feasible guarantee for highly interactive scenarios and enhancing the user experience and commercial application value of the system.

[0055] Furthermore, step 150 also includes step 153.

[0056] Step 153: If the deviation between the user's actual singing speed and the preset speed exceeds the tolerance limit, a re-rendering and cache update of the subsequent reference voice parts is triggered at the boundary of the next song segment corresponding to the current song segment. For example, by monitoring the phase drift or timing shift between the user's singing stream and the original beat grid of the song in real time, when the deviation between the overall speed or local rhythm of the user's singing and the preset speed exceeds a preset threshold, the system does not immediately interrupt the current processing, but marks the state change. Based on the music structure prediction, the system interrupts the original rolling pre-rendering pipeline at the upcoming next predefined switching boundary (such as the next bar line, the end of a musical phrase, or a significant breathing point). Subsequently, the system uses the user's current real-time singing speed as the new reference speed to quickly re-render the reference voice parts for the next predetermined number of measures (such as 1-2 measures) and update the audio cache. This strategy ensures a smooth transition of speed changes, avoids perceptible stuttering or breaks caused by forced switching in the middle of a musical phrase, and limits the computational overhead of re-rendering to a controllable range, maintaining the overall real-time performance of the system.

[0057] As described above, this invention addresses the common real-time tempo variations experienced by users in live performances. By delaying decision-making to the natural boundaries of the musical structure and performing local re-rendering at these boundaries using a new tempo benchmark, it cleverly balances the real-time requirements of vocal synchronization with the coherence requirements of audio generation. This effectively overcomes two extreme approaches commonly found in traditional systems when dealing with tempo changes: one is completely ignoring the change, leading to severe alignment misalignment; the other is high latency and resource fluctuations caused by immediate and frequent full re-rendering. It ensures that even in scenarios where users improvise and the rhythm is unstable, the system maintains the synchronization and auditory naturalness of the chorus, improving the system's robustness to complex and non-ideal vocal inputs, and making the personalized choral experience more realistic, inclusive, and expressive.

[0058] Furthermore, it also includes steps 160 and 161.

[0059] Step 160: Associate and bind the reference singer's authorization information, the unique number of this generation task, and the timestamp. For example, before executing the chorus generation task, the system first verifies the authorization status of the reference singer's acoustic model or dry audio (including scope of use, validity period, applicable scenarios, etc.), and encapsulates and associates this authorization metadata, the unique task identifier assigned by the system to this generation session, and the precise timestamp into the task context. This binding relationship follows the audio data processing flow throughout and is ultimately recorded in the system log and task metadata database, forming an immutable traceability chain, providing a structured basis for copyright verification and usage auditing.

[0060] Step 161: Embed a frequency-domain spread spectrum or neural network watermark into the final output mixed audio signal. For example, in the export stage of generating the final stereo mix or multi-track audio, the system uses a watermark embedding engine to encode the generated associated binding information into an imperceptible digital watermark signal. Specifically, a frequency-domain watermarking algorithm based on spread spectrum technology can be used to disperse and hide the encoded information in the phase or amplitude of multiple frequency bands of the audio; or a neural network watermarking model based on adversarial training can be used to directly modulate the information into the deep features of the audio waveform. The original information can be extracted and restored through a corresponding dedicated decoder or verification service, thereby achieving traceable copyright authentication and piracy tracing of the generated content.

[0061] As described above, this invention systematically addresses core compliance challenges faced by generated content in commercial applications, such as unclear copyright, unknown origin, and difficulty in investigating piracy, by constructing a full-chain compliance and traceability management system that integrates authorization binding and watermark embedding. Real-time binding of authorization information ensures that every content generation occurs within a legally authorized framework. Furthermore, embedding unique task identifiers and timestamps, along with other metadata, imperceptibly into the audio content itself using watermarking technology achieves a physical binding between the content and its source information. This allows key traceability information, such as the source of the audio, the model used, and the generation time, to be effectively verified and extracted even if the audio is copied, disseminated, or reprocessed. This not only provides content creators and platforms with a powerful copyright protection tool and reduces legal risks but also establishes a trustworthy content production and distribution environment through technological means, laying a solid foundation for the large-scale and compliant commercial application of personalized chorus generation technology.

[0062] In summary, this invention constructs a personalized chorus generation method based on vocal analysis. This method integrates multi-source data from user voice modeling, song structure analysis, and reference singer feature matching to build a dynamic optimization framework centered on the user's real-time singing state, achieving intelligent adaptation to the entire chorus generation process. The system can automatically adapt the corresponding generation path and processing strategy according to different application scenarios. For music segments of varying difficulty levels and differences in user abilities, the system can output differentiated role allocation schemes.

[0063] Simultaneously, based on the song's structural features and the user's singing ability, the system obtains reference singers who match specific musical segments and meet artistic performance requirements as the baseline for voice part generation, thereby generating personalized choral content adapted to user characteristics and song requirements. In terms of feature processing, the system employs a hierarchical extraction strategy for various acoustic parameters generated during the user's singing process, collecting relevant features according to different matching needs (e.g., fundamental frequency distribution is mainly collected for range matching, while timbre and singing habits are collected for style matching). Based on the varying degrees of influence of different features on the choral effect, the system integrates the matching results of each feature according to preset weights to obtain a comprehensive adaptation score. This method differs from traditional template-based choral generation methods; through multi-dimensional joint optimization, it can more accurately match users and reference singers, significantly improving the naturalness and harmony of the chorus.

[0064] Furthermore, during the chorus generation process, the system also considers the performance guarantee of real-time interaction, establishing a dual latency control mechanism of pre-rendering caching and streaming processing. When a performance anomaly is detected, the system can trigger a corresponding degradation strategy based on the severity of the anomaly, ensuring the real-time availability of the generated chorus and thus achieving a balance between artistry and fluency in the chorus effect.

[0065] This system is suitable for scenarios requiring personalized choral generation. By constructing a complete intelligent choral generation system, it effectively enhances the artistic expression of choral works and strengthens the user experience.

[0066] See attached document Figure 2 The following describes specific application embodiments of this application. This application can apply the above solution to real-time personalized duet scenarios on online karaoke platforms, including the following steps: 1. The user opens a karaoke app on their mobile device and records approximately 20-40 seconds of a cappella singing and reading samples using their microphone. After processing by the audio preprocessing module, the fundamental frequency, energy, timbre, and style features are extracted to generate structured user vocal parameters, which are then uploaded to the server. This is equivalent to steps 110 and 111 above.

[0067] 2. After receiving the user parameters, the server performs beat detection, measure segmentation, and segment difficulty quantification on the song selected by the user. Simultaneously, based on the user's vocal parameters and the song's difficulty, it matches the most suitable singer from the reference singer database, calculates a comprehensive score, and selects the reference singer. This is equivalent to the matching and selection parts in steps 120, 130, and 140 above.

[0068] 3. Based on the segment difficulty, user ability, and singer matching results, the server uses a dynamic programming algorithm to solve for the optimal role allocation scheme under constraints such as switchable boundaries, minimum continuous length, and chorus ratio. This determines whether each segment is a user solo, a reference singer solo, or a chorus, and outputs the allocation result. This is equivalent to step 160 above.

[0069] 4. According to the allocation scheme, the server starts the multi-path voice part generation engine: if the singing synthesis path is used, the corresponding melody is synthesized based on the reference singer's style and pre-rendered; if the singing conversion path is used, the authorized dry voice is called for timbre conversion; if harmony is required, a multi-part accompaniment is generated based on the chords. The generated reference voice parts are temporarily stored in the cache. This is equivalent to step 170 above.

[0070] 5. The user begins singing, and the client captures the user's audio in real time and streams it. The server uses short-window dynamic time warping to micro-align the user's singing with the reference vocals, applies equalization, compression, and sidechain control, and keeps the end-to-end latency within 40 milliseconds. The mixed audio is then sent back to the user in real time. If a change in the user's singing speed is detected, the system triggers a local re-render at the next measure boundary to maintain synchronization. This is equivalent to step 180 above.

[0071] 6. The final output choral audio is embedded with a verifiable watermark containing the reference singer's authorization information, task ID, and timestamp during export. Related logs and metadata are recorded on the server, supporting subsequent copyright verification and data management. This is equivalent to step 190 above.

[0072] Through the above application examples, the present invention realizes a complete personalized chorus generation process from user vocal analysis, song difficulty quantification, singer intelligent matching, dynamic role allocation to real-time rendering and compliant output. It effectively solves the problems of existing chorus systems lacking personalized adaptation, insufficient real-time performance, and weak copyright management. While improving the naturalness and artistic expression of chorus, it enhances the system's real-time interactive capabilities and commercial compliance.

[0073] Please refer to the appendix. Figure 3 The following describes specific application examples of this application, which can apply the above solution to personalized chorus in live interactive scenarios. Taking a user participating in a real-time chorus live event as an example, the steps include: 1. During the live stream, the user selects the song "The Brightest Star in the Night Sky" and starts the duet mode. The system captures approximately 30 seconds of the user's a cappella singing and recitation audio via microphone in real time. After preprocessing by the terminal noise reduction module, the system extracts fundamental frequency contours, energy distribution, timbre embedding, and style embedding features to generate structured vocal parameters that include the user's comfortable vocal range [#C3-#F4], timbre vectors, and rhythmic stability. These parameters are then uploaded to the cloud server. This is equivalent to steps 110 to 111 above.

[0074] 2. After receiving the user parameters, the cloud server performs beat detection and measure segmentation on the target song, dividing it into 32 segments. The system extracts features such as vocal range, large leap ratio, and note density for each segment, and calculates the difficulty value of each segment through normalization and weighted calculation (e.g., H=0.82 for the chorus segment, H=0.31 for the verse segment). Simultaneously, based on the user's vocal range and style characteristics, the system matches the singer "Singer_A" from the reference singer database, with a timbre similarity of 0.89 and a style matching degree of 0.85, as a reference singer. This is equivalent to steps 120 to 130 above.

[0075] 3. Based on the segment difficulty distribution, user singing ability thresholds, and reference singer suitability, the server uses dynamic programming to derive an optimized allocation scheme under constraints such as "switching is only allowed at bar lines" and "the chorus ratio is controlled between 30% and 50%": bars 1-8 are for user solos, bars 9-16 are for choruses, bars 17-24 are for reference singer solos, and bars 25-32 resume choruses. This is equivalent to steps 140 to 143 above.

[0076] 4. Based on the allocation results, the system initiates multi-path voice part generation: For choral and reference singer solo segments, a singing synthesis engine based on the "Singer_A" acoustic model is used to pre-render and cache the subsequent 4 bars of audio; for user solo segments, harmonic voice parts based on chord progressions are generated. All voice parts undergo formant correction and metallicity suppression post-processing to output a high-quality reference audio stream. This is equivalent to steps 144 to 146 above.

[0077] 5. The user begins singing in real time, and the client uploads the user's singing data in a streaming manner. The server uses a dynamic time warping algorithm with a window length of 40 milliseconds to perform frame-by-frame micro-alignment between the user stream and the reference audio, and dynamically balances the volume of both through sidechain compression. The overall processing pipeline latency is controlled within 38 milliseconds, and the mixed chorus audio is pushed to the live stream in real time. This is equivalent to steps 150 to 152 above.

[0078] 6. After the live stream ends, the system embeds a frequency-domain spread spectrum watermark into the exported final audio. The watermark information includes the "Singer_A" model authorization ID, the task number "TC-20231027-001", and a timestamp. Related generation logs, user data, and authorization records are all encrypted and stored, supporting subsequent copyright tracing and compliance auditing. This is equivalent to steps 160 to 161 above.

[0079] Through the above application embodiments, this application realizes a personalized chorus generation system in highly interactive, low-latency online live streaming scenarios, completing a complete technical loop from user vocal analysis, song difficulty quantification, singer matching, intelligent allocation to real-time rendering and watermark embedding. This solution effectively solves the problems of rigid traditional chorus templates, difficulties in real-time alignment, and lack of copyright management, significantly improving the naturalness, synchronization, and commercial compliance of chorus generation, providing reliable technical support for highly interactive music entertainment scenarios.

[0080] Please refer to the appendix. Figure 4 The following specific application example further illustrates the implementation mechanism of this application in the role allocation optimization step. This example, combined with the illustrated state transition model and constraints, demonstrates how the system performs dynamic role decisions at the fragment level. The specific process is as follows: 1. State space definition and initialization: The system defines a binary state variable for each segment j in the song. ,in Indicates whether the user sings in this segment. This indicates whether the reference singer is performing in this segment. Initially, the first segment is set to be a user solo, i.e. This state representation method breaks away from the traditional "pseudo-role" definition and supports the parallel expression of three singing modes: user solo, reference singer solo, and chorus. This is equivalent to step 141.

[0081] 2. State transition and cost calculation: As shown in the attached diagram, the system makes state transition decisions between adjacent segments. If the state remains unchanged, no switching cost is incurred; if the state changes, a corresponding switching cost is introduced. For example, switching from a user solo (1,0) to a reference singer solo (0,1) or a chorus (1,1) both incur switching costs to encourage performance continuity. The system considers performance adaptation benefits, switching costs, and segment difficulty exceeding limits in its optimization objective function, achieving a balance among multiple objectives. This is equivalent to step 142.

[0082] 3. Dynamic programming solution and constraint application: The system employs a dynamic programming algorithm to solve for the globally optimal state sequence under the following constraints: (1) Switching boundary constraints: State switching is only allowed on the marked switchable boundaries. Proceed at key points, such as bar lines, pauses, or obvious breathing points, to avoid breaks in the middle of musical phrases.

[0083] (2) Minimum continuous length constraint: The solo performance state of the user or reference singer must be maintained continuously for at least This is done in segments to prevent the performance from becoming fragmented.

[0084] (3) Chorus ratio constraint: The proportion of chorus segments to the total number of segments needs to be controlled within the range. Internally, it aims to balance individuality and harmony.

[0085] (4) Maximum number of state transitions constraint: The total number of state transitions in the entire song shall not exceed This ensures structural stability. This set of constraints corresponds to the preset optimization constraints described in step 143.

[0086] 4. Example of output allocation scheme: In a specific instance, based on the above model and constraints, the system outputs the following optimized allocation scheme: Measures 1-8 remain solo performances; measures 9-16 switch to chorus; measures 17-24 switch to a reference singer solo; and measures 25-32 resume chorus performances. This scheme strictly adheres to the boundary switching principle and satisfies the minimum continuous length, chorus ratio, and switching frequency limits, achieving a reasonable allocation of singing difficulty while maintaining naturalness. This is equivalent to the part in step 143 that solves and outputs the allocation result using dynamic programming or integer linear programming.

[0087] This embodiment demonstrates that, by establishing a dynamic programming optimization model based on state transitions and imposing various constraints based on music structure awareness, this application achieves intelligent, dynamic, and artistically sound allocation of singing roles at the segment level. This method effectively solves the problems of abrupt switching, poor coherence, and unreasonable adaptation in traditional template allocation, providing reliable and interpretable decision support for personalized choral generation, and further improving the overall system performance and user experience.

[0088] Please refer to the appendix. Figure 5 The diagram illustrates the real-time audio processing link and latency budget. The following specific application example demonstrates the latency control mechanism of this application in the real-time alignment and mixing steps. Based on the latency allocation and processing module shown in the diagram, this example demonstrates how the system achieves end-to-end low-latency chorus generation of less than or equal to 40 milliseconds in a live streaming scenario. The specific process is as follows: 1. The user sings in real time through a microphone. The audio signal is captured by the client and transmitted to the server via the uplink. The system first performs audio I / O interface processing and buffer management. The latency in this stage is controlled within 5–6 milliseconds to ensure stable signal input. This is equivalent to the audio streaming reception and buffering stage in step 150.

[0089] 2. The system employs a dynamic time warping algorithm with a limited window length (≤40 milliseconds) to perform frame-by-frame micro-alignment between the user's real-time audio stream and the pre-rendered reference voice, while simultaneously performing pitch correction. The latency budget for this core alignment module is 10–20 milliseconds. When system resources are scarce or the load is too high, a degradation strategy can be initiated, such as disabling DTW or shortening the alignment window, to prioritize real-time performance. This is equivalent to steps 150 and 152.

[0090] 3. The aligned dual-channel audio enters the parallel processing pipeline, where dynamic equalization, multi-band compression, sibilance suppression, and limiting are performed sequentially. The total latency of this stage does not exceed 2 milliseconds. The system employs a sidechain control mechanism to enable the reference voice to dynamically respond to changes in the user's singing energy, achieving intelligent volume balance between the two. To enhance the spatial sense and integration of the chorus, the system applies adjustable reverb and stereo field processing to the mixed audio. This module allows for selectable latency of 0–10 milliseconds, which can be dynamically adjusted according to network conditions and real-time requirements. This is equivalent to step 151.

[0091] 4. The reference voice part is pre-rendered and cached for the next 2–4 bars of audio data through singing synthesis or vocal conversion path. This mechanism moves most of the synthesis calculations forward, avoiding excessive latency introduced by real-time rendering and leaving sufficient latency budget for core alignment and processing. This is equivalent to steps 144 to 146.

[0092] 5. The system ensures that the total end-to-end latency from audio input to final mixed output does not exceed 40 milliseconds through fine-grained budget allocation and pipeline optimization of latency for each module. During implementation, the system monitors the time consumed in each stage in real time. If the total latency approaches the threshold, degradation strategies are automatically activated (such as disabling DTW, shortening the window, and reducing rendering complexity) to guarantee the real-time interactive experience in live streaming scenarios. This is equivalent to step 152.

[0093] As can be seen from the above embodiments, this application, by constructing a layered and adjustable latency budget system and integrating short-window alignment, dynamic processing, pre-rendering caching, and intelligent degradation strategies, can still stably achieve low-latency chorus generation of less than or equal to 40 milliseconds even in complex and ever-changing network and computing environments. This latency control mechanism effectively solves the problem that traditional systems cannot meet the real-time interaction requirements under conditions of large-window rendering and offline alignment, significantly improving the synchronization, naturalness, and user experience of the chorus, and providing reliable technical support for scenarios such as online karaoke and live chorus with high real-time requirements.

[0094] Please refer to Figure 6 : A personalized chorus terminal 1 based on vocal analysis includes a memory 3, a processor 2, and a computer program stored in the memory 3 and running on the processor 2. When the processor 2 executes the computer program, it implements various steps in a personalized chorus method based on vocal analysis.

[0095] The above description is merely an embodiment of the present invention and does not limit the patent scope of the present invention. Any equivalent modifications made based on the content of the present invention specification and drawings, or direct or indirect applications in related technical fields, are similarly included within the patent protection scope of the present invention.

Claims

1. A personalized choral singing method based on vocal timbre analysis, characterized in that, include: Receive user audio samples and model them to generate the user's voice parameters; The target song is structurally segmented to obtain song fragments, and the fragment difficulty of each song fragment is calculated; Based on the user's voice parameters and the difficulty of the segment of the target song, a reference singer is matched and selected; For each song segment, role allocation optimization is performed to determine the singing status of the user and the reference singer in each song segment.

2. The personalized choral singing method based on vocal ray analysis according to claim 1, characterized in that, The process of receiving user audio samples and modeling them to generate the user's voice parameters includes: The user audio samples are preprocessed; The fundamental frequency and energy profile are extracted from the preprocessed user audio samples to determine the user's comfortable vocal range; Extract the user's timbre embedding vector and style embedding vector from the preprocessed user audio samples; The user's rhythm and vocal ability parameters are evaluated based on the preprocessed user audio samples. By combining the comfortable vocal range, timbre embedding vector, style embedding vector, and the user's rhythm and vocal ability parameters, structured sound parameters are generated.

3. The personalized choral singing method based on vocal ray analysis according to claim 1, characterized in that, The process of structurally segmenting the target song to obtain song fragments and calculating the segment difficulty of each song fragment includes: The beats, strong beats, and measure positions of the target song are detected, and the target song is divided into song segments with measures as the smallest scheduling unit based on the measure boundaries. Mark the switchable boundaries for each of the aforementioned song segments; Extract features from each song segment to represent the singing difficulty; The extracted features are normalized and weighted summed to calculate the quantized difficulty value of each song segment, which is then used as the difficulty of the song segment.

4. The personalized choral singing method based on vocal ray analysis according to claim 1, characterized in that, The process of matching and selecting a reference singer based on the user's voice parameters and the difficulty of a segment of the target song includes: Calculate the overlap of vocal range, timbre similarity, and style similarity between the user and each preset singer; Based on the user's voice parameters and the characteristics of the target song, assess the overall singability of the target song for the user; Based on the aforementioned vocal range overlap, timbre similarity, style similarity, and overall singability, a comprehensive score is obtained for each of the preset singers. Select the preset singer with the highest overall score as the reference singer.

5. The personalized choral singing method based on vocal ray analysis according to claim 1, characterized in that, The step of optimizing role assignment for each segment of the song, and determining the singing status of the user and the reference singer in each segment of the song, includes: For each song segment, a first decision variable and a second decision variable are defined. The first decision variable is used to indicate whether the user sings in the song segment, and the second decision variable is used to indicate whether the reference singer sings in the song segment. Based on the first and second decision variables of all the song segments, an objective function is constructed. The objective function is used to maximize the overall singing adaptation benefit and minimize the cost introduced by role switching and the penalty introduced by the segment difficulty exceeding the user's singing ability. Under the preset optimization constraints, use dynamic programming or integer linear programming to solve for the optimal allocation scheme.

6. The personalized choral singing method based on vocal ray analysis according to claim 1, characterized in that, The step of optimizing role assignment for each song segment and determining the singing status of the user and the reference singer in each song segment also includes: If a singing synthesis path is adopted, the corresponding melody is synthesized based on the acoustic characteristics of the reference singer, and a lookback window mechanism is used to pre-render and cache the audio of the subsequent preset number of bars; If the vocal conversion path is adopted, the authorized dry vocal material of the song is called, and the dry vocal timbre is converted into the timbre of the reference singer through the timbre conversion model to generate the reference vocal part; If it is necessary to generate harmonic parts, a multi-part accompaniment matching the main melody of the target song is generated based on the chord information of the song or through a neural network model.

7. The personalized choral singing method based on vocal ray analysis according to claim 1, characterized in that, Also includes: A dynamic time warping algorithm with a limited window length is used to fine-tune and align the user's vocal part with the reference vocal part of the reference singer. Audio processing is applied in parallel to the user's vocal part and the reference vocal part, and sidechain energy control is performed; The total latency of the fine-tuning alignment process and the audio processing process is controlled within a preset threshold.

8. The personalized choral singing method based on vocal ray analysis according to claim 7, characterized in that, Also includes: If the deviation between the user's actual singing speed and the preset speed exceeds the tolerance limit, then at the boundary of the next song segment corresponding to the current song segment, a re-rendering and cache update of the subsequent reference voice part is triggered.

9. The personalized choral singing method based on vocal ray analysis according to claim 1, characterized in that, Also includes: Associate and bind the authorization information of the reference singer, the unique number of this generated task, and the timestamp; In the final output mixed audio signal, a watermark of frequency domain spread spectrum or neural network is embedded.

10. A personalized choral terminal based on vocal ray analysis, characterized in that, It includes a memory, a processor, and a computer program stored in the memory and running on the processor, wherein the processor executes the computer program to implement the various steps of the personalized chorus generation method based on vocal ray analysis as described in any one of claims 1 to 9.

Images