Personalized singing cloning system and matching method based on deep voiceprint feature matching

Abstract

The invention relates to the technical field of singing synthesis, and discloses a personalized singing cloning system and matching method based on deep voiceprint feature matching, and the system comprises a user personality modeling module, a style modeling module, a semantic melody joint perception module, a singing synthesis module and an emotion consistency matching module. According to the method, personality and style vectors are generated by extracting voiceprint, emotion and sound production control features of a user, a song sound spectrogram is generated by using a diffusion model in combination with lyric semantics and melody structures, and finally the model is discriminated and optimized through emotion consistency. By extracting the voiceprint feature, the emotion expression feature and the sounding control feature of the user and fusing the features with the style control vector and the semantic melody information, the singing sound which meets the individuation requirement and is consistent in style can be generated, the defects of a singing sound synthesis model in individuation and style consistency expression in the prior art are overcome, and the user experience is improved. And more accurate personalized singing sound generation capability is provided.

Description

Technical Field

[0001] The present invention relates to the technical field of singing synthesis, and specifically to a personalized singing cloning system and matching method based on deep voiceprint feature matching. Background Art

[0002] With the rapid development of artificial intelligence and deep learning, in recent years, singing synthesis technology has made remarkable progress. Traditional singing synthesis methods are mostly based on splicing technology and statistical models. However, the singing generated by these methods often lacks flexibility and is difficult to accurately capture the individual needs and emotional expressions of users.

[0003] In recent years, deep learning-based singing synthesis models have gradually become mainstream. Through end-to-end training, these models can learn complex audio features from a large amount of audio data and then generate high-quality singing. In particular, neural network models such as WaveNet and Tacotron can generate relatively natural and smooth singing through spectrogram generation and waveform prediction. These technologies have achieved remarkable results in sound quality and speech fluency, but they mainly focus on the improvement of audio quality and ignore the needs of personalized and style consistency in the generated singing.

[0004] In addition, most existing singing synthesis technologies rely on fixed training sets and single style features, making it difficult to achieve true personalized customization of the synthesis results. For example, traditional singing synthesis models cannot generate singing with specific emotions or styles according to the specific needs of individual users, lacking a deep understanding and expression of users' personality and style preferences. Therefore, although existing technologies have made breakthroughs in sound quality improvement, they still face the bottleneck of being unable to meet users' needs in terms of personality and style consistency. Summary of the Invention

[0005] Aiming at the deficiencies of the existing technology, the present invention provides a personalized singing cloning system and matching method based on deep voiceprint feature matching, which solves the problems of the existing singing synthesis technology in terms of personalized and style consistency expression.

[0006] To achieve the above objectives, the present invention is realized through the following technical solutions: A personalized singing cloning system based on deep voiceprint feature matching, including:

[0007] A user personality modeling module, used to extract voiceprint features, expression ability features, and vocal control features based on the audio data provided by the user, and fuse them into a personality representation vector;

[0008] A style modeling module, used to construct a high-order association structure between multiple singing style atoms, and generate a style control vector based on this structure;

[0009] A semantic melody joint perception module, which is used to extract semantic information and melody structure information from lyric texts and melody information and fuse them;

[0010] A singing voice synthesis module, which is used to synthesize personalized singing voices according to the personality representation vector, style control vector, semantic information and melody structure information;

[0011] An emotional consistency matching module, which is used to determine whether the synthesized singing voice is consistent with the target personality and style expression.

[0012] Preferably, the user personality modeling module includes:

[0013] A voiceprint extraction unit, which is used to extract the acoustic features of the user's speaking voice;

[0014] An expression ability extraction unit, which is used to extract emotion expression-related features from non-verbal sounds;

[0015] A control ability extraction unit, which is used to extract vocal control features from the user's breathing rhythm and speech rate changes;

[0016] A personality embedding fusion unit, which is used to fuse the above features into a unified personality representation vector through an attention mechanism.

[0017] Preferably, the style modeling module includes:

[0018] A style atomic representation unit, which is used to define several singing style atomic nodes, and each node contains attributes such as style intensity, duration and activation probability;

[0019] A style structure combination unit, which is used to combine two or more style atoms into a high-order style structure;

[0020] A style inference network unit, which is used to infer the style control vector through a hypergraph neural network.

[0021] Preferably, the semantic melody joint perception module includes:

[0022] A lyric semantic encoding unit, which is used to extract the semantic representation of lyric texts through a pre-trained language model;

[0023] A melody structure encoding unit, which is used to extract structural features such as the rhythm and pitch of the melody;

[0024] A conditional control splicing unit, which is used to splice the personality representation vector, style control vector, lyric semantic representation and melody structure features into a control input vector.

[0025] Preferably, the singing voice synthesis module is a neural network model based on a conditional diffusion mechanism, which receives the control vector and Gaussian noise as inputs and generates the corresponding spectrogram frame sequence.

[0026] Preferably, the emotional consistency matching module includes:

[0027] A synesthesia discriminator unit for outputting a consistency discrimination result based on the matching degree between the generated spectrogram and the target personality and style representations;

[0028] A matching optimization unit for optimizing the parameters of the synthesis module according to the discrimination result to improve the consistency performance.

[0029] The present invention also provides a personalized singing voice cloning matching method based on deep voiceprint feature matching, which is characterized in that it is applied to the above-mentioned personalized singing voice cloning system based on deep voiceprint feature matching, and includes the following steps:

[0030] Receiving a user's voice sample, non-verbal expression segment, and vocal control segment;

[0031] Extracting corresponding voiceprint features, expression features, and vocal control features;

[0032] Generating a personality embedding vector based on a multi-scale attention fusion method;

[0033] Constructing a style hypergraph including multiple singing style atoms and their high-order combination relationships, and inferring to obtain a style control vector;

[0034] Extracting song semantic information and melody structure information, and jointly constructing a control vector with the personality vector and the style vector;

[0035] Generating a spectrogram of the target singing voice based on the control vector and the diffusion model;

[0036] Performing emotional consistency discrimination on the synthesis result and using it for model optimization.

[0037] The present invention provides a personalized singing voice cloning system and matching method based on deep voiceprint feature matching.

[0038] It has the following beneficial effects:

[0039] 1. The present invention adopts an emotional consistency matching module, combined with a synesthesia discriminator and a matching optimization unit, which can efficiently evaluate the consistency between the generated singing voice and the target personality and style, and then optimize the synthesis process. Through this technical solution, the generated singing voice not only meets the personalized requirements, but also can accurately express the target emotion and style. Compared with the singing voice generated solely based on audio features in the prior art, it solves the problem of insufficient personality and style consistency.

[0040] 2. The present invention introduces an optimization mechanism based on gradient descent. By dynamically adjusting the parameters of the synthesis module, the matching degree between the generated singing voice and the target features is improved. Different from traditional static generation models, this technical solution can be optimized in real time according to feedback, ensuring the consistency of the singing voice in style and emotional expression, which solves the limitation that traditional models cannot be flexibly adjusted according to individual needs.

[0041] 3. Through the anomaly detection mechanism, the present invention can automatically identify inconsistent emotional expressions during the singing voice generation process and correct them through the feedback mechanism. This intelligent feedback ability ensures that the generated singing voice always highly matches the target requirements in terms of emotion and style. Compared with the manual adjustment and post - correction in traditional methods, the adaptive adjustment of the present invention significantly improves the accuracy and real - time performance of singing voice generation. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] Figure 1 is a three - dimensional view of the system of the present invention;

[0043] Figure 2 is a framework diagram of the user personality modeling module of the present invention;

[0044] Figure 3 is a framework diagram of the style modeling module of the present invention;

[0045] Figure 4 is a framework diagram of the semantic - melody joint perception module of the present invention;

[0046] Figure 5 is a framework diagram of the emotion consistency matching module of the present invention;

[0047] Figure 6 is a flowchart of the matching method of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0048] Next, the technical solutions in the embodiments of the present invention will be clearly and completely described in conjunction with the accompanying drawings of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present invention.

[0049] Please refer to the attached Figure 1 , the embodiments of the present invention provide a personalized singing voice cloning system based on deep voiceprint feature matching, including:

[0050] Please refer to the attached Figure 2 , a user personality modeling module, which is used to extract voiceprint features, expression ability features, and vocal control features based on the audio data provided by the user and fuse them into a personality representation vector;

[0051] To achieve personalized singing synthesis driven by multi-modal voice, it is necessary to first accurately extract the user's personality features from the provided voice data and convert them into a unified embedded expression vector for driving subsequent personality transfer and singing cloning modules. For this purpose, a user personality modeling module is proposed in the present invention. The module is connected after voice data collection and preprocessing and serves as one of the key input nodes in the singing cloning modeling process.

[0052] Generally, the user personality modeling module includes multiple feature modeling branches for separately extracting multi-dimensional features characterizing personality such as acoustic voiceprint, emotional expression ability, and vocal control from the user's voice data, and then integrating them into a unified personality representation vector through a neural fusion mechanism.

[0053] In this embodiment, the user personality modeling module mainly includes the following sub-units: a voiceprint extraction unit, an expression ability extraction unit, a control ability extraction unit, and a personality embedding fusion unit. The implementation method of each unit is as follows:

[0054] In this embodiment, the voiceprint extraction unit is used to extract acoustic features highly related to the individual identity from the user's voice signal. Specifically, this unit first performs frame-level framing and window function weighting on the input voice signal and obtains its spectral features by using the short-time Fourier transform (STFT):

[0055] where x(n) is the nth sampling point, w(n) is the window function, N is the frame length, t is the frame position, and f is the frequency index.

[0056] After obtaining the spectrum, the Mel frequency cepstral coefficients (MFCC) are further calculated:

[0057] MFCC k =∑ f log(|X(t,f)|)·H k (f), where H k (f) represents the kth Mel filter, log(|X(t,f)|) is the logarithmic power spectrum, is the kth order feature component.

[0058] The above features are input into a pre-trained convolutional or temporal neural network, and a voiceprint embedding vector with a fixed dimension is output:

[0059] where f spk (·) represents the voiceprint feature extraction network, represents the output voiceprint vector, and d is the embedding dimension.

[0060] In this embodiment, the expression ability extraction unit is used to obtain the emotional expression characteristics of the user. This module mainly analyzes the fundamental frequency fluctuations, intonation changes, energy changes, and non-verbal features (such as sighs, laughter, etc.) in the speech. For example, the fundamental frequency trajectory F0(t) can be extracted using the YIN or Autocorrelation algorithm:

[0061] where R xx (t,f) represents the autocorrelation function of frequency f at time t.

[0062] To represent the speech rate dynamics, the average speech rate is defined: where N syllable represents the number of syllables in the speech segment, and T speech represents the duration of the speech segment.

[0063] In addition, the short-time energy E(t) of the speech is also calculated:

[0064] The extracted features are mapped to an expression ability embedding vector through an emotion modeling network:

[0065] where f expr (·) is an emotion feature extraction model, with the input being a sequence of audio frames related to non-verbal information and the output being expression ability features with the same dimension.

[0066] In this embodiment, the control ability extraction unit is used to measure the stability and controllability of the user during speech production, which is mainly reflected in aspects such as breathing rhythm, speech rate stability, and pitch smoothness. As an option, the breathing rate is calculated using the pause distribution and duration features:

[0067] where Δt i is the duration of the i-th pause, N pause is the number of pauses, and T total is the total speech duration.

[0068] The pitch smoothness index is calculated:

[0069]

[0070] Based on the above features, they are input into the control ability feature network to extract control features:

[0071] where f ctrl (·) is a control ability modeling function, and the output is a control embedding vector

[0072] The personalized embedding fusion unit is used to fuse the above three embedding vectors into the final personalized vector. As a fusion strategy, a weighted attention mechanism is adopted to dynamically adjust the importance of different features. Specifically, learnable weights are introduced

[0073] i = 1, 2, 3, where W i is the weight vector obtained by training, and e (i) is the i-th type of feature embedding.

[0074] The final fusion output vector: where is the unified personalized representation vector, which serves as the key input condition for the subsequent singing synthesis model. This representation can convey the individual's speech expression style and vocal control characteristics while maintaining the consistency of the voiceprint.

[0075] It should be noted that to improve the robustness of the model, a regularization and Dropout mechanism can be added to each sub-feature channel before fusion. In addition, in actual deployment, a parameter sharing strategy can also be adopted to reduce the model size and improve feature consistency.

[0076] Please refer to Appendix Figure 3 , the style modeling module, which is used to construct the high-order correlation structure between multiple singing style atoms and generate a style control vector based on this structure;

[0077] In this embodiment, the style modeling module includes a style atom representation unit, a style structure combination unit, a style inference network unit, etc. The detailed implementation methods of each sub-unit are as follows:

[0078] In this embodiment, the style atom representation unit is used to define several singing style atom nodes, and each node represents a basic singing style element. Each style atom contains the following three main attributes:

[0079] Style intensity S i ∈[0, 1]: represents the intensity of the style atom. The larger the value, the more obvious the style feature.

[0080] Style duration represents the duration of this style atom, in seconds.

[0081] Activation probability P i ∈[0, 1]: represents the activation probability of this style atom within a specific time period.

[0082] Specifically, for each style atom a i (where i represents the index of the style atom), its attributes can be expressed as:

[0083] ai =(S i , T i , P i ), where S i is the style intensity, reflecting the significance of the style, with a value range of [0, 1]; T i is the style duration, indicating the duration of the style in the singing voice, usually represented by a non - negative real number; P i is the activation probability, representing the probability of the style atom appearing in a specific audio segment.

[0084] In practical applications, style atoms are automatically learned from training data and can be dynamically adjusted according to the specific requirements of the song. For example, some style atoms can be activated in the high - pitch part, while other style atoms are activated in the low - pitch or mid - pitch part to ensure a natural transition of the singing style.

[0085] In this embodiment, the style structure combination unit is used to combine two or more style atoms into a more complex high - order style structure. Generally, the combination of style atoms can be achieved through weighted summation, concatenation, or other forms of operations to capture the interaction relationships between different style atoms.

[0086] A common combination method is to integrate multiple style atoms through weighted summation:

[0087] where e style is the combined style representation vector, representing the final effect after weighting multiple style atoms; α i is the weight of style atom a i , representing the contribution of this style atom in the combination process, and the weight value is usually learned through training the network; N is the number of style atoms.

[0088] As an option, the combination of style atoms can also be achieved through a graph neural network (GNN). Specifically, the relationship between style atoms is expressed through a graph structure, where each node in the graph corresponds to a style atom, and the edges between nodes represent the associations between style atoms. The graph neural network can capture the complex relationships between style atoms through graph convolution operations:

[0089] e style = GConv(A, W), where A is the adjacency matrix between style atoms, representing the connection relationship between style atoms, A ij represents the similarity or influence between style atom a i and a j ; W is the convolution kernel weight, representing the learnable parameter in the graph convolution operation; e style is the style representation vector obtained through the graph convolution operation.

[0090] The graph neural network can effectively capture the high-order correlations between style atoms and generate a more expressive style representation.

[0091] In this embodiment, the style inference network unit is used to infer a style control vector through a hypergraph neural network. Compared with the traditional graph neural network, the hypergraph neural network can better handle the high-order correlations between nodes, connect multiple nodes through hyperedges, and thus capture the complex interactions between multiple style atoms.

[0092] In the hypergraph neural network, each hyperedge e j is composed of multiple style atom nodes, and the hyperedge represents the high-order correlations between multiple style atoms. Specifically, each hyperedge e j is represented as:

[0093] where f GNN (·) represents the message passing function of the graph neural network, which is used to process the information of the style atom nodes in the hyperedge e j ; is the representation of the hyperedge e j , capturing the complex relationships between the multiple style atoms involved in this hyperedge.

[0094] By performing a weighted sum of the representations of all hyperedges, the style control vector is finally obtained:

[0095] where β j is the weight of the hyperedge e j , indicating the contribution of this hyperedge to the final style control vector.

[0096] The style control vector c style is a low-dimensional vector that can effectively control the style features of the generated singing voice. During the singing voice synthesis process, the style control vector will directly affect the synthesized singing voice style, controlling the expression mode, pitch fluctuation, emotional color, etc. of the singing voice.

[0097] Please refer to the attached Figure 4 , the semantic melody joint perception module, which is used to extract semantic information and melody structure information from the lyric text and melody information and fuse them;

[0098] To further enhance the diversity and accuracy of singing voice generation, the semantic melody joint perception module proposed in the present invention aims to generate a comprehensive and accurate control input vector through the in-depth understanding and integration of lyric text and melody structure. This control input vector will be used to drive the singing voice synthesis model, thereby generating singing voices that are personalized, stylized, and meet the requirements of lyric semantics and melody structure. This module plays a crucial role in the singing voice synthesis system. Especially when closely combining text content with music features, it can ensure that the finally generated singing voices express the emotions of the lyrics, the rhythm and pitch changes of the melody, while meeting personalized needs.

[0099] Generally, the semantic melody joint perception module includes a lyric semantic encoding unit, a melody structure encoding unit, and a conditional control splicing unit. The functions of each sub-unit in the overall module complement each other, jointly completing the extraction and integration of lyric text and melody information. Specifically, in this embodiment, the following technical details are used to achieve the comprehensive perception of lyric semantics, melody structure, and personality and style features.

[0100] In this embodiment, the task of the lyric semantic encoding unit is to perform semantic understanding and encoding on the lyric text through a pre-trained language model. Lyric text usually contains rich semantic information, which can convey various levels of information such as emotions, intentions, and atmospheres. Therefore, its semantic encoding is crucial for generating accurate singing voices. Generally, the lyric text L = {l1, l2, …, l n} consists of several words l i . To extract the deep semantic features of the lyrics, this unit uses a pre-trained language model (such as BERT, GPT, etc.) for text encoding.

[0101] Specifically, the pre-trained language model f LM takes the lyric text L as input and outputs the corresponding lyric semantic vector representation e lyric , and the formula is as follows:

[0102] e lyric = f LM (L), where is the semantic representation vector of the lyric text, and d is the dimension of the semantic representation. Through this method, the grammatical structure, word meaning relationship, and context information in the lyrics can be effectively captured, and then a singing voice expression with semantic depth can be generated.

[0103] The melody structure encoding unit is responsible for extracting the structural features such as the rhythm, pitch, and note duration of the melody. In this embodiment, the melody signal M = {m1, m2, …, m n} includes multiple note information, and each note m iIt is determined by its attributes such as pitch, duration, rhythm, etc. To comprehensively describe the structural characteristics of the melody, the melody structure encoding unit abstracts the pitch, rhythm information, etc. of the notes to obtain the structural representation of the melody.

[0104] First, the pitch H = {h1, h2, …, h n} can be expressed as:

[0105] H i = log(f i ), where f i is the frequency of the i-th note, and H i is the corresponding pitch value. Next, the melody encoding maps the information such as pitch and rhythm to the structural feature vector e melody of the melody through the feature extraction network f melody , that is:

[0106] e melody = f melody (M), where is the melody structure representation vector. The encoding of the melody structure takes into account factors such as the temporal distribution of notes, pitch changes, and rhythm changes. Therefore, the output melody structure features can comprehensively reflect the structural information of the melody.

[0107] The task of the conditional control splicing unit is to splice the features obtained from the user personality modeling module, style modeling module, song lyric semantic encoding unit, and melody structure encoding unit to form the final control input vector. This vector will be directly used in the singing synthesis model to determine the performance of the generated singing in terms of personality, style, semantics, and melody structure.

[0108] Specifically, the splicing process combines the following vectors into a unified control input vector:

[0109] Personality representation vector e persona : from the user personality modeling module, representing the personalized speech features of the user;

[0110] Style control vector c style : from the style modeling module, representing the style control features of the singing;

[0111] Song lyric semantic representation e lyric : from the song lyric semantic encoding unit, representing the semantic features of the lyrics;

[0112] Melody structure feature e melody : from the melody structure encoding unit, representing the rhythm and pitch features of the melody.

[0113] The spliced control input vector e control is: e control = [epersona , c style , e lyric , e melody , where · represents the vector concatenation operation, is the final control input vector, and d1, d2, d3, d4 are the dimensions of personality, style, song lyric semantics, and melody structure features respectively.

[0114] Through the concatenated control input vector, the singing voice generation model can consider multiple factors simultaneously during the synthesis process: personalized features, singing voice style, song lyric semantics, and melody structure, so as to generate a singing voice that not only meets the style requirements but also accurately expresses the lyric content and melody.

[0115] The singing voice synthesis module is used to synthesize personalized singing voices according to the personality representation vector, style control vector, semantic information, and melody structure information;

[0116] The singing voice synthesis module in the present invention adopts a conditional diffusion mechanism to generate a sequence of spectrogram frames of the singing voice through a reverse process of gradually denoising. The main task of this module is to use the personality representation vector, style control vector, song lyric semantic information, and melody structure information obtained from the foregoing steps to generate personalized singing voices through a diffusion model. The control input vector encodes these feature information and transmits it to the generation model to drive the model to generate the final spectrogram.

[0117] During the actual operation process, the working process of the singing voice synthesis module is as follows: First, the control input vector and the initial Gaussian noise are sent into the neural network as conditional inputs, and the singing voice spectrogram consistent with the input conditions is gradually generated through the reverse diffusion mechanism. During the training process, the model will learn how to recover high-quality singing voices from the noise through the denoising process.

[0118] The basic idea of the diffusion model is to transform real data (such as audio data) into noise through a forward process, and then recover the original data from the noise through a reverse diffusion process. Specifically, the forward diffusion process transforms the audio data x0 into Gaussian noise z T , and the reverse diffusion process gradually removes the noise and finally generates the target sample.

[0119] For the singing voice synthesis task, the forward diffusion process can be expressed by the following formula:

[0120] where β t is the noise standard deviation of the diffusion process, controlling the increase of noise at each step; z t is the noise state at the t-th step; represents the Gaussian distribution; I is the identity matrix; z t-1 is the noise state at the (t - 1)-th step, representing the noise state of the previous step.

[0121] During the reverse diffusion process, we hope to remove noise and restore the audio signal by controlling the input with conditions (control vector e control ). The formula for the reverse process is:

[0122] where, and represent the mean and variance of the denoising process respectively, which are predicted by the neural network model. The control vector e control plays a key role in this process. It contains all the information of personality, style, song semantics and melody structure, and guides the model to generate a singing voice consistent with the target requirements.

[0123] The control vector e control is a feature representation extracted and fused from multiple information sources (personality, style, song semantics, melody structure). Specifically, the control vector e control can be expressed as:

[0124] e control = [e persona , c style , e lyric , e melody , where e persona is the personality representation vector, representing individualized speech features; c style is the style control vector, determining the style features of the singing voice; e lyric is the song semantics encoding vector, reflecting the emotion and semantic content of the lyrics; e melody is the melody structure encoding vector, representing the rhythm and pitch features of the melody.

[0125] These features from different sources are concatenated through the conditional control concatenation unit, and finally form e control , which is passed as a conditional input to the singing voice synthesis module, guiding the generation of singing voice features in each step of the denoising process.

[0126] In the diffusion model, the initial Gaussian noise z T is the starting point of the reverse diffusion process. This noise vector is usually randomly generated, and its dimension matches the size of the spectrogram frame sequence. The initial noise z T can be expressed as:

[0127] where, represents a Gaussian distribution with zero mean and variance . This noise is used as the input of the reverse diffusion process and is gradually denoised, finally restoring a clear singing voice spectrogram.

[0128] After the reverse diffusion process, the final output is a sequence of spectrogram frames of a singing voice S = {s1, s2, …, s n}. Each frame s i represents the frequency distribution at a moment, reflecting the frequency and amplitude information of the singing voice at that moment. The final spectrogram sequence can be used in the audio synthesis process to generate the actual audio waveform.

[0129] The formula for the generation process is: S = Diffusion(e control , z T ), where Diffusion(·) represents the reverse diffusion process based on the control vector e control and the Gaussian noise z T to output the final sequence of spectrogram frames S.

[0130] Please refer to the appendix Figure 5 for the emotional consistency matching module, which is used to determine whether the synthesized singing voice is consistent with the target personality and style expression.

[0131] To further improve the emotional consistency of singing voice generation, the present invention proposes an emotional consistency matching module to ensure the matching degree between the synthesized singing voice and the target personality and style. The working process of this module includes an empathy discriminator unit and a matching optimization unit. Among them, the former evaluates the similarity between the generated singing voice and the target features, and the latter optimizes the parameters of the synthesis model according to the discrimination results, so as to achieve more accurate personalized and stylized singing voice generation. This technical solution details how to evaluate the consistency between the generated singing voice and the target personality and style, and uses a feedback mechanism to continuously optimize the generation effect to ensure that the finally synthesized singing voice conforms to the target emotion and style.

[0132] In this embodiment, the task of the empathy discriminator unit is to evaluate the matching degree between the generated singing voice and the target personality and style. This unit calculates the similarity or consistency measure between the generated spectrogram S = {s1, s2, …, s n} and the target personality vector e persona and the style control vector c style to output the consistency discrimination result

[0133] In this unit, the input of the neural network model includes the generated singing voice features SSS and the target personality and style feature vectors e persona and c style . By extracting and processing the features of the input data, the neural network calculates the matching degree between the generated singing voice and the target personality and style. Specifically, the network uses a similarity measure function to quantify the distance or similarity between these feature vectors. The formula is as follows:

[0134] Among them, f match represents the neural network model of the empathy discriminator, is the output matching degree value, indicating the similarity between the generated singing voice and the target. If is closer to 1, it indicates that the matching degree between the generated singing voice and the target personality and style is higher; if is closer to 0, it indicates that the matching degree between the two is lower.

[0135] In practical applications, the target personality and style features e persona and c style are usually obtained by feature modeling of the user's input. For example, the personality vector e persona can be generated by analyzing the user's past singing voice or speech features, and the style vector c style is obtained by analyzing the style features of the target song (such as music rhythm, melody change, etc.). Based on the matching of these features, the emotion and style of the singing voice can be ensured to be highly consistent with the target during the generation process.

[0136] The matching optimization unit optimizes the parameters of the synthesis module through the output result of the empathy discriminator to improve the consistency of singing voice synthesis by minimizing the error between the generated singing voice and the target features. This optimization process adopts an optimization method based on gradient descent to minimize the loss function as the goal, thereby adjusting the control parameters of the generated singing voice to make it more in line with the personality and style requirements.

[0137] During the optimization process, the loss function can be calculated through contrastive loss, and the specific expression is:

[0138] Among them, is the cross-entropy loss function, y is the true label, indicating whether the personality and style of the target singing voice meet the expectations, is the output of the empathy discriminator, representing the matching degree between the generated singing voice and the target personality and style. By minimizing the loss function the model can gradually optimize the personalization and style consistency of the generated singing voice.

[0139] Through the backpropagation mechanism, the parameters θ of the generation model are updated:

[0140] Among them, η is the learning rate, is the gradient of the loss function with respect to the model parameters θ. The gradient descent method updates the model parameters according to the loss value in each iteration, thereby continuously improving the consistency between the generated singing voice and the target features.

[0141] In some embodiments, the emotional consistency matching module may further include an anomaly detection mechanism for detecting whether there is a significant inconsistency between the generated singing voice and the target. If the matching degree of the generated singing voice is lower than a predetermined threshold, the system will trigger a feedback mechanism to adjust the parameters in the synthesis process to make up for the deviation in personality and style. This anomaly detection can be achieved by monitoring the fluctuation range of the matching degree. If it exceeds a certain tolerance, it indicates a large inconsistency and the control vector of the synthesis process needs to be adjusted or the model needs to be optimized.

[0142] For more complex situations, the optimization of the emotional consistency matching module can be further improved by combining multi-modal information. For example, more emotional data or emotional vocabulary models can be introduced to further refine the emotional expression of singing voice generation, making the generated singing voice richer in emotional expression. By introducing reinforcement learning techniques, the model can not only optimize the consistency of personality and style, but also enhance the emotional depth and detailed performance of the generated singing voice.

[0143] The personalized singing voice cloning matching method based on deep voiceprint feature matching described below can be referred to in correspondence with the personalized singing voice cloning system based on deep voiceprint feature matching described above.

[0144] Please refer to the appendix Figure 6 , the personalized singing voice cloning matching method based on deep voiceprint feature matching, which is applied to the above-mentioned personalized singing voice cloning system based on deep voiceprint feature matching, includes the following steps:

[0145] Receive the user's voice sample, non-verbal expression segment and vocal control segment;

[0146] Extract the corresponding voiceprint features, expression features and vocal control features;

[0147] Generate a personalized embedding vector based on a multi-scale attention fusion method;

[0148] Construct a style hypergraph containing multiple singing style atoms and their higher-order combination relationships, and infer a style control vector;

[0149] Extract the semantic information of the lyrics and the melody structure information, and jointly construct a control vector with the personality vector and the style vector;

[0150] Generate the spectrogram of the target singing voice based on the control vector and the diffusion model;

[0151] Perform emotional consistency discrimination on the synthesis result and use it for model optimization.

[0152] The method of this embodiment can be used to implement the above system embodiment, and its principle and technical effects are similar, so they will not be elaborated here.

[0153] Although embodiments of the present invention have been shown and described, those of ordinary skill in the art will appreciate that various changes, modifications, substitutions and variations can be made to these embodiments without departing from the principles and spirit of the present invention. The scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. A personalized singing voice cloning system based on deep voiceprint feature matching, characterized in that, It includes: A user personality modeling module, which is used to extract voiceprint features, expression ability features and vocal control features based on the audio data provided by the user, and fuse them into a personality representation vector; A style modeling module, which is used to construct a high-order association structure among multiple singing style atoms, and generate a style control vector based on this structure; A semantic melody joint perception module, which is used to extract semantic information and melody structure information from the lyrics text and melody information and fuse them; A singing synthesis module, which is used to synthesize personalized singing according to the personality representation vector, style control vector, semantic information and melody structure information; An emotional consistency matching module, which is used to judge whether the synthesized singing is consistent with the target personality and style expression.

2. The personalized singing voice cloning system based on deep voiceprint feature matching according to claim 1, wherein The user personality modeling module includes: A voiceprint extraction unit, which is used to extract the acoustic features of the user's speaking voice; An expression ability extraction unit, which is used to extract emotion expression-related features from non-verbal sounds; A control ability extraction unit, which is used to extract vocal control features from the user's breathing rhythm and speech rate changes; A personality embedding fusion unit, which is used to fuse the above features into a unified personality representation vector through an attention mechanism.

3. The personalized singing voice cloning system based on deep voiceprint feature matching according to claim 1, wherein The style modeling module includes: A style atom representation unit, which is used to define several singing style atom nodes, and each node contains attributes such as style intensity, duration and activation probability; A style structure combination unit, which is used to combine two or more style atoms into a high-order style structure; A style inference network unit, which is used to infer a style control vector through a hypergraph neural network.

4. The personalized singing voice cloning system based on deep voiceprint feature matching according to claim 1, wherein The semantic melody joint perception module includes: A song lyric semantic encoding unit, which is used to extract the semantic representation of the lyrics text through a pre-trained language model; A melody structure encoding unit, which is used to extract structural features such as the rhythm and pitch of the melody; A conditional control splicing unit, which is used to splice the personality representation vector, style control vector, song lyric semantic representation and melody structure features into a control input vector.

5. The personalized singing voice cloning system based on deep voiceprint feature matching according to claim 1, characterized in that, The singing synthesis module is a neural network model based on a conditional diffusion mechanism, which receives a control vector and Gaussian noise as inputs, and generates a corresponding sequence of spectrogram frames.

6. The personalized singing voice cloning system based on deep voiceprint feature matching according to claim 1, wherein The emotional consistency matching module includes: A synesthesia discriminator unit, which is used to output a consistency discrimination result based on the matching degree between the generated spectrogram and the target personality and style representation; A matching optimization unit, which is used to optimize the parameters of the synthesis module according to the discrimination result to improve the consistency performance.

7. A personalized singing voice cloning matching method based on deep voiceprint feature matching, characterized in that, Applied to the personalized singing cloning system based on deep voiceprint feature matching described in any one of claims 1-6 above, it includes the following steps: Receive the user's speech sample, non-verbal expression segment and vocal control segment; Extract the corresponding voiceprint features, expression features and vocal control features; Generate a personality embedding vector based on a multi-scale attention fusion method; Construct a style hypergraph containing multiple singing style atoms and their high-order combination relationships, and infer a style control vector; Extract song lyric semantic information and melody structure information, and jointly construct a control vector with the personality vector and style vector; Generate a spectrogram of the target singing based on the control vector and the diffusion model; Perform emotional consistency discrimination on the synthesis result and use it for model optimization.

Images