Method for personalized speech synthesis of AI platform digital human, electronic device and system

Abstract

The application relates to the technical field of voice synthesis, and discloses an AI platform digital person personalized voice synthesis method, an electronic device and a system, which comprise the following steps: acquiring voice signals and spectrograms of a plurality of users asking questions to an AI platform; obtaining a plurality of word groups and word vectors of the user questions, and obtaining spectrogram matrices of the word groups in combination with the spectrograms; analyzing fundamental frequency differences and energy differences of the spectrogram matrices of the same word groups of different users, and acquiring personalized vectors of the word groups of the users; classifying to obtain a plurality of types of word groups; obtaining a plurality of sensitive word groups of the users; obtaining a plurality of sensitive word group combinations of the users based on the distribution of the parts of speech corresponding to the word groups in the user questions and the distribution of the sensitive word groups and other word groups before and after the sensitive word groups in the user questions; pruning a voice synthesis network, and performing personalized voice synthesis of a digital person. The application aims to solve the problem that a neural network model cannot be trained in a targeted manner due to the personalized information contained in user voice.

Description

Technical Field

[0001] This invention relates to the field of speech synthesis technology, specifically to a personalized speech synthesis method, electronic device, and system for AI platform digital humans. Background Technology

[0002] With the rapid development of virtual interaction and artificial intelligence technologies, digital humans have moved from concept to large-scale commercial applications, and are widely used in fields such as virtual anchors, intelligent customer service, online education, game NPCs, and virtual idols. When users interact with digital humans on AI platforms, they ask questions to the digital humans. Based on semantic understanding, the digital humans understand what the users want to express and provide personalized responses.

[0003] While traditional text-to-speech (TTS) technology can achieve clear and understandable speech, its tone is monotonous and lacks emotion and personalization. Response speech synthesis is often generated through neural networks. The speech synthesis network analyzes the user's question and generates a corresponding answer. However, because the training set used in the speech synthesis network training process does not reflect personalized information, that is, it does not include personalized factors such as the user's tone and intonation, and the neural network structure is too redundant, it learns a lot of useless information during training. This results in the generation of answers that do not contain characteristics in personalized responses, thus affecting the speech synthesis effect of the digital human and the user's interaction experience with the digital human. Summary of the Invention

[0004] This invention provides a personalized speech synthesis method, electronic device, and system for digital humans on an AI platform, to solve the problem that existing neural network models cannot be trained specifically because user speech contains personalized information. The specific technical solution adopted is as follows:

[0005] This invention proposes a personalized speech synthesis method for digital humans on an AI platform, which includes the following steps:

[0006] Acquire the voice signals of several users asking questions to the AI ​​platform, and obtain the spectrogram of each voice signal;

[0007] The speech signal of the user's question is identified and segmented to obtain several word groups and word vectors of the user's question. Combined with the spectrogram, the spectrogram matrix of each word group is obtained. The DTW matching of the spectrogram matrices of the same word group of different users is performed based on the energy value. The fundamental frequency difference and energy difference of the two spectrogram matrices at the matching time are analyzed. Combined with the energy DTW distance of the two spectrogram matrices, the personalized difference vector of the same word group of different users is obtained, and then the personalized vector of each word group of each user is obtained.

[0008] Based on the similarity relationship of word vectors of different word groups of the same user, several categories of word groups are classified; the differences between the personalized vectors of different word groups in the same category are analyzed to obtain several sensitive word groups of the user; based on the distribution of the parts of speech of word groups in user questions, several habitual parts of speech combinations of each user are obtained, and combined with the distribution of sensitive word groups and other word groups before and after them in user questions, several sensitive word combinations of the user are obtained.

[0009] The speech synthesis network is pruned based on a combination of sensitive words in the user's questions, and personalized speech synthesis for digital humans is performed.

[0010] Optionally, the specific method for obtaining several word groups and word vectors of the user question, and combining them with a spectrogram to obtain the spectrogram matrix of each word group, includes:

[0011] Take any user as the target user, recognize the speech signal of the target user's question, convert the speech signal into text information, segment the text information into words to obtain several word groups, and perform vector transformation on each word group to obtain the word vector of each word group.

[0012] For the spectrogram of the speech signal of the target user's question, the horizontal axis of the spectrogram represents time, the vertical axis represents frequency, and the pixel value of each data point in the spectrogram represents the energy value. Obtain the time period corresponding to each word group in the spectrogram, and take the spectrogram of any word group of the target user in the spectrogram under the corresponding time period as the spectrogram matrix of the target user's word group. In the spectrogram matrix, the same row corresponds to the same frequency, the same column corresponds to the same time, and the elements are the energy values ​​represented by the pixel values ​​of the data points in the spectrogram.

[0013] Optionally, the specific method for performing time-based DTW matching of the spectrogram matrix for the same word group from different users based on energy values ​​includes:

[0014] For any word group of the target user, obtain the maximum value of the energy value of each column element in the spectrum matrix, and construct a curve in time order to serve as the energy change curve of the word group.

[0015] For any user other than the target user, obtain several word groups whose word vectors are the same as those of the target user for that word group, and use them as several reference word groups for the target user's word group; obtain the energy change curve of any reference word group, and perform DTW matching with the energy change curve of the word group to obtain the DTW distance and several matching time pairs, and use the DTW distance as the energy DTW distance between the target user's word group and the other user's reference word group; the matching time pairs are the matching time pairs formed by DTW matching between each time in the spectrogram matrix of the target user's word group and each time in the spectrogram matrix of the other user's reference word group.

[0016] Optionally, the specific method for obtaining the personalized difference vector of the same word group for different users, and then obtaining the personalized vector of each word group for each user, includes:

[0017] Using the PYIN algorithm, the fundamental frequency trajectory of the spectrogram matrix of any word group of the target user is obtained, as well as the fundamental frequency trajectory of the spectrogram matrix of any reference word group of the target user for any other user other than the target user. For the time of the next two spectrogram matrices under the same matching time pair, the fundamental frequency difference value is calculated to obtain the fundamental frequency difference value of each matching time pair. The mean, variance, maximum value and minimum value of all fundamental frequency difference values ​​are calculated respectively.

[0018] For any column of the spectrogram matrix of the target user's word group, calculate the sum of all energy values ​​corresponding to the elements of the corresponding time, and construct a curve in chronological order as the energy curve of the word group; for the time of the next two spectrogram matrices under the same matching time, calculate the energy and difference values ​​to obtain the energy and difference values ​​of each matching time pair, and calculate the mean, variance, maximum and minimum values ​​of all energy and difference values ​​respectively.

[0019] The mean, variance, maximum and minimum values ​​of all fundamental frequency differences between the target user's word group and the reference word group of other users, the mean, variance, maximum and minimum values ​​of all energy and difference values, and the energy DTW distance are used as elements of the personalized difference vector between the target user's word group and the reference word group of other users to obtain the personalized difference vector between the target user's word group and the reference word group of other users.

[0020] Obtain the personalized difference vectors between the target user's word group and each reference word group of other users besides the target user, obtain the average vector of all personalized difference vectors, and use the average vector as the personalized vector of the target user's word group.

[0021] Optionally, the classification yields several categories of word groups, including specific methods such as:

[0022] For all word groups of the target user, density clustering is performed based on the word vectors of each word group. The distance metric is the difference between the cosine similarity between the word vectors of different word groups minus 1, resulting in several clusters. All word groups in the same cluster are regarded as one word group, resulting in several word groups.

[0023] Optionally, the specific methods for obtaining several sensitive word groups from the user include:

[0024] For any word group of a target user, obtain the cosine similarity between the word group and the personalized vectors of other word groups of the same type. Set a similarity threshold and use the ratio of the number of other word groups of the same type whose cosine similarity with the personalized vector of the word group is less than the similarity threshold to the total number of other word groups of the same type as the sensitivity of the word group.

[0025] The preset filtering ratio selects the word groups in any word group that are filtered first from the highest to the lowest sensitivity level. These selected word groups are then considered as sensitive word groups for that word group. This process yields a number of sensitive word groups for each word group for the target user. Other word groups that are not considered sensitive word groups are then considered as non-sensitive word groups.

[0026] Optionally, the specific method for obtaining several habitual parts of speech combinations of each user is as follows:

[0027] For all phrases in the target user's question, the LLM large model is used to perform part-of-speech tagging on each phrase to obtain the part-of-speech of each phrase. The part-of-speech of all phrases in the target user's question is arranged according to the distribution order of the phrases to form the part-of-speech sequence of the target user's question. Adjacent elements in the part-of-speech sequence are combined as a part-of-speech combination to obtain several part-of-speech combinations.

[0028] A preset habit threshold is set, and the ratio of the number of times any word class combination appears in all word class combinations in the word class sequence to the total number of word class combinations in the word class sequence is used as the habit factor of that word class combination; if the habit factor is greater than the habit threshold, the word class combination is considered as a habitual word class combination, thus obtaining several habitual word class combinations of the target user.

[0029] Optionally, the specific methods for obtaining several combinations of sensitive words from the user include:

[0030] For any sensitive word group of the target user, obtain the preceding and following word groups of the target user in the question. Combine the non-sensitive word group of the two adjacent word groups with the sensitive word group as a candidate word group combination for the sensitive word group. If it is not a non-sensitive word group, no candidate word group combination is constructed.

[0031] Obtain the part-of-speech combination corresponding to the candidate word combination. If it is a habitual part-of-speech combination of the target user, the candidate word combination is used as a habitual word combination of the sensitive word combination. If it is not a habitual part-of-speech combination of the target user, the candidate word combination is used as a related word combination of the sensitive word combination. If the sensitive word combination and its two adjacent word combinations do not constitute a candidate word combination or a habitual word combination, the sensitive word combination is used as a sensitive word combination of the target user.

[0032] For any common word combination of the sensitive word group, obtain the frequency of occurrence of the sensitive word group in the common word combination. The frequency of occurrence is the ratio of the number of times the sensitive word group appears in all word groups in the target user's question to the total number of word groups.

[0033] The frequency of non-sensitive words in the habitual word combination and the frequency of the habitual word combination are obtained. Then, the correlation strength between the sensitive words and non-sensitive words in the habitual word combination is obtained through the point mutual information algorithm.

[0034] A preset association threshold is set. If the association strength is greater than the association threshold, the habitual phrase combination of the sensitive word group is treated as a sensitive word combination.

[0035] This invention also proposes a personalized voice synthesis system for digital humans on an AI platform, the system comprising:

[0036] The voice acquisition module is used to acquire voice signals of several users asking questions to the AI ​​platform and to acquire the spectrogram of each voice signal;

[0037] The speech processing module is used to recognize and segment the speech signal of the user's question, obtain several word groups and word vectors of the user's question, and obtain the spectrogram matrix of each word group by combining the spectrogram. The DTW matching of the spectrogram matrices of the same word group of different users is performed based on the energy value. The fundamental frequency difference and energy difference of the two spectrogram matrices at the matching time are analyzed. Combined with the energy DTW distance of the two spectrogram matrices, the personalized difference vector of the same word group of different users is obtained, and then the personalized vector of each word group of each user is obtained.

[0038] Based on the similarity relationship of word vectors of different word groups of the same user, several categories of word groups are classified; the differences between the personalized vectors of different word groups in the same category are analyzed to obtain several sensitive word groups of the user; based on the distribution of the parts of speech of word groups in user questions, several habitual parts of speech combinations of each user are obtained, and combined with the distribution of sensitive word groups and other word groups before and after them in user questions, several sensitive word combinations of the user are obtained.

[0039] The speech synthesis module is used to prune the speech synthesis network based on a combination of several sensitive words in the user's question, and to perform personalized speech synthesis for the digital human.

[0040] The present invention also proposes a personalized voice synthesis electronic device for AI platform digital humans, the electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, wherein the processor executes the computer program to implement the steps of the above method.

[0041] The beneficial effects of this invention are as follows: This invention receives the voice signal of a user's question on an AI platform, generates a large-scale answer through the speech synthesis network in the AI ​​platform, and then uses a digital human to synthesize the speech to answer the question. Furthermore, it considers the user's personalized expression, such as timbre, intonation, and habitual part-of-speech combinations. Through spectrogram analysis of the voice signal, it analyzes the word vectors and spectrogram matrix changes of each phrase in the user's question. By analyzing the spectrogram matrix of the same phrases from different users, it quantifies personalized vectors to reflect the differences in speech rate, intonation, and other expression methods among different users. By classifying the phrases of the same user based on word vectors, it further identifies the individual differences among similar phrases from the same user. The greater the difference in vector values, the greater the influence of the corresponding word group on the user's expression in different contexts. Accordingly, sensitive word groups are screened. Combining habitual part-of-speech expression methods, habitual part-of-speech combinations are obtained. Sensitive word groups are combined with other word groups before and after them to obtain joint sensitive information. Sensitive word combinations are then used for pruning training of the speech synthesis network. Through neuron pruning of the speech synthesis network, the learning efficiency of the retained neurons for sensitive information of sensitive word combinations is improved, and the learning of redundant and invalid information is reduced, thereby improving the accuracy of the speech synthesis network. Based on this, the corresponding answers to user questions are generated. Finally, personalized speech synthesis of the AI ​​platform's digital human is achieved through speech expression by a digital human. Attached Figure Description

[0042] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0043] Figure 1 This is a schematic diagram of a personalized voice synthesis method for digital humans on an AI platform provided in one embodiment of the present invention;

[0044] Figure 2 This is a block diagram of a personalized speech synthesis system for an AI platform digital human, provided as another embodiment of the present invention. Detailed Implementation

[0045] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0046] Please see Figure 1 The diagram illustrates a flowchart of a personalized speech synthesis method for digital humans on an AI platform, provided by an embodiment of the present invention. The method includes the following steps:

[0047] Step S001: Obtain the voice signals of several users asking questions to the AI ​​platform, and obtain the spectrogram of each voice signal.

[0048] The purpose of this embodiment is to address the issue of users asking questions to AI platforms. When a user asks a question to an AI platform, the AI ​​platform receives the voice signal and performs personalized voice synthesis using a digital human to provide a voice answer to the user's question. Therefore, it is necessary to first obtain the voice signal of the user's question. At the same time, in order to analyze the user's timbre, intonation, and other personalized expression methods, it is necessary to obtain a spectrogram of the voice signal for corresponding analysis.

[0049] Specifically, each user asks a question to the AI ​​platform. The AI ​​platform receives the voice signal and associates each received voice signal with the user as the voice signal for each user's question. For any voice signal, noise reduction is performed through preprocessing. Signal noise reduction is a well-known technique and will not be described in detail in this embodiment. The preprocessed voice signal is then subjected to spectrogram conversion to obtain the spectrogram of each voice signal.

[0050] It should be noted that during the interaction between the user and the AI ​​platform's digital human, the user will ask questions. The digital human will understand the content that the user wants to express based on semantic understanding and provide personalized responses, which is the personalized speech generation process. Semantic understanding and speech generation are usually output through neural networks. However, in speech synthesis networks, due to the redundancy of neural networks, a lot of invalid information will be learned. In the process of providing personalized responses, invalid answers will be generated. Therefore, network pruning is required.

[0051] Step S002: Recognize and segment the speech signal of the user's question to obtain several word groups and word vectors of the user's question. Combine the spectrogram to obtain the spectrogram matrix of each word group. Perform time-based DTW matching on the spectrogram matrices of the same word groups of different users based on the energy value. Analyze the fundamental frequency difference and energy difference of the two spectrogram matrices at the matching time. Combine the energy DTW distance of the two spectrogram matrices to obtain the personalized difference vector of the same word groups of different users, and then obtain the personalized vector of each word group of each user.

[0052] It should be noted that analyzing user questions allows us to determine the contextual information at the time of the question. This contextual information includes the desired response the user wanted at that moment. Therefore, to ensure that the synthesized response is more relevant to the user's question, we need to analyze the context of the question. By analyzing the context, we can generate a reasonable answer and thus a personalized response. Since contextual information can be reflected in the intonation and content of the speech signal, we obtain this context by analyzing the energy value changes in the spectrogram. At the same time, the content of the speech signal is analyzed using character phrases to analyze the user's personalized characteristics through text content and intonation.

[0053] Preferably, in one embodiment of the present invention, the speech signal of the user question is identified and segmented to obtain several word groups and word vectors of the user question, and the spectrogram matrix of each word group is obtained by combining the spectrogram. The specific method includes:

[0054] Taking any user as the target user, the speech signal of the target user's question is recognized, the speech signal is converted into text information, and the text information is segmented into words to obtain several word groups. Each word group is then transformed into a word vector. The speech signal to text information conversion is an existing technology, such as the BERT model and the Transformer model for speech signal to text recognition. The text information is segmented into words using Jieba segmentation, and the Word2vec model is used for word vector transformation. The word vectors of different word groups have the same number of elements, that is, the same number of dimensions. The above technologies are all existing methods, and will not be described in detail in this embodiment.

[0055] Furthermore, for the spectrogram of the speech signal of the target user's question, the horizontal axis of the spectrogram represents time, the vertical axis represents frequency, and the pixel value of each data point in the spectrogram represents energy value. Since each word group obtained from the text information corresponds to a time period on the horizontal axis of the spectrogram, that is, the time axis of the text information corresponds to the time axis of the speech signal and the spectrogram, the time period corresponding to each word group in the spectrogram can be obtained. The spectrogram of any word group of the target user in the corresponding time period in the spectrogram is taken as the spectrogram matrix of the target user's word group. In the spectrogram matrix, the same row corresponds to the same frequency, the same column corresponds to the same time, and the elements are the energy values ​​represented by the pixel values ​​of the data points in the spectrogram.

[0056] It should be further explained that the user's personalized characteristics are represented by constructing feature vectors. The user's personalized characteristics are reflected in the different intonation information of the same text by different users, which means that the user's personalized characteristics are contained. Since different users say a word with different intonation, time alignment is required.

[0057] Preferably, in one embodiment of the present invention, time-based DTW matching is performed on the spectrogram matrices of the same word groups from different users based on energy values. The fundamental frequency difference and energy difference between the two spectrogram matrices at the matching time are analyzed. Combined with the energy DTW distance between the two spectrogram matrices, a personalized difference vector of the same word groups from different users is obtained, thereby obtaining the personalized vector of each word group for each user. The specific method includes:

[0058] For any word phrase of the target user, i.e. the corresponding spectrogram, the maximum value of the energy value of each column element in the spectrogram is obtained, and a curve is constructed in time order as the energy change curve of the word phrase. That is, the maximum pixel value at each time point in the corresponding spectrogram is obtained by referring to the envelope. The energy change curve of the word phrase is subsequently used in the DTW matching process. Time does not have a specific meaning and is only the first time point, the second time point, etc.

[0059] Furthermore, for any user other than the target user, several word groups whose word vectors are the same as those of the target user's word group are obtained, serving as several reference word groups for the target user's word group; the energy change curve of any reference word group is obtained, and DTW matching is performed with the energy change curve of the word group to obtain the DTW distance and several matching time pairs. The DTW distance is used as the energy DTW distance between the target user's word group and the other user's reference word group; where the matching time pair is the matching time pair formed by DTW matching between each time in the spectrogram matrix of the target user's word group and each time in the spectrogram matrix of the other user's reference word group.

[0060] Furthermore, the PYIN algorithm (an existing technique for obtaining fundamental frequency trajectories, which will not be elaborated upon in this embodiment) is used to obtain the fundamental frequency trajectories of the two spectrogram matrices respectively. For the same matching time pair, the fundamental frequency difference value is calculated for the time under the two spectrogram matrices, that is, the absolute value of the difference is calculated based on the data corresponding to the fundamental frequency trajectories of the two time points. The fundamental frequency difference value of each matching time pair is then obtained. The mean, variance, maximum value, and minimum value are calculated for all fundamental frequency difference values ​​respectively. The sum of all energy values ​​corresponding to any column element in the spectrogram matrix of the target user's word group is calculated as the energy sum of the corresponding time point. The curve is then constructed in chronological order as the energy curve of the word group. For the same matching time pair... At each time point in the next two-dimensional spectral matrix, the energy and difference values ​​are calculated. That is, based on the energy and the absolute value of the difference between the two time points, the energy and difference values ​​of each matching time point pair are obtained. The mean, variance, maximum value, and minimum value of all energy and difference values ​​are calculated respectively. The mean, variance, maximum value, and minimum value of all fundamental frequency difference values ​​between the target user's word group and the reference word group of other users, the mean, variance, maximum value, and minimum value of all energy and difference values, and the energy DTW distance are used as elements of the personalized difference vector between the target user's word group and the reference word group of other users, thus obtaining the personalized difference vector between the target user's word group and the reference word group of other users.

[0061] Furthermore, following the above method, the personalized difference vectors of the target user's word group and each reference word group of other users are obtained. The average vector of all personalized difference vectors is obtained, that is, the average value of the elements at the same position in the personalized difference vectors is calculated, and the average vector is used as the personalized vector of the target user's word group.

[0062] It should be noted that after time alignment based on the energy change curve, the differences between energy and fundamental frequency trajectories at the matching time point are further analyzed as features. By combining the fundamental frequency difference value and energy difference value with the energy DTW distance, a personalized difference vector is constructed. The average vector of the personalized difference vectors of the same phrases for all users is obtained, thus obtaining the personalized vector.

[0063] At this point, personalized vectors for each user's word groups are obtained.

[0064] Step S003: Based on the similarity relationship of word vectors of different word groups of the same user, classify several word groups; analyze the differences between the personalized vectors of different word groups in the same category to obtain several sensitive word groups of the user; based on the distribution of the parts of speech of the word groups in the user's questions, obtain several habitual parts of speech combinations of each user, and combine the distribution of sensitive word groups and other word groups before and after them in the user's questions to obtain several sensitive word combinations of the user.

[0065] It should be noted that in the process of users expressing their needs, the same need may be expressed using the same meaning but different phrases. Users have similar expressions for the same type of phrases. Therefore, by comparing users' expressions among phrases with the same meaning, we can determine which phrases are sensitive and unique information for users, and thus obtain the sensitivity of each phrase, thereby filtering sensitive phrases.

[0066] Preferably, in one embodiment of the present invention, several categories of word groups are classified based on the similarity relationship of word vectors of different word groups from the same user. The specific method includes:

[0067] For all word groups of the target user, DBSCAN clustering is performed based on the word vectors of each word group. The distance metric is the difference between the cosine similarity between the word vectors of different word groups minus 1, resulting in several clusters. All word groups in the same cluster are considered as one word group, thus obtaining several word groups.

[0068] Preferably, in one embodiment of the present invention, the method of analyzing the differences between personalized vectors of different word groups within the same category to obtain several sensitive word groups for the user includes:

[0069] For any personalized vector of a target user's word group, the cosine similarity between the personalized vector of the word group and the personalized vectors of other word groups of the same type is obtained. A similarity threshold is preset. In this embodiment, the similarity threshold is described as 0.35. The ratio of the number of other word groups of the same type whose cosine similarity with the personalized vector of the word group is less than the similarity threshold to the total number of other word groups of the same type is used as the sensitivity of the word group.

[0070] Furthermore, a preset screening ratio is used. In this embodiment, the screening ratio is described as 10%. The word groups with the highest and lowest sensitivity in any word group are selected as the sensitive word groups of that word group. This results in a number of sensitive word groups for each type of word group of the target user. Other word groups that are not sensitive word groups in each type of word group are selected as non-sensitive word groups.

[0071] It should be further explained that, since the contextual level of a single phrase is limited, and some non-sensitive phrases and sensitive phrases are combined to express the user's meaning, it is not accurate to judge based solely on a single sensitive phrase. Therefore, a comprehensive representation is made using combinations of non-sensitive and sensitive words. Based on the part-of-speech distribution of each phrase in the user's question, by constructing part-of-speech combinations and filtering habitual part-of-speech combinations, the sensitive and non-sensitive phrases are further analyzed and combined to finally obtain the sensitive word combinations.

[0072] Preferably, in one embodiment of the present invention, based on the distribution of the parts of speech corresponding to phrases in user questions, several habitual parts of speech combinations for each user are obtained. Combined with the distribution of sensitive phrases and other phrases preceding and following them in the user questions, several sensitive word combinations for the user are obtained. The specific method includes:

[0073] For all phrases in the target user's question, part-of-speech tagging is performed on each phrase using an LLM large-scale model to obtain the part-of-speech of each phrase. The part-of-speech of all phrases in the target user's question is arranged according to the distribution order of the phrases, forming the part-of-speech sequence of the target user's question. Adjacent elements in the part-of-speech sequence are considered as a part-of-speech combination, resulting in several part-of-speech combinations. A custom threshold is preset (in this embodiment, the custom threshold is described as 1 / 3). The ratio of the number of times any part-of-speech combination appears in all part-of-speech combinations in the part-of-speech sequence to the total number of all part-of-speech combinations in the part-of-speech sequence is used as the custom factor of that part-of-speech combination. If the custom factor is greater than the custom threshold, the part-of-speech combination is considered a custom part-of-speech combination. It should be noted that since part-of-speech combinations may appear multiple times in the part-of-speech sequence, custom judgment only needs to be performed once for the same part-of-speech combination. Thus, several custom part-of-speech combinations of the target user are obtained.

[0074] Furthermore, for any sensitive word group of the target user, obtain the preceding and following word groups in the target user's question. Combine the non-sensitive word groups from the two adjacent word groups with the sensitive word group as a candidate word group combination (if they are not non-sensitive, no candidate word group combination is constructed), and ensure that there is an order among the word groups in the candidate word group combination. Obtain the part-of-speech combination corresponding to the candidate word group combination. If it is a habitual part-of-speech combination of the target user, then the candidate word group combination is used as a habitual word group combination of the sensitive word group. If it is not a habitual part-of-speech combination of the target user, then the candidate word group combination is used as a related word group combination of the sensitive word group. If the sensitive word group and its two adjacent word groups do not constitute a candidate word group combination or a habitual word group combination, i.e., both adjacent word groups are sensitive word groups or both constitute related word group combinations, then the sensitive word group is directly used as a sensitive word group combination of the target user.

[0075] Furthermore, for any habitual word combination of the sensitive word group, the frequency of occurrence of the sensitive word group in the habitual word combination is obtained. The frequency of occurrence is the ratio of the number of times the sensitive word group appears in all word combinations in the target user's question to the total number of word combinations. A word combination with the same word vector as the sensitive word group is considered as one occurrence of the sensitive word group. The frequency of occurrence of non-sensitive word groups in the habitual word combination and the frequency of occurrence of the habitual word combination are obtained. Then, the correlation strength between the sensitive word group and non-sensitive word groups in the habitual word combination is obtained through the point mutual information algorithm. The point mutual information algorithm calculates the PMI value of the two word groups using existing technology, that is, it is calculated based on the co-occurrence probability (the frequency of occurrence of the habitual word combination) and the frequency of occurrence of the two word groups respectively. This embodiment will not be elaborated further. A preset association threshold is set. In this embodiment, the association threshold is described as 0.5. If the association strength is greater than the association threshold, the habitual word combination of the sensitive word group is considered as a sensitive word combination. It is particularly noted that if the sensitive word group and two adjacent word groups both constitute habitual word combinations and both are considered as sensitive word combinations, the two sensitive word combinations do not need to be merged.

[0076] It should be noted that, based on the candidate word combinations, we filter them by habitual part-of-speech combinations to obtain habitual word combinations. We then further analyze the association strength by analyzing the frequency of occurrence and co-occurrence probability. The stronger the association, the closer the relationship between the corresponding non-sensitive word combinations and sensitive word combinations, while satisfying the user's habitual expression. In the subsequent neural network learning process, we should pay more attention to learning such sensitive word combinations to jointly express the corresponding sensitive information.

[0077] At this point, we have obtained several combinations of sensitive words from each user.

[0078] Step S004: Prune the speech synthesis network based on several sensitive word combinations in the user's question, and perform personalized speech synthesis for the digital human.

[0079] It should be noted that after analyzing user questions and obtaining several combinations of sensitive words, the neurons of the speech synthesis network need to be pruned accordingly. The speech synthesis network is the neural network model used in the personalized speech synthesis of the AI ​​platform digital human. It adopts existing methods related to AI platform digital human technology, and the specific network structure will not be described in this embodiment. During the training of the speech synthesis network through combinations of sensitive words, the gradient changes of each neuron reflect whether it needs to be pruned, that is, the gradient reflects the impact on sensitive information.

[0080] Specifically, in the personalized speech synthesis of digital humans on the AI ​​platform, a speech synthesis network is used for speech synthesis, and pruning training is performed using word phrases from the target user's questions. The loss function uses the cross-entropy loss function to calculate the sensitivity loss of sensitive word combinations. After inputting all sensitive word combinations of the target user into the speech synthesis network, the loss function obtains the gradient values ​​of each neuron in the network model. If the gradient value is large, it indicates that it has a significant impact on the sensitivity loss; if the gradient value is close to 0, it indicates that it is irrelevant to the sensitive information. A preset pruning ratio is set, which is described as 5% in this embodiment. All neurons are arranged in ascending order of their gradient values. Neurons with the previous pruning ratio are marked as neurons to be pruned and pruned. After pruning, input and pruning training are performed again, and pruning training is iteratively performed according to the pruning ratio until the gradient values ​​of the retained neurons all reflect that they have a significant impact on the sensitivity loss. Pruning is then stopped, and the final trained speech synthesis network is obtained. The specific iteration termination operation is a prior art of pruning neurons based on gradient values, which is not specifically limited in this embodiment and will not be described in detail here.

[0081] Furthermore, for the target user's question, the final trained speech synthesis network generates corresponding word groups and arranges them in order to form the answer to the target user's question. The AI ​​platform digital human then plays the answer aloud, thus realizing personalized speech synthesis for the AI ​​platform digital human.

[0082] This concludes the embodiment.

[0083] Please see Figure 2 This illustrates a block diagram of a personalized speech synthesis system for digital humans on an AI platform, provided by another embodiment of the present invention. The system includes:

[0084] Voice acquisition module 101: Acquires voice signals of several users asking questions to the AI ​​platform, and acquires the spectrogram of each voice signal;

[0085] Speech processing module 102: Recognizes and segments the speech signal of the user's question to obtain several word groups and word vectors of the user's question. Combined with the spectrogram, it obtains the spectrogram matrix of each word group. For the spectrogram matrices of the same word group of different users, it performs DTW matching based on the energy value at time. It analyzes the fundamental frequency difference and energy difference of the two spectrogram matrices at the matching time. Combined with the energy DTW distance of the two spectrogram matrices, it obtains the personalized difference vector of the same word group of different users, and then obtains the personalized vector of each word group of each user.

[0086] Based on the similarity relationship of word vectors of different word groups of the same user, several categories of word groups are classified; the differences between the personalized vectors of different word groups in the same category are analyzed to obtain several sensitive word groups of the user; based on the distribution of the parts of speech of word groups in user questions, several habitual parts of speech combinations of each user are obtained, and combined with the distribution of sensitive word groups and other word groups before and after them in user questions, several sensitive word combinations of the user are obtained.

[0087] Speech synthesis module 103: Prunes the speech synthesis network based on a combination of several sensitive words in the user's question and performs personalized speech synthesis for the digital human.

[0088] Another embodiment of the present invention provides a personalized voice synthesis electronic device for an AI platform digital human. The electronic device includes a memory, a processor, and a computer program stored in the memory and running on the processor. When the processor executes the computer program, it implements the above-described method steps S001 to S004.

[0089] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A personalized speech synthesis method for digital humans on an AI platform, characterized in that, The method includes the following steps: Acquire the voice signals of several users asking questions to the AI ​​platform, and obtain the spectrogram of each voice signal; The speech signal of the user's question is identified and segmented to obtain several word groups and word vectors of the user's question. Combined with the spectrogram, the spectrogram matrix of each word group is obtained. The DTW matching of the spectrogram matrices of the same word group of different users is performed based on the energy value. The fundamental frequency difference and energy difference of the two spectrogram matrices at the matching time are analyzed. Combined with the energy DTW distance of the two spectrogram matrices, the personalized difference vector of the same word group of different users is obtained, and then the personalized vector of each word group of each user is obtained. Based on the similarity relationship of word vectors of different word groups of the same user, several categories of word groups are classified; the differences between the personalized vectors of different word groups in the same category are analyzed to obtain several sensitive word groups of the user; based on the distribution of the parts of speech of word groups in user questions, several habitual parts of speech combinations of each user are obtained, and combined with the distribution of sensitive word groups and other word groups before and after them in user questions, several sensitive word combinations of the user are obtained. The speech synthesis network is pruned based on several sensitive word combinations in the user question, and personalized speech synthesis for digital humans is performed. The pruning of the speech synthesis network based on several sensitive word combinations in the user question includes: after inputting the sensitive word combinations into the speech synthesis network, pruning the speech synthesis network based on the gradient values ​​of each neuron, wherein the gradient values ​​are used to reflect the degree of influence of each neuron on the sensitivity loss calculated based on the sensitive word combinations.

2. The personalized speech synthesis method for digital humans on an AI platform according to claim 1, characterized in that, The specific method for obtaining several word groups and word vectors of the user's question, and combining them with a spectrogram to obtain the spectrogram matrix of each word group, includes: Take any user as the target user, recognize the speech signal of the target user's question, convert the speech signal into text information, segment the text information into words to obtain several word groups, and perform vector transformation on each word group to obtain the word vector of each word group. For the spectrogram of the speech signal of the target user's question, the horizontal axis of the spectrogram is time, the vertical axis is frequency value, and the pixel value of each data point in the spectrogram represents the energy value. Obtain the time period corresponding to each word group in the spectrogram, and take a segment of the spectrogram of any word group of the target user under the corresponding time period in the spectrogram as the spectrogram matrix of the target user's word group; In the spectrogram matrix, rows correspond to the same frequency, columns correspond to the same time, and elements are the energy values ​​represented by the pixel values ​​of data points in the spectrogram.

3. The personalized speech synthesis method for digital humans on an AI platform according to claim 2, characterized in that, The specific method for performing time-based DTW matching of the spectrogram matrix for the same word group from different users based on energy values ​​includes: For any word group of the target user, obtain the maximum value of the energy value of each column element in the spectrum matrix, and construct a curve in time order to serve as the energy change curve of the word group. For any user other than the target user, obtain several word groups whose word vectors are the same as those of the target user for that word group, and use them as several reference word groups for the target user's word group; obtain the energy change curve of any reference word group, and perform DTW matching with the energy change curve of the word group to obtain the DTW distance and several matching time pairs, and use the DTW distance as the energy DTW distance between the target user's word group and the other user's reference word group; the matching time pairs are the matching time pairs formed by DTW matching between each time in the spectrogram matrix of the target user's word group and each time in the spectrogram matrix of the other user's reference word group.

4. The personalized speech synthesis method for digital humans on an AI platform according to claim 3, characterized in that, The specific method for obtaining personalized difference vectors of the same word groups from different users, and then obtaining personalized vectors of each word group for each user, includes: Using the PYIN algorithm, the fundamental frequency trajectory of the spectrogram matrix of any word group of the target user is obtained, as well as the fundamental frequency trajectory of the spectrogram matrix of any reference word group of the target user for any other user other than the target user. For the time of the next two spectrogram matrices under the same matching time pair, the fundamental frequency difference value is calculated to obtain the fundamental frequency difference value of each matching time pair. The mean, variance, maximum value and minimum value of all fundamental frequency difference values ​​are calculated respectively. For any column of the spectrogram matrix of the target user's word group, calculate the sum of all energy values ​​corresponding to the elements of the corresponding time, and construct a curve in chronological order as the energy curve of the word group; for the time of the next two spectrogram matrices under the same matching time, calculate the energy and difference values ​​to obtain the energy and difference values ​​of each matching time pair, and calculate the mean, variance, maximum and minimum values ​​of all energy and difference values ​​respectively. The mean, variance, maximum and minimum values ​​of all fundamental frequency differences between the target user's word group and the reference word group of other users, the mean, variance, maximum and minimum values ​​of all energy and difference values, and the energy DTW distance are used as elements of the personalized difference vector between the target user's word group and the reference word group of other users to obtain the personalized difference vector between the target user's word group and the reference word group of other users. Obtain the personalized difference vectors between the target user's word group and each reference word group of other users besides the target user, obtain the average vector of all personalized difference vectors, and use the average vector as the personalized vector of the target user's word group.

5. The personalized speech synthesis method for digital humans on an AI platform according to claim 1, characterized in that, The classification yields several categories of word groups, including the following specific methods: For all word groups of the target user, density clustering is performed based on the word vectors of each word group. The distance metric is the difference between the cosine similarity between the word vectors of different word groups minus 1, resulting in several clusters. All word groups in the same cluster are regarded as one word group, resulting in several word groups.

6. The personalized speech synthesis method for digital humans on an AI platform according to claim 2, characterized in that, The specific methods for obtaining several sensitive word groups from the user include: For any word group of a target user, obtain the cosine similarity between the word group and the personalized vectors of other word groups of the same type. Set a similarity threshold and use the ratio of the number of other word groups of the same type whose cosine similarity with the personalized vector of the word group is less than the similarity threshold to the total number of other word groups of the same type as the sensitivity of the word group. The preset filtering ratio selects the word groups in any word group that are filtered first from the highest to the lowest sensitivity level. These selected word groups are then considered as sensitive word groups for that word group. This process yields a number of sensitive word groups for each word group for the target user. Other word groups that are not considered sensitive word groups are then considered as non-sensitive word groups.

7. The personalized speech synthesis method for a digital human on an AI platform according to claim 6, characterized in that, The specific method for obtaining the various habitual parts of speech combinations of each user is as follows: For all phrases in the target user's question, the LLM large model is used to perform part-of-speech tagging on each phrase to obtain the part-of-speech of each phrase. The part-of-speech of all phrases in the target user's question is arranged according to the distribution order of the phrases to form the part-of-speech sequence of the target user's question. Adjacent elements in the part-of-speech sequence are combined as a part-of-speech combination to obtain several part-of-speech combinations. A preset habit threshold is set, and the ratio of the number of times any word class combination appears in all word class combinations in the word class sequence to the total number of word class combinations in the word class sequence is used as the habit factor of that word class combination; if the habit factor is greater than the habit threshold, the word class combination is considered as a habitual word class combination, thus obtaining several habitual word class combinations of the target user.

8. The personalized speech synthesis method for digital humans on an AI platform according to claim 7, characterized in that, The specific methods for obtaining several combinations of sensitive words from the user include: For any sensitive word group of the target user, obtain the preceding and following word groups of the target user in the question. Combine the non-sensitive word group of the two adjacent word groups with the sensitive word group as a candidate word group combination for the sensitive word group. If it is not a non-sensitive word group, no candidate word group combination is constructed. Obtain the part-of-speech combination corresponding to the candidate word combination. If it is a habitual part-of-speech combination of the target user, the candidate word combination is used as a habitual word combination of the sensitive word combination. If it is not a habitual part-of-speech combination of the target user, the candidate word combination is used as a related word combination of the sensitive word combination. If the sensitive word combination and its two adjacent word combinations do not constitute a candidate word combination or a habitual word combination, the sensitive word combination is used as a sensitive word combination of the target user. For any common word combination of the sensitive word group, obtain the frequency of occurrence of the sensitive word group in the common word combination. The frequency of occurrence is the ratio of the number of times the sensitive word group appears in all word groups in the target user's question to the total number of word groups. The frequency of non-sensitive words in the habitual word combination and the frequency of the habitual word combination are obtained. Then, the correlation strength between the sensitive words and non-sensitive words in the habitual word combination is obtained through the point mutual information algorithm. A preset association threshold is set. If the association strength is greater than the association threshold, the common word combination of the sensitive word group is treated as a sensitive word combination.

9. A personalized voice synthesis system for digital humans on an AI platform, characterized in that, The system includes: The voice acquisition module is used to acquire voice signals from several users asking questions to the AI ​​platform and to acquire the spectrogram of each voice signal. The speech processing module is used to recognize and segment the speech signal of the user's question, obtain several word groups and word vectors of the user's question, and obtain the spectrogram matrix of each word group by combining the spectrogram. The DTW matching of the spectrogram matrices of the same word group of different users is performed based on the energy value. The fundamental frequency difference and energy difference of the two spectrogram matrices at the matching time are analyzed. Combined with the energy DTW distance of the two spectrogram matrices, the personalized difference vector of the same word group of different users is obtained, and then the personalized vector of each word group of each user is obtained. Based on the similarity relationship of word vectors of different word groups of the same user, several categories of word groups are classified; the differences between the personalized vectors of different word groups in the same category are analyzed to obtain several sensitive word groups of the user; based on the distribution of the parts of speech of word groups in user questions, several habitual parts of speech combinations of each user are obtained, and combined with the distribution of sensitive word groups and other word groups before and after them in user questions, several sensitive word combinations of the user are obtained. The speech synthesis module is used to prune the speech synthesis network based on several sensitive word combinations in the user question and to perform personalized speech synthesis for the digital human. The pruning of the speech synthesis network based on several sensitive word combinations in the user question includes: after inputting the sensitive word combinations into the speech synthesis network, pruning the speech synthesis network based on the gradient values ​​of each neuron, wherein the gradient values ​​are used to reflect the degree of influence of each neuron on the sensitivity loss calculated based on the sensitive word combinations.

10. A personalized voice synthesis electronic device for an AI platform digital human, the electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the personalized speech synthesis method for a digital human on an AI platform as described in any one of claims 1-8.

Images