Personalized speech synthesis method, electronic equipment and system for digital human on AI platform

Abstract

The invention relates to the technical field of voice synthesis, and provides an AI platform digital human personalized voice synthesis method, electronic equipment and system, and the method comprises the steps: obtaining voice signals of a plurality of users asking questions to an AI platform and a spectrogram of the voice signals; obtaining a plurality of phrases and word vectors of the user question, and combining the spectrogram to obtain a spectrogram matrix of each phrase; fundamental frequency differences and energy differences of the speech spectrum matrixes of the same phrases of different users are analyzed, and personalized vectors of the phrases of the users are obtained; classifying to obtain a plurality of classes of phrases; obtaining a plurality of sensitive phrases of the user; obtaining a plurality of sensitive word combinations of the user based on the distribution of part-of-speech corresponding to the phrases in the user question and the distribution of the sensitive phrases and other phrases before and after the sensitive phrases in the user question; and pruning the speech synthesis network, and carrying out personalized speech synthesis of the digital human. The objective of the invention is to solve the problem that a neural network model cannot be trained in a targeted manner due to the fact that user voice contains personalized information.

Description

Technical Field

[0001] The present invention relates to the field of speech synthesis technology, and in particular to a personalized speech synthesis method, electronic device, and system for digital humans on an AI platform. Background Art

[0002] With the rapid development of virtual interaction and artificial intelligence technologies, digital humans have evolved from concepts to large-scale commercial applications, and are widely used in virtual anchors, intelligent customer service, online education, game NPCs, virtual idols and other fields. When users interact with digital humans on AI platforms, they ask questions to the digital humans. The digital humans understand the user's meaning based on semantics and provide personalized responses.

[0003] Although traditional text-to-speech (TTS) technology can achieve clear and understandable voice broadcasts, its timbre is monotonous and lacks emotion and personalization. Response voice synthesis is often generated through neural networks. The voice synthesis network analyzes the user's questions and generates corresponding answers. During the training process of the voice synthesis network, the training set used does not reflect personalized information, that is, it does not include personalized factors such as the user's timbre and intonation. At the same time, the neural network structure is too redundant, and a lot of useless information will be learned during the training process. As a result, when providing personalized replies, answers without characteristics are generated, which in turn affects the voice synthesis effect of the digital human and the user's interactive experience with the digital human. Summary of the Invention

[0004] The present invention provides a personalized speech synthesis method, electronic device, and system for AI platform digital humans to solve the existing problem that neural network models cannot be trained specifically due to the personalized information contained in user speech. The technical solutions adopted are as follows: The present invention proposes a personalized speech synthesis method for a digital human on an AI platform, which includes the following steps: Obtain voice signals of several users asking questions to the AI ​​platform, and obtain the spectrogram of each voice signal; The speech signals of user questions are recognized and segmented to obtain several phrases and word vectors of the user questions. The spectrogram is then combined to obtain the spectrum matrix of each phrase. The spectrum matrices of the same phrases of different users are matched at any given moment based on the energy value. The fundamental frequency and energy differences of the two spectrum matrices at the matching moment are analyzed. The energy DTW distance of the two spectrum matrices is combined to obtain the personalized difference vectors of the same phrases of different users, and thus the personalized vectors of each phrase of each user are obtained. Based on the similarity between the word vectors of different phrases for the same user, we classify them into several categories. We analyze the differences between the personalized vectors of different phrases within the same category to obtain several sensitive phrases for the user. Based on the distribution of the corresponding parts of speech of the phrases in the user's questions, we obtain several habitual part-of-speech combinations for each user. Combined with the distribution of sensitive phrases and other phrases before and after them in the user's questions, we obtain several sensitive word combinations for the user. The speech synthesis network is pruned based on several sensitive word combinations in the user's questions, and personalized speech synthesis of the digital human is performed.

[0005] Optionally, the method of obtaining several phrases and word vectors of the user's question and combining them with the spectrogram to obtain the spectrogram matrix of each phrase includes the following specific methods: Take any user as the target user, recognize the voice signal of the target user's question, convert the voice signal into text information, segment the text information into words, obtain several phrases, and convert each phrase into a vector to obtain the word vector of each phrase; For the spectrogram of the speech signal of the target user's question, the horizontal axis in the spectrogram is time, the vertical axis is frequency value, and the pixel value of each data point in the spectrogram represents the energy value; obtain the time period corresponding to each phrase in the spectrogram, and use a spectrogram of any phrase of the target user in the corresponding time period in the spectrogram as the spectrogram matrix of the phrase of the target user; wherein the same row in the spectrogram matrix corresponds to the same frequency, the same column corresponds to the same time, and the elements are the energy values ​​represented by the pixel values ​​of the data points in the spectrogram.

[0006] Optionally, the DTW matching of the spectral matrices of the same phrase of different users based on the energy value at each moment includes the following specific methods: For any phrase of the target user, the maximum value of the energy value of each column element in the spectrum matrix is ​​obtained, and a curve is constructed in chronological order as the energy change curve of the phrase; For any other user except the target user, obtain several phrases of the other user with the same word vector as the phrase of the target user as several reference phrases of the target user; obtain the energy change curve of any reference phrase, and perform DTW matching with the energy change curve of the phrase to obtain the DTW distance and several matching time pairs, and use the DTW distance as the energy DTW distance between the phrase of the target user and the reference phrase of the other user; the matching time pairs are the matching time pairs formed by DTW matching at each moment in the spectral matrix of the phrase of the target user and each moment in the spectral matrix of the reference phrase of the other user.

[0007] Optionally, the method of obtaining personalized difference vectors of the same phrase for different users and then obtaining personalized vectors for each phrase for each user includes the following specific methods: Using the PYIN algorithm, we obtain the fundamental frequency trajectory of the spectral matrix of any phrase of the target user, as well as the fundamental frequency trajectory of the spectral matrix of any reference phrase of the target user's phrase for any user other than the target user. For the moments under the two spectral matrices of the same matching moment, we calculate the fundamental frequency difference value, obtain the fundamental frequency difference value of each matching moment pair, and calculate the mean, variance, maximum and minimum values ​​of all fundamental frequency difference values ​​respectively. Calculate the sum of all energy values ​​corresponding to any column element in the grammar matrix of the target user's phrase as the energy sum at the corresponding moment, and construct a curve in chronological order as the energy curve of the phrase; calculate the energy and difference values ​​for the moments in the two grammar matrices under the same matching moment, and obtain the energy and difference values ​​of each matching moment pair. Calculate the mean, variance, maximum and minimum values ​​of all energy and difference values ​​respectively; The mean, variance, maximum, and minimum of all fundamental frequency difference values ​​between the target user's phrase and the reference phrase of the other user, the mean, variance, maximum, and minimum of all energy and difference values, and the energy DTW distance are used as elements of the personalized difference vector between the target user's phrase and the reference phrase of the other user to obtain the personalized difference vector between the target user's phrase and the reference phrase of the other user; Obtain personalized difference vectors between the target user's phrase and reference phrases of all other users except the target user, obtain an average vector of all personalized difference vectors, and use the average vector as the personalized vector of the target user's phrase.

[0008] Optionally, the classification may obtain several types of phrases, including the following specific methods: For all phrases of the target user, density clustering is performed based on the word vectors of each phrase. The distance metric is 1 minus the difference between the cosine similarity between the word vectors of different phrases to obtain several clusters. All phrases in the same cluster are regarded as a type of phrase, and several types of phrases are obtained.

[0009] Optionally, the method of obtaining the user's sensitive phrases includes: For the personalized vector of any phrase of the target user, obtain the cosine similarity between the personalized vectors of the phrase and other similar phrases. Preset a similarity threshold and use the ratio of the number of similar phrases whose cosine similarity with the personalized vector of the phrase is less than the similarity threshold to the total number of similar phrases as the sensitivity of the phrase. A screening ratio is preset, and phrases with the highest screening ratio in any category of phrases are sorted from most sensitive to least sensitive as sensitive phrases of that category, and several sensitive phrases of each category of target users are obtained, and other phrases in each category that are not sensitive phrases are regarded as non-sensitive phrases.

[0010] Optionally, the specific method for obtaining the habitual part-of-speech combinations of each user is as follows: For all phrases in the target user's question, perform part-of-speech tagging on each phrase using the LLM model to obtain the part-of-speech of each phrase. Arrange the parts-of-speech of all phrases in the target user's question in the order of the phrase distribution to form a part-of-speech sequence for the target user's question. Treat adjacent elements in the part-of-speech sequence as a part-of-speech combination to obtain several part-of-speech combinations. A habit threshold is preset, and the ratio of the number of times any part-of-speech combination appears in all part-of-speech combinations in the part-of-speech sequence to the number of all part-of-speech combinations in the part-of-speech sequence is used as the habit factor of the part-of-speech combination; if the habit factor is greater than the habit threshold, the part-of-speech combination is regarded as a habitual part-of-speech combination, and several habitual part-of-speech combinations of the target user are obtained.

[0011] Optionally, the method of obtaining a plurality of sensitive word combinations of the user includes: For any sensitive phrase of the target user, obtain the previous and next phrases of the sensitive phrase in the target user's question, and combine the non-sensitive phrase of the two adjacent phrases with the sensitive phrase as an alternative phrase for the sensitive phrase. If it is not a non-sensitive phrase, no alternative phrase combination is constructed; Obtain the part-of-speech combination corresponding to the candidate phrase combination. If it is the target user's usual part-of-speech combination, use the candidate phrase combination as the habitual phrase combination of the sensitive phrase. If it is not the target user's usual part-of-speech combination, use the candidate phrase combination as the associated phrase combination of the sensitive phrase. If the sensitive phrase and its two adjacent phrases do not constitute an candidate phrase combination or a habitual phrase combination, use the sensitive phrase as a sensitive phrase combination of the target user. For any habitual phrase combination of the sensitive phrase, obtain the occurrence frequency of the sensitive phrase in the habitual phrase combination, where the occurrence frequency is the ratio of the number of occurrences of the sensitive phrase in all phrases in the target user's question to the number of all phrases; Obtaining the frequency of occurrence of non-sensitive phrases in the habitual phrase combination and the frequency of occurrence of the habitual phrase combination, and then obtaining the correlation strength between the sensitive phrase and the non-sensitive phrase in the habitual phrase combination through a point mutual information algorithm; A correlation threshold is preset, and if the correlation strength is greater than the correlation threshold, the habitual phrase combination of the sensitive phrase is regarded as a sensitive word combination.

[0012] The present invention also proposes a personalized speech synthesis system for digital humans on an AI platform, which includes: The voice acquisition module is used to obtain voice signals of several users asking questions to the AI ​​platform and obtain the spectrogram of each voice signal; The speech processing module is used to recognize and segment the speech signals of user questions, obtain several phrases and word vectors of the user questions, and combine them with the spectrogram to obtain the spectrum matrix of each phrase. The spectrum matrices of the same phrases of different users are matched based on the energy value at the time of DTW. The fundamental frequency difference and energy difference of the two spectrum matrices at the matching time are analyzed. The energy DTW distance of the two spectrum matrices is combined to obtain the personalized difference vectors of the same phrases of different users, and then obtain the personalized vectors of each phrase of each user. Based on the similarity between the word vectors of different phrases for the same user, we classify them into several categories. We analyze the differences between the personalized vectors of different phrases within the same category to obtain several sensitive phrases for the user. Based on the distribution of the corresponding parts of speech of the phrases in the user's questions, we obtain several habitual part-of-speech combinations for each user. Combined with the distribution of sensitive phrases and other phrases before and after them in the user's questions, we obtain several sensitive word combinations for the user. The speech synthesis module is used to prune the speech synthesis network based on several sensitive word combinations in the user's questions and perform personalized speech synthesis of the digital human.

[0013] The present invention also proposes an electronic device for personalized speech synthesis of a digital human on an AI platform. The electronic device includes a memory, a processor, and a computer program stored in the memory and running on the processor. When the processor executes the computer program, the steps of the above method are implemented.

[0014] The beneficial effects of the present invention are as follows: the present invention receives the voice signal of the user asking the question on the AI ​​platform, generates the big data answer through the voice synthesis network in the AI ​​platform, and then performs voice synthesis to answer the question through the digital human, further considering the user's personalized expression, such as timbre, intonation and habitual part-of-speech combination expression and other factors, and through the spectrogram analysis of the voice signal, the word vector and spectrum matrix changes of each phrase in the user's question, and through the spectrum matrix analysis of the same phrase of different users, quantify the personalized vector to reflect the differences in the expression of each phrase of each user in terms of speech speed, intonation, etc.; by classifying the phrases of the same user based on the word vector, the personality of the same phrases of the same user is distinguished. The greater the difference in the quantization vectors, the greater the impact of the corresponding phrases on the differences in the user's expression methods in different contexts, and sensitive phrases are screened accordingly; habitual part-of-speech combinations are obtained by combining the expression methods of habitual part-of-speech combinations, and sensitive phrases are combined with other phrases before and after them to obtain joint sensitive information. Sensitive word combinations are obtained and used for pruning training of the speech synthesis network. Through neuron pruning of the speech synthesis network, the retained neurons learn the sensitive information of sensitive word combinations more efficiently, reduce the learning of redundant and invalid information, and thus improve the accuracy of the speech synthesis network, and generate corresponding answers to user questions. Finally, digital humans perform speech expression to realize personalized speech synthesis of digital humans on the AI ​​platform. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0016] Figure 1 A flowchart of a personalized speech synthesis method for a digital human on an AI platform provided by one embodiment of the present invention; Figure 2 This is a structural block diagram of the personalized speech synthesis system of the AI ​​platform digital human provided by another embodiment of the present invention. DETAILED DESCRIPTION

[0017] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0018] See also Figure 1 , which shows a flow chart of a personalized speech synthesis method for a digital human on an AI platform provided by one embodiment of the present invention, the method comprising the following steps: Step S001: Acquire voice signals of several users asking questions to the AI ​​platform, and obtain a spectrogram of each voice signal.

[0019] The purpose of this embodiment is to use the AI ​​platform's digital human to perform personalized speech synthesis on the voice signal received by the AI ​​platform when the user asks a question, so as to provide a voice answer to the user's question. First, it is necessary to obtain the voice signal of the user's question. At the same time, in order to analyze the user's personalized expression such as timbre and intonation, it is necessary to obtain a spectrogram of the voice signal for corresponding analysis.

[0020] Specifically, each user asks a question to the AI ​​platform, and the AI ​​platform receives a voice signal, and matches each received voice signal with the user as the voice signal of each user's question; for any voice signal, denoising is performed through preprocessing, where signal denoising is a well-known technology and will not be described in detail in this embodiment, and the preprocessed voice signal is converted into a spectrogram to obtain a spectrogram of each voice signal.

[0021] It should be noted that when users interact with the digital human on the AI ​​platform, they will ask questions to the digital human. The digital human will understand what the user wants to express based on semantic understanding and provide personalized responses, which is the personalized voice generation process. Semantic understanding and voice generation are usually output through neural networks. However, in the voice synthesis network, the neural network is too redundant and will learn a lot of invalid information. In the personalized reply, invalid answers will be generated, so network pruning is required.

[0022] Step S002: Identify and segment the voice signal of the user's question to obtain several phrases and word vectors of the user's question, and obtain the spectrum matrix of each phrase by combining the spectrum graph; perform DTW matching on the spectrum matrices of the same phrase of different users based on the energy value at each moment, analyze the fundamental frequency difference and energy difference of the two spectrum matrices at the matching moment, and combine the energy DTW distance of the two spectrum matrices to obtain personalized difference vectors of the same phrase of different users, and then obtain the personalized vector of each phrase of each user.

[0023] It should be noted that by analyzing the information asked by the user, the contextual information of the user when asking the question can be determined, and the user's contextual information includes the context of the answer to the question that the user wants to get at that time. Therefore, in order to make the synthesized reply voice more suitable for the user's question, it is necessary to analyze the user's question context, and generate a reasonable answer by analyzing the question context, and then generate a personalized reply; because the contextual information can be reflected in the tone and content of the voice signal, in order to express the context of this voice signal, it is obtained by analyzing the energy value changes in the spectrogram, and at the same time, the content of the voice signal is analyzed using character phrases, so as to analyze the user's personalized characteristics based on the text content and tone.

[0024] Preferably, in one embodiment of the present invention, the speech signal of the user question is recognized and segmented to obtain several phrases and word vectors of the user question, and the spectrogram is combined to obtain the spectrogram matrix of each phrase, including the specific method of: Any user is taken as the target user, and the voice signal of the target user's question is recognized, the voice signal is converted into text information, and the text information is segmented to obtain several phrases, and each phrase is vector-converted to obtain a word vector for each phrase; wherein the conversion of voice signals into text information is an existing technology, such as the BERT model, the Transformer model, etc., which performs text recognition on voice signals, segments the text information through Jieba segmentation, and uses the Word2vec model for word vector conversion, wherein the number of elements in the word vectors of different phrases is the same, that is, the number of dimensions is the same. The above technologies are all existing methods and will not be repeated in this embodiment.

[0025] Furthermore, for the spectrogram of the speech signal of the target user's question, the horizontal axis in the spectrogram is time, the vertical axis is frequency value, and the pixel value of each data point in the spectrogram represents the energy value. Since each phrase obtained from the text information corresponds to a period of time in the horizontal axis of the spectrogram, that is, the time axis of the text information corresponds to the time axis of the speech signal and the spectrogram, the time period corresponding to each phrase in the spectrogram can be obtained. A spectrogram of any phrase of the target user in the corresponding time period in the spectrogram is used as the spectrogram matrix of the phrase of the target user, where the same row in the spectrogram corresponds to the same frequency, the same column corresponds to the same time, and the element is the energy value represented by the pixel value of the data point in the spectrogram.

[0026] It should be further explained that the user's personalized characteristics are represented by constructing a feature vector. The user's personalized characteristics are reflected in the different intonation information corresponding to the same text of different users, which means that the personalized characteristics of the user are contained. Since different users have different intonations when speaking a word, time alignment is required.

[0027] Preferably, in one embodiment of the present invention, DTW matching is performed on the spectral matrices of the same phrases of different users based on the energy value at the moment, and the fundamental frequency difference and energy difference of the two spectral matrices at the matching moment are analyzed. The energy DTW distance of the two spectral matrices is combined to obtain the personalized difference vectors of the same phrases of different users, and then the personalized vectors of each phrase of each user are obtained. The specific method includes: For the spectral matrix of any phrase of the target user, that is, the corresponding spectrogram, the maximum value of the energy value of each column element in the spectral matrix is ​​obtained, and a curve is constructed in chronological order as the energy change curve of the phrase, that is, the maximum value of the pixel value at each moment in the corresponding spectrogram is obtained by referring to the envelope. The energy change curve of the phrase is subsequently used in the DTW matching process. The time does not have a specific meaning, and is only the first moment, the second moment, etc.

[0028] Furthermore, for any other user except the target user, several phrases of the other user with the same word vector as the phrase of the target user are obtained as several reference phrases of the phrase of the target user; the energy change curve of any reference phrase is obtained, and DTW matching is performed with the energy change curve of the phrase to obtain the DTW distance and several matching time pairs, and the DTW distance is used as the energy DTW distance between the phrase of the target user and the reference phrase of the other user; the matching time pair is the matching time pair formed by DTW matching at each moment in the spectral matrix of the phrase of the target user and each moment in the spectral matrix of the reference phrase of the other user.

[0029] Furthermore, the fundamental frequency trajectories of the two spectral matrices are obtained respectively by using the PYIN algorithm (the existing technology for obtaining fundamental frequency trajectory, which will not be described in detail in this embodiment), and the fundamental frequency difference value is calculated for the moments under the two spectral matrices at the same matching moment, that is, the absolute value of the difference is calculated based on the data corresponding to the respective fundamental frequency trajectories at the two moments, and the fundamental frequency difference value of each matching moment pair is obtained. The mean, variance, maximum value and minimum value of all fundamental frequency difference values ​​are calculated respectively; the sum of all energy values ​​corresponding to any column element in the spectral matrix of the phrase of the target user is calculated as the energy sum of the corresponding moment, and a curve is constructed in chronological order as the energy curve of the phrase; for the same matching moment pair At the moments under the two spectral matrices, the energy and difference values ​​are calculated, that is, the absolute value of the difference is calculated based on the energy and corresponding to the two moments, and the energy and difference values ​​of each matching moment pair are obtained, and the mean, variance, maximum and minimum values ​​of all energy and difference values ​​are calculated respectively; the mean, variance, maximum and minimum values ​​of all fundamental frequency difference values ​​between the target user's phrase and the reference phrase of the other user, the mean, variance, maximum and minimum values ​​of all energy and difference values, and the energy DTW distance are used as the elements of the personalized difference vector between the target user's phrase and the reference phrase of the other user, and the personalized difference vector between the target user's phrase and the reference phrase of the other user is obtained.

[0030] Furthermore, according to the above method, the personalized difference vectors between the target user's phrase and the reference phrases of other users except the target user are obtained, and the average vector of all personalized difference vectors is obtained, that is, the elements in the same position in the personalized difference vectors are averaged respectively, and the average vector is used as the personalized vector of the target user's phrase.

[0031] It should be noted that after time alignment based on the energy change curve, the differences in energy and fundamental frequency trajectories at the matching time are further analyzed as features. The fundamental frequency difference value and energy sum difference value are combined with the energy DTW distance to comprehensively construct a personalized difference vector, and the average vector of the personalized difference vectors of the same phrase for all users is obtained to obtain a personalized vector.

[0032] At this point, the personalized vectors of each user and each phrase are obtained.

[0033] Step S003: Based on the similarity between the word vectors of different phrases of the same user, classify them into several categories of phrases; analyze the differences between the personalized vectors of different phrases in the same category to obtain several sensitive phrases of the user; based on the distribution of the corresponding parts of speech of the phrases in the user's questions, obtain several habitual part-of-speech combinations of each user, and combine the distribution of sensitive phrases and other phrases before and after them in the user's questions to obtain several sensitive word combinations of the user.

[0034] It should be noted that in the process of users expressing their needs, the same need may be expressed with the same meaning but different phrases. Users have similar expressions for the same type of phrases. Therefore, by comparing users' expressions between phrases with the same semantics, we can determine which phrases are sensitive and unique information for users, and then obtain the sensitivity of each phrase to screen out sensitive phrases.

[0035] Preferably, in one embodiment of the present invention, based on the similarity between the word vectors of different word groups of the same user, several categories of word groups are classified, including the following specific methods: For all phrases of the target user, DBSCAN clustering is performed based on the word vectors of each phrase. The distance metric is 1 minus the difference between the cosine similarity between the word vectors of different phrases to obtain several clusters. All phrases in the same cluster are regarded as a type of phrase, and several types of phrases are obtained.

[0036] Preferably, in one embodiment of the present invention, the differences between personalized vectors of different phrases in the same category are analyzed to obtain several sensitive phrases of the user, including the following specific methods: For the personalized vector of any phrase of the target user, the cosine similarity between the personalized vectors of the phrase and other similar phrases is obtained. A similarity threshold is preset. In this embodiment, the similarity threshold is 0.35. The sensitivity of the phrase is determined by the ratio of the number of similar phrases whose cosine similarity with the personalized vector of the phrase is less than the similarity threshold to the total number of similar phrases.

[0037] Furthermore, a screening ratio is preset. In this embodiment, the screening ratio is described as 10%. The phrases in any category of phrases that have the highest sensitivity and the highest screening ratio are sorted from the highest to the lowest, and are used as sensitive phrases for that category of phrases. Then, several sensitive phrases for each category of phrases for the target user are obtained, and other phrases in each category that are not sensitive phrases are used as non-sensitive phrases.

[0038] It should be further explained that since the context expressed by a single phrase is limited, and some non-sensitive phrases and sensitive phrases are combined to express the user's meaning, the accuracy of judgment based solely on a single sensitive phrase is not high. Therefore, a comprehensive representation is made using a combination of non-sensitive words and sensitive words; based on the part-of-speech distribution of each phrase in the user's question, by constructing part-of-speech combinations and screening habitual part-of-speech combinations, further combined analysis of sensitive and non-sensitive phrases is performed, and finally a sensitive word combination is obtained.

[0039] Preferably, in one embodiment of the present invention, based on the distribution of parts of speech corresponding to phrases in user questions, several habitual part-of-speech combinations of each user are obtained, and combined with the distribution of sensitive phrases and other phrases before and after them in user questions, several sensitive word combinations of the user are obtained. The specific method includes: For all phrases in the target user's question, each phrase is tagged with part of speech through the LLM large model to obtain the part of speech of each phrase, and the parts of speech of all phrases in the target user's question are arranged in the order of the distribution of the phrases as the part-of-speech sequence of the target user's question; adjacent elements in the part-of-speech sequence are regarded as a part-of-speech combination, and several part-of-speech combinations are obtained; a habit threshold is preset, and the habit threshold in this embodiment is described as 1 / 3, and the number of times any part-of-speech combination appears in all part-of-speech combinations in the part-of-speech sequence is ratioed to the number of all part-of-speech combinations in the part-of-speech sequence as the habit factor of the part-of-speech combination; if the habit factor is greater than the habit threshold, the part-of-speech combination is regarded as a habitual part-of-speech combination; it should be noted that since the part-of-speech combination may be distributed in the part-of-speech sequence multiple times, the habit judgment only needs to be made once for the same part-of-speech combination; thus, several habitual part-of-speech combinations of the target user are obtained.

[0040] Furthermore, for any sensitive phrase of the target user, the previous phrase and the next phrase of the sensitive phrase in the target user's question are obtained, and the phrase that is a non-sensitive phrase in the two adjacent phrases is used together with the sensitive phrase as an alternative phrase combination for the sensitive phrase (if it is not a non-sensitive phrase, no alternative phrase combination is constructed), and there is an order between the phrases in the alternative phrase combination; the part-of-speech combination corresponding to the alternative phrase combination is obtained, and if it is the target user's habitual part-of-speech combination, the alternative phrase combination is used as the habitual phrase combination of the sensitive phrase; if it is not the target user's habitual part-of-speech combination, the alternative phrase combination is used as the associated phrase combination of the sensitive phrase; if the sensitive phrase and its two adjacent phrases do not constitute an alternative phrase combination or a habitual phrase combination, that is, the two adjacent phrases are both sensitive phrases or constitute an associated phrase combination, then the sensitive phrase is directly used as a sensitive word combination for the target user.

[0041] Furthermore, for any habitual phrase combination of the sensitive phrase group, the frequency of occurrence of the sensitive phrase in the habitual phrase combination is obtained, where the frequency of occurrence is the ratio of the number of occurrences of the sensitive phrase in all phrases in the target user's question to the number of all phrases, where a phrase with the same word vector as the sensitive phrase is deemed to appear once. The frequency of occurrence of non-sensitive phrases in the habitual phrase combination and the frequency of occurrence of the habitual phrase combination are obtained, and then the correlation strength between the sensitive phrase and the non-sensitive phrase in the habitual phrase combination is obtained using a point mutual information algorithm. The point mutual information algorithm calculates the PMI value of two phrases in the prior art, i.e., it is calculated based on the co-occurrence probability (the frequency of occurrence of the habitual phrase combination) and the frequency of occurrence of each of the two phrases, which will not be described in detail in this embodiment. A correlation threshold is preset, and in this embodiment, the correlation threshold is described as 0.5. If the correlation strength is greater than the correlation threshold, the habitual phrase combination of the sensitive phrase group is regarded as a sensitive word combination. It is particularly noted that if the sensitive phrase and two adjacent phrases constitute a habitual phrase combination and are both used as sensitive word combinations, the two sensitive word combinations do not need to be merged.

[0042] What needs to be explained is that, based on the alternative phrase combinations, we screen through the habitual part-of-speech combinations, obtain the habitual phrase combinations, and further analyze the association strength through the frequency of occurrence and co-occurrence probability. The greater the association strength, the closer the association between the corresponding non-sensitive phrases and sensitive phrases on the basis of satisfying the user's habitual expression mode. In the subsequent neural network learning process, we should pay more attention to learning this sensitive word combination to jointly express the corresponding sensitive information.

[0043] At this point, several sensitive word combinations of each user are obtained.

[0044] Step S004: Prune the speech synthesis network based on several sensitive word combinations in the user's question, and perform personalized speech synthesis of the digital human.

[0045] It should be noted that after analyzing the user's questions to obtain several sensitive word combinations, it is necessary to use these to prune the neurons of the speech synthesis network. The speech synthesis network is a neural network model used in the personalized speech synthesis of digital humans on the AI ​​platform. It adopts existing methods of related technologies of digital humans on the AI ​​platform. The specific network structure will not be described in detail in this embodiment. In the process of training the speech synthesis network through sensitive word combinations, the gradient changes of each neuron reflect whether it needs to be pruned, that is, the gradient reflects the impact on sensitive information.

[0046] Specifically, in the personalized speech synthesis of the digital human on the AI ​​platform, a speech synthesis network is used for speech synthesis, and pruning training is performed using phrases in the target user's questions. The loss function uses a cross-entropy loss function to calculate the sensitivity loss of sensitive word combinations. After all sensitive word combinations of the target user are input into the speech synthesis network, the loss function obtains the gradient value of each neuron in the network model. If the gradient value is large, it indicates that it has a large impact on the sensitivity loss. If the gradient value is close to 0, it indicates that it is insignificant to the sensitive information. A preset pruning ratio is used. In this embodiment, the pruning ratio is described as 5%. All neurons are arranged in order of their gradient values ​​from small to large. Neurons with a previous pruning ratio are marked as neurons to be pruned and pruned. After pruning, re-input and pruning training are performed. Pruning training is iteratively performed according to the pruning ratio until the gradient values ​​of the retained neurons all reflect that they will have a large impact on the sensitivity loss. Pruning is stopped to obtain a finally trained speech synthesis network. The specific iterative termination operation is the existing technology of pruning neurons based on gradient values. This embodiment does not specifically limit it and will not be described in detail.

[0047] Furthermore, for the target user's question, the finally trained speech synthesis network generates corresponding phrases and arranges them in order to form the answer to the target user's question. The AI ​​platform digital human plays the answer in voice, thereby realizing the personalized speech synthesis of the AI ​​platform digital human.

[0048] At this point, this embodiment is completed.

[0049] See also Figure 2 , which shows a structural block diagram of a personalized speech synthesis system for a digital human on an AI platform provided by another embodiment of the present invention, the system comprising: Voice collection module 101: acquires voice signals of several users asking questions to the AI ​​platform and obtains spectrograms of each voice signal; Speech processing module 102: Recognizes and segments the speech signal of the user's question to obtain several phrases and word vectors of the user's question. Combined with the spectrogram, it obtains the spectrum matrix of each phrase. It then performs DTW matching on the spectrum matrices of the same phrase of different users based on the energy value at each moment. It analyzes the fundamental frequency difference and energy difference of the two spectrum matrices at the matching moment. Combined with the energy DTW distance of the two spectrum matrices, it obtains personalized difference vectors of the same phrase of different users, and then obtains the personalized vector of each phrase of each user. Based on the similarity between the word vectors of different phrases for the same user, we classify them into several categories. We analyze the differences between the personalized vectors of different phrases within the same category to obtain several sensitive phrases for the user. Based on the distribution of the corresponding parts of speech of the phrases in the user's questions, we obtain several habitual part-of-speech combinations for each user. Combined with the distribution of sensitive phrases and other phrases before and after them in the user's questions, we obtain several sensitive word combinations for the user. Speech synthesis module 103: prunes the speech synthesis network based on several sensitive word combinations in the user's question, and performs personalized speech synthesis of the digital human.

[0050] Another embodiment of the present invention provides an electronic device for personalized speech synthesis of a digital human on an AI platform. The electronic device includes a memory, a processor, and a computer program stored in the memory and running on the processor. When the processor executes the computer program, steps S001 to S004 of the above method are implemented.

[0051] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent replacements, improvements, etc. made within the principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A personalized speech synthesis method for digital humans on an AI platform, characterized in that: The method comprises the following steps: Obtain voice signals of several users asking questions to the AI ​​platform, and obtain the spectrogram of each voice signal; The speech signals of user questions are recognized and segmented to obtain several phrases and word vectors of the user questions. The spectrogram is then combined to obtain the spectrum matrix of each phrase. The spectrum matrices of the same phrases of different users are matched at any given moment based on the energy value. The fundamental frequency and energy differences of the two spectrum matrices at the matching moment are analyzed. The energy DTW distance of the two spectrum matrices is combined to obtain the personalized difference vectors of the same phrases of different users, and thus the personalized vectors of each phrase of each user are obtained. Based on the similarity between the word vectors of different phrases for the same user, we classify them into several categories. We analyze the differences between the personalized vectors of different phrases within the same category to obtain several sensitive phrases for the user. Based on the distribution of the corresponding parts of speech of the phrases in the user's questions, we obtain several habitual part-of-speech combinations for each user. Combined with the distribution of sensitive phrases and other phrases before and after them in the user's questions, we obtain several sensitive word combinations for the user. The speech synthesis network is pruned based on several sensitive word combinations in the user's questions, and personalized speech synthesis of the digital human is performed.

2. The personalized speech synthesis method for digital humans on an AI platform according to claim 1, characterized in that: The specific method of obtaining the multiple phrases and word vectors of the user's question and combining them with the spectrogram to obtain the spectrogram matrix of each phrase is as follows: Take any user as the target user, recognize the voice signal of the target user's question, convert the voice signal into text information, segment the text information into words, obtain several phrases, and convert each phrase into a vector to obtain the word vector of each phrase; For the spectrogram of the speech signal of the target user's question, the horizontal axis in the spectrogram is time, the vertical axis is frequency value, and the pixel value of each data point in the spectrogram represents the energy value; Obtain the time period corresponding to each phrase in the spectrogram, and use a spectrogram of any phrase of the target user in the corresponding time period in the spectrogram as the spectrogram matrix of the phrase of the target user; In the spectrogram matrix, the same row corresponds to the same frequency, the same column corresponds to the same time, and the elements are the energy values ​​represented by the pixel values ​​of the data points in the spectrogram.

3. The personalized speech synthesis method for digital humans on an AI platform according to claim 2, characterized in that: The specific method of performing DTW matching on the spectrum matrices of the same phrase of different users based on energy values ​​at each moment is as follows: For any phrase of the target user, the maximum value of the energy value of each column element in the spectrum matrix is ​​obtained, and a curve is constructed in chronological order as the energy change curve of the phrase; For any other user except the target user, obtain several phrases of the other user with the same word vector as the phrase of the target user as several reference phrases of the target user; obtain the energy change curve of any reference phrase, and perform DTW matching with the energy change curve of the phrase to obtain the DTW distance and several matching time pairs, and use the DTW distance as the energy DTW distance between the phrase of the target user and the reference phrase of the other user; the matching time pairs are the matching time pairs formed by DTW matching at each moment in the spectral matrix of the phrase of the target user and each moment in the spectral matrix of the reference phrase of the other user.

4. The personalized speech synthesis method for digital humans on an AI platform according to claim 3, characterized in that: The specific method of obtaining the personalized difference vectors of the same phrases of different users and then obtaining the personalized vectors of each phrase of each user is as follows: Using the PYIN algorithm, we obtain the fundamental frequency trajectory of the spectral matrix of any phrase of the target user, as well as the fundamental frequency trajectory of the spectral matrix of any reference phrase of the target user's phrase for any user other than the target user. For the moments under the two spectral matrices of the same matching moment, we calculate the fundamental frequency difference value, obtain the fundamental frequency difference value of each matching moment pair, and calculate the mean, variance, maximum and minimum values ​​of all fundamental frequency difference values ​​respectively. Calculate the sum of all energy values ​​corresponding to any column element in the grammar matrix of the target user's phrase as the energy sum at the corresponding moment, and construct a curve in chronological order as the energy curve of the phrase; calculate the energy and difference values ​​for the moments in the two grammar matrices under the same matching moment, and obtain the energy and difference values ​​of each matching moment pair. Calculate the mean, variance, maximum and minimum values ​​of all energy and difference values ​​respectively; The mean, variance, maximum, and minimum of all fundamental frequency difference values ​​between the target user's phrase and the reference phrase of the other user, the mean, variance, maximum, and minimum of all energy and difference values, and the energy DTW distance are used as elements of the personalized difference vector between the target user's phrase and the reference phrase of the other user to obtain the personalized difference vector between the target user's phrase and the reference phrase of the other user; Obtain personalized difference vectors between the target user's phrase and reference phrases of all other users except the target user, obtain an average vector of all personalized difference vectors, and use the average vector as the personalized vector of the target user's phrase.

5. The personalized speech synthesis method for digital humans on an AI platform according to claim 1, characterized in that: The classification results in several types of phrases, including the following specific methods: For all phrases of the target user, density clustering is performed based on the word vectors of each phrase. The distance metric is 1 minus the difference between the cosine similarity between the word vectors of different phrases to obtain several clusters. All phrases in the same cluster are regarded as a type of phrase, and several types of phrases are obtained.

6. The personalized speech synthesis method for digital humans on an AI platform according to claim 2, characterized in that: The specific method of obtaining the user's sensitive phrases includes: For the personalized vector of any phrase of the target user, obtain the cosine similarity between the personalized vectors of the phrase and other similar phrases. Preset a similarity threshold and use the ratio of the number of similar phrases whose cosine similarity with the personalized vector of the phrase is less than the similarity threshold to the total number of similar phrases as the sensitivity of the phrase. A screening ratio is preset, and phrases with the highest screening ratio in any category of phrases are sorted from most sensitive to least sensitive as sensitive phrases of that category, and several sensitive phrases of each category of target users are obtained, and other phrases in each category that are not sensitive phrases are regarded as non-sensitive phrases.

7. The personalized speech synthesis method for digital humans on an AI platform according to claim 6, characterized in that: The specific method for obtaining the habitual part-of-speech combinations of each user is as follows: For all phrases in the target user's question, perform part-of-speech tagging on each phrase using the LLM model to obtain the part-of-speech of each phrase. Arrange the parts-of-speech of all phrases in the target user's question in the order of the phrase distribution to form a part-of-speech sequence for the target user's question. Treat adjacent elements in the part-of-speech sequence as a part-of-speech combination to obtain several part-of-speech combinations. A habit threshold is preset, and the ratio of the number of times any part-of-speech combination appears in all part-of-speech combinations in the part-of-speech sequence to the number of all part-of-speech combinations in the part-of-speech sequence is used as the habit factor of the part-of-speech combination; if the habit factor is greater than the habit threshold, the part-of-speech combination is regarded as a habitual part-of-speech combination, and several habitual part-of-speech combinations of the target user are obtained.

8. The personalized speech synthesis method for digital humans on an AI platform according to claim 7, characterized in that: The specific method of obtaining the user's multiple sensitive word combinations includes: For any sensitive phrase of the target user, obtain the previous and next phrases of the sensitive phrase in the target user's question, and combine the non-sensitive phrase of the two adjacent phrases with the sensitive phrase as an alternative phrase for the sensitive phrase. If it is not a non-sensitive phrase, no alternative phrase combination is constructed; Obtain the part-of-speech combination corresponding to the candidate phrase combination. If it is the target user's usual part-of-speech combination, use the candidate phrase combination as the habitual phrase combination of the sensitive phrase. If it is not the target user's usual part-of-speech combination, use the candidate phrase combination as the associated phrase combination of the sensitive phrase. If the sensitive phrase and its two adjacent phrases do not constitute an candidate phrase combination or a habitual phrase combination, use the sensitive phrase as a sensitive phrase combination of the target user. For any habitual phrase combination of the sensitive phrase, obtain the occurrence frequency of the sensitive phrase in the habitual phrase combination, where the occurrence frequency is the ratio of the number of occurrences of the sensitive phrase in all phrases in the target user's question to the number of all phrases; Obtaining the frequency of occurrence of non-sensitive phrases in the habitual phrase combination and the frequency of occurrence of the habitual phrase combination, and then obtaining the correlation strength between the sensitive phrase and the non-sensitive phrase in the habitual phrase combination through a point mutual information algorithm; A correlation threshold is preset, and if the correlation strength is greater than the correlation threshold, the habitual phrase combination of the sensitive phrase is regarded as a sensitive word combination.

9. A personalized speech synthesis system for digital humans on an AI platform, characterized by: The system includes: The voice acquisition module is used to obtain voice signals of several users asking questions to the AI ​​platform and obtain the spectrogram of each voice signal; The speech processing module is used to recognize and segment the speech signals of user questions, obtain several phrases and word vectors of the user questions, and combine them with the spectrogram to obtain the spectrum matrix of each phrase. The spectrum matrices of the same phrases of different users are matched based on the energy value at the time of DTW. The fundamental frequency difference and energy difference of the two spectrum matrices at the matching time are analyzed. The energy DTW distance of the two spectrum matrices is combined to obtain the personalized difference vectors of the same phrases of different users, and then obtain the personalized vectors of each phrase of each user. Based on the similarity between the word vectors of different phrases for the same user, we classify them into several categories. We analyze the differences between the personalized vectors of different phrases within the same category to obtain several sensitive phrases for the user. Based on the distribution of the corresponding parts of speech of the phrases in the user's questions, we obtain several habitual part-of-speech combinations for each user. Combined with the distribution of sensitive phrases and other phrases before and after them in the user's questions, we obtain several sensitive word combinations for the user. The speech synthesis module is used to prune the speech synthesis network based on several sensitive word combinations in the user's questions and perform personalized speech synthesis of the digital human.

10. An electronic device for personalized speech synthesis of a digital human on an AI platform, the electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that: When the processor executes the computer program, the steps of the personalized speech synthesis method of a digital human on an AI platform as described in any one of claims 1 to 8 are implemented.

Images