Adaptive voice broadcast method and system based on context awareness

Through the adaptive voice broadcasting method based on context perception, using deep learning and co-occurrence matrix to extract features, the problem of insufficient naturalness and adaptability of speech synthesis in the prior art is solved, and a more natural and personalized voice broadcasting effect is achieved.

CN118335056BActive Publication Date: 2024-11-08JIANGSU HUAMING GUOAN TECH CO LTD
View PDF 22 Cites 0 Cited by

Patent Information

Application Number
CN202410593684.4
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-05-14
Publication Date
2024-11-08
Estimated Expiration
2044-05-14

AI Technical Summary

Technical Problem

Existing pronunciation techniques have limitations in nature, adaptability and resource efficiency, and cannot effectively deal with new words or special expressions not included in the pronunciation database, and synthetic pronunciation may lack a sense of nature and sound mechanized.

Method used

Adaptive voice broadcasting method based on context perception is adopted, context features are extracted through effective word conformity set cleaning text, co-occurrence matrix and mutual information, combined with the end-to-end speech synthesis network of deep learning, multi-part loss function is set and gradient descent optimization is used , generate more natural and smooth voice.

Benefits of technology

It improves the naturalness and adaptability of pronunciation, reduces word participle errors and noise interference, significantly improves the personalization and adaptability of broadcasts, and the generated pronunciation is more natural and in line with the actual context.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118335056B_ABST
    Figure CN118335056B_ABST
Patent Text Reader

Abstract

The present invention discloses an adaptive speech broadcast method and system based on context perception, including: S1: using a valid character set to clean the broadcast text, and segmenting the cleaned broadcast text; S2: extracting context features in the segmentation results based on co-occurrence matrix and mutual information; S3: using an end-to-end speech synthesis network based on deep learning to build an adaptive speech broadcast network, setting the input, output and loss function of the adaptive speech broadcast network; S4: using a gradient descent method to update the adaptive speech broadcast network parameters, and obtaining a trained adaptive speech broadcast network; S5: using the trained adaptive speech broadcast network and a vocoder to generate a broadcast voice for an input new broadcast text, and post-processing the generated broadcast voice. The present invention can utilize context feature extraction technology to enable the speech synthesis system to understand and adapt to various text contents and contexts, and generate more natural and fluent speech broadcasts.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the technical field of voice broadcasting, and in particular to a context-aware adaptive voice broadcasting method and system. Background Art

[0002] With the rapid development of information technology, speech synthesis technology has been widely used in various scenarios, such as intelligent assistants, automatic customer service systems, and electronic reading devices. Although current speech synthesis technology has made significant progress, existing methods still have some significant limitations in terms of naturalness, adaptability, and resource efficiency. Splicing-based systems rely on pre-recorded speech databases to generate speech by selecting and splicing stored speech segments. Although this method can produce relatively natural speech output, it has limited adaptability to text and cannot effectively handle new words or special expressions that are not included in the speech database. The splicing points of speech segments may sound incoherent, especially when intonation and emotion conversion are prone to unnatural breaks. Parameter-based systems generate speech through acoustic models, which have better storage efficiency and flexibility. However, the speech they synthesize often lacks naturalness and may sound mechanical, especially when expressing complex voice changes and emotions. In order to achieve natural speech output, this method requires fine-tuning of a large number of parameters. Summary of the invention

[0003] In view of this, the present invention provides a context-aware adaptive speech broadcast method, which aims to utilize context feature extraction technology to enable a speech synthesis system to understand and adapt to various text contents and contexts, and generate more natural and fluent speech broadcasts.

[0004] To achieve the above object, the context-aware adaptive voice broadcast method provided by the present invention comprises the following steps:

[0005] S1: Use the valid character set to clean the broadcast text and perform word segmentation on the cleaned broadcast text;

[0006] S2: Extract context features from word segmentation results based on co-occurrence matrix and mutual information;

[0007] S3: Use an end-to-end speech synthesis network based on deep learning to build an adaptive speech broadcast network, and set the input, output, and loss function of the adaptive speech broadcast network;

[0008] S4: Use the gradient descent method to update the parameters of the adaptive speech broadcast network to obtain a trained adaptive speech broadcast network;

[0009] S5: using the trained adaptive speech broadcast network and the vocoder to generate broadcast speech for the input new broadcast text, and post-processing the generated broadcast speech;

[0010] As a further improvement method of the present invention:

[0011] Optionally, in step S1, the broadcast text is cleaned using a valid character set, and the cleaned broadcast text is segmented, including:

[0012] S11: Cleaning text:

[0013] Define a set C of characters containing valid characters, and for each character t in the broadcast text T i ,like Then remove, specifically:

[0014]

[0015] Wherein, T′ is the cleaned broadcast text; N represents the length of T;

[0016] S12: Participle:

[0017] The bidirectional maximum matching method is based on comparing the cleaned broadcast segmentation book with the dictionary to perform word segmentation, and the bidirectional maximum matching method is divided into forward maximum matching and reverse maximum matching;

[0018] The forward maximum match starts from the beginning of the text and matches the longest word in the dictionary first. If it fails, the length of the match is gradually reduced. Specifically:

[0019]

[0020]

[0021] Among them, W fmm Indicates the forward maximum matching word segmentation result; N′ is the length of the cleaned broadcast text; L is the maximum word length; T′[i′:i′+k] is the characters of the cleaned broadcast text at indexes i′ to i′+k; i′ is the current processed character position; The word that matches starting from index i′;

[0022] The reverse matching starts from the end of the text and matches the longest word in the dictionary, and also decreases the length of the word if it is unsuccessful, specifically:

[0023]

[0024]

[0025] Among them, W bmm Indicates the reverse maximum matching word segmentation result; The word that matches starting from index i′;

[0026] Compare W fmm and W bmm The results with fewer segmentation words are selected to reduce segmentation ambiguity, specifically:

[0027]

[0028] Among them, |W fmm | and |W bmm | are the number of words in the forward maximum match and reverse maximum match word segmentation results respectively; W final Indicates the final word segmentation result;

[0029] Optionally, extracting context features from the word segmentation results based on the co-occurrence matrix and mutual information in step S2 includes:

[0030] S21: Construct a co-occurrence matrix:

[0031] The co-occurrence matrix is ​​used to analyze the relationship between words. The co-occurrence matrix records the number of times each pair of words appears in a window with a window size of τ, specifically:

[0032] [ In the window Appears in]

[0033] in, and W final The words with indexes m and n in ; Words and The number of co-occurrences of |W final | is the number of words in the final word segmentation result; To surround is a window with a center window size of τ;

[0034] Based on the co-occurrence matrix, construct a co-occurrence feature vector for each word:

[0035]

[0036] in, and W final The indexes are 1, 2, and |W final |words; Words The co-occurrence feature vector of contains the co-occurrence frequency of the word with other words in the text;

[0037] S22: Calculate point mutual information:

[0038] The point mutual information is used to measure the frequency of co-occurrence of two words, specifically:

[0039]

[0040] in, Words and words The point mutual information of Words and words The probability of simultaneous occurrence, and Separate words and words The probability of occurrence, and Separate words and words frequency of occurrence;

[0041] Based on the point mutual information, a point mutual information feature vector is constructed for each word:

[0042]

[0043] Optionally, in the step S3, an end-to-end speech synthesis network based on deep learning is used to construct an adaptive speech broadcast network, and the input, output and loss function of the adaptive speech broadcast network are set, including:

[0044] S31: Set input and output:

[0045] The deep learning-based end-to-end speech synthesis network Tacotron2 is used as the framework of the adaptive speech broadcast network. The broadcast text and the context features extracted in step S2 are used as input, and the predicted Mel spectrogram and pause frame probability are output, specifically:

[0046] Y mel , Y stop =Tacotron2(T, V1, V2|θ)

[0047] Wherein: V1 and V2 are the co-occurrence features and point mutual information features of the broadcast text extracted according to steps S1 and S2 respectively; θ is the parameter of the adaptive speech broadcast network; Y mel and Y stop They are the Mel spectrogram and pause frame probability predicted by the adaptive speech broadcast network respectively;

[0048] S32: Setting loss function:

[0049] The loss function of the adaptive speech broadcast network consists of three parts: Mel spectrum loss, pause frame loss and dynamic time warping loss, specifically:

[0050]

[0051] in, and They are Mel spectrum loss, pause frame loss and dynamic time warping loss respectively; and These are real Mel spectrum graphs and pause frames, which are manually extracted from the broadcast text. Mel spectrum loss is represented by mean square error.

[0052] Dynamic time warping loss is achieved by defining a cost matrix, calculating the cumulative cost, finding the optimal path, and calculating the loss. The cost matrix is:

[0053]

[0054] Where x and y are the time frames in the predicted Mel spectrogram and the real Mel spectrogram respectively; Y mel,x and are the values ​​of the predicted Mel-spectrogram and the real Mel-spectrogram at the x-th frame and the y-th frame respectively; ||·|| is the norm;

[0055] Use dynamic programming to calculate the minimum cumulative cost from the starting point to each point in the cost matrix, specifically:

[0056] D(x,y)=cost(x,y)+min(D(x-1,y-1),D(x-1,y),D(x,y-1))

[0057] Where D(x, y) is the minimum cumulative cost to reach point (x, y); D(x-1, y-1) is the minimum cumulative cost to reach point (x-1, y-1); D(x-1, y) is the minimum cumulative cost to reach point (x-1, y); D(x, y-1) is the minimum cumulative cost to reach point (x, y-1);

[0058] Starting from the last point (U, U) of the cost matrix, trace back to the starting point (1, 1) to find the path with the lowest cost. The final dynamic time warping loss is the accumulated cost of the optimal path, specifically:

[0059]

[0060] Among them, path is the optimal path; Z is the total number of points in the optimal path;

[0061] Optionally, in step S4, the adaptive voice broadcast network parameters are updated using a gradient descent method to obtain a trained adaptive voice broadcast network, including:

[0062] S41: Gradient Descent:

[0063] The gradient is calculated based on the loss function value obtained in step S3, specifically:

[0064]

[0065] Among them, g v is the gradient at the vth iteration; θ v The adaptive voice broadcast network parameters at the vth iteration; is the loss function value with respect to θ v The partial derivative of

[0066] Update the adaptive speech broadcast network parameters based on the calculated gradient value, specifically:

[0067]

[0068]

[0069]

[0070] in, and φ v is the intermediate variable at the vth iteration; θ v+1 is the adaptive speech broadcast network parameter at the v+1th iteration; η v is the learning rate at the vth iteration; β1 and β2 are the decay rates;

[0071] S42: Adaptive adjustment of learning rate:

[0072] Use the cyclic learning rate to adjust the learning rate during gradient descent, specifically:

[0073]

[0074] Among them, η max and η min are the maximum and minimum values ​​of the set learning rate respectively; π is the pi;

[0075] Repeating S41 and S42 to obtain a trained adaptive speech broadcast network;

[0076] Optionally, in step S5, the trained adaptive speech broadcast network and vocoder are used to generate a broadcast speech for the input new broadcast text, and the generated broadcast speech is post-processed, including:

[0077] S51: Synthesized Speech:

[0078] Input the text to be broadcast into the trained adaptive speech broadcast network, and output the predicted Mel-spectrogram; input the predicted Mel-spectrogram into the vocoder to obtain the speech O that can be broadcast;

[0079] S52: Post-processing:

[0080] The acquired speech is further post-processed, specifically:

[0081]

[0082] Where R is the length of the filter; b is the speech time frame index; O(vr′) is the value of O at br′; O′ is the post-processed speech; O′(b) is the value of O′ at b′; ω r =Rr;r=0,1,…,R-1;

[0083] The post-processed voice is the final voice used for broadcasting;

[0084] The present invention also discloses an adaptive voice broadcasting system based on context perception, comprising:

[0085] Preprocessing module: Use the valid character set to clean the broadcast text and perform word segmentation on the cleaned broadcast text;

[0086] Feature extraction module: extracts context features from word segmentation results based on co-occurrence matrix and mutual information;

[0087] Network building module: Use end-to-end speech synthesis network based on deep learning to build an adaptive speech broadcast network;

[0088] Network training module: Use gradient descent method to update the parameters of adaptive speech broadcast network;

[0089] Post-processing module: Use the trained adaptive speech broadcast network and vocoder to generate broadcast speech for the input new broadcast text, and post-process the generated broadcast speech;

[0090] Beneficial effects:

[0091] By defining a valid word set and cleaning the broadcast text, the present invention effectively removes irrelevant characters, reduces noise interference, and improves the accuracy and efficiency of subsequent processing. The bidirectional maximum matching method is used to segment the cleaned text, further optimizing the accuracy of segmentation and effectively reducing segmentation errors and ambiguity. The co-occurrence matrix and mutual information are used to conduct an in-depth analysis of the segmentation results, and the present invention can accurately capture the relationship between words and contextual dependencies.

[0092] By using advanced deep learning technology to build an end-to-end speech synthesis network and optimizing it through the gradient descent method, the present invention can automatically learn and simulate human speech patterns. The adaptive speech broadcast network not only improves the naturalness of speech, but also adjusts the broadcast strategy according to different broadcast content and contextual features, which significantly improves the personalization and adaptability of the broadcast. BRIEF DESCRIPTION OF THE DRAWINGS

[0093] Figure 1 A flowchart of a context-aware adaptive voice broadcast method according to an embodiment of the present invention;

[0094] Figure 2 This is a comparison diagram of the Mel-spectrogram according to an embodiment of the present invention. DETAILED DESCRIPTION

[0095] The present invention is further described below in conjunction with the accompanying drawings, but the present invention is not limited in any way. Any changes or substitutions made based on the teachings of the present invention belong to the protection scope of the present invention.

[0096] Embodiment 1: A context-aware adaptive voice broadcast method, such as Figure 1 As shown, the following steps are included:

[0097] S1: Use the valid character set to clean the broadcast text and segment the cleaned broadcast text:

[0098] S11: Cleaning text:

[0099] Define a set C of characters containing valid characters, and for each character t in the broadcast text T i ,like Then remove, specifically:

[0100]

[0101] Wherein, T′ is the cleaned broadcast text; N represents the length of T;

[0102] S12: Participle:

[0103] Based on the bidirectional maximum matching method, the cleaned broadcast segmentation book and the dictionary are compared to perform word segmentation, and the bidirectional maximum matching method is divided into forward maximum matching and reverse maximum matching;

[0104] The forward maximum match starts from the beginning of the text, first matches the longest word in the dictionary, and gradually reduces the length of the match if it fails, specifically:

[0105]

[0106]

[0107] Among them, W fmm Indicates the forward maximum matching word segmentation result; N′ is the length of the cleaned broadcast text; L is the maximum word length; T′[i′:i′+k] is the characters of the cleaned broadcast text at indexes i′ to i′+k; i′ is the current processed character position; The word that matches starting from index i′;

[0108] The reverse matching starts from the end of the text and matches the longest word in the dictionary, and also decreases the length of the word if it is unsuccessful, specifically:

[0109]

[0110]

[0111] Among them, W bmm Indicates the reverse maximum matching word segmentation result; The word that matches starting from index i′;

[0112] Compare W fmm and W bmm The results with fewer segmentation words are selected to reduce segmentation ambiguity, specifically:

[0113]

[0114] Among them, |W fmm | and |W bmm | are the number of words in the forward maximum match and reverse maximum match word segmentation results respectively; W final Indicates the final word segmentation result;

[0115] The quality of text data can be significantly improved by defining a valid character set and removing characters that do not belong to the set. This cleaning process removes noise that may cause analysis errors, such as special symbols, meaningless characters, etc., ensuring that subsequent processing steps are based on more accurate and clean data. The cleaned text is more standardized and consistent, which makes the word segmentation process more efficient and accurate. The bidirectional maximum matching method effectively reduces the word segmentation errors that may be caused by a single direction by segmenting the text from two directions and comparing the results of the two methods to select the best word segmentation.

[0116] S2: Extract context features from word segmentation results based on co-occurrence matrix and mutual information:

[0117] S21: Construct a co-occurrence matrix:

[0118] The co-occurrence matrix is ​​used to analyze the relationship between words. The co-occurrence matrix records the number of times each pair of words appears in a window with a window size of τ, specifically:

[0119] [ In the window Appears in]

[0120] in, and W final The words with indexes m and n in ; Words and The number of co-occurrences of |W final | is the number of words in the final word segmentation result; To surround is a window with a center window size of τ;

[0121] Based on the co-occurrence matrix, construct a co-occurrence feature vector for each word:

[0122]

[0123] in, and W final The indexes are 1, 2, and |W final |words; Words The co-occurrence feature vector of contains the co-occurrence frequency of the word with other words in the text;

[0124] S22: Calculate point mutual information:

[0125] The point mutual information is used to measure the frequency of co-occurrence of two words, specifically:

[0126]

[0127] in, Words and words The point mutual information of Words and words The probability of simultaneous occurrence, and Separate words and words The probability of occurrence, and Separate words and words frequency of occurrence;

[0128] Based on the point mutual information, a point mutual information feature vector is constructed for each word:

[0129]

[0130] The co-occurrence matrix provides a way to quantify the relationship between words. By counting the co-occurrence frequency of words within a statistical window, the correlation between words can be captured. This method can reveal how words are combined in actual language use, thereby helping the speech broadcast system understand the semantics and usage of different word combinations. By calculating the point mutual information, the amount of information about the co-occurrence of two words can be quantified, which not only reflects their co-occurrence frequency, but also takes into account the frequency of their independent occurrence. A high PMI value means that the probability of two words co-occurring in a given text is higher than the probability of their random co-occurrence, which helps to identify semantically closely related or dependent word pairs, thereby improving natural intonation and contextual adaptability in speech synthesis.

[0131] The constructed co-occurrence feature vector and point mutual information feature vector can be directly used as input features of the adaptive speech broadcast network. The addition of these contextual features makes the speech synthesis model not only rely on the speech characteristics of a single word, but also adjust the intonation, speaking speed and emotional expression of the speech according to the relationship between words and the overall text context, making the synthesized speech more natural and in line with the actual context.

[0132] S3: Use an end-to-end speech synthesis network based on deep learning to build an adaptive speech broadcast network, and set the input, output, and loss function of the adaptive speech broadcast network:

[0133] S31: Set input and output:

[0134] The deep learning-based end-to-end speech synthesis network Tacotron2 is used as the framework of the adaptive speech broadcast network. The broadcast text and the context features extracted in step S2 are used as input, and the predicted Mel spectrogram and pause frame probability are output, specifically:

[0135] Y mel , Y stop =Tacotron2(T, V1, V2|θ)

[0136] Where V1 and V2 are the co-occurrence features and point mutual information features of the broadcast text extracted according to steps S1 and S2 respectively; θ is the parameter of the adaptive speech broadcast network; Y mel and Y stop They are the Mel spectrogram and pause frame probability predicted by the adaptive speech broadcast network respectively;

[0137] S32: Setting loss function:

[0138] The loss function of the adaptive speech broadcast network consists of three parts: Mel spectrum loss, pause frame loss and dynamic time warping loss, specifically:

[0139]

[0140] in, and They are Mel spectrum loss, pause frame loss and dynamic time warping loss respectively; and These are real Mel spectrum graphs and pause frames, which are manually extracted from the broadcast text. Mel spectrum loss is expressed using mean square error, specifically:

[0141]

[0142] Where U is the number of time frames; Y mel,u and are the values ​​of the predicted Mel-spectrogram and the real Mel-spectrogram at the uth frame respectively; ||·|| is the norm;

[0143] The pause frame loss is expressed using cross entropy, specifically:

[0144]

[0145] Among them, Y stop,u and are the values ​​of the predicted pause frame and the real pause frame at the uth frame respectively;

[0146] Dynamic time warping loss is achieved by defining a cost matrix, calculating the cumulative cost, finding the optimal path, and calculating the loss. The cost matrix is:

[0147]

[0148] Where x and y are the time frames in the predicted Mel spectrogram and the real Mel spectrogram respectively; Y mel,x and are the values ​​of the predicted Mel spectrogram and the true Mel spectrogram at the xth frame and the yth frame respectively;

[0149] Use dynamic programming to calculate the minimum cumulative cost from the starting point to each point in the cost matrix, specifically:

[0150] D(x,y)=cost(x,y)+min(D(x-1,y-1),D(x-1,y),D(x,y-1))

[0151] Where D(x, y) is the minimum cumulative cost to reach point (x, y); D(x-1, y-1) is the minimum cumulative cost to reach point (x-1, y-1); D(x-1, y) is the minimum cumulative cost to reach point (x-1, y); D(x, y-1) is the minimum cumulative cost to reach point (x, y-1);

[0152] Starting from the last point (U, U) of the cost matrix, trace back to the starting point (1, 1) to find the path with the lowest cost. The final dynamic time warping loss is the accumulated cost of the optimal path, specifically:

[0153]

[0154] Among them, path is the optimal path; Z is the total number of points in the optimal path;

[0155] By adopting an end-to-end speech synthesis approach, the system can automatically generate speech directly from raw text without the need for multi-stage processing with human intervention. This automation greatly simplifies the speech synthesis process and reduces system complexity while ensuring high efficiency and high consistency of output.

[0156] Combining the co-occurrence features and point mutual information features extracted in step S2 as the input of the network enhances the model's ability to process complex texts. These features provide rich semantic and pragmatic information, enabling the adaptive speech broadcast network to more accurately understand the text intent and emotion, and then generate more natural and expressive speech, and generate more accurate mel spectrograms, such as Figure 2 shown.

[0157] By setting a multi-part loss function including Mel spectrum loss, pause frame loss and dynamic time warping loss, the system can optimize the quality of speech synthesis in multiple dimensions. These loss functions ensure the clarity of the synthesized speech, the naturalness of pauses and the overall speech coherence, and each part aims to improve different aspects of the speech synthesis results.

[0158] The dynamic time warping loss is used to further optimize the alignment accuracy of the Mel-spectrogram, so that the model can more accurately learn how to generate time sequences that are highly consistent with real speech during training.

[0159] S4: Use the gradient descent method to update the parameters of the adaptive speech broadcast network to obtain the trained adaptive speech broadcast network:

[0160] S41: Gradient Descent:

[0161] The gradient is calculated based on the loss function value obtained in step S3, specifically:

[0162]

[0163] Among them, g v is the gradient at the vth iteration; θ v The adaptive voice broadcast network parameters at the vth iteration; is the loss function value with respect to θ v The partial derivative of

[0164] Update the adaptive speech broadcast network parameters based on the calculated gradient value, specifically:

[0165]

[0166]

[0167]

[0168] in, and φ v is the intermediate variable at the vth iteration; θ v+1 is the adaptive speech broadcast network parameter at the v+1th iteration; η v is the learning rate at the vth iteration; β1 and β2 are decay rates, which are 0.9 and 0.999 respectively in this embodiment;

[0169] S42: Adaptive adjustment of learning rate:

[0170] Use the cyclic learning rate to adjust the learning rate during gradient descent, specifically:

[0171]

[0172] Among them, η max and η min are the maximum and minimum values ​​of the set learning rate, which are 0.001 and 0.0001 in this embodiment; π is the pi;

[0173] Repeating S41 and S42 to obtain a trained adaptive speech broadcast network;

[0174] By calculating the gradient of the loss function with respect to the network parameters, the gradient descent method can accurately guide the parameter update direction and step size to minimize the loss. This method enables the network to effectively learn and adapt to the training data, thereby improving its performance while maintaining the complexity of the model.

[0175] Using the Adam optimizer, combined with adjustments to the first-order and second-order moment estimates, helps the model converge to the optimal solution faster.

[0176] Through the cyclic learning rate strategy, the system can automatically adjust the learning rate at different stages of training, which helps avoid falling into local minima and accelerates the search for the global optimal solution. The cyclic learning rate is adjusted according to a preset periodic function, allowing the model to make more detailed parameter adjustments when it is close to the optimal solution.

[0177] S5: Use the trained adaptive speech broadcast network and vocoder to generate broadcast speech for the input new broadcast text, and post-process the generated broadcast speech:

[0178] S51: Synthesized Speech:

[0179] Input the text to be broadcast into the trained adaptive speech broadcast network, and output the predicted Mel-spectrogram; input the predicted Mel-spectrogram into the vocoder to obtain the speech O that can be broadcast;

[0180] S52: Post-processing:

[0181] The acquired speech is further post-processed, specifically:

[0182]

[0183] Where R is the length of the filter; b is the speech time frame index; O(br′) is the value of O at br′; O′ is the post-processed speech; O′(b) is the value of O′ at b′; ω r =Rr;r=0,1,…,R-1;

[0184] The post-processed voice is the final voice used for broadcasting;

[0185] The trained adaptive speech broadcasting network can accurately convert text into Mel spectrograms, which are key intermediate representations in speech synthesis. Mel spectrograms retain important features of speech, such as pitch and rhythm, ensuring that the generated speech is both natural and expressive. The vocoder further converts the Mel spectrogram into a clear and smooth audio signal. The quality of the speech is further improved through post-processing steps.

[0186] Embodiment 2: The present invention also discloses a context-aware adaptive voice broadcast system, comprising the following five modules:

[0187] Preprocessing module: Use the valid character set to clean the broadcast text and perform word segmentation on the cleaned broadcast text;

[0188] Feature extraction module: extracts context features from word segmentation results based on co-occurrence matrix and mutual information;

[0189] Network building module: Use end-to-end speech synthesis network based on deep learning to build an adaptive speech broadcast network;

[0190] Network training module: Use gradient descent method to update the parameters of adaptive speech broadcast network;

[0191] Post-processing module: Use the trained adaptive speech broadcast network and vocoder to generate broadcast speech for the input new broadcast text, and post-process the generated broadcast speech;

[0192] It should be noted that the serial numbers of the above embodiments of the present invention are only for description and do not represent the advantages and disadvantages of the embodiments. And the terms "including", "comprising" or any other variants thereof in this article are intended to cover non-exclusive inclusion, so that a process, device, article or method including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, device, article or method. In the absence of further restrictions, an element defined by the sentence "including a ..." does not exclude the presence of other identical elements in the process, device, article or method including the element.

[0193] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) as described above, and includes a number of instructions for a terminal device (which can be a mobile phone, a computer, a server, or a network device, etc.) to execute the methods described in each embodiment of the present invention.

[0194] The above are only preferred embodiments of the present invention, and are not intended to limit the patent scope of the present invention. Any equivalent structure or equivalent process transformation made using the contents of the present invention specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present invention.

Claims

1. The context-aware adaptive speech broadcast method is characterized in that: The following steps are involved: S1: Use the valid character set to clean the broadcast text and perform word segmentation on the cleaned broadcast text; S2: Extract context features from word segmentation results based on co-occurrence matrix and point mutual information; S3: Use an end-to-end speech synthesis network based on deep learning to build an adaptive speech broadcast network, and set the input, output, and loss function of the adaptive speech broadcast network; S4: Use the gradient descent method to update the parameters of the adaptive speech broadcast network to obtain a trained adaptive speech broadcast network; S5: using the trained adaptive speech broadcast network and the vocoder to generate broadcast speech for the input new broadcast text, and post-processing the generated broadcast speech; Wherein, the step S1 includes the following steps: S11: Cleaning text: Define a set C of characters containing valid characters, and for each character t in the broadcast text T i ,like Then remove, specifically: Wherein, T′ is the cleaned broadcast text; N represents the length of T; S12: Participle: Based on the bidirectional maximum matching method, the cleaned broadcast segmentation book and the dictionary are compared to perform word segmentation, and the bidirectional maximum matching method is divided into forward maximum matching and reverse maximum matching; The forward maximum match starts from the beginning of the text, first matches the longest word in the dictionary, and gradually reduces the length of the match if it fails, specifically: Among them, W fmm Indicates the forward maximum matching word segmentation result; N′ is the length of the cleaned broadcast text; L is the maximum word length; T′[i′:i′+k] is the characters of the cleaned broadcast text at indexes i′ to i′+k; i′ is the current processed character position; The word that matches starting from index i′; The reverse matching starts from the end of the text and matches the longest word in the dictionary, and also decreases the length of the word if it is unsuccessful, specifically: Among them, W bmm Indicates the reverse maximum matching word segmentation result; The word that matches starting from index i′; Compare W fmm and W bmm The results with fewer segmentation words are selected to reduce segmentation ambiguity, specifically: Among them, |W fmm | and |W bmm | are the number of words in the forward maximum match and reverse maximum match word segmentation results respectively; W final Indicates the final word segmentation result; Wherein, the step S2 includes the following steps: S21: Construct a co-occurrence matrix: The co-occurrence matrix is ​​used to analyze the relationship between words. The co-occurrence matrix records the number of times each pair of words appears in a window with a window size of τ, specifically: in, and W final The words with indexes m and n in ; Words and The number of co-occurrences of |W final | is the number of words in the final word segmentation result; To surround is a window with a center window size of τ; Based on the co-occurrence matrix, construct a co-occurrence feature vector for each word: in, and W final The indexes are 1, 2, and |W final |words; Words The co-occurrence feature vector of contains the co-occurrence frequency of the word with other words in the text; S22: Calculate point mutual information: The point mutual information is used to measure the frequency of co-occurrence of two words, specifically: in, Words and words The point mutual information of Words and words The probability of simultaneous occurrence, and Separate words and words The probability of occurrence, and Separate words and words frequency of occurrence; Based on the point mutual information, a point mutual information feature vector is constructed for each word: Wherein, the step S3 includes the following steps: S31: Set input and output: The deep learning-based end-to-end speech synthesis network Tacotron2 is used as the framework of the adaptive speech broadcast network. The broadcast text and the context features extracted in step S2 are used as input, and the predicted Mel spectrogram and pause frame probability are output, specifically: AND mel ,AND stop =Tacotron2(T,V1,V2|θ) Wherein, V1 and V2 are the co-occurrence features and point mutual information features of the broadcast text extracted according to steps S1 and S2 respectively; θ is the parameter of the adaptive speech broadcast network; Y mel and Y stop They are the Mel spectrogram and pause frame probability predicted by the adaptive speech broadcast network respectively; S32: Setting loss function: The loss function of the adaptive speech broadcast network consists of three parts: Mel spectrum loss, pause frame loss and dynamic time warping loss, specifically: in, and They are Mel spectrum loss, pause frame loss and dynamic time warping loss respectively; and These are real Mel spectrum graphs and pause frames, which are manually extracted from the broadcast text. Mel spectrum loss is expressed using mean square error, specifically: Where U is the number of time frames; Y mel,u and are the values ​​of the predicted Mel spectrogram and the true Mel spectrogram at the uth frame respectively; The pause frame loss is expressed using cross entropy, specifically: Among them, Y stop,u and are the values ​​of the predicted pause frame and the real pause frame at the uth frame respectively; Dynamic time warping loss is achieved by defining a cost matrix, calculating the cumulative cost, finding the optimal path, and calculating the loss. The cost matrix is: Where x and y are the time frames in the predicted Mel spectrogram and the real Mel spectrogram respectively; Y mel,x and are the values ​​of the predicted Mel-spectrogram and the real Mel-spectrogram at the x-th frame and the y-th frame respectively; ||·|| is the norm; Use dynamic programming to calculate the minimum cumulative cost from the starting point to each point in the cost matrix, specifically: D(x,y)=cost(x,y)+min(D(x-1,y-1),D(x-1,y),D(x,y-1)) Where D(x,y) is the minimum cumulative cost to reach point (x,y); D(x-1,y-1) is the minimum cumulative cost to reach point (x-1,y-1); D(x-1,y) is the minimum cumulative cost to reach point (x-1,y); D(x,y-1) is the minimum cumulative cost to reach point (x,y-1); Starting from the last point (U,U) of the cost matrix, trace back to the starting point (1,1) to find the path with the lowest cost. The final dynamic time warping loss is the accumulated cost of the optimal path, specifically: Among them, path is the optimal path; Z is the total number of points in the optimal path.

2. The context-aware adaptive voice broadcast method according to claim 1, characterized in that: The step S4 includes the following steps: S41: Gradient Descent: The gradient is calculated based on the loss function value obtained in step S3, specifically: Among them, g v is the gradient at the vth iteration; θ v The adaptive voice broadcast network parameters at the vth iteration; is the loss function value with respect to θ v The partial derivative of Update the adaptive speech broadcast network parameters based on the calculated gradient value, specifically: in, and φ v is the intermediate variable at the vth iteration; θ v+1 is the adaptive speech broadcast network parameter at the v+1th iteration; η v is the learning rate at the vth iteration; β1 and β2 are the decay rates; S42: Adaptive adjustment of learning rate: Use the cyclic learning rate to adjust the learning rate during gradient descent, specifically: Among them, η max and η min are the maximum and minimum values ​​of the set learning rate respectively; π is the pi; S41 and S42 are repeatedly executed to obtain a trained adaptive speech broadcast network.

3. The context-aware adaptive voice broadcast method according to claim 2, characterized in that: The step S5 includes the following steps: S51: Synthesized Speech: Input the text to be broadcast into the trained adaptive speech broadcast network, and output the predicted Mel-spectrogram; input the predicted Mel-spectrogram into the vocoder to obtain the speech O that can be broadcast; S52: Post-processing: The acquired speech is further post-processed, specifically: Where R is the length of the filter; b is the speech time frame index; O(br′) is the value of O at br′; O′ is the post-processed speech; O′(b) is the value of O′ at b′; ω r =Rr; r = 0, 1, ..., R-1; The post-processed voice is the final voice used for broadcasting.

4. The context-aware adaptive voice broadcast system is characterized by: include: Preprocessing module: Use the valid character set to clean the broadcast text and perform word segmentation on the cleaned broadcast text; Feature extraction module: extracts context features from word segmentation results based on co-occurrence matrix and point mutual information; Network building module: Use end-to-end speech synthesis network based on deep learning to build an adaptive speech broadcast network; Network training module: Use gradient descent method to update the parameters of adaptive speech broadcast network; Post-processing module: Use the trained adaptive speech broadcast network and vocoder to generate broadcast speech for the input new broadcast text, and post-process the generated broadcast speech; To implement the context-aware adaptive voice broadcast method as described in any one of claims 1-3.

Citation Information

Patent Citations

  • Dictionary-based lucene Chinese word segmentation method

    CN105426539A

  • Voice broadcasting method and system for text

    CN106652995A

  • Speech synthesis method and device, electronic equipment and storage medium

    CN110797002A

  • Emotion analysis method based on emotion dictionary and Transformer

    CN112329449A

  • Speech synthesis method based on deep learning

    CN113112985A