A Chinese speech synthesis method based on Chinese word segmentation optimization, an electronic device, and a storage medium

Abstract

The application discloses a Chinese speech synthesis method based on Chinese word segmentation optimization, which comprises the following steps: firstly, using a Chinese word segmentation model CWS-WSD based on weighted self-distillation to perform word segmentation on reply text generated in the interaction process of a medical service robot, and clearly defining the word boundaries in the text; then, inserting corresponding pause markers according to the word segmentation result; and finally, sending the text after the insertion of the markers into Tacotron2 for speech synthesis. The application adds word boundaries to the text in advance by performing Chinese word segmentation on the Chinese text in the speech synthesis process, and correspondingly adds pauses in the generated speech to alleviate the aforementioned ambiguity and interaction quality problems.

Description

Technical Field

[0001] This invention relates to the field of Chinese speech synthesis technology, specifically to a Chinese speech synthesis method, electronic device, and storage medium based on Chinese word segmentation optimization. Background Technology

[0002] Voice interaction is an important method of human-computer interaction, enabling robots to provide relevant information to users through speech synthesis modules. Taking medical service robots as an example, the interaction process often involves a large number of professional terms without clear lexical boundaries, resulting in synthesized audio without appropriate pauses, which can easily lead to ambiguous medical information. For example, the disease name "methemoglobinemia" contains ambiguities such as "high-speed rail" and "leukemia" (i.e., leukemia), reducing the quality of human-computer interaction.

[0003] Current speech synthesis inputs consist of a string of natural language text output by other modules (such as dialogue systems). In the Chinese domain, to avoid ambiguity or polyphonic characters, the text sequence needs to be segmented into corresponding word sequences—a process known as Chinese word segmentation. As a fundamental task in Chinese natural language processing, the quality of Chinese word segmentation directly impacts the performance of downstream tasks, such as question-answering systems, sentiment recognition, and text classification. For medical texts, Chinese word segmentation breaks down strings containing numerous medical terminologies into word sequences with clear boundary information, reducing ambiguity and providing more accurate medical semantics for downstream tasks. Medical texts often contain a large number of disease names, surgical procedures, drug names, and medical terminology. These specialized terms are mostly out-of-vocabulary (OOV) words compared to general corpora, and the recognition performance of OOV words directly affects the segmentation quality of the model. Wordhood information is crucial for Chinese text modeling. Essentially, it is a text feature that reflects the co-occurrence probability between characters in the text, which can improve the model's ability to handle out-of-vocabulary (OOV) words. However, existing wordhood memory networks that integrate wordhood information do not consider the dependencies between labels during modeling (e.g., label S should be followed by label S or label B, while label B should be followed by label M or label E). This results in a deficiency in the model's word segmentation performance, especially in the recognition of OOV words.

[0004] Research on Chinese word segmentation dates back to the 1980s and has yielded fruitful results in general domains. However, research on domain-specific Chinese word segmentation suffers from significant performance degradation when applying models and tools that perform well in general domains to specialized domains due to the presence of numerous specialized terms. Ding et al. proposed a word segmentation model that utilizes general domain knowledge to handle specialized domains, employing transfer learning to transfer general knowledge from high-resource domains to low-resource domains, thereby improving the model's ability to handle out-of-vocabulary (OV) words and addressing OV-specific word segmentation tasks. Ye et al. proposed a semi-supervised cross-domain Chinese word segmentation model using word embeddings, integrating word embeddings trained from unsupervised specialized domain corpora into the model's training process, and leveraging the domain knowledge within the word embeddings to improve the model's segmentation performance for specialized domain terms. Li Peng et al. proposed a model to address the low quality of OV word recognition in specific domains, by selecting high-confidence n-gram segments from unsupervised corpora and utilizing the prior knowledge within these n-gram segments to assist the model in recognizing OV words, achieving improved segmentation performance in the architecture domain. Tian et al. proposed a word-formation memory network based on key-value networks and attention mechanisms, which utilizes textual word-formation information. They also proposed a Chinese word segmentation model, WMSEG, which utilizes the word-formation memory network, achieving state-of-the-art Chinese word segmentation performance and improving the model's ability to handle OOV words.

[0005] However, these methods have two main shortcomings: (1) Existing models are insufficient in modeling the higher-order semantic relationships between contextual features and semantic features; (2) Current research has failed to link Chinese word segmentation with service robots in the medical field, and cannot further utilize the model. Summary of the Invention

[0006] The purpose of this invention is to address the aforementioned deficiencies in the prior art by providing a Chinese speech synthesis method, electronic device, and storage medium based on Chinese word segmentation optimization.

[0007] To address the shortcomings of the aforementioned technical problems, the present invention adopts the following technical solution: a Chinese speech synthesis method based on Chinese word segmentation optimization: First, the response text generated during the interaction of the medical service robot is segmented using the weighted self-distillation-based Chinese word segmentation model CWS-WSD to clarify the lexical boundaries in the text. Then, corresponding pause markers are inserted according to the word segmentation results. Finally, the text after the markers are inserted is sent to Tacotron2 for speech synthesis.

[0008] As a further optimization of the Chinese speech synthesis method based on Chinese word segmentation optimization of this invention: the Chinese word segmentation specifically includes:

[0009] The input text sequence is semantically encoded and contextually encoded using an encoder to generate a semantic feature vector containing text semantic information and a contextual feature vector containing word formation information.

[0010] The interactive attention mechanism is used to introduce the higher-order semantic relationship between contextual features and semantic features into the model training process, and the weighted self-distillation mechanism is used to constrain the model feature learning and guide the model to learn the contextual features in the text.

[0011] The output features, which contain high-order semantic relationships between contextual and semantic features, are fed into the decoder for word segmentation and annotation.

[0012] As a further optimization of the Chinese speech synthesis method based on Chinese word segmentation optimization of the present invention, the Chinese word segmentation method specifically includes the following steps:

[0013] S1. Construct a dictionary N to filter n-gram segments in the text that contain the corresponding characters;

[0014] S2. Given a text sequence X = [x1, ..., x2], ... i , ..., x X ], where X represents the length of the text sequence, and for each character x in the text sequence i The dictionary N is used to traverse the text sequence to find the character x. i All n-gram fragments, thus obtaining the key list K. i =[k i,1 , ..., k i,j , ..., k i,m Sum of values ​​list V i =[v i,1 , ..., v i,j , ..., v i,m ], m indicates that it contains the character x i The number of n-gram segments, using key embeddings Indicates k i,j and value embedding Indicates v i,j Where i represents the position of the character in the text sequence X, and j represents the n-gram segment k. i,j Its position in the key list;

[0015] S3. Input the text sequence X into the CWS-WSD model, and use the encoder to process each character x in the sequence. i Encode a feature vector h containing textual semantic information i The formula is as follows:

[0016] Encoder(X) = [h1,L,h i ,L,h |X|], where Encoder represents the encoder, specifically the Bert encoder;

[0017] S4. Transfer the semantic feature vector h i Key embedding Value embedding The contextual feature vector o containing word formation information is calculated using the following formula:

[0018]

[0019]

[0020] Where: h i p represents the feature vector containing textual semantic information generated by the encoder. i,j This indicates that the character x is included. i n-gram fragment k i,j The probability distribution; o i This represents the contextual feature vector containing word formation information generated by the word formation memory network;

[0021] S5. Utilize the contextual attention mechanism to model two types of feature-weighted embedding vectors containing contextual features, and utilize the semantic attention mechanism to model two types of feature-weighted embedding vectors containing semantic features. Then, concatenate and fuse the two weighted embedding vectors, and then reduce the dimensionality of the fused feature vector. This integrates the high-order semantic relationship between contextual features and semantic features into the model training process, guides the model to learn features, and finally obtains the output features of the interactive attention mechanism.

[0022] S6. The decoder predicts the text label sequence Y = y1L y based on the output features of the interactive attention mechanism. i L y |X| The formula is as follows:

[0023] Decoder(a′) = Y

[0024] Where: Decoder represents the decoder, a′=a1L a i L a |X| Using CRF as the decoder, the algorithm is defined by the following formula:

[0025]

[0026] Among them: W c ∈R |B|×|B| b c ∈R |B| Both are trainable parameters;

[0027] The S7 and CWS-WSD models first use an encoder to semantically encode the input text sequence, generating feature vectors containing textual semantic information. Then, they use an attention mechanism and a key-value memory network to encode feature vectors containing textual contextual information. Next, they use an interactive attention mechanism to introduce the higher-order semantic relationships between contextual and semantic features into the model's training process. Furthermore, they use a weighted self-distillation mechanism to constrain the model's feature learning and guide the model to learn contextual features in the text. Finally, the output features containing the higher-order semantic relationships between contextual and semantic features are fed into the decoder for word segmentation and annotation.

[0028] As a further optimization of the Chinese speech synthesis method based on Chinese word segmentation optimization of the present invention: the dictionary N is constructed using an unsupervised word formation method, and the dictionary N is constructed based on the original corpus. The algorithm is defined by the following formula:

[0029] AV(n) = min(L(n), R(n))

[0030] AV represents adjacency diversity, L(n) represents the number of different characters to the left of n-gram fragment n, and R(n) represents the number of different characters to the right of n-gram fragment n.

[0031] As a further optimization of the Chinese speech synthesis method based on Chinese word segmentation optimization of the present invention: In step S5, the Q matrix of the contextual feature attention mechanism is an embedding vector containing semantic features, and K and V are embedding vectors containing contextual features; while the Q matrix of the semantic feature attention mechanism is an embedding vector containing contextual features, and K and V are embedding vectors containing semantic features. This achieves the modeling of high-order semantic associations between contextual features and semantic features, and the information flow interaction between the two attention mechanism modules. Assuming that vector H is an embedding feature containing semantic features, and vector O is an embedding feature containing contextual features, the calculation process of the contextual attention mechanism and the semantic attention mechanism is as follows:

[0032] O′=ContextAttention(H,O,O)

[0033] H′=SemanticsAttention(O,H,H)

[0034] ContextAttention(·) is the context attention mechanism, and SemanticsAttention(·) is the semantic attention mechanism. O′ is the attention-weighted embedding vector containing context features, and H′ is the attention-weighted embedding vector containing semantic features. After obtaining the two weighted embedding vectors, the features are concatenated, and then the concatenated feature vector is fed into a fully connected layer for dimensionality reduction. Finally, the output features of the interactive attention mechanism are obtained. The specific calculation is as follows:

[0035] a′=W a′ ·([H′;O′])+b a′

[0036] in, These are the learnable parameters of the model. For the model's bias parameters, [H′; O′] represents the concatenation of embedding vectors containing contextual features and embedding vectors containing semantic features along a certain dimension. A The dimension of the embedding vector, which contains contextual features and higher-order semantic associations of semantic features.

[0037] As a further optimization of the Chinese speech synthesis method based on Chinese word segmentation optimization of the present invention: the weighted self-distillation mechanism in step S7 is specifically as follows:

[0038] The predicted outputs of the teacher model and the student model are scored, and the scores are used as weight coefficients to weight the information flow of the two models, guiding the model to learn the contextual features in the text. The specific scoring calculation process is shown below.

[0039]

[0040]

[0041] Where, num true This indicates the number of tags in the tag sequence corresponding to a given text sequence. This represents the number of correctly labeled labels in the label sequence predicted by the teacher model. w represents the number of correctly labeled labels in the student model's predicted label sequence. t The teacher model score represents the scoring mechanism, w s The scoring mechanism is represented by the student model's score. After obtaining two weight coefficients, the output information streams of the teacher model and the student model are weighted to achieve weighted self-distillation.

[0042] As a further optimization of the Chinese speech synthesis method based on Chinese word segmentation optimization of this invention, the loss of the CWS-WSD model is divided into two parts: the first part is the model loss during the self-training process, using the negative log-likelihood function as the loss function for this part. The second part is the model loss during the self-distillation process. The first part loss, the second part loss, and the total loss function of the model are shown below:

[0043]

[0044]

[0045] L WSD (θ WSD)=λL1+(1-λ)L2

[0046] Where, θ WSD L1(·) represents the model parameters of CWS-WSD, which are continuously optimized during model training; L2(·) represents the model's self-training loss; L2(·) represents the weighted self-distillation loss; L WSD (·) represents the total loss of CWS-WSD; logits t The information flow, logits, is input to the decoder for the teacher model. s The information flow input to the decoder is provided to the student model. λ is a hyperparameter of CWS-WSD, set to balance the two types of loss.

[0047] An electronic device, comprising:

[0048] Memory and at least one computer program located on the memory;

[0049] It also includes at least one processor for processing at least one computer program in the memory, wherein when the processor executes the computer program, it implements the steps of the Chinese speech synthesis method described above.

[0050] A computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of the above-described Chinese speech synthesis method.

[0051] This invention offers the following advantages: By pre-segmenting the Chinese text into words during speech synthesis, it adds lexical boundaries to the text, thereby incorporating pauses into the generated speech to alleviate the aforementioned ambiguity and interaction quality issues. Furthermore, during Chinese word segmentation, this invention proposes a weighted self-distillation-based Chinese word segmentation model, LWSEG, to segment the text to be synthesized by the robot, clearly defining lexical boundaries. Then, based on this boundary information, corresponding pause markers are inserted. Speech synthesis is then performed using the text-to-speech model, Tacotron2, inserting pauses into the synthesized audio. This mitigates the impact of ambiguity in voice interaction, improves the quality of robot voice interaction, provides patients with more accurate medical information, and enhances the quality of robot services. Attached Figure Description

[0052] Figure 1 This is a schematic diagram of the system when the speech synthesis of this invention is applied to a medical robot;

[0053] Figure 2 This is a flowchart illustrating the speech synthesis method of the present invention;

[0054] Figure 3 This is a schematic diagram of the CWS-WSD model used in the speech synthesis method of this invention;

[0055] Figure 4 This is a schematic diagram illustrating the principle of the interactive attention mechanism in the speech synthesis method of the present invention;

[0056] Figure 5 Visualizing cross-domain examples Figure I ;

[0057] Figure 6 Visualize examples within the domain Figure I ;

[0058] Figure 7 Visualizing cross-domain examples Figure II ;

[0059] Figure 8 Visualize examples within the domain Figure II . Detailed Implementation

[0060] To better understand the present invention, the following embodiments further illustrate the content of the present invention, but the content of the present invention is not limited to the following embodiments.

[0061] This invention addresses the speech synthesis component of medical service robots, proposing an improved speech synthesis module by adding pauses during the synthesis process to alleviate ambiguity. The module comprises two parts: a Chinese word segmentation module and a text-to-speech module. The specific solution is as follows:

[0062] First, the weighted self-distillation-based Chinese word segmentation model CWS-WSD is used to segment the response text generated during the interaction with the medical service robot, clarifying the lexical boundaries in the text. Then, appropriate pause markers are inserted based on the segmentation results. The text after the marker insertion is then fed into Tacotron2 for speech synthesis, inserting pauses into the synthesized audio to mitigate the impact of ambiguous information on the interaction quality, ensuring that the user receives the appropriate feedback and realizing the information transmission between the medical service robot and the user. A schematic diagram of the system is shown below. Figure 1 As shown in the diagram. The structure of the improved speech synthesis method for medical robots is shown in the figure below. Figure 2 As shown.

[0063] For the Chinese word segmentation module, taking several text sequences as examples, the processed text sequences are shown in Table 1. Table 1: Sample Word Segmentation Results

[0064]

[0065] For a sequence of Chinese medical text, the word segmentation process includes: using an encoder to perform semantic encoding and contextual encoding on the input text sequence, generating a semantic feature vector containing textual semantic information and a contextual feature vector containing word formation information. Then, an interactive attention mechanism is used to introduce the higher-order semantic relationships between contextual and semantic features into the model's training process. A weighted self-distillation mechanism is used to constrain the model's feature learning, guiding the model to learn contextual features in the text. Finally, the output features containing the higher-order semantic relationships between contextual and semantic features are fed into the decoder for word segmentation and annotation. During model training, the model's loss consists of two parts: the first part is the negative log-likelihood loss during model training, and the second part is the self-distillation loss.

[0066] Reference Figure 3 , Figure 3 This is a schematic diagram of a Chinese word segmentation model based on weighted self-distillation according to the present invention. In this embodiment, the specific process includes:

[0067] The first step is to construct a dictionary N to filter n-gram segments in the text that contain the corresponding characters. The dictionary N is constructed using an unsupervised word formation method: Accessor Variety (AV). The algorithm for constructing dictionary N based on the original corpus is defined as follows:

[0068] AV(n) = min(L(n), R(n))

[0069] Where L(n) represents the number of distinct characters to the left of n-gram segment n, and R(n) represents the number of distinct characters to the right of n-gram segment n. A higher AV score indicates that n-gram segment n is more likely to be a word.

[0070] The second step is to give a text sequence X = [x1, ..., x...] i , ..., x X ], where X represents the length of the text sequence, and for each character x in the text sequence i The dictionary N is used to traverse the text sequence to find the character x. i All n-gram fragments, thus obtaining the key list K. i =[k i,1 , ..., k i,j , ..., k i,m Sum of values ​​list V i =[v i,1 , ..., v i,j , ..., v i,m ], m indicates that it contains the character x iThe number of n-gram segments. For example, in the text sequence "professional ethics construction", x2 = "career", then we get K2 = ["career", "profession", "professional ethics"] and V2 = [V S V E V M ], V S This represents the numerical value after the label is mapped to a number. Key-based embedding is used. Indicates k i,j and value embedding Indicates v i,j Where i represents the position of the character in the text sequence X, and j represents the n-gram segment k. i,j Its position in the list of keys.

[0071] The third step involves inputting the text sequence X into CWS-WSD, first using an encoder to process each character x in the sequence. i Encode a feature vector h containing textual semantic information i The formula is as follows:

[0072] Encoder(X) = [h1,L,h i ,L,h |X| ]

[0073] Here, Encoder represents the encoder, and this paper chooses Bert as the encoder.

[0074] The fourth step is to convert the semantic feature vector h i Key embedding Value embedding The contextual feature vector o containing word formation information is calculated using the following formula:

[0075]

[0076]

[0077] Where: h i p represents the feature vector containing textual semantic information generated by the encoder. i,j This indicates that the character x is included. i n-gram fragment k i,j The probability distribution; o i This represents the contextual feature vector containing word information generated by the word formation memory network.

[0078] The fifth step involves extracting high-order semantic association features between contextual features containing word formation information and semantic features obtained from the encoder through an interactive attention mechanism. This assists the model in feature learning and is essentially a combination of two attention modules: a semantic feature attention mechanism and a contextual feature attention mechanism.

[0079] We use a contextual attention mechanism to model two types of feature-weighted embedding vectors containing contextual features, and a semantic attention mechanism to model two types of feature-weighted embedding vectors containing semantic features. Then, we concatenate and fuse the two weighted embedding vectors, and then reduce the dimensionality of the fused feature vector. In this way, we integrate the high-order semantic relationship between contextual features and semantic features into the model training process and guide the model to learn features.

[0080] The interactive attention mechanism differs from general attention mechanisms used in semantic analysis and the previously proposed label attention mechanism in that the former is a combination of two levels of attention mechanisms and can achieve information interaction between two modules, while the latter two are single-level attention mechanisms. The specific structure diagram is shown below. Figure 4 As shown in the figure, the Q matrix of the contextual feature attention mechanism contains embedding vectors of semantic features, while K and V contain embedding vectors of contextual features. Conversely, the Q matrix of the semantic feature attention mechanism contains embedding vectors of contextual features, while K and V contain embedding vectors of semantic features. This enables the modeling of higher-order semantic relationships between contextual and semantic features and the information flow interaction between the two attention mechanism modules. Assuming vector H is an embedding feature containing semantic features and vector O is an embedding feature containing contextual features, the calculation processes of the contextual and semantic attention mechanisms are as follows.

[0081] O′=ContextAttention(H,O,O)

[0082] H′=SemanticsAttention(O,H,H)

[0083] ContextAttention(·) is the context attention mechanism, and SemanticsAttention(·) is the semantic attention mechanism. O′ is the attention-weighted embedding vector containing context features, and H′ is the attention-weighted embedding vector containing semantic features. After obtaining the two weighted embedding vectors, the two are concatenated, and then the concatenated feature vector is fed into a fully connected layer for dimensionality reduction. Finally, the output features of the interactive attention mechanism are obtained. The specific calculation is shown below.

[0084] a′=W a′ ·([H′;O′])+b a′

[0085] in, These are the learnable parameters of the model. For the model's bias parameters, [H′; O′] represents the concatenation of embedding vectors containing contextual features and embedding vectors containing semantic features along a certain dimension. AThe dimension of the embedding vector, which contains contextual features and higher-order semantic associations of semantic features.

[0086] In the sixth step, the decoder predicts the text label sequence Y = y1Ly based on the output features of the interactive attention mechanism. i Ly X The formula is as follows:

[0087] Decoder(a′) = Y

[0088] Where: Decoder represents the decoder, a′=a1L a i L a X This paper chooses CRF as the decoder, and the algorithm is defined as follows:

[0089]

[0090] Among them: W c ∈R |B|×|B| b c ∈R |B| Both are trainable parameters.

[0091] Step 7: Weighted Self-Distillation Mechanism. To constrain the model's learning of textual features, a weighted self-distillation mechanism is proposed. The model training process using this mechanism differs from traditional self-training. During training, the best-performing historical model is used as the teacher model, and the currently trained model is used as the student model. The teacher model is then used to perform knowledge distillation on the student model, improving model performance. Unlike traditional knowledge distillation methods, this mechanism does not rely on other networks or data, only on the network itself. The teacher model essentially trains the best-performing student model historically, thus avoiding dependence on a powerful teacher model. Furthermore, the knowledge distillation process involves weighting the output information flows of the teacher and student models. Specifically, the predicted outputs of the teacher and student models are scored, and these scores are used as weight coefficients to weight their information flows, guiding the model to learn contextual features in the text. The specific scoring calculation process is shown below.

[0092]

[0093]

[0094] Where, num true This indicates the number of tags in the tag sequence corresponding to a given text sequence. This represents the number of correctly labeled labels in the label sequence predicted by the teacher model. w represents the number of correctly labeled labels in the student model's predicted label sequence. t The teacher model score represents the scoring mechanism, ws The scoring mechanism is represented by the student model's score. After obtaining two weight coefficients, the output information streams of the teacher model and the student model are weighted to achieve weighted self-distillation.

[0095] The model loss of CWS-WSD is divided into two parts: the first part is the model loss during the self-training process, which uses the negative log-likelihood function as the loss function. The second part is the model loss during the self-distillation process. This chapter aligns the distributions of the teacher model output information flow and the student model output information flow by minimizing the mean squared error between the weighted information flows of the teacher model and the student model. The model is trained using the standard backpropagation algorithm. The first part of the loss, the second part of the loss, and the total loss function are shown below.

[0096]

[0097]

[0098] L WSD (θ WSD )=λL1+(1-λ)L2

[0099] Where, θ WSD L1(·) represents the model parameters of CWS-WSD, which are continuously optimized during model training; L2(·) represents the model's self-training loss; L2(·) represents the weighted self-distillation loss; L WSD (·) represents the total loss of CWS-WSD; logits t The information flow, logits, is input to the decoder for the teacher model. s The information flow input to the decoder is provided to the student model. λ is a hyperparameter of CWS-WSD, set to balance the two types of loss.

[0100] To better illustrate the implementation details of CWS-WSD, the model pseudocode is presented as follows:

[0101]

[0102] <Model Comparison>

[0103] The hyperparameter settings for CWS-WSD are shown in the table below:

[0104]

[0105] In addition to the parameters mentioned in the table above, when constructing the dictionary N, it is necessary to set the threshold of the AV score according to the corresponding dataset. This threshold is used to filter n-gram segments in the corpus. The threshold setting is different for different training corpora. When processing the datasets PKU and CITYU, the AV score threshold is set to 2. When processing the datasets AS and MSR, the AV score threshold is set to 5. When processing the four domain datasets, the AV score threshold is set to 2. In addition, the hyperparameter λ in the model's loss function is set to 0.7.

[0106] <Cross-domain comparison and ablation experiments>

[0107] To verify the model's word segmentation performance on medical text data and its cross-domain Chinese word segmentation performance, comparative experiments were conducted with CWS-WSD and several other cross-domain Chinese word segmentation models from recent years on the selected cross-domain Chinese word segmentation datasets. The F1 scores of the word segmentation models were listed for performance comparison. The comparative experimental results of CWS-WSD on the cross-domain benchmark dataset are as follows:

[0108]

[0109] The data in the table above shows that, compared with the model proposed by Degen et al., CWS-WSD improves the F1 score by an average of 0.59% on the four selected cross-domain Chinese word segmentation datasets;

[0110] Compared with the model proposed by Liu et al., CWS-WSD improves the F1 score by an average of 2.15% on four selected cross-domain Chinese word segmentation datasets;

[0111] Compared with the model proposed by Chen et al., CWS-WSD improves the F1 score by an average of 2.34% on the four selected cross-domain Chinese word segmentation datasets;

[0112] Compared with the model proposed by Cai et al., the CWS-WSD F1 score was improved by an average of 2.22% on four selected cross-domain Chinese word segmentation datasets;

[0113] Compared with the model proposed by Huang et al., CWS-WSD improves the F1 score by an average of 1.73% on the four selected cross-domain word segmentation datasets;

[0114] Although CWS-WSD did not outperform the model proposed by Zhao et al. on the Com dataset, it improved the F1 score by an average of 1.30% on the four selected cross-domain Chinese word segmentation datasets.

[0115] Compared with the model proposed by Zhang et al., CWS-WSD improves the F1 score by an average of 0.90% on four selected cross-domain Chinese word segmentation datasets;

[0116] Compared with the model proposed by Tian et al., CWS-WSD improves the F1 score by an average of 0.80% on the four selected datasets.

[0117] The results of comparative experiments on cross-domain benchmark datasets show that CWS-WSD outperforms other mainstream cross-domain word segmentation models on medical datasets, and also outperforms other mainstream cross-domain word segmentation models on literature and finance datasets. Although it does not outperform other models on computer science datasets, its overall performance on the four selected cross-domain Chinese word segmentation datasets is still better than the aforementioned models.

[0118] To further verify the effectiveness of CWS-WSD in cross-domain Chinese word segmentation tasks, ablation experiments were conducted on the model. The improvements to the proposed Chinese word segmentation model are as follows: 1) An interactive attention mechanism is used to introduce high-order semantic association features between textual context features and semantic features into the model, guiding the model's feature learning. 2) A weighted self-distillation mechanism is proposed, using the historical best model as the teacher model for knowledge distillation, and a scoring mechanism is used to score the predictions of the two models. Then, the output information streams of the two models are weighted using these scores to achieve weighted self-distillation and improve model performance.

[0119] To verify the effectiveness of the first improvement, the performance of the CWS-WSDa model, which only introduces the interactive attention mechanism but does not use the weighted self-distillation mechanism, is compared with that of the WMSEG word segmentation model, which does not introduce either the interactive attention mechanism or the weighted self-distillation mechanism. To verify the effectiveness of the second improvement, the performance of CWS-WSD and CWS-WSDa is compared. The ablation experiment results are shown in the table below.

[0120]

[0121] The performance comparison results of CWS-WSDa and WMSEG show that introducing higher-order semantic correlation features between textual context features and semantic features into the model through an interactive attention mechanism improves the model's F1 score by 0.13%, 0.57%, 0.57%, and 0.81% on Med., Lit., Fin., and Com., respectively, and improves the model's Roov by 2.16%, 2.78%, 1.83%, and 1.50%, respectively. In conclusion, introducing higher-order semantic correlation features between textual context features and semantic features can help improve model performance, thus verifying the effectiveness of the first improvement in cross-domain Chinese word segmentation tasks.

[0122] The performance comparison results of CWS-WSD and CWS-WSDa show that by using the weighted self-distillation mechanism and employing the historical best model as the teacher model for weighted self-distillation, the F1 score of the model improved by 0.04% and 0.34% on Med. and Lit., respectively, and the Roov score improved by 0.19% and 1.12%, respectively. On Fin. and Com., although no improvement in Roov was achieved, the F1 score of the model improved by 0.41% and 0.34%, respectively. In conclusion, the weighted self-distillation mechanism can improve the word segmentation performance of the model, thus verifying the effectiveness of the second improvement in cross-domain Chinese word segmentation.

[0123] <Intra-domain comparison and ablation experiments>

[0124] To verify the domain-specific Chinese word segmentation performance of the model proposed in this chapter, this chapter lists CWS-WSD and several other Chinese word segmentation models from recent years, and conducts comparative experiments on domain-specific datasets. The F1 score and Roov scores of the models are listed for performance comparison, and the specific performance is shown in the table below:

[0125]

[0126] As shown in the table above, compared with the model proposed by Duan et al., CWS-WSD improved the F1 score by an average of 1.26% on the four selected datasets.

[0127] Compared with the model proposed by Nguyen et al., CWS-WSD improved the F1 score and Roov score by an average of 0.27% and 1.31% on the four selected datasets, respectively.

[0128] Compared with the model proposed by Qiu et al., CWS-WSD improved the F1 score and Roov score by an average of 1.13% and 5.44% on the four selected datasets, respectively.

[0129] Compared with the model proposed by Chen et al., CWS-WSD improved the F1 score and Roov score by an average of 2.42% and 10.46% on the four selected datasets, respectively.

[0130] Compared with the word segmentation model proposed by Gong et al., CWS-WSD improved the F1 score and Roov score by an average of 1.24% and 14.47% on the four selected datasets, respectively.

[0131] Compared with the model proposed by Wang et al., CWS-WSD improved the F1 score by an average of 1.43% on the four selected datasets;

[0132] Compared with the word segmentation model proposed by Ma et al., CWS-WSD improved the F1 score by an average of 0.68% on the four selected datasets;

[0133] Compared with the word segmentation method proposed by Gan et al., CWS-WSD improved the F1 score on MSR and PKU by an average of 0.85%;

[0134] Compared with the word segmentation method proposed by Yang et al., CWS-WSD improved the F1 score on MSR and PKU by an average of 0.90%;

[0135] Compared with the word segmentation model proposed by Zhang Zhonglin et al., CWS-WSD improves the F1 score by an average of 1.46% on the four selected datasets;

[0136] Compared with the word segmentation model proposed by Zhou Yulin et al., CWS-WSD improved the F1 score by an average of 0.71% on the four selected datasets.

[0137] The results of comparative experiments on benchmark datasets in the field show that CWS-WSD achieves word segmentation performance that is no worse than other mainstream models on four commonly used datasets, demonstrating the performance advantage of this model in Chinese word segmentation tasks in the field.

[0138] To further verify the effectiveness of CWS-WSD in the domain-specific Chinese word segmentation task, this invention conducts ablation experiments on the two improvements to the model, and the results are shown in the table below:

[0139]

[0140] The performance comparison results of CWS-WSDa and WMSEG show that the F1 score of the proposed model is improved by 0.14%, 0.08%, 0.02%, and 0.09% on AS, CITYU, MSR, and PKU, respectively, and the Roov score is improved by 2.09%, 0.79%, 0.84%, and 0.36%, respectively. In summary, introducing higher-order semantic association features between textual context features and semantic features can help the model improve the performance of domain-specific Chinese word segmentation, thus verifying the effectiveness of the first improvement in domain-specific Chinese word segmentation tasks.

[0141] From the comparison results of CWS-WSD and CWS-WSDa, it can be seen that on CITYU, MSR, and PKU, the F1 scores of the model increased by 0.11%, 0.04%, and 0.55% respectively, and the Roov of the model increased by 0.27%, 0.61%, and 0.96% respectively. On AS, although there was no improvement in Roov, the F1 score of the model increased by 0.22%. In summary, the use of the weighted self-distillation mechanism can improve the word segmentation performance of the model, thus verifying the effectiveness of the second improvement of the model in Chinese word segmentation within the domain.

[0142] To better illustrate that CWS-WSD introduces high-order semantic associations between context features and semantic features through the interactive attention mechanism, and uses the weighted self-distillation mechanism to constrain the model for feature learning, which can guide the model to learn context features in the text. In this subsection, cross-domain and intra-domain text sequences where CWS-WSD has correct word segmentation and WMSEG, a word segmentation model that integrates word formation information using a word formation memory network, has incorrect word segmentation are selected, and the weights of candidate n-gram segments in them are visualized. Taking the cross-domain text sequence "4 interstitial fluid secretion" and the intra-domain text sequence "4 strive unceasingly" as examples for case studies, the visualization results are as Figure 5 and Figure 6 shown.

[0143] The horizontal axis of the heat map represents the candidate n-gram segments, and the vertical axis is the text sequence to be segmented. The darker the color of the weight grid, the greater the weight, and vice versa. From Figure 5 Figure (a) in it, for the cross-domain text "4 interstitial fluid secretion", CWS-WSD selects the n-gram segments "4 interstitial fluid" and "4 secretion". From Figure 5 Figure (b) in it, for the cross-domain text "4 interstitial fluid secretion", WMSEG selects the n-gram segments "4 cells", "4 interstitial", and "4 secretion". And the correct word segmentation sequence of the text "4 interstitial fluid secretion" is "4 interstitial fluid / secretion", so it can be obtained that CWS-WSD can learn more context features that conform to the context in cross-domain data. From Figure 6 Figure (a) in it, it can be known that for the intra-domain text "4 strive unceasingly", CWS-WSD selects the n-gram segment "4 strive unceasingly". From Figure 6 Figure (b) in it, for the intra-domain text "4 strive unceasingly", WMSEG selects the n-gram segments "4 strive" and "4 unceasing". And the correct word segmentation sequence of the text "4 strive unceasingly" is "4 strive unceasingly", so it can be obtained that CWS-WSD can learn more context features that conform to the context in intra-domain data.

[0144] To further illustrate that the CWS-WSD uses the interactive attention mechanism to introduce the high-order semantic association between context features and semantic features, and uses the weighted self-distillation mechanism to constrain the model for feature learning, which can guide the model to learn the context features in the text. In this chapter, the weights of candidate n-gram segments in cross-domain and intra-domain text sequences where both CWS-WSD and WMSEG are correctly segmented are visualized. Taking the cross-domain text "urine routine examination" and the intra-domain text "bless the people" as examples for case studies, the visualization results are as Figure 7 and Figure 8 shown.

[0145] From Figure 7 Figure (a) in it, for the cross-domain text "urine routine examination", the n-gram segments taken by CWS-WSD are "urine routine" and "examination". From Figure 7 Figure (b) in it, for the cross-domain text "urine routine examination", the n-gram segments taken by WMSEG are "urine", "routine inspection", and "examination". And the correct segmentation sequence of the text "urine routine examination" is "urine routine / examination", so it can be obtained that CWS-WSD can learn more context features that conform to the context in cross-domain data. From Figure 8 Figure (a) in it, for the intra-domain text "bless the people", the n-gram segments taken by CWS-WSD are "for", "the people", and "bless". From Figure 8 Figure (b) in it, for the intra-domain text "bless the people", the n-gram segments taken by WMSEG are "for people", "people", and "bless". And the correct segmentation sequence of "bless the people" is "for / the people / bless", so it can be obtained that CWS-WSD can learn more context features that conform to the context in intra-domain data.

[0146] In summary, by visualizing the weights of candidate n-gram segments in cross-domain and intra-domain text sequences where CWS-WSD is correctly segmented while WMSEG is wrongly segmented, and in cross-domain and intra-domain text sequences where both are correctly segmented, it can be seen that introducing the high-order semantic association between context features and semantic features through the interactive attention mechanism, and using the weighted self-distillation mechanism to constrain the model for feature learning can guide the model to learn the context features in the text.

[0147] This invention is a Chinese speech synthesis method based on optimized Chinese word segmentation. It segments the text using Chinese word segmentation and inserts appropriate pause markers based on the segmentation results. The text with the inserted markers is then used for speech synthesis, resulting in pauses in the synthesized audio and mitigating the impact of ambiguous information on interaction quality. Besides using Chinese word segmentation, existing methods involve manual annotation to simulate human speech pauses and text semantics, inserting pause markers before speech synthesis. However, manual annotation incurs significant costs. Our method addresses this issue by using a Chinese word segmentation model.

[0148] This invention addresses the challenges of out-of-vocabulary word identification in cross-domain Chinese word segmentation scenarios and high-order semantic association modeling between contextual and semantic features in Chinese word segmentation tasks. It proposes a weighted self-distillation-based Chinese word segmentation model. Furthermore, existing Chinese word segmentation research has not been applied to speech synthesis. Therefore, this invention combines Chinese word segmentation with speech synthesis, using the segmented Chinese text as pause marks for speech synthesis, thus solving the cost problem of manual annotation.

[0149] The specific embodiments of the present invention have been described above. It should be understood that the present invention is not limited to the specific embodiments described above, and those skilled in the art can make various modifications or variations within the scope of the claims, which do not affect the essence of the present invention.

Claims

1. A Chinese speech synthesis method based on Chinese word segmentation optimization, characterized in that: First, the weighted self-distillation-based Chinese word segmentation model CWS-WSD is used to segment the response text generated during the interaction with the medical service robot, clarifying the word boundaries in the text. Then, according to the word segmentation results, corresponding pause markers are inserted. Finally, the text after the markers are inserted is sent to Tacotron2 for speech synthesis. Chinese word segmentation is as follows: The input text sequence is semantically encoded and contextually encoded using an encoder to generate a semantic feature vector containing text semantic information and a contextual feature vector containing word formation information. The interactive attention mechanism is used to introduce the higher-order semantic relationship between contextual features and semantic features into the model training process, and the weighted self-distillation mechanism is used to constrain the model feature learning and guide the model to learn the contextual features in the text. The output features, which contain high-order semantic relationships between contextual and semantic features, are fed into the decoder for word segmentation and annotation.

2. The Chinese speech synthesis method based on Chinese word segmentation optimization as described in claim 1, characterized in that: The specific steps of Chinese word segmentation are as follows: S1. Construct a dictionary N to filter n-gram segments in the text that contain the corresponding characters; S2, Given a text sequence X = [ ,…, ,…, ],in This represents the length of the text sequence, for each character in the text sequence. The characters contained in the text sequence are obtained by traversing the dictionary N. All n-gram fragments, thus obtaining the key list K. i = [ ,…, ,…, Sum of values ​​list V i = [ ,…, ,…, ], m indicates that it contains characters The number of n-gram segments, using key embeddings express and value embedding express ,in The character represents the position of a character in the text sequence X, and j represents an n-gram segment. Its position in the key list; S3. Input the text sequence X into the CWS-WSD model, and use the encoder to process each character in the sequence. Encode a feature vector containing textual semantic information The formula is as follows: Here, Encoder represents the encoder, specifically the Bert encoder; S4. Transfer semantic feature vectors Key embedding Value embedding Calculations are performed to obtain contextual feature vectors containing word formation information. The calculation formula is as follows: in: This represents the feature vector containing textual semantic information generated by the encoder; Indicates about the included character n-gram fragments The probability distribution; This represents the contextual feature vector containing word formation information generated by the word formation memory network; S5. Utilize the contextual attention mechanism to model two types of feature-weighted embedding vectors containing contextual features, and utilize the semantic attention mechanism to model two types of feature-weighted embedding vectors containing semantic features. Then, concatenate and fuse the two weighted embedding vectors, and then reduce the dimensionality of the fused feature vector. This integrates the high-order semantic relationship between contextual features and semantic features into the model training process, guides the model to learn features, and finally obtains the output features of the interactive attention mechanism. S6. The decoder predicts the text label sequence Y based on the output features of the interactive attention mechanism. The formula is as follows: Where: Decoder represents the decoder. Using CRF as the decoder, the algorithm is defined by the following formula: in: , Both are trainable parameters; The S7 and CWS-WSD models first use an encoder to semantically encode the input text sequence, generating feature vectors containing textual semantic information. Then, they use an attention mechanism and a key-value memory network to encode feature vectors containing textual contextual information. Next, they use an interactive attention mechanism to introduce the higher-order semantic relationships between contextual and semantic features into the model's training process. Furthermore, they use a weighted self-distillation mechanism to constrain the model's feature learning and guide the model to learn contextual features in the text. Finally, the output features containing the higher-order semantic relationships between contextual and semantic features are fed into the decoder for word segmentation and annotation.

3. The Chinese speech synthesis method based on Chinese word segmentation optimization as described in claim 2, characterized in that: The dictionary N is constructed using an unsupervised word formation method. The algorithm for constructing dictionary N based on the original corpus is defined as follows: AV represents adjacency diversity. This represents the number of distinct characters to the left of the n-gram segment n. This indicates the number of distinct characters to the right of the n-gram segment n.

4. The Chinese speech synthesis method based on Chinese word segmentation optimization as described in claim 2, characterized in that: The contextual feature attention mechanism in step S5 The matrix represents the embedding vectors containing semantic features, while K and V represent the embedding vectors containing contextual features. The semantic feature attention mechanism... The matrix represents the embedding vector of contextual features, while K and V represent the embedding vectors of semantic features. This enables the modeling of higher-order semantic relationships between contextual and semantic features and the information flow interaction between the two attention mechanism modules. Assuming the vectors... For embedded features containing semantic features, vectors For embedded features that include contextual features, the computation process of the contextual attention mechanism and the semantic attention mechanism is as follows: For contextual attention mechanism, For semantic attention mechanism, This is the attention-weighted embedding vector that includes contextual features. The attention-weighted embedding vector contains semantic features. After obtaining the two weighted embedding vectors, the features of the two are concatenated. Then, the concatenated feature vector is fed into a fully connected layer for dimensionality reduction. Finally, the output features of the interactive attention mechanism are obtained. The specific calculation is as follows: in, These are the learnable parameters of the model. These are the bias parameters of the model. This means concatenating the embedding vector containing contextual features and the embedding vector containing semantic features along a certain dimension. The dimension of the embedding vector, which contains contextual features and higher-order semantic associations of semantic features.

5. The Chinese speech synthesis method based on Chinese word segmentation optimization as described in claim 2, characterized in that: The weighted self-distillation mechanism in step S7 is specifically as follows: The predicted outputs of the teacher model and the student model are scored, and the scores are used as weighting coefficients to weight the information flow of the two models, guiding the model to learn the contextual features in the text. The specific scoring calculation process is as follows: in, This indicates the number of tags in the tag sequence corresponding to a given text sequence. This represents the number of correctly labeled labels in the label sequence predicted by the teacher model. This represents the number of correctly labeled labels in the student model's predicted label sequence. The teacher model scores the scoring mechanism. The scoring mechanism is represented by the student model's score. After obtaining two weight coefficients, the output information streams of the teacher model and the student model are weighted to achieve weighted self-distillation.

6. The Chinese speech synthesis method based on Chinese word segmentation optimization as described in claim 2, characterized in that: The loss of the CWS-WSD model is divided into two parts: the first part is the model loss during the self-training process, which uses the negative log-likelihood function as the loss function; the second part is the model loss during the self-distillation process. The first part loss, the second part loss, and the total loss function are shown below: in, The model parameters for CWS-WSD are continuously optimized during the model training process; This represents the model's self-training loss. Weighted self-distillation loss; This is the total loss of CWS-WSD; The information flow is input to the decoder for the teacher model. The information flow is input to the decoder for the student model. The hyperparameters for CWS-WSD are set to balance the two types of loss.

7. An electronic device, characterized in that, include: Memory and at least one computer program located on the memory; It also includes at least one processor for processing at least one computer program in the memory, wherein when the processor executes the computer program, it implements the steps of the Chinese speech synthesis method as described in any one of claims 1-6.

8. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the steps of the Chinese speech synthesis method according to any one of claims 1-6.

Images