Polyphone prediction method, device and equipment and readable storage medium

Abstract

The invention provides a polyphone prediction method, device and equipment and a readable storage medium, and the method comprises the steps: extracting context embedding of a to-be-tested polyphone for a word segmentation sequence of a processing text containing the to-be-tested polyphone through a pronunciation prediction network which learns semantic information and pronunciation characteristics of the polyphone in different word segmentation modes; a linear projection matrix obtained through training in the pronunciation prediction network is utilized, the target segmented word and part-of-speech information of the target segmented word are fused to determine a conditional weight matrix, and the conditional weight matrix is utilized to dynamically adjust prediction probabilities for different pronunciation labels in the network output process, so that the pronunciation prediction network can better understand the context where polyphones are located, and the pronunciation prediction efficiency is improved. And the accuracy and robustness of polyphone prediction are improved.

Description

Technical Field

[0001] The present application relates to the field of natural language processing, and in particular to a polyphonetic character prediction method, apparatus, device and readable storage medium. Background Art

[0002] In the field of natural language processing, in the process of accurately converting Chinese text into its corresponding pronunciation representation, that is, Mandarin text-to-speech technology, since there are a large number of Chinese characters with multiple pronunciations and their pronunciation is often affected by the context, it is necessary to accurately determine the correct pronunciation of each Chinese character in a specific context.

[0003] Traditional polyphone processing methods mainly rely on rule-based methods, which divide the text into word fragments by building a detailed dictionary and presetting a complex set of rules, and eliminate pronunciation ambiguity based on the pronunciation entries in the dictionary and the rule set.

[0004] However, this method is highly dependent on the completeness of the dictionary and the accuracy of the rules. Since the update of the dictionary often lags behind the development of the language, and the rules are difficult to cover all complex language phenomena, and the rules are static and fixed, they cannot be flexibly adjusted according to the dynamic semantic environment of the text. When processing texts with complex semantic relationships, prediction errors are prone to occur, and there are problems with low accuracy and poor flexibility in processing polyphones. Summary of the invention

[0005] In view of this, in order to solve the above technical problems, the present application provides a polyphonetic character prediction method, device, equipment and readable storage medium.

[0006] Specifically, the present application is implemented through the following technical solutions:

[0007] According to a first aspect of an embodiment of the present application, a polyphonetic character prediction method is provided, which is applied to a pre-trained pronunciation prediction network, wherein the pronunciation prediction network is provided with a pronunciation label set corresponding to all pronunciations of each polyphonetic character; the method comprises:

[0008] For the processed text containing the polyphone to be tested, obtaining an initial word segmentation sequence obtained by segmenting words as units;

[0009] According to the initial word segmentation sequence, obtaining the context embedding of the polyphonetic character to be tested;

[0010] Using a linear projection matrix, feature linear projection is performed on at least one of the target segmentation containing the polyphone to be tested, the part of speech of the target segmentation, and the cross feature of the target segmentation and the part of speech in the initial segmentation sequence to generate a conditional weight matrix; the length of the conditional weight matrix is ​​the same as the length of the pronunciation label set;

[0011] By applying the output activation function of the conditional weight matrix, generating a predicted probability distribution for the pronunciation label set according to the context embedding, and determining the pronunciation of the polyphonetic character to be predicted;

[0012] The pronunciation prediction network is trained by obtaining the predicted pronunciations of polyphones in the same sample sentence under different word segmentation methods and minimizing the weighted results of the loss functions corresponding to the predicted pronunciations.

[0013] Optionally, the obtaining of an initial word segmentation sequence obtained by segmenting words as units includes:

[0014] Using a word-level segmentation tool to perform segmentation processing on the processed text to obtain a first segmentation sequence;

[0015] For each first participle in the first participle sequence, when the first participle fails to match the preset vocabulary, the first participle is updated to a second participle; the second participle is obtained by performing participle processing on the first participle using the participle function set in the pronunciation prediction network;

[0016] The first word segmentation sequence after the update is determined as the initial word segmentation sequence.

[0017] Optionally, obtaining the context embedding of the polyphonetic character to be tested according to the initial word segmentation sequence includes:

[0018] Encode each word in the initial word segmentation sequence to obtain an initial embedding vector corresponding to each word segmentation, including a word embedding vector and a position embedding vector; the word embedding vector is used to represent the semantic information of the word segmentation itself; the position embedding vector is used to represent the position information of the word segmentation itself;

[0019] According to the initial embedding vector of each word segment, the context information is fused to the initial embedding vector of the target word segment through an attention mechanism to generate a context embedding of the polyphone to be tested.

[0020] Optionally, generating a conditional weight matrix includes:

[0021] Obtaining a character encoding vector of the target word segment and a part-of-speech encoding vector of the part-of-speech;

[0022] Extracting a cross feature vector of the character encoding vector and the part-of-speech encoding vector, and obtaining a first projection result of the cross feature by using a first projection matrix in the linear projection matrix;

[0023] Using a second projection matrix in the linear projection matrix, obtaining a second projection result of the character code vector;

[0024] Using the second projection matrix in the linear projection matrix, obtaining a third projection result of the part-of-speech encoding vector;

[0025] The conditional weight matrix is ​​generated according to the first projection result, the second projection result and the third projection result.

[0026] Optionally, the extracting cross-features of the character encoding vector and the part-of-speech encoding vector includes at least any one of the following:

[0027] Obtaining a tensor product of the character encoding vector and the part-of-speech encoding vector as the cross feature;

[0028] Alternatively, an element-by-element addition result of the character encoding vector and the part-of-speech encoding vector is obtained as the cross feature.

[0029] Optionally, generating the conditional weight matrix according to the first projection result, the second projection result and the third projection result includes:

[0030] According to the three set hyperparameters, weighted processing is performed on the first projection result, the second projection result and the third projection result respectively to obtain three corresponding weighted projection results;

[0031] Performing element addition processing on the three weighted projection results to generate an initial weight matrix;

[0032] Each element in the initial weight matrix is ​​converted into a probability value to obtain the conditional weight matrix; the numerical range of the probability value is 0 to 1.

[0033] Optionally, after generating the conditional weight matrix, the method further includes:

[0034] Determining candidate pronunciation labels of the polyphonetic character to be predicted; the candidate pronunciation labels represent all pronunciations of the polyphonetic character to be predicted;

[0035] For the pronunciation label set, setting mask 0 for the other pronunciation labels except the candidate pronunciation label, and setting mask 1 for the candidate pronunciation label, to obtain a mask matrix;

[0036] The mask matrix is ​​multiplied element by element by the conditional weight matrix to obtain an updated conditional weight matrix.

[0037] Optionally, the pronunciation prediction network includes a part-of-speech prediction subnetwork, which is provided with a part-of-speech tag set; the method further includes:

[0038] Mapping the context embedding of the polyphone to be tested into a part-of-speech score vector through the first mapping parameter in the part-of-speech prediction subnetwork; the part-of-speech score vector includes an unnormalized score of each part-of-speech tag in the part-of-speech tag set;

[0039] By using an activation function for part-of-speech classification, a predicted probability distribution for the part-of-speech tag set is generated according to the part-of-speech score vector, and the predicted part-of-speech of the target word is determined as the part-of-speech of the target word.

[0040] Optionally, generating a predicted probability distribution for the pronunciation label set according to the context embedding by applying the output activation function of the conditional weight matrix comprises:

[0041] According to the second mapping parameter learned in the training phase, the context embedding is mapped into a pronunciation score vector; the pronunciation score vector includes an unnormalized score of each pronunciation tag in the pronunciation tag set;

[0042] For each pronunciation tag, obtaining a conditional weight in the conditional weight matrix that is in the same position as the unnormalized score of the pronunciation tag;

[0043] According to the unnormalized score of the pronunciation label and the conditional weight, normalization processing is performed through the output activation function to obtain the probability value of the pronunciation label.

[0044] Optionally, the normalizing process is performed by the output activation function according to the unnormalized score of the pronunciation label and the conditional weight, including:

[0045] Determining an exponential operation result of an unnormalized score of the pronunciation label;

[0046] Obtaining the product of the index operation result and the conditional weight of the pronunciation label as the target operation result;

[0047] The target operation result of each pronunciation label is normalized according to the sum of the target operation results of all pronunciation labels.

[0048] Optionally, the method further comprises a step of training a pronunciation prediction network:

[0049] For each sample sentence, a first segmentation sequence is obtained by segmenting words in single-character units, and a second segmentation sequence is obtained by segmenting words in words; the sample sentence includes a polyphonetic character and a pronunciation tag of the polyphonetic character; the second segmentation sequence also includes a part-of-speech tag of the polyphonetic character;

[0050] Inputting the first segmentation sequence and the second segmentation sequence into the pronunciation prediction network together, obtaining a first predicted pronunciation corresponding to the first segmentation sequence and a second predicted pronunciation corresponding to the second segmentation sequence; the conditional weight matrix is ​​determined according to the segmentations containing polyphones in the second segmentation sequence and the part-of-speech tags;

[0051] According to the first predicted pronunciation and the second predicted pronunciation, a target loss function is obtained, and the model parameters of the pronunciation prediction network are adjusted by minimizing the target loss function; the target loss includes a first loss between the first predicted pronunciation and the pronunciation mark, and a second loss between the second predicted pronunciation and the pronunciation mark.

[0052] Optionally, the pronunciation prediction network includes a part-of-speech prediction subnetwork, and the method further includes:

[0053] Determining the context embedding of the word containing the polyphonetic character by the pronunciation prediction network;

[0054] Inputting the context embedding into the part-of-speech prediction subnetwork to obtain the predicted part-of-speech output by the part-of-speech prediction subnetwork;

[0055] The conditional weight matrix is ​​determined according to the segmentation of the polyphonetic character and the predicted part of speech;

[0056] The target loss function also includes a third loss of the predicted part of speech and the part of speech tag.

[0057] According to a second aspect of an embodiment of the present application, a polyphone prediction device is provided, which is applied to a pre-trained pronunciation prediction network, wherein the pronunciation prediction network is provided with a pronunciation label set corresponding to all pronunciations of each polyphone; the pronunciation prediction network is trained by obtaining predicted pronunciations of polyphones under different word segmentation methods for the same sample sentence, and minimizing the weighted results of the loss functions corresponding to each predicted pronunciation; the device comprises:

[0058] A word segmentation processing module is used to obtain an initial word segmentation sequence obtained by segmenting words as units for a processing text containing a polyphone to be tested;

[0059] A context acquisition module, used for acquiring the context embedding of the polyphonetic character to be tested according to the initial word segmentation sequence;

[0060] A conditional weight matrix generation module is used to use a linear projection matrix to perform feature linear projection on at least one of the target segmentation containing the polyphone to be tested, the part of speech of the target segmentation, and the cross feature of the target segmentation and the part of speech in the initial segmentation sequence to generate a conditional weight matrix; the length of the conditional weight matrix is ​​the same as the length of the pronunciation label set;

[0061] The prediction probability and pronunciation determination module is used to generate a prediction probability distribution for the pronunciation label set according to the context embedding by applying the output activation function of the conditional weight matrix, and determine the pronunciation of the polyphone to be predicted.

[0062] Optionally, the word segmentation processing module is specifically used for:

[0063] Using a word-level segmentation tool to perform segmentation processing on the processed text to obtain a first segmentation sequence;

[0064] For each first participle in the first participle sequence, when the first participle fails to match the preset vocabulary, the first participle is updated to a second participle; the second participle is obtained by performing participle processing on the first participle using the participle function set in the pronunciation prediction network;

[0065] The first word segmentation sequence after the update is determined as the initial word segmentation sequence.

[0066] Optionally, the context acquisition module is specifically used to:

[0067] Encode each word in the initial word segmentation sequence to obtain an initial embedding vector corresponding to each word segmentation, including a word embedding vector and a position embedding vector; the word embedding vector is used to represent the semantic information of the word segmentation itself; the position embedding vector is used to represent the position information of the word segmentation itself;

[0068] According to the initial embedding vector of each word segment, the context information is fused to the initial embedding vector of the target word segment through an attention mechanism to generate a context embedding of the polyphone to be tested.

[0069] Optionally, the conditional weight matrix generating module is specifically used for:

[0070] Obtaining a character encoding vector of the target word segment and a part-of-speech encoding vector of the part-of-speech;

[0071] Extracting a cross feature vector of the character encoding vector and the part-of-speech encoding vector, and obtaining a first projection result of the cross feature by using a first projection matrix in the linear projection matrix;

[0072] Using the second projection matrix in the linear projection matrix, a second projection result of the character encoding vector is obtained; using the second projection matrix in the linear projection matrix, a third projection result of the part-of-speech encoding vector is obtained;

[0073] The conditional weight matrix is ​​generated according to the first projection result, the second projection result and the third projection result.

[0074] Optionally, when the conditional weight matrix generation module is used to extract the cross-features of the character encoding vector and the part-of-speech encoding vector, at least any one of the following methods is included:

[0075] Obtaining a tensor product of the character encoding vector and the part-of-speech encoding vector as the cross feature;

[0076] Alternatively, an element-by-element addition result of the character encoding vector and the part-of-speech encoding vector is obtained as the cross feature.

[0077] Optionally, when the conditional weight matrix generating module is used to generate the conditional weight matrix according to the first projection result, the second projection result and the third projection result, it includes:

[0078] According to the three set hyperparameters, weighted processing is performed on the first projection result, the second projection result and the third projection result respectively to obtain three corresponding weighted projection results;

[0079] Performing element addition processing on the three weighted projection results to generate an initial weight matrix;

[0080] Each element in the initial weight matrix is ​​converted into a probability value to obtain the conditional weight matrix; the numerical range of the probability value is 0 to 1.

[0081] Optionally, after generating the conditional weight matrix, the apparatus further includes:

[0082] Determining candidate pronunciation labels of the polyphonetic character to be predicted; the candidate pronunciation labels represent all pronunciations of the polyphonetic character to be predicted;

[0083] For the pronunciation label set, setting mask 0 for the other pronunciation labels except the candidate pronunciation label, and setting mask 1 for the candidate pronunciation label, to obtain a mask matrix;

[0084] The mask matrix is ​​multiplied element by element by the conditional weight matrix to obtain an updated conditional weight matrix.

[0085] Optionally, the pronunciation prediction network includes a part-of-speech prediction subnetwork, which is provided with a part-of-speech tag set; the device further includes:

[0086] Mapping the context embedding of the polyphone to be tested into a part-of-speech score vector through the first mapping parameter in the part-of-speech prediction subnetwork; the part-of-speech score vector includes an unnormalized score of each part-of-speech tag in the part-of-speech tag set;

[0087] By using an activation function for part-of-speech classification, a predicted probability distribution for the part-of-speech tag set is generated according to the part-of-speech score vector, and the predicted part-of-speech of the target word is determined as the part-of-speech of the target word.

[0088] Optionally, the prediction probability and pronunciation determination module is specifically used for:

[0089] According to the second mapping parameter learned in the training phase, the context embedding is mapped into a pronunciation score vector; the pronunciation score vector includes an unnormalized score of each pronunciation tag in the pronunciation tag set;

[0090] For each pronunciation tag, obtaining a conditional weight in the conditional weight matrix that is in the same position as the unnormalized score of the pronunciation tag;

[0091] According to the unnormalized score of the pronunciation label and the conditional weight, normalization processing is performed through the output activation function to obtain the probability value of the pronunciation label.

[0092] Optionally, when the prediction probability and pronunciation determination module is used to perform normalization processing through the output activation function according to the unnormalized score of the pronunciation label and the conditional weight, it includes:

[0093] Determining an exponential operation result of an unnormalized score of the pronunciation label;

[0094] Obtaining the product of the index operation result and the conditional weight of the pronunciation label as the target operation result;

[0095] The target operation result of each pronunciation label is normalized according to the sum of the target operation results of all pronunciation labels.

[0096] Optionally, the device further comprises a training step of a pronunciation prediction network:

[0097] For each sample sentence, a first segmentation sequence is obtained by segmenting words in single-character units, and a second segmentation sequence is obtained by segmenting words in words; the sample sentence includes a polyphonetic character and a pronunciation tag of the polyphonetic character; the second segmentation sequence also includes a part-of-speech tag of the polyphonetic character;

[0098] Inputting the first segmentation sequence and the second segmentation sequence into the pronunciation prediction network together, obtaining a first predicted pronunciation corresponding to the first segmentation sequence and a second predicted pronunciation corresponding to the second segmentation sequence; the conditional weight matrix is ​​determined according to the segmentations containing polyphones in the second segmentation sequence and the part-of-speech tags;

[0099] According to the first predicted pronunciation and the second predicted pronunciation, a target loss function is obtained, and the model parameters of the pronunciation prediction network are adjusted by minimizing the target loss function; the target loss includes a first loss between the first predicted pronunciation and the pronunciation mark, and a second loss between the second predicted pronunciation and the pronunciation mark.

[0100] Optionally, the pronunciation prediction network includes a part-of-speech prediction subnetwork, and the device further includes: determining the context embedding of the word segment containing the polyphonetic character through the pronunciation prediction network;

[0101] The context embedding is input into the part-of-speech prediction subnetwork to obtain the predicted part-of-speech output by the part-of-speech prediction subnetwork; the conditional weight matrix is ​​determined based on the word segmentation containing polyphones and the predicted part-of-speech; the target loss function also includes a third loss of the predicted part-of-speech and the part-of-speech tag.

[0102] According to a third aspect of an embodiment of the present application, an electronic device is provided, comprising: a memory and a processor; the memory is used to store a computer program; the processor is used to execute the above-mentioned polyphonetic character prediction method by calling the computer program.

[0103] According to a fourth aspect of the embodiments of the present application, a computer-readable storage medium is provided, on which a computer program is stored, and when the program is executed by a processor, the above-mentioned polyphonetic character prediction method is implemented.

[0104] The technical solution provided by the embodiments of the present application may have the following beneficial effects:

[0105] In the technical solution provided by the present application, a pronunciation prediction network that has learned the semantic information and pronunciation features of polyphones under different word segmentation methods is used to extract the context embedding of the polyphone to be tested through the word segmentation sequence of the processed text containing the polyphone to be tested, accurately capture the pronunciation features of the polyphone in the current context, and use the linear projection matrix trained in the pronunciation prediction network to fuse the target word segmentation and its part-of-speech information to determine the conditional weight matrix. The conditional weight matrix is ​​used to dynamically adjust the prediction probability for different pronunciation labels in the network output process, so that the pronunciation prediction network can better understand the context of the polyphone and make more accurate pronunciation predictions, thereby improving the accuracy and robustness of polyphone predictions, reducing the confusion and inconvenience of user understanding, and improving user experience.

[0106] It should be understood that the above general description and the following detailed description are only exemplary and explanatory and cannot limit the present application. In addition, any embodiment in the present application does not need to achieve all the above effects. BRIEF DESCRIPTION OF THE DRAWINGS

[0107] The accompanying drawings here are incorporated into the specification and form a part of this specification, showing embodiments consistent with the present application, and are used together with the specification to explain the principles of the present application.

[0108] Figure 1A It is a schematic flow chart of a method for predicting polyphonic characters shown in an exemplary embodiment of the present application;

[0109] Figure 1B It is a schematic diagram of the network structure of a pronunciation prediction network shown in an exemplary embodiment of the present application;

[0110] Figure 2 It is a schematic flow chart of steps for obtaining an initial word segmentation sequence shown in an exemplary embodiment of the present application;

[0111] Figure 3 It is a schematic flow chart of steps for obtaining the context embedding of the polyphonic character to be measured shown in an exemplary embodiment of the present application;

[0112] Figure 4A It is a schematic flow chart of steps for generating the conditional weight matrix shown in an exemplary embodiment of the present application;

[0113] Figure 4B It is a schematic flow chart of steps for obtaining the conditional weight matrix according to three projection results shown in an exemplary embodiment of the present application;

[0114] Figure 4C It is another schematic flow chart of steps for generating the conditional weight matrix by fusing specific mask information based on the conditional weight shown in an exemplary embodiment of the present application;

[0115] Figure 5 It is a schematic diagram of the training process of a pronunciation prediction network shown in an exemplary embodiment of the present application;

[0116] Figure 6 It is a schematic diagram of the structure of a polyphonic character prediction device shown in an exemplary embodiment of the present application;

[0117] Figure 7 It is a schematic diagram of the hardware of an electronic device shown in an exemplary embodiment of the present application. Detailed implementation manners

[0118] In the field of natural language processing, in the process of accurately converting Chinese text into the corresponding pronunciation representation, that is, the Mandarin text-to-speech technology, due to the existence of a large number of characters with multiple pronunciations in Chinese characters, and the pronunciation of polyphonic characters is often affected by the context. For example, for the polyphonic character "行", the pronunciation is represented as "xing2" in the text "我骑自行车上学" (I go to school by bike) and as "hang2" in the text "去银行" (go to the bank). Therefore, it is necessary to accurately judge the correct pronunciation of each Chinese character in a specific context.

[0119] Traditional methods for handling polyphones rely on rule-based approaches, which work by building detailed dictionaries and presetting complex rule sets. The dictionaries contain a large number of pronunciation entries for Chinese characters, while the rule sets are used to parse the text, divide the text into word fragments, and determine the pronunciation of each Chinese character based on the pronunciation entries in the dictionary and the rule sets to eliminate pronunciation ambiguity of polyphones.

[0120] This method is highly dependent on the completeness of the dictionary and the accuracy of the rules. However, with the continuous development of the language and the emergence of new vocabulary, the update of the dictionary is often difficult to keep pace with the development of the language, resulting in some newly emerging polyphones or special pronunciations cannot be accurately recognized. Moreover, the rule set based on the complexity of the Chinese language is difficult to cover all possible contexts and pronunciation changes. In addition, the rules are static and fixed and cannot be flexibly adjusted according to the dynamic semantic environment of the text. Prediction errors are prone to occur when processing texts with complex semantic relationships. There are problems of low accuracy and poor flexibility in polyphone processing.

[0121] In view of this, the embodiment of the present application proposes a polyphone prediction method implemented by a pre-trained pronunciation prediction network, which uses the pronunciation prediction network to capture the dynamic characteristics and contextual information of polyphones, thereby achieving accurate judgment of the pronunciation of polyphones. In this embodiment, a pronunciation label set is provided in the output layer of the pronunciation prediction network, including the pronunciation labels corresponding to all pronunciations of each polyphone. For example, the number of polyphones in Chinese characters is N, and the number of pronunciations of polyphones is usually between 2 and 5. Then, the pronunciation label set contains the pronunciation labels corresponding to all pronunciations of the N polyphones.

[0122] During the training phase, the pronunciation prediction network obtains the predicted pronunciations of polyphones in the same sample sentence under different word segmentation methods, and minimizes the weighted results of the loss functions corresponding to the predicted pronunciations for training. That is, during the training phase, the pronunciation prediction network uses different word segmentation methods to obtain multiple sample word segmentation sequences for the same sample sentence, predicts the predicted pronunciations of the polyphones in each sample word segmentation sequence, calculates the losses of the predicted pronunciations of the same polyphone in different sample word segmentation sequences, and performs weighted summation of these losses to obtain a weighted loss. The internal parameters of the network are adjusted by minimizing the weighted loss, so that the pronunciation prediction network can better learn the contextual dependency, sentence structure and semantic information of polyphones during the training phase, thereby enhancing the accuracy and robustness of polyphone prediction.

[0123] Based on this pronunciation prediction network, see Figure 1A As shown in the flowchart of the method steps, the polyphone prediction method proposed in this embodiment may at least include the following steps:

[0124] S101, for a processed text containing a polyphone to be tested, obtaining an initial word segmentation sequence obtained by word segmentation;

[0125] The polyphone to be tested refers to a polyphone in a text whose correct pronunciation needs to be determined. A polyphone refers to a Chinese character with two or more pronunciations, and its pronunciation is related to the context in which the polyphone is located, such as a word or sentence.

[0126] In the present embodiment, the processing text containing the polyphone to be tested is used to provide context information for the polyphone to be tested to help the pronunciation of the polyphone to be tested. Therefore, the length of the processing text should be enough to contain the context information required for the polyphone to be tested, so as to be able to clearly express the context and semantic relationship of the polyphone to be tested. In practical applications, the length of the processing text can be the entire sentence, or it can be a part of the sentence such as a phrase or clause containing the polyphone to be tested, which can be dynamically adjusted according to actual conditions. For example, if the polyphone to be tested is located near the beginning or end of a sentence, the entire sentence or even more context sentences can be used as the processing text as a whole; if the polyphone to be tested is located in the middle of the sentence and the words around it have clarified its context, the clause or phrase where the polyphone to be tested is located can be used as the processing text.

[0127] Word segmentation refers to the process of dividing continuous text into meaningful and independent vocabulary units. According to the different granularity of word segmentation, it can be divided into word-level segmentation and single-character-level segmentation. Among them, word-level segmentation refers to segmenting the text according to the boundaries of the vocabulary. Each word segmentation unit is a complete word. This word segmentation method conforms to the reading and understanding habits of humans and can retain more semantic information; single-character-level segmentation refers to segmenting the text by characters, and each word segmentation unit is a separate character. For example, the processed text is "He is a bank accountant.", and the corresponding word segmentation result at the word level is "He / is / a / bank / accountant", and the single-character-level segmentation result is "He / is / a / bank / accountant". In this embodiment, word-level segmentation is performed on the processed text to obtain the corresponding word segmentation sequence as the initial word segmentation sequence.

[0128] S102, obtaining a context embedding of the polyphonetic character to be tested according to the initial word segmentation sequence;

[0129] Contextual embedding is a vector representation that represents the meaning of a word or character in a specific text environment. Taking into account the context information, the word or character and its context information are converted into a high-dimensional vector representation of a fixed length. This vector representation can capture the semantic information and contextual relationship of the word or character in a specific text. The contextual embedding changes according to the context of the word or character. The same word will have different embedding representations in different contexts.

[0130] For example, for sentence 1 "He goes to the bank every day." and sentence 2 "He walks every day." that contain the polyphonic character "行", the context embedding of "行" in sentence 1 tends to mean "noun, financial institution", while the context embedding of "行" in sentence 2 tends to mean "verb, walk".

[0131] In this embodiment, based on the processed text for providing context information for the polyphonic character to be measured, for the polyphonic character to be measured, its context embedding will be constructed according to the initial word segmentation sequence corresponding to the processed text and the sequence position of the polyphonic character to be measured in the initial word segmentation sequence. Regarding the context embedding, the pre-trained word embedding sub-network or the custom embedding layer in the pronunciation prediction network can be used to generate the context embedding of the target word segment containing the polyphonic character to be measured by extracting the semantic information and position information of the target word segment in the initial word segmentation sequence and combining the context relationship information between other word segments and the target word segment in the initial word segmentation sequence through the self-attention mechanism, and use it as the context embedding of the polyphonic character to be measured.

[0132] Among them, the position information may include absolute position information and relative position information. The absolute position information provides the exact position of the polyphonic character to be predicted in the initial word segmentation sequence to better understand the global structure and long-distance dependence relationship of the sentence, and the relative position information provides the relative distance and order between word segments in the initial word segmentation sequence to capture the local dependence relationship in the sentence. The two position information complement each other to better understand and capture the context information of the polyphonic character to be measured.

[0133] S103, using a linear projection matrix, perform feature linear projection on at least one of the target word segment containing the polyphonic character to be measured, the part-of-speech of the target word segment, and the cross feature of the target word segment and the part-of-speech in the initial word segmentation sequence to generate a conditional weight matrix; the length of the conditional weight matrix is the same as the length of the pronunciation label set;

[0134] The linear projection matrix belongs to the learnable parameters in the pronunciation prediction network and is obtained through the training stage. Based on the fact that the pronunciation prediction network is trained by obtaining the predicted pronunciations of polyphonic characters in the same sample sentence under different word segmentation methods and minimizing the weighted results of the loss functions corresponding to each predicted pronunciation, therefore, the linear projection matrix is used as a shared parameter for joint training of sample word segmentation sequences under different word segmentation granularities in the training stage, and is obtained by jointly training the sample word segmentation sequences of the sample sentence under different word segmentation granularities. The linear projection matrix learned in the training stage fuses and learns the influence of different word segmentation granularities on the pronunciation prediction of polyphonic characters, and captures the contribution of the part-of-speech of polyphonic characters in the context and the characters themselves to the semantics.

[0135] The target word segmentation refers to the word segmentation unit that contains the polyphone to be tested among all the word segmentation units included in the initial word segmentation sequence corresponding to the processed text. The part of speech of the target word segmentation is used to describe the grammatical function and category of the target word segmentation in the processed text. For example, the part of speech may include but is not limited to nouns, verbs, adjectives, pronouns, adverbs, prepositions, onomatopeia, etc.

[0136] In this embodiment, the linear projection matrix is ​​used to perform projection transformation on the input features, and a new and more representative comprehensive feature vector is generated by performing feature mapping and linear combination on the target segmentation containing the polyphone to be tested in the initial segmentation sequence and the part-of-speech tagging of the target segmentation, so as to better capture the semantic feature information of the polyphone.

[0137] The linear projection matrix can be used to project the original input features such as the character encoding vector corresponding to the target word segmentation, or the part-of-speech encoding vector corresponding to the part-of-speech, or the cross-features of the target word segmentation and the part-of-speech into a new feature space. The projection can be achieved by matrix multiplication, wherein the original input feature is multiplied by the linear projection matrix to obtain the projected new feature vector. In order to generate the conditional weight matrix, the pronunciation prediction network will use the projected new feature vector for further interaction and convert it into a probability matrix as a conditional weight matrix. The length of the conditional weight matrix is ​​the same as the number of pronunciation labels contained in the pronunciation label set set in the pronunciation prediction network, and the elements in the conditional weight matrix correspond one by one to the pronunciation labels in the pronunciation label set.

[0138] The conditional weight matrix is ​​used to adjust the activation function output of the pronunciation prediction model. It assigns different weights to each pronunciation label in the pronunciation label set set in the pronunciation prediction model. The weight reflects the possibility that the pronunciation label is predicted to be the pronunciation of the polyphone to be tested based on the target character and its part-of-speech information containing the polyphone to be tested. It integrates the contribution of the part-of-speech of the polyphone to be tested in the context and the semantic feature information extracted from the character itself to the pronunciation prediction. Each element in the conditional weight matrix is ​​used to represent the weight or probability of predicting the polyphone to be tested to belong to the corresponding pronunciation label based on the semantic feature information extracted from the part-of-speech of the polyphone to be tested in the context and the character itself.

[0139] S104, by applying the output activation function of the conditional weight matrix, generating a predicted probability distribution for the pronunciation label set according to the context embedding, and determining the pronunciation of the polyphonetic character to be predicted;

[0140] The output activation function refers to a function used to convert linear output into nonlinear output in a pronunciation prediction network, and is applied to the output layer of the pronunciation prediction network to ensure that the output value is within a suitable range. For classification tasks, the softmax function is usually used to convert the output into a probability distribution of the corresponding category. The output activation function uses the logit value (also referred to as unnormalized score or linear prediction) generated by the network for each pronunciation label as a function input, and outputs the probability that the polyphone to be predicted belongs to the pronunciation label. Wherein, the logit value is used to represent the original prediction output of the network for each category (referring to the pronunciation label herein), and the predicted value for each category directly obtained after the context embedding of the aforementioned polyphone to be tested is processed by the network, and is usually calculated by the linear layer of the network such as the fully connected layer, so it is also referred to as a linear prediction value, which reflects the linear division of the network to the category in the input feature space. For example, if the network needs to predict one of n pronunciation labels, the output layer of the network is provided with n neurons, and each neuron outputs the logit value of a pronunciation label.

[0141] In this embodiment, after the conditional weight matrix is ​​generated by the linear projection matrix, the conditional weight matrix participates in the calculation of the output activation function, and the elements corresponding to the pronunciation labels in the conditional weight matrix are used to adjust the predicted probability generated by the output activation function for the pronunciation labels. For example, for the i-th pronunciation label, in the process of the output activation function generating the predicted probability Pi corresponding to the i-th pronunciation label, the calculation of the output activation function will be based on the logit value of the network output corresponding to the i-th pronunciation label, and the weight element corresponding to the i-th pronunciation label in the conditional weight matrix, and the weight element is used to adjust the probability value determined according to the logit value.

[0142] In the pronunciation prediction network, after obtaining the context embedding of the polyphone to be tested, in order to further extract and integrate the feature information of the polyphone to be tested, the context embedding can be further encoded, and the encoding process can be implemented by a simple fully connected layer, or a more complex structure such as a recurrent neural network, a long short-term memory network, a gated recurrent unit or a Transformer encoder to obtain an encoding processing feature, and generate a logit value for each pronunciation label based on the encoding feature. The process of generating the logit value is to map the encoding processing feature to a space with the same dimension as the number of pronunciation labels in the pronunciation label set.

[0143] Based on this, for the step of generating a predicted probability distribution for the pronunciation label set according to the context embedding, the context embedding code can be mapped to the logit value of each pronunciation label, and the logit value is used as the input of the output activation function to obtain the probability of each pronunciation label output by the output activation function applying the conditional weight matrix, thereby forming the predicted probability distribution for the pronunciation label set, and then determining the pronunciation of the polyphone to be predicted based on the predicted probability distribution.

[0144] The pronunciation of the polyphone to be tested can be determined in any of the following ways: determine the pronunciation label with the highest probability value from the predicted probability distribution as the target pronunciation label, and determine the pronunciation corresponding to the target pronunciation label as the pronunciation of the polyphone to be tested. Alternatively, in order to improve the accuracy and reliability of the prediction and avoid misjudgment due to network uncertainty or noise, the pronunciation label with the highest probability value and the probability value greater than the set probability threshold can be determined from the predicted probability distribution as the target pronunciation label, and then the pronunciation corresponding to the target pronunciation label is determined as the pronunciation of the polyphone to be tested. The set probability threshold is a confidence dividing line preset according to task requirements and model performance, and is used to screen out pronunciation labels with sufficiently high confidence. When the highest probability value exceeds the set probability threshold, the network's prediction is considered reliable, and then the pronunciation corresponding to the target pronunciation label is determined as the pronunciation of the polyphone to be tested.

[0145] In the disclosed embodiments, by utilizing a pronunciation prediction network that has learned the semantic information and pronunciation features of polyphones under different word segmentation methods, the context embedding of the polyphone to be tested is extracted from the word segmentation sequence of the processed text containing the polyphone to be tested, and the pronunciation features of the polyphone in the current context are accurately captured. The linear projection matrix trained in the pronunciation prediction network is used to fuse the target word segmentation and its part-of-speech information to determine the conditional weight matrix, and the prediction probability for different pronunciation labels in the network output process is dynamically adjusted based on the conditional weight matrix, so that the pronunciation prediction network can better understand the context in which the polyphone is located, more accurately capture the pronunciation features of the polyphone in the current context, and achieve more accurate pronunciation prediction, thereby improving the accuracy and robustness of polyphone prediction, reducing the user's understanding difficulties caused by pronunciation prediction errors, and improving the user experience.

[0146] Based on the steps of the above polyphone prediction method, Figure 1BA schematic diagram of a pronunciation prediction network structure is shown as an example. The pronunciation prediction network involved in the present application may include at least an input layer, an encoding layer, a conditional weighting layer, and an output layer; wherein the input layer is used to receive a processed text containing a polyphone to be tested, and obtain an initial word segmentation sequence obtained by word segmentation in units of words; the encoding layer is used to obtain the context embedding of the polyphone to be tested based on the initial word segmentation sequence, and can also be used to perform further encoding processing on the context embedding; the parameters of the conditional weighting layer include a linear projection matrix, which is used to process the initial word segmentation sequence based on the linear projection weights, that is, to determine the initial word segmentation sequence contains The target segmentation of the polyphone to be tested, and the part of speech of the target segmentation, and at least one of the cross-features of the target segmentation and the part of speech, are subjected to feature linear projection to generate a conditional weight matrix; the output layer takes the output results of the encoding layer and the conditional weighting layer as input, and obtains the logit value corresponding to each pronunciation label by mapping the output results of the encoding layer to a space with the same dimension as the number of pronunciation labels, and generates a predicted probability distribution for the pronunciation label set according to the logit value by applying the output activation function of the conditional weight matrix, and then outputs the pronunciation of the polyphone to be predicted according to the predicted probability distribution.

[0147] The above-mentioned input layer can be a simple text processing module that can convert a text string into a word segmentation sequence. The input layer can also obtain the initial word segmentation sequence by calling other external tools or additional word segmentation networks for word segmentation processing. The encoding layer can use a pre-trained word embedding network to extract the context embedding of the polyphone to be tested, or the encoding layer can also use a convolutional neural network, a recurrent neural network or a Transformer structure to further encode the context embedding to extract deeper semantic information. The conditional weighted layer can include one or more fully connected layers for feature mapping of the input target word segmentation and its part of speech, generating a vector representation of the target word segmentation, a vector representation of the part of speech and their cross features, and using a linear projection matrix for linear transformation to generate a conditional weight matrix with the same dimension as the number of pronunciation labels. In practical applications, the conditional weighted layer can also use an attention mechanism to dynamically adjust the weights to better capture the pronunciation characteristics of polyphones in different contexts. The output layer can include at least one fully connected layer and a softmax activation function. The fully connected layer is used to map the output results of the encoding layer to a space with the same dimension as the number of pronunciation labels to obtain logit values. The softmax activation function is used to convert the logit values ​​into probability distributions to ensure that the sum of all probabilities is 1. During the training phase, the output layer can use loss functions such as cross entropy loss, negative log-likelihood loss, etc. to optimize network performance.

[0148] It is understandable that the exemplary network hierarchy of the pronunciation prediction network described above can be flexibly adjusted according to design requirements in practical applications, and this application does not limit this.

[0149] In some embodiments, in order to improve the accuracy of text segmentation processing, a vocabulary is pre-set in this embodiment. The preset vocabulary can include at least commonly used Chinese words, which can be set by the top N words with the highest frequency of the word-level segmentation tool used. For example, if the word-level segmentation tool is jieba, the preset vocabulary can include at least the 20,000 words with the highest frequency of jieba. For the aforementioned step S101, this embodiment proposes a new segmentation method based on the vocabulary, see Figure 2 As shown in the step flow chart, this word segmentation method can be implemented by the following steps:

[0150] S201, using a word-level segmentation tool to perform segmentation processing on the processing text to obtain a first segmentation sequence;

[0151] The word-level segmentation tool refers to a tool in natural language processing technology that can identify and segment words in a text, which is trained based on a large number of corpora and can accurately identify vocabulary boundaries in Chinese, English or other languages. In the present embodiment, the word-level segmentation tool includes other segmentation methods except the segmentation function set in the pronunciation prediction network, and can include segmentation tools in related technologies, such as jieba segmentation, SnowNLP, LTP (Language Technology Platform), HanLP, NLPIR, THULAC, FoolNLTK, etc., and can dynamically select a segmentation tool that is more suitable for the current segmentation task according to actual conditions.

[0152] In this step, the processed text is used as the input of the word-level segmentation tool, and the word-level segmentation tool is used to identify and segment the words in the text through a pre-trained model or algorithm to obtain the corresponding segmentation sequence output by the word-level segmentation tool as the first segmentation sequence. For example, the processed text "He is a bank accountant" and the first segmentation sequence output by the word-level segmentation tool can be expressed as "He / is / bank accountant".

[0153] S202, for each first participle in the first participle sequence, determining whether the first participle matches a preset vocabulary;

[0154] The preset vocabulary may include a custom vocabulary and a vocabulary of a pre-trained pronunciation prediction network, and the custom vocabulary may be determined according to the vocabulary of the word-level segmentation tool or other word-level segmentation tools used in step S201. That is, an additional set number of Chinese words may be added to the vocabulary (vocab.txt) of the pronunciation prediction network to enhance the performance of the pronunciation prediction network in Chinese word segmentation.

[0155] In this embodiment, the preset vocabulary is used to determine whether the segmentation result of the word-level segmentation tool meets expectations, by traversing each first segmentation in the first segmentation sequence, detecting whether the first segmentation matches the words included in the preset vocabulary, that is, whether the first segmentation exists in the preset vocabulary. If the first segmentation exists in the preset vocabulary, it means that the first segmentation belongs to a high-frequency word or a known word, and the segmentation result can be directly retained; on the contrary, if the first segmentation does not exist in the preset vocabulary, it means that the first segmentation may belong to a low-frequency word, a new word, or a non-word fragment obtained due to inaccurate segmentation of the word-level segmentation tool, and further segmentation processing is required.

[0156] For example, for the first word segmentation sequence "he / is / bank accountant" mentioned above, the three words "he", "is" and "bank accountant" are checked in turn to see whether they exist in the preset vocabulary to determine whether the first word segmentation matches the preset vocabulary.

[0157] S2031, when the first participle fails to match the preset vocabulary, updating the first participle to a second participle; the second participle is obtained by performing participle processing on the first participle using the participle function set in the pronunciation prediction network;

[0158] If the first word segmentation fails to match the preset vocabulary, it means that the preset vocabulary does not include the first word segmentation, and further processing is required for the first word segmentation. In this embodiment, for the first word segmentation that fails to match, the word segmentation function set in the pronunciation prediction network is called, such as the tokenize function of the pronunciation prediction network, to re-segment the first word segmentation, so as to further divide the first word segmentation into smaller units such as characters or subwords, and obtain the second word segmentation output by the word segmentation function of the network, and the first word segmentation in the first word segmentation sequence is updated to the second word segmentation.

[0159] For example, for the above first word segmentation sequence "He / Is / Bank Accountant", the three word segments "He", "Is", and "Bank Accountant" are sequentially detected. If it is detected that the first word segment "Bank Accountant" does not exist in the preset vocabulary, that is, the first word segment fails to match the preset vocabulary, then the first word segment "Bank Accountant" is re-segmented through the word segmentation function in the pronunciation prediction network. For example, if the second word segment output by the word segmentation function is "Bank / Accountant", then after the first word segment is updated to the second word segment, the first word segmentation sequence is represented as "He / Is / Bank / Accountant".

[0160] Regarding updating the first word segment to the second word segment, if the first word segment is different from the corresponding second word segment, the first word segment can be updated to the second word segment; conversely, if the first word segment is the same as the corresponding second word segment, the first word segment can remain unchanged.

[0161] S2032, in the case where the first word segment successfully matches the preset vocabulary, continue to match the next word segment of the first word segment with the preset vocabulary until all the first word segments in the first word segmentation sequence are matched.

[0162] The successful matching of the first word segment with the preset vocabulary indicates that the preset vocabulary includes the first word segment. Then, skip the first word segment and determine whether the next word segment of the first word segment matches the preset vocabulary. For example, taking the above first word segmentation sequence ":He / Is / Bank Accountant" as an example, if it is detected that the first word segment "He" successfully matches the preset vocabulary, then continue to detect whether the first word segment "Is" matches the preset vocabulary until the matching detection of the last word segment in the first word segmentation sequence is completed.

[0163] In the embodiments of the present disclosure, by presetting a vocabulary containing common Chinese words, using a word-level word segmentation tool for preliminary word segmentation, sequentially matching the word segmentation results with the preset vocabulary, and using the word segmentation function自带 by the pronunciation prediction network for re-word segmentation in the case of a matching failure, it effectively deals with the situation where some rare words, new words, or professional terms are not in the preset vocabulary, improves the generalization ability of the word segmentation process. Based on the fact that the words in the preset vocabulary are more in line with Chinese language habits, the above steps reduce the possibility of word segmentation errors, improving the word segmentation quality while ensuring the word segmentation efficiency.

[0164] In some embodiments, for the obtaining of the context embedding of the to-be-detected polyphonic word according to the initial word segmentation sequence described in the foregoing step S102, refer to Figure 3 The exemplary step flowchart shown can be implemented in the following manner:

[0165] S301. Encode each token in the initial token sequence to obtain an initial embedding vector corresponding to each token, including a word embedding vector and a position embedding vector. The word embedding vector is used to represent the semantic information of the token itself, and the position embedding vector is used to represent the position information of the token itself.

[0166] This initial embedding vector is used to initially capture and represent the semantic information of each token in the token sequence. It includes a word embedding vector and a position embedding vector. Among them, the word embedding vector maps each token unit in the initial token sequence to a vector in the real number space to generate a vector representation. Each token is represented as a vector of a fixed length, which can capture the semantic relationship between words and is used to represent the semantic information of the token itself. For example, for the polyphonic character "行" as part of the token "银行", the word embedding vector contains the overall semantic meaning of the word "银行". The position embedding vector is used to introduce the word order information in the initial token sequence and can be generated in various ways, such as using sine and cosine functions (such as the position encoding in the Transformer model) or a learned position embedding matrix. Since the word embedding itself does not contain the position information of the token unit in the processed text, the position embedding vector is used to supplement this position information, and the position embedding vector is used to represent the position information of the token itself.

[0167] The position information represented by the position embedding vector can include at least one of relative position encoding and absolute position encoding. Relative position encoding is used to capture the relative position relationship between token units. It is calculated based on the relative position between token units. By calculating the position difference between two token units and applying a learnable function or matrix, relative position encoding is generated. Absolute position encoding assigns a fixed encoding to each position, which can make up for the absolute position information that may be lost when there is only relative position encoding.

[0168] In this embodiment, by combining the word embedding vector and the position embedding vector of each token, such as adding or concatenating, the initial embedding vector of each token can be obtained. For example, for a token sequence containing polyphonic characters such as "我 / 今天 / 去 / 银行", each token will have an initial embedding vector Mi. Using Ei to represent the word embedding vector and Si to represent the position embedding vector, the corresponding initial embedding vector can be expressed as: "我": M1 = E1 + S1, "今天": M2 = E2 + S2, "去": M3 = E3 + S3, "银行": M4 = E4 + S4.

[0169] S302. According to the initial embedding vector of each token, use the attention mechanism to fuse the context information of the initial embedding vector of the target token to generate the context embedding of the to-be-detected polyphonic character.

[0170] The attention mechanism allows the network to dynamically focus on different parts of the input sequence when processing text. It is used to calculate the importance of each word segmentation unit in the initial word segmentation sequence for the current task and fuse feature information according to the weight representing the importance.

[0171] In an embodiment, an attention mechanism is used to calculate the correlation or importance between a target word segmentation and each word segmentation in an initial word segmentation sequence, the query vector of the target word segmentation is calculated using the attention weight matrix learned through the network, and the key vector and value vector of each word segmentation unit in the initial word segmentation sequence are calculated, the dot product or other similarity metrics are used to calculate the similarity score between the query vector and each key vector as the attention score, the softmax function is used to normalize the attention score to the attention weight, the weight represents the importance of each word segmentation unit in the initial word segmentation sequence to the target word segmentation, and then the value vectors of all word segmentations are weighted and summed according to the attention weight to obtain a vector representation of the fused context information as the context embedding of the polyphone to be tested, so that the context embedding contains information about the target word segmentation and its surrounding word segments, and can reflect the complete semantics of the polyphone to be tested in the current context.

[0172] For example, taking the initial word segmentation sequence "I / today / go / bank" as an example, the initial embedding vector corresponding to the word segmentation is expressed as: "I": M1=E1S1, "today": M2=E2S2, "go": M3=E3S3, "bank": M4=E4S4, the attention weight matrix learned in the attention mechanism includes Q, K, V, based on the attention machine mechanism Q is the query vector, which indicates the information that needs to be paid attention to at present; K is the key vector, which indicates the characteristics of the candidate information; and V is the value vector, which indicates the actual content of the candidate information. dk is the dimension of the key vector, which is used to scale the dot product result to prevent the gradient from disappearing or exploding. The attention mechanism calculation process can be expressed as follows: (1) For the target word "bank", calculate its query vector QM4; (2) For each word unit in the initial word sequence, calculate its key vector KMi and value vector VMi; (3) Calculate the similarity score score(QM4, KMi) between the query vector and each key vector; (4) Use the softmax function to normalize the score to the attention weight Pi4; (5) Weight the sum of the value vectors VMi of all word segments according to the attention weight Pi4 to obtain the context embedding of the polyphone to be tested.

[0173] In the disclosed embodiments, by combining word embedding vectors, position embedding vectors and attention mechanisms, the pronunciation prediction network can dynamically focus on the part of the initial word segmentation sequence that is related to the polyphone to be tested, and fuse this partial information to generate the context embedding of the polyphone to be tested, so that the network can more accurately understand the meaning of the polyphone to be tested in the current context, so as to improve the accuracy of polyphone pronunciation prediction.

[0174] In some embodiments, the linear projection matrix described in the aforementioned step S103 may include at least one of the first projection matrix, the second projection matrix and the third projection matrix, the first projection matrix is ​​used to project the cross-features formed by the character encoding vector of the target word segmentation and the part-of-speech encoding vector corresponding to the part of speech of the target word segmentation, the second projection matrix is ​​used to project the character encoding vector of the target word segmentation, and the third projection matrix is ​​used to project the part-of-speech encoding vector corresponding to the part of speech of the target word segmentation. In the pronunciation prediction network structure setting stage, the setting of the linear projection matrix is ​​determined according to the input feature type to be projected. For example, in the case of only projecting the cross-features of the character encoding vector of the target word segmentation and the part-of-speech encoding vector corresponding to the part of speech of the target word segmentation, the linear projection matrix may only include the first projection matrix; in the case of simultaneously projecting the cross-features, the target word segmentation, and the part of speech of the target word segmentation, the linear projection matrix includes the first, second and third projection matrices at the same time.

[0175] Based on the linear projection matrix, Figure 4A As shown in the schematic flow chart of the steps, for the aforementioned step S103 of performing feature linear projection on the target segmentation containing the polyphone to be tested in the initial segmentation sequence, the part of speech of the target segmentation, and at least one of the cross-features of the target segmentation and the part of speech to generate a conditional weight matrix, step S401 can be first executed to obtain the character encoding vector of the target segmentation and the part of speech encoding vector of the part of speech, and extract the cross-features of the character encoding vector and the part of speech encoding vector; next, step S4021 can be performed to obtain a first projection result of the cross-feature by using the first projection matrix in the linear projection matrix, or step S4022 can be performed to obtain a second projection result of the character encoding vector by using the second projection matrix in the linear projection matrix, or step S4023 can be performed to obtain a third projection result of the part of speech encoding vector by using the second projection matrix in the linear projection matrix, and then the conditional weight matrix is ​​generated according to the obtained projection results. When the cross-feature, character encoding vector and part-of-speech encoding vector are projected at the same time, step S403 can be executed to generate the conditional weight matrix according to the first projection result, the second projection result and the third projection result. Among them, steps S4021, S4022 and S4023 are three parallel steps, and at least one of the projection processes can be selected to be executed according to the needs to obtain the corresponding projection result.

[0176] The cross feature is a richer feature representation generated by combining character-level information and part-of-speech information, which is used to capture the interaction between the character encoding vector and the part-of-speech encoding vector, and can be understood as combining the character encoding vector and the part-of-speech encoding vector in some form to capture the interaction information between the two. In this embodiment, the character encoding vector and the part-of-speech encoding vector can be processed by element-level multiplication, element-level addition, concatenation, linear transformation, etc. to obtain the cross feature, which can be selected and used according to the needs of specific tasks to improve the representation ability and prediction performance of the network.

[0177] Among them, element-level multiplication generates a new vector as the cross feature by multiplying the character encoding vector and the part-of-speech encoding vector element by element. This method can emphasize the common features between the target word segmentation and the part-of-speech of the target word segmentation. Assuming that the character encoding vector is c and the part-of-speech encoding vector is p, the cross feature vector f can be expressed as: f = c⊙p, ⊙ represents element-by-element multiplication. Correspondingly, element-level addition generates a new vector as the cross feature by adding the character encoding vector and the part-of-speech encoding vector element by element. This method can combine the information of both while retaining their respective characteristics.

[0178] Concatenation refers to concatenating character encoding vectors and part-of-speech encoding vectors to generate a higher-dimensional vector. This method can retain the complete information of both, such as f = [c, p]. Linear transformation refers to combining character encoding vectors and part-of-speech encoding vectors through linear transformation (such as a fully connected layer) to generate a new vector. This method can learn more complex interactions. For example, assuming that W is a weight matrix and b represents a bias vector, the cross-feature vector f can be expressed as f = W[cp] + b.

[0179] In some embodiments, the step of extracting the cross-features of the character encoding vector and the part-of-speech encoding vector can be implemented in any of the following ways: obtaining the tensor product of the character encoding vector and the part-of-speech encoding vector as the cross-feature; or obtaining the element-by-element addition result of the character encoding vector and the part-of-speech encoding vector as the cross-feature.

[0180] Regarding obtaining the tensor product of the character encoding vector and the part-of-speech encoding vector, the tensor product of two vectors (also called outer product, cross product or vector product) is an important and useful concept in linear algebra and vector analysis. Suppose two three-dimensional vectors a = (a1, a2, a3), b = (b1, b2, b3), then the tensor product is a new vector f = a × b, which can be defined as f = (a2b3-a3b2, a3b1-a1b3, a1b2-a2b1), then the tensor product of the character encoding vector and the part-of-speech encoding vector can be generated based on the tensor product calculation principle of the three-dimensional vector. It can be understood that the essence of the tensor product is to map the character encoding vector and the part-of-speech encoding vector to generate a new, higher-dimensional vector and matrix, so the calculation method of the tensor product can also be defined by the developer himself. Non-standard tensor product calculations, such as bilinear transformations, can also be defined.

[0181] In the embodiments of the present disclosure, by obtaining the character encoding vector and part-of-speech encoding vector of the target word segmentation and extracting their cross-features, it is possible to more comprehensively capture the characteristics of the polyphone to be tested in different contexts, and use a linear projection matrix to perform feature linear projection on the character encoding vector, part-of-speech encoding vector and their cross-features to generate a conditional weight matrix, which not only reduces the feature dimension and the amount of calculation, but also effectively learns and integrates the part-of-speech of the polyphone to be tested in the context and the semantic information of the word segmentation itself through the learning ability of the projection matrix, so that the features are more distinctive in the projected space.

[0182] In some embodiments, when the cross-features, character encoding vectors and part-of-speech encoding vectors are projected simultaneously in the aforementioned embodiments to obtain a first projection result, a second projection result and a third projection result, in order to more flexibly adjust the contribution of these projection results when generating a conditional weight matrix, this embodiment proposes three hyperparameters as weight coefficients for these three projection results for dynamically adjusting the importance of each projection result during the training or inference process of the pronunciation prediction network.

[0183] Based on this, see Figure 4BAs shown, the step S403 in the above embodiment generates the conditional weight matrix according to the first projection result, the second projection result and the third projection result, which can be achieved in the following manner, namely: S4031, weighting the first projection result, the second projection result and the third projection result according to the three set hyperparameters, respectively, to obtain the corresponding three weighted projection results; S4032, performing element addition processing on the three weighted projection results to generate an initial weight matrix; S4033, converting each element in the initial weight matrix into a probability value to obtain the conditional weight matrix; the numerical range of the probability value is 0 to 1. In the process of performing element addition processing on the three weighted projection results, a bias term can be set to adjust the addition result.

[0184] Regarding converting each element in the initial weight matrix into a probability value to obtain the conditional weight matrix, the initial weight matrix may be activated by a sigmoid function to convert each element in the initial weight matrix into a probability value.

[0185] For example, the hyperparameters α1, α2, and α3 are set for the first, second, and third projection results respectively. After obtaining the three projection results r1, r2, and r3, the corresponding three weighted projection results can be expressed as α1*r1, α2*r2, and α3*r3. The three weighted projection results are added together to generate an initial weight matrix, which can be expressed as f'=α1*r1+α2*r2+α3*r3. Each element in f' is converted into a probability value to obtain the conditional weight matrix f.

[0186] In the disclosed embodiment, by introducing hyperparameters as weight coefficients, the contribution of cross-features, character encoding vectors, and part-of-speech encoding vectors in generating conditional weight matrices is dynamically adjusted, so that the network can more accurately capture the pronunciation changes of polyphones in different contexts, thereby improving the accuracy and reliability of prediction. In addition, by adjusting the weight coefficients, the pronunciation prediction network can better adapt to different training data and test environments, improving the generalization ability and robustness of the network.

[0187] In some embodiments, after generating the conditional weight matrix, in order to more effectively use the matrix to focus on the candidate pronunciations of the polyphone to be predicted and reduce the interference with other non-candidate pronunciations, this embodiment proposes another processing step of fusing specific mask information based on the conditional weights. Figure 4C As shown, after generating the conditional weight matrix, the following steps may also be included:

[0188] S404, determining candidate pronunciation labels of the polyphonetic character to be predicted; the candidate pronunciation labels represent all pronunciations of the polyphonetic character to be predicted;

[0189] The candidate pronunciation label is selected from the set of pronunciation labels set in the pronunciation prediction network, and all labels whose pronunciations represented by the pronunciation labels belong to the pronunciations of the polyphonic character to be measured. For example, if the polyphonic character to be measured is "行", the candidate pronunciation label refers to all pronunciation labels representing the polyphonic pronunciations of "行" in the set of pronunciation labels.

[0190] In this embodiment, the mapping relationship between the polyphonic character and all its pronunciations can be pre-constructed and stored. For the polyphonic character to be measured, the mapping record containing the polyphonic character to be measured is searched from the mapping relationship, and all pronunciations of the polyphonic character to be measured are extracted from the mapping record. Then, from all the pronunciation labels included in the set of pronunciation labels, the label representing the pronunciation of the polyphonic character to be measured is determined as the candidate pronunciation label.

[0191] S405. For the set of pronunciation labels, set the mask 0 for other pronunciation labels except the candidate pronunciation labels, and set the mask 1 for the candidate pronunciation labels to obtain a mask matrix.

[0192] The mask matrix is a matrix with the same shape as the conditional weight matrix. The length of the matrix is the same as the number of pronunciation labels included in the set of pronunciation labels, and its element values are 0 or 1, which are used to indicate which elements in the conditional weight matrix should be retained (elements with a value of 1), and which elements should be ignored or set to zero (elements with a value of 0).

[0193] That is, for each pronunciation label in the set of pronunciation labels set in the pronunciation prediction network, if the pronunciation represented by the pronunciation label belongs to one of the pronunciations of the polyphonic character to be measured, the element value at the corresponding element position in the mask matrix is set to 1. Conversely, if the pronunciation represented by the pronunciation label does not belong to one of the pronunciations of the polyphonic character to be measured, the element value at the corresponding element position in the mask matrix is set to 0.

[0194] For example, if the set of pronunciation labels includes 100 pronunciation labels, the 51st pronunciation label represents the pronunciation "hang2", and the 77th pronunciation label represents the pronunciation "xing2", then for the polyphonic character "行" to be measured, the length of this mask matrix is 100, including 100 elements. Except that the 51st and 77th elements have a value of 1, the element values at other positions are 0.

[0195] S406. Multiply the mask matrix and the conditional weight matrix element by element to obtain an updated conditional weight matrix.

[0196] The mask matrix is ​​used to ensure that only the conditional weights corresponding to the candidate pronunciation labels of the polyphone to be tested are retained, and the conditional weights of non-candidate pronunciation labels will be set to 0, so that the updated conditional weight matrix is ​​used to dynamically adjust the prediction probability of the candidate pronunciation labels of the polyphone to be tested according to the context information during the pronunciation prediction process. The mask matrix and the conditional weight matrix are multiplied element by element, which can be implemented through a matrix operation library such as NumPy, TensorFlow or PyTorch, where the elements of each corresponding position are multiplied to obtain a new matrix as the updated conditional weight matrix.

[0197] In the embodiment of the present disclosure, a mask matrix is ​​introduced so that the conditional weights corresponding to the candidate pronunciation labels of the polyphone to be tested in the conditional weight matrix are retained, and the conditional weights of the non-candidate pronunciation labels are set to 0, which effectively reduces the interference of other non-candidate pronunciations on the prediction results, allowing the network to focus more on the prediction of candidate pronunciations, thereby improving the accuracy and reliability of the prediction.

[0198] For the part of speech of the target segmentation containing the polyphone to be tested in the initial segmentation sequence described in the aforementioned embodiment, it can be determined by generating the initial segmentation sequence, and the target segmentation is tagged with the part of speech by the part of speech tagging method of the relevant technology. However, in order to further improve the accuracy and efficiency of part of speech tagging, especially when processing the target segmentation containing polyphones, this embodiment proposes a method of obtaining the part of speech of the target segmentation by performing part of speech prediction based on the part of speech prediction subnetwork set in the pronunciation prediction network. Specifically, the pronunciation prediction network in this embodiment not only includes a part for predicting the pronunciation of polyphones, but also additionally sets a part of speech prediction subnetwork, which can use context information related to the target segmentation such as contextual vocabulary, grammatical structure, etc. to predict the part of speech of the target segmentation. The specific implementation of the part of speech prediction subnetwork can be a multi-layer perceptron, a convolutional neural network, a recurrent neural network or its variants such as a long short-term memory network, a gated recurrent unit, etc., or a combination of these networks. The part of speech prediction subnetwork receives the context embedding information of the polyphone to be predicted and the target segmentation, and finally outputs the part of speech prediction result of the target segmentation through a series of calculations and transformations.

[0199] In this embodiment, a part-of-speech prediction subnetwork is provided in the pronunciation prediction network, and the part-of-speech prediction subnetwork is trained along with the pronunciation prediction network. The classification category of the part-of-speech prediction subnetwork is part-of-speech classification, and a part-of-speech tag set is pre-set in the subnetwork, including multiple part-of-speech tags representing different parts of speech. For example, the part-of-speech tags may represent the part-of-speech including but not limited to unknown tags, nouns, verbs, adjectives, quantifiers, pronouns, adverbs, prepositions, conjunctions, auxiliary words, interjections, onomatopoeia, etc. Based on the part-of-speech prediction subnetwork, the method may also include the following steps of obtaining the part-of-speech of the target word segmentation:

[0200] Mapping the context embedding of the polyphone to be tested into a part-of-speech score vector through the first mapping parameter in the part-of-speech prediction subnetwork; the part-of-speech score vector includes an unnormalized score of each part-of-speech tag in the part-of-speech tag set;

[0201] By using an activation function for part-of-speech classification, a predicted probability distribution for the part-of-speech tag set is generated according to the part-of-speech score vector, and the predicted part-of-speech of the target word is determined as the part-of-speech of the target word.

[0202] The first mapping parameter refers to the weights and biases learned by the part-of-speech prediction subnetwork during training, which are used to map the input to the part-of-speech score vector. Each element in the part-of-speech score vector corresponds to an unnormalized score (i.e., logit value) of a part-of-speech tag, which indicates the probability that the target word belongs to the part-of-speech, but has not been normalized, so it cannot be directly used as a probability.

[0203] The activation function is used to introduce nonlinear factors so that the part-of-speech prediction subnetwork can learn complex mapping relationships. In the part-of-speech classification task of this embodiment, the softmax function can be used as the activation function to convert the part-of-speech score vector into a predicted probability distribution. In the predicted probability distribution, each part-of-speech tag corresponds to a probability value, indicating the possibility that the target word belongs to this part of speech.

[0204] Based on the predicted probability distribution, the part-of-speech tag with the largest probability value can be directly selected as the predicted part-of-speech of the target word segmentation. The tag represents the most likely part-of-speech of the target word segmentation determined by the part-of-speech prediction subnetwork in a given context of the processed text.

[0205] In the disclosed embodiment, the part of speech prediction subnetwork is used to accurately predict the part of speech of the polyphone to be tested, thereby providing useful information for subsequent polyphone pronunciation prediction tasks. In addition, since the part of speech prediction subnetwork shares some input features or context information with the pronunciation prediction network, information sharing and collaborative optimization between the two can be achieved to a certain extent.

[0206] In some embodiments, the aforementioned step S104 of generating a predicted probability distribution for the pronunciation label set according to the context embedding by applying the output activation function of the conditional weight matrix can be specifically implemented by the following steps a1-a3:

[0207] a1, mapping the context embedding into a pronunciation score vector according to the second mapping parameter learned in the training phase; the pronunciation score vector includes an unnormalized score of each pronunciation tag in the pronunciation tag set;

[0208] The second mapping parameter refers to the parameters such as weights and biases used in the pronunciation prediction network to map the context embedding of the polyphone to be tested, or the encoding processing features generated by the context embedding after encoding processing, to the output such as the pronunciation score vector. It is understandable that the "second" has no specific indication meaning and is only used to distinguish it from other mapping parameters of the same type in this application.

[0209] The pronunciation score vector is a vector output by the pronunciation prediction network, in which each element corresponds to the unnormalized score of a pronunciation label. The score represents the probability that the context of the polyphone to be tested corresponds to the pronunciation label, but it has not been normalized, so it cannot be directly used as a probability. The unnormalized score of the pronunciation label is the original, unnormalized prediction value output by the pronunciation prediction network, corresponding to the linear prediction result of each pronunciation label, and then converted to a probability value through the normalization of the output activation function (such as the softmax function). For example, in a three-classification problem (assuming the categories are cat, dog, and bird), the network output layer may have three neurons, corresponding to the logits values ​​of the three categories. For the i-th label is "dog", the unnormalized score of the i-th label is the output value of the neuron corresponding to the "dog" category in the network output layer.

[0210] b2, for each pronunciation label, obtaining the conditional weight in the conditional weight matrix that is in the same position as the unnormalized score of the pronunciation label;

[0211] Since the length of the conditional weight matrix is ​​the same as the number of pronunciation tags included in the pronunciation tag set, for each element in the pronunciation score vector, i.e., the unnormalized score of each pronunciation tag, a conditional weight at the same position can be found in the conditional weight matrix. The conditional weight reflects the relative importance of each pronunciation tag under the context of the current processed text.

[0212] b3, performing normalization processing through the output activation function according to the unnormalized score of the pronunciation label and the conditional weight, to obtain the probability value of the pronunciation label.

[0213] In this embodiment, the normalization processing through the output activation function can be implemented in the following manner: first, determine the exponential operation result of the unnormalized score of the pronunciation label, for example, take the natural constant e as the base, and take the unnormalized score logit of the pronunciation label as the exponent, and obtain the exponential operation result corresponding to the pronunciation label, which can be expressed as exp{logit}, that is, e to the logit power; next, obtain the product of the exponential operation result and the conditional weight of the pronunciation label as the target operation result, assuming that the conditional weight of the pronunciation label is Wi, then the target operation result can be expressed as Pi=Wi*exp{logit}; then, according to the sum of the target operation results of all pronunciation labels, the target operation result of each pronunciation label is normalized, that is, first calculate the sum of the target operation results of all pronunciation labels, which can be expressed as ∑ j N ＝1 W j *exp{(logit) j}, N represents the number of pronunciation labels in the pronunciation label set, (logit) j represents the unnormalized score of the jth pronunciation label, and then calculates the ratio of the target operation result of each pronunciation label to the sum of the target operation results of all pronunciation labels, and takes it as the normalized result of the pronunciation label, that is, the normalized result of the i-th pronunciation label can be expressed as The normalized result is used as the probability value of the pronunciation label.

[0214] In some embodiments, the polyphone prediction method provided in the present application is implemented by a pronunciation prediction network, which is trained by obtaining the predicted pronunciations of polyphones in different word segmentation methods for the same sample sentence and minimizing the weighted results of the loss functions corresponding to the predicted pronunciations. Based on this, the pronunciation prediction network can be trained at least through the following steps b1-b3:

[0215] b1, for each sample sentence, obtain a first segmentation sequence obtained by segmenting words in units of single characters, and obtain a second segmentation sequence obtained by segmenting words in units of words; the sample sentence includes a polyphonetic character and a pronunciation tag of the polyphonetic character; the second segmentation sequence also includes a part-of-speech tag of the segmentation word containing the polyphonetic character;

[0216] During the training process described in this embodiment, the same sample sentence will be segmented in two ways: character by character and word by word. The first segmentation sequence obtained by segmenting character by character, where each character in the first segmentation sequence is regarded as an independent word, and the second segmentation sequence obtained by segmenting word by word. The second segmentation sequence also includes the true part-of-speech annotation for the segmentation containing the polyphonic character to be tested in the sample sentence, that is, the part-of-speech tag for the segmentation containing the polyphonic character.

[0217] See Figure 5 In the training process diagram shown, taking the sample sentence "他今天去银行 (He went to the bank today)" as an example, the first segmentation sequence is represented as "他 / 今 / 天 / 去 / 银 / 行 (He / today / go / to / bank)", the second segmentation sequence is represented as "他 / 今天 / 去 / 银行 (He / today / go / to the bank)", and the second segmentation sequence carries the part-of-speech tag "noun" for the segmentation "银行 (bank)".

[0218] b2, input the first segmentation sequence and the second segmentation sequence into the pronunciation prediction network together to obtain the first predicted pronunciation corresponding to the first segmentation sequence and the second predicted pronunciation corresponding to the second segmentation sequence; the conditional weight matrix is determined according to the segmentation containing the polyphonic character in the second segmentation sequence and the part-of-speech tag.

[0219] During the training process of this embodiment, the first segmentation sequence and the second segmentation sequence of the same sample sentence will be input into the pronunciation prediction network for processing at the same time. The process of obtaining the first predicted pronunciations of the first segmentation sequence and the second segmentation sequence is the same. As Figure 5 shown, taking the first segmentation sequence as an example, that is, using the input layer to obtain the context embedding of the polyphonic character to be tested according to the initial segmentation sequence, input the context embedding into the encoding layer to obtain the encoded processing feature output after being processed by the encoding layer, input the encoded processing feature into the output layer, and through the output activation function of the conditional weight matrix, obtain the predicted probability distribution based on the first segmentation sequence, and then select the pronunciation corresponding to the pronunciation label with the highest probability as the first predicted pronunciation.

[0220] Among them, the conditional weight matrix is determined according to the segmentation containing the polyphonic character in the second segmentation sequence and the part-of-speech tag. Therefore, after inputting the second segmentation sequence into the pronunciation prediction network and obtaining the context embedding output by the input layer, input the context embedding into the conditional weighting layer to use the conditional weighting layer to combine the target segmentation "银行 (bank)" containing the polyphonic character "行 (hang)" to be tested and the part-of-speech of the target segmentation "noun" to generate a conditional weight matrix through a learnable linear projection matrix.

[0221] For example, for the first word segmentation sequence represented as "he / today / goes / to / the / bank", the first predicted pronunciation output by the network is "xing2", and for the second word segmentation sequence represented as "he / today / goes / to / the / bank", the first predicted pronunciation output by the network is "hang2".

[0222] b3. Obtain a target loss function based on the first predicted pronunciation and the second predicted pronunciation, and adjust the model parameters of the pronunciation prediction network by minimizing the target loss function; the target loss includes a first loss between the first predicted pronunciation and the pronunciation mark, and a second loss between the second predicted pronunciation and the pronunciation mark.

[0223] In machine learning, the loss function is used to quantify the difference between the network's predicted results and the actual results. The process of training the network is to continuously adjust the network parameters to minimize the loss function. In this embodiment, the target loss function used in the training stage is composed of two weighted loss functions: the first part is a loss function based on the first predicted pronunciation of the first word segmentation sequence, that is, the first loss of the pronunciation mark of the polyphone to be tested in the first predicted pronunciation and the original sentence is calculated. Assuming that the first predicted pronunciation is represented by Ph1 and the pronunciation mark is represented by Ph0, the first loss can be expressed as L1=L(Ph0,Ph1); the second part is a loss function based on the second predicted pronunciation of the second word segmentation sequence, that is, the second loss of the pronunciation mark of the polyphone to be tested in the second predicted pronunciation and the original sentence is calculated. Assuming that the second predicted pronunciation is represented by Ph2, the second loss can be expressed as L2=L(Ph0,Ph2); setting the weight coefficient of the first loss to β1 and the weight coefficient of the second loss to β2, the sum of β1 and β2 is 1, then the target loss function can be expressed as L=β1*L1+β2*L2. Regarding the first loss or the second loss, any loss function suitable for pronunciation loss calculation, such as mean square error, root mean square error, cross entropy loss, etc., can be used, and this application does not limit this.

[0224] The specific processing logic of the pronunciation prediction network based on the training process is the same as the processing logic of the pronunciation prediction network in the aforementioned pronunciation prediction method, and will not be repeated in this embodiment. For the specific process, please refer to the description of the aforementioned embodiment.

[0225] In some embodiments, when a part-of-speech prediction subnetwork is set in the pronunciation prediction network to predict the part-of-speech of the target word segment in the inference phase, the loss of the part-of-speech prediction subnetwork needs to be considered during the training process. Based on the part-of-speech prediction subnetwork, the training process of the pronunciation prediction network may also include the following steps:

[0226] The context embedding of the word segment containing polyphones is determined by the pronunciation prediction network; the context embedding is input into the part-of-speech prediction subnetwork to obtain the predicted part-of-speech output by the part-of-speech prediction subnetwork; the conditional weight matrix is ​​determined based on the word segment containing polyphones and the predicted part-of-speech; the target loss function also includes a third loss of the predicted part-of-speech and the part-of-speech tag.

[0227] That is, when a part-of-speech prediction subnetwork is set up, the target loss function used in the training process will be composed of three parts, including a first loss of the first predicted pronunciation and the pronunciation mark, a second loss of the second predicted pronunciation and the pronunciation mark, and a third loss of the predicted part of speech and the part-of-speech mark in the part-of-speech prediction subnetwork. Assuming that the third loss is expressed as L3, the target loss function can be expressed as L=γ1*L1+γ2*L2+γ3*L3, where γ1, γ2, and γ3 are three hyperparameters used to control the contribution of different loss functions to the target loss function.

[0228] In the disclosed embodiment, by incorporating part-of-speech prediction into the training process, the pronunciation prediction network can learn more contextual information. Part-of-speech not only reflects the grammatical role of a word in a sentence, but is also associated with the pronunciation of the word. Using part-of-speech prediction can help the model more accurately predict the pronunciation of polyphones, thereby improving the overall prediction accuracy. At the same time, the part-of-speech prediction subnetwork provides an additional supervisory signal, which together with the loss of pronunciation prediction guides the parameter update of the pronunciation prediction network, helping the pronunciation prediction network learn a more robust and comprehensive feature representation.

[0229] Corresponding to the above-mentioned embodiment of the polyphone prediction method, see Figure 6 As shown, the present application also provides an embodiment of a polyphone prediction device, which is applied to a pre-trained pronunciation prediction network, wherein the pronunciation prediction network is provided with a pronunciation label set corresponding to all pronunciations of each polyphone; the pronunciation prediction network is trained by obtaining the predicted pronunciations of the polyphones in different word segmentation modes for the same sample sentence, and minimizing the weighted result of the loss function corresponding to each predicted pronunciation; the device comprises:

[0230] The word segmentation processing module 601 is used to obtain an initial word segmentation sequence obtained by word segmentation in units of words for the processed text containing the polyphone to be tested;

[0231] A context acquisition module 602, configured to acquire a context embedding of the polyphonetic character to be tested according to the initial word segmentation sequence;

[0232] The conditional weight matrix generation module 603 is used to perform feature linear projection on at least one of the target segmentation containing the polyphone to be tested, the part of speech of the target segmentation, and the cross feature of the target segmentation and the part of speech in the initial segmentation sequence using a linear projection matrix to generate a conditional weight matrix; the length of the conditional weight matrix is ​​the same as the length of the pronunciation label set;

[0233] The prediction probability and pronunciation determination module 604 is used to generate a prediction probability distribution for the pronunciation label set according to the context embedding by applying the output activation function of the conditional weight matrix, and determine the pronunciation of the polyphone to be predicted.

[0234] In some embodiments, the word segmentation processing module is specifically used to:

[0235] Using a word-level segmentation tool to perform segmentation processing on the processed text to obtain a first segmentation sequence;

[0236] For each first participle in the first participle sequence, when the first participle fails to match the preset vocabulary, the first participle is updated to a second participle; the second participle is obtained by performing participle processing on the first participle using the participle function set in the pronunciation prediction network;

[0237] The first word segmentation sequence after the update is determined as the initial word segmentation sequence.

[0238] In some embodiments, the context acquisition module is specifically used to:

[0239] Encode each word in the initial word segmentation sequence to obtain an initial embedding vector corresponding to each word segmentation, including a word embedding vector and a position embedding vector; the word embedding vector is used to represent the semantic information of the word segmentation itself; the position embedding vector is used to represent the position information of the word segmentation itself;

[0240] According to the initial embedding vector of each word segment, the context information is fused to the initial embedding vector of the target word segment through an attention mechanism to generate a context embedding of the polyphone to be tested.

[0241] In some embodiments, the conditional weight matrix generation module is specifically used to:

[0242] Obtaining a character encoding vector of the target word segment and a part-of-speech encoding vector of the part-of-speech;

[0243] Extracting a cross feature vector of the character encoding vector and the part-of-speech encoding vector, and obtaining a first projection result of the cross feature by using a first projection matrix in the linear projection matrix;

[0244] Using the second projection matrix in the linear projection matrix, a second projection result of the character encoding vector is obtained; using the second projection matrix in the linear projection matrix, a third projection result of the part-of-speech encoding vector is obtained;

[0245] The conditional weight matrix is ​​generated according to the first projection result, the second projection result and the third projection result.

[0246] In some embodiments, the conditional weight matrix generation module, when used to extract the cross-features of the character encoding vector and the part-of-speech encoding vector, includes at least any of the following methods:

[0247] Obtaining a tensor product of the character encoding vector and the part-of-speech encoding vector as the cross feature;

[0248] Alternatively, an element-by-element addition result of the character encoding vector and the part-of-speech encoding vector is obtained as the cross feature.

[0249] In some embodiments, the conditional weight matrix generation module, when used to generate the conditional weight matrix according to the first projection result, the second projection result and the third projection result, includes:

[0250] According to the three set hyperparameters, weighted processing is performed on the first projection result, the second projection result and the third projection result respectively to obtain three corresponding weighted projection results;

[0251] Performing element addition processing on the three weighted projection results to generate an initial weight matrix;

[0252] Each element in the initial weight matrix is ​​converted into a probability value to obtain the conditional weight matrix; the numerical range of the probability value is 0 to 1.

[0253] In some embodiments, after generating the conditional weight matrix, the apparatus further comprises:

[0254] Determining candidate pronunciation labels of the polyphonetic character to be predicted; the candidate pronunciation labels represent all pronunciations of the polyphonetic character to be predicted;

[0255] For the pronunciation label set, setting mask 0 for the other pronunciation labels except the candidate pronunciation label, and setting mask 1 for the candidate pronunciation label, to obtain a mask matrix;

[0256] The mask matrix is ​​multiplied element by element by the conditional weight matrix to obtain an updated conditional weight matrix.

[0257] In some embodiments, the pronunciation prediction network includes a part-of-speech prediction subnetwork, which is provided with a part-of-speech tag set; the device further includes:

[0258] Mapping the context embedding of the polyphone to be tested into a part-of-speech score vector through the first mapping parameter in the part-of-speech prediction subnetwork; the part-of-speech score vector includes an unnormalized score of each part-of-speech tag in the part-of-speech tag set;

[0259] By using an activation function for part-of-speech classification, a predicted probability distribution for the part-of-speech tag set is generated according to the part-of-speech score vector, and the predicted part-of-speech of the target word is determined as the part-of-speech of the target word.

[0260] In some embodiments, the prediction probability and pronunciation determination module is specifically used to:

[0261] According to the second mapping parameter learned in the training phase, the context embedding is mapped into a pronunciation score vector; the pronunciation score vector includes an unnormalized score of each pronunciation tag in the pronunciation tag set;

[0262] For each pronunciation tag, obtaining a conditional weight in the conditional weight matrix that is in the same position as the unnormalized score of the pronunciation tag;

[0263] According to the unnormalized score of the pronunciation label and the conditional weight, normalization processing is performed through the output activation function to obtain the probability value of the pronunciation label.

[0264] In some embodiments, the prediction probability and pronunciation determination module, when used to perform normalization processing through the output activation function according to the unnormalized score of the pronunciation label and the conditional weight, includes:

[0265] Determining an exponential operation result of an unnormalized score of the pronunciation label;

[0266] Obtaining the product of the index operation result and the conditional weight of the pronunciation label as the target operation result;

[0267] The target operation result of each pronunciation label is normalized according to the sum of the target operation results of all pronunciation labels.

[0268] In some embodiments, the apparatus further comprises a step of training a pronunciation prediction network:

[0269] For each sample sentence, a first segmentation sequence is obtained by segmenting words in single-character units, and a second segmentation sequence is obtained by segmenting words in words; the sample sentence includes a polyphonetic character and a pronunciation tag of the polyphonetic character; the second segmentation sequence also includes a part-of-speech tag of the polyphonetic character;

[0270] Inputting the first segmentation sequence and the second segmentation sequence into the pronunciation prediction network together, obtaining a first predicted pronunciation corresponding to the first segmentation sequence and a second predicted pronunciation corresponding to the second segmentation sequence; the conditional weight matrix is ​​determined according to the segmentations containing polyphones in the second segmentation sequence and the part-of-speech tags;

[0271] According to the first predicted pronunciation and the second predicted pronunciation, a target loss function is obtained, and the model parameters of the pronunciation prediction network are adjusted by minimizing the target loss function; the target loss includes a first loss between the first predicted pronunciation and the pronunciation mark, and a second loss between the second predicted pronunciation and the pronunciation mark.

[0272] In some embodiments, the pronunciation prediction network includes a part-of-speech prediction subnetwork, and the device further includes: determining the context embedding of the word segment containing the polyphonetic character through the pronunciation prediction network;

[0273] The context embedding is input into the part-of-speech prediction subnetwork to obtain the predicted part-of-speech output by the part-of-speech prediction subnetwork; the conditional weight matrix is ​​determined based on the word segmentation containing polyphones and the predicted part-of-speech; the target loss function also includes a third loss of the predicted part-of-speech and the part-of-speech tag.

[0274] The implementation process of the functions and effects of each unit in the above-mentioned device is specifically described in the implementation process of the corresponding steps in the above-mentioned method, and will not be repeated here.

[0275] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the present application. Ordinary technicians in this field can understand and implement it without creative work.

[0276] The present application also provides an electronic device. The structural diagram of the electronic device is as follows: Figure 7 As shown, the electronic device 700 includes at least one processor 701, a memory 702 and a bus 703, and the at least one processor 701 is electrically connected to the memory 702; the memory 702 is configured to store at least one computer-executable instruction, and the processor 701 is configured to execute the at least one computer-executable instruction, thereby executing the steps of any polyphonetic character prediction method provided in any embodiment or any optional implementation manner in the present application.

[0277] Furthermore, the processor 701 may be a Field-Programmable Gate Array (FPGA) or other devices with logic processing capabilities, such as a Microcontroller Unit (MCU) or a Central Process Unit (CPU).

[0278] The embodiment of the present application also provides another readable storage medium storing a computer program, which is used to implement the steps of any polyphonetic character prediction method provided by any embodiment or any optional implementation manner in the present application when executed by a processor.

[0279] The readable storage medium provided in the embodiments of the present application includes, but is not limited to, any type of disk (including floppy disk, hard disk, optical disk, CD-ROM, and magneto-optical disk), ROM (Read-Only Memory), RAM (Random Access Memory), EPROM (Erasable Programmable Read-Only Memory), EEPROM (Electrically Erasable Programmable Read-Only Memory), flash memory, magnetic card or optical card. That is, the readable storage medium includes any medium that can store or transmit information in a readable form by a device (e.g., a computer).

[0280] Thus, specific embodiments of the subject matter have been described. Other embodiments are within the scope of the appended claims. In some cases, the actions recited in the claims can be performed in a different order and still achieve the desired results. In addition, the processes depicted in the drawings do not necessarily require the particular order or sequential order shown to achieve the desired results. In some implementations, multitasking and parallel processing may be advantageous.

[0281] Although this specification includes many specific implementation details, these should not be interpreted as limiting the scope of any invention or the scope of protection claimed, but are mainly used to describe the features of the specific embodiments of specific inventions. Certain features described in multiple embodiments in this specification may also be implemented in combination in a single embodiment. On the other hand, the various features described in a single embodiment may also be implemented separately in multiple embodiments or in any suitable sub-combination. In addition, although features may work in certain combinations as described above and even initially claim protection, one or more features from the claimed combination may be removed from the combination in some cases, and the claimed combination may point to a sub-combination or a variation of a sub-combination.

[0282] The above description is only a preferred embodiment of the present application and is not intended to limit the present application. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present application shall be included in the scope of protection of the present application.

Claims

1. A polyphone prediction method, characterized in that: The method is applied to a pre-trained pronunciation prediction network, wherein the pronunciation prediction network is provided with a pronunciation label set corresponding to all pronunciations of each polyphonetic character; the method comprises: For the processed text containing the polyphone to be tested, obtaining an initial word segmentation sequence obtained by segmenting words as units; According to the initial word segmentation sequence, obtaining the context embedding of the polyphonetic character to be tested; Using a linear projection matrix, feature linear projection is performed on at least one of the target segmentation containing the polyphone to be tested, the part of speech of the target segmentation, and the cross feature of the target segmentation and the part of speech in the initial segmentation sequence to generate a conditional weight matrix; the length of the conditional weight matrix is ​​the same as the length of the pronunciation label set; By applying the output activation function of the conditional weight matrix, generating a predicted probability distribution for the pronunciation label set according to the context embedding, and determining the pronunciation of the polyphonetic character to be predicted; The pronunciation prediction network is trained by obtaining the predicted pronunciations of polyphones in the same sample sentence under different word segmentation methods and minimizing the weighted results of the loss functions corresponding to the predicted pronunciations.

2. The method according to claim 1, characterized in that The step of obtaining an initial word segmentation sequence obtained by segmenting words as units includes: Using a word-level segmentation tool to perform segmentation processing on the processed text to obtain a first segmentation sequence; For each first participle in the first participle sequence, when the first participle fails to match the preset vocabulary, the first participle is updated to a second participle; the second participle is obtained by performing participle processing on the first participle using the participle function set in the pronunciation prediction network; The first word segmentation sequence after the update is determined as the initial word segmentation sequence.

3. The method according to claim 1, characterized in that The step of obtaining the context embedding of the polyphonetic character to be tested according to the initial word segmentation sequence includes: Encode each word in the initial word segmentation sequence to obtain an initial embedding vector corresponding to each word segmentation, including a word embedding vector and a position embedding vector; the word embedding vector is used to represent the semantic information of the word segmentation itself; the position embedding vector is used to represent the position information of the word segmentation itself; According to the initial embedding vector of each word segment, the context information is fused to the initial embedding vector of the target word segment through an attention mechanism to generate a context embedding of the polyphone to be tested.

4. The method according to claim 1, characterized in that: The generating conditional weight matrix comprises: Obtaining a character encoding vector of the target word segment and a part-of-speech encoding vector of the part-of-speech; Extracting a cross feature vector of the character encoding vector and the part-of-speech encoding vector, and obtaining a first projection result of the cross feature by using a first projection matrix in the linear projection matrix; Using a second projection matrix in the linear projection matrix, obtaining a second projection result of the character code vector; Using the second projection matrix in the linear projection matrix, obtaining a third projection result of the part-of-speech encoding vector; The conditional weight matrix is ​​generated according to the first projection result, the second projection result and the third projection result.

5. The method according to claim 4, characterized in that The extracting of the cross-features of the character encoding vector and the part-of-speech encoding vector includes at least any of the following: Obtaining a tensor product of the character encoding vector and the part-of-speech encoding vector as the cross feature; Alternatively, an element-by-element addition result of the character encoding vector and the part-of-speech encoding vector is obtained as the cross feature.

6. The method according to claim 4, characterized in that The generating the conditional weight matrix according to the first projection result, the second projection result and the third projection result includes: According to the three set hyperparameters, weighted processing is performed on the first projection result, the second projection result and the third projection result respectively to obtain three corresponding weighted projection results; Performing element addition processing on the three weighted projection results to generate an initial weight matrix; Each element in the initial weight matrix is ​​converted into a probability value to obtain the conditional weight matrix; the numerical range of the probability value is 0 to 1.

7. The method according to claim 4, characterized in that After generating the conditional weight matrix, the method further includes: Determining candidate pronunciation labels of the polyphonetic character to be predicted; the candidate pronunciation labels represent all pronunciations of the polyphonetic character to be predicted; For the pronunciation label set, setting mask 0 for the other pronunciation labels except the candidate pronunciation label, and setting mask 1 for the candidate pronunciation label, to obtain a mask matrix; The mask matrix is ​​multiplied element by element by the conditional weight matrix to obtain an updated conditional weight matrix.

8. The method according to claim 1, characterized in that The pronunciation prediction network includes a part-of-speech prediction subnetwork, which is provided with a part-of-speech tag set; the method further includes: Mapping the context embedding of the polyphone to be tested into a part-of-speech score vector through the first mapping parameter in the part-of-speech prediction subnetwork; the part-of-speech score vector includes an unnormalized score of each part-of-speech tag in the part-of-speech tag set; By using an activation function for part-of-speech classification, a predicted probability distribution for the part-of-speech tag set is generated according to the part-of-speech score vector, and the predicted part-of-speech of the target word is determined as the part-of-speech of the target word.

9. The method according to claim 1, characterized in that: The step of generating a predicted probability distribution for the pronunciation label set according to the context embedding by applying the output activation function of the conditional weight matrix comprises: According to the second mapping parameter learned in the training phase, the context embedding is mapped into a pronunciation score vector; the pronunciation score vector includes an unnormalized score of each pronunciation tag in the pronunciation tag set; For each pronunciation tag, obtaining a conditional weight in the conditional weight matrix that is in the same position as the unnormalized score of the pronunciation tag; According to the unnormalized score of the pronunciation label and the conditional weight, normalization processing is performed through the output activation function to obtain the probability value of the pronunciation label.

10. The method according to claim 9, characterized in that The step of performing normalization processing according to the unnormalized score of the pronunciation label and the conditional weight through the output activation function includes: Determining an exponential operation result of an unnormalized score of the pronunciation label; Obtaining the product of the index operation result and the conditional weight of the pronunciation label as the target operation result; The target operation result of each pronunciation label is normalized according to the sum of the target operation results of all pronunciation labels.

11. The method according to claim 1, characterized in that The method also includes a training step of a pronunciation prediction network: For each sample sentence, a first segmentation sequence is obtained by segmenting words in single-character units, and a second segmentation sequence is obtained by segmenting words in words; the sample sentence includes a polyphonetic character and a pronunciation tag of the polyphonetic character; the second segmentation sequence also includes a part-of-speech tag of the polyphonetic character; Inputting the first segmentation sequence and the second segmentation sequence into the pronunciation prediction network together, obtaining a first predicted pronunciation corresponding to the first segmentation sequence and a second predicted pronunciation corresponding to the second segmentation sequence; the conditional weight matrix is ​​determined according to the segmentations containing polyphones in the second segmentation sequence and the part-of-speech tags; According to the first predicted pronunciation and the second predicted pronunciation, a target loss function is obtained, and the model parameters of the pronunciation prediction network are adjusted by minimizing the target loss function; the target loss includes a first loss between the first predicted pronunciation and the pronunciation mark, and a second loss between the second predicted pronunciation and the pronunciation mark.

12. The method according to claim 11, characterized in that The pronunciation prediction network includes a part-of-speech prediction subnetwork, and the method further includes: Determining the context embedding of the word containing the polyphonetic character by the pronunciation prediction network; Inputting the context embedding into the part-of-speech prediction subnetwork to obtain the predicted part-of-speech output by the part-of-speech prediction subnetwork; The conditional weight matrix is ​​determined according to the segmentation of the polyphonetic character and the predicted part of speech; The target loss function also includes a third loss of the predicted part of speech and the part of speech tag.

13. A polyphonetic character prediction device, characterized in that: The device is applied to a pre-trained pronunciation prediction network, wherein the pronunciation prediction network is provided with a pronunciation label set corresponding to all pronunciations of each polyphonetic character; the device comprises: A word segmentation processing module is used to obtain an initial word segmentation sequence obtained by segmenting words as units for a processing text containing a polyphone to be tested; A context acquisition module, used for acquiring the context embedding of the polyphonetic character to be tested according to the initial word segmentation sequence; A conditional weight matrix generation module is used to use a linear projection matrix to perform feature linear projection on at least one of the target segmentation containing the polyphone to be tested, the part of speech of the target segmentation, and the cross feature of the target segmentation and the part of speech in the initial segmentation sequence to generate a conditional weight matrix; the length of the conditional weight matrix is ​​the same as the length of the pronunciation label set; A prediction probability and pronunciation determination module, for generating a prediction probability distribution for the pronunciation label set according to the context embedding by applying the output activation function of the conditional weight matrix, and determining the pronunciation of the polyphone to be predicted; The pronunciation prediction network is trained by obtaining the predicted pronunciations of polyphones in the same sample sentence under different word segmentation methods and minimizing the weighted results of the loss functions corresponding to the predicted pronunciations.

14. An electronic device, characterized in that: include: Memory, processor; The memory is used to store computer programs; The processor is used to call the computer program to implement the method according to any one of claims 1-12.

15. A readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method according to any one of claims 1 to 12 is implemented.

Images