Chinese polyphone prediction and identification method and device, equipment and storage medium
By constructing a distribution balanced multiphonic word dataset and context enhancement method, combined with the BERT sequence labeling model, the problem of insufficient prediction capabilities of the TTS front-end in specific scenarios is solved, and the generalization ability and prediction accuracy of the model are improved.
Patent Information
- Application Number
- CN202411235408.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-09-04
- Publication Date
- 2024-12-10
AI Technical Summary
Currently, the TTS front-end has weak predictive capabilities for professional terms and names in specific business scenarios, and models trained based on the CPP dataset are difficult to predict the pronunciation of fewer training samples, resulting in a reduced generalization capability of the model.
A preset Chinese polyphonic word data set with distribution balance is constructed, containing multiphonic word sampling data from multiple data sources, and the embedded vector sequence is processed through context enhancement method, and the multiphonic word prediction is used using the BERT sequence labeling model to ensure that each polyphonic word is There is sufficient data volume and diversity.
The model's prediction ability of professional terms and names has been improved, the ability to predict oral tone has been enhanced, the model's generalization ability has been improved, and the problem of insufficient data source and data volume has been solved.
Smart Images
Figure CN119107935A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of speech recognition technology, and in particular to a method, device, equipment and storage medium for predicting and recognizing Chinese polyphones. Background Art
[0002] The TTS (Text-to-Speech) front end refers to the front end part of the technology that converts text to speech, that is, the text input system. The main goal of this technology is to generate correct and natural text phonemes, so that the TTS system can synthesize natural and fluent speech content, allowing computers to interact with users through voice. In applications such as intelligent voice customer service and digital humans, TTS is an important technology that enables customers to interact with applications. A TTS front end that can generate accurate phonemes is the primary prerequisite for users to feel smooth and natural during the interaction process, helping to improve the user experience.
[0003] A major difficulty faced by the current TTS front-end in the pinyin prediction task is the disambiguation of polyphones. Data-driven deep learning methods are an effective way to solve this difficulty. However, there are very few open source Chinese pinyin datasets, and most models are trained based on the CPP dataset.
[0004] However, the CPP dataset has many shortcomings: 1. The data source is single, and all the data comes from Wikipedia; 2. The sampling is completely based on the data distribution of the data source, resulting in the pronunciation of many polyphones in the CPP dataset being highly scarce; 3. The data corresponding to each word is very small, no more than 200. This will make it difficult for the model trained based on the CPP dataset to predict the pronunciation of those with fewer training samples. At the same time, the single data source and small amount of data will reduce the generalization ability of the model to a considerable extent.
[0005] At the same time, the current TTS front-end has weak prediction capabilities for some professional terms and names in specific business scenarios, and it is also difficult to predict some common intonation changes in spoken language (such as the three-three intonation change). Summary of the invention
[0006] The present application provides a method, apparatus, device and storage medium for predicting and recognizing Chinese polyphones, which solves the technical problems that the current TTS front-end has weak prediction ability for some professional terms and names in specific business scenarios, and it is also difficult to predict some common tone changes in spoken language. In addition, the model trained based on the CPP data set is difficult to predict the pronunciations with fewer training samples. At the same time, a single data source and a small amount of data will reduce the generalization ability of the model to a considerable extent.
[0007] In view of this, the first aspect of the present application provides a method for predicting and recognizing Chinese polyphones, the method comprising:
[0008] Get the text content to be recognized;
[0009] Preprocessing the text content by using a preset Chinese polyphone data set to convert the text content into an embedding vector sequence recognizable by a model;
[0010] Processing the embedded vector sequence based on a context enhancement method to obtain a fused feature vector sequence;
[0011] The fused feature vector sequence is used as the input of a pre-trained BERT-based sequence annotation model, and classified output is performed to obtain a pinyin sequence corresponding to the predicted polyphonetic character.
[0012] Optionally, before obtaining the text content to be recognized, the method further includes:
[0013] A preset Chinese polyphonetic character data set is constructed, wherein the preset Chinese polyphonetic character data set includes polyphonetic character sampling data collected from a plurality of data sources, and the number of each polyphonetic character sampling data in the preset Chinese polyphonetic character data set satisfies a uniform distribution.
[0014] Optionally, preprocessing the text content by using a preset Chinese polyphone data set to convert the text content into an embedding vector sequence recognizable by a model specifically includes:
[0015] Matching and replacing the text content based on preset regular expression rules to obtain preprocessed text content;
[0016] Performing word segmentation processing on the preprocessed text content to obtain a vocabulary consisting of word segmentation results of the preprocessed text content;
[0017] Performing part-of-speech tagging on the word segmentation results contained in the vocabulary using a preset tagging tool;
[0018] The word segmentation results and corresponding part-of-speech standards contained in the vocabulary are converted into an embedding vector group through a pre-trained word embedding model, and an embedding vector sequence is formed according to the order of the word segmentation results in the pre-processed text content.
[0019] Optionally, the processing of the embedded vector sequence based on the context enhancement method to obtain a fused feature vector sequence specifically includes:
[0020] Using a context enhancement method, the embedding vector group of each word segment in the embedding vector sequence is fused with the embedding vector groups of the previous and next word segments to generate a fused feature vector sequence.
[0021] Optionally, the context enhancement method is specifically:
[0022] Create a collection window for the embedded vector sequence, configure the offset ω and step size s of the collection window, move the embedded vector sequence along the sequence direction, and fill it with a 0 vector when the movement exceeds the limit. The moving range is
[0023] The fused feature vector sequence of the embedded vector sequence is obtained based on a preset formula group, and the preset formula group is:
[0024]
[0025]
[0026] V context =GELU(E win )+E emb ;
[0027] in, Refers to the set of all translation embedding vector sequences within the window; A set of left-translated embedding vector sequences; A set of embedding vector sequences pointing to the right; refers to the original untranslated embedding vector sequence; E win is the matrix obtained by vertically concatenating all embedded vector sequences in the window; V context It is processed by GELU activation function and combined with E emb The fused feature vector sequence after residual connection; T is the sequence length; D is the dimension of the embedded vector, E emb ∈R T×D .
[0028] A second aspect of the present application provides a Chinese polyphonetic character prediction and recognition device, the device comprising:
[0029] An acquisition unit, used for acquiring text content to be recognized;
[0030] An embedding operation unit, used to pre-process the text content by using a preset Chinese polyphone data set, and convert the text content into an embedding vector sequence recognizable by a model;
[0031] A context enhancement unit, used for processing the embedding vector sequence based on a context enhancement method to obtain a fused feature vector sequence;
[0032] The prediction unit is used to use the fused feature vector sequence as the input of the pre-trained BERT-based sequence annotation model, perform classification output, and obtain the pinyin sequence corresponding to the predicted polyphone.
[0033] Optionally, it also includes:
[0034] The construction unit is used to construct a preset Chinese polyphone data set, wherein the preset Chinese polyphone data set includes polyphone sampling data collected from a plurality of data sources, and the number of each polyphone sampling data in the preset Chinese polyphone data set satisfies a uniform distribution.
[0035] Optionally, the embedded operation unit is specifically used for:
[0036] Matching and replacing the text content based on preset regular expression rules to obtain preprocessed text content;
[0037] Performing word segmentation processing on the preprocessed text content to obtain a vocabulary consisting of word segmentation results of the preprocessed text content;
[0038] Performing part-of-speech tagging on the word segmentation results contained in the vocabulary using a preset tagging tool;
[0039] The word segmentation results and corresponding part-of-speech standards contained in the vocabulary are converted into an embedding vector group through a pre-trained word embedding model, and an embedding vector sequence is formed according to the order of the word segmentation results in the pre-processed text content.
[0040] Optionally, the context enhancement unit is specifically used for:
[0041] Using a context enhancement method, the embedding vector group of each word segment in the embedding vector sequence is fused with the embedding vector groups of the previous and next word segments to generate a fused feature vector sequence.
[0042] Optionally, the shift fusion method is specifically:
[0043] Create a collection window for the embedded vector sequence, configure the offset ω and step size s of the collection window, move the embedded vector sequence along the sequence direction, and fill it with a 0 vector when the movement exceeds the limit. The moving range is
[0044] The fused feature vector sequence of the embedded vector sequence is obtained based on a preset formula group, and the preset formula group is:
[0045]
[0046]
[0047] V context =GELU(E win )+E emb ;
[0048] in, Refers to the set of all translation embedding vector sequences within the window; A set of left-translated embedding vector sequences; A set of embedding vector sequences pointing to the right; refers to the original untranslated embedding vector sequence; E win is the matrix obtained by vertically concatenating all embedded vector sequences in the window; V context It is processed by GELU activation function and combined with E emb The fused feature vector sequence after residual connection; T is the sequence length; D is the dimension of the embedded vector, E emb ∈R T×D .
[0049] Optionally, a mask based on word granularity is provided in the classifier at the end of the BERT-based sequence labeling model.
[0050] A third aspect of the present application provides a Chinese polyphonetic character prediction and recognition device, the device comprising a processor and a memory:
[0051] The memory is used to store program codes and transmit the program codes to the processor;
[0052] The processor is used to execute the steps of the method for predicting and recognizing Chinese polyphonetic characters as described in the first aspect according to the instructions in the program code.
[0053] A fourth aspect of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium is used to store program code, and the program code is used to execute the method described in the first aspect above.
[0054] It can be seen from the above technical solutions that the embodiments of the present application have the following advantages:
[0055] In the present application, a method, apparatus, device and storage medium for predicting and recognizing Chinese polyphones are provided, a preset Chinese polyphone data set with balanced distribution is established, and on the basis of ensuring that each polyphone has the same sufficient amount of data, the data set is designed according to the actual distribution of the pronunciation of the characters, so that the model trained based on the enhanced data set can better fit the actual situation to predict the pronunciation of the polyphones, and at the same time integrate the semantic and structural information of the sentence context to help the model better learn the influence of words and context on the pronunciation of the characters, thereby solving the problems that the current TTS front-end has weak prediction ability for some professional terms and names in specific business scenarios, and it is also difficult to predict some common tone changes in spoken language, and the model trained based on the CPP data set is difficult to predict the pronunciation of those with fewer training samples, and the single data source and small amount of data will reduce the generalization ability of the model to a considerable extent. BRIEF DESCRIPTION OF THE DRAWINGS
[0056] Figure 1A method flow chart of the Chinese polyphone character prediction and recognition method in the embodiment of the present application;
[0057] Figure 2 This is a schematic diagram of the structure of the Chinese polyphonetic character prediction and recognition device in the embodiment of the present application;
[0058] Figure 3 Schematic diagram of the structure of the Chinese polyphone character prediction and recognition device in the embodiment of the present application. DETAILED DESCRIPTION
[0059] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.
[0060] The present application designs a method, apparatus, device and storage medium for predicting and recognizing Chinese polyphones, which solves the technical problems that the current TTS front-end has weak prediction ability for some professional terms and names in specific business scenarios, and it is also difficult to predict some common intonation changes in spoken language. The model trained based on the CPP dataset is difficult to predict the pronunciations with fewer training samples. At the same time, a single data source and a small amount of data will reduce the generalization ability of the model to a considerable extent.
[0061] For easier understanding, see Figure 1 , Figure 1 Flow chart of the method for predicting and recognizing Chinese polyphones in the embodiment of the present application. Figure 1 As shown, specifically:
[0062] Step S1, obtaining text content to be recognized;
[0063] It should be noted that the original text data to be recognized is received.
[0064] In addition, basic text cleaning can be performed, such as removing meaningless symbols, unifying the encoding format, etc.
[0065] Step S2: preprocessing the text content by using a preset Chinese polyphonetic character dataset to convert the text content into an embedding vector sequence recognizable by the model;
[0066] It should be noted that because the pronunciation of a polyphone is closely related to the semantics of the word it is in and its contextual position, we supplemented the Chinese encoding with information such as part of speech and word boundaries (segmentation) and embedded it in the front end of the BERT model.
[0067] Regular Expression Transformation: Use regular expressions to identify and replace specific patterns in text. For example, it can be used to identify email addresses, phone numbers, or dates in a specific format.
[0068] Word segmentation: Split text into smaller units, usually words or phrases. Word segmentation is especially important in Chinese because Chinese text does not have clear word boundaries like English.
[0069] Encoding: Converting text into a numerical form that a machine learning model can process. This usually involves mapping words or phrases into vectors in a high-dimensional space.
[0070] Part-of-speech tagging: Assigning a part-of-speech tag to each word in the text, such as noun, verb, adjective, etc. This helps in understanding the structure and meaning of the text.
[0071] Embedding vector sequence: Ultimately, the data generated by the above steps will be converted into a series of embedding vectors, which can be used to train machine learning models such as text classification, sentiment analysis, or language generation models.
[0072] Step S3, processing the embedded vector sequence based on the context enhancement method to obtain a fused feature vector sequence;
[0073] It should be noted that the context enhancement (shift fusion) method is used to fuse the embedding vector of each word with the embedding vectors of the previous and next words.
[0074] Consider word order and distance, adjust fusion weights, and enhance contextual information.
[0075] The fused contextual information is integrated with the original word embedding vector to form a richer feature vector.
[0076] Ensure that the fused feature vector can reflect the semantic and grammatical roles of words in the sentence.
[0077] Step S4: Use the fused feature vector sequence as the input of the pre-trained BERT-based sequence annotation model, perform classification output, and obtain the pinyin sequence corresponding to the predicted polyphonetic character.
[0078] It should be noted that the pre-trained BERT model is loaded to ensure that it can process the semantic information of the text.
[0079] The fused feature vector is input into the BERT model to leverage its deep semantic understanding capabilities.
[0080] The sequence labeling function of the BERT model is used to predict polyphones for each word.
[0081] The model outputs multiple possible pronunciations for each word and selects the most appropriate pronunciation based on contextual information.
[0082] According to the prediction results of the model, the pinyin sequence corresponding to each polyphone is generated.
[0083] Ensure the accuracy and naturalness of the pinyin sequence and conform to the pronunciation rules of Chinese.
[0084] Furthermore, before step S1, the following steps are also included:
[0085] Step S0: construct a preset Chinese polyphonetic character data set, wherein the preset Chinese polyphonetic character data set includes polyphonetic character sampling data collected from a plurality of data sources, and the number of each polyphonetic character sampling data in the preset Chinese polyphonetic character data set satisfies a uniform distribution.
[0086] It should be noted that more fields are selected when collecting data, such as Wikipedia, question-and-answer websites, forums, news, and other polyphonetic character datasets such as CPP and MPC data sources to better simulate the actual distribution of data.
[0087] The sampling quantity of polyphones is made to obey uniform distribution, and each pronunciation within a single polyphone is sampled according to the actual distribution of the data source.
[0088] Each polyphone must have at least 1,000 data points, and each polyphone must have at least 100 data points for each single pronunciation.
[0089] Furthermore, step S2 specifically includes:
[0090] Match and replace the text content based on the preset regular expression rules to obtain the preprocessed text content;
[0091] Perform word segmentation on the preprocessed text content to obtain a vocabulary consisting of word segmentation results of the preprocessed text content;
[0092] Perform part-of-speech tagging on the word segmentation results contained in the vocabulary using a preset tagging tool;
[0093] The word segmentation results and corresponding part-of-speech standards contained in the vocabulary are converted into an embedding vector group through a pre-trained word embedding model, and an embedding vector sequence is formed according to the order of the word segmentation results in the pre-processed text content.
[0094] Furthermore, step S3 specifically includes:
[0095] Using the context enhancement method, the embedding vector group of each word in the embedding vector sequence is fused with the embedding vector groups of the previous and next word segments to generate a fused feature vector sequence.
[0096] Furthermore, the context enhancement method is specifically as follows:
[0097] Create a collection window for the embedded vector sequence, configure the offset ω and step size s of the collection window, move the embedded vector sequence along the sequence direction, and fill it with a 0 vector when the movement exceeds the limit. The moving range is
[0098]
[0099] The fused feature vector sequence of the embedded vector sequence is obtained based on the preset formula group, and the preset formula group is:
[0100]
[0101] V context =GELU(E win )+E emb ;
[0102] in, Refers to the set of all translation embedding vector sequences within the window; A set of left-translated embedding vector sequences; A set of embedding vector sequences pointing to the right; refers to the original untranslated embedding vector sequence; E win is the matrix obtained by vertically concatenating all embedded vector sequences in the window; V context It is processed by GELU activation function and combined with E emb The fused feature vector sequence after residual connection; T is the sequence length; D is the dimension of the embedded vector, E emb ∈R T×D .
[0103] It should be noted that the preset formula group included in the context enhancement method can be understood in detail as follows:
[0104] Initialize a temporary matrix Its size is [0:T,0:D], where T is the sequence length; D is the dimension of the embedding vector;
[0105] Get a set of left-shifted embedding vector sequences, that is, for each k (starting from 0), copy the sequence from -ks to T to
[0106] Get a set of right-shifted embedding vector sequences, that is, for each k, copy the sequence from 0 to T-ks to
[0107] The original untranslated embedding vector sequence is directly copied to
[0108] All translated embedding vector sequences are concatenated vertically, that is, concatenated along the dimension of the embedding vector to obtain E win ;
[0109] Apply the GELU activation function to E win , and compare the result with the original embedding vector E emb Perform residual connection to obtain the final fusion feature vector sequence V context , which is a vector representing contextual features. It integrates multiple contextual information of the current vocabulary and can better capture the complexity and diversity of language.
[0110] Furthermore, a word-granularity-based mask is set in the classifier at the end of the BERT-based sequence labeling model.
[0111] It should be noted that the classifier at the end of BERT is based on the set L = {label1, label2, label3, ..., label n}, which will introduce obvious noise in the pronunciation prediction of a single polyphone, and have a great impact on the model prediction results.
[0112] Therefore, we introduce word-granularity masks, which mask the pronunciation labels that do not belong to the word when the model predicts the pronunciation of the current polyphone. In this way, the scope of model prediction is narrowed to the several pronunciations corresponding to the word, greatly improving the accuracy of model prediction.
[0113] See also Figure 2 , Figure 2 Schematic diagram of the structure of the Chinese polyphone prediction and recognition device in the embodiment of the present application. Figure 2 As shown, specifically:
[0114] An acquisition unit 201 is used to acquire text content to be recognized;
[0115] The embedding operation unit 202 is used to pre-process the text content by using a preset Chinese polyphone data set to convert the text content into an embedding vector sequence recognizable by the model;
[0116] A context enhancement unit 203, used to process the embedded vector sequence based on a context enhancement method to obtain a fused feature vector sequence;
[0117] The prediction unit 204 is used to use the fused feature vector sequence as the input of the pre-trained BERT-based sequence labeling model, perform classification output, and obtain the pinyin sequence corresponding to the predicted polyphonetic character.
[0118] Furthermore, it also includes:
[0119] A construction unit is used to construct a preset Chinese polyphone data set, which includes polyphone sampling data collected from several data sources, and the overall number of polyphone sampling data satisfies a uniform distribution, and the individual number of each pronunciation of the polyphone sampling data satisfies a preset threshold.
[0120] It should be noted that more fields are selected when collecting data, such as Wikipedia, question-and-answer websites, forums, news, and other polyphonetic character datasets such as CPP and MPC data sources to better simulate the actual distribution of data.
[0121] The sampling quantity of polyphones is made to obey uniform distribution, and each pronunciation within a single polyphone is sampled according to the actual distribution of the data source.
[0122] Each polyphone must have at least 1,000 data points, and each polyphone must have at least 100 data points for its corresponding single pronunciation.
[0123] Furthermore, the embedded operation unit 202 is specifically used for:
[0124] Match and replace the text content based on the preset regular expression rules to obtain the preprocessed text content;
[0125] Perform word segmentation on the preprocessed text content to obtain a vocabulary consisting of word segmentation results of the preprocessed text content;
[0126] Perform part-of-speech tagging on the word segmentation results contained in the vocabulary using a preset tagging tool;
[0127] The word segmentation results and corresponding part-of-speech standards contained in the vocabulary are converted into an embedding vector group through a pre-trained word embedding model, and an embedding vector sequence is formed according to the order of the word segmentation results in the pre-processed text content.
[0128] Further, the context enhancement unit 203 is specifically used for:
[0129] Using the context enhancement method, the embedding vector group of each word in the embedding vector sequence is fused with the embedding vector groups of the previous and next word segments to generate a fused feature vector sequence.
[0130] Furthermore, the context enhancement method is specifically as follows:
[0131] Create a collection window for the embedded vector sequence, configure the offset ω and step size s of the collection window, move the embedded vector sequence along the sequence direction, and fill it with a 0 vector when the movement exceeds the limit. The moving range is
[0132]
[0133] The fused feature vector sequence of the embedded vector sequence is obtained based on the preset formula group, and the preset formula group is:
[0134]
[0135] V context =GELU(E win )+E emb ;
[0136] in, Refers to the set of all translation embedding vector sequences within the window; A set of left-translated embedding vector sequences; A set of embedding vector sequences pointing to the right; refers to the original untranslated embedding vector sequence; E win is the matrix obtained by vertically concatenating all embedded vector sequences in the window; V context It is processed by GELU activation function and combined with E emb The fused feature vector sequence after residual connection; T is the sequence length; D is the dimension of the embedded vector, E emb ∈R T×D .
[0137] Furthermore, a word-granularity-based mask is set in the classifier at the end of the BERT-based sequence labeling model.
[0138] The present application also provides another Chinese polyphonetic character prediction and recognition device, such as Figure 3 For the sake of convenience, only the parts related to the embodiments of the present application are shown. For specific technical details not disclosed, please refer to the method part of the embodiments of the present application. The terminal can be any terminal device including a mobile phone, a tablet computer, a personal digital assistant (English full name: Personal Digital Assistant, English abbreviation: PDA), a sales terminal (English full name: Point of Sales, English abbreviation: POS), a car computer, etc., taking the terminal as a mobile phone as an example:
[0139] Figure 3 FIG. 1 is a block diagram showing a partial structure of a mobile phone related to a terminal provided in an embodiment of the present application. Figure 3 The mobile phone includes: a radio frequency (RF) circuit 1010, a memory 1020, an input unit 1030, a display unit 1040, a sensor 1050, an audio circuit 1060, a wireless fidelity (WiFi) module 1070, a processor 1080, and a power supply 1090. Those skilled in the art can understand that Figure 3The mobile phone structure shown in the figure does not constitute a limitation on the mobile phone, and may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently.
[0140] Combine the following Figure 3 A detailed introduction to the various components of the mobile phone:
[0141] The RF circuit 1010 can be used for receiving and sending signals during information transmission or calls. In particular, after receiving the downlink information of the base station, it is sent to the processor 1080 for processing; in addition, the designed uplink data is sent to the base station. Generally, the RF circuit 1010 includes but is not limited to an antenna, at least one amplifier, a transceiver, a coupler, a low noise amplifier (full name: Low Noise Amplifier, English abbreviation: LNA), a duplexer, etc. In addition, the RF circuit 1010 can also communicate with the network and other devices through wireless communication. The above-mentioned wireless communications can use any communication standard or protocol, including but not limited to Global System of Mobile communications (Global System of Mobile communication, English abbreviation: GSM), General Packet Radio Service (General Packet Radio Service, GPRS), Code Division Multiple Access (Code Division Multiple Access, English abbreviation: CDMA), Wideband Code Division Multiple Access (WCDMA), Long Term Evolution (Long Term Evolution, English abbreviation: LTE), e-mail, Short Messaging Service (SMS), etc.
[0142] The memory 1020 can be used to store software programs and modules. The processor 1080 executes various functional applications and data processing of the mobile phone by running the software programs and modules stored in the memory 1020. The memory 1020 can mainly include a program storage area and a data storage area, wherein the program storage area can store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc.; the data storage area can store data created according to the use of the mobile phone (such as audio data, a phone book, etc.), etc. In addition, the memory 1020 can include a high-speed random access memory, and can also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other volatile solid-state storage devices.
[0143] The input unit 1030 can be used to receive input digital or character information, and to generate key signal input related to the user settings and function control of the mobile phone. Specifically, the input unit 1030 may include a touch panel 1031 and other input devices 1032. The touch panel 1031, also known as a touch screen, can collect the user's touch operation on or near it (such as the user's operation on the touch panel 1031 or near the touch panel 1031 using any suitable object or accessory such as a finger, stylus, etc.), and drive the corresponding connection device according to a pre-set program. Optionally, the touch panel 1031 may include two parts: a touch detection device and a touch controller. Among them, the touch detection device detects the user's touch orientation, detects the signal brought by the touch operation, and transmits the signal to the touch controller; the touch controller receives the touch information from the touch detection device, converts it into contact coordinates, and then sends it to the processor 1080, and can receive and execute commands sent by the processor 1080. In addition, the touch panel 1031 can be implemented in various types such as resistive, capacitive, infrared, and surface acoustic waves. In addition to the touch panel 1031, the input unit 1030 may further include other input devices 1032. Specifically, the other input devices 1032 may include but are not limited to one or more of a physical keyboard, function keys (such as volume control keys, switch keys, etc.), a trackball, a mouse, a joystick, and the like.
[0144] The display unit 1040 can be used to display information input by the user or information provided to the user and various menus of the mobile phone. The display unit 1040 may include a display panel 1041. Optionally, the display panel 1041 may be configured in the form of a liquid crystal display (full name in English: Liquid Crystal Display, English abbreviation: LCD), an organic light-emitting diode (full name in English: Organic Light-Emitting Diode, English abbreviation: OLED), etc. Further, the touch panel 1031 may cover the display panel 1041. When the touch panel 1031 detects a touch operation on or near it, it is transmitted to the processor 1080 to determine the type of touch event. Subsequently, the processor 1080 provides a corresponding visual output on the display panel 1041 according to the type of touch event. Although in Figure 3 In the embodiment, the touch panel 1031 and the display panel 1041 are used as two independent components to realize the input and output functions of the mobile phone, but in some embodiments, the touch panel 1031 and the display panel 1041 can be integrated to realize the input and output functions of the mobile phone.
[0145] The mobile phone may also include at least one sensor 1050, such as a light sensor, a motion sensor, and other sensors. Specifically, the light sensor may include an ambient light sensor and a proximity sensor, wherein the ambient light sensor may adjust the brightness of the display panel 1041 according to the brightness of the ambient light, and the proximity sensor may turn off the display panel 1041 and / or the backlight when the mobile phone is moved to the ear. As a type of motion sensor, the accelerometer sensor can detect the magnitude of acceleration in all directions (generally three axes), and can detect the magnitude and direction of gravity when stationary. It can be used for applications that identify the posture of the mobile phone (such as horizontal and vertical screen switching, related games, magnetometer posture calibration), vibration recognition related functions (such as pedometer, tapping), etc.; as for other sensors that can be configured in the mobile phone, such as gyroscopes, barometers, hygrometers, thermometers, infrared sensors, etc., they will not be repeated here.
[0146] The audio circuit 1060, the speaker 1061, and the microphone 1062 can provide an audio interface between the user and the mobile phone. The audio circuit 1060 can transmit the received audio data to the speaker 1061 after converting the received audio data into an electrical signal, which is converted into a sound signal for output; on the other hand, the microphone 1062 converts the collected sound signal into an electrical signal, which is received by the audio circuit 1060 and converted into audio data, and then the audio data is output to the processor 1080 for processing, and then sent to another mobile phone through the RF circuit 1010, or the audio data is output to the memory 1020 for further processing.
[0147] WiFi is a short-range wireless transmission technology. The mobile phone can help users send and receive emails, browse web pages and access streaming media through the WiFi module 1070. It provides users with wireless broadband Internet access. Figure 3 A WiFi module 1070 is shown, but it is understandable that it is not an essential component of the mobile phone and can be omitted as needed without changing the essence of the invention.
[0148] The processor 1080 is the control center of the mobile phone. It uses various interfaces and lines to connect various parts of the entire mobile phone. By running or executing software programs and / or modules stored in the memory 1020 and calling data stored in the memory 1020, it executes various functions of the mobile phone and processes data, thereby monitoring the mobile phone as a whole. Optionally, the processor 1080 may include one or more processing units; preferably, the processor 1080 may integrate an application processor and a modem processor, wherein the application processor mainly processes the operating system, user interface and application programs, etc., and the modem processor mainly processes wireless communications. It is understandable that the above-mentioned modem processor may not be integrated into the processor 1080.
[0149] The mobile phone also includes a power supply 1090 (such as a battery) for supplying power to various components. Preferably, the power supply can be logically connected to the processor 1080 through a power management system, so that the power management system can manage functions such as charging, discharging, and power consumption.
[0150] Although not shown, the mobile phone may also include a camera, a Bluetooth module, etc., which will not be described in detail here.
[0151] In the embodiment of the present application, the processor 1080 included in the terminal also has the following functions:
[0152] Step S1, obtaining text content to be recognized;
[0153] It should be noted that the original text data to be recognized is received.
[0154] In addition, basic text cleaning can be performed, such as removing meaningless symbols, unifying the encoding format, etc.
[0155] Step S2, preprocessing the text content by using a preset Chinese polyphone data set to convert the text content into an embedding vector sequence recognizable by the model;
[0156] It should be noted that because the pronunciation of a polyphone is closely related to the semantics of the word it is in and its contextual position, we supplemented the Chinese encoding with information such as part of speech and word boundaries (segmentation) and embedded it in the front end of the BERT model.
[0157] Regular Expression Transformation: Use regular expressions to identify and replace specific patterns in text. For example, it can be used to identify email addresses, phone numbers, or dates in a specific format.
[0158] Word segmentation: Split text into smaller units, usually words or phrases. Word segmentation is especially important in Chinese because Chinese text does not have clear word boundaries like English.
[0159] Encoding: Converting text into a numerical form that a machine learning model can process. This usually involves mapping words or phrases into vectors in a high-dimensional space.
[0160] Part-of-speech tagging: Assigning a part-of-speech tag to each word in the text, such as noun, verb, adjective, etc. This helps in understanding the structure and meaning of the text.
[0161] Embedding vector sequence: Ultimately, the data generated by the above steps will be converted into a series of embedding vectors, which can be used to train machine learning models such as text classification, sentiment analysis, or language generation models.
[0162] Step S3, processing the embedded vector sequence based on the context enhancement method to obtain a fused feature vector sequence;
[0163] It should be noted that the context enhancement (shift fusion) method is used to fuse the embedding vector of each word with the embedding vectors of the previous and next words.
[0164] Consider word order and distance, adjust fusion weights, and enhance contextual information.
[0165] The fused contextual information is integrated with the original word embedding vector to form a richer feature vector.
[0166] Ensure that the fused feature vector can reflect the semantic and grammatical roles of words in the sentence.
[0167] Step S4: Use the fused feature vector sequence as the input of the pre-trained BERT-based sequence annotation model, perform classification output, and obtain the pinyin sequence corresponding to the predicted polyphonetic character.
[0168] It should be noted that the pre-trained BERT model is loaded to ensure that it can process the semantic information of the text.
[0169] The fused feature vector is input into the BERT model to leverage its deep semantic understanding capabilities.
[0170] The sequence labeling function of the BERT model is used to predict polyphones for each word.
[0171] The model outputs multiple possible pronunciations for each word and selects the most appropriate pronunciation based on contextual information.
[0172] According to the prediction results of the model, the pinyin sequence corresponding to each polyphone is generated.
[0173] Ensure the accuracy and naturalness of the pinyin sequence and conform to the pronunciation rules of Chinese.
[0174] Furthermore, before step S1, the following steps are also included:
[0175] Step S0: construct a preset Chinese polyphonetic character data set, wherein the preset Chinese polyphonetic character data set includes polyphonetic character sampling data collected from a plurality of data sources, and the number of each polyphonetic character sampling data in the preset Chinese polyphonetic character data set satisfies a uniform distribution.
[0176] It should be noted that more fields are selected when collecting data, such as Wikipedia, question-and-answer websites, forums, news, and other polyphonetic character datasets such as CPP and MPC data sources to better simulate the actual distribution of data.
[0177] The sampling quantity of polyphones is made to obey uniform distribution, and each pronunciation within a single polyphone is sampled according to the actual distribution of the data source.
[0178] Each polyphone must have at least 1,000 data points, and each polyphone must have at least 100 data points for each single pronunciation.
[0179] Furthermore, step S2 specifically includes:
[0180] Matching and replacing the text content based on preset regular expression rules to obtain preprocessed text content;
[0181] Perform word segmentation on the preprocessed text content to obtain a vocabulary consisting of word segmentation results of the preprocessed text content;
[0182] Perform part-of-speech tagging on the word segmentation results contained in the vocabulary using a preset tagging tool;
[0183] The word segmentation results and corresponding part-of-speech standards contained in the vocabulary are converted into an embedding vector group through a pre-trained word embedding model, and an embedding vector sequence is formed according to the order of the word segmentation results in the pre-processed text content.
[0184] Furthermore, step S3 specifically includes:
[0185] Using the context enhancement method, the embedding vector group of each word in the embedding vector sequence is fused with the embedding vector groups of the previous and next word segments to generate a fused feature vector sequence.
[0186] Furthermore, the context enhancement method is specifically as follows:
[0187] Create a collection window for the embedded vector sequence, configure the offset ω and step size s of the collection window, move the embedded vector sequence along the sequence direction, and fill it with a 0 vector when the movement exceeds the limit. The moving range is
[0188] The fused feature vector sequence of the embedded vector sequence is obtained based on a preset formula group, and the preset formula group is:
[0189]
[0190] V context =GELU(E win )+E emb ;
[0191] in, Refers to the set of all translation embedding vector sequences within the window; A set of left-translated embedding vector sequences; A set of embedding vector sequences pointing to the right; refers to the original untranslated embedding vector sequence; E win is the matrix obtained by vertically concatenating all embedded vector sequences in the window; V context It is processed by GELU activation function and combined with E emb The fused feature vector sequence after residual connection; T is the sequence length; D is the dimension of the embedded vector, E emb ∈R T×D .
[0192] It should be noted that the preset formula group included in the context enhancement method can be understood in detail as follows:
[0193] Initialize a temporary matrix Its size is [0:T,0:D], where T is the sequence length; D is the dimension of the embedding vector;
[0194] Get a set of left-shifted embedding vector sequences, that is, for each k (starting from 0), copy the sequence from -ks to T to
[0195] Get a set of right-shifted embedding vector sequences, that is, for each k, copy the sequence from 0 to T-ks to
[0196] The original untranslated embedding vector sequence is directly copied to
[0197] All translated embedding vector sequences are concatenated vertically, that is, concatenated along the dimension of the embedding vector to obtain E win ;
[0198] Apply the GELU activation function to E win , and compare the result with the original embedding vector E emb Perform residual connection to obtain the final fusion feature vector sequence V context , which is a vector representing contextual features. It integrates multiple contextual information of the current vocabulary and can better capture the complexity and diversity of language.
[0199] Furthermore, a word-granularity-based mask is set in the classifier at the end of the BERT-based sequence labeling model.
[0200] It should be noted that the classifier at the end of BERT is based on the set L = {label1, label2, label3, ..., label n}, which will introduce obvious noise in the pronunciation prediction of a single polyphone, and have a great impact on the model prediction results.
[0201] Therefore, we introduce word-granularity masks, which mask the pronunciation labels that do not belong to the word when the model predicts the pronunciation of the current polyphone. In this way, the scope of model prediction is narrowed to the several pronunciations corresponding to the word, greatly improving the accuracy of model prediction.
[0202] The embodiment of the present application also provides a computer-readable storage medium for storing program code, wherein the program code is used to execute any one of the implementation methods of the Chinese polyphonetic character prediction and recognition method described in the aforementioned embodiments.
[0203] In the embodiments of the present application, a method, apparatus, device and storage medium for predicting and recognizing Chinese polyphones are provided, a preset Chinese polyphone data set with balanced distribution is established, and on the basis of ensuring that each polyphone has the same sufficient amount of data, the data set is designed according to the actual distribution of the pronunciation of the characters, so that the model trained based on the enhanced data set can better fit the actual situation to predict the pronunciation of the polyphones, and at the same time integrate the sentence context semantics and structural information to help the model better learn the influence of words and context on the pronunciation of the characters, thereby solving the problems that the current TTS front-end has weak prediction ability for some professional terms and names in specific business scenarios, and it is also difficult to predict some common tone changes in spoken language, and the model trained based on the CPP data set is difficult to predict the pronunciation of those with fewer training samples, and the single data source and small amount of data will reduce the generalization ability of the model to a considerable extent.
[0204] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.
[0205] The terms "first", "second", "third", "fourth", etc. (if any) in the specification of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein, for example. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.
[0206] It should be understood that in the present application, "at least one (item)" means one or more, and "plurality" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the objects associated before and after are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.
[0207] In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.
[0208] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.
[0209] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.
[0210] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (full name in English: Read-Only Memory, English abbreviation: ROM), random access memory (full name in English: Random Access Memory, English abbreviation: RAM), disk or optical disk and other media that can store program codes.
[0211] As described above, the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application.
Claims
1. A method for predicting and recognizing Chinese polyphones, characterized in that: include: Get the text content to be recognized; Preprocessing the text content by using a preset Chinese polyphone data set to convert the text content into an embedding vector sequence recognizable by a model; Processing the embedded vector sequence based on a context enhancement method to obtain a fused feature vector sequence; The fused feature vector sequence is used as the input of a pre-trained BERT-based sequence annotation model, and classified output is performed to obtain a pinyin sequence corresponding to the predicted polyphonetic character.
2. The Chinese polyphone prediction and recognition method according to claim 1, characterized in that: Before obtaining the text content to be identified, the method further includes: A preset Chinese polyphonetic character data set is constructed, wherein the preset Chinese polyphonetic character data set includes polyphonetic character sampling data collected from a plurality of data sources, and the number of each polyphonetic character sampling data in the preset Chinese polyphonetic character data set satisfies a uniform distribution.
3. The Chinese polyphone prediction and recognition method according to claim 1, characterized in that: The preprocessing of the text content by using a preset Chinese polyphone data set to convert the text content into an embedding vector sequence recognizable by a model specifically includes: Matching and replacing the text content based on preset regular expression rules to obtain preprocessed text content; Performing word segmentation processing on the preprocessed text content to obtain a vocabulary consisting of word segmentation results of the preprocessed text content; Performing part-of-speech tagging on the word segmentation results contained in the vocabulary using a preset tagging tool; The word segmentation results and corresponding part-of-speech standards contained in the vocabulary are converted into an embedding vector group through a pre-trained word embedding model, and an embedding vector sequence is formed according to the order of the word segmentation results in the pre-processed text content.
4. The Chinese polyphone prediction and recognition method according to claim 1, characterized in that: The method of processing the embedded vector sequence based on the context enhancement method to obtain a fused feature vector sequence specifically includes: Using a context enhancement method, the embedding vector group of each word segment in the embedding vector sequence is fused with the embedding vector groups of the previous and next word segments to generate a fused feature vector sequence.
5. The Chinese polyphonetic character prediction and recognition method according to claim 4, characterized in that: The context enhancement method is specifically as follows: Create a collection window for the embedded vector sequence, configure the offset ω and step size s of the collection window, move the embedded vector sequence along the sequence direction, and fill it with a 0 vector when the movement exceeds the limit. The moving range is The fused feature vector sequence of the embedded vector sequence is obtained based on a preset formula group, and the preset formula group is: in, Refers to the set of all translation embedding vector sequences within the window; A set of left-translated embedding vector sequences; A set of embedding vector sequences pointing to the right; refers to the original untranslated embedding vector sequence; E win is the matrix obtained by vertically concatenating all embedded vector sequences in the window; V context It is processed by GELU activation function and combined with E emb The fused feature vector sequence after residual connection; T is the sequence length; D is the dimension of the embedded vector, E emb ∈R T×D .
6. The Chinese polyphone prediction and recognition method according to claim 1, characterized in that: A mask based on word granularity is provided in the classifier at the end of the BERT-based sequence labeling model.
7. A Chinese polyphonetic character prediction and recognition device, characterized in that: include: An acquisition unit, used for acquiring text content to be recognized; An embedding operation unit, used to pre-process the text content by using a preset Chinese polyphone data set, and convert the text content into an embedding vector sequence recognizable by a model; A context enhancement unit, used for processing the embedding vector sequence based on a context enhancement method to obtain a fused feature vector sequence; The prediction unit is used to use the fused feature vector sequence as the input of the pre-trained BERT-based sequence annotation model, perform classification output, and obtain the pinyin sequence corresponding to the predicted polyphone.
8. The Chinese polyphonetic character prediction and recognition device according to claim 7, characterized in that: Also includes: The construction unit is used to construct a preset Chinese polyphone data set, wherein the preset Chinese polyphone data set includes polyphone sampling data collected from a plurality of data sources, and the number of each polyphone sampling data in the preset Chinese polyphone data set satisfies a uniform distribution.
9. A Chinese polyphonetic character prediction and recognition device, characterized in that: The device comprises a processor and a memory: The memory is used to store program code and transmit the program code to the processor; The processor is used to execute the Chinese polyphone character prediction and recognition method according to any one of claims 1-6 according to the instructions in the program code.
10. A computer-readable storage medium, characterized in that: The computer-readable storage medium is used to store program codes, and the program codes are used to execute the Chinese polyphone character prediction and recognition method according to any one of claims 1 to 6.