Hybrid speech processing method, electronic device, and computer readable medium
By treating mixed speech as the main language speech processing and using preset models and mixed dictionaries for recognition, the problem of large amount of training data and long recognition time in the existing technology of mixed speech models is solved, and fast and efficient mixed speech recognition is achieved. .
Patent Information
- Application Number
- CN202010705959.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2020-07-21
- Publication Date
- 2024-12-06
- Estimated Expiration
- 2040-07-21
AI Technical Summary
When using deep neural networks to train mixed language pronunciation models in Chinese and English, the prior art requires massive sample data and the recognition time is long, and the search space of the mixed language pronunciation model is huge, resulting in low recognition efficiency.
By treating mixed speech as the main language pronunciation processing, the preset main language pronunciation model is used to obtain the phoneme sequence to be recognized, and the word sequence is searched according to the mixed dictionary, and only the main language phonemes are searched to reduce the search space and improve the recognition efficiency.
It realizes fast recognition of mixed speech, fast response, less time, and does not require a large amount of sample data training, which improves recognition efficiency.
Smart Images

Figure CN114038463B_ABST
Abstract
Description
Technical Field
[0001] The embodiments of the present disclosure relate to the field of speech recognition technology, and in particular to a method, electronic device, and computer-readable medium for hybrid speech processing. Background Art
[0002] With the development of globalization, mixed sentences (such as Chinese and English mixed sentences) are used more and more frequently, such as "download Beyond's songs", "turn on wifi", etc. Speech recognition is also increasingly being used in smart driving, smart speakers, mobile assistants and other fields.
[0003] Using deep neural networks to train a mixed language speech model of Chinese and English to recognize sentences of mixed words requires a huge amount of sample data for training and the recognition time is long. Summary of the invention
[0004] The embodiments of the present disclosure provide a method and device for hybrid speech processing, an electronic device, and a computer-readable medium.
[0005] In a first aspect, an embodiment of the present disclosure provides a method for processing a mixed speech, wherein the mixed speech includes a mixed sentence using a primary language and a secondary language at the same time, wherein the primary language and the secondary language are two different predetermined languages; the method includes: acquiring the mixed speech; obtaining a phoneme sequence to be recognized through a preset primary language speech model; the phoneme sequence to be recognized is a sequence of primary language phonemes corresponding to the mixed speech; querying and obtaining at least one word sequence corresponding to the phoneme sequence to be recognized according to a predetermined mixed dictionary; the word sequence is composed of a plurality of words, each word being a primary language word or a secondary language word; the mixed dictionary includes primary language phoneme combinations corresponding to primary language words, and primary language phoneme combinations corresponding to secondary language words.
[0006] In some embodiments, the process of building a predetermined hybrid dictionary includes: obtaining multiple speech samples of multiple sub-language words, and obtaining the main language phoneme combination of each speech sample of each sub-language word through a preset main language speech conversion model; performing clustering decisions on the main language phoneme combinations of all speech samples of each sub-language word obtained, and obtaining the main language phoneme combination corresponding to each sub-language word.
[0007] In a second aspect, an embodiment of the present disclosure provides a device for processing a mixed speech, wherein the mixed speech includes a mixed sentence using a primary language and a secondary language at the same time, and the primary language and the secondary language are two different predetermined languages; the device includes: an acquisition module, used to acquire the mixed speech; a recognition module, used to obtain a phoneme sequence to be recognized through a preset primary language speech model; the phoneme sequence to be recognized is a sequence of primary language phonemes corresponding to the mixed speech; a query module, used to query and obtain at least one word sequence corresponding to the phoneme sequence to be recognized according to a predetermined mixed dictionary; the word sequence is composed of multiple words, each word is a primary language word or a secondary language word; the mixed dictionary includes a primary language phoneme combination corresponding to a primary language word, and a primary language phoneme combination corresponding to a secondary language word.
[0008] In a third aspect, an embodiment of the present disclosure provides an electronic device, comprising:
[0009] one or more processors;
[0010] A memory having one or more programs stored thereon, wherein when the one or more programs are executed by the one or more processors, the one or more processors implement any one of the above-mentioned hybrid speech processing methods;
[0011] One or more I / O interfaces are connected between the processor and the memory to implement information exchange between the processor and the memory.
[0012] In a fourth aspect, an embodiment of the present disclosure provides a computer-readable medium having a computer program stored thereon, wherein the program, when executed by a processor, implements any of the above-mentioned mixed speech processing methods.
[0013] In the method and device, electronic device, and computer-readable medium for mixed speech processing of the embodiments of the present disclosure, the mixed speech is regarded as the main language speech processing, and the mixed speech is recognized by using a trained main language speech model. During the recognition process, only the main language phonemes need to be searched, the search space is relatively small, the recognition response is fast, and the time taken is short. BRIEF DESCRIPTION OF THE DRAWINGS
[0014] In the accompanying drawings of the embodiments of the present disclosure:
[0015] Figure 1 A flow chart of a method for mixed speech processing provided by an embodiment of the present disclosure;
[0016] Figure 2 A flowchart of another method for mixed speech processing provided by an embodiment of the present disclosure;
[0017] Figure 3 A flowchart of some steps in another method for mixed speech processing provided by an embodiment of the present disclosure;
[0018] Figure 4 A flowchart of some steps in another method for mixed speech processing provided by an embodiment of the present disclosure;
[0019] Figure 5 A flowchart of some steps in another method for mixed speech processing provided by an embodiment of the present disclosure;
[0020] Figure 6 A block diagram of a hybrid speech processing device provided by an embodiment of the present disclosure;
[0021] Figure 7 A block diagram of an electronic device provided in an embodiment of the present disclosure;
[0022] Figure 8 A block diagram of a computer-readable medium provided for an embodiment of the present disclosure. DETAILED DESCRIPTION
[0023] In order to enable those skilled in the art to better understand the technical solutions of the embodiments of the present disclosure, the method and device for hybrid speech processing, electronic device, and computer-readable medium provided by the embodiments of the present disclosure are described in detail below in conjunction with the accompanying drawings.
[0024] The embodiments of the present disclosure will be described more fully below with reference to the accompanying drawings, but the embodiments shown may be embodied in different forms and should not be construed as being limited to the embodiments set forth in the present disclosure. On the contrary, the purpose of providing these embodiments is to make the present disclosure thorough and complete and to enable those skilled in the art to fully understand the scope of the present disclosure.
[0025] The accompanying drawings of the embodiments of the present disclosure are used to provide a further understanding of the embodiments of the present disclosure and constitute a part of the specification. Together with the embodiments of the present disclosure, they are used to explain the present disclosure and do not constitute a limitation of the present disclosure. The above and other features and advantages will become more apparent to those skilled in the art by describing the detailed exemplary embodiments with reference to the accompanying drawings.
[0026] The embodiments of the present disclosure may be described with reference to plan views and / or cross-sectional views by way of ideal schematic diagrams of the present disclosure. Therefore, the exemplary illustrations may be modified according to manufacturing techniques and / or tolerances.
[0027] In the absence of conflict, the various embodiments of the present disclosure and the various features therein may be combined with each other.
[0028] The terms used in the present disclosure are only used to describe specific embodiments and are not intended to limit the present disclosure. The term "and / or" as used in the present disclosure includes any and all combinations of one or more related enumerated items. The singular forms "one" and "the" as used in the present disclosure are also intended to include plural forms, unless the context clearly indicates otherwise. The terms "including", "made of..." as used in the present disclosure specify the presence of the features, wholes, steps, operations, elements and / or components, but do not exclude the presence or addition of one or more other features, wholes, steps, operations, elements, components and / or groups thereof.
[0029] Unless otherwise defined, all terms (including technical and scientific terms) used in this disclosure have the same meaning as commonly understood by those of ordinary skill in the art. It will also be understood that terms such as those defined in commonly used dictionaries should be interpreted as having a meaning consistent with their meaning in the context of the relevant art and this disclosure, and will not be interpreted as having an idealized or overly formal meaning unless explicitly defined in this disclosure.
[0030] The embodiments of the present disclosure are not limited to the embodiments shown in the drawings, but include modifications of the configurations formed based on the manufacturing process. Therefore, the regions illustrated in the drawings have schematic properties, and the shapes of the regions shown in the drawings illustrate the specific shapes of the regions of the elements, but are not intended to be limiting.
[0031] ASR (automatic speech recognition) refers to a technology that analyzes the voice spoken by the user to determine its corresponding content (such as text content). After identifying the content corresponding to the voice, it can further analyze the meaning of the content and perform corresponding operations, such as for corresponding control in the fields of smart driving, smart speakers, mobile phone assistants, etc.
[0032] With the development of the times, the use of mixed speech is increasing. Mixed speech refers to speech including mixed sentences. Mixed sentences refer to the use of two different languages in one sentence, such as "download Beyond's songs" and "turn on WiFi".
[0033] Therefore, when using ASR technology to recognize mixed speech, it is necessary to establish a mixed language speech model, that is, to regard the mixture of multiple languages as a new language, the phoneme set of this language includes the phonemes of all languages in the mixed language, and the mixed language speech model can be obtained by training the model with a large number of mixed speech samples containing these phonemes. Therefore, the mixed language speech model can recognize both mixed speech and any single speech in it.
[0034] For example, when using automatic speech recognition technology to recognize Chinese-English mixed speech, the Chinese-English mixed speech is regarded as a new language different from Chinese and English. The phonemes of this language include all the phonemes of Chinese and all the phonemes of English. By obtaining a large number of Chinese, English and Chinese-English mixed speech samples to train the model, a Chinese-English mixed language speech model is obtained. Therefore, the Chinese-English mixed language speech model can recognize Chinese speech, English speech alone, or Chinese-English mixed speech.
[0035] Among them, "phonemes of a language" refers to all the most basic "single-phonetic" elements in the normal pronunciation of that language, so the speech of each language is composed of many phonemes.
[0036] But obviously, building a mixed language speech model requires more sample data than building a single speech model for training, and because the mixed language speech model essentially recognizes two languages, the search space is larger during recognition, so the recognition time is higher than that of a single speech model. For example, when recognizing a Chinese-English mixed language speech model, it is necessary to search for both Chinese phonemes and English phonemes, and the search space is larger than that of single Chinese or English recognition.
[0037] First, refer to Figure 1 , an embodiment of the present disclosure provides a method for mixed speech processing.
[0038] Mixed speech includes mixed sentences using a primary language and a secondary language at the same time, wherein the primary language and the secondary language are two different predetermined languages, and the primary language and the secondary language are determined in a mixed speech processing process (or the recognition model corresponding to each embodiment of the present disclosure), but may be different in different mixed speech processing processes.
[0039] In other words, the mixed speech processing method of the embodiment of the present disclosure is based on a recognition model, which can recognize a main speech (primary speech, such as Chinese), and can also recognize the speech of a main speech mixed with a secondary language (sub-language) within a sentence (such as the mixed speech of Chinese and English), or it can recognize the main language speech mixed with a small amount of secondary language (such as Chinese speech mixed with a small amount of English), but it cannot recognize pure secondary language speech (such as a sentence of pure English speech).
[0040] For example, in one recognition process (or one recognition model), the main language of the mixed speech "turn on wifi" may be Chinese, and the corresponding secondary language may be English; of course, for another recognition process (or another recognition model), the main language may also be English, and the secondary language may be Chinese.
[0041] The method of the embodiment of the present disclosure may include the following operations:
[0042] S101: Acquire mixed speech.
[0043] The server obtains the mixed voice (such as "turn on wifi") emitted by the user through the microphone of the terminal (such as a mobile phone, smart speaker, car speaker) and other sound collecting devices.
[0044] The server refers to any device that can implement the method of the embodiment of the present disclosure, which can be located on the network side or on the terminal. If the server is located on the network side, the terminal will send the collected mixed voice to the server through the network after collecting the mixed voice emitted by the user; if the server is located on the terminal (such as the server is the processor of the terminal), the server directly obtains the collected mixed voice and processes it.
[0045] S102, obtaining a phoneme sequence to be recognized through a preset primary language speech model; the phoneme sequence to be recognized is a sequence of primary language phonemes corresponding to the mixed speech.
[0046] The server inputs the obtained mixed speech into the main language speech model to obtain a sequence of main language phonemes corresponding to the mixed speech, that is, the speech of the secondary language (such as "wifi") in the mixed speech is also processed as the speech of the main language and converted into main language phonemes.
[0047] Among them, the main language speech model is a model that is pre-trained with the main language speech samples and can convert speech signals into main language phonemes. It can be a deep learning model, such as DNN (Deep Neural Networks), LSTM (Long Short-Term Memory), CTC (Connectionist Temporal Classification) and any other suitable speech model.
[0048] S103. According to a predetermined hybrid dictionary, query and obtain at least one word sequence corresponding to the to-be-recognized phoneme sequence; the word sequence is composed of a plurality of words, each of which is a primary language word or a secondary language word. In one embodiment, the hybrid dictionary may include primary language phoneme combinations corresponding to primary language words, and primary language phoneme combinations corresponding to secondary language words.
[0049] The server searches a predetermined mixed dictionary for a word sequence corresponding to the phoneme sequence to be recognized according to the phoneme sequence to be recognized.
[0050] The mixed dictionary may include both the main language phoneme combinations corresponding to the main language words and the main language phoneme combinations corresponding to the secondary language words. In other words, the mixed dictionary marks the main language phoneme combinations corresponding to many main language words, and also corresponds the secondary language words with the main language phoneme combinations.
[0051] Since the phoneme sequence to be recognized includes multiple phonemes, which may form multiple phoneme combinations, and each phoneme combination may correspond to multiple primary language words or secondary language words in the mixed dictionary, there may be more than one word sequence obtained in this step. Therefore, after obtaining the word sequence, the obtained word sequence can be further screened (detailed description later).
[0052] The method for mixed speech processing in the embodiment of the present disclosure regards mixed speech as main language speech processing, and recognizes the mixed speech through a trained main language speech model. During the recognition process, only the main language phonemes need to be searched, the search space is relatively small, the recognition response is fast, and the time taken is short.
[0053] Obviously, for a small amount of other languages (secondary languages) mixed with one language (primary language), on the one hand, the number of words used is often relatively low. For example, the English words that are often mixed with Chinese are mainly a small number of words such as wifi and beyond. On the other hand, the actual pronunciation of the secondary language is often "primary language-ized". For example, when you say "turn on wifi", the pronunciation of "wifi" is often different from the pronunciation of "wifi" when speaking pure English.
[0054] Therefore, the above method of treating the mixed speech as the main language speech can achieve good recognition effect.
[0055] Reference Figure 2 In some embodiments, after obtaining at least one word sequence corresponding to the to-be-recognized phoneme sequence through querying according to a predetermined hybrid dictionary (S103), the method further includes:
[0056] S1041. According to a preset main language model, determine a word sequence from multiple word sequences as a recognition result.
[0057] After obtaining multiple word sequences corresponding to the mixed speech, the server sends the obtained word sequences to the main language speech model and determines one of the multiple word sequences as the recognition result. This recognition result is displayed as the sentence corresponding to the mixed speech.
[0058] Among them, the primary language language model is pre-trained, and it can calculate the relationship strength between different words (including primary language words and secondary language words) (such as the probability of appearing in the same sentence). For example, "open" and "wifi" obviously have a high probability of appearing in the same sentence, while "match" and "wifi" obviously have only a very small probability of appearing in the same sentence. By judging the probability of different words in a word sequence appearing in the same sentence, it can further judge whether the word sequence is reasonable.
[0059] Using the main language language model, multiple word sequences can be queried in the mixed dictionary according to the phoneme sequence to be recognized, and then screened to obtain a unique word sequence corresponding to the mixed speech, which is the recognition result.
[0060] Reference Figure 2 In some embodiments, before determining a word sequence as a recognition result from a plurality of word sequences according to a preset main language model, the method further includes:
[0061] S1040: Use the mixed sentence to train the main language language model.
[0062] The server obtains multiple preset, correct mixed sentences (such as "turn on wifi, what time does AC Milan's game start tonight, I want to listen to Beyond's songs", etc.) as training samples, and trains the main language language model to obtain the relationship strength between the main language words and the secondary language words, and improve its accuracy in processing the word sequence corresponding to the mixed speech. Among them, the main language language model before the mixed sentence training can be a trained model that can calculate the relationship strength between different main language words.
[0063] Reference Figure 3 In some embodiments, the process of forming a predetermined hybrid dictionary may include:
[0064] S201, obtaining a plurality of speech samples of a plurality of secondary language words, and obtaining a primary language phoneme combination of each speech sample of each secondary language word through a preset primary language speech conversion model.
[0065] The server obtains multiple speech samples of multiple sub-language words (such as sub-language words that often appear in mixed speech), inputs all the obtained speech samples into the primary language speech conversion model, and obtains the primary language phoneme combination of the sub-language word of each speech sample.
[0066] Each sub-language word has a certain number of speech samples, but a large number of samples is not required. Taking Chinese as the main language and English as the secondary language as an example, the language samples corresponding to each sub-language word generally do not exceed 50. Of course, the specific number can also be adjusted according to the specific process.
[0067] S202: Perform clustering decision on the main language phoneme combinations of all speech samples of each acquired sub-language word to obtain the main language phoneme combination corresponding to each sub-language word.
[0068] The server performs clustering decision on the main language phoneme combination recognized from all speech samples of each sub-language word, and obtains the main language phoneme combination corresponding to each sub-language word.
[0069] After obtaining the primary language phoneme combination corresponding to each secondary language word, the secondary language word and its corresponding primary language phoneme combination are written into the dictionary that has already written the primary language word and its corresponding primary language phoneme to form a mixed dictionary.
[0070] Obtaining the primary language phoneme combination corresponding to the secondary language word by clustering the recognition results of multiple samples of each secondary language word is more accurate than using the recognition result of a single sample as the primary language phoneme combination corresponding to the secondary language word, and is also simpler than using a large number of samples.
[0071] In some embodiments, the primary language speech model and the primary language speech conversion model may be the same model.
[0072] That is, the primary language speech model in step S102 may be used to obtain the primary language phoneme combination of the secondary language word of each speech sample.
[0073] The primary language speech model and the primary language speech conversion model are the same model, which can reduce the difference between the primary language phoneme combination corresponding to the same secondary language word in the mixed dictionary and the recognized primary language phoneme combination caused by different models.
[0074] Reference Figure 4 In some embodiments, clustering decision is performed on the main language phoneme combination of all speech samples of each acquired sub-language word, and acquiring the main language phoneme combination corresponding to each sub-language word (S202) may include:
[0075] S2021. Determine a target length of the primary language phoneme combination corresponding to each secondary language word according to the length of the primary language phoneme combination of all speech samples of the secondary language word.
[0076] For each sub-language word, the server first determines the length of the main language phoneme combination corresponding to the sub-language word based on the main language phoneme combinations of all its voice samples, that is, the number of main language phonemes contained in the main language phoneme combination corresponding to the sub-language word.
[0077] S2022. Determine a target main language phoneme combination from the main language phoneme combinations of all speech samples of each sub-language word, and determine the main language phoneme combination corresponding to the sub-language word based on the target main language phoneme combination, wherein the length of the target main language phoneme combination is the same as the target length.
[0078] After determining the length of the primary language phoneme combination corresponding to the secondary language word, the server determines the primary language phoneme combination corresponding to the secondary language word based on the primary language phoneme combination whose length is the determined length of the primary language phoneme combination corresponding to the secondary language word (i.e., the target primary language phoneme combination).
[0079] There is a high probability that a main language phoneme combination whose length is not the length of the main language phoneme combination corresponding to the sub-language word may be recognized inaccurately. Determining the main language phoneme combination corresponding to the sub-language word from the main language phoneme combination whose length is the length of the main language phoneme combination corresponding to the sub-language word can improve the accuracy of the final result.
[0080] Reference Figure 5 In some embodiments, determining the target length of the primary language phoneme combination corresponding to each secondary language word according to the length of the primary language phoneme combination of all speech samples of the secondary language word (S2021) may include:
[0081] S20211. Among all the lengths of the main language phoneme combination of all the speech samples of each sub-language word, the length with the largest number of speech samples corresponding to the sub-language word is the target length of the main language phoneme combination corresponding to the sub-language word.
[0082] The server counts the number of phonemes contained in the main language phoneme combination of all speech samples of each sub-language word, and the main language phoneme combination of the speech sample with the largest number of phonemes among these numbers is the length of the main language phoneme combination corresponding to the sub-language word.
[0083] Determining the primary language phoneme combination corresponding to the secondary language word according to the target primary language phoneme combination (S2022) may include:
[0084] S20221. Determine the main language phoneme with the largest number at each position in all target main language phoneme combinations of the sub-language word as the main language phoneme at that position of the main language phoneme combination corresponding to the sub-language word.
[0085] The first phoneme of all the main language phoneme combinations (i.e., the target main language phoneme combinations) whose length is the determined length of the main language phoneme combination corresponding to the sub-language word is counted, and the phoneme with the largest number of occurrences is used as the first phoneme of the main language phoneme combination corresponding to the sub-language word; the second phoneme of all the main language phoneme combinations whose length is the determined length of the main language phoneme combination corresponding to the sub-language word is counted, and the phoneme with the largest number of occurrences is used as the second phoneme of the main language phoneme combination corresponding to the sub-language word; and so on, until all the phonemes in the main language phoneme combination corresponding to the sub-language word are determined.
[0086] For more than one sub-language word, the length of the main language phoneme combination corresponding to the majority of its speech samples indicates that the length of the main language phoneme combination obtained after most of the sub-language words are fed into the model is this length, so it is reasonable to use the length of the main language phoneme combination corresponding to the majority of the speech samples as the length of the main language phoneme combination corresponding to the sub-language word. The phoneme that appears most frequently at each position also indicates that the position of the main language phoneme combination obtained after most of the sub-language words are fed into the model is this factor, so it is also reasonable to use the phoneme that appears most frequently at each position as the phoneme at that position of the main language phoneme combination corresponding to the sub-language word. Specific embodiment:
[0088] The mixed speech processing method of the embodiment of the present disclosure can be used for Chinese-English mixed speech recognition, and may include the following steps:
[0089] A01. Building a hybrid dictionary
[0090] Collect English words or English abbreviations that are often mixed in Chinese spoken words, such as iPad, wifi, GDP, CPU, WINDOWS, etc., and then record a small number of voice samples for each English word member. Taking wifi as an example, the voice samples recorded for it are recorded as W1, W2, W3..Wn, and the number of samples n here generally does not exceed 50.
[0091] Get the Chinese phoneme combination of all samples of each English word, and determine the Chinese phoneme combination corresponding to each English word. Each Chinese phoneme can be understood as a member of the Chinese initial consonant and final set. Take wifi as an example, input the voice samples W1, W2, W3..Wn of wifi into the trained Chinese acoustic model (the Chinese acoustic model used here is a single Chinese recognition model trained by Chinese speech) to obtain the Chinese phoneme combination Y1, Y2, Y3..Yn corresponding to each sample, such as Y1 is {uu uai1 f ai4}, Y2 is {uu uai1uu uai1 f ai4}, Y3 is {uu uai2 fai4}, Yn is {uu uai1 f ai3}, etc., where the characters separated by spaces are Chinese phonemes (expressed in the International Phonetic Alphabet), and the numbers represent the corresponding tones (referring to the four tones of Chinese).
[0092] The number with the largest phoneme length among Y1, Y2, Y3..Yn is taken as the final Chinese phoneme combination length of wifi. For example, the phoneme length of Y1 is 4, the phoneme length of Y2 is 6, the phoneme length of Y3 is 4, the phoneme length of Yn is 4, etc. Through statistics, it is found that among all the Chinese phoneme combinations obtained from wifi samples, the number with a length of 4 is the largest, so the final Chinese phoneme combination length of wifi is 4.
[0093] Get all the samples of wifi and obtain Chinese phoneme combinations with a length of 4 (such as Y1, Y3, Yn, etc.), and count the most common phonemes in each position from the first position to the fourth position, which are the phonemes in the final Chinese phoneme combination of wifi. For example, if the most common phoneme in the first position is uu, then the phoneme in the first position of the final Chinese phoneme combination of wifi is uu; similarly, if the most common phoneme in the second position is uai1, then the phoneme in the second position of the final Chinese phoneme combination of wifi is uai1; if the most common phoneme in the third position is f, then the phoneme in the third position of the final Chinese phoneme combination of wifi is f; if the most common phoneme in the fourth position is ai4, then the phoneme in the fourth position of the final Chinese phoneme combination of wifi is ai4. That is, the final Chinese phoneme combination of wifi is {uu uai1 f ai4}.
[0094] Similarly, obtain the Chinese phoneme combination corresponding to each English word, and write these English words and their corresponding Chinese phoneme combinations into the Chinese dictionary to obtain a mixed dictionary. Here, the Chinese dictionary contains all the Chinese words and their corresponding Chinese phoneme combinations.
[0095] A02. Build Chinese and English language models.
[0096] Construct common Chinese sentences containing these English words, such as "turn on wifi", "when does AC Milan's game start tonight", "I want to listen to Beyond's songs", etc. These sentences are combined into a sample set to train the Chinese language model to obtain a Chinese-English language model. The Chinese language model is a model that has been trained using Chinese sentences to convert Chinese phonemes into Chinese sentences.
[0097] A03. Obtain the mixed speech to be recognized and recognize it.
[0098] The user's mixed Chinese and English speech is obtained through a microphone or other device, and the speech is input into a Chinese acoustic model to obtain its corresponding Chinese phoneme combination. The corresponding word sequence is queried in a hybrid dictionary based on the Chinese phoneme combination, and the obtained word sequence is sent to the Chinese and English language model to obtain its corresponding Chinese and English sentence.
[0099] Second, refer to Figure 6 , the embodiment of the present disclosure provides a device 1 for mixed speech processing, the mixed speech includes mixed sentences using a primary language and a secondary language at the same time, the primary language and the secondary language are two different predetermined languages;
[0100] The mixed speech processing device 1 of the embodiment of the present disclosure may include:
[0101] An acquisition module 11 is used to acquire mixed speech;
[0102] The recognition module 12 is used to obtain a phoneme sequence to be recognized through a preset primary language speech model; the phoneme sequence to be recognized is a sequence of primary language phonemes corresponding to the mixed speech;
[0103] The query module 13 is used to query and obtain at least one word sequence corresponding to the phoneme sequence to be recognized according to a predetermined hybrid dictionary; the word sequence is composed of multiple words, each word is a main language word or a secondary language word; the hybrid dictionary includes a main language phoneme combination corresponding to the main language word, and a main language phoneme combination corresponding to the secondary language word.
[0104] Thirdly, refer to Figure 7 The present disclosure provides an electronic device 2, which includes:
[0105] one or more processors 21,
[0106] A memory 22, on which one or more programs are stored, when the one or more programs are executed by the one or more processors 21, the one or more processors 21 implement any one of the above-mentioned mixed speech processing methods;
[0107] One or more I / O interfaces 23 are connected between the processor 21 and the memory 22 to implement information exchange between the processor 21 and the memory 22 .
[0108] Among them, the processor 21 is a device with data processing capabilities, including but not limited to a central processing unit (CPU); the memory 22 is a device with data storage capabilities, including but not limited to random access memory (RAM, more specifically SDRAM, DDR, etc.), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory (FLASH); the I / O interface (read-write interface) 23 is connected between the processor 21 and the memory 22, and is used to realize information interaction between the memory 22 and the processor 21, including but not limited to a data bus (Bus), etc.
[0109] In some implementations, the memory 22 may also be a storage device externally connected to the electronic device 2 .
[0110] Fourthly, refer to Figure 8 The embodiment of the present disclosure provides a computer-readable medium 3 on which a computer program is stored. When the program is executed by a processor, any of the above-mentioned mixed speech processing methods is implemented.
[0111] Those skilled in the art will appreciate that all or some of the steps, systems, and functional modules / units in the apparatus disclosed above may be implemented as software, firmware, hardware, or a suitable combination thereof.
[0112] In hardware implementations, the division between functional modules / units mentioned in the above description does not necessarily correspond to the division of physical components; for example, one physical component may have multiple functions, or one function or step may be performed by several physical components in cooperation.
[0113] Some physical components or all physical components may be implemented as software executed by a processor, such as a central processing unit (CPU), a digital signal processor or a microprocessor, or implemented as hardware, or implemented as an integrated circuit, such as an application-specific integrated circuit. Such software may be distributed on a computer-readable medium, which may include a computer storage medium (or non-transitory medium) and a communication medium (or temporary medium). As known to those of ordinary skill in the art, the term computer storage medium includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information (such as computer-readable instructions, data structures, program modules or other data). Computer storage media include, but are not limited to, random access memory (RAM, more specifically SDRAM, DDR, etc.), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory (FLASH) or other disk storage; read-only compact disk (CD-ROM), digital versatile disk (DVD) or other optical disk storage; magnetic cassettes, tapes, disk storage or other magnetic storage; any other medium that can be used to store desired information and can be accessed by a computer. Furthermore, it is well known to those skilled in the art that communication media typically embodies computer readable instructions, data structures, program modules, or other data in a modulated data signal such as a carrier wave or other transport mechanism, and may include any information delivery media.
[0114] The present disclosure has disclosed example embodiments, and although specific terms are employed, they are used and should be interpreted only in a general illustrative sense and not for limiting purposes. In some instances, it will be apparent to those skilled in the art that, unless otherwise expressly stated, features, characteristics, and / or elements described in conjunction with a particular embodiment may be used alone or in combination with features, characteristics, and / or elements described in conjunction with other embodiments. Therefore, those skilled in the art will appreciate that various changes in form and detail may be made without departing from the scope of the present disclosure as set forth in the appended claims.
Claims
1. A method for processing mixed speech, wherein the mixed speech includes mixed sentences using a primary language and a secondary language at the same time, wherein the primary language and the secondary language are two different predetermined languages; the method comprises: Acquire the mixed voice; Obtaining a phoneme sequence to be recognized through a preset primary language speech model; the phoneme sequence to be recognized is a sequence of primary language phonemes corresponding to the mixed speech; According to a predetermined hybrid dictionary, query and obtain at least one word sequence corresponding to the to-be-recognized phoneme sequence; the word sequence is composed of a plurality of words, each word being a primary language word or a secondary language word; the hybrid dictionary includes a primary language phoneme combination corresponding to the primary language word, and a primary language phoneme combination corresponding to the secondary language word; The process of forming the predetermined hybrid dictionary includes: Acquire multiple speech samples of multiple sub-language words, and acquire a primary language phoneme combination of each speech sample of each sub-language word through a preset primary language speech conversion model; A clustering decision is made on the main language phoneme combination of all speech samples of each acquired sub-language word to obtain the main language phoneme combination corresponding to each sub-language word.
2. The method according to claim 1, wherein: The main language speech model and the main language speech conversion model are the same model.
3. The method according to claim 1, wherein: The clustering decision is performed on the main language phoneme combination of all speech samples of each acquired sub-language word to obtain the main language phoneme combination corresponding to each sub-language word, which includes: Determining a target length of the main language phoneme combination corresponding to each sub-language word according to the length of the main language phoneme combination of all speech samples of each sub-language word; A target main language phoneme combination is determined from the main language phoneme combinations of all speech samples of each sub-language word, and the main language phoneme combination corresponding to the sub-language word is determined based on the target main language phoneme combination, and the length of the target main language phoneme combination is the same as the target length.
4. The method according to claim 3, wherein: Determining the target length of the primary language phoneme combination corresponding to each secondary language word according to the length of the primary language phoneme combination of all speech samples of the secondary language word comprises: Among all the lengths of the main language phoneme combination of all the speech samples of each sub-language word, the length with the largest number of speech samples corresponding to the sub-language word is taken as the target length of the main language phoneme combination corresponding to the sub-language word.
5. The method according to claim 3, wherein: The step of determining the primary language phoneme combination corresponding to the secondary language word according to the target primary language phoneme combination includes: The main language phoneme with the largest number at each position in all target main language phoneme combinations of the sub-language word is determined as the main language phoneme at that position of the main language phoneme combination corresponding to the sub-language word.
6. The method according to claim 1, wherein: After obtaining at least one word sequence corresponding to the to-be-recognized phoneme sequence through querying according to a predetermined hybrid dictionary, the method further includes: According to the preset main language language model, a word sequence is determined from multiple word sequences as the recognition result.
7. The method according to claim 6, wherein: Before determining a word sequence as a recognition result from a plurality of word sequences according to a preset main language language model, the method further includes: The primary language language model is trained using the mixed sentences.
8. An electronic device, comprising: one or more processors; A memory having one or more programs stored thereon, wherein when the one or more programs are executed by the one or more processors, the one or more processors implement the hybrid speech processing method according to any one of claims 1 to 7; One or more I / O interfaces are connected between the processor and the memory to implement information exchange between the processor and the memory.
9. A computer-readable medium having a computer program stored thereon, wherein when the program is executed by a processor, the mixed speech processing method according to any one of claims 1 to 7 is implemented.
