Voice interaction method, server and storage medium

Abstract

The application provides a voice interaction method, a server and a storage medium. The voice interaction method comprises the following steps: obtaining phonemes of different languages, and merging the phonemes of different languages into first syllables according to pronunciation rules; identifying combined syllables of pronunciation by using training materials of different languages, and screening second syllables from the combined syllables according to pronunciation adhesion; merging the first syllables and the second syllables to obtain modeling syllables; generating an acoustic model according to the modeling syllables; receiving a voice request issued by a user in a vehicle cabin forwarded by a vehicle; identifying the voice request according to the acoustic model to generate an identification result and issuing the identification result to the vehicle to complete voice interaction. The scheme provided by the application can realize unified modeling of multiple languages, is more convenient for voice recognition and voice interaction of different languages, reduces deployment cost, and avoids waste of machine resources.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of voice, in particular to a voice interaction method, a server and a storage medium. BACKGROUND

[0002] With the continuous development of the automobile industry and human-computer interaction technology, intelligent vehicles also provide voice interaction functions for users.

[0003] The voice interaction function relies on voice recognition technology. At present, the voice data to be recognized may not be only single language voice, but also bilingual mixed voice or multilingual mixed voice, etc., so the voice recognition methods are also different. For example, in Europe, there are many languages and multiple language families, and the number of languages exceeds ten. In the related technology, a context-dependent syllable modeling method is generally used to construct an acoustic model, but the scene is strongly dependent, and it is not possible to achieve unified modeling of multiple languages, which is not conducive to voice recognition and voice interaction of different languages. If a set of voice recognition systems is deployed for each language, the cost will be greatly increased, and machine resources will be wasted. SUMMARY

[0004] To solve or partially solve the problems in the related technology, the present application provides a voice interaction method, a server and a storage medium, which can realize unified modeling of multiple languages, more conveniently voice recognition and voice interaction of different languages, reduce deployment cost and avoid waste of machine resources.

[0005] The first aspect of the present application provides a voice interaction method, comprising: obtaining phonemes of different languages, merging the phonemes of different languages into first syllables according to pronunciation rules; identifying combined syllables of pronunciation by using training materials of different languages, and screening second syllables from the combined syllables according to pronunciation adhesion; merging the first syllables and the second syllables to obtain modeling syllables; generating an acoustic model according to the modeling syllables; receiving a voice request issued by a user in a vehicle cabin forwarded by a vehicle; identifying the voice request according to the acoustic model to generate an identification result and issuing the identification result to the vehicle to complete voice interaction. According to the present application, the phonemes of different languages are merged into first syllables according to pronunciation rules, and the second syllables are screened from the combined syllables of the training materials of different languages according to pronunciation adhesion, and then the modeling syllables are obtained by merging. These syllables are independent of context, so that various different languages can use the same modeling system, thereby realizing unified modeling of multiple languages, more conveniently voice recognition and voice interaction of different languages, reducing deployment cost and avoiding waste of machine resources.

[0006] The merging the phonemes of the different languages into the first syllable according to the phonetic rules comprises: pre-merging the phonemes of the different languages according to the IPA rules; and merging the pre-merged phonemes into the first syllable according to the phonetic rules. The phoneme merging based on the IPA can reduce the number of phonemes, can enable a plurality of mixed languages to use one output layer, and can reduce the calculation amount and the latency.

[0007] The merging the phonemes of the different languages into the first syllable according to the phonetic rules comprises: from the pre-merged phonemes, merging the phonemes of the initial and the rhyme to obtain the first syllable, and separately taking the phonemes of the single initial and the single rhyme as the first syllable. The phoneme merging into the syllable can be performed according to the phonetic rules, so that the merged syllable is more in line with the pronunciation habit.

[0008] The identifying the combined syllable of pronunciation by using the training materials of different languages comprises: identifying the combined syllable of pronunciation by using the audio and / or video training materials of different languages. The audio and / or video can be fully used as the training materials of different languages.

[0009] The screening the second syllable from the combined syllable according to the pronunciation adhesion degree comprises: performing forced frame alignment on the combined syllable to obtain the average pronunciation duration of the combined syllable and the average pronunciation duration of all the combined syllables; taking the ratio of the average pronunciation duration of the combined syllable to the average pronunciation duration of all the combined syllables as the pronunciation adhesion degree, and taking the combined syllable with the pronunciation adhesion degree less than a set threshold as the second syllable. The syllable is screened according to the pronunciation adhesion degree, so that the screened syllable is more accurate.

[0010] The method further comprises: before the performing forced frame alignment on the combined syllable, screening the combined syllable that meets the syllable merging rule from the combined syllable; and the performing forced frame alignment on the combined syllable comprises: performing forced frame alignment on the combined syllable that meets the syllable merging rule. The syllable merging rule is used to filter before the forced frame alignment, so that the workload of subsequent screening is reduced and the processing efficiency is improved.

[0011] The syllable merging rule comprises at least one of the following rules: initial+initial+rhyme; initial+initial+rhyme+special initial; initial+rhyme+special initial. The syllable merging rule can be applied to a plurality of situations.

[0012] The average pronunciation duration of the combined syllable is determined according to the ratio of the total pronunciation duration after the forced frame alignment to the number of times that the combined syllable appears in the training audio. The average pronunciation duration is obtained through mean processing, so that the parameter is more accurate.

[0013] The forced frame alignment of the combined syllable includes: performing forced frame alignment on the combined syllable with a rime as a core. By performing forced frame alignment with a rime as a core, the actual pronunciation situation is more in line with.

[0014] The second aspect of the application provides a server, comprising: a phoneme processing module, configured to obtain phonemes of different languages, and merge the phonemes of different languages into first syllables according to pronunciation rules; a training processing module, configured to identify combined syllables of pronunciation by using training materials of different languages, and screen second syllables from the combined syllables according to pronunciation adhesion; a syllable merging module, configured to merge the first syllables obtained by the phoneme processing module and the second syllables obtained by the training processing module to obtain modeling syllables; a model generation module, configured to generate an acoustic model according to the modeling syllables; a request receiving module, configured to receive a voice request issued by a user in a vehicle cabin forwarded by a vehicle; a voice recognition module, configured to recognize the voice request received by the request receiving module according to the acoustic model generated by the model generation module, and generate a recognition result to be sent to the vehicle to complete voice interaction. According to the application, the phonemes of different languages are merged into first syllables according to pronunciation rules, and the second syllables are screened from the combined syllables of the training materials of different languages according to pronunciation adhesion, and then the modeling syllables are obtained by merging. These syllables are independent of context, so that the same set of modeling systems can be used for various different languages, thereby realizing unified modeling of multiple languages, facilitating voice recognition and voice interaction of different languages, reducing deployment cost, and avoiding waste of machine resources.

[0015] The phoneme processing module comprises: a pre-merging sub-module, configured to pre-merge the phonemes of different languages according to the IPA rules; and a phoneme merging sub-module, configured to merge the phonemes pre-merged into first syllables according to pronunciation rules. According to the application, phoneme merging is performed based on IPA, which can reduce the number of phonemes, use one output layer for tens of mixed languages, and reduce the amount of calculation and delay.

[0016] The training processing module comprises: an alignment and statistics module, configured to perform forced frame alignment on the combined syllables, and obtain the average pronunciation duration of the combined syllables after alignment and the average pronunciation duration of all combined syllables; and a syllable screening module, configured to take the ratio of the average pronunciation duration of the combined syllables to the average pronunciation duration of all combined syllables as the pronunciation adhesion, and take the combined syllables with the pronunciation adhesion less than a set threshold as the second syllables. According to the application, the syllables are screened according to the pronunciation adhesion, which can make the screened syllables more accurate.

[0017] The third aspect of the present application provides a server, comprising: a processor; and a memory having executable code stored thereon, which, when executed by the processor, causes the processor to perform the method as described above.

[0018] The fourth aspect of the present application provides a computer-readable storage medium having executable code stored thereon, which, when executed by a processor of an electronic device, causes the processor to perform the method as described above.

[0019] It should be understood that the general description above and the following detailed description are only exemplary and explanatory, and are not limiting to the present application. BRIEF DESCRIPTION OF DRAWINGS

[0020] The above and other objects, features and advantages of the present application will become more apparent from the following detailed description when taken in conjunction with the accompanying drawings in which like reference characters refer to like parts throughout the several views.

[0021] Figure 1 is a flowchart of a voice interaction method shown in the present application;

[0022] Figure 2 is a flowchart of another voice interaction method shown in the present application;

[0023] Figure 3 is a flowchart of voice modeling in a voice interaction method shown in the present application;

[0024] Figure 4 is an application framework diagram of applying voice modeling for voice recognition shown in the present application;

[0025] Figure 5 is a comparison diagram of context-dependent modeling and context-independent modeling shown in the present application;

[0026] Figure 6 is a modeling unit diagram shown in the present application;

[0027] Figure 7 is a structure diagram of a server shown in the present application;

[0028] Figure 8 is a structure diagram of another server shown in the present application;

[0029] Figure 9 is another structure diagram of a server shown in the present application. DETAILED DESCRIPTION

[0030] Embodiments of the present application will be described in more detail with reference to the drawings. Although embodiments of the present application are shown in the drawings, it should be understood that the present application can be implemented in various forms and should not be limited by the embodiments set forth herein. Rather, these embodiments are provided so that the present application will be thorough and complete, and will fully convey the scope of the present application to those skilled in the art.

[0031] The terminology used in this application is for the purpose of describing particular embodiments only and is not intended to be limiting of the present application. As used in this application and the appended claims, the singular forms "a," "an" and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will also be understood that the term "and / or" as used herein refers to and encompasses any and all possible combinations of one or more of the associated listed items.

[0032] It should be understood that although the terms "first", "second", "third", etc. can be employed in this application to describe various information, these information should not be limited by these terms. These terms are only used to distinguish one piece of information from another piece of information of the same type. For example, the first information can also be referred to as the second information, and similarly, the second information can also be referred to as the first information without departing from the scope of the present application. Therefore, the features defined with "first", "second" can explicitly or implicitly include one or more of the features. In the description of the present application, the meaning of "a plurality of" is two or more, unless otherwise specifically limited.

[0033] In the related art, a context-dependent syllable modeling method is generally used to construct an acoustic model, which cannot achieve unified modeling of multiple languages, and is not conducive to speech recognition and voice interaction of different languages. The present application provides a voice interaction method, which can achieve unified modeling of multiple languages, is more convenient for speech recognition and voice interaction of different languages, reduces deployment cost, and avoids waste of machine resources.

[0034] The technical solutions of the present application will be described in detail below with reference to the drawings.

[0035] Figure 1 is a flowchart of the voice interaction method shown in the present application. The method can be applied to a server.

[0036] Referring to Figure 1 , the method comprises:

[0037] S101, obtaining phonemes of different languages, and merging the phonemes of different languages into a first syllable according to pronunciation rules.

[0038] The phonemes of different languages can be pre-merged according to the IPA rules. For example, after the English phonemes and the French phonemes are merged using the IPA, there are only 56 phonemes, thereby reducing the number of phonemes through the merging process. Then, the phonemes after pre-merging can be merged into the first syllable according to the phonetic rules. For example, the phonemes of the initial and the phonemes of the final can be merged from the phonemes after pre-merging to obtain the first syllable, and the phonemes of the single initial and the phonemes of the single final are separately taken as the first syllable.

[0039] S102, identifying the combined syllable of the pronunciation by using the training materials of different languages, and screening the second syllable from the combined syllable according to the pronunciation adhesion.

[0040] The combined syllable of the pronunciation can be identified by using the training materials of different languages in the form of audio and / or video. The present application can make full use of audio and / or video as training materials of different languages.

[0041] The combined syllable can be forcedly frame-aligned, the average pronunciation duration of the aligned combined syllable and the average pronunciation duration of all the combined syllables are determined, the ratio of the average pronunciation duration of the combined syllable to the average pronunciation duration of all the combined syllables is taken as the pronunciation adhesion, and the combined syllable with the pronunciation adhesion less than a set threshold is taken as the second syllable. The smaller the pronunciation adhesion is, the more the phonemes in the combined syllable should be merged together.

[0042] The syllable merging rules include at least one of the following rules: initial+initial+final, initial+initial+final+special initial, initial+final+special initial.

[0043] The average pronunciation duration of the combined syllable is determined according to the ratio of the total pronunciation duration after forced frame alignment to the number of times of occurrence of the combined syllable in the training audio.

[0044] The forced frame alignment of the combined syllable includes: performing forced frame alignment on the combined syllable with the final as the core.

[0045] It should be noted that there is no sequence relationship between S101 and S102.

[0046] S103, merging the first syllable and the second syllable to obtain the modeling syllable.

[0047] The first syllable and the second syllable obtained by the above different steps are merged to generate the final syllable, i.e., the modeling syllable.

[0048] S104, generating an acoustic model according to the modeling syllable.

[0049] After obtaining the modeling syllable, an acoustic model can be generated according to the modeling syllable by using the existing related technologies.

[0050] S105, receiving a voice request issued by a user in a vehicle cabin forwarded by a vehicle.

[0051] The server can receive a voice request issued by a user in a vehicle cabin forwarded by a vehicle. The voice request can be a French voice request, a German voice request, etc.

[0052] S106, identifying the voice request according to an acoustic model to generate an identification result and issuing the identification result to the vehicle to complete voice interaction.

[0053] After the server receives the voice request issued by the user, the voice request is identified according to an acoustic model to generate an identification result and issuing the identification result to the vehicle to complete voice interaction. The method for identifying the voice request according to the acoustic model can use the existing identification method in the related art, which is not limited in the present application.

[0054] According to the present application, different phonemes of different languages are merged into first syllables according to pronunciation rules, and second syllables are selected from combined syllables of training materials of different languages according to pronunciation adhesion, and then modeling syllables are obtained by merging. These syllables are independent of context, so that the same set of modeling systems can be used for various different languages, thereby realizing unified modeling of multiple languages, facilitating voice recognition and voice interaction of different languages, reducing deployment cost, and avoiding waste of machine resources.

[0055] Figure 2 is another flowchart of a voice interaction method according to the present application. The method can be applied to a server. Figure 2 In the method, training materials of different languages are taken as audio training materials (referred to as training audio) as an example but not limited thereto, and video training materials (referred to as training video) can also be used.

[0056] Referring to Figure 2 The method comprises the following steps.

[0057] S201, obtaining phonemes of different languages, pre-merging the phonemes of different languages according to the International Phonetic Alphabet rules, and merging the pre-merged phonemes into first syllables according to pronunciation rules.

[0058] Among them, the phonemes of initials and vowels can be merged from the pre-merged phonemes to obtain the first syllables, and the phonemes of single initials and single vowels are separately taken as the first syllables.

[0059] S202, obtaining training audio of different languages, identifying combined syllables from the training audio, and selecting combined syllables that meet the syllable merging rules from the combined syllables.

[0060] The syllable combination rule includes at least one of the following rules: initial + initial + final; initial + initial + final + special initial; and initial + final + special initial.

[0061] It should be noted that there is no sequential relationship between S201 and S202.

[0062] S203, the combined syllable meeting the syllable combination rule is forced to frame alignment, and the average pronunciation duration of the aligned combined syllable and the average pronunciation duration of all combined syllables are determined.

[0063] The combined syllable can be forced to frame alignment with the final as the core.

[0064] The average pronunciation duration of the combined syllable is determined according to the ratio of the total pronunciation duration after forced frame alignment to the number of times the combined syllable appears in the training audio.

[0065] S204, the ratio of the average pronunciation duration of the combined syllable to the average pronunciation duration of all combined syllables is taken as the pronunciation adhesion, and the combined syllable with a pronunciation adhesion less than a set threshold is taken as a second syllable.

[0066] S205, the first syllable and the second syllable are combined to obtain a modeling syllable.

[0067] S205 can refer to the description of S103, which will not be repeated here.

[0068] S206, generating an acoustic model according to the modeling syllable.

[0069] S206 can refer to the description of S104, which will not be repeated here.

[0070] S207, receiving a voice request issued by a user in a vehicle cabin forwarded by a vehicle.

[0071] The server can receive a voice request issued by a user in a vehicle cabin forwarded by a vehicle. The voice request can be a French voice request, a German voice request, etc.

[0072] S208, identifying the voice request according to the acoustic model to generate an identification result and issuing the identification result to the vehicle to complete voice interaction.

[0073] After the server receives the voice request issued by the user, the voice request is identified according to the acoustic model to generate an identification result and issue the identification result to the vehicle to complete voice interaction. The method of identifying the voice request according to the acoustic model can use the existing identification method in related technologies, which is not limited in the present application.

[0074] The phoneme merging based on the IPA can enable a mixed use of tens of languages to use one softmax (logistic regression) output layer, thereby reducing the amount of calculation and latency. The syllable generation by fusing pronunciation rules and statistical results of training audio data can increase the modeling unit of a single language from the order of 40 to the order of 500, thereby greatly reducing the learning difficulty, improving the speech recognition rate, and facilitating speech recognition and voice interaction of different languages.

[0075] Figure 3 is a flowchart of a speech modeling process in a voice interaction method according to an embodiment of the present application. The method can be applied to a server. Figure 3 In the method, training materials in different languages are taken as audio training materials (referred to as training audio) for example. The training materials in different languages in the present application can be audio training materials or video training materials, or audio training materials and video training materials are used together.

[0076] Referring to Figure 3 The method comprises the following steps.

[0077] S301, obtaining phonemes in different languages.

[0078] Taking Europe as an example, there are several language families and more than ten languages in Europe.

[0079] This step can obtain phonemes in different languages, for example, French phonemes, English phonemes, German phonemes, etc. For example, the obtained French phonemes include: brem…; and the obtained German phonemes include: kam…

[0080] A phoneme (phone) is the smallest unit of speech divided according to the natural properties of speech, analyzed according to the pronunciation action in a syllable, and one action constitutes one phoneme. Phonemes can be divided into two categories: vowels and consonants. For example, the Chinese syllable "ah" (ā) has only one phoneme, "love" (ài) has two phonemes, and "generation" (dài) has three phonemes. Phonemes are also the smallest unit or smallest speech segment that constitutes a syllable. The phonetic symbol of the International Phonetic Alphabet (also known as "Universal Phonetics") corresponds to the phonemes of all human languages.

[0081] A syllable is the smallest unit of speech in which a single vowel phoneme and a consonant phoneme are combined to pronounce. In phonetics, a syllable refers to a basic unit of speech structure composed of one or more phonemes; and a phoneme is the smallest unit of speech. For example, the syllable of the Chinese character "good" is h_ao3, in which 3 represents the tone. It should be noted that the syllable can also not have a tone.

[0082] S302, pre-merging the phonemes in different languages according to the IPA.

[0083] Different languages in Europe, many languages actually belong to the same language family, and there are many commonalities in pronunciation. The present application fully considers the correlation between different languages in Europe, and merges the phonemes of different languages using the international standard ipa (International Phonetic Alphabet, also known as International Phonetic Alphabet). IPA is a system for marking pronunciation, based on Latin letters, designed by the International Phonetic Association as a standardized notation method for spoken language sounds. The feature of IPA is that the more similar the languages, the more phonemes coincide. For example, in IPA, English has 39 phonemes, French has 36 phonemes, but 19 of them coincide, so after merging the English phonemes and French phonemes using IPA, there are only 56 phonemes, thereby reducing the number of phonemes through merging processing.

[0084] Since the phonemes of multiple different languages are merged, multiple languages can be directly modeled in one model, with only one output layer, but containing all languages, so there is no need to perform language discrimination processing. Through the ipa-based phoneme merging, multiple languages can share modeling units and thus share part of the data, so that different languages can improve each other's effects.

[0085] S303, merging the pre-merged phonemes into a first syllable according to the phonology rules.

[0086] In S303, the phonemes of the initial and the phonemes of the final can be merged from the phonemes pre-merged by IPA to obtain the first syllable, and the phonemes of the single initial and the phonemes of the single final are separately taken as the first syllable.

[0087] The phonology rules of the present application can be to merge the initial and the final as one syllable, and the single initial and the single final left after merging are also taken as one syllable. After merging based on the phonology rules, the syllable obtained generally contains 1-2 phonemes. The initial, which is the consonant before the final, together with the final forms a complete syllable. The consonant is the sound produced by the airflow in the oral cavity when it is blocked by various obstacles. The final composed of a single vowel is called a single final, also called a monophthong final. Some syllables have no initial at the beginning, only a final, which can also be an independent syllable.

[0088] For example: salad (contains 5 phonemes: s ae l ax d) merged -> (3 syllables: s_ae l_axd), so 5 phonemes are merged into 3 syllables.

[0089] For example, the acquired French phoneme brem… and the acquired German phoneme kam… are merged according to pronunciation rules to obtain syllables b_e, r_e, k_a, etc. as first syllables. In the first syllable, generally 1-2 phonemes are included.

[0090] The present application performs phoneme merging based on IPA and is context-independent, so that the context-dependent triphones are converted into context-independent syllables, the number of pronunciation units is sufficient, the distinction between them is greater, and it is easier to learn, while the number of context-independent syllables is taken into account, and the larger pronunciation unit has stronger anti-noise capability.

[0091] Context-dependent refers to the modeling unit and the context in which it is located. The same pronunciation symbol in different contexts is also a different modeling unit. Context-independent refers to the same modeling unit as long as the pronunciation symbol is the same, regardless of the context.

[0092] Using the context-dependent modeling method, the distinction is strong, the same modeling unit has the same context, and the modeling unit is easy to learn, but one of the disadvantages is that the number of modeling units is large. For example, assuming that there are 10 languages, each language has 30 phonemes, then the modeling unit of the bi-phoneme is 300*300=90000, and the modeling unit of the tri-phoneme is 300*300*300. The system generally cannot accept a large number of modeling units. Therefore, multi-language generally cannot use context-dependent modeling units. Another disadvantage of the context-dependent modeling method is poor migration. If it is highly bound to the training corpus, for example, a model trained with music corpus is difficult to migrate to navigation. Therefore, the related art uses context-dependent modeling method, which has strong scene dependence and cannot achieve unified modeling of multi-language.

[0093] Figure 5 is a comparison diagram of the context-dependent modeling and the context-independent modeling shown in the present application. Referring to Figure 5 , take the ae pronunciation symbol as an example. For the English word happy, the phonemes include haepiy, and for the English word salad, the phonemes include saelaxd. If the context-dependent modeling method is used, two modeling units h_ae and p_iy are included, and if the context-independent modeling method of the present application is used, only one modeling unit ae is included.

[0094] Further referring to Figure 6 , Figure 6 is a modeling unit diagram shown in the present application. As Figure 6As shown, the left box is the context-dependent phoneme, x represents the context; the middle box is the context-independent phoneme, and the two ae represent the same modeling unit; and the right box is the context-independent syllable after the context-independent phonemes are merged.

[0095] S304, obtaining training audio of different languages, and identifying the combined syllables from the training audio.

[0096] This step can obtain training audio of different languages, such as French training audio, English training audio, German training audio, etc.

[0097] Using existing speech recognition related technology, the combined syllables can be identified from the training audio, such as b_r_e, b_e_m, r_e_m, r_e_b, etc.

[0098] It should be noted that there is no sequence relationship between S304 and S301.

[0099] S305, screening the combined syllables that meet the syllable merging rule from the combined syllables.

[0100] After identifying the combined syllables from the training audio, it is determined whether the combined syllables meet the syllable merging rule, and the combined syllables that meet the syllable merging rule are screened according to the determination result.

[0101] The syllable merging rule includes at least one of the following rules: initial+initial+final; initial+initial+final+special initial; initial+final+special initial.

[0102] 1) initial+initial+final

[0103] 2) initial+initial+final+special initial

n / m

[0104] For example, b_r_i_n and b_r_e_m, where the special initial n / m is a monophone that is generally pronounced shorter and has a cohesive monophone in monophone statistics.

[0105] 3) initial+final+special initial

n / m

[0106] After identifying the combined syllables such as b_r_e, b_e_m, r_e_m, and r_e_b from the training audio, the combined syllables that meet the syllable merging rule can be screened according to the syllable merging rule, such as b_r_e, b_e_m, and r_e_m. r_e_b is excluded because it is the end of the initial, not the end of the final or the special initial, so it does not meet the syllable merging rule.

[0107] S306, performing forced frame alignment on the combined syllables that meet the syllable merging rule.

[0108] From the training audio, the combined syllable meeting the syllable merging rule is forced frame-aligned to obtain a forced frame alignment result.

[0109] The forced frame alignment refers to a process of obtaining a labeled frame corresponding to each frame, given the audio and the corresponding text label. The combined syllable in the training audio can be forced frame-aligned with the core being a rime.

[0110] For example, for a 200-frame audio, the text label is n i3 h ao3 (where 3 represents a tone), and the forced frame alignment result is n(1-30)i3(31-100)h(101-120)ao3(121-200). That is, the first 30 frames are n, the 31st-100th frames are i3, the 101st-120th frames are h, and the 121st-200th frames are ao3.

[0111] It should be noted that, in speech modeling, tone modeling can be performed or not.

[0112] S307, determine the average pronunciation duration of the aligned combined syllable and the average pronunciation duration of all combined syllables.

[0113] After obtaining the forced frame alignment result, for the aligned combined syllable, the pronunciation duration of all combined syllables (containing phoneme combinations) with lengths of 3 and 4 in different context is counted, and the average value is taken as the average pronunciation duration of the combined syllable (belonging to a consonant-rime combination). The average pronunciation duration of the combined syllable can be determined according to the ratio of the total pronunciation duration after forced frame alignment to the number of times the combined syllable appears in the training audio. In addition, the average pronunciation duration of all combined syllables is also counted.

[0114] For example, the three phonemes b, r, and e of the combined syllable b_r_e appear together 1000 times in the training audio, and the total pronunciation duration of (b+r+e) after alignment is 500s, so the average pronunciation duration of the three phoneme combinations b, r, and e is N(b_r_e) = 500 / 1000 = 0.5s.

[0115] For another example, the a_b phoneme combination appears 1000 times in 10000 sentences, and the average pronunciation duration of the a_b phoneme combination can be obtained by adding up the durations of the 1000 times and dividing by 1000.

[0116] S308, determine the pronunciation adhesion, and take the combined syllable with a pronunciation adhesion less than a set threshold as a second syllable.

[0117] The ratio of the average pronunciation duration of the combined syllable to the average pronunciation duration of all combined syllables can be used as the pronunciation adhesion degree. The combined syllable with a pronunciation adhesion degree less than a set threshold value is the second syllable. The smaller the pronunciation adhesion degree, the more the phonemes in the combined syllable should be combined.

[0118] Assuming that the set threshold value is 0.5, when the pronunciation adhesion degree is less than 0.5, the combined syllable with adhesion is output as the second syllable.

[0119] The following illustrates the determination process of the pronunciation adhesion degree in different situations in the syllable combination rule, where xym represents the initial, z represents the final, s represents any initial / final, N represents the average value of the pronunciation duration, p represents the pronunciation adhesion degree, and T represents the pronunciation duration.

[0120] 1) Initial+Initial+Final: Pronunciation adhesion degree of x,y,z

[0121]

[0122] 2) Initial+Initial+Final+Special Initial: Pronunciation adhesion degree of x,y,z,m

[0123]

[0124] 3) Initial+Final+Special Initial: Pronunciation adhesion degree of x,z,m

[0125]

[0126] For example, the pronunciation adhesion degree of x,y,z, the denominator in the formula represents the average value (average pronunciation duration) of all appearing phoneme combinations x,s,z (all combined syllables); the numerator represents the average value (average pronunciation duration) of the phoneme combination x,y,z (aligned combined syllable). The smaller the pronunciation adhesion degree p, the shorter the duration when x,y,z appears, the more adhesion, and the more they should be combined.

[0127] For example:

[0128] The pronunciation adhesion degree of the combined syllable b_r_e is:

[0129] p(b_r_e) = N(b_r_e) / N(b_*_e)

[0130] Where * represents all initials, N represents the average value of the pronunciation duration, N(b_r_e) represents the average pronunciation duration of the combined syllable b_r_e, and N(b_*_e) represents the average pronunciation duration of all combined syllables of all initials combined with b and e.

[0131] b_r_e If the cohesiveness is high, the average pronunciation duration of b_r_e is generally shorter than the average pronunciation duration of b_*_e.

[0132] The pronunciation cohesiveness of the combined syllable r_e_m is:

[0133] p(r_e_m) = N(r_e_m) / N(r_e_*)

[0134] wherein * represents all initials, N represents the average value of pronunciation duration, N(r_e_m) represents the average pronunciation duration of the combined syllable r_e_m, and N(r_e_*) represents the average pronunciation duration of all combined syllables of all initials combined with r and e.

[0135] For example, the combined syllables b_r_e, b_e_m, and r_e_m that meet the syllable combination rule are subjected to pronunciation cohesiveness judgment, and the combined syllables b_r_e and b_e_m that meet the conditions are obtained as the second syllables. The second syllables after pronunciation cohesiveness judgment generally contain 3-4 phonemes.

[0136] S309, merging the first syllable and the second syllable to obtain a modeling syllable.

[0137] wherein the first syllable and the second syllable obtained in the above different steps are merged to generate the final syllable, i.e., the modeling syllable.

[0138] For example, the first syllables b_e, r_e, and k_a and the second syllables b_r_e and b_e_m are merged to obtain the final modeling syllables b_e, r_e, k_a, b_r_e, and b_e_m.

[0139] The modeling method of the application can be independent of context, which can compress the number of modeling units, different contexts can share the same modeling unit, so that dozens of languages can use the same set of modeling system, thereby realizing unified modeling of multiple languages, and more conveniently recognizing speech and interacting with speech of different languages. The application can realize data sharing, and the same modeling unit can exist in different languages, which can greatly reduce the dependence of the data volume of a single language. The application merges phonemes into syllables based on ipa, generates modeling syllables by fusing pronunciation rules and statistical results of training audio data, which can increase the modeling unit of a single language from 40 orders of magnitude to 500 orders of magnitude, thereby greatly reducing the learning difficulty and improving the recognition rate. The ipa-based syllable modeling technology of the application enables dozens of mixed languages to use one softmax output layer, which not only reduces the computational load and delay, but also enables comparison between different languages to be carried out inside the model, which becomes a learnable target rather than manually giving different language weights for comparison. The ipa-based syllable modeling technology of the application enables the modeling unit to be converted from a context-dependent triphone to a context-independent syllable, which can make the number of pronunciation units sufficient and more distinguishable, making it easier to learn, while taking into account the number of context-independent syllables, and the larger pronunciation unit has stronger anti-noise capability.

[0140] Figure 4 is an application framework schematic diagram for applying speech modeling to perform speech recognition.

[0141] Referring to Figure 4 , the vehicle-mounted system of the vehicle sends a speech request of a user in the cabin to the server after receiving the speech request to be recognized, the server receives the speech request sent by the user in the cabin of the vehicle, and uses the acoustic model and the related decoder established by the application according to the modeling syllable to perform speech recognition on the speech request, to obtain a speech recognition result, and the speech recognition result is sent to the vehicle to complete speech interaction. The construction process of the acoustic model can be seen from Figure 4 The right side, for the speech audio issued by the user, a neural network layer such as an LSTM (Long short-term memory, long short-term memory network) hidden layer can be used to process and output a phoneme feature vector, and then different language phonemes such as German phonemes, English phonemes, and French phonemes are merged based on ipa. Dozens of mixed languages can use one softmax output layer, and subsequent language discrimination processing is not required. The more detailed construction process of the acoustic model of the application can be seen from Figure 3 The flow is described. The phoneme merging based on ipa enables modeling units to be shared between multiple languages, and thus part of the data is shared, so that different languages can improve each other's effects.

[0142] Corresponding to the foregoing application function implementation method, the application further provides a server.

[0143] Figure 7 is a structural schematic diagram of the server shown in the application.

[0144] Referring to Figure 7 The server 70 provided by the application comprises a phoneme processing module 71, a training processing module 72, a syllable merging module 73, a model generation module 74, a request receiving module 75, and a speech recognition module 76.

[0145] The phoneme processing module 71 is configured to obtain phonemes of different languages, and merge the phonemes of different languages into first syllables according to pronunciation rules. The phoneme processing module 71 can pre-merge the phonemes of different languages according to the rules of the International Phonetic Alphabet, and merge the phonemes after pre-merging into first syllables according to pronunciation rules.

[0146] The training processing module 72 is configured to obtain training audio of different languages, identify combined syllables of pronunciation by using training materials of different languages, and select second syllables from the combined syllables according to pronunciation adhesion. The training processing module 72 can perform forced frame alignment on the combined syllables, determine the average pronunciation duration of the aligned combined syllables and the average pronunciation duration of all combined syllables, and take the ratio of the average pronunciation duration of the combined syllables to the average pronunciation duration of all combined syllables as the pronunciation adhesion. The combined syllables with pronunciation adhesion less than a set threshold are taken as the second syllables. The application can identify the combined syllables of pronunciation by using training materials of audio and / or video of different languages.

[0147] The syllable merging module 73 is configured to merge the first syllables obtained by the phoneme processing module 71 and the second syllables obtained by the training processing module 72 to obtain modeling syllables.

[0148] The model generation module 74 is configured to generate an acoustic model according to the modeling syllables.

[0149] The request receiving module 75 is configured to receive a voice request issued by a user in a vehicle cabin forwarded by a vehicle.

[0150] The speech recognition module 76 is configured to recognize the voice request received by the request receiving module 75 according to the acoustic model generated by the model generation module 74, generate a recognition result, and issue the recognition result to the vehicle to complete voice interaction.

[0151] Figure 8 is a structural schematic diagram of another server shown in the application.

[0152] Referring to Figure 8The server 70 provided in the application comprises a phoneme processing module 71, a training processing module 72, a syllable merging module 73, a model generating module 74, a request receiving module 75, and a speech recognition module 76.

[0153] The phoneme processing module 71 comprises a pre-merging sub-module 711 and a phoneme merging sub-module 712.

[0154] The pre-merging sub-module 711 is configured to pre-merge phonemes of different languages according to the International Phonetic Alphabet (IPA) rules.

[0155] The phoneme merging sub-module 712 is configured to merge the pre-merged phonemes into first syllables according to the articulatory phonetics rules. For example, the phoneme merging sub-module 712 can merge the phonemes of initial and final consonants to obtain a first syllable, and separately take the phonemes of single initial consonant and single final consonant as first syllables.

[0156] The training processing module 72 comprises an alignment and statistics module 721 and a syllable screening module 722.

[0157] The alignment and statistics module 721 is configured to perform forced frame alignment on the combined syllables to obtain the average pronunciation duration of the combined syllables and the average pronunciation duration of all the combined syllables.

[0158] The syllable screening module 722 is configured to take the ratio of the average pronunciation duration of the combined syllables to the average pronunciation duration of all the combined syllables as pronunciation adhesion, and take the combined syllables with pronunciation adhesion less than a set threshold as second syllables. The average pronunciation duration of the combined syllables can be determined according to the ratio of the total pronunciation duration after forced frame alignment to the number of times of occurrence of the combined syllables in the training audio.

[0159] The training processing module 72 can further comprise a rule screening module 723.

[0160] The rule screening module 723 screens the combined syllables that meet the syllable merging rules from the combined syllables, and the alignment and statistics module 721 performs forced frame alignment on the combined syllables that meet the syllable merging rules.

[0161] The syllable merging rules comprise at least one of the following rules: initial consonant + initial consonant + final consonant; initial consonant + initial consonant + final consonant + special initial consonant; initial consonant + final consonant + special initial consonant.

[0162] The server of the present application combines different language phonemes into first syllables according to pronunciation rules, filters second syllables from the combined syllables of the training audio according to pronunciation adhesion, and then combines to obtain modeling syllables, which are irrelevant to context, so that different languages can use the same set of modeling system, thereby realizing unified modeling of multiple languages, more convenient voice recognition and voice interaction of different languages, reducing deployment cost, and avoiding waste of machine resources.

[0163] As to the server in the above embodiments, the specific manner in which the various modules perform operations has been described in detail in the embodiments related to the method, and will not be described in detail here.

[0164] Figure 9 FIG. 6 is another structural schematic diagram of the server shown in the present application.

[0165] Referring to FIG. 6, Figure 9 The server 1000 includes a memory 1010 and a processor 1020.

[0166] The processor 1020 can be a central processing unit (CPU), and can also be other general-purpose processors, digital signal processors (DSPs), application specific integrated circuits (ASICs), field programmable gate arrays (FPGAs) or other programmable logic devices, discrete gates or transistor logic, discrete hardware components, etc. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor.

[0167] The memory 1010 can include various types of storage units such as a system memory, a read-only memory (ROM), and a permanent storage device. Among them, the ROM can store static data or instructions required by the processor 1020 or other modules of the computer. The permanent storage device can be a rewritable storage device. The permanent storage device can be a non-volatile storage device that does not lose stored instructions and data even after the computer is powered off. In some embodiments, the permanent storage device employs a mass storage device (e.g., a magnetic or optical disk, a flash memory) as a permanent storage device. In some other embodiments, the permanent storage device can be a removable storage device (e.g., a floppy disk, an optical drive). The system memory can be a readable and writable storage device or a volatile readable and writable storage device such as a dynamic random access memory. The system memory can store some or all of the instructions and data required by the processor during runtime. In addition, the memory 1010 can include a combination of any computer readable storage media, including various types of semiconductor storage chips (e.g., DRAM, SRAM, SDRAM, flash memory, programmable read-only memory), magnetic disks and / or optical disks. In some embodiments, the memory 1010 can include a readable and / or writable removable storage device such as a compact disc (CD), a read-only digital versatile disc (e.g., DVD-ROM, double-layer DVD-ROM), a read-only Blu-ray disc, an ultra-density optical disc, a flash memory card (e.g., an SD card, a min SD card, a Micro-SD card, etc.), a magnetic floppy disk, etc. The computer readable storage medium does not include a carrier wave and an instantaneous electronic signal transmitted by wireless or wired transmission.

[0168] The memory 1010 stores executable code, which, when processed by the processor 1020, can cause the processor 1020 to perform part or all of the above-mentioned methods.

[0169] In addition, the method according to the present application can also be implemented as a computer program or a computer program product, which includes computer program code instructions for performing part or all of the steps of the above-mentioned methods of the present application.

[0170] Alternatively, the present application can also be implemented as a computer readable storage medium (or non-transitory machine readable storage medium or machine readable storage medium) having executable code (or computer program or computer instruction code) stored thereon, which, when executed by a processor of an electronic device (or a server, etc.), causes the processor to perform part or all of the steps of the above-mentioned methods according to the present application.

[0171] Having described various embodiments of the application, it is to be understood that the above description is meant not to limit and not to encompass all of the possible embodiments. Many modifications and variations of this application can be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments. It is intended that the scope of the application be defined by the scope of the patent and by the claims as allowed by the patent office, which can include adaptations based on the description, equivalents, and / or substitutions of elements individually or collectively to the entire disclosure.

Claims

1. A voice interaction method, characterized in that, include: Obtain phonemes from different languages ​​and merge the phonemes from the different languages ​​into a first syllable according to phonetic rules; Using training materials from different languages, the pronunciation of combined syllables is identified. A second syllable is selected from these combined syllables based on pronunciation cohesion. This process includes: forcibly aligning the combined syllables to a frame; determining the average pronunciation duration of the aligned combined syllables and the average pronunciation duration of all combined syllables; using the ratio of the average pronunciation duration of the combined syllables to the average pronunciation duration of all combined syllables as pronunciation cohesion; and selecting combined syllables with pronunciation cohesion less than a set threshold as the second syllable. The first syllable and the second syllable are merged to obtain the modeled syllable; An acoustic model is generated based on the modeled syllables; Receive voice requests from users inside the vehicle's cabin, forwarded by the vehicle. The voice request is recognized based on the acoustic model, and the recognition result is sent to the vehicle to complete the voice interaction.

2. The method according to claim 1, characterized in that, The step of merging phonemes from different languages ​​into a first syllable according to phonetic rules includes: The phonemes of the different languages ​​are pre-merged according to the Universal Phonetic Alphabet (UPA) rules; According to phonetic rules, the pre-merged phonemes are merged into the first syllable.

3. The method according to claim 2, characterized in that, The step of merging the pre-merged phonemes into a first syllable according to phonetic rules includes: From the pre-merged phonemes, the phonemes of the initial consonant and the final vowel are merged to obtain the first syllable, and the remaining individual phonemes of the initial consonant and the individual phonemes of the final vowel are used as the first syllables.

4. The method according to claim 1, characterized in that, The method of identifying syllable combinations of sounds using training materials in different languages ​​includes: Identify syllable combinations of sounds using training materials in different languages, including audio and / or video.

5. The method according to claim 1, characterized in that, Before performing forced frame alignment on the combined syllables, the method further includes: filtering out combined syllables that conform to the syllable merging rules from the combined syllables; The step of forcibly aligning the combined syllables includes: forcibly aligning the combined syllables that conform to the syllable merging rules.

6. The method according to claim 5, characterized in that, The syllable merging rules include at least one of the following rules: Initial consonant + initial consonant + final vowel; Initial consonant + initial consonant + final vowel + special initial consonant; Initial consonant + final vowel + special initial consonant.

7. The method according to claim 1, characterized in that: The average duration of pronunciation of the combined syllable is determined by the ratio of the total duration of pronunciation after forced frame alignment to the number of times the combined syllable appears in the training audio.

8. The method according to claim 1, characterized in that, The step of forcibly aligning the combined syllables includes: The combined syllables are forced to be frame-aligned with the vowel as the core.

9. A server, characterized in that, include: The phoneme processing module is used to acquire phonemes from different languages ​​and merge the phonemes from the different languages ​​into a first syllable according to phonetic rules. The training processing module is used to identify combined syllables of pronunciation using training materials from different languages, and to select second syllables from the combined syllables based on pronunciation cohesion. This includes: performing forced frame alignment on the combined syllables; determining the average pronunciation duration of the aligned combined syllables and the average pronunciation duration of all combined syllables; using the ratio of the average pronunciation duration of the combined syllables to the average pronunciation duration of all combined syllables as the pronunciation cohesion, and selecting combined syllables with a pronunciation cohesion less than a set threshold as the second syllable. The syllable merging module is used to merge the first syllable obtained by the phoneme processing module and the second syllable obtained by the training processing module to obtain the modeled syllable; The model generation module is used to generate an acoustic model based on the modeled syllables; The request receiving module is used to receive voice requests issued by users in the vehicle cabin and forwarded by the vehicle. The speech recognition module is used to recognize the speech requests received by the request receiving module based on the acoustic model generated by the model generation module, and generate a recognition result to send to the vehicle to complete the speech interaction.

10. A server, characterized in that, include: processor; as well as A memory having executable code stored thereon, which, when executed by the processor, causes the processor to perform the method as described in any one of claims 1-8.

11. A computer-readable storage medium having executable code stored thereon, which, when executed by a processor of an electronic device, causes the processor to perform the method as described in any one of claims 1-8.

Images