Phoneme alignment method and device, equipment and medium

Abstract

The invention relates to the field of artificial intelligence, and provides a phoneme alignment method and device, equipment and a medium, and the method comprises the steps: determining text features of a Chinese character sequence based on the Chinese character sequence obtained in advance; extracting time-frequency characteristics of audio information corresponding to the Chinese character sequence through an acoustic model; performing time step alignment on the Chinese character sequence and the audio information through a forced alignment model and the time-frequency features to obtain aligned audio features; fusing the text features and the aligned audio features to generate corresponding joint embedding features; and obtaining a phoneme sequence corresponding to the joint embedding feature through a phoneme decoding model. According to the embodiment of the invention, the text features and the aligned audio features can be fused to generate the corresponding joint embedded features, so that the pronunciation rules can be accurately captured, and the method is suitable for complex actual scene requirements; in addition, according to the embodiment of the invention, a phoneme sequence corresponding to the joint embedding feature can be obtained through a phoneme decoding model, and the accuracy and robustness of phoneme alignment are improved.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence, and in particular, to a phoneme alignment method, apparatus, device, and medium. Background Art

[0002] In the task of converting Chinese pinyin to phonemes (G2P, Grapheme-to-Phoneme), the existing technologies mainly rely on text data to perform speech annotation on Chinese characters and generate corresponding phoneme sequences. However, these technologies have many deficiencies:

[0003] 1. Insufficient context information: Most current G2P models predict based on Chinese character sequences and are difficult to fully capture complex contexts and ambiguities; for example, the pronunciation of "行" in Chinese can be [xing2] or [hang2], and it is difficult to accurately distinguish only relying on text information;

[0004] 2. Lack of utilization of audio: Although speech data can provide more intuitive pronunciation references, the existing G2P technologies rarely combine audio alignment information for modeling, which makes the model perform poorly in dealing with homophones, tone sandhi, etc.;

[0005] 3. Polyphone problem: Polyphones are very common in Chinese. The existing models have low accuracy in predicting polyphones. Especially in complex sentences, due to the lack of effective context information, it is easy to cause phoneme annotation errors;

[0006] 4. High cost of manual annotation: Most G2P systems need to rely on a large amount of annotated data, and the complexity of Chinese phoneme annotation makes the data annotation process time-consuming and laborious, restricting the application of the model in low-resource scenarios;

[0007] 5. Poor robustness: When dealing with complex inputs such as colloquialisms, typos, and dialects, the existing G2P systems have low robustness, and the generated results often deviate from expectations;

[0008] 6. Insufficient transfer learning: The existing G2P models are difficult to quickly adapt in new fields (such as specific industry terms or dialects), resulting in a significant decline in performance outside the field.

[0009] Therefore, the existing phoneme alignment methods have the problem of low alignment accuracy. Summary of the Invention

[0010] Embodiments of the present invention provide a phoneme alignment method, apparatus, device, and medium, aiming to solve the problem of low alignment accuracy existing in the existing phoneme alignment methods.

[0011] In a first aspect, embodiments of the present invention provide a phoneme alignment method, and the method includes:

[0012] Determining text features of the Chinese character sequence based on the pre-acquired Chinese character sequence;

[0013] Extracting time-frequency features of audio information corresponding to the Chinese character sequence through an acoustic model;

[0014] Performing time step alignment on the Chinese character sequence and the audio information by using a forced alignment model and the time-frequency features to obtain aligned audio features;

[0015] Fusing the text features and the aligned audio features to generate corresponding joint embedding features;

[0016] A phoneme sequence corresponding to the joint embedding feature is obtained through a phoneme decoding model.

[0017] In a second aspect, an embodiment of the present invention further provides a phoneme alignment device, the device comprising:

[0018] A first acquisition unit, configured to determine text features of a Chinese character sequence based on a pre-acquired Chinese character sequence;

[0019] An extraction unit, used for extracting the time-frequency features of the audio information corresponding to the Chinese character sequence through an acoustic model;

[0020] An alignment unit, used to perform time step alignment on the Chinese character sequence and the audio information by using a forced alignment model and the time-frequency feature to obtain an aligned audio feature;

[0021] A fusion unit, used to fuse the text features and the aligned audio features to generate corresponding joint embedding features;

[0022] The second acquisition unit is used to acquire a phoneme sequence corresponding to the joint embedding feature through a phoneme decoding model.

[0023] In a third aspect, an embodiment of the present invention further provides an electronic device, which includes a memory and a processor, wherein a computer program is stored in the memory, and when the processor executes the computer program, the method described in the first aspect is implemented.

[0024] In a fourth aspect, an embodiment of the present invention further provides a computer-readable storage medium, wherein the storage medium stores a computer program, wherein the computer program includes program instructions, and wherein the program instructions, when executed by a processor, can implement the method described in the first aspect above.

[0025] The present invention provides a phoneme alignment method, device, equipment and medium, the method comprising: determining the text features of the Chinese character sequence based on a pre-acquired Chinese character sequence; extracting the time-frequency features of the audio information corresponding to the Chinese character sequence through an acoustic model; performing time step alignment on the Chinese character sequence and the audio information through a forced alignment model and the time-frequency features to obtain aligned audio features; fusing the text features and the aligned audio features to generate corresponding joint embedding features; and obtaining the phoneme sequence corresponding to the joint embedding features through a phoneme decoding model. The embodiment of the present invention can generate the corresponding joint embedding features by fusing the text features and the aligned audio features, so as to accurately capture the pronunciation rules and adapt to the complex actual scene requirements; the embodiment of the present invention can also obtain the phoneme sequence corresponding to the joint embedding features through a phoneme decoding model to improve the accuracy and robustness of phoneme alignment. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying any creative work.

[0027] Figure 1 A flowchart of a phoneme alignment method provided by an embodiment of the present invention;

[0028] Figure 2 A schematic block diagram of a phoneme alignment device provided by an embodiment of the present invention;

[0029] Figure 3 A schematic block diagram of an electronic device provided by an embodiment of the present invention;

[0030] Figure 4 A schematic diagram of an application environment of the phoneme alignment method provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0031] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0032] It should be understood that when used in this specification and the appended claims, the terms "include" and "comprises" indicate the presence of described features, integers, steps, operations, elements and / or components, but do not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or combinations thereof.

[0033] It should also be understood that the terms used in this specification of the present invention are only for the purpose of describing specific embodiments and are not intended to limit the present invention. As used in the specification of the present invention and the appended claims, unless the context clearly indicates otherwise, the singular forms "a", "an" and "the" are intended to include plural forms.

[0034] It should be further understood that the term "and / or" used in the present specification and the appended claims refers to any combination and all possible combinations of one or more of the associated listed items, and includes these combinations. The embodiment of the present invention provides a phoneme alignment method, apparatus, device and medium. The phoneme alignment method can be found in Figure 4 , Figure 4 Schematic diagram of the application environment of the phoneme alignment method provided by the embodiment of the present invention. The phoneme alignment method is applied in Figure 4 In an application environment, the terminal communicates with the server through a network. The terminal sends the acquired Chinese character sequence-audio information pair to the server, and the server determines the text features of the Chinese character sequence based on the Chinese character sequence in the Chinese character sequence-audio information pair; extracts the time-frequency features of the audio information in the Chinese character sequence-audio information pair through an acoustic model; performs time step alignment on the Chinese character sequence and the audio information through a forced alignment model and the time-frequency features to obtain the aligned audio features; fuses the text features and the aligned audio features to generate corresponding joint embedding features; obtains the phoneme sequence corresponding to the joint embedding features through a phoneme decoding model to improve the accuracy and robustness of phoneme alignment. Among them, the terminal can be but is not limited to various personal computers, laptops, smart phones, tablet computers and portable wearable devices. The server can be implemented with an independent server or a server cluster consisting of multiple servers. The present invention is described in detail below through specific embodiments.

[0035] Figure 1 Schematic diagram of the process flow of the phoneme alignment method provided by the embodiment of the present invention. Figure 1 As shown, the method includes the following steps S110-S150.

[0036] S110: Determine text features of the Chinese character sequence based on the pre-acquired Chinese character sequence.

[0037] In this embodiment, the Chinese character sequence includes multiple characters. The embodiment of the present invention can determine the text features of the Chinese character sequence through a language processing model (such as a BERT model or a Transformer model), and the text features include context information of each character in the Chinese character sequence; further, the text features also include association information between two adjacent characters in the Chinese character sequence.

[0038] In one embodiment, step S110 includes: obtaining context information of each character in the Chinese character sequence through a language processing model; capturing association information between two adjacent characters based on a self-attention mechanism and each context information; and fusing each context information and each association information to obtain the text feature.

[0039] In this embodiment, the Chinese character sequence is embedded into a high-dimensional vector space, and the context information of each character in the Chinese character sequence is obtained through a language processing model; wherein the language processing model can be a BERT model or a Transformer model; based on the self-attention mechanism and each context information, the association information between two adjacent characters is captured; wherein the association information is used to represent the correlation between two adjacent characters; each context information and each association information is fused to obtain the text feature.

[0040] S120. Extracting time-frequency features of the audio information corresponding to the Chinese character sequence through an acoustic model.

[0041] In this embodiment, the server obtains a Chinese character sequence-audio information pair from the terminal (i.e., the Chinese character sequence and the audio information pair are in a one-to-one correspondence), and extracts the time-frequency features of the audio information in the Chinese character sequence-audio information pair through an acoustic model; wherein the acoustic model can be a Wav2Vec2.0 model or an MFCC model, and the time-frequency features are used to describe the changes of the sound signal in time and frequency.

[0042] S130, performing time step alignment on the Chinese character sequence and the audio information through a forced alignment model and the time-frequency features to obtain aligned audio features.

[0043] In this embodiment, the Chinese character sequence and the audio information are aligned in time steps through a forced alignment model and the time-frequency features to obtain aligned audio features; wherein the forced alignment model can be a Kaldi model or a self-supervised learning model.

[0044] Specifically, the starting position and the ending position of the audio information are obtained according to the time-frequency feature, and the time step alignment of the Chinese character sequence and the audio information is performed according to the starting position and the ending position.

[0045] S140. Fuse the text features and the aligned audio features to generate corresponding joint embedding features.

[0046] In this embodiment, the text features and the aligned audio features are fused to generate joint embedding features that are sensitive to context and audio information, so as to accurately capture the pronunciation rules and meet the requirements of complex actual scenarios. Taking an actual scenario as an example: the input Chinese character sequence is "this line of text". In the prior art, if only relying on the Chinese character sequence to perform phonetic annotation on Chinese characters and generate the corresponding phoneme sequence, the phoneme sequence [zhe4, hang2, wen2, zi4] will be obtained. However, when dealing with complex inputs such as colloquialism, typos, and dialects, the pronunciation of "hang" may be [xing2]. To meet the requirements of complex actual scenarios, the present invention can more accurately capture the pronunciation rules according to the joint embedding features that are sensitive to context and audio information. That is, when the pronunciation of "hang" is [xing2], the output phoneme sequence is [zhe4, xing2, wen2, zi4].

[0047] For the convenience of description, taking the medical business application scenario as an example: the target object asks the question "Who is my attending doctor?" on the interaction interface of the terminal. The server obtains the corresponding Chinese character sequence-audio information pair according to the question raised by the target object and the pre-selected dialect information. Based on the question raised by the target object, the obtained Chinese character sequence may be "Your attending doctor is Li Le". If only relying on the Chinese character sequence to perform phonetic annotation on Chinese characters and generate the corresponding phoneme sequence, the phoneme sequence [nin2, de, zhu3, zhi4, yi1, sheng1, shi4, li3, le4] will be obtained. However, when dealing with complex inputs such as colloquialism, typos, and dialects, the pronunciation of "Le" may be [le2]. To meet the requirements of complex actual scenarios, the present invention can more accurately capture the pronunciation rules by combining the aligned audio features on the basis of the Chinese character sequence. That is, in the audio information, when the pronunciation of "Le" is [le2], the output phoneme sequence is [nin2, de, zhu3, zhi4, yi1, sheng1, shi4, li3, le2].

[0048] S150. Obtain the phoneme sequence corresponding to the joint embedding features through a phoneme decoding model.

[0049] In this embodiment, the combined embedding features are input into the phoneme decoding model. The phoneme decoding model predicts the phonemes corresponding to each character based on the combined embedding features, and combines the phonemes corresponding to each character to obtain a corresponding phoneme sequence. Since the combined embedding features are obtained by fusing the text features and the aligned audio features, the generated phoneme sequence can ensure that each character is exactly matched with the corresponding phoneme. Among them, the phoneme decoding model can be a Transformer model or an RNN-sequence-to-sequence model. For the convenience of description, taking the financial business application scenario as an example, the intelligent customer service of the terminal guides the user to sign in a specified area, such as "Please sign at the last line". If only the Chinese character sequence is used to perform speech annotation on Chinese characters and generate a corresponding phoneme sequence, the phoneme sequence [qing3, zai4, zui4, hou4, yi1, hang2, qian1, ming2] will be obtained. However, when dealing with complex inputs such as colloquialisms, typos, and dialects, the pronunciation of "行" may be [xing2]. In order to ensure that the generated phoneme sequence is consistent with the actual pronunciation of the audio information, in the audio information, when the pronunciation of "行" is [xing2], the output phoneme sequence is [qing3, zai4, zui4, hou4, yi1, xing2, qian1, ming2]. At this time, the intelligent customer service will perform voice playback according to the output phoneme sequence.

[0050] In one embodiment, after step S150, it further includes: obtaining at least one new forced alignment model according to a preset restriction condition; performing time-step alignment on the Chinese character sequence and the audio information through the new forced alignment model and the time-frequency features to obtain at least one new aligned audio feature; fusing the text features and the new aligned audio features to generate at least one new combined embedding feature; and obtaining the new phoneme sequences corresponding to the new combined embedding features according to the phoneme decoding module.

[0051] In this embodiment, in order to further ensure that the generated phoneme sequence is consistent with the actual pronunciation of the audio information, the present invention can preset restriction conditions, and the restriction conditions include at least one restriction item. Specifically, the restriction conditions include but are not limited to changing the alignment method, introducing background noise in the audio, changing the speech rate change of the audio, etc. When the restriction condition contains two restriction items, two new phoneme sequences can be obtained.

[0052] For example, the initial alignment method is left alignment. The present invention can pre-set restriction conditions and re-acquire a new phoneme sequence by diagonal alignment; specifically, when the restriction condition is "adopting diagonal alignment", a new forced alignment model is obtained according to the restriction condition; the Chinese character sequence and the audio information are aligned in time steps through the new forced alignment model and the time-frequency features to obtain a new aligned audio feature; the text features and the new aligned audio features are fused to generate a new joint embedded feature; the new phoneme sequence corresponding to the new joint embedded feature is obtained according to the phoneme decoding module, and then the generated phoneme sequence is ensured to be consistent with the actual pronunciation of the audio information according to the deviation value between the phoneme sequence and the new phoneme sequence.

[0053] In one embodiment, after obtaining the new phoneme sequences corresponding to the new joint embedded features according to the phoneme decoding module, it also includes: obtaining the deviation values ​​corresponding to the new phoneme sequences according to the phoneme sequences; judging whether there is a deviation value greater than a preset deviation threshold among the multiple deviation values, and obtaining a deviation value judgment result; if the deviation value judgment result is yes, returning to execute the step of performing time step alignment on the Chinese character sequence and the audio information through the forced alignment model and the time-frequency features to obtain the aligned audio features.

[0054] In this embodiment, the deviation values ​​corresponding to each of the new phoneme sequences are obtained according to the phoneme sequence; it is determined whether there is a deviation value greater than a preset deviation threshold among the multiple deviation values ​​to obtain a deviation value determination result; wherein the preset deviation threshold can be set according to actual applications; for example, the restriction condition includes a restriction item, and a new phoneme sequence can be obtained according to the restriction condition, and it is verified whether the obtained phoneme sequence meets the requirements according to the deviation value between the phoneme sequence and the new phoneme sequence.

[0055] If the deviation value judgment result is yes, then return to execute the step of performing time step alignment on the Chinese character sequence and the audio information through the forced alignment model and the time-frequency feature to obtain the aligned audio feature.

[0056] In one embodiment, the step of judging whether there is a deviation value greater than a preset deviation threshold among the multiple deviation values, after obtaining the deviation value judgment result, further includes: if the deviation value judgment result is no, then taking the phoneme sequence as a phoneme sequence prediction result and outputting it.

[0057] In this embodiment, if the deviation value judgment result is no, it means that the acquired phoneme sequence meets the requirements. At this time, the phoneme sequence can be used as a phoneme sequence prediction result and output.

[0058] In one embodiment, if the deviation value judgment result is no, then before the phoneme sequence is used as a phoneme sequence prediction result and outputted, it also includes: dynamically adjusting the first weight of the text feature and the second weight of the time-frequency feature; constructing a total loss calculation function based on the adjusted first weight and second weight and returning to execute the step of determining the text feature of the Chinese character sequence based on the pre-acquired Chinese character sequence.

[0059] In this embodiment, a phoneme alignment model deployed in the server is used to obtain a phoneme sequence consistent with the actual pronunciation of the audio information, and the phoneme alignment model includes multiple sub-models, such as a language processing model, an acoustic model, a forced alignment model, a phoneme decoding model, etc. Specifically, the server inputs the Chinese character sequence-audio information pair from the terminal into the phoneme alignment model, firstly, the text features of the Chinese character sequence in the Chinese character sequence-audio information pair are determined by the language processing model (such as a BERT model or a Transformer model) in the phoneme alignment model; then, the time-frequency features of the audio information in the Chinese character sequence-audio information pair are extracted by the acoustic model in the phoneme alignment model; then, the Chinese character sequence and the audio information are aligned in time steps by the forced alignment model in the phoneme alignment model and the time-frequency features to obtain the aligned audio features; then, the text features and the aligned audio features are fused to generate corresponding joint embedding features; finally, the joint embedding features are input into the phoneme decoding model, and the phoneme sequence corresponding to the joint embedding features is obtained by the phoneme decoding model.

[0060] In the first round of phoneme alignment tasks, the phoneme alignment model uses the initial first weight and the initial second weight to obtain the phoneme sequence. In order to further improve the robustness of the phoneme alignment model, the first weight of the text feature and the second weight of the time-frequency feature can be dynamically adjusted in the multi-task loss; then, a total loss calculation function is constructed based on the adjusted first weight and the second weight, and the step of determining the text features of the Chinese character sequence based on the pre-acquired Chinese character sequence is returned to execute; wherein the sum of the first weight and the second weight is always 1.

[0061] Furthermore, before determining the text features of the Chinese character sequence based on the pre-acquired Chinese character sequence, a large amount of Chinese data with audio (i.e., Chinese character sequence-audio information pairs) needs to be input into a pre-built phoneme alignment model for model training until the model converges.

[0062] In one embodiment, after constructing the total loss calculation function based on the adjusted first weight and second weight, it also includes: obtaining a first loss value based on the prediction result of the text feature; obtaining a second loss value based on the alignment result of the time-frequency feature; inputting the first loss value and the corresponding second loss value into the total loss calculation function to obtain a total loss value and storing it in a target area; determining a minimum total loss value based on the total loss value in the target area; and determining a target first weight and a target second weight based on the minimum total loss value.

[0063] In this embodiment, a first loss value loss1 is obtained according to the prediction result of the text feature; a second loss value loss2 is obtained according to the alignment result of the time-frequency feature; the first loss value loss1 and the corresponding second loss value loss2 are input into the total loss calculation function loss_all=a loss1+b loss2 to obtain a total loss value and store it in the target area; wherein loss_all is the total loss value, a is the first weight, loss1 is the first loss value, b is the second weight, and loss2 is the second loss value; since the present invention dynamically adjusts the first weight of the text feature and the second weight of the time-frequency feature, the number of weight adjustments can be set according to actual applications, and the number of weight adjustments can be set to 2000, therefore, a plurality of total loss values ​​will be stored in the target area, and the present invention determines the minimum total loss value according to the total loss value in the target area; wherein the target area stores an initial loss value (that is, the loss value obtained when the initial first weight and the initial second weight are used to obtain the phoneme sequence); determine the target first weight and the target second weight according to the minimum total loss value (that is, when the target first weight and the target second weight are used, the total loss value obtained is the smallest) to improve the robustness of the phoneme alignment model; in the next phoneme alignment task, the phoneme alignment model uses the target first weight as the initial first weight and the target second weight as the initial second weight to perform phoneme alignment on the newly input Chinese character sequence-audio information to obtain the corresponding phoneme sequence, thereby improving the accuracy of phoneme prediction and the accuracy of phoneme alignment.

[0064] Specifically, the phoneme alignment method provided by the present invention has the following beneficial effects:

[0065] 1. Effectively solve the problem of polyphones: By fusing the text features and the aligned audio features, the accuracy of polyphone prediction is significantly improved, and the pronunciation rules can be captured more accurately;

[0066] 2. Improved robustness: For inputs containing noise, dialects, or colloquialisms, the phoneme alignment model can still maintain a high prediction accuracy and adapt to complex practical scenario requirements;

[0067] 3. High alignment efficiency: By adopting audio alignment technologies (such as Kaldi models or self-supervised learning models), the model can perform phoneme alignment. The model can still effectively learn even with less labeled data, improving its applicability in low-resource scenarios;

[0068] 4. Enhanced transfer ability: The transfer learning performance of the phoneme alignment model in new fields has been greatly improved, and it can quickly adapt to the pronunciation rules of specific fields without a large amount of labeled data;

[0069] 5. Wide application prospects: The phoneme alignment model can be applied to various scenarios such as speech synthesis, speech recognition, and educational assistance. It is especially suitable for the Chinese scenario where the pronunciation is complex and easily affected by ambiguity, providing more accurate basic component support for intelligent speech systems. For the sake of description, taking the financial business application scenario as an example, according to the questions raised by the target object and the selected dialect information, the corresponding Chinese character sequence-audio information pair is obtained, and the Chinese character sequence-audio information pair is input into the phoneme alignment model to output a phoneme sequence that is consistent with the actual pronunciation of the audio information. For example, when the input Chinese character sequence is "this line of text", if only relying on the Chinese character sequence to perform speech annotation on the Chinese characters and generate the corresponding phoneme sequence, the phoneme sequence [zhe4, hang2, wen2, zi4] will be obtained. However, when dealing with complex inputs such as colloquialisms, typos, and dialects, the pronunciation of "hang" may be [xing2]. In order to adapt to the complex actual scenario requirements, the present invention can more accurately capture the pronunciation rules by combining the aligned audio features on the basis of the Chinese character sequence. That is, in the audio information, when the pronunciation of "hang" is [xing2], the output phoneme sequence is [zhe4, xing2, wen2, zi4].

[0070] In summary, the embodiments of the present invention can generate corresponding joint embedding features by fusing the text features and the aligned audio features to accurately capture the pronunciation rules and adapt to the complex actual scenario requirements; the embodiments of the present invention can also obtain the phoneme sequence corresponding to the joint embedding features through the phoneme decoding model to improve the accuracy and robustness of phoneme alignment.

[0071] Figure 2 is a schematic block diagram of the phoneme alignment device provided by the embodiments of the present invention. As Figure 2 shown, corresponding to the above phoneme alignment method, the present invention also provides a phoneme alignment device, and the device is configured as Figure 4In an application environment, wherein, the terminal communicates with the server through a network. The terminal sends the acquired Chinese character sequence-audio information pair to the server, and the server determines the text features of the Chinese character sequence based on the Chinese character sequence in the Chinese character sequence-audio information pair; extracts the time-frequency features of the audio information in the Chinese character sequence-audio information pair through an acoustic model; performs time-step alignment on the Chinese character sequence and the audio information through a forced alignment model and the time-frequency features to obtain the aligned audio features; fuses the text features and the aligned audio features to generate corresponding joint embedding features; obtains the phoneme sequence corresponding to the joint embedding features through a phoneme decoding model to improve the accuracy and robustness of phoneme alignment. Among them, the terminal can be but is not limited to various personal computers, laptops, smart phones, tablet computers and portable wearable devices. The server can be implemented with an independent server or a server cluster consisting of multiple servers. Specifically, please refer to Figure 2 , the phoneme alignment device 700 comprises:

[0072] A first acquisition unit 701 is used to determine text features of a Chinese character sequence based on a pre-acquired Chinese character sequence;

[0073] An extraction unit 702, configured to extract time-frequency features of audio information corresponding to the Chinese character sequence through an acoustic model;

[0074] An alignment unit 703 is used to perform time step alignment on the Chinese character sequence and the audio information by using a forced alignment model and the time-frequency feature to obtain an aligned audio feature;

[0075] A fusion unit 704, configured to fuse the text feature and the aligned audio feature to generate a corresponding joint embedding feature;

[0076] The second acquisition unit 705 is used to acquire a phoneme sequence corresponding to the joint embedding feature through a phoneme decoding model.

[0077] In some embodiments, after executing the step of acquiring the phoneme sequence corresponding to the joint embedding feature through the phoneme decoding model, the second acquiring unit 705 is further configured to:

[0078] Obtain at least one new forced alignment model according to preset constraints;

[0079] Performing time step alignment on the Chinese character sequence and the audio information using the new forced alignment model and the time-frequency feature to obtain at least one new aligned audio feature;

[0080] Fusing the text features and the newly aligned audio features to generate at least one new joint embedding feature;

[0081] According to the phoneme decoding module, new phoneme sequences corresponding to the new joint embedding features are obtained.

[0082] In some embodiments, after executing the step of acquiring new phoneme sequences corresponding to the new joint embedding features according to the phoneme decoding module, the second acquiring unit 705 is further configured to:

[0083] Acquire deviation values ​​corresponding to each of the new phoneme sequences according to the phoneme sequence;

[0084] Determine whether there is a deviation value greater than a preset deviation threshold value among the plurality of deviation values, and obtain a deviation value determination result;

[0085] If the deviation value judgment result is yes, then return to execute the step of performing time step alignment on the Chinese character sequence and the audio information through the forced alignment model and the time-frequency feature to obtain the aligned audio feature.

[0086] In some embodiments, after executing the step of determining whether there is a deviation value greater than a preset deviation threshold among the plurality of deviation values ​​and obtaining the deviation value determination result, the second acquisition unit 705 is further configured to:

[0087] If the deviation value judgment result is no, the phoneme sequence is taken as a phoneme sequence prediction result and outputted.

[0088] In some embodiments, before executing the step of taking the phoneme sequence as a phoneme sequence prediction result and outputting it if the deviation value judgment result is no, the second acquisition unit 705 is further used to:

[0089] Dynamically adjusting the first weight of the text feature and the second weight of the time-frequency feature;

[0090] A total loss calculation function is constructed according to the adjusted first weight and second weight, and the step of determining the text features of the Chinese character sequence based on the pre-acquired Chinese character sequence is returned to be executed.

[0091] In some embodiments, after executing the step of constructing a total loss calculation function according to the adjusted first weight and the second weight, the second acquisition unit 705 is further used to:

[0092] Obtaining a first loss value according to the prediction result of the text feature;

[0093] Obtaining a second loss value according to the alignment result of the time-frequency features;

[0094] Inputting the first loss value and the corresponding second loss value into the total loss calculation function to obtain a total loss value and storing it in the target area;

[0095] Determining a minimum total loss value according to the total loss value in the target area;

[0096] The target first weight and the target second weight are determined according to the minimum total loss value.

[0097] In some embodiments, when the first acquisition unit 701 performs the step of determining the text features of the Chinese character sequence based on the pre-acquired Chinese character sequence, it is specifically used to:

[0098] Acquire context information of each character in the Chinese character sequence through a language processing model;

[0099] Capturing the association information between two adjacent characters based on the self-attention mechanism and each of the context information;

[0100] The context information and the associated information are fused to obtain the text features.

[0101] It should be noted that those skilled in the art can clearly understand that the specific implementation process of the above-mentioned phoneme alignment device and each unit can refer to the corresponding description in the aforementioned method embodiment, and for the convenience and brevity of description, it will not be repeated here.

[0102] The above-mentioned phoneme alignment device can be implemented in the form of a computer program. The computer program can be used in Figure 3 The electronic device shown is running.

[0103] See also Figure 3 , Figure 3 800 is a schematic block diagram of an electronic device provided by an embodiment of the present invention. The electronic device 800 may be a terminal or a server, wherein the terminal may be an electronic device with a communication function. The server may be an independent server or a server cluster composed of multiple servers.

[0104] See also Figure 3 The electronic device 800 includes a processor 802 , a memory and a network interface 805 connected via a system bus 801 , wherein the memory may include a non-volatile storage medium 803 and an internal memory 804 .

[0105] The non-volatile storage medium 803 may store an operating system 8031 ​​and a computer program 8032. The computer program 8032 includes program instructions, and when the program instructions are executed, the processor 802 may execute a phoneme alignment method.

[0106] The processor 802 is used to provide computing and control capabilities to support the operation of the entire electronic device 800 .

[0107] The internal memory 804 provides an environment for the operation of the computer program 8032 in the non-volatile storage medium 803. When the computer program 8032 is executed by the processor 802, the processor 802 can execute a phoneme alignment method.

[0108] The network interface 805 is used to communicate with other devices over the network. Figure 3 The structure shown in the figure is merely a block diagram of a partial structure related to the solution of the present invention, and does not constitute a limitation on the electronic device 800 to which the solution of the present invention is applied. The specific electronic device 800 may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0109] The processor 802 is used to run the computer program 8032 stored in the memory to implement the following steps:

[0110] Determining text features of the Chinese character sequence based on the pre-acquired Chinese character sequence;

[0111] Extracting time-frequency features of audio information corresponding to the Chinese character sequence through an acoustic model;

[0112] Performing time step alignment on the Chinese character sequence and the audio information by using a forced alignment model and the time-frequency features to obtain aligned audio features;

[0113] Fusing the text features and the aligned audio features to generate corresponding joint embedding features;

[0114] A phoneme sequence corresponding to the joint embedding feature is obtained through a phoneme decoding model.

[0115] In some embodiments, after implementing the step of acquiring the phoneme sequence corresponding to the joint embedding feature through the phoneme decoding model, the processor 802 is further configured to:

[0116] Obtain at least one new forced alignment model according to preset constraints;

[0117] Performing time step alignment on the Chinese character sequence and the audio information using the new forced alignment model and the time-frequency feature to obtain at least one new aligned audio feature;

[0118] Fusing the text features and the newly aligned audio features to generate at least one new joint embedding feature;

[0119] According to the phoneme decoding module, new phoneme sequences corresponding to the new joint embedding features are obtained.

[0120] In some embodiments, after implementing the step of acquiring new phoneme sequences corresponding to the new joint embedding features according to the phoneme decoding module, the processor 802 is further configured to:

[0121] Acquire deviation values ​​corresponding to each of the new phoneme sequences according to the phoneme sequence;

[0122] Determine whether there is a deviation value greater than a preset deviation threshold value among the plurality of deviation values, and obtain a deviation value determination result;

[0123] If the deviation value judgment result is yes, then return to execute the step of performing time step alignment on the Chinese character sequence and the audio information through the forced alignment model and the time-frequency feature to obtain the aligned audio feature.

[0124] In some embodiments, after the processor 802 implements the step of determining whether there is a deviation value greater than a preset deviation threshold among the plurality of deviation values ​​and obtaining the deviation value determination result, it is further configured to:

[0125] If the deviation value judgment result is no, the phoneme sequence is taken as a phoneme sequence prediction result and outputted.

[0126] In some embodiments, before implementing the step of taking the phoneme sequence as a phoneme sequence prediction result and outputting the result if the deviation value judgment result is no, the processor 802 is further configured to:

[0127] Dynamically adjusting the first weight of the text feature and the second weight of the time-frequency feature;

[0128] A total loss calculation function is constructed according to the adjusted first weight and second weight, and the step of determining the text features of the Chinese character sequence based on the pre-acquired Chinese character sequence is returned to be executed.

[0129] In some embodiments, after implementing the step of constructing a total loss calculation function according to the adjusted first weight and the second weight, the processor 802 is further configured to:

[0130] Obtaining a first loss value according to the prediction result of the text feature;

[0131] Obtaining a second loss value according to the alignment result of the time-frequency features;

[0132] Inputting the first loss value and the corresponding second loss value into the total loss calculation function to obtain a total loss value and storing it in the target area;

[0133] Determining a minimum total loss value according to the total loss value in the target area;

[0134] The target first weight and the target second weight are determined according to the minimum total loss value.

[0135] In some embodiments, when implementing the step of determining the text features of the Chinese character sequence based on the pre-acquired Chinese character sequence, the processor 802 is specifically configured to:

[0136] Acquire context information of each character in the Chinese character sequence through a language processing model;

[0137] Capturing the association information between two adjacent characters based on the self-attention mechanism and each of the context information;

[0138] The context information and the associated information are fused to obtain the text features.

[0139] It should be understood that in the embodiment of the present invention, the processor 802 may be a central processing unit (CPU), and the processor 802 may also be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. Among them, the general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.

[0140] It can be understood by those skilled in the art that all or part of the processes in the method for implementing the above embodiment can be completed by instructing the relevant hardware through a computer program. The computer program includes program instructions, and the computer program can be stored in a storage medium, which is a computer-readable storage medium. The program instructions are executed by at least one processor in the computer system to implement the process steps of the embodiment of the above method.

[0141] Therefore, the present invention also provides a storage medium. The storage medium may be a computer-readable storage medium. The storage medium stores a computer program, wherein the computer program includes program instructions. When the program instructions are executed by a processor, the processor executes the following steps:

[0142] Determining text features of the Chinese character sequence based on the pre-acquired Chinese character sequence;

[0143] Extracting time-frequency features of audio information corresponding to the Chinese character sequence through an acoustic model;

[0144] Performing time step alignment on the Chinese character sequence and the audio information by using a forced alignment model and the time-frequency features to obtain aligned audio features;

[0145] Fusing the text features and the aligned audio features to generate corresponding joint embedding features;

[0146] A phoneme sequence corresponding to the joint embedding feature is obtained through a phoneme decoding model.

[0147] In one embodiment, after executing the program instructions to implement the step of acquiring a phoneme sequence corresponding to the joint embedding feature through a phoneme decoding model, the processor is further configured to:

[0148] Obtain at least one new forced alignment model according to preset constraints;

[0149] Performing time step alignment on the Chinese character sequence and the audio information using the new forced alignment model and the time-frequency feature to obtain at least one new aligned audio feature;

[0150] Fusing the text features and the newly aligned audio features to generate at least one new joint embedding feature;

[0151] According to the phoneme decoding module, new phoneme sequences corresponding to the new joint embedding features are obtained.

[0152] In one embodiment, after executing the program instructions to implement the step of acquiring new phoneme sequences corresponding to the new joint embedding features according to the phoneme decoding module, the processor is further configured to:

[0153] Acquire deviation values ​​corresponding to each of the new phoneme sequences according to the phoneme sequence;

[0154] Determine whether there is a deviation value greater than a preset deviation threshold value among the plurality of deviation values, and obtain a deviation value determination result;

[0155] If the deviation value judgment result is yes, then return to execute the step of performing time step alignment on the Chinese character sequence and the audio information through the forced alignment model and the time-frequency feature to obtain the aligned audio feature.

[0156] In one embodiment, after the processor executes the program instructions to determine whether there is a deviation value greater than a preset deviation threshold among the plurality of deviation values ​​and obtains the deviation value determination result step, it is further configured to:

[0157] If the deviation value judgment result is no, the phoneme sequence is taken as a phoneme sequence prediction result and outputted.

[0158] In one embodiment, before the processor executes the program instructions to implement the step of taking the phoneme sequence as a phoneme sequence prediction result and outputting it if the deviation value judgment result is no, it is further configured to:

[0159] Dynamically adjusting the first weight of the text feature and the second weight of the time-frequency feature;

[0160] A total loss calculation function is constructed according to the adjusted first weight and second weight, and the step of determining the text features of the Chinese character sequence based on the pre-acquired Chinese character sequence is returned to be executed.

[0161] In one embodiment, after executing the program instructions to implement the step of constructing a total loss calculation function according to the adjusted first weight and the second weight, the processor is further configured to:

[0162] Obtaining a first loss value according to the prediction result of the text feature;

[0163] Obtaining a second loss value according to the alignment result of the time-frequency features;

[0164] Inputting the first loss value and the corresponding second loss value into the total loss calculation function to obtain a total loss value and storing it in the target area;

[0165] Determining a minimum total loss value according to the total loss value in the target area;

[0166] The target first weight and the target second weight are determined according to the minimum total loss value.

[0167] In one embodiment, when the processor executes the program instructions to implement the step of determining the text features of the Chinese character sequence based on the pre-acquired Chinese character sequence, it is specifically configured to:

[0168] Acquire context information of each character in the Chinese character sequence through a language processing model;

[0169] Capturing the association information between two adjacent characters based on the self-attention mechanism and each of the context information;

[0170] The context information and the associated information are fused to obtain the text features.

[0171] The storage medium may be a USB flash drive, a mobile hard disk, a read-only memory (ROM), a magnetic disk, or an optical disk, etc., which are computer-readable storage media that can store program codes.

[0172] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in terms of function in the above description. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.

[0173] In the several embodiments provided by the present invention, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of each unit is only a logical function division, and there may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed.

[0174] The steps in the method of the embodiment of the present invention can be adjusted in order, combined and deleted according to actual needs. The units in the device of the embodiment of the present invention can be combined, divided and deleted according to actual needs. In addition, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0175] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a storage medium. Based on this understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for enabling an electronic device (which can be a personal computer, terminal, or network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention.

[0176] The above is only a specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can easily think of various equivalent modifications or replacements within the technical scope disclosed by the present invention, and these modifications or replacements should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention shall be based on the protection scope of the claims.

Claims

1. A phoneme alignment method, characterized in that: The method comprises: Determining text features of the Chinese character sequence based on the pre-acquired Chinese character sequence; Extracting time-frequency features of audio information corresponding to the Chinese character sequence through an acoustic model; Performing time step alignment on the Chinese character sequence and the audio information by using a forced alignment model and the time-frequency features to obtain aligned audio features; Fusing the text features and the aligned audio features to generate corresponding joint embedding features; A phoneme sequence corresponding to the joint embedding feature is obtained through a phoneme decoding model.

2. The phoneme alignment method according to claim 1, characterized in that: After obtaining the phoneme sequence corresponding to the joint embedding feature through the phoneme decoding model, the method further includes: Obtain at least one new forced alignment model according to preset constraints; Performing time step alignment on the Chinese character sequence and the audio information using the new forced alignment model and the time-frequency feature to obtain at least one new aligned audio feature; Fusing the text features and the newly aligned audio features to generate at least one new joint embedding feature; According to the phoneme decoding module, new phoneme sequences corresponding to the new joint embedding features are obtained.

3. The phoneme alignment method according to claim 2, characterized in that: After obtaining the new phoneme sequences corresponding to the new joint embedding features according to the phoneme decoding module, the method further includes: Acquire deviation values ​​corresponding to each of the new phoneme sequences according to the phoneme sequence; Determine whether there is a deviation value greater than a preset deviation threshold value among the plurality of deviation values, and obtain a deviation value determination result; If the deviation value judgment result is yes, then return to execute the step of performing time step alignment on the Chinese character sequence and the audio information through the forced alignment model and the time-frequency feature to obtain the aligned audio feature.

4. The phoneme alignment method according to claim 3, characterized in that: After determining whether there is a deviation value greater than a preset deviation threshold among the plurality of deviation values ​​and obtaining the deviation value determination result, the method further includes: If the deviation value judgment result is no, the phoneme sequence is taken as a phoneme sequence prediction result and outputted.

5. The phoneme alignment method according to claim 4, characterized in that: If the deviation value judgment result is no, before outputting the phoneme sequence as a phoneme sequence prediction result, the method further includes: Dynamically adjusting the first weight of the text feature and the second weight of the time-frequency feature; A total loss calculation function is constructed according to the adjusted first weight and second weight, and the step of determining the text features of the Chinese character sequence based on the pre-acquired Chinese character sequence is returned to be executed.

6. The phoneme alignment method according to claim 5, characterized in that: After constructing the total loss calculation function according to the adjusted first weight and the second weight, the method further includes: Obtaining a first loss value according to the prediction result of the text feature; Obtaining a second loss value according to the alignment result of the time-frequency features; Inputting the first loss value and the corresponding second loss value into the total loss calculation function to obtain a total loss value and storing it in the target area; Determining a minimum total loss value according to the total loss value in the target area; The target first weight and the target second weight are determined according to the minimum total loss value.

7. The phoneme alignment method according to claim 1, characterized in that: The determining of the text features of the Chinese character sequence based on the pre-acquired Chinese character sequence includes: Acquire context information of each character in the Chinese character sequence through a language processing model; Capturing the association information between two adjacent characters based on the self-attention mechanism and each of the context information; The context information and the associated information are fused to obtain the text features.

8. A phoneme alignment device, characterized in that: The device comprises: A first acquisition unit, configured to determine text features of a Chinese character sequence based on a pre-acquired Chinese character sequence; An extraction unit, used for extracting the time-frequency features of the audio information corresponding to the Chinese character sequence through an acoustic model; An alignment unit, used to perform time step alignment on the Chinese character sequence and the audio information by using a forced alignment model and the time-frequency feature to obtain an aligned audio feature; A fusion unit, used to fuse the text features and the aligned audio features to generate corresponding joint embedding features; The second acquisition unit is used to acquire a phoneme sequence corresponding to the joint embedding feature through a phoneme decoding model.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the phoneme alignment method according to any one of claims 1 to 7 is implemented.

10. A computer-readable storage medium, characterized in that: The storage medium stores a computer program, wherein the computer program includes program instructions, and when the program instructions are executed by a processor, the processor executes the phoneme alignment method according to any one of claims 1 to 7.

Images