Training method and device for end-to-end model, electronic equipment and storage medium

Abstract

Embodiments of the present application provide a training method and device for an end-to-end model, electronic equipment and a storage medium. A multi-phonetic word in a target training text is determined. The multi-phonetic word is phonetized, and a corpus is generated. The corpus includes the multi-phonetic word and phonetic information of the multi-phonetic word. Audio information corresponding to the phonetic information is obtained. The end-to-end model is trained using the corpus and the audio information, and an acoustic model is generated, thereby improving the accuracy of training the end-to-end model.

Description

Technical Field

[0001] This invention relates to the field of training technology for end-to-end models, and in particular to a training method for end-to-end models, a training apparatus for end-to-end models, an electronic device, and a computer-readable storage medium. Background Technology

[0002] Automatic Speech Recognition (ASR) is a technology that studies how to recognize and convert human speech into text. It can be applied to services such as voice dialing, voice navigation, indoor device control, voice document retrieval, and simple dictation data entry.

[0003] End-to-end models are important for speech recognition. Generally, when performing Chinese speech recognition based on end-to-end models, Chinese characters need to be selected as modeling units. However, Chinese characters have polyphonic forms. If Chinese characters are directly used as modeling units, during the training process, two Chinese characters with the same pronunciation but different pronunciations will be associated with the same label for training, which will cause inaccurate model training. Summary of the Invention

[0004] The present invention provides a training method, apparatus, electronic device, and computer-readable storage medium for an end-to-end model to solve the problem that speech recognition cannot recognize polyphonic characters.

[0005] This invention discloses a training method for an end-to-end model, which may include:

[0006] Identify polyphonic words in the target training text;

[0007] The polyphonic characters are annotated with phonetic symbols, and a corpus is generated; the corpus includes the polyphonic characters and phonetic symbols for the polyphonic characters.

[0008] Obtain the audio information corresponding to the phonetic information;

[0009] The end-to-end model is trained using the corpus and the audio information, and an acoustic model is generated.

[0010] Optionally, the acoustic model is used to output a text sequence vector, the text sequence vector having a phonetic feature sequence vector for expressing the phonetic information, and the method may further include:

[0011] The acoustic model is used to identify the audio information and generate identified text; the identified text includes phonetic information of the identification result.

[0012] Delete the phonetic information in the recognition result.

[0013] Optionally, the end-to-end model has a corresponding text-to-speech module, and the step of annotating the polyphonic characters with phonetic symbols may further include:

[0014] The text-to-speech module is used to annotate the polyphonic characters and words.

[0015] Optionally, the step of annotating the polyphonic characters with phonetic symbols may further include:

[0016] The polyphonic words are annotated using a speech recognition alignment algorithm.

[0017] This invention also discloses a training apparatus for an end-to-end model, which may include:

[0018] The polyphonic word identification module is used to identify polyphonic words in the target training text;

[0019] The corpus generation module is used to annotate the polyphonic words with phonetic symbols and generate a corpus; the corpus includes the polyphonic words and phonetic symbols for the polyphonic words;

[0020] An audio information acquisition module is used to acquire audio information corresponding to the phonetic information;

[0021] An acoustic model generation module is used to train an end-to-end model using the corpus and the audio information, and to generate an acoustic model.

[0022] Optionally, the acoustic model is used to output a text sequence vector, the text sequence vector having a phonetic feature sequence vector for expressing the phonetic information, and the device may further include:

[0023] The recognition text generation module is used to recognize the audio information using the acoustic model and generate recognition text; the recognition text includes phonetic information of the recognition result;

[0024] The module for deleting phonetic information from the recognition results is used to delete the phonetic information from the recognition results.

[0025] Optionally, the end-to-end model has a corresponding text-to-speech module, and the corpus generation module may further include:

[0026] The first corpus generation submodule is used to annotate the polyphonic words with the text-to-speech module.

[0027] Optionally, the corpus generation module may further include:

[0028] The second corpus generation submodule is used to annotate the polyphonic words with phonetic symbols based on a speech recognition alignment algorithm.

[0029] This invention also discloses an electronic device, including a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other through the communication bus;

[0030] The memory is used to store computer programs;

[0031] When the processor executes a program stored in the memory, it implements the method described in the embodiments of the present invention.

[0032] This invention also discloses a computer-readable storage medium storing instructions that, when executed by one or more processors, cause the processors to perform the methods described in this invention.

[0033] The embodiments of the present invention have the following advantages:

[0034] In this embodiment of the invention, polyphonic words in the target training text are identified; the polyphonic words are annotated with phonetic symbols, and a corpus is generated; the corpus includes the polyphonic words and phonetic symbols for the polyphonic words; audio information corresponding to the phonetic symbols is obtained; and an end-to-end model is trained using the corpus and the audio information to generate an acoustic model, thereby improving the accuracy of end-to-end model training. Attached Figure Description

[0035] Figure 1 This is a flowchart illustrating the steps of a training method for an end-to-end model provided in an embodiment of the present invention.

[0036] Figure 2 This is a structural block diagram of a training device for an end-to-end model provided in an embodiment of the present invention;

[0037] Figure 3 This is a hardware structure block diagram of an electronic device provided in various embodiments of the present invention. Detailed Implementation

[0038] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0039] Traditional speech recognition systems consist of many modules, including an acoustic model, a pronunciation dictionary, and a language model. The acoustic and language models require training. These modules are typically trained independently, each with its own objective function. For example, the training objective of the acoustic model is to maximize the probability of the training speech, while the training objective of the language model is to minimize perplexity.

[0040] Because the modules cannot complement each other during training, and the training objective function deviates from the overall system performance metrics (usually word error rate, WER), the trained network often fails to achieve optimal performance.

[0041] This problem led to the development of end-to-end models.

[0042] An end-to-end model refers to a system that no longer has independent acoustic models, pronunciation dictionaries, language models, etc. Instead, it uses a single neural network to connect the input (speech waveform or feature sequence vector) to the output (word or character sequence vector). This neural network performs the functions of all the original modules. An example is Connected Temporal Classification (CTC), a neural network-based temporal classification method that can be understood as a recognition method for continuous sequence vectors. Specifically in speech recognition, the input sequence vector (audio) can be mapped to X = [x1, x2, ..., xT], and the corresponding output sequence vector is Y = [y1, y2, ..., yU]. The training goal of CTC is to maximize the match between X and Y, i.e., to maximize the output probability P(X|Y). Maximizing P(Y|X) is equivalent to establishing an accurate mapping between X and Y.

[0043] In practical applications, Chinese speech recognition based on end-to-end models requires selecting Chinese characters as modeling units. However, Chinese characters have multiple pronunciations. If Chinese characters are used directly as modeling units, during the training process, two Chinese characters with the same pronunciation but different pronunciations will be assigned to the same label for training, which will cause inaccurate model training.

[0044] Reference Figure 1 The diagram illustrates a flowchart of a training method for an end-to-end model provided in an embodiment of the present invention, which may specifically include the following steps:

[0045] Step 101: Identify polyphonic words in the target training text;

[0046] Step 102: Annotate the pronunciation of the polyphonic words and generate a corpus; the corpus includes the polyphonic words and the phonetic annotation information for the polyphonic words;

[0047] Step 103: Obtain the audio information corresponding to the phonetic information;

[0048] Step 104: Train the end-to-end model using the corpus and the audio information, and generate an acoustic model.

[0049] In a specific implementation, the training text of this invention can be text information including words. In this invention, polyphonic words in the target training text can be identified first.

[0050] Exemplarily, since the standard pronunciation of a character may change over time, a polyphonic character table can be constructed based on the latest stipulated Chinese pronunciation standard (e.g., the latest version of Xinhua Dictionary). Then, by comparing the characters in the target training text with the polyphonic character table, the polyphonic words in the target training text can be determined.

[0051] After determining the polyphonic words in the target training text, the embodiments of the present invention can phonetically annotate the polyphonic words. For example, for "我的" and "的确", the phonetic annotation results are "我的(_de)" and "的(_di)确". After completing the phonetic annotation, a corpus is generated.

[0052] In practical applications, a corpus is language material. The corpus of the embodiments of the present invention can include the polyphonic characters in the target training text and the phonetic annotation information for the polyphonic characters. Optionally, the phonetic annotation information can include Chinese pinyin and Chinese phonetic notation. For example, the corpus can be "除了(_le)" or can be "除了(_ㄌㄜ)".

[0053] The embodiments of the present invention can obtain audio information corresponding to the phonetic annotation information. Specifically, the audio information corresponding to the phonetic annotation information can be an audio with the same pronunciation as the phonetic annotation information. For example, the phonetic annotation information for the word "除了" is "(_le)", so audio information with the pronunciation of "除了(_le)" can be obtained from local or a server. Of course, it can also be obtained through manual input. The embodiments of the present invention do not limit this.

[0054] After generating the corpus and obtaining the audio information corresponding to the phonetic annotation information, the embodiments of the present invention can use the corpus and the audio information to train an end-to-end model and generate an acoustic model.

[0055] Exemplarily, the corpus containing polyphonic characters and the main cause information for the polyphonic characters can be first converted into a text sequence vector, and the audio information corresponding to the phonetic annotation information can be converted into an audio sequence vector. Then, the text training vector expressing the corpus is input into the end-to-end model to converge the end-to-end model, and finally, the end-to-end model is trained into an acoustic model. Among them, the text sequence vector expressing the corpus including the phonetic annotation information has a corresponding independent label, that is, the text sequence vector expressing the corpus including the phonetic annotation information does not correspond to the same label as other text sequence vectors expressing the same character with different pronunciations.

[0056] In an embodiment of the present invention, by determining the polyphonic words in the target training text; phonetically annotating the polyphonic words and generating a corpus; the corpus includes the polyphonic words and the phonetic annotation information for the polyphonic words; obtaining the audio information corresponding to the phonetic annotation information; training an end-to-end model using the corpus and the audio information, and generating an acoustic model, thereby improving the accuracy of training for the end-to-end model.

[0057] Based on the above embodiment, a variant embodiment of the above embodiment is proposed. Here, it should be noted that for the sake of brevity of description, only the differences from the above embodiment are described in the variant embodiment.

[0058] In an optional embodiment of the present invention, the method further includes:

[0059] Using the acoustic model to recognize the audio information and generating a recognition text; the recognition text includes recognition result phonetic annotation information;

[0060] Deleting the recognition result phonetic annotation information.

[0061] In practical applications, when the acoustic model performs speech recognition, the generated text will also include phonetic annotation information. If the phonetic annotation information is not processed, it will lead to data redundancy. Therefore, in order to avoid the recognition result still containing phonetic annotation information and further avoid information redundancy in the recognition result, the embodiment of the present invention can delete the recognition result phonetic annotation information when using the acoustic model to recognize the audio information and generating a recognition text including the recognition result phonetic annotation information.

[0062] For example, the target training text contains the word "的确". After training the end-to-end model with the corpus and the audio information and generating an acoustic model, an audio with the pronunciation of "的(_di)确" can be provided to the acoustic model for recognition. When the acoustic model listens to the audio with the pronunciation of "的(_di)确", the generated recognition text is "的(_di)确". At this time, "__di" can be deleted so that the recognition text "的确" does not contain the recognition result phonetic annotation information "__di".

[0063] In a specific implementation, when the acoustic model recognizes the audio information, it can first output a text sequence vector and generate a recognition text based on this text sequence vector. The text sequence vector output by the generated acoustic model contains a phonetic feature sequence vector for expressing the recognition result phonetic annotation information. Therefore, in an optional embodiment of the present invention, the recognition result phonetic annotation information can also be deleted by deleting the phonetic feature sequence vector.

[0064] In an embodiment of the present invention, the audio information is recognized by using the acoustic model, and a recognition text is generated; the recognition text includes recognition result phonetic information; the recognition result phonetic information is deleted, so as to avoid the recognition result still containing phonetic information, and further avoid the redundancy of the recognition result information.

[0065] In an optional embodiment of the present invention, the end-to-end model has a corresponding text-to-speech module, and the step of phoneticizing the polyphonic words further includes:

[0066] Phonetically annotating the polyphonic words by using the text-to-speech module.

[0067] In practical applications, an embodiment of the present invention can configure a text-to-speech module TTS (text-to-speech) for the end-to-end model. TTS is also called the front-end module. An embodiment of the present invention can phonetically annotate polyphonic words through the text-to-speech module TTS. Specifically, the text-to-speech module TTS can first analyze the text structure. For example, it can determine what language the polyphonic words in the target training text are. After determining the language of the polyphonic words, the text-to-speech module TTS can divide the sentence pattern in the target training text. After the text-to-speech module TTS finishes analyzing the text structure, it can regularize the polyphonic words in the target training text. In practical applications, the text regularization of the text-to-speech module TTS can convert punctuation or numbers that are not Chinese characters into Chinese characters. For example, in the sentence "This operation 666", the text-to-speech module TTS can convert "666" into "six six six". After the text-to-speech module TTS finishes regularizing the polyphonic words in the target training text, it can convert the polyphonic words in the target training text into phonemes, that is, convert the polyphonic words in the target training text into pinyin.

[0068] Of course, due to the existence of polyphonic words in Chinese, it is impossible to guarantee the accuracy of phonetic annotation for polyphonic words. Therefore, in an optional embodiment of the present invention, when the phonetic annotation method for polyphonic words is to use a text-to-speech module to phonetically annotate polyphonic words, and the result of the pronunciation test is different from the phonetic annotation information, the text-to-speech module can be used again to phonetically annotate the polyphonic words and generate new corpus. For example, the target training text contains the word "的确", and the text-to-speech module TTS phonetically annotates "的确" as "的(_de)确". After training the end-to-end model with the corpus and generating an acoustic model, the text containing the word "的确" can be provided to the acoustic model for reading again to test the pronunciation of the acoustic model when reading the word "的确". If the pronunciation of the test sound is "的(_de)确", it is determined that the training of the end-to-end model fails. Therefore, the text-to-speech module TTS can re-phonetically annotate "的确" as "的(_di)确" until the pronunciation of the acoustic model when reading the word "的确" is "的(_di)确".

[0069] In the embodiment of the present invention, by using the text-to-speech module to phonetically annotate the polyphonic words, the automatic phonetic annotation of polyphonic words is realized, thereby improving the training efficiency of the end-to-end model.

[0070] In an optional embodiment of the present invention, the end-to-end model has a corresponding text-to-speech module, and the step of phonetically annotating the polyphonic words further includes:

[0071] Phonetically annotating the polyphonic words based on a speech recognition alignment algorithm.

[0072] In a specific implementation, the embodiment of the present invention can phonetically annotate polyphonic words based on a speech recognition alignment algorithm.

[0073] For example, we can first define a phoneme set for Chinese characters, which includes all Chinese characters, initials, and finals (a, aa, ai, an, ang, ao, b, c, ch, d, e, ee, ei, en, eng, er, f, g, h, i, ia, ian, iang, iao, ie, ii, in, ing, iong, iu, ix, iy, iz, j, k, l, m, n, o, ong, oo, ou, p, q, r, s, sh, t, u, ua). uai, uan, uang, ueng, ui, un, uo, uu, v, van, ve, vn, vv, x, z, zh), and the mapping relationships between all Chinese characters and their initials and finals. Once the polyphonic words in the target training text are determined, the shortest path between the polyphonic character (element) and its initial and final (element) can be determined based on the speech recognition alignment algorithm through the mapping relationship between the polyphonic character (element) and its initial and final (element), and the pinyin corresponding to the polyphonic character (element) can be determined based on this path.

[0074] This invention, in its embodiment, uses a speech recognition alignment algorithm to annotate the pronunciation of polyphonic words. This achieves automatic pronunciation annotation of polyphonic words, thereby improving the training efficiency of end-to-end models.

[0075] It should be noted that, for the sake of simplicity, the method embodiments are all described as a series of actions. However, those skilled in the art should understand that the embodiments of the present invention are not limited to the described order of actions, because according to the embodiments of the present invention, some steps can be performed in other orders or simultaneously. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are preferred embodiments, and the actions involved are not necessarily essential to the embodiments of the present invention.

[0076] Reference Figure 2 The diagram illustrates a structural block diagram of a training device for an end-to-end model provided in an embodiment of the present invention, which may specifically include the following modules:

[0077] The polyphonic word identification module 201 is used to identify polyphonic words in the target training text;

[0078] The corpus generation module 202 is used to annotate the polyphonic words with phonetic symbols and generate a corpus; the corpus includes the polyphonic words and phonetic symbols for the polyphonic words;

[0079] Audio information acquisition module 203 is used to acquire audio information corresponding to the phonetic information;

[0080] The acoustic model generation module 204 is used to train the end-to-end model using the corpus and the audio information, and generate an acoustic model.

[0081] Optionally, the acoustic model is used to output a text sequence vector, the text sequence vector having a phonetic feature sequence vector for expressing the phonetic information, and the device may further include:

[0082] The recognition text generation module is used to recognize the audio information using the acoustic model and generate recognition text; the recognition text includes phonetic information of the recognition result;

[0083] The module for deleting phonetic information from the recognition results is used to delete the phonetic information from the recognition results.

[0084] Optionally, the end-to-end model has a corresponding text-to-speech module, and the corpus generation module may further include:

[0085] The first corpus generation submodule is used to annotate the polyphonic words with the text-to-speech module.

[0086] Optionally, the corpus generation module may further include:

[0087] The second corpus generation submodule is used to annotate the polyphonic words with phonetic symbols based on a speech recognition alignment algorithm.

[0088] As the device embodiment is basically similar to the method embodiment, the description is relatively simple, and relevant parts can be found in the description of the method embodiment.

[0089] In addition, this invention also provides an electronic device, including: a processor, a memory, and a computer program stored in the memory and executable on the processor. When the computer program is executed by the processor, it implements the various processes of the above-described training method embodiments for end-to-end models and achieves the same technical effect. To avoid repetition, it will not be described again here.

[0090] This invention also provides a computer-readable storage medium storing a computer program. When executed by a processor, the computer program implements the various processes of the above-described training method embodiments for end-to-end models and achieves the same technical effects. To avoid repetition, these will not be described again here. The computer-readable storage medium may be a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0091] Figure 3 A schematic diagram of the hardware structure of an electronic device for implementing various embodiments of the present invention.

[0092] The electronic device 300 includes, but is not limited to, components such as: a radio frequency unit 301, a network module 302, an audio output unit 303, an input unit 304, a sensor 305, a display unit 306, a user input unit 307, an interface unit 308, a memory 309, a processor 310, and a power supply 311. Those skilled in the art will understand that... Figure 3 The electronic device structures shown are not intended to limit the electronic device. An electronic device may include more or fewer components than shown, or combine certain components, or have different component arrangements. In embodiments of the present invention, the electronic device includes, but is not limited to, mobile phones, tablet computers, laptops, PDAs, in-vehicle terminals, wearable devices, and pedometers.

[0093] It should be understood that, in this embodiment of the invention, the radio frequency unit 301 can be used for receiving and transmitting signals during information transmission or calls. Specifically, it receives downlink data from the base station and processes it with the processor 310; additionally, it transmits uplink data to the base station. Typically, the radio frequency unit 301 includes, but is not limited to, an antenna, at least one amplifier, a transceiver, a coupler, a low-noise amplifier, a duplexer, etc. Furthermore, the radio frequency unit 301 can also communicate with networks and other devices through a wireless communication system.

[0094] The electronic device provides users with wireless broadband internet access through network module 302, such as helping users send and receive emails, browse web pages, and access streaming media.

[0095] The audio output unit 303 can convert audio data received by the radio frequency unit 301 or the network module 302 or stored in the memory 309 into audio signals and output them as sound. Furthermore, the audio output unit 303 can also provide audio output related to specific functions performed by the electronic device 300 (e.g., call signal reception sound, message reception sound, etc.). The audio output unit 303 includes a speaker, a buzzer, and a receiver, etc.

[0096] Input unit 304 is used to receive audio or video signals. Input unit 304 may include a graphics processing unit (GPU) 3041 and a microphone 3042. The GPU 3041 processes image data of still images or videos acquired by an image capture device (such as a camera) in video capture mode or image capture mode. The processed image frames can be displayed on display unit 306. The image frames processed by GPU 3041 can be stored in memory 309 (or other storage media) or transmitted via radio frequency unit 301 or network module 302. Microphone 3042 can receive sound and process such sound into audio data. The processed audio data can be converted into a format that can be transmitted to a mobile communication base station via radio frequency unit 301 in telephone call mode.

[0097] The electronic device 300 also includes at least one sensor 305, such as a light sensor, a motion sensor, and other sensors. Specifically, the light sensor includes an ambient light sensor and a proximity sensor. The ambient light sensor can adjust the brightness of the display panel 3061 according to the ambient light level, and the proximity sensor can turn off the display panel 3061 and / or backlight when the electronic device 300 is moved to the ear. As a type of motion sensor, an accelerometer sensor can detect the magnitude of acceleration in various directions (generally three axes). When stationary, it can detect the magnitude and direction of gravity and can be used to identify the posture of the electronic device (such as landscape / portrait switching, related games, magnetometer posture calibration), vibration recognition related functions (such as pedometer, tapping), etc. The sensor 305 may also include a fingerprint sensor, pressure sensor, iris sensor, molecular sensor, gyroscope, barometer, hygrometer, thermometer, infrared sensor, etc., which will not be described in detail here.

[0098] The display unit 306 is used to display information input by the user or information provided to the user. The display unit 306 may include a display panel 3061, which may be configured in the form of a liquid crystal display (LCD), an organic light-emitting diode (OLED), or the like.

[0099] User input unit 307 can be used to receive input numerical or character information, and generate key signal inputs related to user settings and function control of electronic devices. Specifically, user input unit 307 includes a touch panel 3071 and other input devices 3072. Touch panel 3071, also known as a touch screen, can collect touch operations performed by the user on or near it (such as operations performed by the user using a finger, stylus, or any suitable object or accessory on or near touch panel 3071). Touch panel 3071 may include two parts: a touch detection device and a touch controller. The touch detection device detects the user's touch position and the signal generated by the touch operation, and transmits the signal to the touch controller; the touch controller receives touch information from the touch detection device, converts it into touch point coordinates, and sends it to the processor 310, which receives and executes commands from the processor 310. In addition, touch panel 3071 can be implemented using various types such as resistive, capacitive, infrared, and surface acoustic wave. In addition to touch panel 3071, user input unit 307 may also include other input devices 3072. Specifically, other input devices 3072 may include, but are not limited to, physical keyboards, function keys (such as volume control buttons, power buttons, etc.), trackballs, mice, joysticks, etc., which will not be described in detail here.

[0100] Furthermore, the touch panel 3071 can cover the display panel 3061. When the touch panel 3071 detects a touch operation on or near it, it transmits the information to the processor 310 to determine the type of touch event. Subsequently, the processor 310 provides corresponding visual output on the display panel 3061 based on the type of touch event. Although in Figure 3 In this embodiment, the touch panel 3071 and the display panel 3061 are two independent components to realize the input and output functions of the electronic device. However, in some embodiments, the touch panel 3071 and the display panel 3061 can be integrated to realize the input and output functions of the electronic device. The specific implementation is not limited here.

[0101] Interface unit 308 serves as an interface for connecting external devices to electronic device 300. For example, external devices may include a wired or wireless headphone port, an external power supply (or battery charger) port, a wired or wireless data port, a memory card port, a port for connecting a device with an identification module, an audio input / output (I / O) port, a video I / O port, a headphone port, and so on. Interface unit 308 can be used to receive input from external devices (e.g., data, power, etc.) and transmit the received input to one or more components within electronic device 300, or it can be used to transmit data between electronic device 300 and external devices.

[0102] The memory 309 can be used to store software programs and various data. The memory 309 may primarily include a program storage area and a data storage area. The program storage area may store the operating system, applications required for at least one function (such as sound playback, image playback, etc.), etc.; the data storage area may store data created based on the use of the mobile phone (such as audio data, phonebook, etc.). Furthermore, the memory 309 may include high-speed random access memory, and may also include non-volatile memory, such as at least one disk storage device, flash memory device, or other volatile solid-state storage device.

[0103] The processor 310 is the control center of the electronic device. It connects various parts of the electronic device via various interfaces and lines. By running or executing software programs and / or modules stored in the memory 309, and by calling data stored in the memory 309, it performs various functions and processes data, thereby providing overall monitoring of the electronic device. The processor 310 may include one or more processing units; preferably, the processor 310 may integrate an application processor and a modem processor. The application processor mainly handles the operating system, user interface, and applications, while the modem processor mainly handles wireless communication. It is understood that the modem processor may not be integrated into the processor 310.

[0104] The electronic device 300 may also include a power supply 311 (such as a battery) for supplying power to various components. Preferably, the power supply 311 can be logically connected to the processor 310 through a power management system, thereby enabling functions such as managing charging, discharging, and power consumption through the power management system.

[0105] In addition, the electronic device 300 includes some functional modules not shown, which will not be described in detail here.

[0106] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0107] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes several instructions to cause a terminal (which may be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in the various embodiments of the present invention.

[0108] The embodiments of the present invention have been described above with reference to the accompanying drawings. However, the present invention is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other forms under the guidance of the present invention without departing from the spirit and scope of the claims, and all of these forms are within the protection scope of the present invention.

[0109] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed in this invention can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this invention.

[0110] Those skilled in the art will understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0111] In the embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative. For instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0112] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0113] In addition, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0114] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, essentially, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, ROM, RAM, magnetic disks, or optical disks.

[0115] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A training method for an end-to-end model, characterized in that, include: Identify polyphonic words in the target training text; The polyphonic characters and words are annotated with phonetic symbols, and a corpus is generated. The corpus includes the polyphonic words and the phonetic annotation information for the polyphonic words; wherein the polyphonic words and the corresponding phonetic annotation information are bound as independent tags; Obtain the audio information corresponding to the phonetic information; The end-to-end model is trained using the corpus and the audio information, and an acoustic model is generated. The step of training the end-to-end model using the corpus and the audio information to generate an acoustic model includes: The corpus is then converted into a text sequence vector with phonetic feature sequence vectors; The audio information is converted into an audio sequence vector; The end-to-end model is trained to generate the acoustic model; The acoustic model outputs a text sequence vector, which contains a phonetic feature sequence vector used to distinguish the pronunciation of polyphonic characters.

2. The method according to claim 1, characterized in that, The method further includes: The acoustic model is used to identify the audio information and generate identified text; the identified text includes phonetic information of the identification result. Delete the phonetic information in the recognition result.

3. The method according to claim 1, characterized in that, The end-to-end model has a corresponding text-to-speech module, and the step of annotating the polyphonic words with phonetic symbols further includes: The text-to-speech module is used to annotate the polyphonic characters and words.

4. The method according to claim 1, characterized in that, The step of annotating the pronunciation of the polyphonic characters further includes: The polyphonic words are annotated using a speech recognition alignment algorithm.

5. A training device for an end-to-end model, characterized in that, include: The polyphonic word identification module is used to identify polyphonic words in the target training text; The corpus generation module is used to annotate the polyphonic characters and words with phonetic symbols and generate corpus. The corpus includes the polyphonic words and the phonetic annotation information for the polyphonic words; wherein the polyphonic words and the corresponding phonetic annotation information are bound as independent tags; An audio information acquisition module is used to acquire audio information corresponding to the phonetic information; An acoustic model generation module is used to train an end-to-end model using the corpus and the audio information, and to generate an acoustic model. The step of training the end-to-end model using the corpus and the audio information to generate an acoustic model includes: The corpus is then converted into a text sequence vector with phonetic feature sequence vectors; The audio information is converted into an audio sequence vector; The end-to-end model is trained to generate the acoustic model; The acoustic model is used to output a text sequence vector, which has a phonetic feature sequence vector for expressing the phonetic information.

6. The apparatus according to claim 5, characterized in that, The device further includes: The recognition text generation module is used to recognize the audio information using the acoustic model and generate recognition text; the recognition text includes phonetic information of the recognition result; The module for deleting phonetic information from the recognition results is used to delete the phonetic information from the recognition results.

7. The apparatus according to claim 5, characterized in that, The end-to-end model has a corresponding text-to-speech module, and the corpus generation module further includes: The first corpus generation submodule is used to annotate the polyphonic words with the text-to-speech module.

8. The apparatus according to claim 5, characterized in that, The corpus generation module further includes: The second corpus generation submodule is used to annotate the polyphonic words with phonetic symbols based on a speech recognition alignment algorithm.

9. An electronic device, characterized in that, It includes a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other through the communication bus; The memory is used to store computer programs; When the processor executes a program stored in the memory, it implements the method as described in any one of claims 1-4.

10. A computer-readable storage medium having instructions stored thereon that, when executed by one or more processors, cause the processors to perform the method as described in any one of claims 1-4.

Images