Speech recognition method, electronic device, and computer-readable storage medium

Abstract

The application provides a speech recognition method, an electronic device and a computer readable storage medium. In the method, the electronic device can determine multiple candidate characters based on the prediction result of the acoustic model when recognizing the character corresponding to each pronunciation, and then perform multiple screenings based on the pronunciation and acoustic probability score of the multiple candidate characters. Finally, the electronic device can further screen the finally determined candidate characters based on the language model to determine the final recognition result. The method can reduce the computational amount of the electronic device while ensuring the accuracy of speech recognition, thereby improving the speed of the electronic device when recognizing speech.

Description

Technical Field

[0001] This application relates to the field of terminal and communication technology, and in particular to speech recognition methods, electronic devices and computer-readable storage media. Background Technology

[0002] Electronic devices typically need to recognize the speech content within the audio during recording, and then provide one or more functions to the user based on the recognized speech content. For example, after recognizing the speech content, the electronic device can display it in a visual form such as text or images for the user to view and edit, or the electronic device can engage in dialogue with the user based on the speech content. Sometimes, if the electronic device's speech content recognition speed is too slow, it will reduce the speed at which it provides subsequent functions, thus affecting the user experience. Summary of the Invention

[0003] This application provides a speech recognition method, an electronic device, and a computer-readable storage medium. The electronic device can first determine multiple candidate characters corresponding to a single audio frame based on a single audio frame and a sequence of recognized characters. Then, the electronic device can perform multiple filtering processes on the candidate characters based on their acoustic probability scores and linguistic probability scores, ultimately determining the character corresponding to the audio frame from the filtered candidate characters. After multiple filtering processes, the electronic device reduces the number of candidate characters, lowering the computational load during speech recognition. This reduces user waiting time and improves the user experience when using speech recognition-related functions.

[0004] In a first aspect, this application provides a speech recognition method applied to an electronic device. The method includes: the electronic device receiving a first audio segment, the first audio segment comprising a Kth frame of audio and N frames of audio preceding the Kth frame, where K is a positive integer and N is a positive integer; the electronic device determining multiple character sequences based on a first prediction result of the Kth frame of audio and a first character sequence corresponding to the recognized N frames of audio, wherein any one of the multiple character sequences contains a candidate character corresponding to the Kth frame of audio in the first character sequence, and the first prediction result includes multiple candidate characters of the Kth frame of audio; the electronic device selecting L1 character sequences from the multiple character sequences, where the acoustic probability score of the L1 character sequences is the highest among the multiple character sequences, and the acoustic probability score of a character sequence represents the Kth frame of audio and the N frames of audio. The electronic device selects L2 character sequences from multiple character sequences based on L1 character sequences. The pronunciation of the candidate character in the Kth frame audio of the L2 character sequences satisfies the first condition with the pronunciation of the candidate character in the Kth frame audio of any character sequence in the L1 character sequences. L2 is a positive integer. The electronic device selects L3 character sequences from the L2 character sequences. The acoustic probability score of the L3 character sequences satisfies the second condition. L3 is a positive integer. The electronic device selects a second character sequence from the L3 character sequences based on the second prediction result of the Kth frame audio and determines the second character sequence as the speech recognition result of the Kth frame audio and the previous N frames audio. The second prediction result is obtained by the electronic device based on the semantic prediction of the speech content of the Kth frame audio.

[0005] In conjunction with the first aspect, in some embodiments, the first condition includes any one of the following: the two pronunciations have the same vowel, the two pronunciations differ only in tone, and the two pronunciations are front and back nasal sounds.

[0006] The first audio segment can be a valid audio segment corresponding to a complete sentence. The first N frames of audio preceding the Kth frame in the first audio segment contain pronunciations, each pronunciation corresponding to a character. The electronic device can determine a first prediction result based on the first character sequence, where the first prediction result may include the probability distribution matrix of the character corresponding to the Kth frame audio output by the acoustic model. The character sequence A can contain multiple character sequences, and therefore the acoustic model can obtain a probability distribution matrix for each character sequence. A probability distribution matrix represents the acoustic probability score of multiple candidate characters under a character sequence. Figure 4Taking the illustrated embodiment as an example, the characters in the (K-1)th frame include "chew" and "pay". The character sequence A may include the character sequence corresponding to the audio before and including the (K-1)th frame. That is to say, the character sequence A includes both the character sequence composed of the recognized characters under the branch of "chew" and "chew", and the character sequence composed of the recognized characters under the branch of "pay" and "pay". The multiple character sequences determined by the electronic device based on the first prediction result include the recognized character sequence and the candidate characters of the audio in the Kth frame under this character sequence, so as to Figure 4Taking the illustrated embodiment as an example, the recognized character sequences include "嚼" and "缴". Among them, the candidate characters for the audio of the K-th frame under the character sequence where "嚼" is located include: "口", "抠", "寇", "扣", etc. The candidate characters for the audio of the K-th frame under the character sequence where "缴" is located include: "扣", "获", "口", "叩", etc. Thus, the multiple character sequences determined by the electronic device include: "嚼口", "嚼抠", "嚼寇", "嚼扣", "缴扣", "缴获", "缴口", "缴叩", etc. Furthermore, the electronic device can screen out L1 character sequences with the highest acoustic probability scores from the multiple character sequences, that is, screen out the character with the highest acoustic probability score corresponding to the audio of the K-th frame under a certain character sequence A. For example, screen out "嚼口", "嚼抠", "缴扣", "缴获" with the highest acoustic probability scores from the multiple character sequences. Furthermore, the electronic device can determine the pronunciation of the character corresponding to the audio of the K-th frame in the screened character sequences, and then screen out the characters with the same or similar pronunciation as the character corresponding to the above-mentioned audio of the K-th frame from the multiple character sequences to form L2 new character sequences. For example, the electronic device selects the pronunciation with the same or similar pronunciation as "口" and "抠" from all the candidate characters of the K-th frame to form new character sequences, thus expanding the candidate character sequences to "嚼口", "嚼抠", "嚼寇", "嚼扣", etc. Here, the L2 character sequences include both the above-mentioned L1 character sequences and the character sequences expanded based on the L1 character sequences. Then the electronic device can screen out L3 character sequences with the highest acoustic probability scores from the L2 character sequences. Finally, the electronic device can select the second character sequence from the L3 character sequences based on the second prediction result of the language model. Among them, the second preset condition can be that the character sequence is one of the character sequences with the highest acoustic probability scores in the L2 character sequences. Here, the highest acoustic probability score can refer to the top Q1 in the acoustic probability scores, Q1 is a positive integer, and Q1 can be 1, 2, 3, etc. The second prediction result can include the probability distribution matrix output by the language model, and the probability distribution matrix contains the language probability scores of the candidate characters corresponding to the audio of the K-th frame under each character sequence A. Among them, the electronic device will look up the language probability scores of the characters corresponding to the audio of the K-th frame in the L3 character sequences from the second prediction result, and then calculate the total score based on the acoustic probability scores and language probability scores of the L3 character sequences. Finally, the electronic device can determine the second character sequence based on the total score and use the second character sequence as the speech recognition result of the audio of the K-th frame and the N frames of audio before the K-th frame. Among them, the second character sequence can be the character sequence with the highest total score among the L3 character sequences. The value of L1 can be small, and the value of L3 can be relatively large compared to L1. For example, L1 can be 3, 4, or 5, etc., and L3 can be 8, 9, or 10, etc.

[0007] That is to say, the electronic device can first determine L1 character sequences with the highest acoustic probability scores from all character sequences, then screen out L2 character sequences from all character sequences based on the pronunciation of the Kth frame audio in the L1 character sequences, and finally screen out the character sequence with the highest acoustic probability score from the L2 character sequences. Compared with determining the final recognition result from a large number of character sequences, the electronic device can reduce the amount of calculation and improve the speed during speech recognition by using the above method.

[0008] Combined with the first aspect, in some embodiments, the electronic device screens out L3 character sequences from the L2 character sequences, specifically including: the electronic device screens out L4 character sequences from the L2 character sequences, and the pronunciation of the character corresponding to the Kth frame audio in the L4 character sequences appears with the highest frequency, where L4 is a positive integer; the electronic device screens out L3 character sequences from the L4 character sequences, and the L4 character sequences are those with the highest acoustic probability scores among the L3 character sequences.

[0009] Reference Figure 4 In the embodiment shown, after the electronic device also uses the characters with the same or similar pronunciations as the candidate characters among the screened candidate characters, it can first screen out the character with the highest pronunciation frequency from the candidate characters. In the branch where the (K - 1)th frame is "缴", candidate characters such as "扣", "获", "口", "叩" are screened and "获" is removed, where "获" is a character with a relatively low pronunciation frequency. The above highest pronunciation frequency can mean that the pronunciation ranks among the top Q2 in terms of the number of occurrences in all candidate characters. Then the electronic device further screens out the candidate character with the highest acoustic probability score from the characters with the highest pronunciation frequency, and obtains L3 character sequences, including: "缴扣", "缴口", "缴叩".

[0010] That is to say, after the electronic device expands the candidate characters of the Kth frame based on the pronunciation of the existing candidate characters, it can then filter out some characters with relatively low pronunciation frequencies. In this way, the electronic device can remove some candidate characters introduced by the prediction network and irrelevant to the pronunciation in the Kth frame audio (such as Figure 4 "获" in the embodiment shown), which can ensure that the candidate characters for the Kth frame audio to be recognized determined by the acoustic model depend more on the audio features of the Kth frame audio, reducing the amount of calculation and ensuring the final recognition accuracy.

[0011] Combined with the first aspect, in some embodiments, the first character sequence includes a third character sequence and a fourth character sequence, and any one of the multiple character sequences contains a candidate character corresponding to the third character sequence and the Kth frame audio, or contains a candidate character corresponding to the fourth character sequence and the Kth frame audio.

[0012] That is to say, the first character sequence can contain multiple character sequences. Taking Figure 4Taking the illustrated embodiment as an example, according to the differences in the content recognized from N frames before the Kth frame, the first character sequence may include the character sequence where "chew" is located and the character sequence where "mouth" is located. Here, the character sequence where "chew" is located can be referred to as the third character sequence, and the character sequence where "mouth" is located can be referred to as the fourth character sequence. Then, among the multiple character sequences determined by the electronic device, there are both the third character sequence and the candidate characters of the Kth frame audio recognized based on the third character sequence (including character sequences such as "chew mouth", "chew pick", "chew kou", "chew buckle", etc.), and the fourth character sequence and the candidate characters of the Kth frame audio recognized based on the fourth character sequence (including character sequences such as "seize buckle", "seize capture", "seize mouth", "seize kowtow", etc.). When the electronic device screens L1 character sequences from the multiple character sequences, it can be screened from the third character sequence or from the fourth character sequence.

[0013] In combination with the first aspect, in some embodiments, the method further includes: The electronic device makes predictions based on the Kth frame audio and the first character sequence to obtain multiple first candidate characters of the Kth frame audio, and determines the acoustic probability scores of the multiple first candidate characters. The first prediction result includes the acoustic probability scores; The electronic device makes semantic predictions on the Kth frame audio according to the first character sequence to obtain multiple second candidate characters of the Kth frame audio, and determines the language probability scores of the multiple second candidate characters. The language probability score includes the probability that the character corresponding to the Kth frame audio under the first character sequence is each candidate character among the second candidate characters. Where the second prediction result includes the language probability scores.

[0014] In combination with the first aspect, in some embodiments, the electronic device selects the second character sequence from L3 character sequences based on the second prediction result of the character corresponding to the Kth frame audio, including: The electronic device calculates the total score of each character sequence among the L3 character sequences. The total score is equal to the product of the language probability score and the first weight plus the acoustic probability score; The electronic device selects the second character sequence based on the total scores of the L3 character sequences.

[0015] Among them, the electronic device can make predictions to obtain the first candidate characters based on the Kth frame audio and the first character sequence through an acoustic model, and determine the second candidate characters according to the semantics of the first character sequence through a language model. Furthermore, the electronic device can determine the total score based on the acoustic probability score and the language probability score of each character sequence. Referring to Figure 5 the illustrated embodiment, the electronic device calculates the total score of each character sequence respectively, and then obtains the total scores of "chew mouth", "chew pick", "chew kou", "seize buckle", "seize mouth", "seize kowtow". Finally, the electronic device can select the character sequence with the highest total score as the final recognition result. For example Figure 5In the illustrated embodiment, "chewing" has the highest total score, and "chewing" is the second character sequence ultimately determined by the electronic device. Optionally, the electronic device can input multiple character sequences with high total scores back into the acoustic model and language model, allowing the acoustic model and language model to identify the character in the (K+1)th frame based on the currently recognized character sequence. For example, the electronic device can input "chewing" and "deduction" into the acoustic model and language model.

[0016] In conjunction with the first aspect, in some embodiments, the first weight gradually increases as the value of N increases.

[0017] In other words, electronic devices increase the initial weight based on the recognized character sequence. When the recognized character sequence is small, the language model has limited context to refer to when predicting characters, resulting in relatively low accuracy. As the number of recognized character sequences increases, the language model can predict the next character based on more context, and its prediction accuracy gradually improves. Electronic devices can gradually increase the initial weight as the length of the recognized character sequence increases, meaning they gradually increase the impact of the language probability score on the total score. This allows the electronic device to determine the corresponding character based more on the acoustic model when the number of recognized characters is small, while the language model can provide more semantic correction to the acoustic model's predictions when the number of recognized characters is large, thus improving the accuracy of speech recognition.

[0018] In conjunction with the first aspect, in some embodiments, when K is 1, the first weight is 0.

[0019] When electronic devices recognize the first character of a sentence, the language model has no previously recognized character sequences to refer to. In this case, the language model relies more on training data to determine the language probability scores of candidate characters, resulting in lower accuracy. Therefore, when recognizing the first character, electronic devices can set the first weight to 1, meaning they ignore the language probability score predicted by the language model for the first character. This way, the total score of the candidate characters for the first character depends only on their acoustic probability scores, without being affected by the language model, thus improving the accuracy of first character recognition.

[0020] In a second aspect, this application provides an electronic device including a display screen, a memory, and a processor coupled to the memory; the display screen is used to display an interface, the memory stores a computer program, and when the processor executes the computer program, it causes the electronic device to implement the method described in any one of the first aspects.

[0021] Thirdly, this application provides a computer-readable storage medium storing a computer program or computer instructions, which are executed by a processor to implement the method described in any of the first aspects above.

[0022] Fourthly, embodiments of this application provide a computer program product, which, when executed by a processor, implements the method described in any of the first aspects above.

[0023] Fifthly, embodiments of this application provide a chip including a processor and a memory, wherein the memory is used to store computer programs or computer instructions, and the processor is used to execute the computer programs or computer instructions stored in the memory, causing the chip to perform the method described in any of the first aspects above.

[0024] The solutions provided in the second to fifth aspects above are used to implement or cooperate with the methods provided in the first aspect above, and therefore can achieve the same or corresponding beneficial effects as the methods in the first aspect, which will not be elaborated here. Attached Figure Description

[0025] Figure 1 This is a schematic diagram of the structure of the electronic device provided in the embodiments of this application;

[0026] Figure 2 This is a schematic diagram of the architecture of the speech recognition system 200 provided in an embodiment of this application;

[0027] Figure 3 This is a flowchart of the speech recognition method provided in the embodiments of this application;

[0028] Figures 4-5 This is a specific example diagram of the electronic device recognizing speech provided in the embodiments of this application. Detailed Implementation

[0029] The terminology used in the following embodiments of this application is for the purpose of describing particular embodiments only and is not intended to be limiting of this application. As used in the specification and appended claims of this application, the singular expressions “a,” “an,” “the,” “the,” “the,” and “this” are intended to include the plural expressions as well, unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used in this application refers to and includes any or all possible combinations of one or more of the listed items.

[0030] Hereinafter, the terms "first" and "second" are used for descriptive purposes only and should not be construed as implying or suggesting relative importance or implicitly indicating the number of indicated technical features. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature, and in the description of the embodiments of this application, unless otherwise stated, "multiple" means two or more.

[0031] Electronic devices typically recognize the speech content within the audio while recording it, and then provide users with one or more functions based on this speech content. These functions may include, but are not limited to, voice notes, voice translation, and human-computer dialogue. Voice notes refer to the electronic device displaying the speech content recognized from the audio in a visual format such as text or images for the user to view and edit. Voice translation refers to the electronic device converting the recognized speech content from one language to another and playing the converted speech content, or displaying the converted speech content as text. Human-computer dialogue refers to the electronic device determining the meaning of the speech content after recognizing it in the audio, and then being able to converse with the user or execute commands contained in the speech content. Sometimes, the speed at which electronic devices recognize speech content from audio is too slow, causing them to take a long time to provide the corresponding functions to the user. For example, in the voice notes function, the electronic device may have received the audio for a long time before displaying the corresponding speech content text; in the human-computer dialogue function, the electronic device may have received the user's voice command for a long time before executing the action. This results in excessively long waiting times for the user, degrading the user experience.

[0032] To improve the speed of speech recognition in electronic devices, embodiments of this application provide a speech recognition method, an electronic device, and a computer-readable storage medium. In this method, for a single phonation, the electronic device first filters out a small group of candidate characters based on the acoustic probability scores of a large number of candidate characters. Then, the electronic device determines the character corresponding to the phonation from this group of candidate characters based on the acoustic probability scores and linguistic probability scores of the filtered candidate characters. The acoustic probability score is the probability predicted by the acoustic model based on the acoustic features of the phonation for each candidate character, and the linguistic probability score is the probability predicted by the language model based on the linguistic features of the context for each candidate character. Using this method, the electronic device can reduce its computational load during speech recognition, improve the speed of speech recognition, thereby reducing user waiting time and improving the user experience when using speech recognition-related functions.

[0033] For ease of description and better understanding, the following embodiments of this application use the recognition of Chinese by an electronic device as an example to introduce the speech recognition method. However, the method is not limited to recognizing Chinese; the electronic device can also recognize speech in other languages ​​using this speech recognition method. For methods of recognizing speech in other languages, please refer to the description of the method for recognizing Chinese by an electronic device in the following embodiments; this application will not elaborate on these methods.

[0034] The exemplary electronic device 100 provided in the embodiments of this application will be introduced first below.

[0035] Figure 1This is a schematic diagram of the structure of the electronic device 100 provided in the embodiments of this application.

[0036] The following description uses electronic device 100 as an example to illustrate the embodiment. It should be understood that electronic device 100 may have more than Figure 1 The more or fewer components shown can be combined into two or more components, or they can have different component configurations. Figure 1 The various components shown can be implemented in hardware, software, or a combination of hardware and software, including one or more signal processing and / or application-specific integrated circuits.

[0037] Electronic device 100 may include: a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, a charging management module 140, a power management module 141, a battery 142, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, a headphone jack 170D, a sensor module 180, a display screen 190, etc. The sensor module 180 may include a pressure sensor 180A, a touch sensor 180B, etc.

[0038] Processor 110 may include one or more processing units, such as: application processor (AP), modem processor, graphics processing unit (GPU), image signal processor (ISP), controller, memory, digital signal processor (DSP), baseband processor, and / or neural network processing unit (NPU), etc. Different processing units may be independent devices or integrated into one or more processors.

[0039] The controller can be the nerve center and command center of the electronic device 100. The controller can generate operation control signals according to the instruction opcode and timing signals to complete the control of fetching and executing instructions.

[0040] The processor 110 may also include a memory for storing instructions and data. In some embodiments, the memory in the processor 110 is a cache memory. This memory can store instructions or data that the processor 110 has just used or that are used repeatedly. If the processor 110 needs to use the instruction or data again, it can retrieve it directly from the memory. This avoids repeated accesses, reduces the waiting time of the processor 110, and thus improves the efficiency of the system.

[0041] The charging management module 140 is used to receive charging input from the charger. The charger can be a wireless charger or a wired charger.

[0042] The power management module 141 is used to connect the battery 142, the charging management module 140, and the processor 110. The power management module 141 receives input from the battery 142 and / or the charging management module 140 to power the processor 110, internal memory 121, external memory, display 190, etc.

[0043] The modem processor may include a modulator and a demodulator. The modulator modulates a low-frequency baseband signal to be transmitted into a mid-to-high frequency signal. The demodulator demodulates the received electromagnetic wave signal into a low-frequency baseband signal. The demodulator then transmits the demodulated low-frequency baseband signal to the baseband processor for processing. After processing by the baseband processor, the low-frequency baseband signal is transmitted to the application processor. The application processor outputs an audio signal through an audio device (not limited to a speaker 170A, receiver 170B, etc.) or displays an image or video through a display screen 190. In some embodiments, the modem processor may be a separate device.

[0044] Electronic device 100 implements display functions through a GPU, a display screen 190, and an application processor. The GPU is a microprocessor for image processing, connected to the display screen 190 and the application processor. The GPU is used to perform mathematical and geometric calculations and for graphics rendering. Processor 110 may include one or more GPUs, which execute program instructions to generate or modify display information.

[0045] The display screen 190 is used to display images, videos, etc. The display screen 190 includes a display panel. The display panel may be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (FLED), a Mini LED, a MicroLED, a Micro-OLED, a quantum dot light-emitting diode (QLED), etc. In some embodiments, the electronic device 100 may include one or N display screens 190, where N is a positive integer greater than 1.

[0046] Digital signal processors (DSPs) are used to process digital signals. Besides digital image signals, they can also process other digital signals. For example, when electronic device 100 selects a frequency, the DSP can perform Fourier transforms on the frequency energy.

[0047] NPU stands for Neural-Network (NN) Computing Processor. By drawing inspiration from the structure of biological neural networks, such as the transmission patterns between neurons in the human brain, it can quickly process input information and continuously learn on its own.

[0048] The external storage interface 120 can be used to connect an external memory card, such as a Micro SD card, to expand the storage capacity of the electronic device 100. The external memory card communicates with the processor 110 through the external storage interface 120 to perform data storage functions. For example, music, video, and other files can be saved on the external memory card.

[0049] Internal memory 121 can be used to store computer executable program code, which includes instructions. Processor 110 executes various functional applications and data processing of electronic device 100 by running the instructions stored in internal memory 121. Internal memory 121 may include a program storage area and a data storage area. The program storage area may store the operating system, at least one application required for a function (such as facial recognition, fingerprint recognition, mobile payment, etc.). The data storage area may store data created during the use of electronic device 100 (such as facial information template data, fingerprint information templates, etc.). Furthermore, internal memory 121 may include high-speed random access memory and non-volatile memory, such as at least one disk storage device, flash memory device, universal flash storage (UFS), etc.

[0050] Electronic device 100 can implement audio functions, such as music playback and recording, through audio module 170, speaker 170A, receiver 170B, microphone 170C, headphone jack 170D, and application processor.

[0051] Audio module 170 is used to convert digital audio information into analog audio signal output, and also to convert analog audio input into digital audio signal. Audio module 170 can also be used for encoding and decoding audio signals.

[0052] The speaker 170A, also known as a "loudspeaker," is used to convert audio electrical signals into sound signals. The electronic device 100 can listen to music or make hands-free calls through the speaker 170A.

[0053] The receiver 170B, also known as the "earpiece," is used to convert audio electrical signals into sound signals. When the electronic device 100 answers a telephone call or voice message, the receiver 170B can be brought close to the ear to listen to the voice.

[0054] The microphone 170C, also known as a "microphone" or "voice transducer," is used to convert sound signals into electrical signals.

[0055] The 170D headphone jack is used to connect wired headphones. The 170D headphone jack can be a USB 130 interface or a 3.5mm Open Mobile Terminal Platform (OMTP) standard interface, a CTIA (Cellular Telecommunications Industry Association of the USA) standard interface.

[0056] Pressure sensor 180A is used to sense pressure signals and convert them into electrical signals. In some embodiments, pressure sensor 180A may be disposed on display screen 190. There are many types of pressure sensors 180A, such as resistive pressure sensors, inductive pressure sensors, and capacitive pressure sensors. A capacitive pressure sensor may include at least two parallel plates with conductive material. When a force is applied to pressure sensor 180A, the capacitance between the electrodes changes. Electronic device 100 determines the pressure intensity based on the change in capacitance. When a touch operation is applied to display screen 190, electronic device 100 detects the intensity of the touch operation based on pressure sensor 180A. Electronic device 100 may also calculate the touch position based on the detection signal from pressure sensor 180A.

[0057] Touch sensor 180B, also known as a "touch panel," can be located on display screen 190. The touch sensor 180B and display screen 190 together form a touchscreen, also known as a "touch screen." Touch sensor 180B is used to detect touch operations applied to or near it. The touch sensor can transmit the detected touch operation to the application processor to determine the type of touch event. Visual output related to the touch operation can be provided through display screen 190. In other embodiments, touch sensor 180B may also be located on the surface of electronic device 100, in a different position than display screen 190.

[0058] In some embodiments, the electronic device can record an audio stream via microphone 170C. Audio module 170 can convert the analog audio data output from microphone 170C into a digital signal. After resampling and denoising the digital audio signal output from audio module 170, the DSP can send the audio data to NPU. Then, NPU can recognize the speech content in the audio. The method for NPU to recognize speech content in audio can be found in the following embodiments, and will not be elaborated here.

[0059] The electronic device 100 may be equipped with a voice recognition system, which is used to recognize the voice content in the audio received by the electronic device.

[0060] The following describes the speech recognition system provided in the embodiments of this application.

[0061] Figure 2 This is a schematic diagram of the architecture of the speech recognition system 200 provided in an embodiment of this application. Figure 2 As shown, the speech recognition system 200 may include an acoustic model 201, a language model 202, and a decoder 203.

[0062] The acoustic model 201 can receive audio input and then determine the probability distribution matrix corresponding to each pronunciation based on the audio features. The probability distribution matrix output by the acoustic model 201 can be a four-dimensional tensor, which can contain the acoustic probability scores of multiple candidate characters corresponding to the pronunciation. The acoustic probability scores are used to characterize the probability of the character corresponding to the pronunciation being recognized as each candidate character.

[0063] For example, the acoustic model 201 may include an encoder 204, a prediction network 205, and a fusion network 206. The encoder 204 encodes audio data to obtain an acoustic coded sequence (or acoustic coded features). The prediction network 205 receives the identified text and encodes it to obtain a text coded sequence (or text coded features). The prediction network 205 then predicts subsequent text based on the text coded sequence, generating a new text coded sequence. The fusion network 206 concatenates the text coded sequence and the acoustic coded sequence to form a new coded sequence, and then outputs a probability distribution matrix to the decoder 203 based on the concatenated coded sequence.

[0064] Figure 2 The acoustic model 201 shown can also be referred to as an acoustic model based on the Transducer architecture. For ease of description and better understanding, subsequent embodiments of this application will use an acoustic model based on the Transducer architecture to introduce the speech recognition method, but it is not limited to an acoustic model based on the Transducer architecture; acoustic model 201 can also employ other architectures. Furthermore, acoustic model 201 can include more than... Figure 2 The embodiments of this application do not limit the architecture of the acoustic model 201, which may include more or fewer modules, or combine certain modules, or add or remove certain modules.

[0065] The language model 202 can receive text input and predict subsequent characters based on the semantic features of the recognized text. For example, the language model 202 can determine the character corresponding to the fifth pronunciation in the current sentence based on the characters corresponding to the first four recognized pronunciations. The more characters already recognized in the current sentence, the more text the language model 202 can handle during prediction, and the stronger its semantic correction function. The language model 202 can output a probability distribution matrix to the decoder 203. The probability distribution matrix output by the language model 202 can contain language probability scores for multiple candidate characters, which represent the probability of the character corresponding to the pronunciation to be recognized as each candidate character. The language model 202 can be trained based on text from the corresponding domain when recognizing speech content from different domains. For example, when an electronic device uses the speech recognition system 200 to recognize everyday speech, the language model 202 can be trained based on common everyday phrases; when an electronic device uses the speech recognition system 200 to recognize speech from the scientific and technological field, the language model 202 can be trained based on the professional terminology of the corresponding scientific and technological field. In this way, the speech recognition system 200 in the electronic device can achieve good speech recognition results in different application scenarios.

[0066] The language model 202 can be a neural network model based on convolutional neural networks (CNN), recurrent neural networks (RNN), long short-term memory (LSTM), deep neural networks (DNN), generative pre-training transformers (GPT), and so on.

[0067] Decoder 203 is used to fuse the prediction results of acoustic model 201 and language model 202 to generate the recognized speech content text. The method of decoder 203 fusing acoustic model 201 and language model 202 can be referred to the description in the following embodiments, and will not be elaborated here.

[0068] After recording an audio stream via microphone 170C, electronic device 100 can segment the audio stream to obtain valid audio segments. The aforementioned valid audio segments can refer to audio segments containing human voices. The electronic device can segment the audio stream to obtain valid audio segments when there are long pauses in the audio stream. The audio stream is continuous audio data transmitted using streaming technology. The electronic device 100 can filter out valid audio from the audio stream by calculating the short-time energy value or short-time zero-crossing rate of audio segments. For example, the electronic device 100 can calculate the short-time energy value within each time window of the audio stream using a sliding time window. When the short-time energy value within a certain time window of the audio stream is lower than a preset threshold, the electronic device can select the start time point of that time window to segment the audio stream. Not limited to this method, the electronic device 100 can also segment valid audio segments using other methods, which are not limited in this embodiment. The electronic device can segment valid audio segments according to a preset duration to obtain multiple frames of audio. The aforementioned preset duration can be 500 milliseconds, meaning that the duration of each frame of audio is 500 milliseconds. Then, the electronic device 100 can input the audio frame by frame into the speech recognition system to recognize the speech content.

[0069] Exemplarily, after the electronic device inputs the first-frame audio into the acoustic model 201, the encoder 204 in the acoustic model 201 generates an acoustic coding sequence corresponding to the first-frame audio. When the electronic device is recognizing the first-frame audio and there are no recognized characters, at this time, the fusion network 206 mainly determines the characters corresponding to the pronunciation included in the first frame based on the acoustic coding sequence output by the encoder 204. The fusion network 206 can input the probability distribution matrix corresponding to the first-frame audio into the decoder 203. Similarly, because there are no recognized characters at present, the decoder 203 can determine one or more candidate characters with higher acoustic probability scores based on the probability distribution matrix output by the fusion network 206, and input the one or more candidate characters into the language model 202 and the prediction network model 205. For example, if the first-frame audio contains the pronunciation "wo", after receiving the first-frame audio, the acoustic model 201 can input the probability distribution matrix into the decoder 203, and the decoder 203 can determine multiple candidate characters with higher acoustic probability scores from the probability distribution matrix, such as "我", "窝", "喔". Since there are no recognized characters at present, the language model 202 cannot give an accurate prediction based on the context. Therefore, the decoder 203 can use the acoustic probability score of each candidate character output by the acoustic model 201 as the total score of the candidate character. Finally, the decoder 203 can input the several candidate characters with higher total scores determined into the language model 202 and the prediction network 205. Optionally, in some scenarios, the electronic device can display the candidate character with the highest total score on the screen. For example, in the voice note function, when the total score of "我" is the highest among "我", "窝", "喔", the electronic device can first display the character "我".

[0070] Then, the electronic device can input the second-frame audio into the acoustic model 201, and the encoder 204 can generate an acoustic coding sequence corresponding to the second-frame audio. The prediction network 205 can predict the candidate characters of the second frame based on the previous context (at this time, the previous context only includes the candidate characters of the first frame). For example, when the candidate characters of the first-frame audio input into the prediction network 205 by the decoder 203 include "我 (wǒ)", "窝 (wō)", and "喔 (wō)", the prediction network can predict the characters of the second frame based on each candidate character respectively, and then generate a text coding sequence, which is used to strengthen some characters in the acoustic coding sequence based on the previous semantic context. The second-frame audio may include the pronunciation "shi". Taking the candidate character "我 (wǒ)" of the first frame as an example, the prediction network 205 can determine that the character "是 (shì)" can follow "我 (wǒ)", and then the prediction network 205 can strengthen "是 (shì)" through the text coding sequence. Although "shi" can be recognized as "是 (shì)", "试 (shì)", or "四 (sì)", the acoustic probability score of "是 (shì)" in the subsequent probability distribution matrix will be increased due to the strengthening of the text coding sequence. Finally, the fusion network 206 will splice the text coding sequence and the acoustic coding sequence to obtain the probability distribution matrix corresponding to the acoustic model. After receiving the language text "我 (wǒ)" corresponding to the first-frame audio, the language model 202 will also predict the characters of the second frame to obtain the probability distribution matrix corresponding to the language model. Finally, the decoder 203 can fuse the probability distribution matrix corresponding to the acoustic model and the probability distribution matrix corresponding to the language model to determine one or more candidate characters of the second frame. The method for the decoder 203 to fuse the probability distribution matrix corresponding to the acoustic model and the probability distribution matrix corresponding to the language model can refer to the introduction in the subsequent embodiments and will not be elaborated here.

[0071] The method for the speech recognition system to recognize the speech content in the subsequent-frame audio can refer to the method for recognizing the speech content in the first and second frames of audio described above, which will not be repeated here.

[0072] It should be noted that the speech recognition system 200 determines multiple candidate characters for each frame of audio, and then recognizes subsequent characters based on each candidate character respectively. In this way, the speech recognition system 200 will obtain multiple sets of speech content texts. Exemplarily, when the speech recognition system 200 determines two candidate characters for each frame, after recognizing the above first frame of audio and the second frame of audio, it can obtain four sets of speech texts. For example, the speech recognition system 200 determines two candidate characters, "我 (I)" and "窝 (wo)", based on the first frame of audio, and then determines "是 (am)" and "试 (try)" based on "我 (I)", and determines "是 (am)" and "市 (city)" based on "窝 (wo)". In this way, four sets of speech texts are obtained: "我是 (I am)", "我试 (I try)", "窝是 (wo am)", "窝市 (wo city)". Furthermore, the speech recognition system will determine the candidate characters corresponding to the third frame of audio based on the above four sets of speech texts. Finally, after recognizing a valid audio segment, the speech recognition system can select the set of speech texts with the highest total score as the final recognition result of this valid audio segment.

[0073] The following combines Figure 2 the speech recognition system 200 shown to specifically introduce the speech recognition method provided by the embodiments of the present application.

[0074] Figure 3 is a flowchart of the speech recognition method provided by the embodiments of the present application. Since this speech recognition method includes two screenings of candidate characters, this speech recognition method can also be called a speech recognition method based on double-topk.

[0075] As Figure 3 shown, this method may include but is not limited to the following steps:

[0076] S301. The electronic device receives an audio segment A, and the audio segment A includes the Kth frame of audio and the N frames of audio before the Kth frame of audio.

[0077] Among them, the audio segment A may be a frame of audio in a valid audio segment, and the audio segment A may include pronunciation A. In Chinese recognition, each pronunciation may correspond to a Chinese character. In English recognition, a pronunciation may correspond to a phoneme. In the embodiments of the present application, the audio segment A may also be referred to as the first audio segment.

[0078] The electronic device may determine the speech content corresponding to a valid audio segment as a sentence. Among them, a valid audio segment is an audio with a pause duration less than a preset duration within a speech content. The electronic device can split a valid audio segment from the audio stream through the short-time energy value or short-time zero-crossing rate of the audio within a sliding window, etc.

[0079] S302. The electronic device determines multiple character sequences based on the character sequence A corresponding to the identified N frames of audio and the prediction result A of the character corresponding to the K-th frame of audio. Any one of the multiple character sequences contains a candidate character corresponding to the character sequence A and the K-th frame of audio, where the prediction result A is determined based on the K-th frame of audio and the character sequence A.

[0080] The electronic device may input the audio segment A into the acoustic model 201, and the acoustic model 201 may determine the prediction result A based on the K-th frame of audio and the character sequence A. The character sequence A may include the characters corresponding to the N frames of audio that the electronic device has identified before the K-th frame of audio. The prediction result A may include a probability distribution matrix of the character corresponding to the K-th frame of audio by the acoustic model 201, and the probability distribution matrix may include the acoustic probability score of the character corresponding to the K-th frame of audio being a certain candidate character under a character sequence. The character sequence may be a sequence containing one or more characters, and each character sequence corresponds to an identification result of the K-th frame of audio and the N frames of audio before the K-th frame of audio. In the embodiments of the present application, the prediction result A may also be referred to as the first prediction result, and the character sequence A may also be referred to as the first character sequence.

[0081] In the embodiments of the present application, since the speech recognition system 200 determines the candidate character corresponding to the current frame of audio based on the character sequence identified before a frame of audio during recognition, the probability of the current frame candidate character is related to the identified character sequence. Also, because the speech recognition system 200 can identify the candidate character corresponding to the current frame of audio based on multiple character sequences respectively, for the sake of easy distinction, the acoustic probability score of the candidate character corresponding to this frame of audio under different character sequences can be referred to as the acoustic probability score of the entire character sequence. For example, "我是", "我试", "窝是", "窝市" are four character sequences, and the acoustic probability score of the character sequence "我是" is the acoustic probability score that the character corresponding to the next frame of audio is "是" when "我" has been identified, which is used to indicate the probability that the character corresponding to the next frame of audio is "是" when "我" has been identified.

[0082] In some embodiments, there may be multiple character results A. This is because the recognition result of the decoder 203 for each frame of audio may include multiple, so that after the decoder 203 recognizes each frame of audio, multiple character sequences are input into the language model 202 and the prediction network 205. The acoustic model 201 determines the candidate character corresponding to the K-th frame of audio based on the character sequence A corresponding to the N frames of audio identified in the audio segment, and finally the acoustic model 201 can determine multiple character sequences, and each character sequence is composed of the character sequence A and a candidate character corresponding to the K-th frame of audio.

[0083] S303. The electronic device selects M1 character sequences from multiple character sequences. The acoustic probability score of the M1 character sequences is the highest among the multiple character sequences. The acoustic probability score of a character sequence represents the probability that the speech content of the Kth frame audio and the previous N frames audio is this character sequence.

[0084] The electronic device can select the M1 character sequences with the highest acoustic probability scores from multiple character sequences based on the prediction result A. Specifically, the electronic device can sort the acoustic probability scores of each character sequence from high to low, and then determine the top M1 character sequences with the highest acoustic probability scores. Since this step selects the top few character sequences with the highest acoustic probability scores, it can also be called the first topk step of this speech recognition method. M1 can be a relatively small number to reduce the computational load; for example, M1 could be 5.

[0085] S304. The electronic device selects M2 character sequences from multiple character sequences based on M1 character sequences. The pronunciation of the candidate character in the Kth frame audio of the M2 character sequences is the same as or similar to the pronunciation of the candidate character in the Kth frame audio of any character sequence in the M1 character sequences.

[0086] In step S303, only the top M1 character sequences with high acoustic probability scores are determined. This may result in correct character sequences having low acoustic probability scores and not being included in the M1 character sequences, thus excluding correct character sequences. In step S304, the electronic device can expand the candidate character sequences based on the pronunciation of the characters in the M1 character sequences.

[0087] The electronic device can determine one or more pronunciations of the Kth frame audio in the aforementioned M1 character sequences, and then filter from multiple character sequences (i.e., the total character sequences) a character sequence whose candidate character corresponding to the Kth frame audio is the same as or similar to the aforementioned one or more pronunciations. Finally, M2 character sequences are selected from the multiple character sequences. The M2 character sequences include both the aforementioned M1 character sequences and one or more newly added character sequences. The pronunciation of the candidate character corresponding to the Kth frame audio in the newly added character sequences is similar to one or more pronunciations corresponding to the Kth frame audio in the aforementioned M1 character sequences.

[0088] The term "similar pronunciation" can include any of the following: two pronunciations are nasal consonants (e.g., "yin" and "ying"); two pronunciations have the same vowel (e.g., "qing" and "xing"); or two pronunciations have different tones (e.g., the pronunciations of "qing" and "qing"). In some embodiments, two pronunciations can also be considered similar if they simultaneously include multiple of the above conditions. For example, "qing" and "xing" have the same vowel but different tones, so their pronunciations can be considered similar. Similarly, "yin" and "ying" are nasal consonants (e.g., "yin" and "ying" have different tones, so their pronunciations can also be considered similar.

[0089] The pronunciation of the character corresponding to the Kth frame audio in the expanded character sequence is more diverse. This avoids the situation where the correct character sequence has a low acoustic probability score due to non-standard pronunciation in the audio, such as incorrect nasal sounds or incorrect pitch, which would lead to the correct result being excluded in step S303.

[0090] S305. The electronic device selects M3 character sequences from M2 character sequences, where the pronunciation of the character corresponding to the Kth frame audio in the M3 character sequences appears most frequently.

[0091] After the expansion in step S304, the number of character sequences is increased, meaning there are now a large number of recognition results for the Kth frame audio and the N frames preceding it. Since the prediction network 205 in the speech recognition system 200 enhances the acoustic probability scores of some candidate characters based on the semantics of the N frames preceding the Kth frame audio when determining the character corresponding to it, this may increase the acoustic probability scores of character sequences containing candidate characters whose pronunciations are not close to the actual pronunciations in the Kth frame audio. To exclude these candidate character sequences, the electronic device can determine the frequency of the pronunciation of the character corresponding to the Kth frame audio in the M3 character sequences, and then determine the character corresponding to one or more pronunciations with the highest frequency. Then, the electronic device can filter out the M3 character sequences with the highest pronunciation frequency of the character corresponding to the Kth frame audio from the M2 character sequences.

[0092] The electronic device may not be limited to the M3 character sequences with the highest pronunciation frequency corresponding to the character in the Kth frame audio. In some embodiments, the electronic device may also select character sequences from the M2 character sequences whose pronunciation frequency exceeds a certain preset threshold.

[0093] For example, if an electronic device determines that the third tone "ni" appears most frequently in the audio corresponding to the Kth frame of an M2 character sequence, it can filter out all character sequences from the M2 character sequences whose corresponding characters in the audio corresponding to the Kth frame are pronounced as the third tone "ni".

[0094] S306. The electronic device selects M4 character sequences from M3 character sequences, and the acoustic probability score of the M4 character sequences is the highest among the M3 character sequences.

[0095] The electronic device can sort the acoustic probability scores of M3 character sequences from high to low, and then determine the top M4 character sequences with the highest acoustic probability scores. This can be called the second topk step of the speech recognition method. M4 can be relatively large compared to M1. For example, when M1 is 5, M4 can be 10. This can improve the diversity of candidate character sequences without increasing the amount of computation too much.

[0096] S307. The electronic device selects a character sequence from M4 character sequences based on the prediction result B of the character corresponding to the Kth frame audio, and determines this character sequence as the speech recognition result of the Kth frame audio and the audio of the N frames preceding the Kth frame audio, wherein the prediction result B is determined based on the character sequence A.

[0097] Both acoustic model 201 and language model 202 output probability distribution matrices. Prediction result A contains the probability distribution matrix output by acoustic model 201, and prediction result B contains the probability distribution matrix output by language model 202. Prediction result B is determined based on the semantic features of the recognized character sequence A. In this embodiment, prediction result B can also be referred to as the second prediction result.

[0098] After determining the M4 character sequences, electronic device 100 can look up the language probability scores of the M4 character sequences in the language model. Here, the language probability scores are denoted as Score. lm The acoustic probability score is denoted as Score. am The total score for each character sequence is Score. final =Score am +α·Score lm α is the weight of the language probability score. The value of α can gradually increase as the number of recognized character sequences (i.e., the value of N) increases; that is, the weight α of the language probability score is proportional to the length of the recognized character sequence. For example, when the speech recognition system 200 recognizes the fifth character in the same character sequence, the weight α can be 0.2, and when it recognizes the sixth character, the weight α can be 0.22. In this embodiment, the weight α can also be called the first weight.

[0099] When calculating the total score, the weight α of the electronic device increases as the length of the recognized text sequence increases. In this way, when there are fewer recognized characters, the context (i.e., the recognized characters) that the language model 202 can refer to is less, so the accuracy may also be lower. At this time, the total score of the candidate characters determined by the speech recognition system 200 depends more on the acoustic probability score of the candidate character. When the recognized characters gradually increase, since the language model 202 can make predictions based on more context (i.e., the recognized characters), the accuracy of its prediction will relatively increase. At this time, increasing the weight α by the speech recognition system can play a semantic correction function for the prediction result of the acoustic model 201, and improve the accuracy of the speech recognition system 200 in recognizing speech content. Taking the example of the electronic device recognizing the audio corresponding to the sentence "In ancient times, what age does the term 'huajia' refer to". When the electronic device always uses the same weight to recognize the audio, it may always be mainly affected by the prediction result of the acoustic model, which may lead to its recognition of "is referred to as" as "market value", "is worth", "set", etc. When the electronic device adjusts and increases the weight of the language probability score as the recognized character sequence grows, the language probability score of the language model can play a semantic correction role for the prediction result of the acoustic model. It can be seen that when the electronic device dynamically adjusts the weight of the language probability score, the correct result "In ancient times, what age does the term 'huajia' refer to" will have a higher total score.

[0100] When N is 0, that is, there are no recognized characters in the current sentence. At this time, the weight α can be 0. That is to say, when the speech recognition system 200 recognizes the first character in a sentence, the weight α can be 0. At this time, the total score of each character only depends on its acoustic probability score. Taking the example of the electronic device recognizing the audio corresponding to the sentence "Fire has no shadow". When the weight of the language model is non-zero when the electronic device recognizes the first character, the language model may determine that the probability of "I" as the first character in a sentence is relatively large from a semantic perspective, which may lead to the sequence with the highest total score in the final recognition result being "I have no shadow", making the recognition result of the first character deviate from its original pronunciation "huo". When the weight of the language model is 0 when the electronic device recognizes the first character, the speech recognition system determines the total score of each character only based on the recognition result of the acoustic model, and the total score of "Fire has no shadow" in the final recognition result will be higher. It can be seen that ignoring the language probability score of the first character in each sentence when the electronic device calculates the total score can improve the accuracy of the first character, and avoid the language model recognizing the first character only based on the statistical probability in the training data, which affects the final recognition result.

[0101] Finally, the electronic device can use the character sequence with the highest total score as the speech recognition result of the Kth frame audio and the N frames of audio before the Kth frame.

[0102] In some embodiments, the electronic device may input multiple character sequences with higher total scores into the language model 202 and the prediction network 205, so that when the speech recognition system 200 recognizes the character corresponding to the (K + 1)-th frame of audio, it can predict based on the first N + 1 audios of the (K + 1)-th frame of audio, and then obtain multiple character sequences.

[0103] When the speech recognition system 200 determines that a sentence ends, the speech recognition system 200 may select a character sequence with the highest total score as the final recognition result of the sentence. The above-mentioned sign that a sentence ends may be that the speech recognition system 200 has recognized all the frame audios in a valid audio segment (such as audio segment A).

[0104] The following uses a specific example to introduce the speech recognition method provided by the embodiments of the present application.

[0105] Figures 4-5 It is a specific example diagram when the electronic device provided by the embodiments of the present application recognizes speech.

[0106] Here, an example is given in which the speech recognition system 200 selects two characters for input into the language model 202 and the prediction network 205 when recognizing each frame of audio. This example introduces the method of the electronic device when recognizing partial speech content "chew gum" in a sentence. Here, the pronunciations to be recognized are: "jiao", "kou", "xiang", "tang". For the convenience of description and better understanding, Figure 4 、 Figure 5 only two characters corresponding to the K-th frame of audio are shown. It can be understood that there may be more frame audios before the K-th frame of this sentence, and each frame of audio may include two recognition results (characters). The multiple characters recognized from multiple frames of audio together form multiple character sequences.

[0107] As Figure 4 shown, the electronic device determines that the characters corresponding to the pronunciation in the (K - 1)-th frame of audio are "chew" and "pay". The character sequences where "chew" and "pay" are located may be the two character sequences with the highest total scores ranked in descending order when the electronic device recognizes the (K - 1)-th frame of audio. In this example, the character sequences where "chew" and "pay" are located can be called two character sequences in character sequence A.

[0108] After the electronic device 100 obtains the K-th frame of audio from the valid audio segment, it can input the K-th frame of audio into the acoustic model 201. Subsequently, the acoustic model 201 will output a probability distribution matrix corresponding to the K-th frame of audio, and the probability distribution matrix contains the probabilities that the pronunciation corresponding characters in the K-th frame of audio are each character in the total character set. Among them, the acoustic model can determine the probabilities that the corresponding characters of the audio are each character in the total character set every time it recognizes audio. The electronic device has input two character sequences in total: the character sequences where the corresponding characters of the K-th frame of audio are "嚼" and "缴". The acoustic model 201 will obtain two probability distribution matrices for "嚼" and "缴" respectively. The probability distribution matrix can indicate the probability that the K-th frame of audio and the first N frames of the K-th frame of audio are a certain character sequence. Among them, when the pronunciation corresponding character in the (K - 1)-th frame of audio is "嚼", the candidate characters with the highest to lowest acoustic probability scores in the K-th frame of audio are "口", "抠", "寇", "扣", etc., and it can also be said that the acoustic probability scores of the character sequences from the highest to the lowest are "嚼口", "嚼抠", "嚼寇", "嚼扣", etc. When the pronunciation corresponding character in the (K - 1)-th frame of audio is "缴", the candidate characters with the highest to lowest acoustic probability scores in the K-th frame can be "扣", "获", "口", "叩", etc., and it can also be said that the acoustic probability scores of the character sequences from the highest to the lowest are "缴扣", "缴获", "缴口", "缴叩". Subsequently, the electronic device can execute step S303 to determine M1 characters based on the probability distribution matrix corresponding to the pronunciation in the (K - 1)-th frame of audio. Figure 4 In the illustrated embodiment, M1 can be taken as 2. Since the acoustic probability scores of the character sequences "嚼口" and "嚼抠" are relatively high, the electronic device 100 can use "口" and "抠" as the candidate characters for the K-th frame of audio when the (K - 1)-th frame of audio is recognized as "嚼". Similarly, the electronic device can use "扣" and "获" as the candidate characters for the K-th frame of audio when the (K - 1)-th frame of audio is recognized as "缴".

[0109] Then the electronic device can execute step S304 to expand the candidate character set based on the pronunciations of the above candidate characters, that is to say, the electronic device uses the characters with the same or similar pronunciations as the selected candidate characters as the candidate characters for the K-th frame of audio. For example, the electronic device can also use the characters with the same or similar pronunciations as the candidate characters excluded in step S303 as candidate characters. As Figure 4 shown, under the branch (hereinafter referred to as branch 1) where the pronunciation corresponding character in the (K - 1)-th frame of audio is "嚼", the electronic device can determine the characters such as "寇" and "扣" with pronunciations similar to "口" and "抠" as the candidate characters corresponding to the K-th frame of audio under this branch. Similarly, the electronic device can also expand branch 2. For example, it can also use "口" and "叩" with pronunciations similar to "扣" as candidate characters. There are fewer characters with the same pronunciation as "获" among the candidate characters, so "获" is excluded from the candidate characters.

[0110] Next, the electronic device can execute step S305 to screen one or more characters corresponding to the pronunciations with the highest occurrence frequencies in each branch. In branch 1, the characters corresponding to three pronunciations are retained, namely, "抠" corresponding to the first tone "kou", "口" corresponding to the third tone "kou", and "寇" and "扣" corresponding to the fourth tone "kou". In branch 2, the pronunciation corresponding to "获" has a relatively low occurrence frequency, so "获" is excluded, and only "扣", "口", and "叩" are left as candidate characters.

[0111] Finally, the electronic device can screen the M4 characters with the highest acoustic probability scores from the candidate characters. In this way, under branch 1, the electronic device can determine the three characters with the highest acoustic probability scores, namely, "口", "抠", and "寇", from the candidate characters such as "口", "抠", "寇", and "扣". Under branch 2, the electronic device can determine the three characters with the highest acoustic probability scores under this branch: "扣", "口", and "叩".

[0112] As Figure 5 shown, after the decoder determines the candidate characters in branch 1 and branch 2, it can determine the total scores of each character sequence based on the candidate characters in branch 1 and branch 2 respectively. Taking branch 1 as an example, after the decoder 203 determines that the candidate characters are "口", "抠", and "寇", it will look up the language probability scores of "口", "抠", and "寇" in the language model 202. The language probability scores of "口", "抠", and "寇" in the language model 202 are determined by the language model 202 based on the prediction result "嚼" of the Kth frame. Further, the decoder 203 can calculate the total score of each character sequence. Here, taking the weight α as 0.2 as an example, the decoder 203 can calculate the total score of the character sequence "嚼口" = acoustic probability score (0.15) + weight (0.2) × language probability score (0.1) = 0.17. Similarly, the decoder 203 can calculate the total scores of the character sequences "嚼抠", "嚼寇" in turn, and the total scores of the character sequences "缴扣", "缴口", and "缴叩" under branch 2. Finally, the decoder 203 can compare the total scores of all character sequences to determine the recognition results of the (K - 1)th frame and the Kth frame of audio.

[0113] In some embodiments, the electronic device needs to display the recognized speech content while performing speech recognition. Since the total score of the character sequence "嚼口" exceeds the other 5 total scores, the electronic device can use "嚼口" as the current recognition result and display it on the display screen. It should be noted that the speech recognition content displayed by the electronic device here does not represent the final recognition result of this sentence of speech. The electronic device will continue to recognize the subsequent frames and select the character sequence with the highest total score in the subsequent frames as the new recognition result.

[0114] In this example, when the speech recognition system 200 recognizes each frame of audio, it selects two characters and inputs them into the language model 202 and the prediction network 205. The decoder 203 determines the two character sequences with the highest total scores, including the "bit" sequence with a total score of 0.17 in branch 1 and the "withhold" sequence with a total score of 0.158 in branch 2. Then, these two character sequences are input into the language model 202 and the prediction network 205 to predict the characters corresponding to the (K + 1)-th frame of audio. That is to say, after calculating the total scores of all character sequences, the decoder 203 determines the final recognition result based on the total scores of all branch character sequences.

[0115] When the speech recognition system recognizes the (K + 1)-th frame of audio, the prediction network 205 in the acoustic model 201 will increase the acoustic probability scores of some candidate characters based on the recognized character sequences (including the "bit" character sequence and the "withhold" character sequence). The language model 202 will also determine the language probability scores of the characters corresponding to the (K + 1)-th frame of audio based on the recognized character sequences. Finally, the decoder 203 can determine the character sequence containing the characters corresponding to the (K + 1)-th frame of audio based on the prediction results of the acoustic model 201 and the language model 202. The method by which the decoder 203 determines the (K + 1)-th frame of audio based on the prediction results of the acoustic model 201 and the language model 202 can refer to the description of the above embodiments and will not be elaborated here.

[0116] As described above, the above embodiments are only used to illustrate the technical solutions of the present application and are not intended to limit them; although the present application has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that they can still modify the technical solutions described in the foregoing embodiments, or perform equivalent replacements on some of the technical features; and these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present application.

[0117] As used in the above embodiments, depending on the context, the term "when..." can be interpreted to mean "if...", or "after...", or "in response to determining...", or "in response to detecting...". Similarly, depending on the context, the phrase "when determining..." or "if detecting (the stated condition or event)" can be interpreted to mean "if determining...", or "in response to determining...", or "when detecting (the stated condition or event)", or "in response to detecting (the stated condition or event)".

[0118] In the above embodiments, implementation can be achieved entirely or partially through software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented entirely or partially in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid-state drive), etc.

[0119] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. This program can be stored in a computer-readable storage medium, and when executed, it can include the processes described in the above method embodiments. The aforementioned storage medium includes various media capable of storing program code, such as ROM or random access memory (RAM), magnetic disks, or optical disks.

Claims

1. A speech recognition method, characterized in that, The method is applied to an electronic device, and the method includes: The electronic device receives a first audio segment, which includes the Kth frame of audio and the N frames of audio preceding the Kth frame, where K is a positive integer and N is a positive integer. The electronic device determines multiple character sequences based on the first prediction result of the Kth frame audio and the first character sequence corresponding to the identified previous N frames audio. Any one of the multiple character sequences contains a candidate character corresponding to the Kth frame audio in the first character sequence. The first prediction result includes multiple candidate characters of the Kth frame audio. The electronic device selects L1 character sequences from the plurality of character sequences. The acoustic probability score of the L1 character sequences is the highest among the plurality of character sequences. The acoustic probability score of a character sequence represents the probability that the speech content of the Kth frame audio and the previous N frames audio is the character sequence. L1 is a positive integer. The electronic device selects L2 character sequences from the plurality of character sequences based on the L1 character sequences. The pronunciation of the candidate character of the Kth frame audio in the L2 character sequences satisfies a first condition with the pronunciation of the candidate character of the Kth frame audio in any of the L1 character sequences. L2 is a positive integer. The first condition includes any one of the following: the two pronunciations have the same vowel, the two pronunciations differ only in tone, and the two pronunciations are front and back nasal sounds. The electronic device selects L3 character sequences from the L2 character sequences. The acoustic probability scores of the L3 character sequences satisfy a second condition, where L3 is a positive integer. The second condition includes the highest acoustic probability score among the L2 character sequences. The electronic device selects a second character sequence from the L3 character sequences based on the second prediction result of the Kth frame audio, and determines the second character sequence as the speech recognition result of the Kth frame audio and the previous N frames audio. The second prediction result is obtained by the electronic device based on the semantic prediction of the speech content of the Kth frame audio.

2. The method according to claim 1, characterized in that, The electronic device selects the L3 character sequence from the L2 character sequences, specifically including: The electronic device selects L4 character sequences from the L2 character sequences, wherein the pronunciation of the character corresponding to the Kth frame audio appears most frequently in the L4 character sequences, and L4 is a positive integer; The electronic device selects the L3 character sequence from the L4 character sequences, where the L3 character sequence is the one with the highest acoustic probability score among the L4 character sequences.

3. The method according to claim 1 or 2, characterized in that, The first character sequence includes a third character sequence and a fourth character sequence. Any one of the multiple character sequences contains a candidate character corresponding to the third character sequence and the audio of the Kth frame, or contains a candidate character corresponding to the fourth character sequence and the audio of the Kth frame.

4. The method according to claim 1 or 2, characterized in that, The method further includes: The electronic device predicts multiple first candidate characters of the Kth frame audio based on the Kth frame audio and the first character sequence, and determines the acoustic probability score of the multiple first candidate characters. The first prediction result includes the acoustic probability score. The electronic device performs semantic prediction on the Kth frame audio based on the first character sequence to obtain multiple second candidate characters of the Kth frame audio, and determines the language probability score of the multiple second candidate characters. The language probability score includes the probability that the character corresponding to the Kth frame audio under the first character sequence is each of the second candidate characters. The second prediction result includes the language probability score.

5. The method according to claim 4, characterized in that, The electronic device selects a second character sequence from the L3 character sequences based on the second prediction result of the character corresponding to the audio of the Kth frame, including: The electronic device calculates the total score for each character sequence in the L3 character sequences, and the total score is equal to the product of the language probability score and the first weight plus the acoustic probability score; The electronic device selects the second character sequence based on the total score of the L3 character sequences.

6. The method according to claim 5, characterized in that, The first weight gradually increases as the value of N increases.

7. The method according to claim 5 or 6, characterized in that, When K is 1, the first weight is 0.

8. An electronic device, characterized in that, The electronic device includes: a display screen, a memory, and a processor coupled to the memory; the display screen is used to display a user interface, the memory stores a computer program, and the processor executes the computer program to cause the electronic device to perform the method as described in any one of claims 1 to 7.

9. A computer-readable storage medium storing computer instructions, characterized in that, When the computer instructions are executed on an electronic device, the electronic device causes the electronic device to perform the method as described in any one of claims 1 to 7.

Images