Speech recognition method and electronic device

Abstract

The present application provides a voice recognition method and an electronic device, which can improve the voice recognition accuracy of the electronic device and can be applied to scenarios such as voice assistants, voice input methods, dictation systems, smart homes, and in-vehicle navigation, and relates to the voice recognition technology in the field of artificial intelligence. The method includes: the terminal obtains N target phoneme sequences with the highest probabilities according to the probability matrix of the phonemes corresponding to the target voice signal frame and the probability matrix of the next possible phoneme of the first phoneme sequence. Then, translate the N target phoneme sequences to obtain at least one first text sequence corresponding to each of the N target phoneme sequences and the third probability of each first text sequence in the at least one first text sequence, and determine the first text sequence with the highest probability in the at least one first text sequence according to the third probability and the target probability corresponding to the N target phoneme sequences, and display the first text sequence with the highest probability.

Description

Technical Field

[0001] This application relates to the technical field of terminals, and in particular, to a voice recognition method and an electronic device. Background Art

[0002] With the improvement of the computing power of electronic devices, the current trend of voice recognition is to deploy a voice interaction system on an electronic device to improve the user experience of voice recognition and reduce costs. However, due to the limited storage capacity of electronic devices, the space usually used to store the voice interaction system is small, which limits the size of the space occupied by the voice interaction system.

[0003] A traditional voice interaction system usually consists of two parts: an acoustic model (such as a hidden Markov model (HMM), a Gaussian mixture model (GMM)) and a language model (such as an N-gram). The language model is trained based on a large amount of text data, so that the voice interaction system has high recognition accuracy. However, the storage space occupied by the language model is usually large. If you want to deploy a traditional voice interaction system on an electronic device, you need to compress the language model. However, using the compressed voice interaction system for voice recognition will greatly reduce the recognition accuracy.

[0004] An existing streaming end-to-end model can be deployed on an electronic device, and audio can be directly mapped into a text sequence, such as a recurrent neural network transducer (RNN-T). However, this model is only trained based on audio data, and the training samples used are limited, resulting in low recognition accuracy of the streaming end-to-end model for a large number of texts that have not been trained during the voice recognition process. Summary of the Invention

[0005] Embodiments of this application provide a voice recognition method and an electronic device, which can improve the voice recognition accuracy of the electronic device.

[0006] To achieve the above object, this application adopts the following technical solutions:

[0007] In a first aspect, a speech recognition method is provided. The speech recognition method includes: obtaining N target phoneme sequences with the highest target probabilities according to a first probability matrix and a second probability matrix, translating the N target phoneme sequences, obtaining third probability matrices respectively corresponding to the N target phoneme sequences, determining, according to the third probability matrices and the target probabilities corresponding to the N target phoneme sequences, a first text sequence with the highest probability among at least one first text sequences, and displaying the first text sequence with the highest probability. In one embodiment, several first text sequences with the highest probabilities (such as the top three) may also be determined for display.

[0008] Among them, the first probability matrix includes the probabilities of each phoneme in a plurality of phonemes included in a phoneme vocabulary corresponding to a phoneme of a target speech signal frame, the second probability matrix includes the probabilities of each phoneme in a plurality of phonemes included in a phoneme vocabulary corresponding to the next possible phoneme of a plurality of first phoneme sequences, each first phoneme sequence in the plurality of first phoneme sequences includes a starting phoneme and a phoneme sequence among a plurality of phoneme sequences corresponding to one or more speech signal frames before the target speech signal frame, the target probability is the probability of each target phoneme sequence, each target phoneme sequence is composed of a first phoneme sequence among the plurality of first phoneme sequences and a phoneme corresponding to the target speech signal frame, N is an integer greater than or equal to 1, and the target speech signal frame is a speech signal frame to be currently recognized. The third probability matrix includes the probabilities that a target phoneme sequence among the N target phoneme sequences is translated into at least one first text sequence.

[0009] Based on the speech recognition method described in the first aspect, the terminal obtains N target phoneme sequences with the highest target probabilities according to the probability matrix of the phoneme corresponding to the target speech signal frame and the probability matrix of the next possible phoneme of the first phoneme sequence. Then, the N target phoneme sequences are translated to obtain at least one first text sequence respectively corresponding to the N target phoneme sequences and the third probability of each first text sequence in the at least one first text sequence. According to the third probability and the target probabilities corresponding to the N target phoneme sequences, a first text sequence with the highest probability among the at least one first text sequences is determined, and the first text sequence with the highest probability is displayed, thereby improving the accuracy of speech recognition.

[0010] In a possible design, the speech recognition method described in the first aspect may further include: obtaining one or more of the following based on the target speech signal frame and one of the multiple first phoneme sequences: a first probability matrix, at least one second phoneme sequence, and a fourth probability matrix. Each second phoneme sequence in the at least one second phoneme sequence is composed of one of the multiple first phoneme sequences and a phoneme corresponding to the target speech signal frame. The fourth probability matrix includes the probabilities corresponding to each second phoneme sequence in the at least one second phoneme sequence, and the second phoneme sequence is used as the first phoneme sequence corresponding to the next speech signal frame of the target speech signal frame. In this way, the first probability matrix, the second phoneme sequence, and the fourth probability matrix corresponding to the phoneme of the target speech signal frame can be obtained to improve the accuracy of speech recognition.

[0011] In a possible design, the speech recognition method described in the first aspect may further include: obtaining a second probability matrix according to the semantics of the first phoneme sequence. The semantics of the first phoneme sequence may represent the language meaning expressed by the first phoneme sequence. In this way, the probability of the next possible phoneme of the first phoneme sequence can be obtained according to the language meaning of the first phoneme sequence to improve the accuracy of speech recognition.

[0012] In a possible design, the obtaining of the N target phoneme sequences with the largest target probabilities according to the first probability matrix and the second probability matrix may include: fusing the first probability matrix and the second probability matrix to obtain a fifth probability matrix, and obtaining the N target phoneme sequences with the largest target probabilities according to the fifth probability matrix corresponding to one or more speech signal frames before the target speech signal frame and the target speech signal frame respectively. The fifth probability matrix may include the fusion probabilities of each phoneme in the phoneme vocabulary corresponding to a phoneme of the target speech signal frame. That is to say, after fusing the first probability matrix and the second probability matrix, the accuracy of the probability matrix of the next possible phoneme obtained is higher, thereby improving the accuracy of speech recognition.

[0013] In a possible design, the speech recognition method described in the first aspect may further include: translating one or more phonemes in the phoneme vocabulary corresponding to the target speech signal frame to obtain text probability matrices corresponding to the one or more phonemes respectively. The text probability matrix may include the probabilities that the corresponding phonemes are translated into each text symbol in the text vocabulary. In this way, one or more phonemes corresponding to one or more speech signal frames included in the speech stream can be translated respectively to obtain text probability matrices corresponding to the respective phonemes.

[0014] In a possible design, the speech recognition method described in the first aspect may further include: obtaining a first grid structure. The abscissa of the first grid structure is each phoneme in the complete phoneme sequence, and the ordinate is the text probability matrix corresponding to each phoneme in the complete phoneme sequence. The complete phoneme sequence is the target phoneme sequence obtained when the last speech signal frame in the speech stream is used as the target speech signal frame. That is to say, the obtained first grid structure includes the text probability matrix corresponding to each phoneme in the complete phoneme sequence.

[0015] In a possible design, the speech recognition method described in the first aspect may further include: cropping the first grid structure to obtain a second grid structure, decoding the second grid structure, and obtaining and displaying a second text sequence. In this way, by cropping the first grid structure and adjusting the probabilities of the text symbols in the first grid structure, the correction of ambiguous sounds is realized, and a more accurate text sequence is obtained, thereby further improving the accuracy of speech recognition.

[0016] In a possible design, the above-mentioned cropping the first grid structure to obtain a second grid structure may include: obtaining a phoneme-text confusion matrix based on a confusion phoneme matrix and a phoneme-text matrix, and cropping the first grid structure based on the phoneme-text confusion matrix to obtain a second grid structure. The confusion phoneme matrix is used to represent the probability that each phoneme in the multiple phonemes included in the phoneme vocabulary is recognized as each phoneme in the multiple phonemes included in the phoneme vocabulary, and the phoneme-text matrix is used to represent the correspondence between each text symbol in the multiple text symbols included in the text vocabulary and each phoneme in the multiple phonemes included in the phoneme vocabulary. In this way, by cropping the first grid structure with the phoneme-text confusion matrix, the accuracy of the text probability corresponding to each phoneme in the obtained second grid structure is higher, thereby further improving the accuracy of speech recognition.

[0017] In a second aspect, an electronic device is provided. The electronic device includes: a processor, a memory, and a display screen. The memory and the display screen are coupled to the processor. The memory is used to store computer program code, and the computer program code includes computer instructions. When the processor reads the computer instructions from the memory, the electronic device executes the speech recognition method described in any one of the possible implementation manners in the first aspect.

[0018] The technical effects of the electronic device described in the second aspect can refer to the technical effects of the speech recognition method described in the first aspect, which will not be elaborated here.

[0019] In a third aspect, a computer-readable storage medium is provided. A computer program or instruction is stored on the computer-readable storage medium. When the computer program or instruction runs on a computer, the computer executes the speech recognition method described in any one of the possible implementation manners in the first aspect.

[0020] In a fourth aspect, a computer program product is provided, which includes a computer program or instructions that, when running on a computer, cause the computer to execute the speech recognition method described in any possible implementation manner of the first aspect.

[0021] In a fifth aspect, an embodiment of the present application provides a chip system, including at least one processor and at least one interface circuit. The at least one interface circuit is configured to perform transceiver functions and send instructions to the at least one processor. When the at least one processor executes the instructions, the at least one processor executes the speech recognition method described in the first aspect and any possible implementation manner thereof as described above. Description of the Drawings

[0022] Figure 1 Schematic structural diagram of the electronic device provided by the embodiment of the present application Figure 1 ;

[0023] Figure 2 Software structural block diagram of the electronic device provided by the embodiment of the present application;

[0024] Figure 3 Schematic structural diagram of the speech recognition module provided by the embodiment of the present application;

[0025] Figure 4 Schematic structural diagram of the acoustic model, phoneme predictor, and phoneme translator provided by the embodiment of the present application;

[0026] Figure 5 Schematic structural diagram of the rectifier provided by the embodiment of the present application;

[0027] Figure 6 Schematic flow diagram of the speech recognition method provided by the embodiment of the present application Figure 1 ;

[0028] Figure 7 Schematic application diagram of the speech recognition method provided by the embodiment of the present application Figure 1 ;

[0029] Figure 8 Schematic application diagram of the speech recognition method provided by the embodiment of the present application Figure 2 ;

[0030] Figure 9 Schematic application diagram of the speech recognition method provided by the embodiment of the present application Figure 3 ;

[0031] Figure 10 Schematic application diagram of the speech recognition method provided by the embodiment of the present application Figure 4 ;

[0032] Figure 11 Schematic application diagram of the speech recognition method provided by the embodiments of the present application Figure 5 ;

[0033] Figure 12 Schematic application diagram of the speech recognition method provided by the embodiments of the present application Figure 6 ;

[0034] Figure 13 Schematic application diagram of the speech recognition method provided by the embodiments of the present application Figure 7 ;

[0035] Figure 14 Schematic flow diagram of the speech recognition method provided by the embodiments of the present application Figure 2 ;

[0036] Figure 15 Schematic structural diagram of the first grid structure provided by the embodiments of the present application;

[0037] Figure 16 Schematic application diagram of the speech recognition method provided by the embodiments of the present application Figure 8 ;

[0038] Figure 17 Schematic application diagram of the speech recognition method provided by the embodiments of the present application Figure 9 ;

[0039] Figure 18 Schematic structure of the electronic device provided by the embodiments of the present application Figure 2 。 Detailed implementation manners

[0040] The speech recognition method and the electronic device provided by the embodiments of the present application will be described in detail below with reference to the accompanying drawings.

[0041] As used in the description of the present application, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusion. For example, a process, method, system, product or device that includes a series of steps or units is not limited to the listed steps or units, but optionally further includes other steps or units not listed, or optionally further includes other steps or units inherent to these processes, methods, products or devices.

[0042] It should be noted that in the embodiments of the present application, words such as "exemplary" or "for example" are used to indicate examples, illustrations or explanations. Any embodiment or design solution described as "exemplary" or "for example" in the embodiments of the present application should not be construed as being more preferred or having more advantages than other embodiments or design solutions. Rather, the use of words such as "exemplary" or "for example" is intended to present relevant concepts in a specific manner.

[0043] In the description of the present application, unless otherwise specified, the meaning of "a plurality of" refers to two or more. The "and / or" herein is merely an association relationship describing associated objects, indicating that three relationships may exist. For example, A and / or B may represent: A exists alone, both A and B exist simultaneously, and B exists alone.

[0044] In the embodiments of the present application, sometimes a subscript such as W1 may be miswritten as a non-subscript form such as W1. When the difference is not emphasized, their intended meanings are the same.

[0045] First, for the sake of easy understanding, the relevant terms and concepts that may be involved in the embodiments of the present application are introduced below.

[0046] (1) Phoneme

[0047] The phonemes in the embodiments of the present application may include, but are not limited to, one or more of the following: pinyin with tones, pinyin without tones, vowel phonemes, and consonant phonemes. Taking the speech stream as Chinese as an example, for example, the phoneme can be pinyin with tones, such as "tiān" or "tian1", where "1" can be used to represent the first tone. Again, for example, the phoneme can be pinyin without tones, such as "tian" composed of vowel phonemes and consonant phonemes. It should be noted that for the sake of easy description, the embodiments of the present application use the numbers after the pinyin to represent tones. For example, "1" can be used to represent the first tone, "2" is used to represent the second tone, "3" is used to represent the third tone, and "4" is used to represent the fourth tone. Taking the speech stream as a language other than Chinese as an example, the phonemes in the embodiments of the present application may include vowel phonemes and / or consonant phonemes. For example, the phonemes corresponding to the English word "good" can be "gu" and "d". The embodiments of the present application do not limit the specific implementation manner of phonemes.

[0048] (2) Word vector

[0049] A word vector can also be referred to as a word embedding, which is used to map the text in the text vocabulary into a real number vector.

[0050] (3) Mel-frequency cepstral coefficients

[0051] Mel-scale frequency cepstral coefficients (MFCC) are cepstral parameters extracted in the Mel-scale frequency domain. The Mel scale is used to describe the non-linear characteristics of the human ear's frequency. MFCC can include pre-emphasis, framing, windowing, fast Fourier transform (FFT), Mel filter bank, discrete cosine transform, etc. MFCC is used to extract acoustic features from a segment of speech. Since some information in the speech signal is irrelevant to speech recognition and can make speech recognition more cumbersome, extracting acoustic features from the speech signal can describe the speech signal through a given number of signal components and extract signals helpful for speech recognition.

[0052] Embodiments of this application provide a speech recognition method and an electronic device, which can be applied to scenarios such as voice assistants, voice input methods, dictation systems, smart homes, and in-vehicle navigation, and can improve the accuracy of speech recognition. The speech recognition method provided by the embodiments of this application can be applied in the electronic device 100 or in a system including the electronic device 100.

[0053] Optionally, the electronic device 100 can specifically be a mobile phone, a tablet computer, a vehicle-mounted device, an augmented reality (AR) / virtual reality (VR) device, a laptop computer, an ultra-mobile personal computer (UMPC), a netbook, a personal digital assistant (PDA), an artificial intelligence device, a wearable device, etc., which are terminal devices with speech recognition functions. The wearable device can be a smart watch, a smart bracelet, wireless earphones, smart glasses, a smart helmet, a blood glucose meter, a blood pressure monitor, etc. Embodiments of this application do not impose any restrictions on the specific type of the electronic device 100.

[0054] Exemplarily, Figure 1A schematic structural diagram of an electronic device 100 is shown. The electronic device 100 may include a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, a charging management module 140, a power management module 141, a battery 142, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a sensor module 190, a button 190, a motor 191, an indicator 192, a camera 193, a display screen 194, and a subscriber identification module (SIM) card interface 195, etc.

[0055] The processor 110 may include one or more processing units. For example, the processor 110 may include an application processor (AP), a modem processor, a graphics processing unit (GPU), an image signal processor (ISP), a controller, a memory, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural-network processing unit (NPU), etc. Among them, different processing units may be independent devices or integrated in one or more processors.

[0056] Among them, the controller may be the nerve center and command center of the electronic device 100. The controller may generate operation control signals according to the instruction operation code and timing signal to complete the control of fetching and executing instructions.

[0057] A memory may also be provided in the processor 110 for storing instructions and data. In some embodiments, the memory in the processor 110 is a cache memory. This memory may save the instructions or data that the processor 110 has just used or recycled. If the processor 110 needs to use the instruction or data again, it can be directly called from the memory. This avoids repeated accesses, reduces the waiting time of the processor 110, and thus improves the efficiency of the system.

[0058] In some embodiments of the present application, the electronic device 100 may use the processor 110 to process the voice stream to obtain a text sequence.

[0059] The charging management module 140 is used to receive a charging input from a charger. Among them, the charger may be a wireless charger or a wired charger.

[0060] The power management module 141 is used to connect the battery 142, the charging management module 140, and the processor 110. The power management module 141 receives inputs from the battery 142 and / or the charging management module 140 and supplies power to the processor 110, the internal memory 121, the external memory, the display screen 194, the camera 193, the wireless communication module 160, etc.

[0061] The wireless communication function of the electronic device 100 can be implemented by the antenna 1, the antenna 2, the mobile communication module 150, the wireless communication module 160, the modulation and demodulation processor, the baseband processor, etc.

[0062] The mobile communication module 150 can provide solutions for wireless communications such as 2G / 3G / 4G / 5G applied to the electronic device 100. The mobile communication module 150 can include at least one filter, switch, power amplifier, low noise amplifier (LNA), etc.

[0063] The wireless communication module 160 can provide solutions for wireless communications applied to the electronic device 100, including wireless local area networks (WLAN) (such as wireless fidelity (Wi-Fi) networks), Bluetooth (BT), global navigation satellite system (GNSS), frequency modulation (FM), near field communication (NFC), infrared technology (IR), etc.

[0064] The electronic device 100 realizes the display function through the GPU, the display screen 194, and the application processor, etc. The GPU is a microprocessor for image processing, connected to the display screen 194 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. The processor 110 may include one or more GPUs, which execute program instructions to generate or change display information.

[0065] The display screen 194 is used to display images, videos, etc. In some embodiments, the electronic device 100 may include 1 or N display screens 194, where N is a positive integer greater than 1. In some embodiments of the present application, the display screen 194 can be used to display text symbols, text sequences.

[0066] The electronic device 100 can realize the shooting function through the ISP, the camera 193, the video codec, the GPU, the display screen 194, and the application processor, etc.

[0067] The camera 193 is used to capture still images or videos.

[0068] The external memory interface 120 can be used to connect an external memory card, such as a Micro SD card, to implement the storage capacity expansion of the electronic device 100.

[0069] The internal memory 121 can be used to store computer-executable program codes, and the executable program codes include instructions. The internal memory 121 can include a program storage area and a data storage area. The internal memory 121 can include a high-speed random access memory, and can also include a non-volatile memory, such as at least one disk storage device, a flash memory device, a universal flash storage (UFS), etc. The processor 110 executes various functional applications and data processing of the electronic device 100 by running the instructions stored in the internal memory 121, and / or the instructions stored in the memory provided in the processor. In some embodiments of the present application, the internal memory 121 can be used to store an acoustic model, a phoneme predictor, a phoneme translator, etc. Optionally, the internal memory 121 can also be used to store a rectifier.

[0070] The audio module 170 includes a speaker, a receiver, a microphone, a headphone interface, etc.

[0071] The audio module 170 is used to convert digital audio data into an analog audio electrical signal for output, and is also used to convert the input analog audio electrical signal into digital audio data. The audio module 170 can include an analog-to-digital converter and a digital-to-analog converter. In some embodiments of the present application, the audio module 170 can be used to collect a voice stream, an audio signal, a voice signal, etc.

[0072] In some embodiments, the electronic device 100 can implement audio functions through the audio module 170 and an application processor, etc. Such as music playback, recording, etc.

[0073] The sensor module 190 can include a pressure sensor, a gyroscope sensor, a barometric pressure sensor, a magnetic sensor, an acceleration sensor, a distance sensor, a proximity light sensor, a fingerprint sensor, a temperature sensor, a touch sensor, an ambient light sensor, a bone conduction sensor, etc.

[0074] It can be understood that the structure schematically shown in the embodiments of the present application does not constitute a specific limitation on the electronic device 100. In other embodiments of the present application, the electronic device 100 can include more or fewer components than those shown in the figure, or combine certain components, or split certain components, or have different component arrangements. The components shown in the figure can be implemented in hardware, software, or a combination of software and hardware.

[0075] The software system of the electronic device 100 may adopt a layered architecture, an event-driven architecture, a microkernel architecture, a microservices architecture, or a cloud architecture. In the embodiments of the present invention, taking the Android system with a layered architecture as an example, the software structure of the electronic device 100 will be exemplarily described.

[0076] Figure 2 It is a software structure block diagram of the electronic device 100 according to the embodiments of the present invention.

[0077] The layered architecture divides the software into several layers, and each layer has a clear role and division of labor. The layers communicate with each other through software interfaces. In some embodiments, the Android system is divided into four layers, from top to bottom, namely the application layer, the application framework layer, the Android runtime and system libraries, and the kernel layer.

[0078] The application layer may include a series of application packages.

[0079] As Figure 2 shown, the application packages may include applications such as a camera, a calendar, a map, WLAN, music, short messages, a gallery, a call, navigation, and a first application.

[0080] Among them, the voice recognition in the embodiments of the present application may be implemented as a first application, and the first application is used to process a voice stream to obtain a text sequence.

[0081] The application framework layer provides application programming interfaces (APIs) and programming frameworks for the applications in the application layer. The application framework layer includes some predefined functions.

[0082] As Figure 2 shown, the application framework layer may include a window manager, a content provider, a view system, a telephone manager, a resource manager, a notification manager, etc.

[0083] In some embodiments, the voice recognition may also be implemented as a module in the application framework layer of the electronic device, such as a voice recognition module.

[0084] The window manager is used to manage window programs. The window manager can obtain the display screen size, determine whether there is a status bar, lock the screen, capture the screen, etc.

[0085] The content provider is used to store and obtain data, and make this data accessible to applications. The data may include videos, images, audio, dialed and answered calls, browsing history and bookmarks, a phone book, etc.

[0086] The view system includes visual controls, such as controls for displaying text, controls for displaying pictures, etc. The view system can be used to build applications. The display interface can be composed of one or more views. For example, a display interface including a short message notification icon can include a view for displaying text and a view for displaying pictures.

[0087] The phone manager is used to provide the communication functions of the electronic device 100. For example, the management of call status (including answering, hanging up, etc.).

[0088] The resource manager provides various resources for applications, such as localized strings, icons, pictures, layout files, video files, and so on.

[0089] The notification manager enables applications to display notification information in the status bar. It can be used to convey notification-type messages, which can automatically disappear after a short stay without user interaction. For example, the notification manager is used to inform that the download is completed, message reminders, etc. The notification manager can also be a notification that appears in the system top status bar in the form of a chart or scroll bar text, such as the notification of a background-running application, and can also be a notification that appears on the screen in the form of a dialogue window. For example, prompting text information in the status bar, emitting a prompt tone, and the indicator light flashing, etc.

[0090] Android Runtime includes a core library and a virtual machine. Android runtime is responsible for the scheduling and management of the Android system.

[0091] The core library consists of two parts: one part is the functional functions that need to be called by the Java language, and the other part is the core library of Android.

[0092] The application layer and the application framework layer run in the virtual machine. The virtual machine executes the Java files of the application layer and the application framework layer as binary files. The virtual machine is used to perform functions such as the management of object life cycles, stack management, thread management, security and exception management, and garbage collection.

[0093] The system library can include multiple functional modules. For example: surface manager, Media Libraries, 3D graphics processing library (such as: OpenGL ES), 2D graphics engine (such as: SGL), etc.

[0094] The surface manager is used to manage the display subsystem and provides the fusion of 2D and 3D layers for multiple applications.

[0095] The media library supports the playback and recording of a variety of common audio and video formats, as well as static image files, etc. The media library can support a variety of audio and video coding formats, such as: MPEG4, H.264, MP3, AAC, AMR, JPG, PNG, etc.

[0096] The 3D graphics processing library is used to implement 3D graphics drawing, image rendering, synthesis, and layer processing, etc.

[0097] The 2D graphics engine is a graphics engine for 2D drawing.

[0098] The kernel layer is the layer between hardware and software. The kernel layer at least includes a display driver, a camera driver, an audio driver, and a sensor driver.

[0099] Figure 3 This is a schematic structural diagram of the speech recognition module provided for the embodiments of this application.

[0100] As Figure 3 shown, the speech recognition module provided for the embodiments of this application may include an acoustic model (AM) 310, a phoneme predictor 320, and a phoneme translator 330. Optionally, the speech recognition system may further include a rectifier 340.

[0101] Among them, the acoustic model 310 may be a neural network model with a transducer structure. The training samples of the acoustic model 310 may be audio data. The acoustic model 310 mainly obtains relevant information of the corresponding phoneme based on the target speech signal frame. For example, the acoustic model 310 may obtain relevant information of the corresponding phoneme according to the target speech signal frame and the decoded output phoneme sequence.

[0102] In the embodiments of this application, the target speech signal frame may be a speech signal frame to be currently recognized. The decoded output phoneme sequence may include a starting phoneme and a phoneme sequence among one or more phoneme sequences corresponding to the speech signal frames before the target speech signal frame. The decoded output phoneme sequence may be obtained through the acoustic model 310. The relevant information of the phoneme corresponding to the target speech signal frame includes but is not limited to: the first probability matrix, the phoneme sequence, and the probability of the phoneme sequence.

[0103] Among them, the first probability matrix includes the probabilities of each phoneme in the phoneme vocabulary table corresponding to a phoneme corresponding to the target speech signal frame. The phoneme vocabulary table includes phonemes corresponding to one or more Chinese characters and / or phonemes corresponding to one or more English words. It can be understood that the first probability matrix includes the probabilities of the target speech signal frame being recognized as each phoneme in the phoneme vocabulary table.

[0104] Figure 4 This is a schematic structural diagram of an acoustic model, a phoneme predictor, and a phoneme translator provided by an embodiment of the present application.

[0105] As Figure 4 shown in (a) therein, the acoustic model 310 may include an encoder 311, a predict network 312, and a joint network 313.

[0106] For example, the encoder 311 may be a neural network such as a convolutional neural network (CNN) structure, a recurrent neural network (RNN) structure, or a transformer structure. Among them, the encoder 311 may be implemented as a residual neural network (ResNet) structure, a multi-layer long short-term memory (LSTM) structure, or a transformer structure, etc. As Figure 4 shown in (a) therein, the input of the encoder 311 is an audio signal processed by the Mel power spectrum, and the output is a feature vector of a fixed dimension.

[0107] For example, the predict network 312 may be a neural network model such as a CNN structure or an RNN structure. Among them, the predict network 312 may be implemented as a one-dimensional convolutional structure or a multi-layer LSTM structure, etc. As Figure 4 shown in (a) therein, the input of the predict network 312 is a phoneme sequence of the decoded output processed by the word vector, such as "jin1 / tian1 / tian1", and the output is a feature vector of a fixed dimension.

[0108] For example, the joint network 313 may include one or more fully connected layers. Among them, if each node in the hidden layer is connected to all nodes in the previous layer and combines the features extracted in the previous layer, then the hidden layer can be called a fully connected layer. As Figure 4 shown in (a) therein, the joint network 313 performs an operation of concatenating or directly adding the feature vector of a fixed dimension output by the encoder 311 and the feature vector of a fixed dimension output by the predict network 312, predicts the next phoneme, and obtains relevant information of the phoneme corresponding to the target speech signal frame.

[0109] Exemplarily, the phoneme predictor 320 may be a neural network model of a CNN structure, a RNN structure, or a transformer structure. For example, the phoneme predictor 320 may be implemented as a multi-layer LSTM structure, or a transformer structure. The training sample of the phoneme predictor 320 may be text data. The phoneme predictor 320 mainly obtains relevant information of the phoneme based on semantics. Figure 4 As shown in (b), the phoneme predictor 320 can obtain relevant information of the next phoneme of the decoded output phoneme sequence based on the decoded output phoneme sequence processed by the word vector.

[0110] In the embodiment of the present application, the relevant information of the next phoneme includes but is not limited to: a second probability matrix, a phoneme sequence, and the probability of the phoneme sequence. The second probability matrix includes the probability of each of the multiple phonemes included in the phoneme vocabulary corresponding to the next possible phoneme of the multiple decoded output phoneme sequences.

[0111] Exemplarily, the phoneme translator 330 may be a neural network model of a CNN structure, a RNN structure, or a transformer structure. For example, the phoneme translator 330 may be implemented as a 1-dimensional convolution structure, or a transformer structure. The training sample of the phoneme translator 330 may be text data. Figure 4 As shown in (c), the phoneme translator 330 can translate the phoneme sequence processed by the word vector into the corresponding text sequence, such as translating "jin1 / tian1 / tian1 / qi4" into "today's weather", and can also obtain the translation probability corresponding to the text sequence, and the phoneme sequence corresponds to the text sequence one by one. Optionally, the length of the text sequence output by the phoneme translator is less than or equal to the length of the phoneme sequence. If the length of the text sequence is less than the length of the phoneme sequence, a placeholder can be added to the text sequence to make the length of the text sequence equal to the length of the phoneme sequence.

[0112] In some embodiments, the phoneme translator 330 may also be used to obtain the probability that each phoneme in the phoneme sequence is translated into each text symbol in the plurality of text symbols included in the text vocabulary. Figure 4 As shown in (c), the probability of "jin1" being translated into "今" is 0.8, which is not listed here. After the phoneme translator completes the translation of the complete phoneme sequence corresponding to the speech stream, a first grid structure is obtained, which includes a probability matrix of the phoneme sequence corresponding to the text.

[0113] Exemplarily, the acoustic model 310, the phoneme predictor 320, and the phoneme translator 330 can be used to perform streaming recognition on the speech stream. Taking the speech stream "The weather is good today" as an example, after the acoustic model 310 processes the speech signal frame "today", the acoustic model 310 starts to process the speech signal "day", which can improve the recognition efficiency. Similarly, after the phoneme translator 330 processes the data corresponding to the speech signal frame "today", it displays the text corresponding to the speech signal frame "today", and then processes the data corresponding to the speech signal "day", and stream-displays the text sequence corresponding to the speech signal sequence "today". Similarly, the streaming recognition of the entire sentence speech stream is completed, and the streaming recognition result can be the final recognition result.

[0114] The acoustic model 310 is a model trained based on audio data, and can obtain information related to phonemes based on target speech signal frames. The phoneme predictor 320 is a model trained based on text data, and can obtain relevant information about the next phoneme based on semantics. The phoneme translator 330 is a model trained based on text data, and can translate phonemes into text. These models take up little storage space and can be deployed in electronic devices to achieve stream recognition, with rich training samples, including both audio data and text data, and high recognition accuracy.

[0115] Optionally, combined Figure 5 , the rectifier 340 may include a confuse sound matrix, a phoneme text matrix, and a weighted finite state transducer (WFST) ( Figure 5 ). The rectifier 340 may use the confusion sound matrix and the phoneme text matrix to trim the first grid structure, obtain the second grid structure, and use the WFST to perform Viterbi algorithm decoding on the second grid structure to obtain the final text sequence.

[0116] For example, the confusion matrix is obtained by statistically analyzing the acoustic model 310 using training samples, and the confusion matrix is used to represent the probability that each phoneme in the phoneme vocabulary is recognized by the acoustic model 310 as each phoneme in the entire phoneme. Figure 5 , the probability of phoneme "a1" being recognized as phoneme "a1" is 0.5, and the probability of being recognized as phoneme "a2" is 0.2, which will not be repeated here.

[0117] For example, the phoneme text matrix is a 0-1 matrix used to represent the phonemes corresponding to the text symbols. Taking Chinese speech recognition as an example, the phoneme text matrix can represent the correspondence between Chinese characters and pinyin, for example, Figure 5, the Chinese character "啊" can make the pinyin "a1", so the value of the phoneme "a1" corresponding to "啊" is 1. The Chinese character "啊" cannot make the pinyin "a2", so the value of the phoneme "a2" corresponding to "啊" is 0.

[0118] After completing the streaming recognition of the entire speech stream, the rectifier 340 can correct the ambiguous sound and fine-tune the probability of the text in the first grid structure to obtain a more accurate text sequence, thereby further improving the accuracy of speech recognition.

[0119] It should be noted that the embodiments of the present application do not limit the specific implementation methods of the acoustic model 310, the phoneme predictor 320, the phoneme translator 330 and the rectifier 340. For example, there is no limitation on which neural network model is used to implement them, and the corresponding functions recorded in the embodiments of the present application shall be realized.

[0120] The following will be based on electronic equipment Figure 1 and Figure 2 Taking the terminal with the shown structure as an example, the speech recognition method provided in the embodiment of the present application is explained.

[0121] For example, Figure 6 Schematic diagram of the process of the speech recognition method provided in the embodiment of the present application Figure 1 See also Figure 6 The speech recognition process includes S601-S606. The following takes the speech stream "The weather is good today" as an example to describe the technical solution provided by the embodiment of the present application in detail in conjunction with the accompanying drawings.

[0122] S601: The terminal obtains a first probability matrix according to a target speech signal frame and a first phoneme sequence among a plurality of first phoneme sequences.

[0123] The voice stream includes at least one voice signal frame, and the terminal can use audio power spectrum, Mel power spectrum, or MFCC to extract acoustic features of the voice stream. The target voice signal frame is a voice signal frame to be recognized. For example, if the voice stream is "The weather is good today", if the terminal interface currently displays "Today's weather", the voice signal frame "Today's weather" is the recognized voice signal frame, and the voice signal frame "air" can be the target voice signal frame.

[0124] Exemplarily, the number of first phoneme sequences may be multiple, and each of the multiple first phoneme sequences may include a phoneme sequence among multiple phoneme sequences corresponding to a starting phoneme and one or more speech signal frames before the target speech signal frame. The first phoneme sequence may be word vector processed, and the starting phoneme may include a null character.

[0125] For example, when the target speech signal frame is the first speech signal frame in the speech stream, such as when the target speech signal frame is "今", the first phoneme sequence may be a null character.

[0126] For example, when the target speech signal frame is the first "天" in the speech stream, each first phoneme sequence may include a phoneme sequence corresponding to the speech signal frame "今", such as "jin1", "xin1", or "a1", etc. For another example, when the target speech signal frame is "氣", each first phoneme sequence may include a phoneme sequence corresponding to the speech signal frame sequence "今日天", such as "jin1 / tian1 / tian1", or "jin1 / tian1 / tian2", etc.

[0127] Optionally, the terminal may respectively perform the above S601 according to the target speech signal frame and each of the multiple first phoneme sequences. For example, the target speech signal frame is "qi", and the multiple first phoneme sequences include "jin1 / tian1 / tian1", and "jin1 / tian1 / tian2". The terminal may obtain the corresponding first probability matrix according to the target speech signal frame "qi" and the first phoneme sequence "jin1 / tian1 / tian1", and obtain the corresponding first probability matrix according to the target speech signal frame "qi" and the first phoneme sequence "jin1 / tian1 / tian2".

[0128] In some embodiments, the first probability matrix may include the probability of each of the multiple phonemes included in the phoneme vocabulary corresponding to a phoneme corresponding to the target speech signal frame, wherein the phoneme vocabulary includes phonemes corresponding to one or more Chinese characters and / or phonemes corresponding to one or more English words.

[0129] For example, as shown in Table 1, when the target speech signal frame is "今", the first probability matrix may include that the target speech signal frame "今" is determined to be "jin1" in the phoneme vocabulary with a first probability of 0.7, is determined to be "jing1" with a first probability of 0.2, is determined to be "jun1" with a first probability of 0.05, is determined to be "jin4" with a first probability of 0.02, etc., and the embodiments of the present application do not list them one by one.

[0130] Table 1

[0131] phoneme first probability jin1 0.7 jing1 0.2 jun1 0.05 jun4 0.02 … …

[0132] For another example, as shown in Table 2, when the target voice signal frame is "qi", the first probability matrix may include that the first probability of determining the target voice signal frame "qi" as "qi4" in the phoneme vocabulary is 0.7, the first probability of determining it as "qu4" is 0.1, the first probability of determining it as "qie4" is 0.05, the first probability of determining it as "ji4" is 0.01, etc. The embodiments of the present application do not list them one by one.

[0133] Table 2

[0134] phoneme first probability qi4 0.7 qu4 0.1 qie4 0.05 ji4 0.01 … …

[0135] It should be noted that the embodiments of the present application do not list one by one the first probability matrices corresponding to each voice signal frame in the voice stream of "The weather is good today".

[0136] In a possible design, the terminal obtains at least one second phoneme sequence and / or a fourth probability matrix according to the target voice signal frame and one of the multiple first phoneme sequences. Each second phoneme sequence in the at least one second phoneme sequence is composed of one of the multiple first phoneme sequences and a phoneme corresponding to the target voice signal frame. The fourth probability matrix includes the probabilities corresponding to each second phoneme sequence in the at least one second phoneme sequence. The fourth probability matrix may include a fourth probability, and the fourth probability is the probability corresponding to each second phoneme sequence.

[0137] Taking the target voice signal frame as the first voice signal frame in the voice stream as an example, for example, the target voice signal frame is "jin". Combining Table 3, the at least one second phoneme sequence determined by the terminal according to the target voice signal frame "jin" and the empty character may include "jin1", "jing1", "jun1", and "jin4", etc. Among them, the fourth probability corresponding to the second phoneme sequence "jin1" is 0.7, the fourth probability corresponding to "jing1" is 0.2, the fourth probability corresponding to "jun1" is 0.05, the fourth probability corresponding to "jun4" is 0.02, etc. The embodiments of the present application do not list them one by one.

[0138] Table 3

[0139] second phoneme sequence fourth probability jin1 0.7 jing1 0.2 jun1 0.05 jun4 0.02 … …

[0140] Table 4

[0141] second phoneme sequence fourth probability jin1 / tian1 / tian1 / qi4 0.4 jin1 / tian1 / tian1 / qu4 0.1 jin1 / tian1 / tian1 / qie4 0.03 jin1 / tian1 / tian1 / ji4 0.01 … …

[0142] Taking the example that the target speech signal frame is not the first speech signal frame in the speech stream. For example, the target speech signal frame is "qi", and the first phoneme sequence is "jin1 / tian1 / tian1". As shown in Table 4, the at least one determined second phoneme sequence may include "jin1 / tian1 / tian1 / qi4", "jin1 / tian1 / tian1 / qu4", "jin1 / tian1 / tian1 / qie4", and "jin1 / tian1 / tian1 / ji4", etc. Among them, the fourth probability corresponding to the second phoneme sequence "jin1 / tian1 / tian1 / qi4" is 0.4, the fourth probability corresponding to "jin1 / tian1 / tian1 / qu4" is 0.1, the fourth probability corresponding to "jin1 / tian1 / tian1 / qie4" is 0.03, the fourth probability corresponding to "jin1 / tian1 / tian1 / ji4" is 0.01, etc. The embodiments of the present application do not list them one by one.

[0143] It should be noted that the present application does not limit the order of obtaining the first probability matrix and obtaining the at least one second phoneme sequence and / or the fourth probability matrix. For example, the terminal can obtain the at least one second phoneme sequence and / or the fourth probability matrix while obtaining the first probability matrix.

[0144] In some embodiments, the second phoneme sequence corresponding to the target speech signal frame can be used as the first phoneme sequence corresponding to the next speech signal frame of the target speech signal frame.

[0145] For example, in combination with Table 3 above, when the target speech signal frame is "jin", the at least one obtained second phoneme sequence "jin1", "jing1", "jun1", and "jin4", etc. can be used as the first phoneme sequence corresponding to the first "tian" in the speech stream. That is to say, when the first "tian" in the speech signal frame is the target speech signal frame, the terminal can use "jin1", "jing1", "jun1", or "jin4", etc. as the first phoneme sequence to execute the method provided by the embodiments of the present application. For example, when executing S601 above, the terminal can obtain the first probability matrix according to the speech signal frame "tian" and the first phoneme sequence ("jin1", "jing1", "jun1", or "jin4", etc.).

[0146] For another example, in combination with Table 4 above, when the target speech signal frame is "qi", at least one obtained second phoneme sequence such as "jin1 / tian1 / tian1 / qi4", "jin1 / tian1 / tian1 / qu4", "jin1 / tian1 / tian1 / qie4", and "jin1 / tian1 / tian1 / ji4" can be used as the corresponding first phoneme sequence for the speech signal frame "hao" in the speech stream. That is to say, when the speech signal frame "hao" is the target speech signal frame, the terminal can use "jin1 / tian1 / tian1 / qi4", "jin1 / tian1 / tian1 / qu4", "jin1 / tian1 / tian1 / qie4", or "jin1 / tian1 / tian1 / ji4", etc. as the first phoneme sequence to execute the method provided in the embodiments of the present application. For example, the terminal can obtain at least one second phoneme sequence and / or a fourth probability matrix according to the target speech signal frame "hao" and the first phoneme sequence ("jin1 / tian1 / tian1 / qi4", "jin1 / tian1 / tian1 / qu4", "jin1 / tian1 / tian1 / qie4", or "jin1 / tian1 / tian1 / ji4", etc.).

[0147] Optionally, the terminal can use each second phoneme sequence in at least one second phoneme sequence corresponding to the target speech signal frame as the first phoneme sequence for the next speech signal frame corresponding to the target speech signal frame, or the terminal can use some second phoneme sequences in at least one second phoneme sequence as the first phoneme sequence for the next speech signal frame corresponding to the target speech signal frame. For example, the terminal can determine the first phoneme sequence according to the probability corresponding to the second phoneme sequence, such as using the second phoneme sequence with the highest probability as the first phoneme sequence for the next speech signal frame corresponding to the target speech signal frame.

[0148] In a possible design, the terminal can use an acoustic model to process the target speech signal frame and one of the multiple first phoneme sequences to obtain one or more of the following: a first probability matrix, at least one second phoneme sequence, and a fourth probability matrix.

[0149] Combined with Figure 7 In (a) above, taking the target speech signal frame being the speech signal frame "jin" as an example, the acoustic model 310 processes the target speech signal frame "jin" and the first phoneme sequence empty character to obtain the first probability matrix corresponding to the target speech signal frame "jin", multiple second phoneme sequences corresponding to the target speech signal frame "jin", and the fourth probabilities corresponding to the multiple second phoneme sequences respectively.

[0150] Combined with Figure 7In (b), taking the target speech signal frame as the speech signal frame "qi" and the corresponding first phoneme sequence as "jin1 / tian1 / tian1" as an example, the acoustic model 310 can process the target speech signal frame "qi" and the first phoneme sequence "jin1 / tian1 / tian1" to obtain the first probability matrix corresponding to the target speech signal frame "qi", multiple second phoneme sequences corresponding to the target speech signal frame "qi", and the fourth probabilities respectively corresponding to the multiple second phoneme sequences.

[0151] In some embodiments, the terminal can use the encoder 311 of the acoustic model 310 to process the target speech signal frame to obtain a feature vector F with a fixed dimension encoder , use the prediction network 312 of the acoustic model 310 to process the first phoneme sequence to obtain a feature vector F with a fixed dimension predNet , and the embodiments of the present application do not limit the order of obtaining F encoder and F predNet . Then, use the joint network 313 to splice F encoder and F predNet , or directly add them, etc., to obtain the first probability matrix corresponding to the target speech signal frame, multiple second phoneme sequences, and the fourth probabilities respectively corresponding to the multiple second phoneme sequences. The first probability matrix includes the first probability P AM (t d ).

[0152] Wherein, t represents the target speech signal frame, t can be the sorting of the target speech signal frame in the speech stream, t is an integer greater than or equal to 1, the length of the first speech sequence can be t - 1, d represents a phoneme, and P AM (t d ) represents the probability of the phoneme d corresponding to the target speech signal frame t.

[0153] Combined with Figure 7 in (c), taking the target speech signal frame as "qi" and the corresponding first phoneme sequence as "jin1 / tian1 / tian1" as an example. The encoder 311 processes the target speech signal frame "qi" to obtain a feature vector F with a fixed dimension encoder . The prediction network 312 processes the first phoneme sequence "jin1 / tian1 / tian1" to obtain a feature vector F with a fixed dimension predNet . The joint network 313 splices F encoder and F predNet , or directly adds them, etc., to obtain the first probability matrix corresponding to the target speech signal frame "qi", multiple second phoneme sequences, and the fourth probabilities respectively corresponding to the multiple second phoneme sequences.

[0154] S602, the terminal obtains a second probability matrix according to the semantics of the first phoneme sequence.

[0155] Optionally, the semantics of the first phoneme sequence represents the linguistic meaning expressed by the first phoneme sequence, and the second probability matrix includes the probabilities of each phoneme among the multiple phonemes included in the phoneme vocabulary corresponding to the next possible phoneme of the multiple first phoneme sequences. In this way, the terminal can obtain the probabilities of each phoneme among the multiple phonemes included in the phoneme vocabulary being determined as the next possible phoneme of the first phoneme sequence according to the semantics of the first phoneme sequence.

[0156] For example, when the target speech signal frame is the first speech signal frame in the speech stream, referring to Table 5, the terminal can obtain, based on the null character, that the probability of the first phoneme of the speech stream being determined as the phoneme "jin1" in the phoneme vocabulary is 0.2, the probability of being determined as the phoneme "wo3" is 0.1, the probability of being determined as the phoneme "jing1" is 0.01, the second probability of being determined as the phoneme "hao3" is 0.01, etc. The embodiments of the present application will not list them one by one.

[0157] Table 5

[0158] phoneme second probability jin1 0.2 wo3 0.1 jing1 0.01 hao3 0.01 … …

[0159] Taking the first phoneme sequence "jin1 / tian1 / tian1" as an example, referring to Table 6, the terminal can obtain, based on the possible linguistic meaning to be expressed by "jin1 / tian1 / tian1", that the probability of the next possible phoneme of the first phoneme sequence "jin1 / tian1 / tian1" being determined as the phoneme "qi4" is 0.2, the probability of being determined as the phoneme "qu4" is 0.01, the probability of being determined as the phoneme "leng3" is 0.05, the probability of being determined as the phoneme "yin1" is 0.05, etc. The embodiments of the present application will not list them one by one.

[0160] Table 6

[0161] phoneme second probability qi4 0.2 qing2 0.1 leng3 0.05 qu4 0.01 … …

[0162] In some embodiments, the terminal can use a phoneme predictor to obtain a second probability matrix according to the semantics of the first phoneme sequence, and the second probability matrix includes the second probability P Predictor (t d ).

[0163] Wherein, t is the sorting of the speech signal frame corresponding to the next possible phoneme of the first phoneme sequence (i.e., the target speech signal frame) in the speech stream, t is an integer greater than or equal to 1, the length of the first phoneme sequence can be t - 1, d represents a phoneme, and P Predictor (t d) represents the probability of phoneme d among multiple phonemes included in the phoneme vocabulary corresponding to the next possible phoneme t of the first phoneme sequence.

[0164] Combined with Figure 8 , taking the first phoneme sequence as "jin1 / tian1 / tian1" and t = 4 as an example, the phoneme predictor 320 can obtain the probability P that the next possible phoneme of the first phoneme sequence "jin1 / tian1 / tian1" is determined to be phoneme "qi4" according to the semantics of the first phoneme sequence "jin1 / tian1 / tian1". Predictor (4 qi4 ) = 0.2, the probability P that it is determined to be phoneme "qing2". Predictor (4 qing2 ) = 0.1, the probability P that it is determined to be phoneme "leng3". Predictor (4 leng3 ) = 0.05, the probability P that it is determined to be phoneme "qu4". Predictor (4 qu4 ) = 0.01, etc. The embodiments of the present application will not list them one by one.

[0165] S603. The terminal obtains N target phoneme sequences with the largest target probabilities according to the first probability matrix and the second probability matrix.

[0166] Among them, each target phoneme sequence is composed of a phoneme corresponding to the fourth step and the target speech signal frame, N is preset or pre-configured, N is an integer greater than or equal to 1, and the target probability is the probability of each target phoneme sequence. That is to say, the terminal can retain the N target phoneme sequences with the highest probabilities, which can avoid only selecting the phoneme with the highest probability and missing the globally optimal phoneme sequence, and can improve the accuracy of speech recognition.

[0167] In a possible design, the above S603 may include: Step 1, the terminal obtains a fifth probability matrix according to the first probability matrix and the second probability matrix. Step 2, the terminal obtains N target phoneme sequences with the largest target probabilities according to the fifth probability matrix corresponding to one or more speech signal frames before the target speech signal frame and the target speech signal frame respectively.

[0168] Next, a specific description is given of Step 1, where the terminal obtains a fifth probability matrix according to the first probability matrix and the second probability matrix.

[0169] Optionally, the fifth probability matrix includes the fusion probabilities of each phoneme among multiple phonemes included in the phoneme vocabulary corresponding to a phoneme corresponding to the target speech signal frame. The fifth probability matrix includes a fifth probability, and the fifth probability is the probability of one phoneme among multiple phonemes included in the phoneme vocabulary corresponding to a phoneme corresponding to the target speech signal frame. That is to say, the terminal can fuse the first probability matrix and the second probability matrix to obtain the final probability of the next possible phoneme.

[0170] In some embodiments, the following formula (1) can be used to fuse the first probability P AM (t d ) and the second probability P Predictor (t d ) to obtain the fifth probability P(t d ).

[0171] P(t d ) = P AM (t d ) + λ1 × P Predictor (t d ) (1)

[0172] In the above formula (1), P(t d ) is the fusion probability of the phoneme d corresponding to the target speech signal frame t, λ1 is a fusion parameter, and the value range of λ1 is 0 ≤ λ1 ≤ 1. λ1 can be adjusted according to the training samples used to train the acoustic model and / or the phoneme predictor.

[0173] Taking the target speech signal frame as the first speech signal frame in the speech stream, t as 1, and d as the phoneme "jin1" as an example. Combining Figure 9 in (a), P AM (1 jin1 ) = 0.7, P Predictor (1 jin1 ) = 0.2. Assuming λ1 = 0.3, the fusion probability of the first speech signal frame in the speech stream corresponding to the phoneme "jin1" is P(1 jin1 ) = 0.7 + 0.3 × 0.2 = 0.76. Similarly, the fusion probabilities of the first speech signal frame in the speech stream corresponding to other phonemes d can also be calculated using the above formula (1), which will not be listed one by one in the embodiments of this application.

[0174] Taking the target speech signal frame as the fourth speech signal frame in the speech stream, t as 4, and d as the phoneme "qi4" as an example. Combining Figure 9 in (b), P AM (4 qi4 ) = 0.7, P Predictor (4 qi4) = 0.2. Assuming λ1 = 0.5, the fusion probability of the fourth speech signal frame in the speech stream corresponding to the phoneme "qi4" is P(4 qi4 ) = 0.7 + 0.5 × 0.2 = 0.8. Similarly, the fusion probability of the fourth speech signal frame in the speech stream corresponding to other phonemes d can also be calculated using the above formula (1), which will not be listed one by one in the embodiments of this application.

[0175] In some embodiments, the following formula (2) can be used to fuse the first probability P AM (t d ) and the second probability P Predictor (t d ) to obtain the fifth probability P(t d ).

[0176] P(t d ) = λ2 × P AM (t d ) + λ1 × P Predictor (t d ) (2)

[0177] In the above formula (2), λ2 is a fusion parameter, and the value range of λ2 is 0 ≤ λ2 ≤ 1. λ2 is obtained based on the training samples used to train the acoustic model and / or the phoneme predictor.

[0178] Taking the target speech signal frame as the first speech signal frame in the speech stream, t as 1, and d as the phoneme "jing1" as an example. Combining Figure 9 in (a), P AM (1 jing1 ) = 0.2, P Predictor (1 jing1 ) = 0.01. Assuming λ2 = 0.2 and λ1 = 1, the fusion probability of the first speech signal frame in the speech stream corresponding to the phoneme "jing1" is P(1 jing1 ) = 0.2 × 0.2 + 0.01 × 1 = 0.05. Similarly, the fusion probability of the first speech signal frame in the speech stream corresponding to other phonemes d can also be calculated using the above formula (2), which will not be elaborated in the embodiments of this application.

[0179] In some embodiments, the following formula (3) can be used to fuse the first probability P AM (t d ) and the second probability P Predictor (t d ) to obtain the fifth probability P(t d ).

[0180] P(t d ) = λ2 × P AM (t d) + P Predictor (t d ) (3)

[0181] Taking the case where the target speech signal frame is the fourth speech signal frame in the speech stream, t is 4, and d is the phoneme "qu4" as an example. Combining Figure 9 in (b), P AM (4 qu4 ) = 0.1, P Predictor (4 qu4 ) = 0.01, assuming λ2 = 0.4, then the fusion probability of the first speech signal frame in the speech stream corresponding to the phoneme "qu4" is P(4 qu4 ) = 0.4 × 0.1 + 0.01 = 0.05. Similarly, the above formula (3) can also be used to calculate the fusion probability of the fourth speech signal frame in the speech stream corresponding to other phonemes d. Examples of this application are not listed one by one.

[0182] It should be noted that this application does not limit the value of λ1. Exemplarily, when obtaining the fifth probability P(t d ), when the speech signal frames t are the same but the phonemes d are different, the values of λ1 used can be different. For example, when the target speech signal frame is the first speech signal frame in the speech stream, the value of λ1 used to obtain P(1 jin1 ) can be different from the value of λ1 used to obtain P(1 jun1 ). Exemplarily, when obtaining the fifth probability matrix P(t d ), when the speech signal frames t are different, the values of λ1 used can be different. For example, the value of λ1 used to obtain P(1 jin1 ) can be different from the value of λ1 used to obtain P(4 qi4 ). Similarly, this application does not limit the value of λ2, which will not be elaborated here.

[0183] It should be noted that the terminal can use one or more of the above formulas (1), (2), and (3) to obtain the fifth probability P(t d ). Exemplarily, the above formulas (1), (2), and (3) can be used alone. For example, when recognizing the speech stream "The weather is good today", the terminal only uses the above formula (1) to obtain the fifth probability P(t d ). Another example is that the above formulas (1), (2), and (3) can be used in combination. For example, when the target speech signal frame is the first speech signal frame in the speech stream, the terminal uses formula (1) to obtain the fifth probability P(t d ), and when the target speech signal frame is the second speech signal in the speech stream, the terminal uses formula (3) to obtain the fifth probability P(t d)。 Alternatively, when the target speech signal frame is the first speech signal frame in the speech stream, the terminal uses the above formula (1) to obtain the probability P(1 of the phoneme "jin1" corresponding to the speech signal frame jin1 ) and uses the above formula (2) to obtain the probability P(1 of other phonemes (such as "jing1") corresponding to the speech signal frame jing1 ).

[0184] The following specifically elaborates on the above step 2, where the terminal obtains N target phoneme sequences with the largest target probabilities according to the fifth probability matrices respectively corresponding to one or more speech signal frames before the target speech signal frame and the target speech signal frame.

[0185] Taking the target speech signal frame as the first speech signal frame in the speech stream as an example, combined with Figure 10 in (a), assuming N is 3, the terminal can sort the probabilities P(1 d ) of the phonemes corresponding to the first speech signal frame "jin" to obtain the target probability P phone-seq (i) of the largest 3 target phoneme sequences, such as "jin1", "jing1", "jun1" respectively. Here, i represents the target phoneme sequence.

[0186] Taking the target speech signal frame as the fourth speech signal frame in the speech stream as an example, combined with Figure 10 in (b), assuming N is 3, the terminal can, based on the probability P(1 d ) of the phonemes corresponding to the first speech signal frame "jin" to the probability P(4 d ) of the phonemes corresponding to the fourth speech signal frame "qi", obtain the target probability P phone-seq (i) of the largest 3 target phoneme sequences, such as "jin1 / tian1 / tian1 / qi4", "jin1 / tian1 / tian1 / qu4", "jin1 / tian1 / tian1 / qing2" respectively.

[0187] In some embodiments, the above step 2 may include: the terminal obtains at least one target phoneme sequence and at least one target probability according to the fifth probability matrices respectively corresponding to one or more speech signal frames before the target speech signal frame and the target speech signal frame. Then, from the at least one target phoneme sequence, the terminal obtains N target phoneme sequences with the largest target probabilities. Among them, the target phoneme sequence and the target probability correspond one by one.

[0188] Taking the target speech signal frame as the fourth speech signal frame in the speech stream as an example, combined with Figure 10 in (c), assuming N is 3, the terminal can, based on the probability P(1 d) to the fourth speech signal frame "气" corresponding to the phoneme probability P(4 d ), and obtain the target probability P corresponding to multiple target phoneme sequences i and the target phoneme sequence phone-seq (i) The terminal can obtain the target probability P from multiple target phoneme sequences. phone-seq (i) The largest three target phoneme sequences are “jin1 / tian1 / tian1 / qi4”, “jin1 / tian1 / tian1 / qu4” and “jin1 / tian1 / tian1 / qing2”.

[0189] In some embodiments, the terminal may use a beam search algorithm to obtain N target phoneme sequences with the highest target probabilities based on the fifth probability matrices corresponding to one or more speech signal frames before the target speech signal frame and the target speech signal frame.

[0190] The beam search algorithm is a search strategy in the sequence-to-sequence deep natural language processing algorithm. The beam search algorithm selects multiple alternatives for the input sequence at each time step based on conditional probability. In this way, the terminal retains the N target phoneme sequences with the highest probability, which can avoid selecting only the phonemes with the highest probability and missing the globally optimal phoneme sequence, thereby improving the accuracy of speech recognition.

[0191] S604: The terminal translates the N target phoneme sequences to obtain a third probability matrix.

[0192] The third probability matrix includes the probability that one of the N target phoneme sequences is translated into at least one first text sequence. Exemplarily, the terminal can obtain one or more first text sequences and the corresponding third probability matrix based on the same target phoneme sequence. The third probability matrix includes the third probability, the third probability P translator (i e ) is the probability that the target phoneme sequence i is translated into the first text sequence e, where e represents the first text sequence or text symbol.

[0193] Taking the target speech signal frame as the first speech signal frame in the speech stream as an example, combined with Figure 11 In (a), the terminal can translate the target phoneme sequence "jin1" to obtain at least one first text sequence, such as "今", P translator (jin1 今 )=0.75, and for example, "金", etc., which are not listed one by one in the embodiments of the present application. Translate the target phoneme sequence "jing1" to obtain at least one first text sequence, such as "经", P translator (jing1 经)=0.5; translate the target phoneme sequence "jun1" to obtain at least one first text sequence, such as "军", P translator (jun1 军 )=0.6.

[0194] Take the target speech signal frame as the fourth speech signal frame in the speech stream as an example, combined with Figure 11 In (b), the terminal can translate the target phoneme sequence "jin1 / tian1 / tian1 / qi4" to obtain at least one first text sequence, such as "today's weather", P translator (jin1 / tian1 / tian1 / qi4 今天天气 )=0.98, and for example, "今日天器", etc., which are not listed one by one in the embodiments of this application. Translate the target phoneme sequence "jin1 / tian1 / tian1 / qu4" to obtain at least one first text sequence, such as "今日天去", P translator (jin1 / tian1 / tian1 / qu4 今天天去 )=0.84; translate the target phoneme sequence “jin1 / tian1 / tian1 / qing2” to obtain at least one first text sequence, such as “今日天晴”, P translator (jin1 / tian1 / tian1 / qing2 今天天晴 )=0.9.

[0195] In a possible design, the terminal may translate one or more phonemes in the phoneme vocabulary corresponding to the target speech signal frame to obtain a text probability matrix. The text probability matrix includes the probability of the corresponding phoneme being translated into each text symbol in the multiple text symbols included in the text vocabulary, and the text probability matrix includes the text probability H(t e ), text probability H(t e ) includes the probability that phoneme t is translated into a text symbol e among a plurality of text symbols included in the text vocabulary.

[0196] For example, taking the target speech signal frame as "今" as an example, combined with Figure 11 In (c), for example, the phoneme corresponding to the target speech signal frame "今" is "jin1", and the terminal can obtain the probability H(jin1) that the phoneme "jin1" is translated into the text symbol "今". 今 )=0.75, the probability of being translated into the text symbol "金" H(jin1 金 )=0.10, the probability of being translated into the text symbol "晶" is H(jin1 晶 )=0.05, the probability of being translated into the text symbol "津" H(jin1 津)=0.02, the probability of being translated into the text symbol "军" H(jin1 军 )=0.01, the embodiments of the present application are not listed one by one.

[0197] For example, taking the target speech signal frame as "今" as an example, combined with Figure 11 In (d), for example, the phoneme corresponding to the target speech signal frame "今" is "jing1", and the terminal can obtain the probability H(jing1) that the phoneme "jing1" is translated into the text symbol "经". 经 )=0.5, the probability of being translated into the text symbol “晶” is H(jing1 晶 )=0.2, the probability of being translated into the text symbol "惊" H(jing1 惊 )=0.05, the probability H(jing1) of being translated into the text symbol “京” 京 )=0.02, the probability of being translated into the text symbol "今" H(jing1 今 )=0.01, the embodiments of the present application are not listed one by one.

[0198] For example, taking the target speech signal frame as "气" as an example, combined with Figure 11 In (e), for example, the phoneme corresponding to the target speech signal frame "qi" is "qi4", and the terminal can obtain the probability H(qi4) that the phoneme "qi4" is translated into the text symbol "qi". 气 )=0.6, the probability of being translated into the text symbol "器" is H(qi4 器 )=0.2, the probability of being translated into the text symbol “弃” is H(qi4 弃 )=0.1, the probability of being translated into the text symbol "去" is H(qi4 去 )=0.02, the probability of being translated into the text symbol "却" is H(qi4 却 )=0.01, the embodiments of the present application are not listed one by one.

[0199] It should be noted that the terminal can obtain the probability that other phonemes corresponding to the target voice signal frame "今" (such as "jun1") are translated into each of the multiple text symbols included in the text vocabulary, and can obtain the probability that phonemes corresponding to other voice signals in the voice stream (such as "好") are translated into each of the multiple text symbols included in the text vocabulary, which are not listed one by one in the embodiments of the present application.

[0200] In a possible design, the terminal may use a phoneme translator to translate N target phoneme sequences to obtain third probability matrices corresponding to the N target phoneme sequences respectively.

[0201] Take the target speech signal frame as the fourth speech signal frame in the speech stream as an example, combined with Figure 12The terminal can use the phoneme translator 330 to translate the target phoneme sequence "jin1 / tian1 / tian1 / qi4" to obtain at least one first text sequence, such as "today's weather", P translator (jin1 / tian1 / tian1 / qi4 今天天气 )=0.98, and for example, "今日天器", etc., which are not listed one by one in the embodiments of the present application. Of course, the terminal can translate the target phoneme sequence "jin1 / tian1 / tian1 / qu4", "jin1 / tian1 / tian1 / qing2", which are not listed one by one here.

[0202] In a possible design, the terminal may translate one or more phonemes among a plurality of phonemes included in a phoneme vocabulary corresponding to the target speech signal frame using a phoneme translator to obtain a text probability matrix.

[0203] For example, in combination Figure 12 When the target speech signal frame is "今", the terminal can use the phoneme translator 330 to translate the phoneme "jin1" corresponding to the target speech signal frame "今" to obtain the probability H(jin1) that the phoneme "jin1" is translated into the text symbol "今". 今 )=0.75, the probability of being translated into the text symbol "金" H(jin1 金 )=0.01, which are not listed here one by one. The terminal can also use the phoneme translator 330 to obtain the probability that other phonemes (such as "jing1", "jun1", etc.) corresponding to the target voice signal frame "今" are translated into each of the multiple text symbols included in the text vocabulary. Similarly, when each voice signal frame in the voice stream is used as the target voice signal frame, the terminal can use the phoneme translator 330 to translate one or more phonemes of the multiple phonemes included in the phoneme vocabulary corresponding to the target voice signal frame to obtain the corresponding text probability matrix, which are not listed one by one in the embodiments of the present application.

[0204] Optionally, the terminal may store the text probability matrix and may adjust the text probability matrix corresponding to the voice signal frame sorted in front according to the text probability matrix corresponding to the voice signal frame sorted in the back in the voice stream.

[0205] Combination Figure 12 When the target speech signal frame is "今", the terminal can use the phoneme translator 330 to obtain H(jin1 今 )=0.75,H(jin1 金 )=0.10, and store it. When the target speech signal frame becomes the next speech signal frame "day" of "jin", the terminal can adjust H(jin1) by the text probability matrix corresponding to the target speech signal frame "day". 今), and / or H(jin1 金 ) value. When the target speech signal frame becomes "qi", the terminal can adjust H(jin1) according to the text probability matrix corresponding to the target speech signal frame "qi". 今 ), and / or H(jin1 金 ) value.

[0206] It should be noted that the present application does not limit the order in which the terminal translates one or more phonemes among the multiple phonemes included in the phoneme vocabulary corresponding to the target speech signal frame and the terminal translates the N target phoneme sequences.

[0207] S605: The terminal determines a first text sequence with the highest probability among at least one first text sequence according to the third probability matrix and the target probabilities corresponding to the N target phoneme sequences.

[0208] In some embodiments, the target probability P can be calculated using the following formula (4): phone-seq (i) and the third probability P translator (i e ) to obtain the probability P(i e ).

[0209] P(i e )=P phone-seq (i)+λ3×P translator (i e ) (4)

[0210] In the above formula (4), i represents the target phoneme sequence, e represents the first text sequence, and the probability P(i e ) is the fusion probability of the target phoneme sequence i corresponding to the first text sequence e, λ3 is the fusion parameter, the value range of λ3 is 0≤λ3≤1, and λ3 can be adjusted according to the training samples, such as the training samples used to train the phoneme translator.

[0211] Combination Figure 13 In (a), taking the target speech signal frame as the speech signal frame "今" as an example, P phone-seq (jin1)=0.8,P translator (jin1 今 )=0.75, assuming λ3=0.2, then the fusion probability of the target phoneme sequence “jin1” corresponding to the first text sequence “今” is P(jin1 今 )=0.8+0.2×0.75=0.95. Similarly, the above formula (4) can also be used to calculate the fusion probability of the target phoneme sequence "jin1" corresponding to other first text sequences (such as "金"), which will not be described in detail in the embodiment of the present application.

[0212] CombinationFigure 13 In (b), taking the target speech signal frame as the speech signal frame "qi" as an example, P phone-seq (jin1 / tian1 / tian1 / qi4) = 0.5, P translator (jin1 / tian1 / tian1 / qi4 今天天气 ) = 0.98. Assuming λ3 = 0.2, the fusion probability of the target phoneme sequence "jin1 / tian1 / tian1 / qi4" corresponding to the first text sequence "today's weather" is P(jin1 / tian1 / tian1 / qi4 今天天气 ) = 0.5 + 0.2×0.98 = 0.696. Similarly, the fusion probability of the target phoneme sequence "jin1 / tian1 / tian1 / qi4" corresponding to other first text sequences (such as "today's sky paint") can also be calculated using the above formula (4), which will not be elaborated in the embodiments of this application.

[0213] In some embodiments, the following formula (5) can be used to fuse the target probability P phone-seq (i) and the third probability P translator (i e ) to obtain the probability P(i e ).

[0214] P(i e ) = λ4×P phone-seq (i) + λ3×P translator (i e ) (5)

[0215] In the above formula (5), λ4 is a fusion parameter, and the value range of λ4 is 0 ≤ λ4 ≤ 1. λ4 is obtained based on the training samples, such as the training samples for training the phoneme translator.

[0216] Combined with Figure 13 In (b), taking the target speech signal frame as the speech signal frame "qi" as an example, P phone-seq (jin1 / tian1 / tian1 / qu4) = 0.03, P translator (jin1 / tian1 / tian1 / qu4 今天天去 ) = 0.84. Assuming λ4 = 0.9, λ3 = 0.3, the fusion probability of the target phoneme sequence "jin1 / tian1 / tian1 / qi4" corresponding to the first text sequence "today's sky goes" is P(jin1 / tian1 / tian1 / qu4 今天天去)=0.9×0.03+0.3×0.84=0.279. Similarly, the above formula (5) can also be used to calculate the fusion probability of the target phoneme sequence "jin1 / tian1 / tian1 / qu4" corresponding to other first text sequences (such as "今日天趣"), which will not be repeated in the embodiments of this application.

[0217] In some embodiments, the target probability P can be calculated using the following formula (6): phone-seq (i) and the third probability P translator (i e ) to obtain the probability P(i e ).

[0218] P(i e )=λ4×P phone-seq (i)+P translator (i e ) (6)

[0219] Combination Figure 13 In (a), taking the target speech signal frame as the speech signal frame "今" as an example, P phone-seq (jing1)=0.05,P translator (jing1 经 )=0.05, assuming λ4=0.9, then the fusion probability of the target phoneme sequence “jing1” corresponding to the first text sequence “jing” is P(jing1 经 )=0.9×0.05+0.05=0.095. Similarly, the above formula (6) can also be used to calculate the fusion probability of the target phoneme sequence "jing1" corresponding to other first text sequences (such as "晶"), which will not be described in detail in the embodiment of the present application.

[0220] It should be noted that the terminal may use one or more of the above formula (4), formula (5) and formula (6) to obtain the probability of the first text sequence. Exemplarily, the above formula (4), formula (5) and formula (6) may be used alone or in combination.

[0221] In this way, the terminal can obtain a more accurate probability of the first text sequence by fusing the target probability and the third probability, thereby further improving the accuracy of speech recognition.

[0222] S606: The terminal displays a first text sequence with the highest probability.

[0223] That is to say, the terminal can determine the probabilities corresponding to the first text sequences and display a first text sequence with the highest probability.

[0224] Combination Figure 13In (a), when the first text sequence is the text sequence corresponding to the target phoneme sequence when the target speech signal frame is "jin1", the terminal sorts the first text sequence according to the probability of the first text sequence. Among them, the first text sequence with the highest probability corresponding to the target phoneme sequence "jin1" is "qi4", the first text sequence with the highest probability corresponding to the target phoneme sequence "jing1" is "jing1", and the first text sequence with the highest probability corresponding to the target phoneme sequence "jun1" is "jun1". The terminal sorts the probabilities corresponding to the first text sequences "qi4", "jing1", and "jun1", and obtains that the probability of the first text sequence "jin1" is the highest, which is 0.95. Then, the terminal displays the first text sequence "jin1".

[0225] Combined with Figure 13 In (b), when the first text sequence is the text sequence corresponding to the target phoneme sequence when the target speech signal frame is "qi4", the terminal sorts the first text sequence according to the probability of the first text sequence. Similar to the above situation when the first text sequence is the text sequence corresponding to the target phoneme sequence when the target speech signal frame is "jin1" and the terminal sorts the first text sequence according to the probability, the terminal sorts the probabilities corresponding to the first text sequences "today's weather", "today's sky goes", and "today's sky is clear", and obtains that the probability of the first text sequence "today's weather" is the highest, which is 0.696. Then, the terminal displays the first text sequence "today's weather". Similarly, the streaming recognition of the entire sentence speech stream is completed, and the streaming recognition result can be the final recognition result.

[0226] Based on Figure 6 The speech recognition method shown, the terminal obtains N target phoneme sequences with the highest target probabilities according to the probability matrix of the phonemes corresponding to the target speech signal frame and the probability matrix of the next possible phonemes of the first phoneme sequence. Then, translate the N target phoneme sequences, obtain at least one first text sequence corresponding to each of the N target phoneme sequences and the third probability of each first text sequence in the at least one first text sequence. According to the third probability and the target probability corresponding to the N target phoneme sequences, determine the first text sequence with the highest probability among the at least one first text sequence, and display the first text sequence with the highest probability, thereby improving the accuracy of speech recognition.

[0227] Exemplarily, Figure 14 is the flow diagram of the speech recognition method provided by the embodiment of the present application Figure 2 . The above S601-S606 can be used as the process of the terminal streaming display of the text sequence. When the terminal executes the above S601-S606 and streams and displays the complete text sequence corresponding to the speech stream (such as "today's weather is good"), it can also execute the following S1401-S1403. For details, see Figure 14 .

[0228] S1401, the terminal obtains the first lattice structure.

[0229] Optionally, the abscissa of the first lattice structure is each phoneme in the complete phoneme sequence corresponding to the speech stream, and the ordinate is the text probability matrix corresponding to each phoneme in the complete phoneme sequence.

[0230] Figure 15 This is the schematic diagram of the structure of the first lattice structure provided by the embodiments of the present application. Combining Figure 15 , taking the complete phoneme sequence "jin1 / tian1 / tian1 / qi4 / hao3" corresponding to the speech stream as an example, the first lattice structure includes the text probability matrix corresponding to the phoneme "jin1", the text probability matrix corresponding to the phoneme "tian1", the text probability matrix corresponding to the phoneme "tian1", the text probability matrix corresponding to the phoneme "qi4", and the text probability matrix corresponding to the phoneme "hao3".

[0231] Exemplarily, the size of the first lattice structure can be T×L e , L e is the number of text symbols in the text vocabulary, and T is the number of speech signal frames included in the speech stream or the number of phonemes in the complete phoneme sequence corresponding to the speech stream. For example, Figure 15 the size of the first lattice structure shown can be 4×L e .

[0232] In some embodiments, the terminal can decode at least one complete phoneme sequence to obtain the first lattice structure. Among them, the complete phoneme sequence can be the target phoneme sequence obtained when the target speech signal frame is the last speech signal frame in the speech stream.

[0233] Exemplarily, the speech stream can correspond to one or more complete phoneme sequences. For example, the speech stream "Today the weather is good" corresponds to the complete phoneme sequences "jin1 / tian1 / tian1 / qi4 / hao3", "jin1 / tian1 / tian1 / qi4 / huo3", "jin1 / tian1 / tian1 / qi4 / gao1", etc. The terminal can obtain the first lattice structures corresponding to the complete phoneme sequences "jin1 / tian1 / tian1 / qi4 / hao3", "jin1 / tian1 / tian1 / qi4 / huo3", "jin1 / tian1 / tian1 / qi4 / gao1" respectively.

[0234] Optionally, the terminal can decode the complete phoneme sequence corresponding to the complete text sequence with the highest probability to obtain a first lattice structure. Among them, the complete text sequence with the highest probability is the complete text sequence displayed during the streaming display process (such as "The weather is good today"). In this way, the terminal can only decode the phoneme sequence corresponding to the complete text sequence with the highest probability to reduce the complexity of speech recognition.

[0235] It should be noted that the embodiments of the present application do not limit the order of S1401 and the above S604 - S606. For example, the first lattice structure can be obtained during or after the terminal translates the target phoneme sequence, or can be obtained during or after the text sequence is displayed. The present application does not limit this.

[0236] S1402, the terminal clips the first lattice structure to obtain a second lattice structure.

[0237] In some embodiments, the terminal can use a rectifier to clip the first lattice structure to obtain a second lattice structure. Specifically, it can include the following steps three to four.

[0238] Step three, the terminal obtains a phoneme text confusion matrix M d-d and a phoneme text matrix M d-e . confusion .

[0239] Among them, the size of the confusion phoneme matrix is L d ×L d , the size of the phoneme text matrix is L d ×L e , L d is the number of phonemes in the phoneme vocabulary, and L e is the number of text symbols in the text vocabulary.

[0240] Optionally, the phoneme text confusion matrix M confusion is obtained by multiplying the confusion phoneme matrix M d-d and the phoneme text matrix M d-e , for example, using the following formula (7).

[0241] M confusion = M d-d ×M d-e (7)

[0242] In the above formula (7), the size of the phoneme text confusion matrix M confusion is L d ×L e . Combining Figure 16 in (a), the confusion phoneme matrix M d-d and the phoneme text matrix M d-eMatrix multiplication to obtain the phoneme text confusion matrix M confusion .

[0243] Step 4: The terminal is based on the phoneme text confusion matrix M confusion The first lattice structure Lattice is trimmed to obtain a second lattice structure.

[0244] Optionally, the second grid structure can be a phoneme text confusion matrix M confusion The value is obtained by multiplying the corresponding element of the first lattice structure Lattice, such as using the following formula (8).

[0245] Lattice'(t) = Lattice(t)×M confusion (d t ) (8)

[0246] In the above formula (8), Lattice'(t) represents the column data corresponding to the phoneme t in the second grid structure Lattice', Lattice(t) represents the column data corresponding to the phoneme t in the first grid structure, M confusion (d t ) represents the phoneme text confusion matrix M confusion The row data corresponding to the phoneme t in .

[0247] Combination Figure 16 In (b), the first column of the first grid structure Lattice is trimmed as an example. The first column of the first grid structure Lattice is the data corresponding to the phoneme "jin1", which is recorded as Lattice (jin1). Phoneme text confusion matrix M confusion The row data corresponding to the phoneme "jin1" in is denoted as M confusion (d jin1 ). The value of the phoneme "jin1" corresponding to the text symbol "今" in the first grid structure Lattice is 0.75, and the phoneme text confusion matrix M confusion The phoneme "jin1" in the Chinese text symbol "今" has a value of 1 ( Figure 16 Not shown in the figure, it is assumed that 1) is multiplied to obtain the value 0.75 of the phoneme "jin1" corresponding to the text symbol "今" in the second grid structure Lattice'. Similarly, the other data in the first column are calculated and the values with the calculated result of 0 are removed. Similarly, the other columns of the first grid structure Lattice are trimmed by the same method to obtain the second grid structure Lattice'. It should be noted that Figure 16 Just one example for this application.

[0248] S1403: The terminal decodes the second grid structure, obtains and displays a second text sequence.

[0249] Optionally, the second text sequence may be the same as or different from the first text sequence with the highest probability. The first text sequence is the text sequence obtained when the target speech signal frame is the last speech signal frame in the speech stream.

[0250] Exemplarily, the Viterbi algorithm is used to decode the second lattice structure by using a weighted finite state transducer (WFST) to obtain the second text sequence, which can be used as the final text sequence.

[0251] Exemplarily, the second text sequence may or may not be the same as the first text sequence. If there are easily confused speech signal frames in the speech stream, such as the speech signal frame "Zou" in the speech stream "Call Zou Jielun", this speech signal frame "Zou" is easily confused with the speech signal frame "Zhou". Through Figure 7 the method shown, the first text sequence obtained may be "Call Zhou Jielun", and through Figure 14 the method described, "Call Zhou Jielun" can be corrected to "Call Zou Jielun" to obtain the second text sequence "Call Zou Jielun".

[0252] Based on Figure 14 the speech recognition method shown, by cropping the first lattice structure, the probabilities of the text symbols in the first lattice structure can be finely adjusted. The order of the text symbols corresponding to the phonemes in the obtained second lattice structure Lattice' may be different from that of the first lattice structure. Decoding the second lattice structure to obtain the second text sequence can achieve the correction of ambiguous sounds. For example, correcting "Call Zhou Jielun" to "Call Zou Jielun" can further improve the accuracy of speech recognition.

[0253] Figure 17 This is an application schematic of the speech recognition method provided by the embodiments of the present application. Figure 9 Taking the speech stream as "The weather is good today" and the target speech signal frame as "qi" as an example, the speech recognition method provided by the embodiments of the present application will be described below.

[0254] As Figure 17 shown, the acoustic model 310 processes the target speech signal frame "qi" and the first phoneme sequence "jin1 / tian1 / tian1" to obtain the first probability matrix corresponding to the target speech signal frame "qi". The phoneme predictor 320 obtains the second probability matrix corresponding to the next possible phoneme of the first phoneme sequence "jin1 / tian1 / tian1" according to the semantics of the first phoneme sequence "jin1 / tian1 / tian1". Then, the first probability of each phoneme in the first probability matrix and the second probability of the corresponding phoneme in the second probability matrix are fused by using the fusion parameter λ1 and / or the fusion parameter λ2 to obtain the fifth probability matrix.

[0255] According to the fifth probability matrix corresponding to one or more speech signal frames before the target speech signal frame ("今", "天", "天") and the target speech signal frame "气", the three target phoneme sequences with the largest target probabilities are obtained, such as "jin1 / tian1 / tian1 / qi4", "jin1 / tian1 / tian1 / qu4", "jin1 / tian1 / tian1 / qing2" and "jin1 / tian1 / tian1 / qi4", "jin1 / tian1 / tian1 / qu4", "jin1 / tian1 / tian1 / qing2".

[0256] The three target phoneme sequences with the largest target probabilities are translated by the phoneme translator 330 to obtain at least one first text sequence corresponding to the three target phoneme sequences and the probability ( Figure 17 In the figure, only one corresponding first text sequence and the third probability corresponding to the first text sequence are shown for each target phoneme sequence). The third probability corresponding to each first text sequence in at least one first text sequence and the corresponding target probability are fused using the fusion parameter λ1 and / or the fusion parameter λ2 to obtain the probability of the first text sequence, such as P(jin1 / tian1 / tian1 / qi4 今天天气 )=0.696, the terminal displays a first text sequence with the highest probability "Today's weather". Similarly, taking the target speech signal frame as "氣", a first text sequence with the highest corresponding probability is obtained and displayed, completing the stream recognition of the entire speech stream.

[0257] Optionally, the terminal uses a confusion sound matrix M d-d and phoneme text matrix M d-e The first grid structure is cropped to obtain a second grid structure, and the Viterbi search of WFST is used to obtain a second text sequence according to the second grid structure. The second text sequence may be the same as the first text sequence or may be different. The terminal displays the second text sequence, which can correct ambiguous sounds and further improve the accuracy of speech recognition.

[0258] It is understandable that in order to implement the above functions, the electronic device includes hardware and / or software modules corresponding to the execution of each function. In combination with the algorithm steps of each example described in the embodiments disclosed herein, the present application can be implemented in the form of hardware or a combination of hardware and computer software. Whether a function is executed in the form of hardware or computer software driving hardware depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application in combination with the embodiments, but such implementation should not be considered to be beyond the scope of the present application.

[0259] In this embodiment, the functional modules of the electronic device can be divided according to the above method examples. For example, each functional module can be divided corresponding to each function, or two or more functions can be integrated into one processing module. The above integrated module can be implemented in the form of hardware. It should be noted that the division of modules in this embodiment is illustrative, only a logical function division, and there can be other division methods in actual implementation.

[0260] As Figure 18 shown, an embodiment of the present application discloses a schematic structural diagram of an electronic device. The electronic device 1800 can be used to implement the methods described in the above method embodiments. Exemplarily, the electronic device 1800 may specifically include: an acquisition unit 1801 and an output unit 1802. Among them, the acquisition unit 1801 is used to support the electronic device 1800 to execute Figure 6 the steps S601 - S605 in Figure 14 and / or, the acquisition unit 1801 is used to support the electronic device 1800 to execute Figure 6 the steps S1401 - S1403 in Figure 14 and / or, the acquisition unit 1801 is further used to support the electronic device 1800 to execute other steps executed by the electronic device in the embodiments of the present application. The output unit 1802 is used to support the electronic device 1800 to execute

[0261] the step S606 in Figure 18 and / or, the output unit 1802 is used to support the electronic device 1800 to execute Figure 6 the step S1403 in Figure 14 and / or, the output unit 1802 is further used to support the electronic device 1800 to execute other steps executed by the electronic device in the embodiments of the present application.

[0261] Optionally, Figure 18 the electronic device 1800 shown may further include a collection unit 1803, and the collection unit 1803 is used to support the electronic device 1800 to execute the steps of collecting a voice stream or a voice signal frame executed by the electronic device in the embodiments of the present application.

[0262] Optionally, Figure 18 the electronic device 1800 shown may further include a storage unit ( Figure 18 not shown in Figure 18 ), and the storage unit stores programs or instructions. When the processing unit executes the programs or instructions, the Figure 18 shown electronic device 1800 can execute Figure 6 and Figure 14 the voice recognition methods shown.

[0263] Figure 18 The technical effects of the electronic device 1800 shown can be referred to Figure 6 and Figure 14The technical effects of the voice recognition method shown are not elaborated here.

[0264] Figure 18 The acquisition unit involved in the electronic device 1800 shown can be implemented by a processor or processor-related circuit components and can be a processor or a processing module. The acquisition unit 1803 and the output unit 1802 can be combined into a transceiver unit, and the transceiver unit can be implemented by a transceiver or transceiver-related circuit components and can be a transceiver or a transceiver module.

[0265] The embodiments of the present application further provide a chip system, including: a processor, where the processor is coupled to a memory, and the memory is used to store programs or instructions. When the programs or instructions are executed by the processor, the chip system implements the methods in any of the above method embodiments.

[0266] Optionally, the processor in the chip system can be one or more. The processor can be implemented by hardware or by software. When implemented by hardware, the processor can be a logic circuit, an integrated circuit, etc. When implemented by software, the processor can be a general-purpose processor that implements by reading the software code stored in the memory.

[0267] Optionally, the memory in the chip system can also be one or more. The memory can be integrated with the processor or can be separately arranged from the processor, which is not limited in the present application. Exemplarily, the memory can be a non-transitory processor, such as a read-only memory ROM, which can be integrated with the processor on the same chip or can be separately arranged on different chips. The present application does not make specific limitations on the type of the memory and the setting manner of the memory and the processor.

[0268] Exemplarily, the chip system can be a field programmable gate array (FPGA), can be an application specific integrated circuit (ASIC), can also be a system on chip (SoC), can also be a central processor unit (CPU), can also be a network processor (NP), can also be a digital signal processing circuit (DSP), can also be a micro controller unit (MCU), can also be a programmable logic device (PLD) or other integrated chips.

[0269] It should be understood that each step in the above method embodiments can be completed by the integrated logic circuit of the hardware in the processor or the instructions in the form of software. The method steps disclosed in combination with the embodiments of the present application can be directly embodied as being executed and completed by the hardware processor, or executed and completed by the combination of the hardware and software modules in the processor.

[0270] The embodiments of the present application provide a computer-readable storage medium, on which a computer program or instruction is stored. When the computer program or instruction runs on a computer, the computer is enabled to execute the speech recognition method described in the above method embodiments.

[0271] The embodiments of the present application provide a computer program product, which includes: a computer program or instruction. When the computer program or instruction runs on a computer, the computer is enabled to execute the speech recognition method described in the above method embodiments.

[0272] In addition, the embodiments of the present application further provide a device, which may specifically be a component or a module. The device may include a processor and a memory connected to each other; wherein, the memory is used to store computer execution instructions. When the device runs, the processor may execute the computer execution instructions stored in the memory, so that the device executes the web page authentication method in the above method embodiments.

[0273] Among them, the terminal device, computer-readable storage medium, computer program product or chip provided by the embodiments of the present application are all used to execute the corresponding methods provided above. Therefore, the beneficial effects that can be achieved can refer to the beneficial effects in the corresponding methods provided above, and will not be elaborated here.

[0274] Through the description of the above embodiments, those skilled in the art can clearly understand that for the convenience and brevity of description, only the above division of each functional module is used as an example for illustration. In actual applications, the above functions can be allocated to different functional modules according to needs, that is, the internal structure of the device is divided into different functional modules to complete all or part of the functions described above. The specific working processes of the systems, devices, and units described above can refer to the corresponding processes in the foregoing method embodiments, and will not be elaborated here.

[0275] In several embodiments provided in the present application, it should be understood that the disclosed method can be implemented in other ways. For example, the terminal device embodiments described above are merely illustrative. For example, the division of the modules or units is only a logical function division. In actual implementation, there may be other division methods. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the displayed or discussed couplings or direct couplings or communication connections to each other can be through some interfaces, and the indirect couplings or communication connections of the modules or units can be in electrical, mechanical or other forms.

[0276] The units described as separate components may or may not be physically separated. The components displayed as units may or may not be physical units, that is, they can be located in one place or distributed to multiple network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0277] In addition, in each embodiment of the present application, the functional units can be integrated in a processing unit, or each unit can exist physically alone, or two or more units can be integrated in one unit. The above-mentioned integrated units can be implemented in the form of hardware or in the form of software functional units.

[0278] If the above-mentioned integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, in essence, or the part that contributes to the prior art, or all or part of this technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) or a processor to execute all or part of the steps of the method described in each embodiment of the present application. The foregoing storage medium includes: flash memory, mobile hard disk, read-only memory, random access memory, magnetic disk or optical disk and other media that can store program instructions.

[0279] The above is only the specific implementation manner of the present application, but the protection scope of the present application is not limited thereto. Any changes or substitutions within the technical scope disclosed in the present application should be covered by the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims

1. A speech recognition method, characterized in that, Including: Obtaining N target phoneme sequences with the largest target probability according to a first probability matrix and a second probability matrix; wherein, the first probability matrix includes the probabilities of each phoneme in multiple phonemes included in a phoneme vocabulary corresponding to a phoneme of a target speech signal frame, the second probability matrix includes the probabilities of each phoneme in multiple phonemes included in a phoneme vocabulary corresponding to the next possible phoneme of multiple first phoneme sequences, each first phoneme sequence in the multiple first phoneme sequences includes a starting phoneme and a phoneme sequence in multiple phoneme sequences corresponding to one or more speech signal frames before the target speech signal frame, the target probability is the probability of each target phoneme sequence, each target phoneme sequence is composed of a first phoneme sequence in the multiple first phoneme sequences and a phoneme corresponding to the target speech signal frame, N is an integer greater than or equal to 1, and the target speech signal frame is a speech signal frame to be currently recognized; Translating the N target phoneme sequences to obtain third probability matrices respectively corresponding to the N target phoneme sequences; wherein, the third probability matrix includes the probabilities that a target phoneme sequence in the N target phoneme sequences is translated into at least one first text sequence; Determining a first text sequence with the largest probability in the at least one first text sequence according to the third probability matrix and the target probabilities corresponding to the N target phoneme sequences; Displaying the first text sequence with the largest probability.

2. The voice recognition method according to claim 1, wherein Further including: Obtaining one or more of the following according to the target speech signal frame and a first phoneme sequence in the multiple first phoneme sequences: the first probability matrix, at least one second phoneme sequence, and a fourth probability matrix; wherein, each second phoneme sequence in the at least one second phoneme sequence is composed of a first phoneme sequence in the multiple first phoneme sequences and a phoneme corresponding to the target speech signal frame, the fourth probability matrix includes the probabilities corresponding to each second phoneme sequence in the at least one second phoneme sequence, and the second phoneme sequence serves as the first phoneme sequence corresponding to the next speech signal frame of the target speech signal frame.

3. The voice recognition method according to claim 1 or 2, characterized in that, Further including: Obtaining the second probability matrix according to the semantics of the first phoneme sequence; wherein, the semantics of the first phoneme sequence represents the language meaning expressed by the first phoneme sequence.

4. The voice recognition method according to any one of claims 1-3, characterized in that, The step of obtaining N target phoneme sequences with the largest target probability according to the first probability matrix and the second probability matrix includes: Fusing the first probability matrix and the second probability matrix to obtain a fifth probability matrix; wherein, the fifth probability matrix includes the fusion probabilities of each phoneme in multiple phonemes included in a phoneme vocabulary corresponding to a phoneme of the target speech signal frame; Obtaining N target phoneme sequences with the largest target probability according to the fifth probability matrices respectively corresponding to one or more speech signal frames before the target speech signal frame and the target speech signal frame.

5. The voice recognition method according to any one of claims 1-4, characterized in that, Further including: Translate one or more phonemes included in the phoneme vocabulary corresponding to the target speech signal frame to obtain a text probability matrix corresponding to each of the one or more phonemes; wherein, the text probability matrix includes the probability that the corresponding phoneme is translated into each text symbol included in the text vocabulary.

6. The voice recognition method according to claim 5, wherein Further included: Obtain a first grid structure; wherein, the abscissa of the first grid structure is each phoneme in the complete phoneme sequence, the ordinate is the text probability matrix corresponding to each phoneme in the complete phoneme sequence, and the complete phoneme sequence is the target phoneme sequence obtained when the last speech signal frame in the speech stream is used as the target speech signal frame.

7. The voice recognition method according to claim 6, characterized in that, Further included: Crop the first grid structure to obtain a second grid structure; Decode the second grid structure to obtain and display a second text sequence.

8. The speech recognition method according to claim 7, wherein The step of cropping the first grid structure to obtain a second grid structure includes: Based on the confusion phoneme matrix and the phoneme text matrix, obtain a phoneme text confusion matrix; wherein, the confusion phoneme matrix is used to represent the probability that each phoneme included in the phoneme vocabulary is recognized as each phoneme included in the phoneme vocabulary, and the phoneme text matrix is used to represent the correspondence between each text symbol included in the text vocabulary and each phoneme included in the phoneme vocabulary; Crop the first grid structure based on the phoneme text confusion matrix to obtain the second grid structure.

9. An electronic device, characterized in that, Including: A processor, a memory, and a display screen, the memory and the display screen are coupled to the processor, the memory is used to store computer program code, the computer program code includes computer instructions, and when the processor reads the computer instructions from the memory, the electronic device executes the speech recognition method according to any one of claims 1-8.

10. A computer-readable storage medium, characterized in that, A computer program or instruction is stored on the computer-readable storage medium, and when the computer program or instruction runs on the computer, the computer executes the speech recognition method according to any one of claims 1-8.

11. A computer program product, characterized in that, The computer program product includes: a computer program or instruction, and when the computer program or instruction runs on the computer, the computer executes the speech recognition method according to any one of claims 1-8.

Images