Speech recognition method and electronic device

Abstract

Embodiments of the present application provide a speech recognition method and an electronic device. The method is performed by the electronic device and includes: initializing a speech recognition model in a case where a first service function is started, wherein the speech recognition model is used to recognize characters corresponding to speech information and categories to which the characters belong; receiving speech information input through the first service function, and inputting the speech information and cached categories to which the characters belong into the speech recognition model to obtain a speech recognition result, wherein the cached categories to which the characters belong include categories obtained by recognizing historical speech information. The method can improve the accuracy of the speech recognition result.

Description

Technical Field

[0001] This application relates to the field of electronic technology, specifically to a speech recognition method and an electronic device. Background Technology

[0002] With the continuous development of electronic devices, their rich functionality has greatly facilitated users' lives. For example, when the voice assistant function is enabled on an electronic device, users can have voice conversations with it, and the device can respond to the user's voice commands and convert them into text for display. Another example is when the "see and speak" function is enabled on an electronic device, allowing users to control various operations using their voice.

[0003] In the aforementioned scenarios, electronic devices all need to use automatic speech recognition (ASR) to identify voice information and thus recognize the corresponding voice commands. However, the accuracy of speech recognition in related technologies still needs improvement. Summary of the Invention

[0004] This application provides a speech recognition method and an electronic device that can improve the accuracy of speech recognition results.

[0005] In a first aspect, this application provides a speech recognition method, which is executed by an electronic device and includes: initializing a speech recognition model when a first service function is enabled, wherein the speech recognition model is used to recognize characters corresponding to speech information and the categories to which the characters belong; receiving speech information input through the first service function, and inputting the speech information and cached character categories into the speech recognition model to obtain a speech recognition result, wherein the cached character categories include categories obtained by recognizing historical speech information.

[0006] The first business function may include, but is not limited to, voice assistant, "see and speak" functionality, and contextualized commands; that is, different business functions correspond to different business scenarios. The process of initializing the speech recognition model on the electronic device can be a process of loading configuration parameters, that is, loading the configuration parameters obtained during the training of the speech recognition model. Then, the electronic device can input the received voice information and the categories of the cached characters into the speech recognition model to determine the speech recognition result.

[0007] In the above implementation, because the speech recognition model takes into account the category of the cached characters when recognizing speech information, the output results can be closer to the potential characters of that category, making it more suitable for the current business scenario and improving the accuracy of speech recognition results.

[0008] In conjunction with the first aspect, in some implementations of the first aspect, the above-mentioned inputting the speech information and the category of the cached characters into the speech recognition model to obtain the speech recognition result includes: inputting the speech information, the cached recognition result and the category of the cached characters into the speech recognition model to obtain the fusion probability distribution, wherein the cached recognition result includes the text information obtained by recognizing historical speech information; and determining the speech recognition result based on the fusion probability distribution.

[0009] In addition to considering the categories of voice information and cached characters, the speech recognition model can also take into account the cached recognition results, which are the text information obtained by recognizing historical voice information, i.e., the characters that have already been recognized. Therefore, when processing, the speech recognition model can predict the probability of the next character based on the characters that have already been recognized, so as to further improve the accuracy of the speech recognition results.

[0010] In conjunction with the first aspect, in some implementations of the first aspect, the speech recognition model includes a first model and a second model. The above-mentioned inputting speech information, cached recognition results, and the category of cached characters into the speech recognition model to obtain a fusion probability distribution includes: inputting speech information into the first model to obtain a first probability distribution, the first probability distribution representing the probability distribution of the characters to which the speech information belongs; inputting the cached recognition results and the category of cached characters into the second model to obtain a second probability distribution, the second probability distribution representing the probability distribution of the next character predicted by the second model for the cached recognition results; and determining the fusion probability distribution based on the first probability distribution and the second probability distribution.

[0011] The first model can be an acoustic model, and the second model can be a language model. Speech recognition can be achieved through the combined effect of these two models.

[0012] Optionally, when the speech recognition model includes a first model and a second model, the electronic device can input the speech information (or the audio features corresponding to the speech information) into the first model and output a first probability distribution. For example, for the speech "da", the first model can output the probability that it belongs to the word "da" as a, the probability that it belongs to the word "da" as b, the probability that it belongs to the word "da" as c, etc., based on the audio features of the user's speech "da". The sum of each probability is equal to 1.

[0013] In addition, the electronic device can input the historical cache recognition results and the category to which the cached characters belong into the language model, and output the predicted probability distribution of the next character (i.e., the second probability distribution). For example, in the scenario of a voice assistant, the voice command input by the user is "turn on the flashlight". When it comes to the recognition process of the character "开", it means that the character "打" and the category to which the character "打" belongs have been recognized previously. The predicted probability distribution of the next character can be: the probability of belonging to the character "开" is d, the probability of belonging to the character "凯" is e, the probability of belonging to the character "楷" is f, and so on.

[0014] Then, the electronic device can determine the fusion probability distribution based on the first probability distribution and the second probability distribution. Optionally, the electronic device can perform weighted summation on the first probability distribution and the second probability distribution to obtain the fusion probability distribution.

[0015] In the above implementation manner, the electronic device processes the voice information through the first model and predicts the characters that have been recognized historically through the second model to jointly determine the character corresponding to the current voice information, improving the accuracy of the voice recognition result.

[0016] Combined with the first aspect, in some implementation manners of the first aspect, determining the voice recognition result based on the fusion probability distribution includes: decoding the fusion probability distribution in the manner of beam search to obtain the voice recognition result.

[0017] In one implementation manner, assuming the number of beams is B, the electronic device can select K characters corresponding to the fusion probability distribution for each beam to generate a candidate character queue of B×K. The K characters are the characters corresponding to the first K probabilities after the fusion probability distribution is sorted in descending order of probability (i.e., the TopK of the probability). Then, B characters are selected from the B×K candidate character queues. The B characters are the characters corresponding to the first B probabilities after the B×K candidate character queues are sorted in descending order of probability (i.e., the TopB of the probability). Finally, the character with the largest probability among the B characters (i.e., the Top1 of the probability) is used as the voice recognition result. Optionally, the generated B characters can be added to the above cache recognition results as the input data of the language model in the next round. Optionally, the number of beams B can be determined by the B characters selected in the previous round.

[0018] Combined with the first aspect, in some implementation manners of the first aspect, the second model also outputs a third probability distribution, and the third probability distribution represents the probability distribution of the category to which the next character predicted by the second model belongs.

[0019] For the second model mentioned above, since its input data consists of historical cached recognition results and the category to which the cached characters belong, the second model can also output a third probability distribution, that is, the probability distribution of the predicted category to which the next character belongs, so that the next round of prediction can be made based on the probability distribution obtained this time.

[0020] In some implementations, the electronic device can use beam search to decode the third probability distribution, determine the categories corresponding to the B characters, and add the categories corresponding to the B characters to the cache as input data for the second model in the next round.

[0021] Optionally, assuming the number of beams is B, for the third probability distribution, the top-ranked beam with the highest probability is selected, generating B×1 candidate character category queues. Then, the categories corresponding to the aforementioned B candidate character queues are searched within these B×1 queues, generating B candidate character category queues. These B queues are then added to the previously recorded character categories. This continuous enrichment of the cached character categories can provide assistance during speech recognition, thereby improving the accuracy of the speech recognition results.

[0022] In conjunction with the first aspect, in some implementations of the first aspect, the first model is the RNNT acoustic model, which includes an encoding module, a prediction module, and a joint module. The encoding module is used to encode the audio features corresponding to the speech information, the prediction module is used to process the text features corresponding to the speech information, and the joint module is used to integrate the outputs of the encoding module and the prediction module to obtain the first probability distribution.

[0023] In conjunction with the first aspect, in some implementations of the first aspect, the second model is a Transformer network structure, a Long Short-Term Memory (LSTM) network structure, or a convolutional network structure.

[0024] The language model can receive character sequences and character category sequences as input, generate latent features of characters and character categories through character embedding layers and character category embedding layers respectively, and add the two as the latent features of the current character, which are then fed into the language model for prediction calculation. Finally, it outputs the probability distribution of the next character and the probability distribution of the category to which the next character belongs.

[0025] In the training process of a language model, training data can consist of short sentences, which may include words (entity words) of different categories. This training data can be constructed through template matching, model-assisted generation, and other methods. The language model proposed in this application can predict the probability of the next character's category based on historical recognition results, conforming to language habits in real-world scenarios and possessing good application potential.

[0026] Secondly, this application provides an apparatus included in an electronic device, which has the function of implementing the behaviors of the electronic device in the first aspect and possible implementations thereof. The function can be implemented by hardware or by hardware executing corresponding software. The hardware or software includes one or more modules or units corresponding to the above-described functions. For example, a receiving module or unit, a processing module or unit, etc.

[0027] Thirdly, this application provides an electronic device, which includes a processor, a memory, and an interface; the processor, memory, and interface cooperate with each other to enable the electronic device to execute any one of the methods in the first aspect of the technical solution.

[0028] Fourthly, this application provides a chip including a processor. The processor is used to read and execute a computer program stored in a memory to perform the methods in the first aspect and any possible implementation thereof.

[0029] Optionally, the chip may also include a memory, which is connected to the processor via a circuit or wire.

[0030] Alternatively, the chip may also include a communication interface.

[0031] Fifthly, this application provides a computer-readable storage medium storing a computer program, which, when executed by a processor, causes the processor to perform any one of the methods in the first aspect of the technical solution.

[0032] Sixthly, this application provides a computer program product, which includes computer program code that, when executed on an electronic device, causes the electronic device to perform any one of the methods in the first aspect of the technical solution. Attached Figure Description

[0033] Figure 1 This is a schematic diagram of an example voice assistant scenario provided in an embodiment of this application;

[0034] Figure 2 This is a schematic diagram illustrating the setup process of a voice assistant function provided in an embodiment of this application;

[0035] Figure 3 This is a schematic diagram of an example of a visible and describable scenario provided in an embodiment of this application;

[0036] Figure 4 This is a schematic diagram illustrating the setup process of an example of the "see and speak" function provided in an embodiment of this application;

[0037] Figure 5 This is a schematic diagram of an example of a scenario-based instruction scenario provided in an embodiment of this application;

[0038] Figure 6 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application;

[0039] Figure 7 This is a software structure block diagram of an electronic device provided in an embodiment of this application;

[0040] Figure 8 This is a flowchart illustrating an example of a speech recognition method provided in an embodiment of this application;

[0041] Figure 9 This is a schematic diagram of the structure of an acoustic model provided in an embodiment of this application;

[0042] Figure 10 This is a schematic diagram illustrating the input and output method of an example language model provided in an embodiment of this application;

[0043] Figure 11 This is a schematic diagram of the processing procedure of an example speech recognition method provided in an embodiment of this application;

[0044] Figure 12 This is a schematic diagram illustrating an example of a beam search method provided in an embodiment of this application;

[0045] Figure 13 This is a flowchart illustrating another example of a speech recognition method provided in an embodiment of this application;

[0046] Figure 14 This is a schematic diagram of the structure of an example internal language model provided in an embodiment of this application;

[0047] Figure 15 This is a schematic diagram of the structure of an example external language model provided in an embodiment of this application;

[0048] Figure 16 This is an example diagram illustrating the generation of multiple business texts based on a prompt, provided in an embodiment of this application.

[0049] Figure 17 This is a schematic diagram illustrating the processing procedure of another speech recognition method provided in this application embodiment. Detailed Implementation

[0050] The technical solutions of the embodiments of this application will be described below with reference to the accompanying drawings. In the description of the embodiments of this application, unless otherwise stated, " / " means "or," for example, A / B can mean A or B; "and / or" in this text is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, and B existing alone. Furthermore, in the description of the embodiments of this application, "multiple" refers to two or more than two.

[0051] Hereinafter, the terms "first," "second," and "third" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first," "second," or "third" may explicitly or implicitly include one or more of that feature.

[0052] To facilitate a clear description of the technical solutions in the embodiments of this application, some terms and technologies involved in the embodiments of this application are briefly introduced below:

[0053] (1) Voice Assistant: This is the name of a voice control function. Users can turn the voice assistant on or off on electronic devices using preset switches. After the voice assistant function is turned on, users need to wake it up first, such as by using a wake word or the power button. After the voice assistant on the electronic device is woken up, users can then speak voice commands to control the electronic device to perform corresponding service operations.

[0054] (2) Visible and Talkable: This is another voice control function. Users can turn the Visible and Talkable function on or off via a preset switch on their electronic devices. Unlike voice assistants, when the Visible and Talkable function is turned on, after the fence on which it depends is triggered, the electronic device enters a continuous sound reception state (or long sound reception state). The microphone of the electronic device will continuously collect ambient sound. Users do not need to trigger a wake-up operation; they can directly speak voice commands to control the electronic device to perform corresponding service operations. After the service operation is performed, the electronic device does not turn off the sound reception. Here, the fence mainly refers to the fence of scene events, such as the fence of alarm clock scene events, the fence of telephone scene events, etc.

[0055] (3) Contextualized commands: This is another type of voice control function. Users can turn the contextualized command function on or off on electronic devices via preset switches. Unlike voice assistants and the "see and speak" function, when the contextualized command function is turned on, the electronic device enters a short-tone reception state when a specific scenario (e.g., an alarm clock rings) is triggered. Users can then directly speak voice commands to control the electronic device to perform the corresponding service operation (e.g., turn off the alarm clock). After the specific scenario ends (e.g., the alarm clock turns off), the electronic device turns off the reception.

[0056] With the continuous development of electronic devices, their rich functions have greatly facilitated users' lives. For example, when an electronic device has its voice assistant function enabled and activated, the user can engage in voice conversations with the device. If the user inputs a voice sentence, the electronic device can respond with a voice message. For instance, if the user inputs "What's the weather like today?", the electronic device can reply with "Today's lowest temperature is 15 degrees Celsius, and the highest temperature is 25 degrees Celsius." Alternatively, the user can also control the electronic device with voice; for example, the user can input... Electronic devices can respond with "OK" via voice and then enter... Interface. See also Figure 1 The voice input interface shown allows the electronic device to display the corresponding text when a user inputs "What's the weather like today?". After recognizing the voice, the device can query today's weather information online in the background and convert the corresponding text into speech for output, such as "Today's low temperature is 15 degrees Celsius, and the high temperature is 25 degrees Celsius." In other words, the electronic device needs to recognize the user's voice command and display the corresponding text information on the current interface.

[0057] The user setup process for enabling the voice assistant function may include, for example, such as... Figure 2 As shown, clicking the settings icon on the desktop interface of an electronic device allows the device to enter its settings interface. This interface includes various function options, such as WLAN, Bluetooth, display and brightness, and smart assistant options. Clicking the smart assistant option control 21 redirects the device to the smart assistant settings interface. The smart assistant settings interface includes various smart functions, such as YOYO suggestions, the negative one screen, smart text, smart search, and smart voice. Clicking the smart voice option control 22 redirects the device to the smart voice settings interface. The smart voice settings interface includes different wake-up methods for the voice assistant, such as voice wake-up, power button wake-up, breath wake-up, headphone in-line control wake-up, and Bluetooth device wake-up. Users can choose the appropriate wake-up method to activate the voice assistant. For example, if voice wake-up is selected, the user can later activate the voice assistant by voice inputting "Hello, YOYO"; if power button wake-up is selected, the user can later activate the voice assistant by pressing the power button for one second. It is understood that once a user selects a wake-up method in the smart voice settings interface, the voice assistant function is activated. Subsequently, the voice recognition method of this application embodiment can be used to recognize the user's input voice information and convert the voice information into text information to be displayed on the interface.

[0058] As another example, when the electronic device has the "see-and-say" function enabled, the electronic device can display a prompt icon, such as... Figure 3 As shown, after the "See and Speak" switch is turned on, the status bar of the electronic device's display shows a prompt icon 31, indicating that the "See and Speak" function is enabled. Once enabled, the corresponding recording channel for "See and Speak" is activated in the electronic device's system and drivers, and the electronic device enters a "long-receive" state, continuously collecting ambient sound. Users can issue commands to the electronic device at any time via voice, without needing to input a wake-up word or other wake-up methods. For example, if a user inputs the voice command "Open video," the electronic device will respond by launching the video application and displaying its user interface.

[0059] The user settings process for enabling the "visible and speak" feature may include, for example, such as... Figure 4 As shown, clicking the settings icon on the desktop interface of an electronic device allows the device to enter its settings interface. This interface includes various function options, such as WLAN, Bluetooth, display and brightness, and smart assistant options. Clicking the smart assistant option control 41 redirects the device to the smart assistant settings interface. The smart assistant settings interface includes various smart functions, such as YOYO suggestions, a negative one screen, smart text, smart search, and voice control. Clicking the voice control option control 42 redirects the device to the voice control settings interface. The voice control settings interface includes a "Visible and Talkable" control 43. Clicking this control redirects the device to the "Visible and Talkable" settings interface. Optionally, the "Visible and Talkable" settings interface includes a first prompt information area 44 to instruct the user on how to use the "Visible and Talkable" function. The interface also includes a "Visible and Talkable" toggle control 45. Users can click the toggle control 45 to turn the "Visible and Talkable" function on or off. In one example, the electronic device receives a click on the "Visible and Talkable" toggle control 45, and the "Visible and Talkable" function is activated. Optionally, at this time, the "See and Say" settings interface displays a second prompt information area 46 to notify the user that the "See and Say" function has been successfully enabled.

[0060] For example, when the electronic device has its contextual command function enabled, it enters a short-tone recording state when a specific event is triggered. The device then shuts off the recording after the event ends. For instance, when an alarm clock or telephone rings, the electronic device enters this short-tone recording state, and its microphone shuts off after the alarm is turned off or the call is answered. Figure 5As shown, after the contextual command function is turned on, when the alarm rings, the electronic device displays the main interface 51, a prompt window 52 pops up at the top of the main interface 51, and a prompt message 53 is displayed below the prompt window 52. The user can then speak the voice command "Turn off the alarm" according to this prompt message, and the electronic device will then turn off the alarm. The process of turning on the contextual command function can be found above. Figure 4 The process shown includes a contextual command control in the voice control settings interface, which can be used to enable contextual command functionality, and will not be elaborated further here.

[0061] In related technologies, electronic devices typically use the same speech recognition model (or engine) to recognize user-inputted speech information. However, user-inputted speech information may include different categories of entity words. For example, voice assistant scenarios often involve short, instructive voice commands such as control commands, navigation, and encyclopedia questions. The characters or phrases in these short voice commands generally correspond to specific categories, such as singer names, song titles, movie / TV show titles, city / location names, and app names. If speech recognition is based solely on audio features without considering the categories of the characters in the speech information, it may result in low accuracy.

[0062] In view of this, embodiments of this application provide a speech recognition method, proposing a speech recognition model that considers the category to which characters belong in speech information. This model can predict the category to which the next character belongs based on historical categories, making the recognition result closer to the potential characters of that category, thereby improving the accuracy of speech recognition.

[0063] It is also understood that the speech recognition method provided in this application embodiment can be applied to electronic devices with speech recognition functions or services, such as mobile phones, tablets, wearable devices, in-vehicle devices, augmented reality (AR) / virtual reality (VR) devices, laptops, ultra-mobile personal computers (UMPCs), netbooks, and personal digital assistants (PDAs). This application embodiment does not impose any restrictions on the specific type of electronic device.

[0064] For example, Figure 6This is a schematic diagram of the structure of an electronic device 100 provided in an embodiment of this application. Taking a mobile phone as an example, the electronic device 100 may include a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, a charging management module 140, a power management module 141, a battery 142, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, a headphone jack 170D, a sensor module 180, buttons 190, a motor 191, an indicator 192, a camera 193, a display screen 194, and a subscriber identification module (SIM) card interface 195, etc. The sensor module 180 may include a pressure sensor 180A, a gyroscope sensor 180B, a barometric pressure sensor 180C, a magnetic sensor 180D, an accelerometer sensor 180E, a distance sensor 180F, a proximity sensor 180G, a fingerprint sensor 180H, a temperature sensor 180J, a touch sensor 180K, an ambient light sensor 180L, a bone conduction sensor 180M, etc.

[0065] Processor 110 may include one or more processing units, such as: application processor (AP), modem processor, graphics processing unit (GPU), image signal processor (ISP), controller, memory, video codec, digital signal processor (DSP), baseband processor, and / or neural network processing unit (NPU), etc. Different processing units may be independent devices or integrated into one or more processors.

[0066] The processor 110 may also include a memory for storing instructions and data. In some embodiments, the memory in the processor 110 is a cache memory. This memory can store instructions or data that the processor 110 has just used or that are used repeatedly. If the processor 110 needs to use the instruction or data again, it can retrieve it directly from the memory. This avoids repeated accesses, reduces the waiting time of the processor 110, and thus improves the efficiency of the system.

[0067] In some embodiments, the processor 110 may include one or more interfaces. Interfaces may include an inter-integrated circuit (I2C) interface, an inter-integrated circuit sound (I2S) interface, a pulse code modulation (PCM) interface, a universal asynchronous receiver / transmitter (UART) interface, a mobile industry processor interface (MIPI), a general-purpose input / output (GPIO) interface, a subscriber identity module (SIM) interface, and / or a universal serial bus (USB) interface, etc.

[0068] The wireless communication function of electronic device 100 can be realized through antenna 1, antenna 2, mobile communication module 150, wireless communication module 160, modem processor and baseband processor, etc.

[0069] The mobile communication module 150 can provide solutions for wireless communication, including 2G / 3G / 4G / 5G, applied to the electronic device 100. The mobile communication module 150 may include at least one filter, switch, power amplifier, low noise amplifier (LNA), etc. The mobile communication module 150 can receive electromagnetic waves via antenna 1, and perform filtering, amplification, and other processing on the received electromagnetic waves before transmitting them to a modem processor for demodulation. The mobile communication module 150 can also amplify the signal modulated by the modem processor and convert it into electromagnetic waves for radiation via antenna 1. In some embodiments, at least some functional modules of the mobile communication module 150 may be housed in the processor 110. In some embodiments, at least some functional modules of the mobile communication module 150 and at least some modules of the processor 110 may be housed in the same device.

[0070] Electronic device 100 implements display functions through a GPU, a display screen 194, and an application processor. The GPU is a microprocessor for image processing, connected to the display screen 194 and the application processor. The GPU is used to perform mathematical and geometric calculations and for graphics rendering. Processor 110 may include one or more GPUs, which execute program instructions to generate or modify display information.

[0071] Display screen 194 is used to display images, videos, etc. Display screen 194 includes a display panel. The display panel may be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (FLED), a miniature LED, a microLED, a quantum dot light-emitting diode (QLED), etc. In some embodiments, electronic device 100 may include one or N displays 194, where N is a positive integer greater than 1.

[0072] Electronic device 100 can perform shooting functions through ISP, camera 193, video codec, GPU, display 194 and application processor.

[0073] An NPU (Neural Processing Unit) is a neural network (NN) computing processor that, by borrowing from the structure of biological neural networks, such as the transmission patterns between neurons in the human brain, rapidly processes input information and can continuously learn on its own. NPUs can enable intelligent cognitive applications in electronic devices, such as image recognition, facial recognition, speech recognition, and text understanding. In this embodiment, the electronic device can call the NPU to implement the functions of a speech recognition model to complete the speech recognition process and obtain the speech recognition result.

[0074] Internal memory 121 can be used to store computer executable program code, which includes instructions. Processor 110 executes various functional applications and data processing of electronic device 100 by running the instructions stored in internal memory 121. Internal memory 121 may include a program storage area and a data storage area. The program storage area may store the operating system, at least one application program required for a function (such as sound playback, image playback, etc.), etc. The data storage area may store data created during the use of electronic device 100 (such as audio data, phonebook, etc.). Furthermore, internal memory 121 may include high-speed random access memory and may also include non-volatile memory, such as at least one disk storage device, flash memory device, universal flash storage (UFS), etc.

[0075] Electronic device 100 can implement audio functions, such as music playback and recording, through audio module 170, speaker 170A, receiver 170B, microphone 170C, headphone jack 170D, and application processor.

[0076] Microphone 170C, also known as a "microphone" or "voice transducer," is used to convert sound signals into electrical signals. When making a phone call or sending a voice message, the user can speak by bringing their mouth close to microphone 170C, inputting the sound signal into microphone 170C. Electronic device 100 may have at least one microphone 170C. In some embodiments, electronic device 100 may have two microphones 170C, which, in addition to collecting sound signals, can also perform noise reduction. In other embodiments, electronic device 100 may also have three, four, or more microphones 170C, which can collect sound signals, reduce noise, identify the sound source, and perform directional recording, etc. In the embodiments of this application, for example, in a voice call scenario, the user can input voice commands through the microphone to input voice information.

[0077] It is understood that the structures illustrated in the embodiments of this application do not constitute a specific limitation on the electronic device 100. In other embodiments of this application, the electronic device 100 may include more or fewer components than illustrated, or combine some components, or split some components, or have different component arrangements. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

[0078] It is understandable that the above Figure 6 The illustration shown is merely an example when the electronic device is a mobile phone. If the electronic device is a tablet, PC, PDA, wearable device, or other types of device, the structure of the electronic device may include more... Figure 6 The fewer structures shown can also include more than Figure 6 The structures shown are not limited here.

[0079] The software system of electronic device 100 and the process of cooperation between the software system and the hardware are described below.

[0080] The software system of electronic device 100 can adopt a layered architecture, event-driven architecture, microkernel architecture, microservice architecture, or cloud architecture. This application embodiment uses the layered architecture Android system as an example to exemplify the software structure of electronic device 100.

[0081] Figure 7This is a software structure block diagram of an electronic device 100 according to an embodiment of this application. The layered architecture divides the software into several layers, each with a clear role and function. Layers communicate with each other through software interfaces. In some embodiments, the Android system is divided into four layers, from top to bottom: the application layer, the application framework layer, the Android runtime and system libraries, and the kernel layer. The application layer may include a series of application packages, such as Android application packages (APKs).

[0082] like Figure 7 As shown, the application package may include applications such as voice assistants and "see and speak" applications, and may also include an APK that integrates a speech recognition model. This APK with integrated speech recognition model can realize speech recognition function, and different applications can call the APK with integrated speech recognition model when needed to convert speech information into text information.

[0083] In addition, the application package may also include applications such as camera, gallery, calendar, call, map, navigation, WLAN, Bluetooth, music, video, and SMS. Figure 7 Not shown in the image.

[0084] The application framework layer provides application programming interfaces (APIs) and a programming framework for applications in the application layer. The application framework layer includes some predefined functions.

[0085] like Figure 7 As shown, the application framework layer may include a window manager, content provider, view system, phone manager, resource manager, notification manager, etc.

[0086] The window manager manages windowed applications. It can retrieve screen size, determine the presence of a status bar, lock the screen, and capture screenshots. The content provider stores and retrieves data, making it accessible to applications. The view system includes visual controls, such as controls for displaying text and images. The view system can be used to build applications. The phone manager provides communication functionality for electronic devices, such as managing call status (including connection and disconnection). The resource manager provides applications with various resources, such as localized strings, icons, images, layout files, video files, etc. The notification manager allows applications to display notifications in the status bar, conveying informational messages. Notifications can appear as icons or scrollbar text in the top status bar, or as dialog boxes on the screen.

[0087] The Android runtime consists of core libraries and a virtual machine. The Android runtime is responsible for scheduling and managing the Android system.

[0088] The core library consists of two parts: one part is the functionalities that need to be called by the Java language, and the other part is the Android core library.

[0089] The application layer and application framework layer run in a virtual machine. The virtual machine executes the Java files of the application layer and application framework layer as binary files. The virtual machine is used to perform functions such as object lifecycle management, stack management, thread management, security and exception management, and garbage collection.

[0090] System libraries can include multiple functional modules. For example: surface manager, media libraries, 3D graphics processing libraries (e.g., OpenGL ES), 2D graphics engines (e.g., SGL), etc.

[0091] The Surface Manager manages the display subsystem and provides fusion of 2D and 3D layers for multiple applications. The Media Library supports playback and recording of various common audio and video formats, as well as still image files. The Media Library supports multiple audio and video encoding formats. The 3D Graphics Processing Library implements 3D graphics drawing, image rendering, compositing, and layer processing. The 2D Graphics Engine is the drawing engine for 2D graphics.

[0092] The kernel layer is the layer between hardware and software. The kernel layer contains at least the display driver, camera driver, audio driver, and sensor driver.

[0093] For ease of understanding, the following embodiments of this application will be described using the following methods: Figure 6 and Figure 7 Taking the electronic device with the structure shown as an example, the speech recognition method provided in this application embodiment will be specifically described in conjunction with the accompanying drawings and application scenarios.

[0094] Figure 8 This is a flowchart illustrating an example of a speech recognition method provided in an embodiment of this application. This method can be executed by an electronic device and may specifically include:

[0095] S101 initializes the speech recognition model when the first business function is enabled.

[0096] The first business function may include, but is not limited to, voice assistant, "see and speak" functionality, and contextualized commands, meaning different business functions correspond to different business scenarios. The process of initializing the speech recognition model on the electronic device can be a process of loading configuration parameters, that is, loading the configuration parameters obtained during the training of the speech recognition model to prepare for subsequent data processing. The training process of the speech recognition model is detailed in the following embodiments. The trained speech recognition model can recognize speech information based on the speech information and the category to which the characters belong.

[0097] In some embodiments, a speech recognition model may include multiple models. Optionally, the speech recognition model may include an acoustic model and a language model, which work together to achieve speech recognition. Of course, this application does not limit the specific number and type of models.

[0098] Optionally, the acoustic model can be an RNNT (recurrent neural network transducer) acoustic model. For example, the structure of an RNNT acoustic model can be as follows: Figure 9 As shown, it includes, but is not limited to, an encoder module, a predictor module, and a joint module. After receiving the user's input voice information, the encoder module can be used to encode the audio features corresponding to the voice information, the predictor module can be used to predict the text features corresponding to the voice information, and the joint module can be used to integrate the outputs of the encoder module and the predictor module to output an acoustic probability distribution.

[0099] The input to a language model is a character (or character feature) and a character category. The output is the predicted probability of the next character and the probability of the category corresponding to the next character. For example, the input and output of a language model can be as follows: Figure 10 As shown, the language model can receive character sequences and character category sequences as input. It generates latent features for characters and character categories through character embedding layers and character category embedding layers, respectively. These two features are then added together to form the latent feature of the current character, which is fed into the language model for prediction calculation. Finally, it outputs the probability distribution of the next character and the probability distribution of the next character's category, respectively. Optionally, the language model can be a Transformer network structure, a long short-term memory (LSTM) network structure, a convolutional network structure, etc.

[0100] Optionally, when the first business function is enabled, the application corresponding to the first business function can call the speech recognition model and read the corresponding configuration parameters to initialize the speech recognition model.

[0101] S102. Input the received voice information and the category of the cached characters into a voice recognition model to obtain a voice recognition result.

[0102] Among them, the voice information received by the electronic device may include voice commands input by the user in a voice assistant scenario, voice commands input by the user in a speakable scene, etc. The category of the cached characters may include the category obtained by recognizing historical voice information. After the electronic device inputs the voice information and the category of the cached characters into the voice recognition model, through the processing of the voice recognition model, a voice recognition result can be obtained.

[0103] Optionally, when the voice recognition model includes an acoustic model and a language model, when the electronic device inputs the received voice information and the category of the cached characters into the voice recognition model, the acoustic model and the language model need to perform corresponding processing processes respectively.

[0104] Exemplarily, as Figure 11 shown, when the electronic device inputs the voice information (or the audio features corresponding to the voice information) into the acoustic model, an acoustic probability distribution (abbreviated as the first probability distribution) can be output. Here, the acoustic probability distribution can be the probability of the character to which the voice belongs. For example, in a voice assistant scenario, if the voice command input by the user is "turn on the flashlight", then the electronic device can input the voices corresponding to "turn on", "hand", "electric", and "flashlight" into the acoustic model in sequence to obtain the acoustic probability distribution of each character. Optionally, the acoustic probability distribution can be understood as a vocabulary table, which includes the characters that the voice may correspond to and the probability of each character. Finally, the character with the highest probability can be selected as the recognition result. For example, for the voice of the character "da", the acoustic model outputs the probability that it belongs to the character "hit" as a, the probability that it belongs to the character "big" as b, the probability that it belongs to the character "answer" as c, etc. Among them, the sum of the probabilities is equal to 1. Another example is for the voice of the character "kai", the acoustic model can also output the probabilities that it belongs to the characters "open", "kai", and "kai" respectively. Optionally, since the character "open" has the character "hit" in front, in addition to considering the probability of the character "open" alone, the acoustic model can also combine the probability of the character "hit" to jointly determine the probability of the character to which the voice "turn on" belongs.

[0105] Continue to refer to Figure 11, optionally, the electronic device may input the historical cache recognition result and the category to which the character belongs into the language model, and may output the predicted probability distribution of the next character (abbreviated as the second probability distribution) and the probability distribution of the category to which the next character belongs (abbreviated as the third probability distribution). The cache recognition result input here may be the recognized text information (or text features, hidden features, etc. corresponding to the text information). For example, in the voice assistant scenario, the voice command input by the user is "turn on the flashlight". When the recognition process of the character "开" is in progress, it means that the character "打" has been recognized before. Then the historical cache recognition result is the character "打" or the text feature corresponding to the character "打". The predicted probability distribution of the next character may be: the probability of belonging to the character "开" is d, the probability of belonging to the character "凯" is e, the probability of belonging to the character "楷" is f, and so on. The category to which the character belongs input here is the category to which the recognized character belongs. For example, the category to which the recognized character "打" belongs is 0. The predicted probability distribution of the category to which the next character belongs may be: the probability of belonging to category 0 is x, the probability of belonging to category 1 is y, the probability of belonging to category 2 is z, and so on. It can be understood that in the embodiment of the present application, the probability distribution of the next character and the probability distribution of the category to which it belongs are predicted by the language model through the historical cache recognition result and the category to which the character belongs, so as to assist the above acoustic model to obtain the speech recognition result and further improve the accuracy of the speech recognition result.

[0106] Optionally, each character corresponds to a category to which it belongs. Here, the identifier of the character category may be a preset identifier. For example, the character category of the APP name is 1, the character category of the singer's name is 2, the character category of the song name is 3, the character category of the subject-predicate verb, etc. is 0, and so on.

[0107] It can be understood that there is a corresponding relationship between the historical cache recognition result and the category to which the character belongs. For example, if the historical cache recognition result stores the character "打开", then the category to which the historical character belongs stores the category to which "打开" belongs.

[0108] Optionally, when starting to process the first character voice in the voice command input by the user, there is no corresponding historical speech recognition result in the cache recognition result. Then the language model may use the preset padding character as the input to output the probability distribution of the next character. Here, the preset padding character may be a character set according to the business function or a character obtained according to the experimental data. The embodiment of the present application does not limit this.

[0109] Optionally, the embodiment of the present application does not limit the execution order of the above acoustic model and language model. It may be executed in sequence according to the acoustic model and the language model, or in sequence according to the language model and the acoustic model, or the acoustic model and the language model may be executed simultaneously.

[0110] After obtaining the first probability distribution and the second probability distribution, the electronic device can fuse the two probability distributions to obtain a fused probability distribution. Optionally, the electronic device can fuse the first probability distribution and the second probability distribution by weighted summation. It can be understood that the weights corresponding to the first probability distribution and the second probability distribution can be weights obtained after a large number of experimental verifications, or weights learned after training. The embodiments of the present application do not limit this. Exemplarily, taking the word "kai" in speech as an example, after probability fusion, the obtained fused probability distribution can be: the probability of belonging to the word "开" is g, the probability of belonging to the word "凯" is h, the probability of belonging to the word "楷" is i, and so on.

[0111] Then, the electronic device can decode the fused probability distribution by means of beam search. Assuming the number of beams is B, for the fused probability distribution, each beam takes the TopK of the probabilities to generate B×K candidate character queues. Then, the TopB of the probabilities of the B×K candidate character queues are taken to generate B candidate character queues. Finally, the Top1 of the probabilities of the B candidate character queues is selected to obtain the speech recognition result. Furthermore, the electronic device can display the text corresponding to the speech recognition result on the interface. Optionally, the generated B candidate character queues can be added to the above-mentioned cached recognition results as the input data of the language model in the next round. Optionally, the number of beams B can be determined by the B candidate character queues selected in the previous round.

[0112] Exemplarily, such as Figure 12As shown, assume that the number of beams B = 3 and TopK is Top3; the recognition result of the previous round of the voice character "da" is "打". In this round, the fusion probability distribution obtained by the electronic device is that the probability of belonging to the character "开" is g, the probability of belonging to the character "凯" is h, the probability of belonging to the character "楷" is i... (arranged in descending order of probability). In the case of 3 beams, the candidate character queue obtained in this round is "打开", "打凯", "打楷", and the corresponding probabilities are g, h, i respectively. Then in the next round, the fusion probability distribution corresponding to the voice character "shou" is that the probability of belonging to the character "手" is t, the probability of belonging to the character "收" is m, the probability of belonging to the character "受" is n... (arranged in descending order of probability). Assume that the Top3 probabilities are taken for each beam, that is, "手", "收", "受" are taken, and the corresponding probabilities are t, m, n respectively. Then, the electronic device combines "打开", "打凯", "打楷" and "手", "收", "受" to generate 3×3 candidate character queues, and the probability of each candidate character queue can be obtained. For example, the probability of "打开手" is g×t (or g + t), the probability of "打开受" is g×m (or g + m), etc. Then, take the candidate character queues with the Top3 probabilities from the 3×3 candidate character queues to generate 3 candidate character queues. For example, the Top3 candidate character queues are "打开手", "打开收", "打开受". Optionally, the electronic device can add "打开手", "打开收", "打开受" and the corresponding probabilities to the above cached recognition results. Finally, for the Top3 candidate character queues "打开手", "打开收", "打开受", select the character with the Top1 probability as the voice recognition result. For example, if "打开手" is the character with the highest probability, then "打开手" is the voice recognition result, and so on.

[0113] It can be understood that when the electronic device recognizes a certain voice character, after connecting it with the previously obtained candidate character queue, the probability sorting of the previously obtained candidate character queue may change. Then, when the electronic device displays the text corresponding to the voice recognition result, the previously displayed text will be updated. Exemplarily, for the case of the user's voice input "打开手电筒", after recognizing the three voices "da kai shou", the character with the highest determined probability is "打开收", that is, the electronic device displays the three characters "打开收"; but after recognizing the four voices "da kai shoudian", the character with the highest determined probability is "打开手电", then the electronic device will update "打开收" to "打开手电", that is, update the character corresponding to the "shou" character.

[0114] Furthermore, electronic devices can use beam search to decode the third probability distribution. Assuming the number of beams is B, for the third probability distribution, the top 1 probability of each beam is selected to generate B×1 candidate character category queues. Then, the categories corresponding to the above B candidate character queues are searched in the B×1 candidate character category queues to generate B candidate character category queues. These B candidate character category queues are then added to the above historical character categories to serve as input data for the language model in the next round.

[0115] For example, assuming the number of beams is 1, for the voice command "turn on the flashlight," if the character category of the word "turn on" is "00," and the probability of category "1" being the predicted category of the next character is the highest (i.e., Top 1), then category "1" can be used as the category of the next character. Furthermore, if "1" represents the category of the app name, the electronic device can directly predict that the category of the next character should also be "1," until the character categories corresponding to the app name are predicted.

[0116] In the aforementioned speech recognition method, after the electronic device activates its first business function, it initializes the speech recognition model, which then outputs the speech recognition result. Because the speech recognition model can distinguish the categories to which different characters belong, it can make the output result more closely resemble the potential characters of that category, making it more suitable for the current business scenario and improving the accuracy of the speech recognition result.

[0117] As can be seen from the above embodiments, the speech recognition model needs to be trained to obtain configuration parameters. Optionally, when the speech recognition model includes both an acoustic model and a language model, the training process of the speech recognition model may include:

[0118] For the acoustic model, its structure is exemplarily as described above. Figure 9 As shown, electronic devices can train acoustic models based on training data (including audio data and text data) until convergence.

[0119] For language models, training data can consist of short sentences, which may include words (entity words) of different categories. This training data can be constructed through template matching, model-assisted generation, etc. For example, training data could be "character: Open WeChat, character category: 0011; character: I want to listen to Andy Lau's 'Stupid Kid,' character category: 0002220333…", where each character corresponds to a category. The character category identifier can be a preset identifier, such as APP name character category 1, singer name character category 2, song name character category 3, subject-verb verb character category 0, and so on. Based on the training data, and based on the above… Figure 10The input / output method shown can output the probability distribution of the next character and the probability distribution of the category to which the next character belongs. Then, the electronic device can calculate the loss function based on the output results to complete the training process of the external language model.

[0120] It is understandable that the training process of a speech recognition model may take a long time. Therefore, the training process can be performed by a server (such as a cloud server), and the speech recognition model can be ported to electronic devices for use after the training is completed.

[0121] Optionally, the above-mentioned speech recognition method can also be executed by a server. The electronic device can send the received speech information to the server, and after the server determines the speech recognition result, it returns the result to the electronic device for display or other processing.

[0122] In other embodiments, different business functions correspond to different business scenarios, and the voice content and style vary in different business scenarios. For example, voice assistant scenarios often involve voice commands such as control instructions, navigation, and encyclopedic Q&A; in other words, the scenarios are mostly control commands. Other scenarios, such as call captions, often involve conversational voice, such as customer service, chat, and office communication. Therefore, if an electronic device uses the same voice recognition model for different business scenarios, it may result in low voice recognition accuracy. Thus, embodiments of this application can configure different voice recognition models for different business scenarios to further improve the accuracy of voice recognition results.

[0123] Figure 13 This is a flowchart illustrating another speech recognition method provided in an embodiment of this application. This method can be executed by an electronic device and may specifically include:

[0124] S201, when the first service function is enabled, initialize the speech recognition model according to the configuration parameters corresponding to the first service function.

[0125] Different business functions can correspond to different configuration parameters. Therefore, when the first business function is enabled, the electronic device can initialize the speech recognition model according to the configuration parameters corresponding to the first business function. It can be understood that the configuration parameters corresponding to different business functions can be obtained during the training process of the speech recognition model.

[0126] In some embodiments, a speech recognition model may include multiple models. Optionally, the speech recognition model may include an acoustic model, an internal language model, and an external language model, which work together to achieve speech recognition. Here, the acoustic model can perform the functions of the acoustic model in the above embodiments, and the external language model can perform the functions of the language model in the above embodiments.

[0127] Optionally, when the speech recognition model includes an acoustic model, an internal language model, and an external language model, the acoustic model and the internal language model can be models with fixed parameters, that is, the configuration parameters corresponding to different business functions are the same, while the external language model can be a model with variable parameters, that is, the electronic device needs to initialize the external language model according to the configuration parameters corresponding to the first business function.

[0128] Internal language models can be derived from acoustic models, for example, such as... Figure 14 As shown, a lightweight feedforward neural network (FFN) can be introduced between the prediction module and the joint module of the RNNT acoustic model to obtain the internal language model. The input of the internal language model is historical text features, which are processed by the prediction module, the FFN network, and the joint module, and then output the predicted probability of the next character. It can be understood that the acoustic model and the internal language model focus more on the user's audio or timbre features.

[0129] The training process for the internal language model, being an improvement upon the acoustic model, is structured as exemplarily described above. Figure 14 As shown, the electronic device can fix the parameters of the pre-trained acoustic model and train it using the corresponding training data. During training, only the parameters of the lightweight FFN network need to be updated. Specifically, during training, the training data is input from the prediction module, processed by the prediction module, the FFN network, and the joint module, and the result is output from the joint module. The loss is calculated using the cross-entropy loss function to complete the training process. The training method provided in this embodiment reduces the requirements for storage space and computing power, and does not require changing the acoustic model parameters during internal language model training, thus offering greater flexibility.

[0130] The difference between an external language model and an internal language model is that the configuration parameters in an external language model correspond to the first business function, making it more suitable for the current business function. In other words, the external language model focuses more on the business scenario and, under the influence of the configuration parameters, can output the predicted probability of the next character and the probability of its corresponding category based on the input character and character category. For example, the structure of an external language model can be as follows: Figure 15As shown, this includes fixed parameters and variable parameters. Fixed parameters are those that do not change based on the business scenario (e.g., W), while variable parameters change based on the business scenario. Variable parameters can be obtained by adding the parameter matrix (e.g., the q, k, v parameter matrix) from the attention module to the trained matrix. The trained matrix can be the result of multiplying trained matrices A and B. That is, different business functions correspond to different matrices A and B. Initialization involves loading the matrices A and B corresponding to the first business function to complete the initialization. The column width of matrix A is equal to the row height of matrix B; for example, the column width of matrix A and the row height of matrix B are both r. Optionally, the external language model can be a Transformer network structure, an LSTM network structure, a convolutional network structure, etc.

[0131] For the training process of the external language model, based on the training data consisting of short sentences corresponding to the aforementioned language model, electronic devices can also employ low-rank adaptation of large language models (LoRA) to fine-tune the parameters of the external language model. This involves fixing the fixed parameters in the external language model and replacing the original parameter matrix with the result of multiplying matrix A and matrix B by the parameter matrix in the attention module (e.g., the q, k, v parameter matrices). Matrix A and matrix B are then updated during training to complete the training process. Ultimately, different matrices A and B can be obtained corresponding to different business functions.

[0132] Optionally, the training data used by the external language model can also be the corresponding business text generated by the generative model based on a prompt specified by the business function (or business scenario). This prompt can be a manually written prompt or a prompt generated in batches using historical scenario data; that is, for each business scenario, there is a corresponding set of training data. For example, an example of the generative model generating multiple business texts based on a prompt can be found here. Figure 16 In other words, for navigation scenarios, when traffic information needs to be turned off, the generative model can construct various different business texts. Furthermore, the category to which each character belongs can be added to each set of training data to form the final training data.

[0133] S202, input the received voice information and the category of the cached character into the speech recognition model to obtain the fusion probability distribution and the probability distribution of the category of the next character.

[0134] For example, such as Figure 17As shown, when the speech recognition model includes an acoustic model, an internal language model, and an external language model, the electronic device can input speech information (or the audio features corresponding to the speech information) into the acoustic model, which can output an acoustic probability distribution (i.e., the first probability distribution). It can also input historical cached recognition results into the internal language model, which can output the probability distribution of the next character predicted by the internal language model (referred to as the fourth probability distribution). Furthermore, it can input historical cached recognition results and the historical character categories into the external language model, which can output the probability distribution of the next character predicted by the external language model (i.e., the second probability distribution) and the probability distribution of the next character's category (i.e., the third probability distribution). The form of the probability distribution of the next character predicted by the internal language model is similar to that predicted by the external language model, and will not be elaborated further.

[0135] After obtaining the first probability distribution, the second probability distribution, and the fourth probability distribution, the electronic device can fuse these three probability distributions to obtain a fused probability distribution. Optionally, the electronic device can fuse the first probability distribution, the second probability distribution, and the fourth probability distribution using a weighted summation method. Preferably, the electronic device can subtract the weighted fourth probability distribution after weighted summation of the first probability distribution and the second probability distribution to obtain the fused probability distribution.

[0136] S203, determine the speech recognition result based on the fusion probability, and cache the probability distribution of the category to which the next character belongs.

[0137] After obtaining the fusion probability, continue to see Figure 17 The electronic device can decode the fusion probability distribution using beam search. Assuming the number of beams is B, for each beam, the top K probabilities are selected to generate B×K candidate character queues. Then, the top B probabilities from these B×K queues are selected to generate B candidate character queues. Finally, the top 1 probability from these B queues is selected to obtain the speech recognition result. The electronic device can then display the text corresponding to this speech recognition result on the interface. Optionally, the generated B candidate character queues can be added to the aforementioned cached recognition result as input data for the internal and external language models in the next round. Furthermore, the electronic device can decode the probability distribution of the next character's category (i.e., the third probability distribution) using beam search to cache the probability distribution of the next character's category.

[0138] In the aforementioned speech recognition method, after the electronic device activates the first business function, it initializes the speech recognition model according to the configuration parameters corresponding to the first business function, and then the speech recognition model outputs the speech recognition result. Because the speech recognition model is initialized according to the configuration parameters corresponding to the first business function, it is adapted to the current business scenario and can perform targeted speech recognition, thereby improving the accuracy of the speech recognition result. Furthermore, the speech recognition model can distinguish the category to which different characters belong, allowing the output result to align with the potential characters of that category, making it more suitable for the current business scenario and further improving the accuracy of the speech recognition result.

[0139] The foregoing has detailed examples of the speech recognition method provided in the embodiments of this application. It is understood that, in order to achieve the above functions, the electronic device includes hardware and / or software modules corresponding to the execution of each function. Those skilled in the art should readily recognize that, based on the units and algorithm steps of the examples described in conjunction with the embodiments disclosed herein, this application can be implemented in hardware or a combination of hardware and computer software. Whether a function is executed by hardware or by computer software driving hardware depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application in conjunction with the embodiments, but such implementation should not be considered beyond the scope of this application.

[0140] This application embodiment can divide the electronic device into functional modules according to the above method example. For example, each function can be divided into a separate functional module, such as a detection unit, a processing unit, a display unit, etc., or two or more functions can be integrated into one module. The integrated module can be implemented in hardware or as a software functional module. It should be noted that the module division in this application embodiment is illustrative and only represents one logical functional division. In actual implementation, there may be other division methods.

[0141] It should be noted that all relevant content of each step involved in the above method embodiments can be referenced from the functional description of the corresponding functional module, and will not be repeated here.

[0142] The electronic device provided in this embodiment is used to execute the above-described speech recognition method, and therefore can achieve the same effect as the above-described implementation method.

[0143] When using integrated units, the electronic device may further include a processing module, a storage module, and a communication module. The processing module is used to control and manage the operation of the electronic device. The storage module supports the execution of stored program code and data. The communication module supports communication between the electronic device and other devices.

[0144] The processing module can be a processor or a controller. It can implement or execute various exemplary logic blocks, modules, and circuits described in conjunction with the disclosure of this application. The processor can also be a combination of functions that implement computing capabilities, such as a combination of one or more microprocessors, a digital signal processor (DSP), and a microprocessor, etc. The storage module can be a memory. The communication module can specifically be a radio frequency circuit, a Bluetooth chip, a Wi-Fi chip, or other devices that interact with other electronic devices.

[0145] In one embodiment, when the processing module is a processor and the storage module is a memory, the electronic device involved in this embodiment can be a device having... Figure 6 The device with the structure shown.

[0146] This application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, causes the processor to perform the speech recognition method of any of the above embodiments.

[0147] This application also provides a computer program product that, when run on a computer, causes the computer to perform the aforementioned steps to implement the speech recognition method described in the above embodiments.

[0148] In addition, embodiments of this application also provide an apparatus, which may specifically be a chip, component or module. The apparatus may include a connected processor and a memory; wherein the memory is used to store computer execution instructions, and when the apparatus is running, the processor may execute the computer execution instructions stored in the memory to cause the chip to execute the speech recognition method in the above method embodiments.

[0149] In this embodiment, the electronic device, computer-readable storage medium, computer program product or chip are all used to execute the corresponding methods provided above. Therefore, the beneficial effects that can be achieved can be referred to the beneficial effects of the corresponding methods provided above, and will not be repeated here.

[0150] Through the above description of the embodiments, those skilled in the art will understand that, for the sake of convenience and brevity, only the division of the above functional modules is used as an example. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above.

[0151] The units described as separate components may or may not be physically separate. A component shown as a unit can be one or more physical units; that is, it can be located in one place or distributed in multiple different locations. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0152] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0153] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a readable storage medium. Based on this understanding, the technical solutions of the embodiments of this application, in essence, or the parts that contribute to the prior art, or all or part of the technical solutions, can be embodied in the form of a software product. This software product is stored in a storage medium and includes several instructions to cause a device (which may be a microcontroller, chip, etc.) or processor to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0154] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A voice recognition method, the method being performed by an electronic device, and the method comprising: The method comprises: in the case where the first service function is started, initializing a speech recognition model, wherein the speech recognition model is used to recognize characters corresponding to speech information and categories to which the characters belong, and the speech recognition model comprises a first model and a second model; receiving speech information input through the first service function, and inputting the speech information and cached categories to which characters belong into the speech recognition model to obtain a speech recognition result, wherein the cached categories to which characters belong include categories obtained by recognizing historical speech information; the inputting of the speech information and the cached categories to which characters belong into the speech recognition model to obtain a speech recognition result comprises: inputting the speech information into the first model to obtain a first probability distribution, wherein the first probability distribution represents a probability distribution of characters to which the speech information belongs; inputting a cached recognition result and the cached categories to which characters belong into the second model to obtain a second probability distribution, wherein the second probability distribution represents a probability distribution of a next character predicted by the second model for the cached recognition result, and the cached recognition result includes text information obtained by recognizing the historical speech information; determining a fusion probability distribution based on the first probability distribution and the second probability distribution; determining the speech recognition result based on the fusion probability distribution.

2. The method of claim 1, wherein, the determining of the fusion probability distribution based on the first probability distribution and the second probability distribution comprises: performing weighted summation on the first probability distribution and the second probability distribution to obtain the fusion probability distribution.

3. The method of claim 1, wherein, the determining of the speech recognition result based on the fusion probability distribution comprises: decoding the fusion probability distribution in a beam search manner to obtain the speech recognition result.

4. The method of claim 3, wherein, the decoding of the fusion probability distribution in the beam search manner to obtain the speech recognition result comprises: selecting K characters corresponding to the fusion probability distribution for each beam to generate B×K candidate character queues, wherein B is the number of beams, K characters are characters corresponding to the first K probabilities after the fusion probability distribution is sorted in descending order of probability, B is the number of beams, and the B characters are characters corresponding to the first B probabilities after the B×K candidate character queues are sorted in descending order of probability. selecting B characters from the B×K candidate character queues, wherein the B characters are characters corresponding to the first B probabilities after the B×K candidate character queues are sorted in descending order of probability. taking a character with the largest probability in the B characters as the speech recognition result.

5. The method of claim 4, wherein, after the selection of the B characters, the method further comprises: adding the B characters into the cached recognition result to form a new cached recognition result.

6. The method according to any one of claims 1 to 5, characterized in that, The second model further outputs a third probability distribution, wherein the third probability distribution represents a probability distribution of categories to which a next character predicted by the second model belongs.

7. The method of claim 6, wherein, The method further comprises: decoding the third probability distribution in a beam search manner to determine categories corresponding to the B characters, and adding the categories corresponding to the B characters into the cached categories to which characters belong.

8. The method according to any one of claims 1 to 5, characterized in that, The first model is an RNNT acoustic model, comprising an encoding module, a prediction module, and a joint module. The encoding module is configured to encode audio features corresponding to the speech information. The prediction module is configured to process text features corresponding to the speech information. The joint module is configured to integrate outputs of the encoding module and the prediction module to obtain a first probability distribution.

9. The method according to any one of claims 1 to 5, characterized in that, The second model is a Transformer network structure, a long short-term memory network structure LSTM, or a convolutional network structure.

10. The method according to any one of claims 1 to 5, characterized in that, The first service function includes a voice assistant function, a see-and-say function, or a scenario-based instruction function.

11. An electronic device, comprising: Comprise: one or more processors; one or more memories; The memory stores one or more programs, which, when executed by the processor, cause the electronic device to perform the method of any one of claims 1-10.

12. A computer-readable storage medium, characterized in that, The computer readable storage medium stores a computer program, which, when executed by a processor, causes the processor to perform the method of any one of claims 1-10.

Images