Speech recognition method and electronic equipment

Abstract

The embodiment of the invention provides a voice recognition method and electronic equipment, and the method is executed by the electronic equipment, and comprises the steps: carrying out the initialization of a voice recognition model under the condition that a first service function is enabled, and enabling the voice recognition model to be used for recognizing a character corresponding to voice information and the type of the character; and receiving voice information input through the first service function, and inputting the voice information and categories to which the cached characters belong into the voice recognition model to obtain a voice recognition result, the categories to which the cached characters belong comprising categories obtained by recognizing historical voice information. The method can improve the accuracy of the speech recognition result.

Description

Technical Field

[0001] The present application relates to the field of electronic technology, and in particular to a speech recognition method and electronic equipment. Background Art

[0002] With the continuous development of electronic devices, their rich functionality has greatly facilitated users' lives. For example, with the voice assistant function enabled on an electronic device, users can have voice conversations with the electronic device, which can respond to the user's voice commands and convert them into text for display. For another example, with the "see and speak" function enabled on an electronic device, users can control the electronic device to perform various operations through voice.

[0003] In the above scenarios, electronic devices need to use automatic speech recognition (ASR) to recognize voice information and identify corresponding voice commands. However, the recognition accuracy of electronic devices in the speech recognition process in related technologies needs to be improved. Summary of the Invention

[0004] The present application provides a speech recognition method and electronic device, which can improve the accuracy of speech recognition results.

[0005] In a first aspect, the present application provides a speech recognition method, which is executed by an electronic device, and includes: initializing a speech recognition model when a first business function is turned on, wherein the speech recognition model is used to identify characters corresponding to speech information and the categories to which the characters belong; receiving speech information input through the first business function, and inputting the voice information and the cached character categories into the speech recognition model to obtain a speech recognition result, wherein the cached character categories include the categories obtained by recognizing historical voice information.

[0006] Among them, the first business function may include but is not limited to functions such as voice assistant, see and speak, and scenario-based instructions, that is, different business functions correspond to different business scenarios. The process of initializing the speech recognition model by the electronic device can be a process of loading configuration parameters, that is, loading the configuration parameters trained during the speech recognition model training process. Then, the electronic device can input the received voice information and the category of the cached characters into the above-mentioned speech recognition model to determine the speech recognition result.

[0007] In the above implementation, because the speech recognition model takes into account the category of the cached characters when recognizing speech information, the output results can be closer to the potential characters of that category, which is more suitable for the current business scenario and improves the accuracy of the speech recognition results.

[0008] In combination with the first aspect, in some implementations of the first aspect, the above-mentioned inputting the voice information and the category to which the cached characters belong into the voice recognition model to obtain the voice recognition result includes: inputting the voice information, the cached recognition result and the category to which the cached characters belong into the voice recognition model to obtain a fused probability distribution, the cached recognition result including text information obtained by recognizing historical voice information; and determining the voice recognition result based on the fused probability distribution.

[0009] Among them, in addition to considering the voice information and the categories to which the cached characters belong, the speech recognition model can also take into account the cached recognition results. The cached recognition results are the text information obtained by recognizing historical voice information, that is, the characters that have been recognized. Then, when the speech recognition model is processing, it can predict the possibility of the next character through the characters that have been recognized to further improve the accuracy of the speech recognition results.

[0010] In combination with the first aspect, in some implementations of the first aspect, the speech recognition model includes a first model and a second model, and the speech information, cached recognition results, and the category of the cached characters are input into the speech recognition model to obtain a fused probability distribution, including: inputting the speech information into the first model to obtain a first probability distribution, the first probability distribution representing the probability distribution of the characters to which the speech information belongs; inputting the cached recognition results and the category of the cached characters into the second model to obtain a second probability distribution, the second probability distribution representing the probability distribution of the next character predicted by the second model for the cached recognition results; and determining the fused probability distribution based on the first probability distribution and the second probability distribution.

[0011] The first model may be an acoustic model, and the second model may be a language model. The two models work together to achieve speech recognition.

[0012] Optionally, when the speech recognition model includes a first model and a second model, the electronic device can input the speech information (or the audio features corresponding to the speech information) into the first model and output a first probability distribution. For example, for the speech word "da", the first model can output the probability of it belonging to the word "da" as a, the probability of it belonging to the word "da" as b, the probability of it belonging to the word "da" as c, and so on based on the audio features of the user's speech word "da", where the sum of the probabilities is equal to 1.

[0013] In addition, the electronic device can input the historical cache recognition results and the category to which the cached characters belong into the language model, and output the predicted probability distribution of the next character (i.e., the second probability distribution). For example, in a voice assistant scenario, the voice command input by the user is "turn on the flashlight". When the recognition process reaches the character "开", it means that the character "打" and the category to which the character "打" belongs have been recognized previously. The predicted probability distribution of the next character can be: the probability of belonging to the character "开" is d, the probability of belonging to the character "凯" is e, the probability of belonging to the character "楷" is f, and so on.

[0014] Then, the electronic device can determine the fusion probability distribution based on the first probability distribution and the second probability distribution. Optionally, the electronic device can perform weighted summation on the first probability distribution and the second probability distribution to obtain the fusion probability distribution.

[0015] In the above implementation, the electronic device processes the voice information through the first model and predicts the characters that have been recognized historically through the second model to jointly determine the character corresponding to the current voice information, improving the accuracy of the voice recognition result.

[0016] Combined with the first aspect, in some implementations of the first aspect, determining the voice recognition result based on the fusion probability distribution includes: decoding the fusion probability distribution by means of beam search to obtain the voice recognition result.

[0017] In one implementation, assuming the number of beams is B, the electronic device can select K characters corresponding to the fusion probability distribution for each beam to generate a candidate character queue of B×K. The K characters are the characters corresponding to the first K probabilities after the fusion probability distribution is sorted in descending order of probability (i.e., the TopK of the probability). Then, select B characters from the B×K candidate character queues. The B characters are the characters corresponding to the first B probabilities after the B×K candidate character queues are sorted in descending order of the corresponding probabilities (i.e., the TopB of the probability). Finally, take the character with the largest probability among the B characters (i.e., the Top1 of the probability) as the voice recognition result. Optionally, the generated B characters can be added to the above cache recognition result as the input data of the language model in the next round. Optionally, the number of beams B can be determined by the B characters selected in the previous round.

[0018] Combined with the first aspect, in some implementations of the first aspect, the second model also outputs a third probability distribution, and the third probability distribution represents the probability distribution of the category to which the next character predicted by the second model belongs.

[0019] For the above-mentioned second model, since its input data is the historical cached recognition results and the categories to which the cached characters belong, the second model can also output a third probability distribution, that is, the probability distribution of the category to which the next character is predicted to belong, so as to facilitate the next round of predictions based on the probability distribution obtained this time.

[0020] In some implementations, the electronic device can use a beam search method to decode the third probability distribution, determine the categories corresponding to the above B characters, and add the categories corresponding to the B characters to the categories of the cached characters as input data for the second model in the next round.

[0021] Alternatively, assuming the number of beams is B, for the third probability distribution, the top 1 probability of each beam is selected to generate B×1 candidate character category queues. The categories corresponding to the B candidate character queues are then searched within the B×1 candidate character category queues to generate B candidate character category queues. These B candidate character category queues are then added to the historical character categories. This continuously enriches the cached character categories, which can assist in the speech recognition process and improve the accuracy of the speech recognition results.

[0022] In combination with the first aspect, in some implementations of the first aspect, the first model is an RNNT acoustic model, including an encoding module, a prediction module and a joint module. The encoding module is used to encode the audio features corresponding to the speech information, the prediction module is used to process the text features corresponding to the speech information, and the joint module is used to integrate the output of the encoding module and the output of the prediction module to obtain a first probability distribution.

[0023] In combination with the first aspect, in some implementations of the first aspect, the second model is a Transformer network structure, a long short-term memory network structure LSTM, or a convolutional network structure.

[0024] Among them, the language model can receive character sequences and character category sequences as input, generate latent features of characters and character categories through the character embedding layer and the character category embedding layer respectively, and add the two as the latent features of the current character to be sent to the language model for prediction calculation, and finally output the probability distribution of the next character and the probability distribution of the category to which the next character belongs.

[0025] During language model training, training data can consist of short sentences that can include words (entity words) of different categories. This training data can be constructed through template matching, model-assisted generation, and other methods. The language model proposed in this application can predict the probability of the next character belonging to a category based on historical recognition results, conforming to language habits in real scenarios and having good application scenarios.

[0026] In a second aspect, the present application provides a device, which is included in an electronic device and has the function of implementing the electronic device behavior described in the first aspect and possible implementations of the first aspect. The function can be implemented through hardware or through hardware executing corresponding software implementations. The hardware or software includes one or more modules or units corresponding to the above functions. For example, a receiving module or unit, a processing module or unit, etc.

[0027] In a third aspect, the present application provides an electronic device, which includes: a processor, a memory, and an interface; the processor, the memory, and the interface cooperate with each other so that the electronic device executes any one of the methods in the technical solution of the first aspect.

[0028] In a fourth aspect, the present application provides a chip including a processor, wherein the processor is configured to read and execute a computer program stored in a memory to perform the method of the first aspect and any possible implementation thereof.

[0029] Optionally, the chip further includes a memory, and the memory is connected to the processor via a circuit or wire.

[0030] Further optionally, the chip also includes a communication interface.

[0031] In a fifth aspect, the present application provides a computer-readable storage medium, in which a computer program is stored. When the computer program is executed by a processor, the processor executes any one of the methods in the technical solution of the first aspect.

[0032] In a sixth aspect, the present application provides a computer program product, which includes: a computer program code, which, when the computer program code runs on an electronic device, enables the electronic device to execute any one of the methods in the technical solution of the first aspect. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] Figure 1 This is a schematic diagram of a voice assistant scenario provided by an embodiment of the present application;

[0034] Figure 2 This is a schematic diagram of the process of setting up a voice assistant function provided in an embodiment of the present application;

[0035] Figure 3 This is a schematic diagram of an example of a "seeing is saying" scenario provided in an embodiment of the present application;

[0036] Figure 4 This is a schematic diagram of a setting process of the see-and-tell function provided in an embodiment of the present application;

[0037] Figure 5 This is a schematic diagram of a scenario-based instruction scenario provided in an embodiment of the present application;

[0038] Figure 6 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application;

[0039] Figure 7 This is a software structure diagram of an electronic device provided in an embodiment of the present application;

[0040] Figure 8 This is a flow chart of a speech recognition method provided in an embodiment of the present application;

[0041] Figure 9 is a schematic diagram of the structure of an acoustic model provided in an embodiment of the present application;

[0042] Figure 10 This is a schematic diagram of the input and output methods of a language model provided in an embodiment of the present application;

[0043] Figure 11 This is a schematic diagram of a processing process of a speech recognition method provided in an embodiment of the present application;

[0044] Figure 12 This is a schematic diagram of a beam search process provided by an embodiment of the present application;

[0045] Figure 13 1 is a flow chart of another speech recognition method provided in an embodiment of the present application;

[0046] Figure 14 This is a schematic diagram of the structure of an internal language model provided in an embodiment of the present application;

[0047] Figure 15 This is a schematic diagram of the structure of an external language model provided in an embodiment of the present application;

[0048] Figure 16 This is an example diagram of generating multiple business texts based on prompt provided in an embodiment of the present application;

[0049] Figure 17 This is a schematic diagram of the processing process of another speech recognition method provided in an embodiment of the present application. DETAILED DESCRIPTION

[0050] The technical solutions in the embodiments of the present application will be described below in conjunction with the accompanying drawings in the embodiments of the present application. In the description of the embodiments of the present application, unless otherwise specified, " / " means or, for example, A / B can mean A or B; "and / or" in this article is merely a description of the association relationship of associated objects, indicating that three relationships can exist, for example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. In addition, in the description of the embodiments of the present application, "multiple" means two or more than two.

[0051] In the following, the terms "first," "second," and "third" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the quantity of the technical features indicated. Therefore, a feature specified as "first," "second," or "third" may explicitly or implicitly include one or more of the features.

[0052] To facilitate a clear description of the technical solutions of the embodiments of the present application, the following briefly introduces some of the terms and technologies involved in the embodiments of the present application:

[0053] (1) Voice Assistant: This is the name of a voice control function that users can turn on or off through a preset switch on an electronic device. After the voice assistant function is turned on, the user needs to first wake up the voice assistant, for example, by using a wake-up word or the power button. After the voice assistant of the electronic device is woken up, the user can then speak voice commands to control the electronic device to perform the corresponding service operations.

[0054] (2) See and Speak: This is the name of another voice control function. Users can turn on or off the See and Speak function through a preset switch on the electronic device. Unlike voice assistants, after the switch of the See and Speak function is turned on, after the fence that the See and Speak function depends on is triggered, the electronic device enters a continuous reception state (or long reception state). The microphone of the electronic device will continuously collect the surrounding sounds. The user does not need to trigger the wake-up operation, but can directly speak voice commands to control the electronic device to perform the corresponding service operation. After the service operation is performed, the electronic device does not turn off the reception. The fence here mainly refers to the fence of scene events, such as the fence of the alarm scene event, the fence of the phone scene event, etc.

[0055] (3) Scenario-based commands: This is another name for a voice control function. Users can turn on or off the scenario-based command function on electronic devices through a preset switch. Unlike voice assistants and see-and-speak functions, after the scenario-based command function is turned on, when a specific scenario (for example, an alarm rings) is triggered, the electronic device enters a short reception state. The user directly speaks a voice command to control the electronic device to perform the corresponding service operation (for example, turning off the alarm). After the specific scenario ends (for example, the alarm turns off), the electronic device turns off the reception.

[0056] With the continuous development of electronic devices, the rich functions of electronic devices have provided great convenience to users' lives. For example, when the voice assistant function of an electronic device is turned on and the voice assistant is awakened, the user can have a voice conversation with the electronic device. If the user inputs a voice, the electronic device can make a voice reply based on the voice. For example, if the user inputs "How is the weather today", the electronic device can make a voice reply "Today's minimum temperature is 15 degrees Celsius and the maximum temperature is 25 degrees Celsius"; or, the user can also control the electronic device by voice, for example, if the user inputs The electronic device can respond with "OK" by voice and enter Interface. See Figure 1 In the voice input interface shown, when a user inputs "What's the weather like today?", the electronic device can display the corresponding text. After the electronic device recognizes the voice, it can query today's weather information through the network in the background and convert the corresponding text into speech output, for example, the speech output is "Today's minimum temperature is 15 degrees Celsius and the maximum temperature is 25 degrees Celsius." In other words, the electronic device needs to recognize the user's voice command and display the corresponding text information on the current interface.

[0057] The user setting process of enabling the voice assistant function may include: illustratively, Figure 2 As shown, a user clicking the settings icon on the electronic device's desktop interface will cause the electronic device to enter its settings interface. The settings interface includes setting options for various functions, such as WLAN options, Bluetooth options, display and brightness options, and smart assistant options. When the user clicks the smart assistant option control 21, the electronic device will jump to the smart assistant settings interface. The smart assistant settings interface includes various smart functions, such as YOYO suggestions, negative one screen, smart text, smart search, and smart voice. When the user clicks the smart voice option control 22, the electronic device will jump to the smart voice settings interface. The smart voice settings interface includes different voice assistant wake-up methods, such as voice wake-up, power button wake-up, breath wake-up, headphone remote wake-up, and Bluetooth device wake-up. The user can select the corresponding wake-up method to wake up the voice assistant as needed. For example, if the voice wake-up method is selected, the user can then voice-enter "Hello, YOYO" to wake up the voice assistant function; if the power button wake-up method is selected, the user can then press the power button for one second to wake up the voice assistant function. It can be understood that after the user selects the wake-up method in the smart voice setting interface, the voice assistant function is turned on. The voice recognition method of the embodiment of the present application can then be used to recognize the voice information input by the user to convert the voice information into text information and display it on the interface.

[0058] For example, when the electronic device turns on the "see and speak" function, the electronic device may display a prompt icon, such as Figure 3 As shown, after the "See and Speak" switch is turned on, the status bar of the display interface of the electronic device displays a prompt icon 31, indicating that the "See and Speak" function has been turned on. After "See and Speak" is turned on, the corresponding recording channel of "See and Speak" is started in the system and driver of the electronic device, and the electronic device enters a "long reception" state, continuously collecting surrounding sounds. The user can give instructions to the electronic device by voice at any time, without having to input a wake-up word or other wake-up method to wake up the electronic device. For example, the user inputs the voice "turn on the video" to the electronic device, and the electronic device responds to the user's input voice "turn on the video" to start the video application and display the user interface of the video application.

[0059] The user setting process of enabling the see-and-speak function may include: Figure 4 As shown, a user clicking the settings icon on the desktop interface of an electronic device will cause the electronic device to enter its settings interface. The settings interface includes setting options for various functions, such as WLAN options, Bluetooth options, display and brightness options, and smart assistant options. When a user clicks the smart assistant options control 41, the electronic device will be redirected to the smart assistant settings interface. The smart assistant settings interface includes various smart functions, such as YOYO suggestions, negative one screen, smart text, smart search, and voice control. When a user clicks the voice control options control 42, the electronic device will be redirected to the voice control settings interface. The voice control settings interface includes a see-and-speak control 43. When a user clicks the see-and-speak control 43, the electronic device will be redirected to the see-and-speak settings interface. Optionally, the see-and-speak settings interface includes a first prompt information area 44 for instructing the user on how to use the see-and-speak feature. The see-and-speak settings interface also includes a see-and-speak switch control 45. The user can click the see-and-speak switch control 45 to turn the see-and-speak feature on or off. In one example, upon receiving a user click on the see-and-speak switch control 45, the see-and-speak feature is turned on. Optionally, at this time, the visible-to-speak setting interface displays a second prompt information area 46 for prompting the user that the visible-to-speak function has been successfully enabled.

[0060] For example, when the electronic device turns on the scene-based command function, when an event in a specific scene is triggered, the electronic device enters a short reception state, and the electronic device will turn off the reception after the event in the specific scene ends. For example, when the alarm clock rings or the phone rings, the electronic device enters a short reception state, and the microphone of the electronic device will turn off the reception after the alarm clock turns off or the phone is answered. Figure 5As shown, after the scene-based command function is turned on, when the alarm rings, the electronic device displays the main interface 51, a prompt window 52 pops up above the main interface 51, and a prompt message 53 is displayed below the prompt window 52. The user can say the voice command "turn off the alarm" according to the prompt message, and the electronic device will execute the operation of turning off the alarm. The process of turning on the scene-based command function can be found in the above Figure 4 The process shown includes a scenario-based command control in the voice control setting interface, through which the scenario-based command function can be enabled, which will not be repeated here.

[0061] In related technologies, electronic devices typically use the same speech recognition model (or engine) to recognize user-input voice information. However, the user-input voice information may include entity words of different categories. For example, voice assistant scenarios often use short, command-like voice messages, such as control commands, navigation, and encyclopedia questions and answers. The characters or phrases in these short voice messages generally correspond to specific categories, such as singer names, song titles, film and television drama titles, city / place names, and app names. If the categories of the characters in the voice information are not considered and voice recognition is performed solely based on the audio features in the voice information, the accuracy of voice recognition may be low.

[0062] In view of this, an embodiment of the present application provides a speech recognition method and proposes a speech recognition model that takes into account the categories of characters in speech information. It can predict the category of the next character based on historical categories, so that the recognition results are closer to the potential characters of the category, thereby improving the accuracy of speech recognition.

[0063] It can also be understood that the speech recognition method provided in the embodiments of the present application can be applied to electronic devices with speech recognition functions or services, such as mobile phones, tablet computers, wearable devices, vehicle-mounted devices, augmented reality (AR) / virtual reality (VR) devices, laptops, ultra-mobile personal computers (UMPCs), netbooks, and personal digital assistants (PDAs). The embodiments of the present application do not impose any restrictions on the specific types of electronic devices.

[0064] For example, Figure 61 is a schematic diagram of the structure of an electronic device 100 provided in an embodiment of the present application. Taking the electronic device as a mobile phone as an example, the electronic device 100 may include a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, a charging management module 140, a power management module 141, a battery 142, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, an earphone interface 170D, a sensor module 180, a button 190, a motor 191, an indicator 192, a camera 193, a display 194, and a subscriber identification module (SIM) card interface 195, etc. The sensor module 180 may include a pressure sensor 180A, a gyroscope sensor 180B, an air pressure sensor 180C, a magnetic sensor 180D, an acceleration sensor 180E, a distance sensor 180F, a proximity light sensor 180G, a fingerprint sensor 180H, a temperature sensor 180J, a touch sensor 180K, an ambient light sensor 180L, a bone conduction sensor 180M, etc.

[0065] The processor 110 may include one or more processing units. For example, the processor 110 may include an application processor (AP), a modem processor, a graphics processing unit (GPU), an image signal processor (ISP), a controller, a memory, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural-network processing unit (NPU). The different processing units may be independent devices or integrated into one or more processors.

[0066] Processor 110 may also include a memory for storing instructions and data. In some embodiments, the memory in processor 110 is a cache memory. This memory can store instructions or data that have just been used or are being recycled by processor 110. If processor 110 needs to use the same instruction or data again, it can directly retrieve it from the memory. This avoids duplicate accesses, reduces processor 110 latency, and thus improves system efficiency.

[0067] In some embodiments, the processor 110 may include one or more interfaces. The interfaces may include an inter-integrated circuit (I2C) interface, an inter-integrated circuit sound (I2S) interface, a pulse code modulation (PCM) interface, a universal asynchronous receiver / transmitter (UART) interface, a mobile industry processor interface (MIPI), a general-purpose input / output (GPIO) interface, a subscriber identity module (SIM) interface, and / or a universal serial bus (USB) interface.

[0068] The wireless communication function of the electronic device 100 can be implemented through the antenna 1, the antenna 2, the mobile communication module 150, the wireless communication module 160, the modem processor and the baseband processor.

[0069] The mobile communication module 150 can provide solutions for wireless communications including 2G / 3G / 4G / 5G applied to the electronic device 100. The mobile communication module 150 may include at least one filter, a switch, a power amplifier, a low noise amplifier (LNA), etc. The mobile communication module 150 can receive electromagnetic waves from the antenna 1, and filter, amplify, and process the received electromagnetic waves, and transmit them to the modulation and demodulation processor for demodulation. The mobile communication module 150 can also amplify the signal modulated by the modulation and demodulation processor, and convert it into electromagnetic waves for radiation through the antenna 1. In some embodiments, at least some of the functional modules of the mobile communication module 150 can be set in the processor 110. In some embodiments, at least some of the functional modules of the mobile communication module 150 can be set in the same device as at least some of the modules of the processor 110.

[0070] Electronic device 100 implements display functionality through a GPU, display screen 194, and an application processor. A GPU is a microprocessor for image processing that connects display screen 194 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. Processor 110 may include one or more GPUs that execute program instructions to generate or modify display information.

[0071] Display screen 194 is used to display images, videos, and the like. Display screen 194 includes a display panel. The display panel can be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (FLED), a MiniLED, a MicroLED, a Micro-oLed, or a quantum dot light-emitting diode (QLED). In some embodiments, electronic device 100 may include one or N display screens 194, where N is a positive integer greater than one.

[0072] The electronic device 100 can implement a shooting function through an ISP, a camera 193, a video codec, a GPU, a display screen 194, and an application processor.

[0073] The NPU is a neural network (NN) computing processor that quickly processes input information by drawing on the structure of biological neural networks, such as the transmission mode between neurons in the human brain, and can also continuously self-learn. The NPU can realize applications such as intelligent cognition of the electronic device 100, such as image recognition, face recognition, speech recognition, text understanding, etc. In the embodiment of the present application, the electronic device can call the NPU to implement the function of the speech recognition model to complete the speech recognition process and obtain the speech recognition result.

[0074] The internal memory 121 can be used to store computer executable program codes, and the executable program codes include instructions. The processor 110 executes various functional applications and data processing of the electronic device 100 by running the instructions stored in the internal memory 121. The internal memory 121 may include a program storage area and a data storage area. Among them, the program storage area can store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc. The data storage area can store data created during the use of the electronic device 100 (such as audio data, a phone book, etc.), etc. In addition, the internal memory 121 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, a universal flash storage (UFS), etc.

[0075] The electronic device 100 can implement audio functions such as music playback and recording through the audio module 170, the speaker 170A, the receiver 170B, the microphone 170C, the headphone jack 170D, and the application processor.

[0076] Microphone 170C, also known as "microphone" or "microphone", is used to convert sound signals into electrical signals. When making a call or sending a voice message, the user can speak by putting their mouth close to the microphone 170C to input the sound signal into the microphone 170C. The electronic device 100 can be provided with at least one microphone 170C. In other embodiments, the electronic device 100 can be provided with two microphones 170C, which can not only collect sound signals but also realize noise reduction function. In other embodiments, the electronic device 100 can also be provided with three, four or more microphones 170C to collect sound signals, reduce noise, identify the source of sound, realize directional recording function, etc. In an embodiment of the present application, for example, in the scenario of voice call function, the user can input voice commands through the microphone to realize the input of voice information.

[0077] It should be understood that the structures illustrated in the embodiments of the present application do not constitute a specific limitation on the electronic device 100. In other embodiments of the present application, the electronic device 100 may include more or fewer components than shown, or may combine or separate certain components, or arrange the components differently. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

[0078] It is also understandable that the above Figure 6 The following is only an example of an electronic device being a mobile phone. If the electronic device is a tablet computer, PC, PDA, wearable device or other type of device, the structure of the electronic device may include more than Figure 6 The structure shown in the figure is less than Figure 6 More structures are shown in the figure, which are not limited here.

[0079] The following describes the software system of the electronic device 100 and the coordination process between the software system and the hardware.

[0080] The software system of the electronic device 100 can adopt a layered architecture, an event-driven architecture, a micro-kernel architecture, a micro-service architecture, or a cloud architecture. In the embodiment of the present application, the Android system with a layered architecture is used as an example to illustrate the software structure of the electronic device 100.

[0081] Figure 7: This is a software structure diagram of the electronic device 100 according to an embodiment of the present application. The layered architecture divides the software into several layers, each with clear roles and division of labor. The layers communicate with each other through software interfaces. In some embodiments, the Android system is divided into four layers: the application layer, the application framework layer, the Android runtime (Android runtime) and system libraries, and the kernel layer. The application layer can include a series of application packages, such as the Android application package (APK).

[0082] like Figure 7 As shown, the application package may include applications such as voice assistant and see-and-speak, and may also include an APK integrated with a voice recognition model. The APK integrated with the voice recognition model can realize the voice recognition function, and different applications can call the APK integrated with the voice recognition model when needed to convert voice information into text information.

[0083] In addition, the application package can also include applications such as camera, gallery, calendar, call, map, navigation, WLAN, Bluetooth, music, video, short message, etc. Figure 7 Not shown.

[0084] The application framework layer provides an application programming interface (API) and programming framework for the applications in the application layer. The application framework layer includes some predefined functions.

[0085] like Figure 7 As shown, the application framework layer may include a window manager, a content provider, a view system, a phone manager, a resource manager, a notification manager, and the like.

[0086] The window manager is used to manage window programs. The window manager can obtain the size of the display screen, determine whether there is a status bar, lock the screen, take screenshots, etc. The content provider is used to store and obtain data and make this data accessible to applications. The view system includes visual controls, such as controls for displaying text, controls for displaying pictures, etc. The view system can be used to build applications. The phone manager is used to provide communication functions for the electronic device 100. For example, the management of call status (including answering, hanging up, etc.). The resource manager provides various resources to applications, such as localized strings, icons, pictures, layout files, video files, etc. The notification manager enables applications to display notification information in the status bar, which can be used to convey notification-type messages. The notification manager can also be a notification that appears in the status bar at the top of the system in the form of a chart or scroll bar text, or a notification that appears on the screen in the form of a dialog window.

[0087] The Android runtime includes the core library and the virtual machine. The Android runtime is responsible for scheduling and management of the Android system.

[0088] The core library consists of two parts: one is the function that needs to be called by the Java language, and the other is the Android core library.

[0089] The application layer and application framework layer run in a virtual machine. The virtual machine executes Java files in the application layer and application framework layer as binary files. The virtual machine manages object lifecycles, stack management, thread management, security and exception management, and garbage collection.

[0090] The system library can include multiple functional modules, such as surface manager, media libraries, 3D graphics processing library (such as OpenGL ES), 2D graphics engine (such as SGL), etc.

[0091] The surface manager manages the display subsystem and provides fusion of 2D and 3D layers for multiple applications. The media library supports playback and recording of various common audio and video formats, as well as static image files. It also supports a variety of audio and video encoding formats. The 3D graphics processing library implements 3D graphics drawing, image rendering, compositing, and layer processing. The 2D graphics engine is the drawing engine for 2D drawing.

[0092] The kernel layer is the layer between hardware and software. The kernel layer includes at least display driver, camera driver, audio driver, and sensor driver.

[0093] For ease of understanding, the following examples of this application will be described with Figure 6 and Figure 7 Taking the electronic device with the structure shown as an example, the speech recognition method provided in the embodiment of the present application is specifically described in combination with the accompanying drawings and application scenarios.

[0094] Figure 8 This is a flow chart of a speech recognition method provided in an embodiment of the present application. The method can be performed by an electronic device and may specifically include:

[0095] S101 , when the first service function is enabled, initializing the speech recognition model.

[0096] Among them, the first business function may include but is not limited to functions such as voice assistant, see and speak, and scenario-based instructions, that is, different business functions correspond to different business scenarios. The process of initializing the speech recognition model by the electronic device can be a process of loading configuration parameters, that is, loading the configuration parameters trained during the speech recognition model training process to prepare for subsequent data processing. The training process of the speech recognition model is described in detail in the following embodiment. The trained speech recognition model can recognize speech information according to the speech information and the category to which the character belongs.

[0097] In some embodiments, the speech recognition model may include multiple models. Optionally, the speech recognition model may include an acoustic model and a language model, and speech recognition can be achieved under the combined action of these two models. Of course, the embodiments of the present application do not limit the specific number and type of models.

[0098] Optionally, the acoustic model may be a RNNT (recurrent neural network transducer) acoustic model. For example, the structure of the RNNT acoustic model may be as follows: Figure 9 As shown, it includes but is not limited to an encoding module (encoder), a prediction module (predict) and a joint module (joint); after receiving the voice information input by the user, the encoding module can be used to encode the audio features corresponding to the voice information, the prediction module can be used to predict the text features corresponding to the voice information, and the joint module can be used to integrate the output of the encoding module and the output of the prediction module to output an acoustic probability distribution.

[0099] The input of the language model is the character (or character feature) and the character category, and the output is the predicted probability of the next character and the category probability corresponding to the next character. For example, the input and output of the language model can be as follows: Figure 10 As shown, the language model can receive a character sequence and a character category sequence as input, generate latent features of the character and character category through the character embedding layer and the character category embedding layer respectively, and add the two as the latent features of the current character to be fed into the language model for prediction calculation, and finally output the probability distribution of the next character and the probability distribution of the category to which the next character belongs. Optionally, the language model can be a Transformer network structure, a long short-term memory network structure (long short-term memory, LSTM), a convolutional network structure, etc.

[0100] Optionally, when the first business function is enabled, the speech recognition model may be called by an application corresponding to the first business function, and the application may read corresponding configuration parameters to initialize the speech recognition model.

[0101] S102. Input the received voice information and the category of the cached characters into a voice recognition model to obtain a voice recognition result.

[0102] Among them, the voice information received by the electronic device may include voice commands input by the user in a voice assistant scenario, voice commands input by the user in a visible and speakable scenario, etc. The category of the cached characters may include the categories obtained by recognizing historical voice information. After the electronic device inputs the voice information and the category of the cached characters into the voice recognition model, through the processing of the voice recognition model, a voice recognition result can be obtained.

[0103] Optionally, when the voice recognition model includes an acoustic model and a language model, when the electronic device inputs the received voice information and the category of the cached characters into the voice recognition model, the acoustic model and the language model need to perform corresponding processing processes respectively.

[0104] Exemplarily, as Figure 11 shown, when the electronic device inputs the voice information (or the audio features corresponding to the voice information) into the acoustic model, an acoustic probability distribution (abbreviated as the first probability distribution) can be output. Here, the acoustic probability distribution can be the probability of the character to which the voice belongs. For example, in a voice assistant scenario, if the voice command input by the user is "turn on the flashlight", then the electronic device can input the voices corresponding to "turn on", "hand", "electric", and "flashlight" into the acoustic model in sequence to obtain the acoustic probability distribution of each character. Optionally, the acoustic probability distribution can be understood as a vocabulary table, which includes the characters that the voice may correspond to and the probability of each character. Finally, the character with the highest probability can be selected as the recognition result. For example, for the voice "da", the acoustic model outputs the probability that it belongs to the character "hit" as a, the probability that it belongs to the character "big" as b, the probability that it belongs to the character "answer" as c, etc. Among them, the sum of the probabilities is equal to 1. Another example is for the voice "kai", the acoustic model can also output the probabilities that it belongs to the characters "open", "kai", and "kai" respectively. Optionally, since the character "open" has the character "hit" in front, in addition to considering the probability of the character "open" alone, the acoustic model can also combine the probability of the character "hit" to jointly determine the probability of the character to which the voice "turn on" belongs.

[0105] Continue to refer to Figure 11, optionally, the electronic device may input the historical cache recognition result and the category to which the character belongs into the language model, and may output the predicted probability distribution of the next character (abbreviated as the second probability distribution) and the probability distribution of the category to which the next character belongs (abbreviated as the third probability distribution). The cache recognition result input here may be the recognized text information (or text features, hidden features, etc. corresponding to the text information). For example, in the voice assistant scenario, the voice command input by the user is "turn on the flashlight". When it comes to the recognition process of the character "开", it means that the character "打" has been recognized before. Then the historical cache recognition result is the character "打" or the text feature corresponding to the character "打". The predicted probability distribution of the next character may be: the probability of belonging to the character "开" is d, the probability of belonging to the character "凯" is e, the probability of belonging to the character "楷" is f, and so on. The category to which the character belongs input here is the category to which the recognized character belongs. For example, the category to which the recognized character "打" belongs is 0. The predicted probability distribution of the category to which the next character belongs may be: the probability of belonging to category 0 is x, the probability of belonging to category 1 is y, the probability of belonging to category 2 is z, and so on. It can be understood that in the embodiments of the present application, the probability distribution of the next character and the probability distribution of the category to which it belongs are predicted by the language model through the historical cache recognition result and the category to which the character belongs, so as to assist the above acoustic model to obtain the speech recognition result and further improve the accuracy of the speech recognition result.

[0106] Optionally, each character corresponds to a category to which it belongs. The identifier of the character category here may be a preset identifier. For example, the character category of the APP name is 1, the character category of the singer's name is 2, the character category of the song name is 3, the character category of the subject-predicate verb, etc. is 0, and so on.

[0107] It can be understood that there is a corresponding relationship between the historical cache recognition result and the category to which the character belongs. For example, if the historical cache recognition result stores the character "打开", then the category to which the historical character belongs stores the category to which "打开" belongs.

[0108] Optionally, when starting to process the first character voice in the voice command input by the user, there is no corresponding historical speech recognition result in the cache recognition result. Then the language model may use a preset padding character as the input to output the probability distribution of the next character. The preset padding character here may be a character set according to the service function or a character obtained according to the experimental data. The embodiments of the present application do not limit this.

[0109] Optionally, the embodiments of the present application do not limit the execution order of the above acoustic model and language model. It may be executed sequentially in the order of the acoustic model and the language model, or may be executed sequentially in the order of the language model and the acoustic model, or the acoustic model and the language model may be executed simultaneously.

[0110] After obtaining the first probability distribution and the second probability distribution, the electronic device can fuse the two probability distributions to obtain a fused probability distribution. Optionally, the electronic device can fuse the first probability distribution and the second probability distribution by weighted summation. It can be understood that the weights corresponding to the first probability distribution and the second probability distribution can be weights obtained after a large number of experimental verifications, or they can be weights learned after training, and the embodiments of the present application do not limit this. For example, taking the phonetic word "kai" as an example, after probability fusion, the obtained fused probability distribution can be: the probability of belonging to the word "kai" is g, the probability of belonging to the word "kai" is h, the probability of belonging to the word "kai" is i, and so on.

[0111] Then, the electronic device can use the beam search method to decode the fused probability distribution. Assuming that the number of beams is B, for the fused probability distribution, each beam takes the TopK of the probability to generate B×K candidate character queues, and then takes the TopB of the probability for the B×K candidate character queues to generate B candidate character queues. Finally, the Top1 of the probability is selected for the B candidate character queues to obtain the speech recognition result, and then the electronic device can display the text corresponding to the speech recognition result on the interface. Optionally, the generated B candidate character queues can be added to the above-mentioned cached recognition results as input data for the language model in the next round. Optionally, the number of beams B can be determined by the B candidate character queues selected in the previous round.

[0112] For example, Figure 12As shown, assuming the number of beams B = 3 and TopK is the Top3; the recognition result for the speech character "da" in the previous round is "da". In this round, the electronic device obtains a fused probability distribution with a probability of g for "kai", a probability of h for "kai", a probability of i for "kai", and so on (arranged in descending order of probability). Given a beam number of 3, the candidate character queues obtained in this round are "kai kai", "da kai", and "da kai", with corresponding probabilities of g, h, and i, respectively. Then, in the next round, the fused probability distribution corresponding to the speech character "shou" is a probability of t for "hand", a probability of m for "shou", and a probability of n for "shou", and so on (arranged in descending order of probability). Assuming each beam takes the Top3 probabilities, that is, "hand", "shou", and "shou", with corresponding probabilities of t, m, and n, respectively. Then, the electronic device combines "open", "da kai", "da kai" and "hand", "shou", and "shou" to generate 3×3 candidate character queues, and can obtain the probability of each candidate character queue, for example, the probability of "open hand" is g×t (or g+t), the probability of "open receive" is g×m (or g+m), and so on. Then, the candidate character queues with the top 3 probabilities are taken from the 3×3 candidate character queues to generate 3 candidate character queues. For example, the top 3 candidate character queues are "open hand", "open receive", and "open receive". Optionally, the electronic device can add "open hand", "open receive", "open receive" and the corresponding probabilities to the above cached recognition results. Finally, for the top 3 candidate character queues "open hand", "open receive", and "open receive", the character with the top 1 probability is selected, which is the speech recognition result. For example, if "open hand" is the character with the highest probability, then "open hand" is the speech recognition result, and so on.

[0113] It is understandable that when an electronic device recognizes a certain voice character, after connecting it with the previously obtained candidate character queue, it may change the probability ranking of the previously obtained candidate character queue. Then, when the electronic device displays the text corresponding to the voice recognition result, it will update the previously displayed text. For example, in the case of user voice input of "turn on the flashlight", after recognizing the three voices of "da kai shou", the character with the highest probability is determined to be "kai shou", that is, the electronic device displays the three characters of "kai shou"; but after recognizing the four voices of "da kai shoudian", the character with the highest probability is determined to be "kai shoudian", then the electronic device will update "kai shoudian" to "kai shoudian", that is, it updates the character corresponding to the word "shou".

[0114] In addition, the electronic device can use a beam search method to decode the third probability distribution. Assuming that the number of beams is B, for the third probability distribution, each beam takes the Top1 of the probability to generate B×1 candidate character category queues, and then searches the B×1 candidate character category queues for the categories corresponding to the above B candidate character queues to generate B candidate character category queues, and adds the B candidate character category queues to the categories to which the above historical characters belong as input data for the language model in the next round.

[0115] For example, assuming the number of beams is 1, for the voice command "turn on the flashlight", if the character category of the word "turn on" is "00", and the probability of category "1" among the predicted categories of the next character is the highest (i.e., Top1), then category "1" can be used as the category of the next character. In addition, if "1" represents the category of the app name, the electronic device can also directly predict that the category of the next character should also be "1", until the character category corresponding to the app name is predicted.

[0116] In the aforementioned speech recognition method, after the electronic device activates the first service function, it initializes the speech recognition model, which then outputs speech recognition results. Because the speech recognition model can distinguish the categories to which different characters belong, it can align the output results with the potential characters in that category, making them more suitable for the current service scenario and improving the accuracy of the speech recognition results.

[0117] As can be seen from the above embodiment, the speech recognition model needs to be trained to obtain configuration parameters. In the case where the speech recognition model includes an acoustic model and a language model, the training process of the speech recognition model may optionally include:

[0118] For the acoustic model, its structure is exemplarily as described above. Figure 9 As shown, the electronic device can train the acoustic model according to the training data (including audio data and text data) until convergence.

[0119] For language models, training data can be composed of short sentences, which can include words of different categories (entity words). The training data can be constructed through template matching, model-assisted generation, etc. For example, the training data can be "character: open WeChat, character category: 0011; character: I want to listen to Andy Lau's Stupid Child, character category: 0002220333...", where each character corresponds to its category, and the identifier of the character category can be a preset identifier, such as the character category of the APP name is 1, the character category of the singer name is 2, the character category of the song name is 3, the character category of the subject, predicate, verb, etc. is 0, and so on. On the basis of the training data, based on the above Figure 10The input and output method shown can output the probability distribution of the next character and the probability distribution of the category to which the next character belongs. Then, the electronic device can calculate the loss function based on the output result to complete the training process of the external language model.

[0120] It is understandable that since the training process of the speech recognition model may take a long time, the training process can be performed by a server (such as a cloud server), and after the training is completed, the speech recognition model can be transplanted to the electronic device for use.

[0121] Optionally, the execution process of the above-mentioned voice recognition method can also be executed by a server. The electronic device can send the received voice information to the server. After the server determines the voice recognition result, it returns the result to the electronic device, which is displayed or otherwise processed by the electronic device.

[0122] In other embodiments, because different business functions correspond to different business scenarios, the voice content and style in different business scenarios vary. For example, in voice assistant scenarios, most voice commands are such as control commands, navigation, and encyclopedia questions and answers. It can be seen that most of the voice commands in the scene are control commands. In other scenarios, for example, the voice of the call subtitles is mostly conversational, such as customer service, chat, office communication, etc. Therefore, if the electronic device uses the same voice recognition model for different business scenarios, it may also result in a lower voice recognition accuracy. Therefore, the embodiments of the present application can also configure different voice recognition models for different business scenarios to further improve the accuracy of the voice recognition results.

[0123] Figure 13 This is a flow chart of another example of a speech recognition method provided in an embodiment of the present application. The method can be performed by an electronic device and may specifically include:

[0124] S201 , when a first service function is enabled, initialize a speech recognition model according to configuration parameters corresponding to the first service function.

[0125] Different service functions may correspond to different configuration parameters. When a first service function is enabled, the electronic device may initialize the speech recognition model according to the configuration parameters corresponding to the first service function. It is understood that the configuration parameters corresponding to different service functions may be obtained during the speech recognition model training process.

[0126] In some embodiments, the speech recognition model may include multiple models. Optionally, the speech recognition model may include an acoustic model, an internal language model, and an external language model. The three models work together to achieve speech recognition. The acoustic model here can implement the functions of the acoustic model in the above-mentioned embodiment, and the external language model can implement the functions of the language model in the above-mentioned embodiment.

[0127] Optionally, in the case where the speech recognition model includes an acoustic model, an internal language model and an external language model, the acoustic model and the internal language model can be models with fixed parameters, that is, the configuration parameters corresponding to different business functions are the same, and the external language model can be a model with variable parameters, that is, the electronic device needs to initialize the external language model according to the configuration parameters corresponding to the first business function.

[0128] The internal language model can be improved based on the acoustic model, for example, Figure 14 As shown, a lightweight feedforward neural network (FFN) can be introduced between the prediction module and the joint module of the RNNT acoustic model to obtain an internal language model. The internal language model takes historical text features as input and, after processing by the prediction module, the FFN network, and the joint module, outputs the predicted probability of the next character. It can be understood that the acoustic model and the internal language model focus more on the user's audio or voice characteristics.

[0129] For the training process of the internal language model, since it is improved on the basis of the acoustic model, the structure is exemplarily as described above. Figure 14 As shown, the electronic device can fix the parameters of the trained acoustic model and use the training data corresponding to the acoustic model for training. During the training process, only the parameters of the lightweight FFN network need to be updated. During training, the training data is input from the prediction module, calculated by the prediction module, the FFN network, and the joint module, and the result is output from the joint module. The loss is calculated using the cross-entropy loss function to complete the training process. The training method provided in the embodiment of the present application can reduce the requirements for storage space and computing power, and when training the internal language model, there is no need to change the parameters of the acoustic model, which is more flexible.

[0130] The difference between the external language model and the internal language model is that the configuration parameters in the external language model are parameters corresponding to the first business function, which is more suitable for the current business function. That is, the external language model pays more attention to the business scenario, and can output the predicted probability of the next character and the category probability corresponding to the next character based on the input characters and character categories under the action of the configuration parameters. For example, the structure of the external language model can be as follows Figure 15As shown, it includes fixed parameters and variable parameters, wherein the fixed parameters are parameters that do not change according to the business scenario (for example, W), and the variable parameters are parameters that change according to the business scenario. For variable parameters, it can be obtained by adding the parameter matrix in the attention module (for example, q, k, v parameter matrix) to the matrix obtained by training. Here, the matrix obtained by training can be the result of multiplying the trained matrix A and matrix B. In other words, different business functions correspond to different matrices A and B. During initialization, the matrix A and matrix B corresponding to the first business function are loaded to complete the initialization. Among them, the column width of matrix A is equal to the row height of matrix B. For example, the column width of matrix A and the row height of matrix B are both r. Optionally, the external language model can be a Transformer network structure, an LSTM network structure, a convolutional network structure, and the like.

[0131] For the training process of the external language model, based on the training data consisting of short sentences corresponding to the above language model, the electronic device can also use the low-rank adaptation of large language models (LoRA) method to fine-tune the parameters of the external language model. That is, the fixed parameters in the external language model are fixed, and the parameter matrix in the attention module (such as the q, k, and v parameter matrices) plus the result of multiplying matrix A and matrix B are used to replace the original parameter matrix. Matrix A and matrix B are updated during the training process to complete the training process. In the end, different matrices A and B corresponding to different business functions can be obtained.

[0132] Optionally, the training data used by the external language model can also be the corresponding business text generated by the generative model based on the prompt formulated by the business function (or business scenario). The prompt can be a manually written prompt or a prompt produced in batches based on the historical data of the scenario. That is, for each business scenario, there is a corresponding set of training data. For example, an example of a generative model generating multiple business texts based on a prompt can be seen in Figure 16 In other words, for navigation scenarios where you want to turn off traffic information, the generative model can construct a variety of different business texts. At the same time, you can also add the category of each character to each set of training data to form the final training data.

[0133] S202 , inputting the received voice information and the cached character category into a voice recognition model to obtain a fusion probability distribution and a probability distribution of the category to which the next character belongs.

[0134] For example, Figure 17As shown, when the speech recognition model includes an acoustic model, an internal language model and an external language model, the electronic device can input the speech information (or the audio features corresponding to the speech information) into the acoustic model, and can output the acoustic probability distribution (i.e., the first probability distribution). And by inputting the historical cached recognition results into the internal language model, the next character probability distribution predicted by the internal language model (referred to as the fourth probability distribution) can be output. And by inputting the historical cached recognition results and the historical character categories into the external language model, the next character probability distribution predicted by the external language model (i.e., the second probability distribution) and the probability distribution of the category to which the next character belongs (i.e., the third probability distribution) can be output. Among them, the form of the next character probability distribution predicted by the internal language model is similar to that predicted by the external language model, and will not be repeated.

[0135] After obtaining the first probability distribution, the second probability distribution, and the fourth probability distribution, the electronic device may fuse the three probability distributions to obtain a fused probability distribution. Optionally, the electronic device may fuse the first probability distribution, the second probability distribution, and the fourth probability distribution using a weighted summation method. Preferably, the electronic device may perform a weighted summation of the first probability distribution and the second probability distribution, and then subtract the weighted fourth probability distribution to obtain the fused probability distribution.

[0136] S203: Determine the speech recognition result according to the fusion probability, and cache the probability distribution of the category to which the next character belongs.

[0137] After obtaining the fusion probability, continue to refer to Figure 17 , the electronic device can use the beam search method to decode the fusion probability distribution. Assuming that the number of beams is B, for the fusion probability distribution, each beam takes the TopK of the probability to generate B×K candidate character queues, and then takes the TopB of the probability of the B×K candidate character queues to generate B candidate character queues. Finally, the Top1 of the probability of the B candidate character queues is selected to obtain the speech recognition result, and then the electronic device can display the text corresponding to the speech recognition result on the interface. Optionally, the generated B candidate character queues can be added to the above-mentioned cached recognition results as input data for the internal language model and the external language model in the next round. In addition, the electronic device can use the beam search method to decode the probability distribution of the category to which the next character belongs (i.e., the third probability distribution) to cache the probability distribution of the category to which the next character belongs.

[0138] In the above-mentioned speech recognition method, after the electronic device turns on the first business function, it will initialize the speech recognition model according to the configuration parameters corresponding to the first business function, and then the speech recognition model will output the speech recognition results. Because the speech recognition model is initialized according to the configuration parameters corresponding to the first business function, the speech recognition model is adapted to the current business scenario and can perform targeted speech recognition, thereby improving the accuracy of the speech recognition results. In addition, the speech recognition model can also distinguish the categories to which different characters belong, so that the output results can be closer to the potential characters of the category, which is more adapted to the current business scenario and further improves the accuracy of the speech recognition results.

[0139] The above describes in detail an example of the speech recognition method provided by the embodiment of the present application. It is understandable that, in order to implement the above functions, the electronic device includes hardware and / or software modules corresponding to the execution of each function. Those skilled in the art should easily realize that, in combination with the units and algorithm steps of each example described in the embodiments disclosed herein, the present application can be implemented in the form of hardware or a combination of hardware and computer software. Whether a function is executed in a hardware or computer software driven hardware manner depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application in combination with the embodiments, but such implementation should not be considered to be beyond the scope of the present application.

[0140] The embodiment of the present application can divide the functional modules of the electronic device according to the above method example. For example, each function can be divided into various functional modules, such as a detection unit, a processing unit, a display unit, etc., or two or more functions can be integrated into one module. The above-mentioned integrated module can be implemented in the form of hardware or in the form of a software functional module. It should be noted that the division of modules in the embodiment of the present application is schematic and is only a logical function division. There may be other division methods in actual implementation.

[0141] It should be noted that all relevant contents of each step involved in the above method embodiment can be referred to the functional description of the corresponding functional module and will not be repeated here.

[0142] The electronic device provided in this embodiment is used to execute the above-mentioned speech recognition method, and thus can achieve the same effect as the above-mentioned implementation method.

[0143] When integrated, the electronic device may also include a processing module, a storage module, and a communication module. The processing module may be used to control and manage the operation of the electronic device. The storage module may be used to support the execution of program code and data stored in the electronic device. The communication module may be used to support communication between the electronic device and other devices.

[0144] The processing module may be a processor or a controller. It may implement or execute the various exemplary logic blocks, modules, and circuits described in conjunction with the disclosure of this application. The processor may also be a combination that implements computing functions, such as a combination of one or more microprocessors, a combination of a digital signal processor (DSP) and a microprocessor, and so on. The storage module may be a memory. The communication module may specifically be a device that interacts with other electronic devices, such as a radio frequency circuit, a Bluetooth chip, or a Wi-Fi chip.

[0145] In one embodiment, when the processing module is a processor and the storage module is a memory, the electronic device involved in this embodiment may be a Figure 6 Device with the structure shown.

[0146] An embodiment of the present application further provides a computer-readable storage medium, in which a computer program is stored. When the computer program is executed by a processor, the processor executes the speech recognition method of any of the above embodiments.

[0147] The embodiment of the present application also provides a computer program product. When the computer program product is run on a computer, it enables the computer to execute the above-mentioned related steps to implement the speech recognition method in the above-mentioned embodiment.

[0148] In addition, an embodiment of the present application also provides a device, which can specifically be a chip, component or module, and the device may include a connected processor and memory; wherein the memory is used to store computer-executable instructions, and when the device is running, the processor can execute the computer-executable instructions stored in the memory to enable the chip to execute the speech recognition method in the above-mentioned method embodiments.

[0149] Among them, the electronic device, computer-readable storage medium, computer program product or chip provided in this embodiment are all used to execute the corresponding methods provided above. Therefore, the beneficial effects that can be achieved can refer to the beneficial effects in the corresponding methods provided above, and will not be repeated here.

[0150] Through the description of the above implementation methods, technical personnel in the relevant field can understand that for the convenience and simplicity of description, only the division of the above-mentioned functional modules is used as an example. In actual applications, the above-mentioned functions can be distributed and completed by different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above.

[0151] Units described as separate components may or may not be physically separate, and components shown as units may be one physical unit or multiple physical units, that is, they may be located in one place or distributed in multiple places. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0152] In addition, the functional units in the various embodiments of the present application may be integrated into a single processing unit, or each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or software functional units.

[0153] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a readable storage medium. Based on this understanding, the technical solution of the embodiment of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for enabling a device (which can be a single-chip microcomputer, chip, etc.) or a processor (processor) to execute all or part of the steps of the various embodiments of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0154] The above content is merely a specific embodiment of the present application, but the scope of protection of the present application is not limited thereto. Any changes or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in this application should be included in the scope of protection of the present application. Therefore, the scope of protection of the present application should be based on the scope of protection of the claims.

Claims

1. A speech recognition method, the method being executed by an electronic device, characterized in that: The method comprises: When the first service function is enabled, initializing a speech recognition model, wherein the speech recognition model is used to recognize characters corresponding to speech information and categories to which the characters belong; The voice information inputted through the first service function is received, and the voice information and the cached character category are inputted into the voice recognition model to obtain a voice recognition result, wherein the cached character category includes a category obtained by recognizing historical voice information.

2. The method according to claim 1, characterized in that The step of inputting the voice information and the category to which the cached characters belong into the voice recognition model to obtain a voice recognition result includes: Inputting the voice information, the cached recognition result and the category to which the cached character belongs into the voice recognition model to obtain a fusion probability distribution, wherein the cached recognition result includes text information obtained by recognizing the historical voice information; The speech recognition result is determined based on the fused probability distribution.

3. The method according to claim 2, characterized in that The speech recognition model includes a first model and a second model, and the speech information, the cached recognition result, and the category to which the cached character belongs are input into the speech recognition model to obtain a fusion probability distribution, including: Inputting the voice information into the first model to obtain a first probability distribution, wherein the first probability distribution represents a probability distribution of characters to which the voice information belongs; Inputting the cached recognition result and the category to which the cached character belongs into the second model to obtain a second probability distribution, wherein the second probability distribution represents a probability distribution of a next character predicted by the second model for the cached recognition result; The fused probability distribution is determined based on the first probability distribution and the second probability distribution.

4. The method according to claim 3, characterized in that The determining the fused probability distribution based on the first probability distribution and the second probability distribution includes: The first probability distribution and the second probability distribution are weightedly summed to obtain the fused probability distribution.

5. The method according to claim 3 or 4, characterized in that: The determining the speech recognition result based on the fusion probability distribution includes: The fused probability distribution is decoded by using a beam search method to obtain the speech recognition result.

6. The method according to claim 5, characterized in that The step of decoding the fused probability distribution in a beam search manner to obtain the speech recognition result includes: For each beam, K characters corresponding to the fusion probability distribution are selected to generate B×K candidate character queues, where B is the number of beams, and the K characters are the characters corresponding to the first K probabilities after the fusion probability distribution is sorted from large to small. Selecting B characters from the B×K candidate character queues, the B characters being characters corresponding to the first B probabilities after the probabilities corresponding to the B×K candidate character queues are sorted from large to small; The character with the highest probability among the B characters is taken as the speech recognition result.

7. The method according to claim 6, characterized in that After selecting B characters, the method further includes: The B characters are added to the cache recognition result to form a new cache recognition result.

8. The method according to any one of claims 3 to 7, characterized in that The second model also outputs a third probability distribution, which represents the probability distribution of the category to which the next character predicted by the second model belongs.

9. The method according to claim 8, characterized in that The method further comprises: The third probability distribution is decoded by using a beam search method to determine the categories corresponding to the B characters, and the categories corresponding to the B characters are added to the categories to which the cached characters belong.

10. The method according to any one of claims 3 to 9, characterized in that The first model is an RNNT acoustic model, including an encoding module, a prediction module and a joint module. The encoding module is used to encode the audio features corresponding to the speech information, the prediction module is used to process the text features corresponding to the speech information, and the joint module is used to integrate the output of the encoding module and the output of the prediction module to obtain a first probability distribution.

11. The method according to any one of claims 3 to 10, characterized in that The second model is a Transformer network structure, a long short-term memory network structure LSTM or a convolutional network structure.

12. The method according to any one of claims 1 to 11, characterized in that The first business function includes a voice assistant function, a see-and-speak function, or a scenario-based instruction function.

13. An electronic device, characterized in that: include: one or more processors; one or more memories; The memory stores one or more programs, and when the one or more programs are executed by the processor, the electronic device executes the method according to any one of claims 1 to 12.

14. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the processor is caused to perform the method according to any one of claims 1 to 12.

Images