Speech recognition method and electronic device

Abstract

Embodiments of the present application provide a speech recognition method and an electronic device. The method is performed by the electronic device and includes: in a case where a first service function is started, initializing a speech recognition model according to configuration parameters corresponding to the first service function, wherein different service functions correspond to different configuration parameters, and the speech recognition model is a model containing multiple sets of configuration parameters and trained according to training data; receiving speech information input through the first service function, and inputting the speech information into the speech recognition model to obtain a speech recognition result. The method can improve the accuracy of the speech recognition result.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of electronics, and particularly relates to a speech recognition method and an electronic device. BACKGROUND

[0002] With the continuous development of electronic devices, the rich functions of electronic devices provide great convenience for user life. For example, in the case that the electronic device opens the voice assistant function, the user can have a voice conversation with the electronic device, and the electronic device can respond to the voice instruction of the user and convert the voice instruction into text for display. For another example, in the process of the electronic device calling an opposite electronic device, if the call caption function is opened, the electronic device can display the call content in text on the display interface.

[0003] In the above scenarios, the electronic device needs to use the speech recognition function (automatic speech recognition, ASR) to recognize the voice information to convert it into corresponding text information for display. However, in the related art, the recognition accuracy of the electronic device needs to be improved for the speech recognition process in different business scenarios. SUMMARY

[0004] The present application provides a speech recognition method and an electronic device, which can improve the accuracy of the speech recognition result.

[0005] In a first aspect, the present application provides a speech recognition method, which is executed by an electronic device, and includes: in the case that a first business function is opened, initializing a speech recognition model according to configuration parameters corresponding to the first business function, wherein different business functions correspond to different configuration parameters, and the speech recognition model is a model containing multiple sets of configuration parameters which is trained according to training data; receiving voice information input through the first business function, and inputting the voice information into the speech recognition model to obtain a speech recognition result.

[0006] The first business function can include but is not limited to voice assistant, call caption, visible-to-speak, voice instruction and the like, that is, different business functions correspond to different business scenarios. When the speech recognition model is trained, different configuration parameters can be trained for different business functions according to the training data, so that in the case that the first business function is opened, the electronic device can initialize the speech recognition model according to the configuration parameters corresponding to the first business function.

[0007] The voice information received by the electronic device can include voice instructions input by a user in a voice assistant scenario, call voice of an opposite user in a call caption scenario, etc. After the electronic device inputs the voice information into a voice recognition model corresponding to the first service function, a voice recognition result can be obtained. Since the voice recognition model is initialized according to the configuration parameters corresponding to the first service function, the voice recognition model is adapted to the current service function and can perform voice recognition accordingly, thereby improving the accuracy of the voice recognition result.

[0008] In some implementations of the first aspect, the voice recognition model includes a first model, a second model, and a third model. The parameters in the first model and the parameters in the second model remain unchanged under different service functions. The voice recognition model is initialized according to the configuration parameters corresponding to the first service function, including initializing the third model according to the configuration parameters corresponding to the first service function.

[0009] The first model can be an acoustic model, the second model can be an internal language model, and the third model can be an external language model. Under the joint action of the three models, voice recognition can be achieved. Optionally, the acoustic model and the internal language model can be fixed-parameter models, i.e., the configuration parameters corresponding to different service functions are the same, while the external language model can be a variable-parameter model, i.e., the electronic device needs to initialize the external language model according to the configuration parameters corresponding to the first service function. Thus, the external language model can be adapted to the current service function and perform voice recognition accordingly, thereby improving the accuracy of the voice recognition result.

[0010] In some implementations of the first aspect, inputting the voice information into the voice recognition model to obtain the voice recognition result includes inputting the voice information and a cached recognition result into the voice recognition model to obtain a fusion probability distribution, and the cached recognition result includes text information obtained by recognizing historical voice information. The voice recognition result is determined based on the fusion probability distribution.

[0011] Since the cached recognition result includes text information obtained by recognizing historical voice information, it can assist the voice recognition model in recognizing the current voice information, thereby further improving the accuracy of the voice recognition result.

[0012] The electronic device can input the voice information and the cached recognition result into the voice recognition model to obtain a fusion probability distribution, and then determine the voice recognition result based on the fusion probability distribution. In the case where the voice recognition model includes a first model, a second model, and a third model, the first model, the second model, and the third model need to perform corresponding processing processes respectively.

[0013] In an implementation, the electronic device can input the voice information into the first model to obtain a first probability distribution, the first probability distribution representing a probability distribution of characters to which the voice information belongs. For example, for the voice "da", the first model can output, according to audio features of the voice "da" of the user, a probability of a "dǎ" character being a "dǎ" character, a probability of the "dǎ" character being a "dà" character, a probability of the "dǎ" character being a "dá" character, and so on, where the sum of the probabilities is equal to 1.

[0014] In addition, the electronic device can input the buffered recognition result into the second model to obtain a second probability distribution, the second probability distribution representing a predicted next character probability distribution of the second model for the buffered recognition result. For example, in the case of a voice instruction "turn on the flashlight" input by the user, when the recognition process of the "kāi" character is performed, it is indicated that the "dǎ" character has been recognized previously, and the historical buffered recognition result is the "dǎ" character or the text feature corresponding to the "dǎ" character. The predicted next character probability distribution can be: a probability of the "kāi" character being a "kāi" character, a probability of the "kāi" character being a "kě" character, a probability of the "kāi" character being a "kěi" character, and so on.

[0015] In addition, the electronic device can input the buffered recognition result into the third model to obtain a third probability distribution, the second probability distribution representing a predicted next character probability distribution of the third model for the buffered recognition result. The form of the third probability distribution is similar to that of the second probability distribution.

[0016] Then, the electronic device can determine a fusion probability distribution based on the first probability distribution, the second probability distribution, and the third probability distribution. In an implementation, the electronic device can perform weighted summation on the first probability distribution, the second probability distribution, and the third probability distribution to obtain the fusion probability distribution. Preferably, the electronic device can perform weighted summation on the first probability distribution and the third probability distribution, and then subtract the second probability distribution with a weight to obtain the fusion probability distribution, that is, the weights of the first probability distribution and the third probability distribution are positive numbers, and the weight of the second probability distribution is a negative number.

[0017] In combination with the first aspect, in some implementations of the first aspect, the determining the voice recognition result based on the fusion probability distribution includes: decoding the fusion probability distribution in a beam search manner to obtain the voice recognition result.

[0018] In an implementation manner, assuming that the number of beams is B, the electronic device can select K characters corresponding to the fusion probability distribution for each beam, generate a BxK candidate character queue, the K characters are characters corresponding to top K probabilities after the fusion probability distribution is sorted in descending order of probability, then select B characters in the BxK candidate character queue, the B characters are characters corresponding to top B probabilities after the BxK candidate character queue is sorted in descending order of probability, and finally the character with the largest probability (i.e., Top1 probability) in the B characters is taken as the speech recognition result. Optionally, the generated B characters can be added to the above-mentioned cache recognition result as input data of the internal language model and the external language model in the next round. Optionally, the number of beams B can be determined by the B characters selected in the last round.

[0019] Based on the above implementation manner, after the first service function is started, the electronic device initializes the speech recognition model according to the configuration parameters corresponding to the first service function, and then outputs the speech recognition result from the speech recognition model. Since the speech recognition model is initialized according to the configuration parameters corresponding to the first service function, the speech recognition model is adapted to the current service scenario and can perform targeted speech recognition, thereby improving the accuracy of the speech recognition result. At the same time, since the present application only needs to use the external language recognition model, which is only initialized according to the service function when used, the storage occupation of the electronic device can also be reduced.

[0020] In combination with the first aspect, in some implementation manners of the first aspect, the first model is an RNNT acoustic model, including an encoding module, a prediction module, and a joint module, the encoding module is configured to encode audio features corresponding to the speech information, the prediction module is configured to process text features corresponding to the speech information, and the joint module is configured to integrate the output of the encoding module and the output of the prediction module to obtain the first probability distribution.

[0021] In combination with the first aspect, in some implementation manners of the first aspect, the second model includes a feedforward neural network (FFN), a prediction module, and a joint module, the FFN is located between the prediction module and the joint module, the prediction module is configured to process the cache recognition result to obtain a first result, the FFN is configured to process the first result again to obtain a second result, and the joint module is configured to integrate the first result and the second result to obtain the second probability distribution.

[0022] In training the acoustic model, the electronic device can train the acoustic model according to training data (including audio data and text data). On the basis of the trained acoustic model, the electronic device can fix the parameters of the acoustic model, and train the acoustic model again using the training data corresponding to the acoustic model. In the training process, only the parameters of the lightweight FFN network need to be updated to obtain the internal language model (i.e., the second model). In the retraining process, the training data is input from the prediction module, and the result is output from the joint module through the calculation of the prediction module, the FFN network, and the joint module. The loss is calculated through the cross-entropy loss function to complete the training process. The training method provided in the present application can reduce the requirements for storage space and computing power, and when training the internal language model, the parameters of the acoustic model do not need to be changed, and the flexibility is better.

[0023] In combination with the first aspect, in some implementations of the first aspect, the third model is a Transformer network structure, a long short-term memory network structure LSTM, or a convolutional network structure, and the third model is a model containing multiple sets of configuration parameters trained according to training data.

[0024] Since the third model is a parameter-variable model, the electronic device needs to initialize the third model according to the configuration parameters corresponding to the business function when using the third model. Therefore, the parameters in the third model need to be trained. Alternatively, the training data can be the corresponding business text generated by the generation model according to the prompt formulated according to the business function (or business scenario). The prompt can be a manually written prompt, or a batch-produced prompt combined with scenario historical data, that is, for a business scenario, there is a set of training data. In the training process, the electronic device can also use the low-order adaptation method LoRA of the large language model to fine-tune the parameters of the third model, that is, replace the original parameter matrix with the result of multiplying the parameter matrix (such as the q, k, and v parameter matrices) in the attention module by matrix A and matrix B, and update matrix A and matrix B in the training process to complete the training process. The training method provided in the present application can adapt to multiple different business scenarios in the case of limited storage capacity and computing power, improve the accuracy of the speech recognition result, and reduce various computing costs.

[0025] In the second aspect, the present application provides a device, which is included in an electronic device and has functions to realize the behaviors of the electronic device in the above-mentioned first aspect and possible implementations of the above-mentioned first aspect. The functions can be realized by hardware or by hardware executing corresponding software. The hardware or software includes one or more modules or units corresponding to the above-mentioned functions. For example, a receiving module or unit, a processing module or unit, and the like.

[0026] In a third aspect, the present application provides an electronic device, comprising a processor, a memory and an interface; the processor, the memory and the interface cooperate with each other to enable the electronic device to perform any of the methods in the technical solutions of the first aspect.

[0027] In a fourth aspect, the present application provides a chip comprising a processor. The processor is configured to read and execute a computer program stored in a memory to perform the method in the first aspect and any possible implementation manner thereof.

[0028] Optionally, the chip further comprises a memory, and the memory is connected to the processor through a circuit or a wire.

[0029] Further optionally, the chip further comprises a communication interface.

[0030] In a fifth aspect, the present application provides a computer readable storage medium, and the computer readable storage medium stores a computer program. When the computer program is executed by a processor, the processor performs any of the methods in the technical solutions of the first aspect.

[0031] In a sixth aspect, the present application provides a computer program product, and the computer program product comprises computer program code. When the computer program code runs on an electronic device, the electronic device performs any of the methods in the technical solutions of the first aspect. BRIEF DESCRIPTION OF DRAWINGS

[0032] Figure 1 FIG. 1 is a schematic diagram of a voice assistant scenario provided by an embodiment of the present application;

[0033] Figure 2 FIG. 2 is a schematic diagram of a call caption scenario provided by an embodiment of the present application;

[0034] Figure 3 FIG. 3 is a structural schematic diagram of an electronic device provided by an embodiment of the present application;

[0035] Figure 4 FIG. 4 is a software structure block diagram of an electronic device provided by an embodiment of the present application;

[0036] Figure 5 FIG. 5 is a schematic diagram of a voice assistant function setting process provided by an embodiment of the present application;

[0037] Figure 6 FIG. 6 is a flowchart of a voice recognition method provided by an embodiment of the present application;

[0038] Figure 7 FIG. 7 is a structural schematic diagram of an acoustic model provided by an embodiment of the present application;

[0039] Figure 8FIG. 1 is a structural schematic diagram of an internal language model provided by an embodiment of the present application;

[0040] Figure 9 FIG. 2 is a structural schematic diagram of an external language model provided by an embodiment of the present application;

[0041] Figure 10 FIG. 3 is a processing process schematic diagram of a speech recognition method provided by an embodiment of the present application;

[0042] Figure 11 FIG. 4 is a process schematic diagram of a beam search method provided by an embodiment of the present application;

[0043] Figure 12 FIG. 5 is an example diagram of generating multiple service texts based on a prompt provided by an embodiment of the present application;

[0044] Figure 13 FIG. 6 is a flow schematic diagram of another speech recognition method provided by an embodiment of the present application;

[0045] Figure 14 FIG. 7 is an input and output method schematic diagram of an external language model provided by an embodiment of the present application;

[0046] Figure 15 FIG. 8 is a processing process schematic diagram of another speech recognition method provided by an embodiment of the present application. DETAILED DESCRIPTION

[0047] The technical solutions in the embodiments of the present application will be described below with reference to the accompanying drawings in the embodiments of the present application. In the description of the embodiments of the present application, unless otherwise specified, " / " represents the meaning of or, for example, A / B can represent A or B; in this document, "and / or" only describes the association relationship of the associated objects, which means that there can be three relationships, for example, A and / or B, which means that there are three cases of A alone, A and B together, and B alone. In addition, in the description of the embodiments of the present application, "multiple" means two or more than two.

[0048] Hereinafter, the terms "first", "second", "third" are only for descriptive purposes, and cannot be understood as indicating or implying relative importance or implicitly indicating the number of indicated technical features. Therefore, the features defined with "first", "second", "third" can explicitly or implicitly include one or more features.

[0049] In order to clearly describe the technical solutions of the embodiments of the present application, the following briefly introduces some terms and technologies involved in the embodiments of the present application:

[0050] (1) Voice Assistant: This is the name of a voice control function. Users can turn the voice assistant on or off on electronic devices using preset switches. After the voice assistant function is turned on, users need to wake it up first, such as by using a wake word or the power button. After the voice assistant on the electronic device is woken up, users can then speak voice commands to control the electronic device to perform corresponding service operations.

[0051] (2) Call text: This is the name of an auxiliary call function. When an electronic device answers a call, a call text control can be displayed on the incoming call interface or the call interface after the call is connected. Users can turn on the call text function to display their own and the other party's voice in text form.

[0052] With the continuous development of electronic devices, their rich functions have greatly facilitated users' lives. For example, when an electronic device has its voice assistant function enabled and activated, the user can engage in voice conversations with the device. If the user inputs a voice sentence, the electronic device can respond with a voice message. For instance, if the user inputs "What's the weather like today?", the electronic device can reply with "Today's lowest temperature is 15 degrees Celsius, and the highest temperature is 25 degrees Celsius." Alternatively, the user can also control the electronic device with voice, for example, by inputting "Open..." The electronic device can reply "okay" via voice and then enter... Interface. See also Figure 1 The voice input interface shown allows the electronic device to display the corresponding text when a user inputs "What's the weather like today?". After recognizing the voice, the device can query today's weather information online in the background and convert the corresponding text into speech for output, such as "Today's low temperature is 15 degrees Celsius, and the high temperature is 25 degrees Celsius." In other words, the electronic device needs to recognize the user's voice command and display the corresponding text information on the current interface.

[0053] As another example, during a call between two electronic devices, if the call caption function is enabled, the electronic device can display the call content as text on its display screen. For example, such as... Figure 2As shown, when the electronic device A receives the incoming call of the electronic device B, the electronic device A displays an incoming call interface, which can include information such as a user name, an incoming call number, a number home location, and an operator, and can also include an answer control 21, a reject control 22, and a first control 23. Optionally, the first control 23 can be a touch pop-up window, a card, a control, a floating ball, or other display forms on the electronic device for user interaction. Optionally, the first control 23 can include a text prompt option area for starting the "call caption" function. If the user answers the incoming call on the electronic device A and clicks the first control 23, the electronic device A can update and display the first control 23 as a first window 200, i.e., the user can directly open the call caption while answering the call. At this time, the electronic device A can convert the voice information 1 transmitted by the electronic device B into corresponding text information 1 and display the text information 1 in the first window 200, for example Figure 2 The text information 1 shown in FIG. 13B is "Hello, I am financial consultant Xiao Li". That is, the electronic device needs to recognize the call content of the opposite user by voice recognition and display the corresponding text information on the call interface. At the same time, the user can also input the reply content in the input box of the first window 200 and click the send control 24 to display the reply content.

[0054] In related technologies, the electronic device usually uses the same voice recognition model (or engine) to recognize voice information. However, due to differences in voice content and style in different business scenarios, such as instruction phrases in voice assistants, such as control instructions, navigation, and encyclopedia questions and answers, and conversational phrases in call captions, such as customer service, chat, and office communication. Therefore, if the electronic device uses the same voice recognition model for different business scenarios, the accuracy of voice recognition may be low. If different voice recognition models are developed for different business scenarios, the training efficiency is low, and considering the limitations of the computing and storage capabilities of the current electronic device, it may also not support the operation of multiple different voice recognition models.

[0055] Therefore, the embodiments of the present application provide a voice recognition method, and propose a voice recognition model construction method that can adapt to multiple business scenarios. The corresponding voice information can be automatically recognized according to different business scenarios to improve the accuracy of voice recognition in different scenarios. It can be understood that different business scenarios can also be referred to as different vertical scenarios.

[0056] It can also be understood that the speech recognition method provided by the embodiments of the present application can be applied to electronic devices with speech recognition function or service, such as mobile phones, tablet computers, wearable devices, vehicle-mounted devices, augmented reality (AR) / virtual reality (VR) devices, notebook computers, ultra-mobile personal computers (UMPCs), netbooks, personal digital assistants (PDAs), and the like. The embodiments of the present application do not make any limitation on the specific type of electronic device.

[0057] Exemplarily, Figure 3 is a structural schematic diagram of an example electronic device 100 provided by the embodiments of the present application. Taking the electronic device as a mobile phone for example, the electronic device 100 can include a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, a charging management module 140, a power management module 141, a battery 142, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a loudspeaker 170A, a receiver 170B, a microphone 170C, an earphone interface 170D, a sensor module 180, a key 190, a motor 191, an indicator 192, a camera 193, a display screen 194, and a subscriber identification module (SIM) card interface 195, and the like. The sensor module 180 can include a pressure sensor 180A, a gyroscope sensor 180B, a barometric pressure sensor 180C, a magnetic sensor 180D, an acceleration sensor 180E, a distance sensor 180F, a proximity light sensor 180G, a fingerprint sensor 180H, a temperature sensor 180J, a touch sensor 180K, an ambient light sensor 180L, a bone conduction sensor 180M, and the like.

[0058] The processor 110 can include one or more processing units, for example: the processor 110 can include an application processor (AP), a modem processor, a graphics processing unit (GPU), an image signal processor (ISP), a controller, a memory, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural-network processing unit (NPU), etc. Among them, different processing units can be independent devices, or can be integrated in one or more processors.

[0059] The processor 110 can also be provided with a memory for storing instructions and data. In some embodiments, the memory in the processor 110 is a cache memory. The memory can hold instructions or data that the processor 110 has just used or recycled. If the processor 110 needs to use the instructions or data again, it can be directly called from the memory. Avoiding repeated access reduces the waiting time of the processor 110, thus improving the efficiency of the system.

[0060] In some embodiments, the processor 110 can include one or more interfaces. The interface can include an inter-integrated circuit (I2C) interface, an inter-integrated circuit sound (I2S) interface, a pulse code modulation (PCM) interface, a universal asynchronous receiver / transmitter (UART) interface, a mobile industry processor interface (MIPI), a general-purpose input / output (GPIO) interface, a subscriber identity module (SIM) interface, and / or a universal serial bus (USB) interface, etc.

[0061] The wireless communication function of the electronic device 100 can be realized through the antenna 1, the antenna 2, the mobile communication module 150, the wireless communication module 160, the modem processor, and the baseband processor, etc.

[0062] The mobile communication module 150 can provide a solution including 2G / 3G / 4G / 5G, etc. wireless communication applied on the electronic device 100. The mobile communication module 150 can include at least one filter, a switch, a power amplifier, a low noise amplifier (LNA), etc. The mobile communication module 150 can receive electromagnetic waves by the antenna 1, and perform filtering, amplification, etc. on the received electromagnetic waves, and transfer to a modem processor for demodulation. The mobile communication module 150 can also amplify the signal modulated by the modem processor, and radiate as electromagnetic waves through the antenna 1. In some embodiments, at least part of the functional modules of the mobile communication module 150 can be disposed in the processor 110. In some embodiments, at least part of the functional modules of the mobile communication module 150 can be disposed in the same device as at least part of the modules of the processor 110.

[0063] The electronic device 100 implements a display function through a GPU, a display screen 194, and an application processor, etc. The GPU is a microprocessor for image processing, connected to the display screen 194 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. The processor 110 can include one or more GPUs that execute program instructions to generate or change display information.

[0064] The display screen 194 is used to display images, videos, etc. The display screen 194 includes a display panel. The display panel can adopt a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode (AMOLED), a flex light-emitting diode (FLED), a Miniled, a MicroLed, a Micro-oLed, a quantum dot light emitting diode (QLED), etc. In some embodiments, the electronic device 100 can include 1 or N display screens 194, and N is a positive integer greater than 1.

[0065] The electronic device 100 can implement a shooting function through an ISP, a camera 193, a video codec, a GPU, a display screen 194, and an application processor, etc.

[0066] The NPU is a neural-network (NN) calculation processor. By drawing on the structure of a biological neural network, for example, by drawing on the transmission mode between human brain neurons, the NPU can quickly process input information and can also constantly self-learn. Through the NPU, intelligent cognition and other applications of the electronic device 100 can be implemented, for example, image recognition, face recognition, speech recognition, text understanding, and the like. In embodiments of the present application, the electronic device can invoke the NPU to implement the function of a speech recognition model, to complete a speech recognition process and obtain a speech recognition result.

[0067] The internal memory 121 can be used to store computer executable program codes, and the executable program codes include instructions. The processor 110 executes various function applications and data processing of the electronic device 100 by running the instructions stored in the internal memory 121. The internal memory 121 can include a program storage area and a data storage area. The program storage area can store an operating system, at least one application program required by a function (such as a sound playing function, an image playing function, etc.), and the like. The data storage area can store data created during use of the electronic device 100 (such as audio data, a phone book, etc.), and the like. In addition, the internal memory 121 can include a high-speed random access memory, and can also include a non-volatile memory, for example, at least one magnetic disk storage device, a flash memory device, a universal flash storage (UFS), and the like.

[0068] The electronic device 100 can implement audio functions through an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, an earphone interface 170D, an application processor, and the like. For example, music playing, recording, and the like.

[0069] The microphone 170C, also known as a "microphone", "sound transducer", is used to convert a sound signal into an electrical signal. When making a phone call or sending voice information, a user can make a sound by approaching the microphone 170C with his mouth, and input the sound signal into the microphone 170C. The electronic device 100 can be provided with at least one microphone 170C. In other embodiments, the electronic device 100 can be provided with two microphones 170C, in addition to collecting sound signals, noise reduction functions can also be implemented. In other embodiments, the electronic device 100 can also be provided with three, four or more microphones 170C, to implement the functions of collecting sound signals, noise reduction, and also to identify the source of the sound, to implement the function of directional recording, and the like. In embodiments of the present application, in the scenario of a voice call function, a user can input voice instructions through the microphone to implement the input of voice information.

[0070] It can be understood that the structure shown in the embodiments of the present application does not constitute a specific limitation on the electronic device 100. In other embodiments of the present application, the electronic device 100 can include more or fewer components than shown, or combine certain components, or split certain components, or different arrangement of components. The components shown can be implemented in hardware, software or a combination of software and hardware.

[0071] It can also be understood that the above Figure 3 The structure shown is only an exemplary description when the electronic device is a mobile phone. If the electronic device is a tablet computer, PC, PDA, wearable device or other types of devices, the structure of the electronic device can include fewer structures than those shown in Figure 3 , or more structures than those shown in Figure 3 , which is not limited herein.

[0072] The software system of the electronic device 100 and the cooperation process of the software system and the hardware are described below.

[0073] The software system of the electronic device 100 can adopt a layered architecture, an event-driven architecture, a microkernel architecture, a microservice architecture, or a cloud architecture. The embodiments of the present application take the Android system with a layered architecture as an example to exemplarily describe the software structure of the electronic device 100.

[0074] Figure 4 is a software structure block diagram of the electronic device 100 of the embodiments of the present application. The layered architecture divides the software into several layers, and each layer has a clear role and division of labor. The layers communicate with each other through software interfaces. In some embodiments, the Android system is divided into four layers, from top to bottom, the application layer, the application framework layer, the Android runtime and the system library, and the kernel layer. The application layer can include a series of application packages, such as Android application packages (APKs).

[0075] As shown in Figure 4 , the application package can include applications such as voice assistants and call subtitles, and can also include an APK integrated with a speech recognition model, which can implement a speech recognition function, and different applications can call the APK integrated with the speech recognition model when needed to convert voice information into text information.

[0076] In addition, the application package can also include applications such as cameras, galleries, calendars, calls, maps, navigation, WLAN, Bluetooth, music, videos, short messages, etc. Figure 4 , which is not shown.

[0077] The application framework layer provides an application programming interface (API) and programming framework for applications in the application layer. The application framework layer includes some pre-defined functions.

[0078] As shown in Figure 4 The application framework layer can include a window manager, a content provider, a view system, a telephony manager, a resource manager, a notification manager, etc.

[0079] The window manager is used to manage window programs. The window manager can acquire the size of the display screen, determine whether there is a status bar, lock the screen, and intercept the screen, etc. The content provider is used to store and acquire data, and make the data accessible to the application program. The view system includes visual controls, such as controls for displaying text, controls for displaying pictures, etc. The view system can be used to build an application program. The telephony manager is used to provide the communication function of the electronic device 100. For example, the management of the call state (including connection, hang-up, etc.). The resource manager provides various resources for the application program, such as localized strings, icons, pictures, layout files, video files, etc. The notification manager enables the application program to display notification information in the status bar, which can be used to convey the message of the notification type. The notification manager can also be a notification in the form of a chart or a scroll bar text appearing in the top status bar of the system, or a notification in the form of a dialog window appearing on the screen.

[0080] The Android runtime includes a core library and a virtual machine. The Android runtime is responsible for the scheduling and management of the Android system.

[0081] The core library includes two parts: one part is the function function that the java language needs to call, and the other part is the core library of Android.

[0082] The application layer and the application framework layer run in the virtual machine. The virtual machine executes the java files of the application layer and the application framework layer into binary files. The virtual machine is used to perform the management of the object life cycle, the management of the stack, the management of the thread, the management of the security and the exception, and the garbage collection, etc.

[0083] The system library can include a plurality of functional modules. For example: a surface manager, media libraries, a three-dimensional graphics processing library (for example: OpenGL ES), a 2D graphics engine (for example: SGL), etc.

[0084] The surface manager is used to manage the display subsystem and provides a fusion of 2D and 3D layers for multiple applications. The media library supports playback and recording of a variety of commonly used audio, video formats, and static image files. The media library can support a variety of audio and video encoding formats. The three-dimensional graphics processing library is used to realize three-dimensional graphics drawing, image rendering, synthesis, and layer processing. The 2D graphics engine is a drawing engine for 2D drawing.

[0085] The kernel layer is a layer between hardware and software. The kernel layer at least includes display drivers, camera drivers, audio drivers, and sensor drivers.

[0086] For ease of understanding, the following embodiments of the present application will take an electronic device with the structure shown in Figure 3 and Figure 4 as an example, and the voice recognition method provided by the embodiments of the present application will be described in detail in combination with the drawings and application scenarios.

[0087] The following will take the scenario of starting the voice assistant function as an example to introduce the user setting process. Exemplarily, as shown in Figure 5 , the user clicks the setting icon on the desktop interface of the electronic device, and the electronic device can enter the setting interface. In the setting interface, there are setting options of different functions, such as WLAN option, Bluetooth option, display and brightness option, smart assistant option, etc. In the case that the user clicks the smart assistant option control 51, the electronic device can jump to the smart assistant setting interface. In the smart assistant setting interface, there are different smart functions, such as YOYO suggestion, negative screen, smart text, smart search, smart voice, etc. In the case that the user clicks the smart voice option control 52, the electronic device can jump to the smart voice setting interface. In the smart voice setting interface, there are different wake-up ways of the voice assistant, such as voice wake-up, power key wake-up, breath wake-up, earphone line control wake-up, and Bluetooth device wake-up. The user can select the corresponding wake-up way to wake up the voice assistant according to the demand. For example, in the case that the voice wake-up way is selected, the user can input "Hello, YOYO" by voice in the subsequent to wake up the voice assistant function; in the case that the power key wake-up way is selected, the user can press the power key for 1 second in the subsequent to wake up the voice assistant function. It can be understood that after the user selects the wake-up way in the smart voice setting interface, the voice assistant function is started, and then the voice recognition method of the embodiments of the present application can be used to recognize the voice information input by the user to convert the voice information into text information and display the text information on the interface.

[0088] Figure 6 is a flowchart of an example of the voice recognition method provided by the embodiments of the present application. The method can be executed by an electronic device, and specifically can include:

[0089] S101, in a case where the first service function is started, initializing the speech recognition model according to configuration parameters corresponding to the first service function.

[0090] The first service function can include, but is not limited to, a voice assistant, call captioning, speak-to-text, voice instruction, and the like. Different service functions correspond to different service scenarios. Different service functions can correspond to different configuration parameters. In a case where the first service function is started, the electronic device can initialize the speech recognition model according to configuration parameters corresponding to the first service function. It can be understood that the configuration parameters corresponding to different service functions can be obtained during training of the speech recognition model. The training process of the speech recognition model is described in detail in the following embodiments.

[0091] In some embodiments, the speech recognition model can include multiple models. Optionally, the speech recognition model can include an acoustic model, an internal language model, and an external language model, which can realize speech recognition under the joint action of the three models. Of course, the number and type of specific models are not limited in the embodiments of the present application.

[0092] Optionally, in a case where the speech recognition model includes an acoustic model, an internal language model, and an external language model, the acoustic model and the internal language model can be fixed parameter models, i.e., the configuration parameters corresponding to different service functions are the same, and the external language model can be a parameter variable model, i.e., the electronic device needs to initialize the external language model according to the configuration parameters corresponding to the first service function.

[0093] Optionally, the acoustic model can be an RNNT (recurrent neural network transducer) acoustic model. As an example, the structure of the RNNT acoustic model can be as shown in Figure 7 After receiving the user input speech information, the encoder can be used to encode the audio features corresponding to the speech information, the predictor can be used to predict the text features corresponding to the speech information, and the joint module can be used to integrate the output of the encoder and the output of the predictor to output the acoustic probability distribution. The internal language model can be improved on the basis of the acoustic model. As an example, as shown in Figure 8As shown, a lightweight feedforward neural network (FFN) can be introduced between the prediction module and the joint module of the RNNT acoustic model to obtain the internal language model. The input of the internal language model is historical text features, which are processed by the prediction module, the FFN network, and the joint module, and then output the predicted probability of the next character. It can be understood that the acoustic model and the internal language model focus more on the user's audio or timbre features.

[0094] The input to the external language model is also historical text features, and it can also output the predicted probability of the next character. Unlike the internal language model, the configuration parameters in the external language model are those corresponding to the first business function, making it more suitable for the current business function. In other words, the external language model focuses more on the business scenario. For example, the structure of the external language model can be as follows: Figure 9 As shown, this includes fixed parameters and variable parameters. Fixed parameters are those that do not change based on the business scenario (e.g., W), while variable parameters change based on the business scenario. Variable parameters can be obtained by adding the parameter matrix (e.g., the q, k, v parameter matrix) from the attention module to the trained matrix. The trained matrix can be the result of multiplying trained matrices A and B. That is, different business functions correspond to different matrices A and B. Initialization involves loading the matrices A and B corresponding to the first business function to complete the initialization. The column width of matrix A is equal to the row height of matrix B; for example, the column width of matrix A and the row height of matrix B are both r. Optionally, the external language model can be a Transformer network structure, a long short-term memory (LSTM) network structure, a convolutional network structure, etc.

[0095] Optionally, when the first business function is enabled, the application corresponding to the first business function can call the speech recognition model and read the corresponding configuration parameters to initialize the speech recognition model.

[0096] S102, input the received voice information into the voice recognition model to obtain the voice recognition result.

[0097] The voice information received by the electronic device may include voice commands input by the user in a voice assistant scenario, and voice messages from the other party in a call caption scenario.

[0098] The electronic device can obtain a speech recognition result after inputting the speech information into the speech recognition model corresponding to the first service function. Since the speech recognition model is initialized according to the configuration parameters corresponding to the first service function, the speech recognition model is adapted to the current service function, can perform targeted speech recognition, and thus improves the accuracy of the speech recognition result.

[0099] Optionally, in a case where the speech recognition model includes an acoustic model, an internal language model, and an external language model, when the electronic device inputs the received speech information into the speech recognition model, the acoustic model, the internal language model, and the external language model need to perform corresponding processing processes respectively.

[0100] Exemplarily, as shown in Figure 10 , the electronic device can input the speech information (or audio features corresponding to the speech information) into the acoustic model, and can output an acoustic probability distribution (referred to as a first probability distribution for short). The acoustic probability distribution here can be the probability of a character to which the speech belongs. For example, in a voice assistant scenario, a user inputs a voice instruction of “turn on the flashlight”, and the electronic device can input the voice corresponding to each character of “dǎ”, “kāi”, “shǒu”, “diàn”, and “tǒng” into the acoustic model in sequence, and obtain the acoustic probability distribution of each character. Optionally, the acoustic probability distribution can be understood as a word table including characters to which the speech can correspond and the probability of each character, and finally the character with the maximum probability can be selected as the recognition result. For example, for the voice “da” word, the acoustic model can output the probability that the voice “da” word belongs to the “dǎ” word as a, the probability that the voice “da” word belongs to the “dà” word as b, the probability that the voice “da” word belongs to the “dá” word as c, and so on, where the sum of the probabilities is equal to 1. For example, for the voice “kai” word, the acoustic model can also output the probability that the voice “kai” word belongs to the “kāi” word, the “kǎi” word, and the “kěi” word. Optionally, since the “kāi” word is preceded by the “dǎ” word, the acoustic model can determine the probability that the “kāi” word belongs to the “kāi” word in combination with the probability that the “dǎ” word belongs to the “dǎ” word, in addition to considering the probability that the “kāi” word belongs to the “kāi” word alone.

[0101] Continuing to refer to Figure 10 , the electronic device can input the historical buffered recognition result into the internal language model, and can output a next character probability distribution (referred to as a second probability distribution for short) predicted by the internal language model. The buffered recognition result input here can be recognized text information (or text features, hidden features, etc. corresponding to the text information), for example, in a voice assistant scenario, a user inputs a voice instruction of “turn on the flashlight”, and when the recognition process of the “kāi” word is performed, it is indicated that the “dǎ” word has been recognized previously, and thus the historical buffered recognition result is the “dǎ” word or the text features corresponding to the “dǎ” word. The predicted next character probability distribution can be that the probability that the “kāi” word belongs to the “kāi” word is d, the probability that the “kāi” word belongs to the “kǎi” word is e, the probability that the “kāi” word belongs to the “kěi” word is f, and so on.

[0102] Continuing to refer to Figure 10 , the electronic device inputs the historical cache recognition result into the external language model, and the external language model can output a predicted next character probability distribution (referred to as a third probability distribution for short). The input cache recognition result here can also be the recognized character information (or text features corresponding to the character information), and the form of the predicted next character probability distribution is similar to that of the internal language model, which will not be repeated. It can be understood that the embodiments of the present application use the historical cache recognition result to predict the probability distribution of the next character by the internal language model and the external language model to assist the acoustic model to obtain the speech recognition result, and further improve the accuracy of the speech recognition result.

[0103] Optionally, when starting to process the first character of the speech instruction input by the user, there is no corresponding historical speech recognition result in the cache recognition result, and then the internal language model and the external language model can input a preset fill character as input to output the next character probability distribution. The preset fill character here can be a character set according to the business function, or a character obtained according to experimental data, and the embodiments of the present application do not limit this.

[0104] Optionally, the embodiments of the present application do not limit the execution order of the above-mentioned acoustic model, internal language model and external language model, and can be executed in the order of acoustic model, internal language model and external language model, or in the order of acoustic model, external language model and internal language model, or can be executed simultaneously, etc.

[0105] After obtaining the first probability distribution, the second probability distribution and the third probability distribution, the electronic device can fuse the three probability distributions to obtain a fused probability distribution. Optionally, the electronic device can fuse the first probability distribution, the second probability distribution and the third probability distribution by weighted summation. Preferably, the electronic device can subtract the second probability distribution with weight after weighted summation of the first probability distribution and the third probability distribution to obtain the fused probability distribution. It can be understood that the weights corresponding to the first probability distribution, the second probability distribution and the third probability distribution can be weights obtained after a large number of experimental verification, or can be weights learned after training, and the embodiments of the present application do not limit this. Exemplarily, taking the speech "kai" as an example, after probability fusion, the fused probability distribution obtained can be: the probability of belonging to "open" is g, the probability of belonging to "Kay" is h, the probability of belonging to "Kai" is i, etc.

[0106] Then, the electronic device can decode the fused probability distribution by means of beam search. Assuming the number of beams is B, for the fused probability distribution, each beam takes the TopK of the probabilities to generate B×K candidate character queues. Then, the TopB of the probabilities is taken for the B×K candidate character queues to generate B candidate character queues. Finally, the Top1 of the probabilities is selected from the B candidate character queues to obtain the speech recognition result. Furthermore, the electronic device can display the text corresponding to the speech recognition result on the interface. Optionally, the generated B candidate character queues can be added to the above cached recognition results as input data for the internal language model and the external language model in the next round. Optionally, the number of beams B can be determined by the B candidate character queues selected in the previous round.

[0107] Exemplarily, as Figure 11 shown, assume the number of beams B = 3 and TopK is Top3; the recognition result of the previous round of the speech "da" character is "打". The fused probability distribution obtained by the electronic device in this round is that the probability of belonging to the character "开" is g, the probability of belonging to the character "凯" is h, the probability of belonging to the character "楷" is i... (arranged in descending order of probability). In the case of the number of beams being 3, the candidate character queues obtained in this round are "打开", "打凯", "打楷", and the corresponding probabilities are g, h, i respectively. Then in the next round, the fused probability distribution corresponding to the speech "shou" character is that the probability of belonging to the character "手" is t, the probability of belonging to the character "收" is m, the probability of belonging to the character "受" is n... (arranged in descending order of probability). Assume that each beam takes the Top3 of the probabilities, that is, it takes "手", "收", "受", and the corresponding probabilities are t, m, n respectively. Then, the electronic device combines "打开", "打凯", "打楷" and "手", "收", "受" to generate 3×3 candidate character queues, and the probability of each candidate character queue can be obtained. For example, the probability of "打开手" is g×t (or g + t), the probability of "打开受" is g×m (or g + m), etc. Then, the Top3 candidate character queues are selected from the 3×3 candidate character queues to generate 3 candidate character queues. For example, the Top3 candidate character queues are "打开手", "打开收", "打开受". Optionally, the electronic device can add "打开手", "打开收", "打开受" and the corresponding probabilities to the above cached recognition results. Finally, for the Top3 candidate character queues "打开手", "打开收", "打开受", the character with the Top1 probability is selected as the speech recognition result. For example, if "打开手" is the character with the highest probability, then "打开手" is the speech recognition result, and so on.

[0108] It can be understood that when the electronic device identifies a certain phonetic character, the probability ranking of the previously obtained candidate character queue may be changed after being connected with the previously obtained candidate character queue. Therefore, when the electronic device displays the text corresponding to the speech recognition result, the previously displayed text may be updated. For example, for the case that the user inputs "open the flashlight", after recognizing three phonetics "da kai shou", the character with the highest probability is "open the flashlight", that is, the electronic device displays three characters "open the flashlight"; but after recognizing four phonetics "da kai shoudian", the character with the highest probability is "open the flashlight", that is, the electronic device updates "open the flashlight" to "open the flashlight", that is, the character corresponding to the word "shou" is updated.

[0109] In the above speech recognition method, after the electronic device starts the first service function, the speech recognition model is initialized according to the configuration parameters corresponding to the first service function, and then the speech recognition result is output by the speech recognition model. Since the speech recognition model is initialized according to the configuration parameters corresponding to the first service function, the speech recognition model is adapted to the current service scenario and can perform targeted speech recognition, thereby improving the accuracy of the speech recognition result. At the same time, since the present application only needs to use one speech recognition model, which is only initialized according to the service function when used, the storage occupation of the electronic device can also be reduced.

[0110] As can be seen from the above embodiments, the configuration parameters corresponding to different service functions can be obtained during the training of the speech recognition model. In the case where the speech recognition model includes an acoustic model, an internal language model and an external language model, the training process of the speech recognition model can optionally include:

[0111] For the acoustic model, an example of its structure is shown in the above Figure 7 The electronic device can train the acoustic model according to the training data (including audio data and text data) until convergence. For the internal language model, since it is improved on the basis of the acoustic model, an example of its structure is shown in the above Figure 8 The electronic device can fix the parameters of the trained acoustic model and train the acoustic model corresponding training data. Only the parameters of the lightweight FFN network need to be updated during the training process. During the training, the training data is input from the prediction module, calculated through the prediction module, FFN network and joint module, and the output result is output from the joint module. The loss is calculated through the cross-entropy loss function to complete the training process. The training method provided by the present application can reduce the requirement for storage space and computing power, and the parameters of the acoustic model do not need to be changed during the training of the internal language model, which is more flexible.

[0112] For the external language model, an exemplary structure is as described above Figure 9 As the external language model is a parameter-variable model, the electronic device needs to initialize the external language model according to the configuration parameters corresponding to the business function when using, and then the parameters in the external language model need to be trained. Alternatively, the training data can be the corresponding business text generated by the generation model according to the prompt formulated according to the business function (or business scenario). The prompt can be a manually written prompt, or a batch-produced prompt combined with historical scenario data, that is, for a business scenario, there is a set of training data. Exemplarily, see Figure 12 That is, for the navigation scenario, when you want to turn off the traffic information, the generation model can construct a plurality of different business texts. In the training process, the electronic device can also use the low-rank adaptation of large language models (LoRA) to fine-tune the parameters of the external language model, that is, fix the fixed parameters in the external language model, replace the original parameter matrix with the result of multiplying the parameter matrix (such as q, k, v parameter matrix) in the attention module by matrix A and matrix B, and update matrix A and matrix B in the training process to complete the training process. Then, different matrix A and matrix B corresponding to different business functions can be obtained. For the training method provided in the embodiments of the present application, multiple different business scenarios can be adapted in the case of limited storage capacity and computing power, the accuracy of the speech recognition result can be improved, and various computing costs can be reduced.

[0113] It can be understood that the training process of the speech recognition model can take a long time, so the training process can be performed by a server (such as a cloud server), and after the training is completed, the speech recognition model is transplanted to the electronic device for use.

[0114] Alternatively, the execution process of the above speech recognition method can also be performed by a server, and the electronic device can send the received speech information to the server, and after the server determines the speech recognition result, the result is returned to the electronic device, and the electronic device is displayed or other processing.

[0115] In some other embodiments, for the above Figure 6In the illustrated embodiment, the speech recognition model can also only include an acoustic model and an external language model, and then the acoustic model can output an acoustic probability distribution (i.e., the first probability distribution), and the external language model can output a next character probability distribution (i.e., the third probability distribution). Then, the electronic device can fuse the first probability distribution and the third probability distribution to obtain a fused probability distribution. Alternatively, the electronic device can perform weighted summation on the first probability distribution and the third probability distribution to obtain the fused probability distribution. Finally, the electronic device can decode the fused probability distribution in a beam search manner to output a language recognition result. In this embodiment, the working principles of the acoustic model and the external language model, and the beam search manner are similar to those of the above-mentioned embodiments, and will not be described here.

[0116] In still other embodiments, for a certain service scenario, such as a voice assistant scenario, the user input is mostly instructional short voice, such as control instructions, navigation, encyclopedia questions and answers, etc. The characters or word groups in these short voices generally correspond to specific categories, such as singer names, song names, film and television drama names, city / place names, APP names, etc. Then, the electronic device can also consider the category to which each character belongs in the process of voice recognition, so as to further improve the accuracy of the voice recognition result.

[0117] Figure 13 is another flowchart of a voice recognition method provided by an embodiment of the present application. The method can be executed by an electronic device and can specifically include:

[0118] S201, in the case where the first service function is started, initializing the speech recognition model according to the configuration parameters corresponding to the first service function.

[0119] The process of initializing the speech recognition model can refer to the description of S101 step above, and the implementation principle is similar, which will not be described here. However, in the case where the speech recognition model includes an acoustic model, an internal language model and an external language model, the external language model can predict not only the next character probability distribution (i.e., the third probability distribution), but also the probability distribution of the category to which the next character belongs (referred to as the fourth probability distribution). Alternatively, the external language model in this embodiment can also be a Transformer network structure, an LSTM network structure, a convolutional network structure, etc., but the external language model in this embodiment needs to consider both character and character category data when training.

[0120] Optionally, when training the external language model, the training data can be composed of short sentences, which can include different categories of words (entity words), and the training data can be constructed by template matching, model-assisted generation, etc. For example, the training data can be "character: open WeChat, character category: 0011; character: I want to listen to Liu Dehua's Stupid Child, character category: 0002220333...". Each character corresponds to a category, and the character category identifier can be a preset identifier, such as APP name character category 1, singer name character category 2, song name character category 3, subject-predicate-verb character category 0, etc.

[0121] Based on the training data, the input and output mode of the external language model can be as shown in Figure 14 The external language model can receive a character sequence and a character category sequence as input, generate latent features of characters and character categories through character embedding layers and character category embedding layers respectively, and add them together as the latent features of the current character to be sent to the external language model for prediction calculation. Finally, the probability distribution of the next character and the probability distribution of the category to which the next character belongs are output. In addition, the electronic device can calculate a loss function according to the output result to complete the training process of the external language model.

[0122] S202, input the received voice information into the speech recognition model to obtain a fusion probability distribution and a probability distribution of the category to which the next character belongs.

[0123] For example, as shown in Figure 15 When the speech recognition model includes an acoustic model, an internal language model, and an external language model, the electronic device can input the voice information (or audio features corresponding to the voice information) into the acoustic model, which can output an acoustic probability distribution (i.e., a first probability distribution). In addition, the historical buffered recognition results are input into the internal language model, which can output a next character probability distribution predicted by the internal language model (i.e., a second probability distribution). In addition, the historical buffered recognition results and the historical character category are input into the external language model, which can output a next character probability distribution predicted by the external language model (i.e., a third probability distribution) and a probability distribution of the category to which the next character belongs (i.e., a fourth probability distribution). Optionally, the predicted probability distribution of the category to which the next character belongs can be: the probability of belonging to category 0 is x, the probability of belonging to category 1 is y, the probability of belonging to category 2 is z, etc.

[0124] Then, the electronic device can fuse the first probability distribution, the second probability distribution, and the third probability distribution to obtain a fusion probability distribution. The fusion method can refer to the description of S102 in the above embodiments, which will not be repeated here.

[0125] S203, determine the speech recognition result according to the fusion probability, and cache the probability distribution of the category to which the next character belongs.

[0126] After obtaining the fusion probability, continuing to refer to Figure 15 , the electronic device can decode the fusion probability distribution in a beam search manner. Assuming that the number of beams is B, for the fusion probability distribution, each beam takes the TopK of the probability to generate BxK candidate character queues, and then takes the TopB of the probability from the BxK candidate character queues to generate B candidate character queues, and finally takes the Top1 of the probability from the B candidate character queues, that is, the speech recognition result is obtained, and then the electronic device can display the text corresponding to the speech recognition result on the interface. Alternatively, the generated B candidate character queues can be added to the above-mentioned cached recognition result as input data of the internal language model and the external language model in the next round.

[0127] In addition, the electronic device can decode the probability distribution of the category to which the next character belongs (i.e., the fourth probability distribution) in a beam search manner. Assuming that the number of beams is B, for the probability distribution of the category to which the next character belongs, each beam takes the Top1 of the probability to generate Bx1 candidate character category queues, and then searches for the categories corresponding to the above-mentioned B candidate character queues in the Bx1 candidate character category queues to generate B candidate character category queues, and adds the B candidate character category queues to the above-mentioned historical character categories as input data of the external language model in the next round.

[0128] Exemplarily, assuming that the number of beams is 1, for the speech instruction of “turn on the flashlight”, if the character category of “turn on” is “00”, the probability of the predicted category “1” to which the next character belongs is the highest (i.e., Top1), then the category “1” can be taken as the category to which the next character belongs. In addition, in the case where “1” represents the category of the APP name, the electronic device can also directly predict that the category to which the next character belongs should also be “1”, and so on, until the character category corresponding to the APP name is predicted.

[0129] It can be understood that the historical cached recognition result has a corresponding relationship with the historical character category, for example, the historical cached recognition result stores the “turn on” character, and the historical character category stores the category to which “turn on” belongs.

[0130] The voice recognition method, after the electronic device starts the first service function, the voice recognition model is initialized according to the configuration parameters corresponding to the first service function, and then the voice recognition result is output by the voice recognition model. Because the voice recognition model is initialized according to the configuration parameters corresponding to the first service function, the voice recognition model is adapted to the current service scenario, and can be targeted for voice recognition, thereby improving the accuracy of the voice recognition result. Moreover, the voice recognition model can also distinguish the categories to which different characters belong, can make the output result approach the potential characters of the category, and is more suitable for the current service scenario, further improving the accuracy of the voice recognition result.

[0131] The above describes an example of the voice recognition method provided by the embodiments of the present application. It can be understood that the electronic device includes hardware and / or software modules corresponding to each function to implement the above functions. Those skilled in the art should easily realize that, in combination with the units and algorithm steps of each example described in the embodiments disclosed herein, the present application can be realized in the form of hardware or a combination of hardware and computer software. Whether a certain function is executed in hardware or computer software driven hardware depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application in combination with the embodiments, but such implementation should not be considered beyond the scope of the present application.

[0132] The embodiments of the present application can divide the function modules of the electronic device according to the above method examples. For example, each function module such as a detection unit, a processing unit, and a display unit can be divided, or two or more functions can be integrated into one module. The integrated module can be realized in the form of hardware or software function module. It should be noted that the division of modules in the embodiments of the present application is illustrative, and is only a logical function division. Actual implementation can have another division method.

[0133] It should be noted that all related contents of each step involved in the above method embodiments can be cited to the function description of the corresponding function module, which will not be repeated here.

[0134] The electronic device provided by the embodiments of the present application is used to execute the above voice recognition method, and thus can achieve the same effect as the above implementation method.

[0135] In the case of integrated units, the electronic device can further include a processing module, a storage module, and a communication module. The processing module can be used to control and manage the actions of the electronic device. The storage module can be used to support the electronic device to execute program codes and data. The communication module can be used to support the communication between the electronic device and other devices.

[0136] The processing module can be a processor or a controller. It can implement or execute various exemplary logical blocks, modules, and circuits described in combination with the disclosure of the present application. The processor can also be a combination of computing functions, such as a combination of one or more microprocessors, a combination of a digital signal processor (DSP) and a microprocessor, and the like. The storage module can be a memory. The communication module can be a device for interacting with other electronic devices, such as a radio frequency circuit, a Bluetooth chip, a Wi-Fi chip, and the like.

[0137] In one embodiment, when the processing module is a processor and the storage module is a memory, the electronic device involved in the embodiment can be a device with the structure shown in the figure. Figure 3

[0138] The embodiments of the present application also provide a computer readable storage medium, which stores a computer program. When the computer program is executed by a processor, the processor executes the voice recognition method of any one of the above embodiments.

[0139] The embodiments of the present application also provide a computer program product, which, when running on a computer, causes the computer to execute the above related steps to implement the voice recognition method in the above embodiments.

[0140] In addition, the embodiments of the present application also provide a device, which can be a chip, a component or a module. The device can include a processor and a memory connected to each other. The memory is used to store computer execution instructions. When the device is running, the processor can execute the computer execution instructions stored in the memory to enable the chip to execute the voice recognition method in the above method embodiments.

[0141] The electronic device, the computer readable storage medium, the computer program product or the chip provided by the embodiments of the present application are used to execute the corresponding methods provided above, so the beneficial effects that can be achieved are referred to the beneficial effects of the corresponding methods provided above, which will not be described here.

[0142] Through the description of the above embodiments, those skilled in the art can understand that, for the convenience and brevity of description, only the division of the above functional modules is taken as an example for illustration. In actual application, the above functions can be completed by different functional modules according to needs, that is, the internal structure of the device is divided into different functional modules to complete all or part of the functions described above.

[0143] ​The units described as separate components may or may not be physically separate, and the components displayed as units may be a physical unit or multiple physical units, that is, may be located in one place, or also can be distributed to multiple different places. Part or all of the units can be selected according to actual needs to achieve the purpose of the embodiment scheme.

[0144] In addition, each functional unit in each embodiment of the present application can be integrated in one processing unit, or each unit can be physically present alone, or two or more units can be integrated in one unit. The integrated unit can be realized in the form of hardware or in the form of a software functional unit.

[0145] The integrated unit, if realized in the form of a software functional unit and sold or used as an independent product, can be stored in a readable storage medium. Based on such understanding, the technical scheme of the embodiments of the present application essentially or the part that contributes to the prior art or the whole or part of the technical scheme can be embodied in the form of a software product, which is stored in a storage medium and includes a number of instructions to make a device (which can be a single-chip microcomputer, a chip, etc.) or a processor execute all or part of the steps of the method of each embodiment of the present application. The foregoing storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, and various program code storage media.

[0146] The above is only a specific embodiment of the present application, but the protection scope of the present application is not limited thereto, any person skilled in the art can easily think of changes or replacements within the technical scope disclosed in the present application, which should be covered in the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims

1. A voice recognition method, the method being performed by an electronic device, and the method comprising: The method comprises: in the case where the first service function is started, initializing a speech recognition model according to configuration parameters corresponding to the first service function, wherein different service functions correspond to different configuration parameters, the speech recognition model is a model containing multiple sets of configuration parameters obtained by training training data, and the speech recognition model comprises a first model, a second model and a third model; receiving speech information input through the first service function, and inputting the speech information into the speech recognition model to obtain a speech recognition result; wherein the inputting of the speech information into the speech recognition model to obtain a speech recognition result comprises: inputting the speech information into the first model to obtain a first probability distribution, the first probability distribution representing the probability distribution of the character to which the speech information belongs; inputting a cache recognition result into the second model to obtain a second probability distribution, the second probability distribution representing the next character probability distribution predicted by the second model for the cache recognition result, and the cache recognition result comprising text information obtained by recognizing historical speech information; inputting the cache recognition result into the third model to obtain a third probability distribution, the third probability distribution representing the next character probability distribution predicted by the third model for the cache recognition result; determining a fusion probability distribution based on the first probability distribution, the second probability distribution and the third probability distribution; determining the speech recognition result based on the fusion probability distribution.

2. The method of claim 1, wherein, The parameters in the first model and the parameters in the second model remain unchanged under different service functions, and the initialization of the speech recognition model according to the configuration parameters corresponding to the first service function comprises: initializing the third model according to the configuration parameters corresponding to the first service function.

3. The method of claim 1, wherein, The determination of the fusion probability distribution based on the first probability distribution, the second probability distribution and the third probability distribution comprises: weighting and summing the first probability distribution, the second probability distribution and the third probability distribution to obtain the fusion probability distribution.

4. The method of claim 3, wherein, The weights of the first probability distribution and the third probability distribution are positive numbers, and the weight of the second probability distribution is a negative number.

5. The method of claim 1, wherein, The determination of the speech recognition result based on the fusion probability distribution comprises: decoding the fusion probability distribution in a beam search manner to obtain the speech recognition result.

6. The method of claim 5, wherein, The decoding of the fusion probability distribution in a beam search manner to obtain the speech recognition result comprises: selecting K characters corresponding to the fusion probability distribution for each beam to generate B×K candidate character queues, B being the number of beams, and the K characters being the characters corresponding to the first K probabilities after the fusion probability distribution is sorted in descending order of probability; selecting B characters in the B×K candidate character queues, the B characters being the characters corresponding to the first B probabilities after the B×K candidate character queues are sorted in descending order of probability; taking the character with the largest probability in the B characters as the speech recognition result.

7. The method of claim 6, wherein, After the B characters are selected, the method further includes: adding the B characters into the cache recognition result to form a new cache recognition result.

8. The method according to any one of claims 1 to 7, characterized in that, The first model is an RNNT acoustic model, which includes an encoding module, a prediction module, and a joint module. The encoding module is configured to encode audio features corresponding to the voice information. The prediction module is configured to process text features corresponding to the voice information. The joint module is configured to integrate an output of the encoding module and an output of the prediction module to obtain a first probability distribution.

9. The method of claim 8, wherein, The second model includes a feed-forward neural network (FFN), the prediction module, and the joint module. The FFN is located between the prediction module and the joint module. The prediction module is configured to process the cache recognition result to obtain a first result. The FFN is configured to process the first result again to obtain a second result. The joint module is configured to integrate the first result and the second result to obtain a second probability distribution.

10. The method according to any one of claims 1 to 7, characterized in that, The third model is a Transformer network structure, a long short-term memory (LSTM) network structure, or a convolutional network structure. The third model is a model including multiple sets of configuration parameters, which is trained according to training data.

11. The method of claim 10, wherein, The training data includes text information generated according to a prompt corresponding to the first service function.

12. The method according to any one of claims 1 to 7, characterized in that, The first service function includes a voice assistant function or a call caption function.

13. An electronic device, comprising: comprise: one or more processors; one or more memories; The memory stores one or more programs, which, when executed by the processor, cause the electronic device to perform the method of any one of claims 1 to 12.

14. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, which, when executed by a processor, causes the processor to perform the method of any one of claims 1 to 12.

Images