Speech recognition method and electronic equipment

Abstract

The embodiment of the invention provides a voice recognition method and electronic equipment, and the method is executed by the electronic equipment, and comprises the steps: carrying out the initialization of a voice recognition model according to a configuration parameter corresponding to a first service function under the condition that the first service function is started, and enabling different service functions to correspond to different configuration parameters, the voice recognition model is a model which is obtained by training according to training data and comprises multiple groups of configuration parameters; and receiving voice information input through the first service function, and inputting the voice information into the voice recognition model to obtain a voice recognition result. The method can improve the accuracy of the speech recognition result.

Description

Technical Field

[0001] The present application relates to the field of electronic technology, and in particular to a speech recognition method and electronic equipment. Background Art

[0002] With the continuous development of electronic devices, the rich functions of electronic devices have provided great convenience for users' lives. For example, when the voice assistant function of an electronic device is turned on, the user can have a voice conversation with the electronic device, and the electronic device can respond to the user's voice command and convert the voice command into text for display. For another example, during a call between an electronic device and the other electronic device, if the call subtitle function is turned on, the electronic device can display the call content in text on the display interface.

[0003] In the above scenarios, electronic devices need to use automatic speech recognition (ASR) to recognize voice information and convert it into corresponding text information for display. However, in the related art, the recognition accuracy of electronic devices for speech recognition processes in different business scenarios needs to be improved. Summary of the invention

[0004] The present application provides a speech recognition method and an electronic device, which can improve the accuracy of speech recognition results.

[0005] In a first aspect, the present application provides a speech recognition method, which is executed by an electronic device, and includes: when a first business function is turned on, initializing a speech recognition model according to configuration parameters corresponding to the first business function, wherein different business functions correspond to different configuration parameters, and the speech recognition model is a model trained according to training data and includes multiple groups of configuration parameters; receiving voice information input through the first business function, and inputting the voice information into the speech recognition model to obtain a speech recognition result.

[0006] The first business function may include but is not limited to voice assistant, call captions, see and speak, voice commands and other functions, that is, different business functions correspond to different business scenarios. When training the speech recognition model, different configuration parameters can be trained for different business functions according to the training data. Then, when the first business function is turned on, the electronic device can initialize the speech recognition model according to the configuration parameters corresponding to the first business function.

[0007] The voice information received by the electronic device may include voice commands input by the user in the voice assistant scenario, the call voice of the other user in the call subtitle scenario, etc. After the electronic device inputs the voice information into the voice recognition model corresponding to the first business function, it can obtain a voice recognition result. Because the voice recognition model is initialized according to the configuration parameters corresponding to the first business function, the voice recognition model is adapted to the current business function and can perform targeted voice recognition, thereby improving the accuracy of the voice recognition result.

[0008] In combination with the first aspect, in some implementations of the first aspect, the above-mentioned speech recognition model includes a first model, a second model and a third model, the parameters in the first model and the parameters in the second model remain unchanged under different business functions, and the speech recognition model is initialized according to the configuration parameters corresponding to the first business function, including: initializing the third model according to the configuration parameters corresponding to the first business function.

[0009] Among them, the first model can be an acoustic model, the second model can be an internal language model, and the third model can be an external language model. Speech recognition can be achieved under the joint action of these three models. Optionally, the acoustic model and the internal language model can be fixed parameter models, that is, the configuration parameters corresponding to different business functions are the same, and the external language model can be a variable parameter model, that is, the electronic device needs to initialize the external language model according to the configuration parameters corresponding to the first business function. In this way, the external language model can be adapted to the current business function and perform speech recognition in a targeted manner, thereby improving the accuracy of the speech recognition results.

[0010] In combination with the first aspect, in some implementations of the first aspect, the above-mentioned inputting voice information into the voice recognition model to obtain the voice recognition result includes: inputting the voice information and the cached recognition result into the voice recognition model to obtain a fused probability distribution, the cached recognition result includes text information obtained by recognizing historical voice information; and determining the voice recognition result based on the fused probability distribution.

[0011] Among them, since the cached recognition results include text information obtained by recognizing historical voice information, it can assist the voice recognition model to recognize current voice information, so as to further improve the accuracy of the voice recognition results.

[0012] The electronic device can input the voice information and the cached recognition results into the above-mentioned voice recognition model to obtain a fused probability distribution, and then determine the voice recognition result based on the fused probability distribution. In the case where the voice recognition model includes the first model, the second model and the third model, the first model, the second model and the third model need to perform corresponding processing processes respectively.

[0013] In one implementation, the electronic device may input voice information into a first model to obtain a first probability distribution, which represents the probability distribution of the characters to which the voice information belongs. For example, for the voice character "da", the first model may output, based on the audio features of the user's voice character "da", that the probability of it belonging to the character "打 (dǎ)" is a, the probability of it belonging to the character "大 (dà)" is b, the probability of it belonging to the character "答 (dā)" is c, and so on, where the sum of the respective probabilities is equal to 1.

[0014] In addition, the electronic device may input the cached recognition result into a second model to obtain a second probability distribution, which represents the probability distribution of the next character predicted by the second model for the cached recognition result. For example, when the voice command input by the user is "turn on the flashlight", when the recognition process reaches the character "开 (kāi)", it indicates that the character "打 (dǎ)" has been recognized previously. Then the historical cached recognition result is the character "打 (dǎ)" or the text feature corresponding to the character "打 (dǎ)", and the predicted probability distribution of the next character may be: the probability of belonging to the character "开 (kāi)" is d, the probability of belonging to the character "凯 (kǎi)" is e, the probability of belonging to the character "楷 (kǎi)" is f, and so on.

[0015] In addition, the electronic device may input the cached recognition result into a third model to obtain a third probability distribution, which represents the probability distribution of the next character predicted by the third model for the cached recognition result. Among them, the form of the third probability distribution is similar to the form of the second probability distribution.

[0016] Then, the electronic device may determine a fused probability distribution based on the first probability distribution, the second probability distribution, and the third probability distribution. In one implementation, the electronic device may perform weighted summation on the first probability distribution, the second probability distribution, and the third probability distribution to obtain the fused probability distribution. Preferably, the electronic device may perform weighted summation on the first probability distribution and the third probability distribution and then subtract the weighted second probability distribution to obtain the fused probability distribution, that is, the weights of the first probability distribution and the third probability distribution are positive, and the weight of the second probability distribution is negative.

[0017] Combined with the first aspect, in some implementations of the first aspect, determining the speech recognition result based on the fused probability distribution includes: decoding the fused probability distribution by means of beam search to obtain the speech recognition result.

[0018] In one implementation, assuming that the number of beams is B, the electronic device can select K characters corresponding to the fusion probability distribution for each beam, generate B×K candidate character queues, K characters are the characters corresponding to the first K probabilities after the fusion probability distribution is sorted from large to small according to probability (that is, the TopK of probability), and then select B characters from the B×K candidate character queues, B characters are the characters corresponding to the first B probabilities after the probabilities corresponding to the B×K candidate character queues are sorted from large to small (that is, the TopB of probability), and finally use the character with the highest probability among the B characters (that is, the Top1 of probability) as the speech recognition result. Optionally, the generated B characters can be added to the above-mentioned cached recognition results as input data for the internal language model and the external language model in the next round. Optionally, the number of beams B can be determined by the B characters selected in the previous round.

[0019] Based on the above implementation, after the electronic device turns on the first business function, it will initialize the speech recognition model according to the configuration parameters corresponding to the first business function, and then the speech recognition model will output the speech recognition result. Because the speech recognition model is initialized according to the configuration parameters corresponding to the first business function, the speech recognition model is adapted to the current business scenario, and can perform targeted speech recognition, thereby improving the accuracy of the speech recognition result. At the same time, since the present application only needs to use an external language recognition model, it only needs to initialize its parameters according to the business function when in use, so it can also reduce the storage occupation of the electronic device.

[0020] In combination with the first aspect, in some implementations of the first aspect, the first model is an RNNT acoustic model, including an encoding module, a prediction module and a joint module, the encoding module is used to encode audio features corresponding to the speech information, the prediction module is used to process text features corresponding to the speech information, and the joint module is used to integrate the output of the encoding module and the output of the prediction module to obtain a first probability distribution.

[0021] In combination with the first aspect, in some implementations of the first aspect, the second model includes a feedforward neural network FFN, a prediction module and a joint module, the FFN is located between the prediction module and the joint module, the prediction module is used to process the cache recognition result to obtain a first result, the FFN is used to process the first result again to obtain a second result, and the joint module is used to integrate the first result and the second result to obtain a second probability distribution.

[0022] Among them, when the acoustic model is trained, the electronic device can train it according to the training data (including audio data and text data). Based on the trained acoustic model, the electronic device can fix the parameters of the acoustic model and use the training data corresponding to the acoustic model for training again. During the training process, only the parameters of the lightweight FFN network need to be updated to obtain the internal language model (ie, the second model). When training again, the training data is input from the prediction module, and after calculation by the prediction module, the FFN network, and the joint module, the result is output from the joint module, and the loss is calculated by the cross entropy loss function to complete the training process. For the training method provided in this application, the requirements for storage space and computing power can be reduced, and when the internal language model is trained, the parameters of the acoustic model do not need to be changed, and the flexibility is better.

[0023] In combination with the first aspect, in some implementations of the first aspect, the third model is a Transformer network structure, a long short-term memory network structure LSTM or a convolutional network structure, and the third model is a model trained according to training data and contains multiple sets of configuration parameters.

[0024] Because the third model is a model with variable parameters, the electronic device needs to initialize the third model according to the configuration parameters corresponding to the business function when in use, so it is necessary to train the parameters in the third model. Optionally, the training data can be the corresponding business text generated by the generation model according to the prompt formulated by the business function (or business scenario). The prompt can be a manually written prompt or a prompt produced in batches in combination with the scene historical data, that is, for a business scenario, there is a set of training data. During the training process, the electronic device can also use the low-order adaptation method LoRA of the large language model to fine-tune the parameters of the third model, that is, fix the fixed parameters in the third model, and replace the original parameter matrix by the parameter matrix in the attention module (such as q, k, v parameter matrix) plus the result of multiplying matrix A and matrix B, and update matrix A and matrix B during the training process to complete the training process. For the training method provided in this application, it can adapt to multiple different business scenarios under limited storage capacity and computing power, improve the accuracy of speech recognition results, and reduce various computing costs.

[0025] In a second aspect, the present application provides a device, which is included in an electronic device, and the device has the function of implementing the electronic device behavior in the above-mentioned first aspect and the possible implementation of the above-mentioned first aspect. The function can be implemented by hardware, or it can be implemented by hardware executing corresponding software. The hardware or software includes one or more modules or units corresponding to the above-mentioned functions. For example, a receiving module or unit, a processing module or unit, etc.

[0026] In a third aspect, the present application provides an electronic device, comprising: a processor, a memory, and an interface; the processor, the memory, and the interface cooperate with each other so that the electronic device executes any one of the methods in the technical solution of the first aspect.

[0027] In a fourth aspect, the present application provides a chip, including a processor. The processor is used to read and execute a computer program stored in a memory to perform the method in the first aspect and any possible implementation thereof.

[0028] Optionally, the chip also includes a memory, and the memory is connected to the processor via a circuit or wire.

[0029] Further optionally, the chip also includes a communication interface.

[0030] In a fifth aspect, the present application provides a computer-readable storage medium, in which a computer program is stored. When the computer program is executed by a processor, the processor executes any one of the methods in the technical solution of the first aspect.

[0031] In a sixth aspect, the present application provides a computer program product, the computer program product comprising: a computer program code, when the computer program code runs on an electronic device, the electronic device executes any one of the methods in the technical solution of the first aspect. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] Figure 1 is a schematic diagram of a voice assistant scenario provided in an embodiment of the present application;

[0033] Figure 2 is a schematic diagram of a call subtitle scenario provided by an embodiment of the present application;

[0034] Figure 3 is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application;

[0035] Figure 4 is a software structure block diagram of an electronic device provided in an embodiment of the present application;

[0036] Figure 5 This is a schematic diagram of a setting process of a voice assistant function provided in an embodiment of the present application;

[0037] Figure 6 is a flowchart of a speech recognition method provided in an embodiment of the present application;

[0038] Figure 7 is a schematic diagram of the structure of an acoustic model provided in an embodiment of the present application;

[0039] Figure 8is a schematic diagram of the structure of an internal language model provided in an embodiment of the present application;

[0040] Fig. 9 is a schematic diagram of the structure of an external language model provided in an embodiment of the present application;

[0041] Fig.10 is a schematic diagram of a processing process of a speech recognition method provided in an embodiment of the present application;

[0042] Fig.11 This is a schematic diagram of a beam search process provided by an embodiment of the present application;

[0043] Fig.12 This is an example diagram of generating multiple business texts based on prompt provided in an embodiment of the present application;

[0044] Fig.13 is a flowchart of another speech recognition method provided in an embodiment of the present application;

[0045] Fig.14 This is a schematic diagram of an input and output method of an external language model provided in an embodiment of the present application;

[0046] Fig.15 This is a schematic diagram of the processing process of another speech recognition method provided in an embodiment of the present application. DETAILED DESCRIPTION

[0047] The technical solutions in the embodiments of the present application will be described below in conjunction with the drawings in the embodiments of the present application. In the description of the embodiments of the present application, unless otherwise specified, " / " means or, for example, A / B can mean A or B; "and / or" in this article is only a description of the association relationship of associated objects, indicating that there can be three relationships, for example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. In addition, in the description of the embodiments of the present application, "multiple" means two or more than two.

[0048] In the following, the terms "first", "second", and "third" are used for descriptive purposes only and are not to be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Thus, a feature defined as "first", "second", and "third" may explicitly or implicitly include one or more of the features.

[0049] In order to clearly describe the technical solutions of the embodiments of the present application, some of the terms and technologies involved in the embodiments of the present application are briefly introduced below:

[0050] (1) Voice assistant: is the name of a voice control function. Users can turn on or off the voice assistant through a preset switch on the electronic device. After the voice assistant function is turned on, the user needs to wake up the voice assistant first, for example, by waking up with a wake-up word or by pressing the power button. After the voice assistant of the electronic device is awakened, the user can speak voice commands to control the electronic device to perform the corresponding service operation.

[0051] (2) Call subtitles: is the name of a call-assisting function. When an electronic device receives a call from the other party, a call subtitle control can be displayed on the incoming call interface or the call interface after the call is connected. The user can turn on the call subtitle function to display the voice of himself and the other party in text form.

[0052] With the continuous development of electronic devices, the rich functions of electronic devices have provided great convenience for users' lives. For example, when the voice assistant function of an electronic device is turned on and the voice assistant is awakened, the user can have a voice conversation with the electronic device. If the user inputs a voice, the electronic device can make a voice reply based on the voice. For example, if the user inputs "What's the weather like today", the electronic device can make a voice reply "Today's minimum temperature is 15 degrees Celsius and the maximum temperature is 25 degrees Celsius"; or, the user can also control the electronic device by voice, for example, if the user inputs "Turn on the weather", the electronic device can make a voice reply based on the voice. ”, the electronic device can reply “OK” by voice and enter Interface. Figure 1 In the voice input interface shown, when the user inputs "how is the weather today" by voice, the electronic device can display the text corresponding to the voice. After the electronic device recognizes the voice, it can query today's weather information through the network in the background, and convert the text corresponding to the weather information into voice output, for example, the voice output is "today's minimum temperature is 15 degrees Celsius, and the maximum temperature is 25 degrees Celsius". In other words, the electronic device needs to recognize the voice command input by the user and display the corresponding text information on the current interface.

[0053] For example, during a call between an electronic device and another electronic device, if the call subtitle function is turned on, the electronic device can display the call content in text on the display interface. Figure 2As shown, when electronic device A receives an incoming call from electronic device B, electronic device A displays an incoming call interface, which may include information such as user name, caller number, number location, and operator, and may also include an answer control 21, a reject control 22, and a first control 23. Optionally, the first control 23 may be a display form for user interaction on electronic devices such as a touch pop-up window, a card, a control, or a floating ball. Optionally, the first control 23 may include a text prompt option area for turning on the "call subtitles" function. If the user answers the call on electronic device A and clicks on the first control 23, electronic device A may update the first control 23 to display as a first window 200, that is, the user can directly turn on the call subtitles while answering the call. At this time, electronic device A may convert the voice message 1 transmitted by electronic device B into a corresponding text message 1, and display the text message 1 in the first window 200, for example Figure 2 The text message 1 shown in the figure is "Hello, I am Xiao Li, the financial advisor". That is, the electronic device needs to recognize the content of the call of the other user by voice and display the corresponding text message on the call interface. At the same time, the user can also input the reply content into the input box of the first window 200 and click the send control 24 to display the reply content.

[0054] In related technologies, electronic devices usually use the same speech recognition model (or engine) to recognize voice information. However, due to differences in voice content and style in different business scenarios, for example, voice assistant scenarios are mostly short, command-like voices, such as control commands, navigation, encyclopedia questions and answers, etc., and call subtitle scenarios are mostly conversational field voices, such as customer service, chat, office communication, etc. Then, if electronic devices use the same speech recognition model for different business scenarios, it may result in low accuracy of speech recognition. If different speech recognition models are developed for different business scenarios, their training efficiency is low, and considering the limitations of the computing power and storage capacity of current electronic devices, it may not be able to support the operation of multiple different speech recognition models.

[0055] In view of this, the embodiment of the present application provides a speech recognition method, and proposes a speech recognition model construction method that can adapt to multiple business scenarios, which can automatically identify corresponding speech information according to different business scenarios to improve the accuracy of speech recognition in different scenarios. It can be understood that different business scenarios can also be called different vertical domain scenarios.

[0056] It can also be understood that the speech recognition method provided in the embodiments of the present application can be applied to mobile phones, tablet computers, wearable devices, vehicle-mounted devices, augmented reality (AR) / virtual reality (VR) devices, laptop computers, ultra-mobile personal computers (UMPC), netbooks, personal digital assistants (PDA), and other electronic devices with speech recognition functions or services. The embodiments of the present application do not impose any restrictions on the specific types of electronic devices.

[0057] For example, Figure 3 1 is a schematic diagram of the structure of an example of an electronic device 100 provided in an embodiment of the present application. Taking the electronic device as a mobile phone as an example, the electronic device 100 may include a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, a charging management module 140, a power management module 141, a battery 142, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, an earphone interface 170D, a sensor module 180, a button 190, a motor 191, an indicator 192, a camera 193, a display screen 194, and a subscriber identification module (SIM) card interface 195, etc. The sensor module 180 may include a pressure sensor 180A, a gyroscope sensor 180B, an air pressure sensor 180C, a magnetic sensor 180D, an acceleration sensor 180E, a distance sensor 180F, a proximity light sensor 180G, a fingerprint sensor 180H, a temperature sensor 180J, a touch sensor 180K, an ambient light sensor 180L, a bone conduction sensor 180M, etc.

[0058] The processor 110 may include one or more processing units, for example, the processor 110 may include an application processor (AP), a modem processor, a graphics processor (GPU), an image signal processor (ISP), a controller, a memory, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural-network processing unit (NPU), etc. Different processing units may be independent devices or integrated into one or more processors.

[0059] The processor 110 may also be provided with a memory for storing instructions and data. In some embodiments, the memory in the processor 110 is a cache memory. The memory may store instructions or data that the processor 110 has just used or cyclically used. If the processor 110 needs to use the instruction or data again, it may be directly called from the memory. This avoids repeated access, reduces the waiting time of the processor 110, and thus improves the efficiency of the system.

[0060] In some embodiments, the processor 110 may include one or more interfaces. The interface may include an inter-integrated circuit (I2C) interface, an inter-integrated circuit sound (I2S) interface, a pulse code modulation (PCM) interface, a universal asynchronous receiver / transmitter (UART) interface, a mobile industry processor interface (MIPI), a general-purpose input / output (GPIO) interface, a subscriber identity module (SIM) interface, and / or a universal serial bus (USB) interface, etc.

[0061] The wireless communication function of the electronic device 100 can be implemented through the antenna 1, the antenna 2, the mobile communication module 150, the wireless communication module 160, the modem processor and the baseband processor.

[0062] The mobile communication module 150 can provide solutions for wireless communications including 2G / 3G / 4G / 5G, etc., applied to the electronic device 100. The mobile communication module 150 may include at least one filter, a switch, a power amplifier, a low noise amplifier (LNA), etc. The mobile communication module 150 can receive electromagnetic waves from the antenna 1, and filter, amplify, and process the received electromagnetic waves, and transmit them to the modulation and demodulation processor for demodulation. The mobile communication module 150 can also amplify the signal modulated by the modulation and demodulation processor, and convert it into electromagnetic waves for radiation through the antenna 1. In some embodiments, at least some of the functional modules of the mobile communication module 150 can be set in the processor 110. In some embodiments, at least some of the functional modules of the mobile communication module 150 can be set in the same device as at least some of the modules of the processor 110.

[0063] The electronic device 100 implements the display function through a GPU, a display screen 194, and an application processor. The GPU is a microprocessor for image processing, which connects the display screen 194 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. The processor 110 may include one or more GPUs that execute program instructions to generate or change display information.

[0064] The display screen 194 is used to display images, videos, etc. The display screen 194 includes a display panel. The display panel can be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode or an active-matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (FLED), Miniled, MicroLed, Micro-oLed, a quantum dot light-emitting diode (QLED), etc. In some embodiments, the electronic device 100 may include 1 or N display screens 194, where N is a positive integer greater than 1.

[0065] The electronic device 100 can realize the shooting function through ISP, camera 193, video codec, GPU, display screen 194 and application processor.

[0066] NPU is a neural network (NN) computing processor. By drawing on the structure of biological neural networks, such as the transmission mode between neurons in the human brain, it can quickly process input information and can also continuously self-learn. NPU can realize applications such as intelligent cognition of electronic device 100, such as image recognition, face recognition, speech recognition, text understanding, etc. In the embodiment of the present application, the electronic device can call NPU to implement the function of the speech recognition model to complete the speech recognition process and obtain the speech recognition result.

[0067] The internal memory 121 can be used to store computer executable program codes, and the executable program codes include instructions. The processor 110 executes various functional applications and data processing of the electronic device 100 by running the instructions stored in the internal memory 121. The internal memory 121 may include a program storage area and a data storage area. Among them, the program storage area may store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc. The data storage area may store data created during the use of the electronic device 100 (such as audio data, a phone book, etc.), etc. In addition, the internal memory 121 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, a universal flash storage (UFS), etc.

[0068] The electronic device 100 can implement audio functions such as music playing and recording through the audio module 170, the speaker 170A, the receiver 170B, the microphone 170C, the headphone jack 170D, and the application processor.

[0069] Microphone 170C, also called "microphone" or "microphone", is used to convert sound signals into electrical signals. When making a call or sending a voice message, the user can speak by putting their mouth close to the microphone 170C to input the sound signal into the microphone 170C. The electronic device 100 can be provided with at least one microphone 170C. In other embodiments, the electronic device 100 can be provided with two microphones 170C, which can not only collect sound signals but also realize noise reduction function. In other embodiments, the electronic device 100 can also be provided with three, four or more microphones 170C to collect sound signals, reduce noise, identify the source of sound, realize directional recording function, etc. In an embodiment of the present application, for example, in the scenario of voice call function, the user can input voice commands through the microphone to realize the input of voice information.

[0070] It is to be understood that the structure illustrated in the embodiment of the present application does not constitute a specific limitation on the electronic device 100. In other embodiments of the present application, the electronic device 100 may include more or fewer components than shown in the figure, or combine some components, or split some components, or arrange the components differently. The components shown in the figure may be implemented in hardware, software, or a combination of software and hardware.

[0071] It is also understandable that the above Figure 3 The electronic device shown is only an exemplary description when it is a mobile phone. If the electronic device is a tablet computer, PC, PDA, wearable device or other types of devices, the structure of the electronic device may include more Figure 3 The structure shown in the figure is less than Figure 3 More structures are shown in the figure, which are not limited here.

[0072] The following describes the software system of the electronic device 100 and the coordination process between the software system and hardware.

[0073] The software system of the electronic device 100 may adopt a layered architecture, an event-driven architecture, a micro-core architecture, a micro-service architecture, or a cloud architecture. The embodiment of the present application takes the Android system of the layered architecture as an example to exemplify the software structure of the electronic device 100.

[0074] Figure 4 1 is a software structure diagram of the electronic device 100 of the embodiment of the present application. The layered architecture divides the software into several layers, each layer has a clear role and division of labor. The layers communicate with each other through software interfaces. In some embodiments, the Android system is divided into four layers, from top to bottom, namely, the application layer, the application framework layer, the Android runtime (Android runtime) and the system library, and the kernel layer. The application layer can include a series of application packages, such as the Android application package (Android application package, APK).

[0075] like Figure 4 As shown, the application package may include applications such as voice assistants and call subtitles, and may also include an APK integrated with a speech recognition model. The APK integrated with a speech recognition model can realize the speech recognition function, and different applications can call the APK integrated with the speech recognition model when needed to convert voice information into text information.

[0076] In addition, the application package can also include applications such as camera, gallery, calendar, call, map, navigation, WLAN, Bluetooth, music, video, short message, etc. Figure 4 Not shown in FIG.

[0077] The application framework layer provides an application programming interface (API) and a programming framework for the applications in the application layer. The application framework layer includes some predefined functions.

[0078] like Figure 4 As shown, the application framework layer may include a window manager, a content provider, a view system, a phone manager, a resource manager, a notification manager, and the like.

[0079] The window manager is used to manage window programs. The window manager can obtain the size of the display screen, determine whether there is a status bar, lock the screen, capture the screen, etc. The content provider is used to store and obtain data and make the data accessible to applications. The view system includes visual controls, such as controls for displaying text, controls for displaying pictures, etc. The view system can be used to build applications. The phone manager is used to provide communication functions for the electronic device 100. For example, the management of call status (including connected, hung up, etc.). The resource manager provides various resources to the application, such as localized strings, icons, pictures, layout files, video files, etc. The notification manager enables the application to display notification information in the status bar, which can be used to convey notification-type messages. The notification manager can also be a notification that appears in the system top status bar in the form of a chart or scroll bar text, or a notification that appears on the screen in the form of a dialog window.

[0080] Android runtime includes core libraries and virtual machines. Android runtime is responsible for scheduling and management of the Android system.

[0081] The core library consists of two parts: one part is the function that needs to be called by the Java language, and the other part is the Android core library.

[0082] The application layer and the application framework layer run in a virtual machine. The virtual machine executes the Java files of the application layer and the application framework layer as binary files. The virtual machine is used to perform functions such as object life cycle management, stack management, thread management, security and exception management, and garbage collection.

[0083] The system library may include multiple functional modules, such as surface manager, media libraries, 3D graphics processing library (such as OpenGL ES), 2D graphics engine (such as SGL), etc.

[0084] The surface manager is used to manage the display subsystem and provide fusion of 2D and 3D layers for multiple applications. The media library supports playback and recording of multiple common audio and video formats, as well as static image files. The media library can support multiple audio and video encoding formats. The 3D graphics processing library is used to implement 3D graphics drawing, image rendering, synthesis, and layer processing. The 2D graphics engine is a drawing engine for 2D drawing.

[0085] The kernel layer is the layer between hardware and software. The kernel layer contains at least display driver, camera driver, audio driver, and sensor driver.

[0086] For ease of understanding, the following embodiments of the present application will be described in detail with Figure 3 and Figure 4 Taking the electronic device with the structure shown as an example, the speech recognition method provided in the embodiment of the present application is specifically described in combination with the accompanying drawings and application scenarios.

[0087] The following uses the scenario of enabling the voice assistant function as an example to introduce the user setting process. Figure 5 As shown, the user clicks the setting icon on the desktop interface of the electronic device to enable the electronic device to enter the setting interface. In the setting interface, there are setting options for different functions, such as WLAN options, Bluetooth options, display and brightness options, smart assistant options, etc. When the user clicks the smart assistant option control 51, the electronic device can jump to the smart assistant setting interface. In the smart assistant setting interface, there are different smart functions, such as YOYO suggestions, negative one screen, smart text, smart search, smart voice, etc. When the user clicks the smart voice option control 52, the electronic device can jump to the smart voice setting interface. In the smart voice setting interface, there are different wake-up methods for the voice assistant, such as voice wake-up, power button wake-up, breath wake-up, headphone line control wake-up and Bluetooth device wake-up. The user can choose the corresponding wake-up method to wake up the voice assistant according to the needs. For example, when the voice wake-up method is selected, the subsequent user can voice input "Hello, YOYO" to wake up the voice assistant function; when the power button wake-up method is selected, the subsequent user can press the power button for 1 second to wake up the voice assistant function. It can be understood that after the user selects the wake-up method in the smart voice setting interface, the voice assistant function is turned on. The voice recognition method of the embodiment of the present application can then be used to recognize the voice information input by the user to convert the voice information into text information and display it on the interface.

[0088] Figure 6 : is a flow chart of a speech recognition method provided in an embodiment of the present application. The method can be executed by an electronic device and may specifically include:

[0089] S101, when a first service function is enabled, initializing a speech recognition model according to configuration parameters corresponding to the first service function.

[0090] Among them, the first business function may include but is not limited to functions such as voice assistant, call subtitles, visible and speaking, voice commands, etc., that is, different business functions correspond to different business scenarios. Different business functions may correspond to different configuration parameters. Then, when the first business function is turned on, the electronic device can initialize the speech recognition model according to the configuration parameters corresponding to the first business function. It can be understood that the configuration parameters corresponding to different business functions can be obtained by training during the speech recognition model training process. The training process of the speech recognition model is described in detail in the following embodiment.

[0091] In some embodiments, the speech recognition model may include multiple models. Optionally, the speech recognition model may include an acoustic model, an internal language model, and an external language model, and speech recognition can be achieved under the joint action of these three models. Of course, the embodiments of the present application do not limit the specific number and type of models.

[0092] Optionally, in the case where the speech recognition model includes an acoustic model, an internal language model and an external language model, the acoustic model and the internal language model can be models with fixed parameters, that is, the configuration parameters corresponding to different business functions are the same, and the external language model can be a model with variable parameters, that is, the electronic device needs to initialize the external language model according to the configuration parameters corresponding to the first business function.

[0093] Optionally, the acoustic model may be a RNNT (recurrent neural network transducer) acoustic model. Exemplarily, the structure of the RNNT acoustic model may be as follows: Figure 7 As shown, it includes but is not limited to an encoding module (encoder), a prediction module (predict) and a joint module (join); after receiving the voice information input by the user, the encoding module can be used to encode the audio features corresponding to the voice information, the prediction module can be used to predict the text features corresponding to the voice information, and the joint module can be used to integrate the output of the encoding module and the output of the prediction module to output an acoustic probability distribution. The internal language model can be improved on the basis of the acoustic model. For example, Figure 8As shown, a lightweight feedforward neural network (FFN) can be introduced between the prediction module and the joint module of the RNNT acoustic model to obtain an internal language model. The input of the internal language model is the historical text features. After being processed by the prediction module, the FFN network, and the joint module, the predicted probability of the next character is output. It can be understood that the acoustic model and the internal language model focus more on the audio or voice characteristics of the user.

[0094] The input of the external language model is also historical text features, and it can also output the predicted probability of the next character. Unlike the internal language model, the configuration parameters in the external language model are parameters corresponding to the first business function, which is more suitable for the current business function, that is, the external language model pays more attention to the business scenario. For example, the structure of the external language model can be as follows: Fig. 9 As shown, it includes fixed parameters and variable parameters, wherein the fixed parameters are parameters that do not change according to the business scenario (for example, W), and the variable parameters are parameters that change according to the business scenario. For variable parameters, it can be obtained by adding the parameter matrix in the attention module (for example, q, k, v parameter matrix) to the matrix obtained by training. Here, the matrix obtained by training can be the result of multiplying the matrix A and matrix B obtained by training, that is, different business functions correspond to different matrices A and B. When initializing, the matrix A and matrix B corresponding to the first business function are loaded to complete the initialization. Among them, the column width of matrix A is equal to the row height of matrix B, for example, the column width of matrix A and the row height of matrix B are both r. Optionally, the external language model can be a Transformer network structure, a long short-term memory network structure (longshort-term memory, LSTM), a convolutional network structure, and the like.

[0095] Optionally, when the first business function is enabled, the speech recognition model may be called by an application corresponding to the first business function, and the application may read corresponding configuration parameters to initialize the speech recognition model.

[0096] S102, inputting the received voice information into a voice recognition model to obtain a voice recognition result.

[0097] Among them, the voice information received by the electronic device may include voice commands input by the user in the voice assistant scenario, the call voice of the other user in the call subtitle scenario, etc.

[0098] After the electronic device inputs the voice information into the voice recognition model corresponding to the first service function, a voice recognition result can be obtained. Since the voice recognition model is initialized according to the configuration parameters corresponding to the first service function, the voice recognition model is adapted to the current service function and can perform voice recognition specifically, thereby improving the accuracy of the voice recognition result.

[0099] Optionally, when the voice recognition model includes an acoustic model, an internal language model, and an external language model, when the electronic device inputs the received voice information into the voice recognition model, the acoustic model, the internal language model, and the external language model need to perform corresponding processing procedures respectively.

[0100] Exemplarily, as Fig.10 shown, when the electronic device inputs the voice information (or the audio feature corresponding to the voice information) into the acoustic model, an acoustic probability distribution (referred to as the first probability distribution for short) can be output. Here, the acoustic probability distribution can be the probability of the character to which the voice belongs. For example, in the voice assistant scenario, if the voice command input by the user is "turn on the flashlight", the electronic device can input the voices corresponding to "turn", "on", "hand", "electric", and "flashlight" into the acoustic model in sequence to obtain the acoustic probability distribution of each character. Optionally, the acoustic probability distribution can be understood as a word list, which includes the characters that the voice may correspond to and the probability of each character. Finally, the character with the highest probability can be selected as the recognition result. For example, for the voice "da", the acoustic model outputs the probability that it belongs to the character "打" as a, the probability that it belongs to the character "大" as b, the probability that it belongs to the character "答" as c, etc., where the sum of the probabilities is equal to 1. Another example is for the voice "kai", the acoustic model can also output the probabilities that it belongs to the characters "开", "凯", and "楷" respectively. Optionally, since the character "开" is preceded by the character "打", in addition to considering the probability of the character "开" alone, the acoustic model can also combine the probability of the character "打" to jointly determine the probability of the character to which the voice "打开" belongs.

[0101] Continuing to refer to Fig.10 , when the electronic device inputs the historical cached recognition result into the internal language model, the probability distribution of the next character predicted by the internal language model (referred to as the second probability distribution for short) can be output. The cached recognition result input here can be the recognized text information (or the text feature, hidden feature, etc. corresponding to the text information). For example, in the voice assistant scenario, if the voice command input by the user is "turn on the flashlight", when it comes to the recognition process of the character "开", it means that the character "打" has been recognized before. Then the historical cached recognition result is the character "打" or the text feature corresponding to the character "打", and the probability distribution of the next character predicted can be: the probability that it belongs to the character "开" is d, the probability that it belongs to the character "凯" is e, the probability that it belongs to the character "楷" is f, etc.

[0102] See more Fig.10 , the electronic device inputs the historical cached recognition results into the external language model, and can output the next character probability distribution predicted by the external language model (referred to as the third probability distribution). The cached recognition results input here can also be the recognized text information (or the text features corresponding to the text information). The form of the predicted next character probability distribution is similar to the form of the next character probability distribution predicted by the internal language model, and will not be repeated. It can be understood that the embodiment of the present application predicts the probability distribution of the next character through the internal language model and the external language model through the historical cached recognition results, so as to assist the above-mentioned acoustic model in obtaining the speech recognition results, and further improve the accuracy of the speech recognition results.

[0103] Optionally, when starting to process the first character voice in the voice command input by the user, if there is no corresponding historical voice recognition result in the cached recognition result, the internal language model and the external language model can use the preset filler character as input to output the next character probability distribution. The preset filler character here can be a character set according to the business function, or a character obtained according to experimental data, which is not limited in the embodiment of the present application.

[0104] Optionally, the embodiments of the present application do not limit the execution order of the above-mentioned acoustic model, internal language model and external language model. They can be executed in the order of acoustic model, internal language model, external language model, or in the order of acoustic model, external language model, internal language model. The acoustic model, internal language model and external language model can also be executed simultaneously, and so on.

[0105] After obtaining the first probability distribution, the second probability distribution and the third probability distribution, the electronic device can fuse the three probability distributions to obtain a fused probability distribution. Optionally, the electronic device can fuse the first probability distribution, the second probability distribution and the third probability distribution by weighted summation. Preferably, the electronic device can perform weighted summation on the first probability distribution and the third probability distribution, and then subtract the second probability distribution with weights to obtain a fused probability distribution. It can be understood that the weights corresponding to the first probability distribution, the second probability distribution and the third probability distribution can be weights obtained after a large number of experimental verifications, or can be weights learned after training, and the embodiments of the present application do not limit this. Exemplarily, taking the phonetic word "kai" as an example, after probability fusion, the fused probability distribution obtained can be: the probability of belonging to the word "kai" is g, the probability of belonging to the word "kai" is h, the probability of belonging to the word "kai" is i, and so on.

[0106] Then, the electronic device can decode the fused probability distribution by means of beam search. Assuming the number of beams is B, for the fused probability distribution, each beam takes the TopK of the probabilities to generate B×K candidate character queues. Then, the TopB of the probabilities is taken from the B×K candidate character queues to generate B candidate character queues. Finally, the Top1 of the probabilities is selected from the B candidate character queues, that is, the speech recognition result is obtained. Furthermore, the electronic device can display the text corresponding to the speech recognition result on the interface. Optionally, the B candidate character queues generated can be added to the above-mentioned cached recognition results as the input data of the internal language model and the external language model in the next round. Optionally, the number of beams B can be determined by the B candidate character queues selected in the previous round.

[0107] Exemplarily, as Fig.11 shown, assume the number of beams B = 3 and TopK is Top3; the recognition result of the word "da" in the previous round of speech is "打". In this round, the fused probability distribution obtained by the electronic device shows that the probability of belonging to the word "开" is g, the probability of belonging to the word "凯" is h, the probability of belonging to the word "楷" is i... (arranged in descending order of probability). In the case of the number of beams being 3, the candidate character queues obtained in this round are "打开", "打凯", "打楷", and the corresponding probabilities are g, h, i respectively. Then, in the next round, the fused probability distribution corresponding to the word "shou" in the speech shows that the probability of belonging to the word "手" is t, the probability of belonging to the word "收" is m, the probability of belonging to the word "受" is n... (arranged in descending order of probability). Assume that each beam takes the Top3 of the probabilities, that is, it takes "手", "收", "受", and the corresponding probabilities are t, m, n respectively. Then, the electronic device combines "打开", "打凯", "打楷" and "手", "收", "受" to generate 3×3 candidate character queues, and the probability of each candidate character queue can be obtained. For example, the probability of "打开手" is g×t (or g + t), the probability of "打开受" is g×m (or g + m), and so on. Then, the candidate character queues with the Top3 of the probabilities are taken from the 3×3 candidate character queues to generate 3 candidate character queues. For example, the Top3 candidate character queues are "打开手", "打开收", "打开受". Optionally, the electronic device can add "打开手", "打开收", "打开受" and the corresponding probabilities to the above-mentioned cached recognition results. Finally, for the Top3 candidate character queues "打开手", "打开收", "打开受", the character with the Top1 of the probabilities is selected as the speech recognition result. For example, if "打开手" is the character with the highest probability, then "打开手" is the speech recognition result, and so on.

[0108] It can be understood that when an electronic device recognizes a certain voice character, after connecting it to the previously obtained candidate character queue, it may change the probability ranking of the previously obtained candidate character queue. Then, when the electronic device displays the text corresponding to the voice recognition result, the previously displayed text will be updated. For example, in the case of user voice input "turn on the flashlight", after recognizing the three voices of "da kai shou", the character with the highest probability is determined to be "kai shou", that is, the electronic device displays the three characters of "kai shou"; but after recognizing the four voices of "da kai shoudian", the character with the highest probability is determined to be "kai shoudian", then the electronic device will update "kai shoudian" to "kai shoudian", that is, the character corresponding to the word "shou" is updated.

[0109] In the above-mentioned speech recognition method, after the electronic device turns on the first business function, it will initialize the speech recognition model according to the configuration parameters corresponding to the first business function, and then the speech recognition model will output the speech recognition result. Because the speech recognition model is initialized according to the configuration parameters corresponding to the first business function, the speech recognition model is adapted to the current business scenario, and can perform targeted speech recognition, thereby improving the accuracy of the speech recognition result. At the same time, since the embodiment of the present application only needs to use one speech recognition model, it only needs to initialize its parameters according to the business function when in use, so the storage occupation of the electronic device can also be reduced.

[0110] It can be seen from the above embodiments that the configuration parameters corresponding to different business functions can be obtained by training in the speech recognition model training process. In the case where the speech recognition model includes an acoustic model, an internal language model and an external language model, the training process of the speech recognition model can optionally include:

[0111] For the acoustic model, its structure is exemplarily as described above. Figure 7 As shown, the electronic device can train the acoustic model according to the training data (including audio data and text data) until convergence. As for the internal language model, since it is improved on the basis of the acoustic model, the structure is exemplarily as described above. Figure 8 As shown, the electronic device can fix the parameters of the trained acoustic model and use the training data corresponding to the acoustic model for training. During the training process, only the parameters of the lightweight FFN network need to be updated; wherein, during training, the training data is input from the prediction module, and after calculation by the prediction module, the FFN network, and the joint module, the result is output from the joint module, and the loss is calculated by the cross entropy loss function to complete the training process. For the training method provided in the embodiment of the present application, the requirements for storage space and computing power can be reduced, and when the internal language model is trained, the parameters of the acoustic model do not need to be changed, and the flexibility is better.

[0112] For the external language model, its structure is exemplarily as described above Fig. 9 As shown, since the external language model is a model with variable parameters, the electronic device needs to initialize the external language model according to the configuration parameters corresponding to the business function when in use, then the parameters in the external language model need to be trained. Optionally, the training data can be the corresponding business text generated by the generation model according to the prompt formulated by the business function (or business scenario). The prompt can be a manually written prompt or a prompt mass-produced in combination with the scene historical data, that is, for one business scenario, there is a corresponding set of training data. Exemplarily, an example of a generation model generating multiple business texts based on a prompt can be seen in Fig.12 , that is, for the navigation scenario, when you want to turn off the traffic information, the generation model can construct a variety of different business texts. During the training process, the electronic device can also use the low-rank adaptation of large language models (LoRA) to fine-tune the parameters of the external language model, that is, fix the fixed parameters in the external language model, and replace the original parameter matrix with the parameter matrix in the attention module (such as q, k, v parameter matrix) plus the result of multiplying matrix A and matrix B, and update matrix A and matrix B during the training process to complete the training process. Then, different matrices A and matrix B corresponding to different business functions can be obtained in the end. For the training method provided in the embodiment of the present application, it is possible to adapt to multiple different business scenarios under limited storage capacity and computing power, improve the accuracy of speech recognition results, and reduce various computing costs.

[0113] It is understandable that since the training process of the speech recognition model may take a long time, the training process can be performed by a server (such as a cloud server), and the speech recognition model can be transplanted to the electronic device for use after the training is completed.

[0114] Optionally, the execution process of the above-mentioned speech recognition method can also be executed by a server, and the electronic device can send the received voice information to the server. After the server determines the speech recognition result, it returns the result to the electronic device, which is displayed or otherwise processed by the electronic device.

[0115] In other embodiments, for the above Figure 6In the embodiment shown, the speech recognition model may also include only an acoustic model and an external language model, so the acoustic model can output an acoustic probability distribution (i.e., a first probability distribution), and the external language model can output a next character probability distribution (i.e., a third probability distribution). Then, the electronic device can fuse the first probability distribution and the third probability distribution to obtain a fused probability distribution. Optionally, the electronic device can perform a weighted summation of the first probability distribution and the third probability distribution to obtain a fused probability distribution. Finally, the electronic device can decode the fused probability distribution using a beam search method and output a speech recognition result. The working principle of the acoustic model and the external language model in this embodiment, and the beam search method are similar to those in the above-mentioned embodiment, and will not be repeated here.

[0116] In some other embodiments, for a certain business scenario, such as a voice assistant scenario, because the user input is mostly short command-like voice, such as control commands, navigation, encyclopedia questions and answers, etc., the characters or phrases in these short voices generally correspond to specific categories, such as singer names, song names, film and television drama names, city / place names, APP names and other categories. In this case, the electronic device can also consider the category to which each character belongs during the voice recognition process to further improve the accuracy of the voice recognition results.

[0117] Fig.13 : is a flowchart of another example of a speech recognition method provided in an embodiment of the present application. The method can be executed by an electronic device and may specifically include:

[0118] S201, when a first service function is enabled, initializing a speech recognition model according to configuration parameters corresponding to the first service function.

[0119] Among them, the process of initializing the speech recognition model can refer to the description of the above-mentioned step S101, and its implementation principle is similar and will not be repeated here. However, in the case where the speech recognition model includes an acoustic model, an internal language model and an external language model, the external language model can not only predict the probability distribution of the next character (i.e., the third probability distribution), but also predict the probability distribution of the category to which the next character belongs (referred to as the fourth probability distribution). Optionally, the external language model in this embodiment can also be a Transformer network structure, an LSTM network structure, a convolutional network structure, etc., but the external language model in this embodiment needs to take into account both character and character category data during training.

[0120] Optionally, when training an external language model, the training data may consist of short sentences, which may include words of different categories (entity words), and the training data may be constructed by template matching, model-assisted generation, etc. Exemplarily, the training data may be "character: open WeChat, character category: 0011; character: I want to listen to Andy Lau's Stupid Child, character category: 0002220333...", where each character corresponds to a category, and the identifier of the character category may be a preset identifier, such as the character category of the APP name is 1, the character category of the singer name is 2, the character category of the song name is 3, the character category of the subject, predicate, verb, etc. is 0, and so on.

[0121] Based on the training data, for example, the input and output of the external language model can be as follows Fig.14 As shown, the external language model can receive a character sequence and a character category sequence as input, generate latent features of the character and the character category through the character embedding layer and the character category embedding layer respectively, and add the two as the latent features of the current character to be sent to the external language model for prediction calculation, and finally output the probability distribution of the next character and the probability distribution of the category to which the next character belongs respectively. And, the electronic device can calculate the loss function according to the output result to complete the training process of the external language model.

[0122] S202, inputting the received voice information into a voice recognition model to obtain a fusion probability distribution and a probability distribution of the category to which the next character belongs.

[0123] For example, Fig.15 As shown, in the case where the speech recognition model includes an acoustic model, an internal language model and an external language model, the electronic device can input the speech information (or the audio features corresponding to the speech information) into the acoustic model, and can output the acoustic probability distribution (i.e., the first probability distribution). And the historical cached recognition results are input into the internal language model, and the probability distribution of the next character predicted by the internal language model (i.e., the second probability distribution) can be output. And the historical cached recognition results and the historical character categories are input into the external language model, and the probability distribution of the next character predicted by the external language model (i.e., the third probability distribution) and the probability distribution of the category to which the next character belongs (i.e., the fourth probability distribution) can be output. Optionally, the predicted probability distribution of the category to which the next character belongs can be: the probability of belonging to category 0 is x, the probability of belonging to category 1 is y, the probability of belonging to category 2 is z, and so on.

[0124] Then, the electronic device may fuse the first probability distribution, the second probability distribution and the third probability distribution to obtain a fused probability distribution. The fusion method here can refer to the description of step S102 in the above embodiment, which will not be repeated here.

[0125] S203, determining the speech recognition result according to the fusion probability, and caching the probability distribution of the category to which the next character belongs.

[0126] After obtaining the fusion probability, continue to refer to Fig.15 , the electronic device can decode the fused probability distribution by beam search. Assuming that the number of beams is B, for the fused probability distribution, each beam takes the TopK of probability to generate B×K candidate character queues, and then takes the TopB of probability for the B×K candidate character queues to generate B candidate character queues. Finally, the Top1 of probability is selected for the B candidate character queues to obtain the speech recognition result, and then the electronic device can display the text corresponding to the speech recognition result on the interface. Optionally, the generated B candidate character queues can be added to the above-mentioned cached recognition results as input data for the internal language model and the external language model in the next round.

[0127] In addition, the electronic device can use a beam search method to decode the probability distribution of the category to which the next character belongs (i.e., the fourth probability distribution). Assuming that the number of beams is B, for the probability distribution of the category to which the next character belongs, each beam takes the Top1 of the probability to generate B×1 candidate character category queues, and then searches the B×1 candidate character category queues for the categories corresponding to the above B candidate character queues to generate B candidate character category queues, and adds the B candidate character category queues to the above historical character categories as input data for the external language model in the next round.

[0128] For example, assuming that the number of beams is 1, for the voice command "turn on the flashlight", if the character category of the word "turn on" is "00", the probability of category "1" in the predicted next character category is the highest (i.e. Top1), then category "1" can be used as the category of the next character. In addition, when "1" represents the category of the APP name, the electronic device can also directly predict that the category of the next character should also be "1", until the character category corresponding to the APP name is predicted.

[0129] It can be understood that the historical cache recognition results have a corresponding relationship with the historical character categories. For example, if the historical cache recognition results store the character "open", the historical character categories store the category to which "open" belongs.

[0130] In the above-mentioned speech recognition method, after the electronic device turns on the first business function, it will initialize the speech recognition model according to the configuration parameters corresponding to the first business function, and then the speech recognition model will output the speech recognition result. Because the speech recognition model is initialized according to the configuration parameters corresponding to the first business function, the speech recognition model is adapted to the current business scenario, and can perform targeted speech recognition, thereby improving the accuracy of the speech recognition results. In addition, the speech recognition model can also distinguish the categories to which different characters belong, so that the output results can be closer to the potential characters of the category, which is more suitable for the current business scenario, further improving the accuracy of the speech recognition results.

[0131] The above describes in detail an example of a speech recognition method provided by an embodiment of the present application. It is understandable that, in order to implement the above functions, the electronic device includes hardware and / or software modules corresponding to the execution of each function. Those skilled in the art should easily realize that, in combination with the units and algorithm steps of each example described in the embodiments disclosed herein, the present application can be implemented in the form of hardware or a combination of hardware and computer software. Whether a function is executed in the form of hardware or computer software driving hardware depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application in combination with the embodiments, but such implementation should not be considered to exceed the scope of the present application.

[0132] The embodiment of the present application can divide the functional modules of the electronic device according to the above method example. For example, each function can be divided into various functional modules, such as a detection unit, a processing unit, a display unit, etc., or two or more functions can be integrated into one module. The above integrated module can be implemented in the form of hardware or in the form of a software functional module. It should be noted that the division of modules in the embodiment of the present application is schematic and is only a logical function division. There may be other division methods in actual implementation.

[0133] It should be noted that all relevant contents of each step involved in the above method embodiment can be referred to the functional description of the corresponding functional module and will not be repeated here.

[0134] The electronic device provided in this embodiment is used to execute the above-mentioned speech recognition method, and thus can achieve the same effect as the above-mentioned implementation method.

[0135] In the case of an integrated unit, the electronic device may further include a processing module, a storage module and a communication module. The processing module may be used to control and manage the actions of the electronic device. The storage module may be used to support the electronic device to execute stored program codes and data, etc. The communication module may be used to support the electronic device to communicate with other devices.

[0136] Among them, the processing module can be a processor or a controller. It can implement or execute various exemplary logic boxes, modules and circuits described in conjunction with the disclosure of this application. The processor can also be a combination that implements computing functions, such as a combination of one or more microprocessors, a combination of a digital signal processor (DSP) and a microprocessor, and so on. The storage module can be a memory. The communication module can specifically be a device that interacts with other electronic devices, such as a radio frequency circuit, a Bluetooth chip, a Wi-Fi chip, etc.

[0137] In one embodiment, when the processing module is a processor and the storage module is a memory, the electronic device involved in this embodiment may be a Figure 3 Device of the structure shown.

[0138] An embodiment of the present application also provides a computer-readable storage medium, in which a computer program is stored. When the computer program is executed by a processor, the processor executes the speech recognition method of any of the above embodiments.

[0139] The embodiment of the present application also provides a computer program product. When the computer program product is run on a computer, the computer is caused to execute the above-mentioned related steps to implement the speech recognition method in the above-mentioned embodiment.

[0140] In addition, an embodiment of the present application also provides a device, which can specifically be a chip, component or module, and the device may include a connected processor and memory; wherein the memory is used to store computer-executable instructions, and when the device is running, the processor can execute the computer-executable instructions stored in the memory so that the chip executes the speech recognition method in the above-mentioned method embodiments.

[0141] Among them, the electronic device, computer-readable storage medium, computer program product or chip provided in this embodiment are all used to execute the corresponding methods provided above. Therefore, the beneficial effects that can be achieved can refer to the beneficial effects in the corresponding methods provided above and will not be repeated here.

[0142] Through the description of the above implementation methods, technical personnel in the relevant field can understand that for the convenience and simplicity of description, only the division of the above-mentioned functional modules is used as an example. In actual applications, the above-mentioned functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above.

[0143] The units described as separate components may or may not be physically separated, and the components shown as units may be one physical unit or multiple physical units, that is, they may be located in one place or distributed in multiple different places. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0144] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0145] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a readable storage medium. Based on this understanding, the technical solution of the embodiment of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, which is stored in a storage medium, including several instructions to enable a device (which can be a single-chip microcomputer, chip, etc.) or a processor (processor) to perform all or part of the steps of the various embodiments of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read only memory (ROM), random access memory (RAM), disk or optical disk and other media that can store program code.

[0146] The above contents are only specific implementation methods of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A speech recognition method, the method being executed by an electronic device, characterized in that: The method comprises: When the first service function is enabled, the speech recognition model is initialized according to the configuration parameters corresponding to the first service function, wherein different service functions correspond to different configuration parameters, and the speech recognition model is a model obtained by training according to the training data and includes multiple groups of configuration parameters; The voice information inputted through the first service function is received, and the voice information is inputted into the voice recognition model to obtain a voice recognition result.

2. The method according to claim 1, characterized in that The speech recognition model includes a first model, a second model and a third model, parameters in the first model and parameters in the second model remain unchanged under different business functions, and initializing the speech recognition model according to the configuration parameters corresponding to the first business function includes: The third model is initialized according to the configuration parameters corresponding to the first business function.

3. The method according to claim 2, characterized in that The step of inputting the speech information into the speech recognition model to obtain a speech recognition result includes: Inputting the voice information and the cached recognition results into the voice recognition model to obtain a fusion probability distribution, wherein the cached recognition results include text information obtained by recognizing the historical voice information; The speech recognition result is determined based on the fused probability distribution.

4. The method according to claim 3, characterized in that The step of inputting the speech information and the cached recognition result into the speech recognition model to obtain a fusion probability distribution includes: Inputting the voice information into the first model to obtain a first probability distribution, wherein the first probability distribution represents a probability distribution of characters to which the voice information belongs; Inputting the cached recognition result into the second model to obtain a second probability distribution, wherein the second probability distribution represents a next character probability distribution predicted by the second model for the cached recognition result; Inputting the cache recognition result into the third model to obtain a third probability distribution, wherein the third probability distribution represents a probability distribution of a next character predicted by the third model for the cache recognition result; The fused probability distribution is determined based on the first probability distribution, the second probability distribution, and the third probability distribution.

5. The method according to claim 4, characterized in that The determining the fused probability distribution based on the first probability distribution, the second probability distribution, and the third probability distribution includes: The first probability distribution, the second probability distribution, and the third probability distribution are weightedly summed to obtain the fused probability distribution.

6. The method according to claim 5, characterized in that The weights of the first probability distribution and the third probability distribution are positive numbers, and the weight of the second probability distribution is a negative number.

7. The method according to any one of claims 3 to 6, characterized in that The determining the speech recognition result based on the fusion probability distribution includes: The fused probability distribution is decoded by using a beam search method to obtain the speech recognition result.

8. The method according to claim 7, characterized in that The step of decoding the fused probability distribution in a beam search manner to obtain the speech recognition result includes: For each beam, K characters corresponding to the fusion probability distribution are selected to generate B×K candidate character queues, where B is the number of beams, and the K characters are the characters corresponding to the first K probabilities after the fusion probability distribution is sorted from large to small. Selecting B characters from the B×K candidate character queues, the B characters being characters corresponding to the first B probabilities after the probabilities corresponding to the B×K candidate character queues are sorted from large to small; The character with the highest probability among the B characters is taken as the speech recognition result.

9. The method according to claim 8, characterized in that After selecting B characters, the method further includes: The B characters are added to the cache recognition result to form a new cache recognition result.

10. The method according to any one of claims 2 to 9, characterized in that The first model is an RNNT acoustic model, including an encoding module, a prediction module and a joint module. The encoding module is used to encode the audio features corresponding to the speech information, the prediction module is used to process the text features corresponding to the speech information, and the joint module is used to integrate the output of the encoding module and the output of the prediction module to obtain a first probability distribution.

11. The method according to claim 10, characterized in that The second model includes a feedforward neural network FFN, the prediction module and the joint module, the FFN is located between the prediction module and the joint module, the prediction module is used to process the cache recognition result to obtain a first result, the FFN is used to process the first result again to obtain a second result, and the joint module is used to integrate the first result and the second result to obtain a second probability distribution.

12. The method according to any one of claims 2 to 11, characterized in that The third model is a Transformer network structure, a long short-term memory network structure LSTM or a convolutional network structure, and the third model is a model obtained by training according to training data and includes multiple groups of configuration parameters.

13. The method according to claim 12, characterized in that The training data includes text information generated according to the prompt corresponding to the first business function.

14. The method according to any one of claims 1 to 13, characterized in that The first service function includes a voice assistant function or a call subtitle function.

15. An electronic device, characterized in that: include: one or more processors; one or more memories; The memory stores one or more programs, and when the one or more programs are executed by the processor, the electronic device executes the method according to any one of claims 1 to 14.

16. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the processor is caused to perform the method according to any one of claims 1 to 14.

Images