Hot word enhancement method in speech recognition

Abstract

The present disclosure relates to a method for enhancing hot words in speech recognition, which includes: obtaining a hot word set of information currently displayed on the screen of an electronic device, and obtaining the features of each frame of the audio to be recognized received by the electronic device; obtaining the probabilities of all candidate characters corresponding to each frame of the audio to be recognized according to the features, and obtaining the top K candidate characters with the largest probability for each frame as a candidate character set, wherein K is a natural number; for each frame, when the candidate character set does not contain any hot word in the hot word set, obtaining the top M hot words with the largest probability among the hot words in the hot word set, and adding the top M hot words to the candidate character set, wherein the hot word is a word included in the hot word set, and M is a natural number; and decoding the audio to be recognized according to the candidate character set of each frame and the probability of each character therein, and obtaining the speech recognition result of the audio to be recognized. Using this method, the recognition accuracy can be improved.

Description

Technical Field

[0001] The present disclosure relates to a hot word enhancement method in speech recognition, a computing device, a storage medium, a program product, and the like. Background Art

[0002] At present, in some scenarios where users interact with electronic devices through voice, it is necessary to support what they see is what they say, that is, what is displayed on the screen can be recognized by voice. For example, when a user asks what delicious food is nearby, the electronic device searches for the names of nearby restaurants and food and displays them on the screen. The user may be interested in a restaurant displayed on the screen and then asks to navigate to the restaurant. At this time, the voice recognition system is required to accurately recognize the restaurant name spoken by the user. Summary of the invention

[0003] Embodiments of the present disclosure provide methods for enhancing hot words in speech recognition and corresponding computing devices, non-transitory machine-readable storage media, and computer program products for executing these methods.

[0004] According to a first aspect of an embodiment of the present disclosure, a method for enhancing hot words in speech recognition is provided, comprising: obtaining a hot word set of information currently displayed on a screen of an electronic device, and obtaining features of each frame of audio to be recognized received by the electronic device; obtaining the probabilities of all candidate characters corresponding to each frame of the audio to be recognized according to the features, and obtaining the top K candidate characters with the largest probabilities for each frame as a candidate character set, wherein K is a natural number; for each frame, when the candidate character set does not contain any hot word in the hot word set, obtaining the top M hot words with the largest probabilities among all the hot words in the hot word set, and adding the top M hot words to the candidate character set, wherein the hot word is a word included in the hot word set, and M is a natural number; and decoding the audio to be recognized according to the candidate character set of each frame and the probability of each character therein to obtain a speech recognition result of the audio to be recognized.

[0005] Optionally, the audio to be recognized is decoded to obtain a speech recognition result of the audio to be recognized, including: decoding the audio to be recognized according to the candidate character set of each frame and the probability of each character therein to obtain multiple decoding paths and their decoding scores; for each decoding path, obtaining a hot word additional score of the decoding path according to the hot words contained in the decoding path, wherein the hot words are hot words contained in the hot word set; and adding the hot word additional score of each decoding path to the decoding score to obtain a final score of each decoding path, and obtaining the speech recognition result of the audio to be recognized according to the final score of each decoding path.

[0006] Optionally, the hot word additional score is positively correlated with the number of hot words and / or the number of hot characters, wherein the number of hot characters is the total number of all hot words contained in the decoding path, and the number of hot words is the number of hot words contained in the decoding path.

[0007] Optionally, the hot word additional score includes the sum of a first partial score and a second partial score, wherein the first partial score is the product of the number of hot words and a first additional score, and the second partial score is the product of the number of hot words, the number of hot words and the second additional score.

[0008] Optionally, the audio to be recognized is decoded, including: for each frame, normalizing the probabilities of all characters in the candidate character set to obtain the normalized probabilities of all characters in the candidate character set; and decoding the audio to be recognized based on the candidate character set of each frame and the normalized probability of each character therein.

[0009] Optionally, adding the first M hot words to the candidate character set includes: merging the first K candidate characters of each frame with the first M hot words as the candidate character set; or replacing the M candidate characters with the smallest probability among the first K candidate characters with the first M hot words of each frame, and using the replaced K candidate characters as the candidate character set, where M is less than K.

[0010] Optionally, the display information includes results searched by the electronic device based on user interaction and displayed on the screen.

[0011] Optionally, the hot word set includes entity names belonging to non-general fields in the displayed information.

[0012] Optionally, decoding the audio to be recognized includes: using a decoding module to decode the audio to be recognized, wherein the decoding module adopts a CTC decoding method or a WFST decoding method based on a language model.

[0013] According to a second aspect of an embodiment of the present disclosure, an electronic device is provided, comprising: a processor; and a memory on which executable code is stored, and when the executable code is executed by the processor, the processor executes the method described in any scheme of the first aspect above.

[0014] According to a third aspect of an embodiment of the present disclosure, a non-temporary machine-readable storage medium is provided, on which an executable code is stored. When the executable code is executed by a processor of an electronic device, the processor executes a method as described in any of the schemes in the first aspect above.

[0015] According to a fourth aspect of an embodiment of the present disclosure, a computer program product is provided, comprising an executable code. When the executable code is executed by a processor of an electronic device, the processor is caused to execute a method as described in any one of the schemes in the first aspect above. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] The above and other objects, features and advantages of the present disclosure will become more apparent through a more detailed description of exemplary embodiments of the present disclosure in conjunction with the accompanying drawings, wherein like reference numerals generally represent like components in the exemplary embodiments of the present disclosure.

[0017] Figure 1 A schematic structural diagram of a mobile phone as an example of an electronic device according to at least one embodiment of the present disclosure is exemplarily shown.

[0018] Figure 2 A schematic flowchart of a hot word enhancement method in speech recognition according to at least one embodiment of the present disclosure is exemplarily shown.

[0019] Figure 3 A schematic flowchart of a hot word enhancement method in speech recognition according to at least one embodiment of the present disclosure is exemplarily shown.

[0020] Figure 4 A schematic flowchart of extracting audio features according to at least one embodiment of the present disclosure is exemplarily shown.

[0021] Figure 5 A structural diagram of a computing device according to at least one embodiment of the present disclosure is exemplarily shown. DETAILED DESCRIPTION

[0022] The preferred embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although the preferred embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be limited by the embodiments described herein. On the contrary, these embodiments are provided to make the present disclosure more thorough and complete, and to fully convey the scope of the present disclosure to those skilled in the art.

[0023] First, some explanations of terms involved in this disclosure are given:

[0024] VAD: Voice Activity Detection, which can be used to determine the start and end time points of voice activity in a voice signal;

[0025] DFSMN: Deep Feedforward Sequential Memory Network, which can be used as the backbone network for speech recognition, command word recognition, and voice wake-up.

[0026] WFST: Weighted Finite State Transducer, weighted finite state transition machine;

[0027] MFCC: Mel Frequency Cepstral Coefficient, which is the cepstral parameter extracted in the Mel scale frequency domain. It is a feature widely used in automatic speech and speaker recognition.

[0028] FBANK: filter bank, an audio feature;

[0029] LSTM: Long Stort-Term Memory, long short-term memory network;

[0030] Transformer model: a sequence language model based on attention mechanism;

[0031] CNN: Convolutional Neural Networks, convolutional neural network;

[0032] CTC: Connectionist Temporal Classification, which determines the most probable correspondence between two sequences (X, Y);

[0033] CTC Greedy search:CTC greedy search;

[0034] CTC prefix beam search: CTC prefix beam search;

[0035] LM: Language Model, language model.

[0036] In the aforementioned scenario of supporting what you see is what you say, the restaurant name spoken by the user may not be a normal word or sentence, and it is difficult for the commonly used speech recognition system to recognize it accurately. At this time, in order to improve the accuracy of speech recognition, hot word enhancement technology can be used. For example, the restaurant names currently displayed on the screen are added to the speech recognition system for enhancement, which helps the system to recognize words and sentences that tend to be these restaurant names.

[0037] However, the present disclosure finds that there are many problems with the current mainstream hot word enhancement technology.

[0038] For example, most of the current hot word updates are based on WFST, which requires a long time to run to generate WFST weights, usually taking more than a few seconds. However, the user interface display is updated frequently, and the speed of hot word updates may not keep up with the speed of display updates.

[0039] In addition, if there are no hot words (i.e. words in hot words) in the N-Best (i.e. the top N best results) of the acoustic model, adding N-Best will significantly increase the amount of calculation and content. For example, if the user's Mandarin is not standard or the restaurant name is too obscure and the user cannot pronounce it correctly, it will result in the inability to use WFST weights to compensate for hot words in the future.

[0040] In view of this, the embodiment of the present disclosure proposes a method for enhancing hot words in speech recognition, which can ensure that the candidate character set corresponding to each frame of the audio to be recognized contains hot words, improve the matching degree of subsequent decoding recognition for the hot words displayed on the current screen, and thus improve the accuracy of speech recognition. This is particularly beneficial for the "what you see is what you say" scenario of various electronic devices (such as smart glasses, mobile phones, etc.).

[0041] The method for enhancing hot words in speech recognition provided by the embodiments of the present disclosure can be applied to various types of handheld devices (such as mobile phones, personal digital assistants (PDAs), etc.), various types of computers (such as tablet computers, laptops, ultra-mobile personal computers (UMPCs), netbooks, laptop computers, etc.), wearable devices (such as smart glasses, etc.), vehicle-mounted devices, augmented reality (AR) / virtual reality (VR) devices and other electronic devices, and can also be applied to databases, servers and terminal artificial intelligence-based service response systems, etc. The embodiments of the present disclosure do not impose any restrictions on the specific types of application devices such as electronic devices, as long as the application device has the need to enhance hot words in speech recognition as described above.

[0042] As an example but not limitation, when the terminal device is a wearable device, the wearable device can also be a general term for wearable devices that are developed by applying wearable technology to intelligently design daily wearables, such as gloves, watches, AR head-mounted display devices, VR head-mounted display devices or MR (Mixed Reality) head-mounted display devices equipped with far-field communication modules and / or near-field communication modules.

[0043] In some embodiments, the electronic device may be a device having Figure 1 The exemplary hardware structure of the exemplary mobile phone 100 shown in FIG. Figure 1 As shown, the mobile phone 100 may specifically include: a radio frequency (RF) circuit 110, a memory 120, an input unit 130, a display unit 140, a sensor 150, an audio circuit 160, a short-range wireless communication module 170, a processor 180, and a power supply 190. Those skilled in the art will appreciate that Figure 1 The structure of the mobile phone 100 shown in the figure does not constitute a limitation on the electronic device, and the electronic device may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently.

[0044] Combine the following Figure 1 The components of the mobile phone 100 are described in detail:

[0045] The RF circuit 110 can be used for receiving and sending signals during information transmission or calls. In particular, after receiving the downlink information of the base station, it is sent to the processor 180 for processing; in addition, the designed uplink data is sent to the base station. Generally, the RF circuit includes but is not limited to an antenna, at least one amplifier, a transceiver, a coupler, a low noise amplifier (LNA), a duplexer, etc. In addition, the RF circuit 110 can also communicate with the network and other devices through wireless communication. The above-mentioned wireless communication can use any communication standard or protocol, and the wireless communication can include Global System For Mobile Communications (GSM), General Packet Radio Service (GPRS), Code Division Multiple Access (CDMA), Wideband Code Division Multiple Access (WCDMA), Time Division Code Division Multiple Access (TD-SCDMA), Long Term Evolution (LTE), New Radio (NR), GNSS, FM, low-orbit satellite connection and / or IR technology, etc. The GNSS may include a Global Positioning System (GPS), a Global Navigation Satellite System (GLONASS), a BeiDou Navigation Satellite System (BDS), a Quasi-Zenith Satellite System (QZSS) and / or a Satellite Based Augmentation System (SBAS), etc.

[0046] The memory 120 can be used to store software programs and modules. The processor 180 executes various functional applications and data processing of the mobile phone by running the software programs and modules stored in the memory 120. The memory 120 may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc.; the data storage area may store data created according to the use of the mobile phone (such as pictures, audio data, phone books, etc.), etc. In addition, the memory 120 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other volatile solid-state storage devices. Specifically, the memory 120 may store a program with a voice recognition function.

[0047] The input unit 130 can be used to receive input digital or character information, and to generate key signal input related to the user settings and function control of the mobile phone 100. Specifically, the input unit 130 may include a touch panel 131 and other input devices 132. The touch panel 131, also known as a touch screen, can collect user touch operations on or near it (such as operations performed by the user using fingers (such as single fingers or multiple fingers), styluses, or any other suitable objects or accessories on or near the touch panel 131), and drive the corresponding connection device according to a pre-set program. For example, a user can input corresponding characters on the touch panel 131 through an input method application, so that the input method application displays associated text and / or words or any combination thereof based on the input characters. Optionally, the touch panel 131 may include two parts: a touch detection device and a touch controller. Among them, the touch detection device detects the user's touch direction, detects the signal brought by the touch operation, and transmits the signal to the touch controller; the touch controller receives the touch information from the touch detection device, converts it into the touch point coordinates, and then sends it to the processor 180, and can receive the command sent by the processor 180 and execute it. In addition, the touch panel 131 can be implemented in various types such as resistive, capacitive, infrared and surface acoustic wave. In addition to the touch panel 131, the input unit 130 can also include other input devices 132. Specifically, other input devices 132 can include but are not limited to one or more of a physical keyboard, function keys (such as volume control keys, switch keys, etc.), a trackball, a mouse, a joystick, etc.

[0048] The display unit 140 can be used to display information input by the user or information provided to the user and various menus of the mobile phone. For example, the display unit 140 can be used as a screen in the embodiments described later to display information to the user. The display unit 140 may include a display panel 141. Optionally, the display panel 141 may be configured in the form of a liquid crystal display (LCD), a light emitting diode (LED), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode (AMOLED), etc. Further, the touch panel 131 may cover the display panel 141. When the touch panel 131 detects a touch operation on or near it, it is transmitted to the processor 180 to determine the type of touch event. Subsequently, the processor 180 provides a corresponding visual output on the display panel 141 according to the type of touch event. Although in Figure 1 In the embodiment, the touch panel 131 and the display panel 141 are used as two independent components to realize the input and output functions of the mobile phone, but in some embodiments, the touch panel 131 and the display panel 141 can be integrated to realize the input and output functions of the mobile phone.

[0049] The mobile phone 100 may also include at least one sensor 150, such as a light sensor, a motion sensor, and other sensors. Specifically, the light sensor may include an ambient light sensor and a proximity sensor, wherein the ambient light sensor may adjust the brightness of the display panel 141 according to the brightness of the ambient light, and the proximity sensor may turn off the display panel 141 and / or the backlight when the mobile phone is moved to the ear. As a type of motion sensor, the accelerometer sensor can detect the magnitude of acceleration in each direction (generally three axes), and can detect the magnitude and direction of gravity when stationary. It can be used for applications that identify the posture of the mobile phone (such as horizontal and vertical screen switching, related games, magnetometer posture calibration), vibration recognition related functions (such as pedometer, tapping, etc.); as for other sensors that can be configured in the mobile phone, such as gyroscopes, barometers, hygrometers, thermometers, infrared sensors, etc., they will not be repeated here.

[0050] The audio circuit 160, the speaker 161, and the microphone 162 can provide an audio interface between the user and the mobile phone. The audio circuit 160 can transmit the received audio data to the speaker 161 after converting the electrical signal, and the speaker 161 converts the electrical signal into a sound signal for output; on the other hand, the microphone 162 converts the collected sound signal into an electrical signal, which is received by the audio circuit 160 and converted into audio data, and then the audio data is output to the processor 180 for processing, and then sent to another electronic device through the RF circuit 110, or the audio data is output to the memory 120 for further processing. For example, the microphone 162 can collect the user's voice as the audio to be recognized in the embodiments described later.

[0051] Communication technologies such as Wi-Fi, Bluetooth, Near Field Communication (NFC) and Ultra Wide Band (UWB) belong to short-range wireless transmission technologies. The mobile phone can help users send and receive emails, browse web pages and access streaming media through the short-range wireless module 170, which provides users with wireless broadband Internet access. The above-mentioned short-range wireless module 170 may include a Wi-Fi chip, a Bluetooth chip, an NFC chip and a UWB chip. The Wi-Fi chip can realize the function of Wi-Fi Direct connection between the mobile phone 100 and other electronic devices, and can also make the mobile phone 100 work in an AP mode (Access Point mode) that can provide wireless access services and allow other wireless devices to access, or work in a STA mode (Station mode) that can connect to an AP but does not accept wireless devices to access, thereby establishing point-to-point communication between the mobile phone 100 and other Wi-Fi devices.

[0052] The processor 180 is the control center of the mobile phone. It uses various interfaces and lines to connect various parts of the entire mobile phone. By running or executing software programs and / or modules stored in the memory 120, and calling data stored in the memory 120, it executes various functions of the mobile phone and processes data, thereby monitoring the mobile phone as a whole. Optionally, the processor 180 may include one or more processing units; optionally, the processor 180 may include, for example, an application processor (Application Processor, AP), a modem processor, a graphics processor (Graphics Processing Unit, GPU), an image signal processor (Image Signal Processor, ISP), a controller, a video codec, a digital signal processor (Digital Signal Processor, DSP), a baseband processor, and / or a neural network processor (Neural-Network Processing Unit, NPU) (for example, for running the neural network model described in any embodiment of this document), etc. Among them, different processing units can be independent devices or integrated in one or more processors.

[0053] The mobile phone 100 also includes a power source 190 (such as a battery) for supplying power to various components. Preferably, the power source can be logically connected to the processor 180 through a power management system, so that the power management system can manage charging, discharging, power consumption and other functions.

[0054] The following will be combined Figures 2 to 4 Some embodiments of the present disclosure are described in detail. In the present disclosure, "plurality" or similar expressions refer to two or more.

[0055] Figure 2 The following is a schematic flow chart showing a method for enhancing hot words in speech recognition according to at least one embodiment of the present disclosure. Figure 3 Given Figure 2 A specific implementation example of the method, and Figure 4 Given Figure 3 A specific implementation example of step S2 in .

[0056] like Figure 2 As shown, in step S210, a set of hot words of information currently displayed on the screen of the electronic device is obtained, and features of each frame of audio to be recognized received by the electronic device are obtained.

[0057] In some embodiments, the display information includes the results that the electronic device searches for and displays on the screen based on the user's interaction. For example, the user can voice-request the electronic device to search for nearby restaurants, and then the electronic device displays the search results (such as restaurant names, restaurant addresses, etc.) as the display information on the screen.

[0058] In some embodiments, the hot word set includes entity names belonging to non-general fields in the displayed information. General fields refer to fields that are widely applicable, basic, and generally involved in society. Non-general fields, on the other hand, usually refer to specific fields, which are often professional, specific, and targeted. In some cases, the speech recognition system is usually trained with vocabulary in general fields (including various entity names), so the recognition effect of vocabulary in general fields is better, and these speech recognition systems are not familiar with entity names in non-general fields (such as various niche restaurant names, hotel names, scenic spot names, etc.), so it is necessary to enhance the hot words of entity names in non-general fields.

[0059] For example, in a "what you see is what you say" scenario, when the electronic device displays search results in a certain field, it can be predicted that the user will select one or more items from the displayed search results for the next voice interaction, so the entity names in the search results can be used as hot words to obtain a hot word set. For example, when a user requests to search for nearby restaurants, the hot word set may include: Lijiacun Restaurant, Eden Garden Restaurant, Yuduo Private Kitchen, and the hot words are the words in the hot words, so the hot words may include: Li, Jia, Cun, Fan, Dian, Garden, Wine, Yu, Duo, Private, Room, Cuisine.

[0060] The present disclosure does not limit the execution order of the two operations of obtaining the hot word set and obtaining the features of each frame of the audio to be recognized in step S210, which can be executed sequentially or in parallel in any manner. Figure 2 The flowchart defines that the operation of obtaining the hot word set is performed before step S220, but the present disclosure is not limited thereto. The operation of obtaining the hot word set only needs to be completed before step S230, and it can be performed before or after step S220 or in parallel with step S220 according to actual needs.

[0061] For example, in Figure 3 In the example shown, steps S1 and S2 correspond to the two operations of obtaining a hot word set and obtaining features of each frame of the audio to be recognized in step S210, respectively, and their execution is independent of each other, and step S1 only needs to be completed before step S5.

[0062] Step S2 obtains the features of each frame of the audio to be recognized by performing feature extraction on the audio to be recognized.

[0063] The input of step S2 is a segment of audio to be recognized, the audio sampling rate is, for example, 16kHz, the bit width is 16bit, and the single channel. In order to process real-time, the audio is cut into segments, usually 10ms as a frame. Generally speaking, a segment of 10ms of audio can extract meaningful features and results. In the subsequent processing, in order to prevent spectrum leakage, the method of partially overlapping the previous and next frames is adopted during processing. Each frame length is 25ms, of which 15ms is historical information, that is, 15ms overlap, and only 10ms is actually moved.

[0064] The output of step S2 is the feature value of each frame, which usually uses the spectral features of the audio. Typical features include MFCC features and FBANK features. FBANK is obtained by summing the squares of the power spectrum of the Mel filter and then taking the logarithm. Compared with MFCC, the discrete cosine transform calculation step is omitted. For example, if an input audio segment is 160ms long and is divided into 16 frames, and the features of each frame are 40-dimensional data, the output 16-frame features are a 16×40 matrix.

[0065] Figure 4 An exemplary process for calculating the FBANK feature in step S2 is given: generally, the audio file is pre-emphasized first, in order to enhance the high-frequency signal; then frame division and windowing are performed. Framing is to separate the audio signal into frames of 10ms, and windowing is to prevent spectrum leakage. Each time, 25ms of signal is used to calculate the feature, that is, each time it moves 10ms, and 25ms of signal is actually used, with 15ms of historical overlapping information. Then, the time domain signal is used to obtain the frequency domain signal, that is, the spectrum, through Fourier transform. The frequency domain of a certain time is accumulated to obtain the spectrum (the power spectrum is taken and then the amplitude is squared), and then the frequency is mapped to the Mel frequency scale through the Mel filter bank, and finally the logarithm is taken to obtain the FBANK feature.

[0066] Back to Figure 2 After obtaining the audio features, in step S220, the probabilities of all candidate characters corresponding to each frame of the audio to be recognized are obtained according to the features, and the top K candidate characters with the largest probabilities are obtained for each frame as a candidate character set, where K is a natural number.

[0067] For example, in Figure 3 In the example shown, steps S3 and S4 correspond to step S220, wherein in S3 the features of each frame of audio to be recognized are input into the acoustic model, and the acoustic model (which is a neural network model) is used for reasoning to obtain the probabilities of all candidate characters corresponding to each frame of audio to be recognized, and in S4 the top K (topK) candidate characters are sorted in descending order according to probability to obtain as the candidate character set.

[0068] For example, the input of step S3 is the MFCC or FBANK features of each frame obtained in the aforementioned step S2, and the output of step S3 is the probability of all candidate characters corresponding to each frame, that is, the temporal label matrix inferred by the acoustic model. Assuming that 16 frames of MFCC feature data are input, and assuming that the acoustic model has a total of 2000 candidate characters, the output matrix is ​​a 16× 2000 matrix. For example, the candidate characters may include commonly used Chinese characters.

[0069] In step S3, various neural network-based acoustic models can be used to implement the mapping of audio features (such as MFCC or FBANK features) to characters, such as neural network models based on CNN, DFSMN, LSTM, transformer, or WENET (a commonly used open source speech recognition network). The DFSMN model has a relatively small number of parameters and a simple structure, and is currently a commonly used acoustic model.

[0070] Then in step S4, the probabilities of all the words output by the acoustic model (assuming there are 2000 words, there are 2000 floating point numbers) are sorted from large to small, and the top K (topK) are taken. K can be any empirical value, for example, K=20.

[0071] Back to Figure 2 After obtaining the candidate character set and the hot word set, in step S230, for each frame, when the candidate character set does not contain any hot word in the hot word set, obtain the top M hot words with the highest probability among all the hot words in the hot word set, and add the top M hot words to the candidate character set, wherein the hot word is a word included in the hot word set, and M is a natural number.

[0072] In some implementations, the first K candidate characters and the first M hot characters of each frame may be combined as the candidate character set, so that the final candidate character set includes (K+M) candidate characters.

[0073] Alternatively, in some other implementations, the first M hot words of each frame may be used to replace the M candidate characters with the lowest probabilities among the first K candidate characters (i.e., the last M candidate characters), and the replaced K candidate characters are used as the candidate character set, where M is less than K. Therefore, the final candidate character set still only includes K candidate characters.

[0074] Because the subsequent decoding operation only considers the candidate character set obtained previously, if it does not contain hot words, the final recognition result will not include hot words or hot words. Therefore, in order to enhance the recognition of hot words, the operation of inserting hot words is performed in this step. The purpose is to include the option of hot words in subsequent decoding. However, it does not mean that the hot words will be included in the final decoding result after adding them. The hot words may also fail in the final decoding due to low scores in the decoding process.

[0075] For example, in Figure 3 In the example shown, steps S5 and S6 correspond to step S230, wherein in step S5, it is determined for each frame whether the topK of the acoustic model output results contains any hot word in the hot word set; if the topK contains any hot word, no additional processing is performed; if the topK does not contain any hot word, go to step S6 to insert the M hot words with the highest probability.

[0076] For example, in S6, the probabilities of all hot words can be searched in the inference results of the acoustic model, and the top M hot words with the highest probabilities are selected and added to topK according to the probability sorting. The new candidate character sets obtained by the two aforementioned methods of adding hot words become (K+M) or remain K candidate characters.

[0077] M can be any experience value, such as 2.

[0078] Then return to Figure 2 In step S240, the audio to be recognized is decoded according to the candidate character set of each frame and the probability of each character therein obtained in step S230 to obtain a speech recognition result of the audio to be recognized.

[0079] In some embodiments, since the last layer of the aforementioned acoustic model is generally a softmax layer, the result of the acoustic model reasoning ensures that the sum of the probability values ​​of all characters is 1. However, after subsequent sorting and other operations, only the topK or topK+M candidate characters are taken from the reasoning results of the acoustic model of each frame for decoding. In order to make it easier to control the weights of hot characters in subsequent decoding, a normalization constraint can be made for all candidate characters in the current candidate character set to obtain new acoustic reasoning scores so that the sum of their scores is still 1. The softmax function can also be used for normalization.

[0080] That is to say, in some embodiments, normalization processing can be performed before decoding the audio to be recognized, that is: for each frame, the probabilities of all characters in the candidate character set are normalized to obtain the normalized probabilities of all characters in the candidate character set; and according to the candidate character set of each frame and the normalized probability of each character therein, the audio to be recognized is decoded.

[0081] By normalizing the probabilities (for example, using the softmax function) before decoding, the discrimination can be enhanced and the accuracy of subsequent decoding predictions can be improved. For example, the sum of the probabilities of the original 2,000 characters is 1, and the 20 characters with the highest probabilities are taken as the candidate character set. The probability distribution interval is small, the difference between them is small, and the discrimination is small. After normalization, the sum of the probabilities of these 20 characters becomes 1, and these 20 probability values ​​are distributed over the entire interval from 0 to 1, the difference between them becomes larger, and the discrimination becomes larger.

[0082] In some implementations, a decoding module may be used to decode the audio to be recognized, wherein the decoding module adopts a CTC decoding method or a WFST decoding method based on a language model.

[0083] For example, in Figure 3 In the example, step S7 corresponds to step S240, wherein CTC or WFST decoding is adopted in S7.

[0084] CTC decoding belongs to streaming decoding, which means inferring and decoding some audio features and feeding back the decoding results to the user in real time. For example, if a user says "please help me check the train ticket to Shanghai tomorrow", the streaming recognition process may be as follows:

[0085] 1st CTC decoding: Help me

[0086] Second CTC decoding: Help me check

[0087] 3rd CTC decoding: Help me check tomorrow

[0088] 4th CTC decoding: Help me check it out tomorrow

[0089] The 5th CTC decoding: Help me check if I am going to Shanghai tomorrow

[0090] The 6th CTC decoding: Please help me find the tickets to Shanghai tomorrow.

[0091] The 7th CTC decoding: Please help me check the train tickets to Shanghai tomorrow.

[0092] In CTC decoding, the input can be a time-series label matrix consisting of the normalized probabilities of the candidate character sets of multiple consecutive frames obtained in the above steps. For example, for 16 frames of FBANK feature data, assuming that the acoustic model has a total of 2000 characters, the input matrix is ​​a 16×2000 matrix, in which each element represents the probability of a character. For example, the probability of a character that does not belong to the candidate character set obtained previously can be set to 0, and the probability of a character that belongs to the candidate character set can be set to the above normalized probability.

[0093] CTC decoding can use algorithms such as greedy search, beam search, or prefix beam search to find N best decoding paths (such as the decoding paths with the highest probability or score) in the input matrix, and obtain the final speech recognition result based on this.

[0094] In addition to the sequential label matrix similar to the aforementioned CTC decoding, the input of WFST decoding also includes a list of LMs (language models). WFST decoding uses different language models to re-score the input matrix, finds the decoding path with the highest score, and obtains the final speech recognition result based on this. Generally, the weighted finite state transducer (WFST) is used as the language model.

[0095] Similar to CTC decoding, WFST decoding also completes the decoding of a sequence. The difference is that CTC decoding only considers the probability of the sequence itself, while WFST decoding also considers the relationship between sequences, that is, the sequence transition weight. The Chinese characters before and after in a sentence are strongly correlated. For example, the two characters "你" and "好" may appear as "你好" and "好你" in a sentence, and the former is obviously more probable than the latter. WFST decoding is this decoding method. Its decoding method is similar to CTC, but the probability of the whole sentence adds the transition probability of the previous and subsequent characters.

[0096] As mentioned above, both CTC decoding and WFST decoding can find multiple best decoding paths in the candidate character sets of each frame of the input. In some embodiments, in order to further enhance the recognition of hotwords, after obtaining multiple decoding paths through normal decoding operations, additional scores can be assigned to the decoding paths containing hotwords to further improve the accuracy of hotword recognition.

[0097] For example, in some embodiments, according to the candidate character set of each frame and the probabilities of the characters therein, the audio to be recognized is decoded (such as conventional CTC decoding or WFST decoding). After obtaining multiple decoding paths and their decoding scores, for each decoding path, a hotword additional score of the decoding path is obtained according to the hotwords included in the decoding path, where the hotword is a hotword included in the hotword set; then the hotword additional score of each decoding path is added to the decoding score to obtain the final score of each decoding path, and the speech recognition result of the audio to be recognized is obtained according to the final score of each decoding path.

[0098] In some examples, the aforementioned hotword additional score can be positively correlated with the number of hotwords and / or the number of hot characters, where the number of hot characters is the total number of characters of all hotwords included in the decoding path, and the number of hotwords is the number of hotwords included in the decoding path.

[0099] For example, the additional score of the hot word can be positively correlated with the number of hot words and the number of hot characters. For example, the additional score of the hot word can include the sum of the first part score and the second part score, wherein the first part score is the product of the number of hot characters and the first additional score, and the second part score is the product of the number of hot characters, the number of hot words and the second additional score.

[0100] The following is an exemplary description of the above additional scoring process using CTC decoding as an example. It should be understood that similar additional scoring processes can also be performed for cases where WFST decoding or other decoding methods are used.

[0101] After CTC decoding obtains multiple decoding paths and their scores, for each decoding path, in addition to the CTC score, the hot word additional score is added to get the final score.

[0102] The formula for adding points to hot words is as follows:

[0103] Hot word additional score = (context_score + i × incremental_context_score) ×Length(words);

[0104] in:

[0105] i is the number of hot words, that is, the number of hot words contained in the decoding path;

[0106] Length(words) is the number of hot words, that is, the total number of words in all hot words included in the decoding path;

[0107] context_score and incremental_context_score are the first additional score and the second additional score respectively. Their values ​​are hyperparameters and can be set as needed.

[0108] You can set the values ​​of context_score and incremental_context_score based on experience or experiments, for example, context_score=5, incremental_context_score=3.

[0109] Alternatively, the values ​​of context_score and incremental_context_score may be dynamically adjusted according to the effect of the final speech recognition result (such as accuracy, etc.). For example, if a hot word is not recognized in the end, the first additional score and / or the second additional score may be increased, and if a hot word is mistakenly recognized, the first additional score and / or the second additional score may be decreased.

[0110] For example, if there are hot words: "Xinjiang", "Hand-Pulled Meat", the user says "Look for Xinjiang Hand-Pulled Meat nearby".

[0111] Assuming that the first additional score context_score = 5 and the second additional score incremental_context_score = 3, the number of hot words decoded into the path of "find Xinjiang hand-pulled meat nearby" is 2, and the number of hot characters is 2+3=5, so the additional score of hot words is:

[0112] (context_score + i × incremental_context_score) × Length(words) =(5 + 2 × 3) × 5 = 55.

[0113] In one possible implementation, the decoding process can be performed sentence by sentence. That is, after the sentence is recognized, the best decoding path (such as the path with the highest score) is obtained according to the final score of each decoding path as the speech recognition result of the audio to be recognized. Exemplarily, the VAD model can be used to identify the audio segment with voice activity in the audio collected by the electronic device as the audio to be recognized, and the sentence segment in the audio segment can be identified according to the time interval between the voice frames with audio activity in the audio segment. For another example, the sentence segment in the audio to be recognized can be identified according to the received key press time. For example, in some application scenarios, the user needs to press and hold the function key to enter the voice command, so the audio received during the period of pressing the function key can be regarded as a sentence.

[0114] Figure 5 A schematic diagram of the structure of a computing device that can be used to implement the hot word enhancement method in speech recognition according to at least one embodiment of the present disclosure is shown.

[0115] See also Figure 5 , the computing device 500 includes a memory 510 and a processor 520 .

[0116] The processor 520 may be a multi-core processor or may include multiple processors. In some embodiments, the processor 520 may include a general-purpose main processor and one or more special coprocessors, such as a graphics processing unit (GPU), a digital signal processor (DSP), etc. In some embodiments, the processor 520 may be implemented using a customized circuit, such as an application-specific integrated circuit (ASIC) or a field programmable gate array (FPGA).

[0117] The memory 510 may include various types of storage units, such as system memory, read-only memory (ROM), and permanent storage devices. Among them, ROM can store static data or instructions required by the processor 520 or other modules of the computer. The permanent storage device may be a readable and writable storage device. The permanent storage device may be a non-volatile storage device that does not lose the stored instructions and data even after the computer is powered off. In some embodiments, the permanent storage device uses a large-capacity storage device (such as a magnetic or optical disk, flash memory) as a permanent storage device. In some other embodiments, the permanent storage device may be a removable storage device (such as a floppy disk, optical drive). The system memory may be a readable and writable storage device or a volatile readable and writable storage device, such as a dynamic random access memory. The system memory may store some or all instructions and data required by the processor at runtime. In addition, the memory 510 may include any combination of computer-readable storage media, including various types of semiconductor memory chips (DRAM, SRAM, SDRAM, flash memory, programmable read-only memory), and disks and / or optical disks may also be used. In some embodiments, the memory 510 may include a readable and / or writable removable storage device, such as a laser disc (CD), a read-only digital versatile disc (such as a DVD-ROM, a double-layer DVD-ROM), a read-only Blu-ray disc, an ultra-density optical disc, a flash memory card (such as an SD card, a mini SD card, a Micro-SD card, etc.), a magnetic floppy disk, etc. The computer-readable storage medium does not include carrier waves and transient electronic signals transmitted wirelessly or wired.

[0118] The memory 510 stores executable codes. When the executable codes are processed by the processor 520, the processor 520 can execute the hot word enhancement method in speech recognition mentioned above.

[0119] In addition, the method according to the present disclosure may also be implemented as a computer program or a computer program product, which includes computer program code instructions for executing the above steps defined in the above method of the present disclosure.

[0120] Alternatively, the present disclosure may also be implemented as a non-transitory machine-readable storage medium (or computer-readable storage medium, or machine-readable storage medium) on which executable code (or computer program, or computer instruction code) is stored. When the executable code (or computer program, or computer instruction code) is executed by a processor of an electronic device (or computing device, server, etc.), the processor executes the various steps of the above-mentioned method according to the present disclosure.

[0121] Those skilled in the art will further appreciate that the various illustrative logical blocks, modules, circuits, and algorithm steps described in connection with the disclosure herein may be implemented as electronic hardware, computer software, or a combination of both.

[0122] The flow charts and block diagrams in the accompanying drawings show the possible architecture, functions and operations of the system and method according to multiple embodiments of the present disclosure. In this regard, each box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, a program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flow chart, and the combination of the boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs the specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0123] The embodiments of the present disclosure have been described above, and the above description is exemplary, not exhaustive, and is not limited to the disclosed embodiments. Many modifications and changes will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments. The selection of terms used herein is intended to best explain the principles of the embodiments, practical applications, or improvements to the technology in the market, or to enable other persons of ordinary skill in the art to understand the embodiments disclosed herein.

Claims

1. A method for enhancing hot words in speech recognition, comprising: Obtaining a set of hot words for information currently displayed on a screen of an electronic device, and obtaining features of each frame of audio to be recognized received by the electronic device; Obtaining the probabilities of all candidate characters corresponding to each frame of the audio to be recognized according to the features, and obtaining the top K candidate characters with the largest probabilities for each frame as a candidate character set, where K is a natural number; For each frame, when the candidate character set does not contain any hot word in the hot word set, obtain the top M hot words with the highest probability among all the hot words in the hot word set, and add the top M hot words to the candidate character set, wherein the hot word is a word included in the hot word set, and M is a natural number; as well as According to the candidate character set of each frame and the probability of each character therein, the audio to be recognized is decoded to obtain a speech recognition result of the audio to be recognized.

2. The method according to claim 1, wherein: Decoding the audio to be recognized to obtain a speech recognition result of the audio to be recognized, including: According to the candidate character set of each frame and the probability of each character therein, the audio to be recognized is decoded to obtain multiple decoding paths and decoding scores thereof; For each decoding path, obtaining a hot word additional score for the decoding path according to the hot words included in the decoding path, wherein the hot words are hot words included in the hot word set; and The hot word additional score of each decoding path is added to the decoding score to obtain a final score of each decoding path, and a speech recognition result of the audio to be recognized is obtained according to the final score of each decoding path.

3. The method according to claim 2, wherein: The hot word additional score is positively correlated with the number of hot words and / or the number of hot characters, wherein the number of hot characters is the total number of all hot words contained in the decoding path, and the number of hot words is the number of hot words contained in the decoding path.

4. The method according to claim 3, wherein: The hot word additional score includes the sum of a first part score and a second part score, wherein the first part score is the product of the number of hot words and the first additional score, and the second part score is the product of the number of hot words, the number of hot words and the second additional score.

5. The method according to claim 1, wherein: The step of decoding the audio to be recognized includes: For each frame, normalizing the probabilities of all characters in the candidate character set to obtain normalized probabilities of all characters in the candidate character set; and The audio to be recognized is decoded according to the candidate character set of each frame and the normalized probability of each character therein.

6. The method according to claim 1, wherein: Adding the first M hot words to the candidate character set includes: Combining the first K candidate characters and the first M hot characters of each frame as the candidate character set; or The first M hot words of each frame are used to replace the M candidate characters with the smallest probabilities among the first K candidate characters, and the replaced K candidate characters are used as the candidate character set, where M is less than K.

7. The method according to claim 1, wherein: The displayed information includes the results searched by the electronic device according to the user's interaction and displayed on the screen; and / or The hot word set includes entity names belonging to non-general fields in the displayed information.

8. The method according to claim 1, wherein: The step of decoding the audio to be recognized includes: The audio to be recognized is decoded by using a decoding module, wherein the decoding module adopts a CTC decoding method or a WFST decoding method based on a language model.

9. An electronic device, comprising: processor; as well as A memory having executable codes stored thereon, which, when executed by the processor, causes the processor to execute the method according to any one of claims 1 to 8.

10. A non-transitory machine-readable storage medium having executable codes stored thereon, which, when executed by a processor of an electronic device, causes the processor to execute the method according to any one of claims 1 to 8.

11. A computer program product, comprising executable codes, which, when executed by a processor of an electronic device, causes the processor to execute the method according to any one of claims 1 to 8.

Images