Speech recognition method, interaction method, storage medium and program product

Abstract

This application provides a speech recognition method, an interaction method, a storage medium, and a program product. The speech recognition method includes: obtaining multiple candidate character sequences and their corresponding first scores based on the speech features of the speech to be recognized; obtaining preferred words of the pronunciation object corresponding to the speech to be recognized; matching the multiple candidate character sequences with the preferred words, and determining second scores corresponding to the multiple candidate character sequences based on the matching results; and determining a target character sequence as the speech recognition result from the multiple candidate character sequences based on the first score and the second score corresponding to each of the multiple candidate character sequences.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to a speech recognition method, interaction method, storage medium, and program product. Background Technology

[0002] Speech recognition is a technology that automatically converts speech into text using machines. With the continuous development of machine learning algorithms, speech recognition has achieved very high accuracy rates, making it the interaction interface for many smart devices and making human-computer interaction more natural, faster, and simpler.

[0003] However, because neural network models implemented using machine learning algorithms require significant resources, speech recognition neural network models typically run in the cloud, with multiple users sharing a single machine learning algorithm. However, neural network models generally used for speech recognition are designed to meet the needs of most users and general scenarios, and cannot provide personalized support for specific users or specific use cases. Furthermore, training and using neural network models to support specific users or scenarios would be extremely inefficient. Summary of the Invention

[0004] In view of this, embodiments of this application provide a speech recognition scheme to at least partially solve the above-mentioned problems.

[0005] According to a first aspect of the embodiments of this application, a speech recognition method is provided, comprising: obtaining a plurality of candidate character sequences corresponding to the speech to be recognized and their corresponding first scores based on the speech features of the speech to be recognized; obtaining preferred words of the pronunciation object corresponding to the speech to be recognized; matching the plurality of candidate character sequences with the preferred words, and determining second scores corresponding to the plurality of candidate character sequences respectively based on the matching results; and determining a target character sequence as a speech recognition result from the plurality of candidate character sequences based on the first scores and second scores corresponding to the plurality of candidate character sequences.

[0006] According to a second aspect of the embodiments of this application, an interaction method for a smart IoT device is provided, comprising: receiving and sending speech to be recognized through the smart IoT device, and determining a preferred word corresponding to the pronunciation object of the speech to be recognized; performing speech recognition based on the speech features of the speech to be recognized and the preferred word to obtain a first speech recognition result matching the preferred word and a second speech recognition result not matching the preferred word, wherein the speech features are used to determine candidate character sequences and a first score as candidate speech recognition results, and the preferred word is used to determine a second score based on the matching result of the candidate character sequences and the preferred word; outputting the first speech recognition result and its matching preferred word, and the second speech recognition result through the smart IoT device; and reducing the second score corresponding to the output preferred word if a negative operation from a user is received.

[0007] According to a third aspect of the embodiments of this application, a computer storage medium is provided, on which a computer program is stored, which, when executed by a processor, implements the speech recognition method as described above.

[0008] According to a fourth aspect of the embodiments of this application, a computer program product is provided, including computer instructions that instruct a computing device to perform operations corresponding to the speech recognition method described above.

[0009] According to the speech recognition scheme provided in the embodiments of this application, multiple candidate character sequences and their corresponding first scores are obtained based on the speech features of the speech to be recognized. The first score can be used to characterize the probability of using the candidate character sequence as the speech recognition result based on the speech features. Preferred words of the pronunciation object corresponding to the speech to be recognized are obtained. The multiple candidate character sequences are matched with the preferred words, and a second score is determined for each of the multiple candidate character sequences based on the matching result. The second score is used to characterize the probability of using the candidate character sequence as the speech recognition result based on the preferred words. Based on the first score and second score corresponding to each of the multiple candidate character sequences, a target character sequence as the speech recognition result is determined from the multiple candidate character sequences. This approach comprehensively considers existing experience in speech recognition based on speech features and the preferred words that the pronunciation object hopes to hit, resulting in a personalized speech recognition result for the pronunciation object. Furthermore, in this embodiment, the second score is determined by matching candidate character sequences with the preferred words, thus decoupling the determination processes of the first and second scores. This allows for flexible adjustment of the preferred words for the pronunciation object as needed, without affecting the determination process of the first score. This makes the speech recognition method provided in this embodiment highly transferable and allows for the addition of a large number of preferred words. Moreover, the solution provided in this embodiment can directly add preferred word matching logic to an existing decoder. If preferred word matching logic is not needed, the second score can be directly set to 0. The solution provided in this embodiment can be obtained with minor modifications to existing language recognition models, making it simple and convenient to implement. Attached Figure Description

[0010] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in the embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings.

[0011] Figure 1A This is a flowchart illustrating the steps of a speech recognition method according to Embodiment 1 of this application;

[0012] Figure 1B for Figure 1A A schematic diagram of a scenario example in the illustrated embodiment;

[0013] Figure 2A A flowchart illustrating the steps of a speech recognition method provided in this application embodiment;

[0014] Figure 2Bfor Figure 2A A schematic diagram of the structure of a neural network model in the embodiment shown;

[0015] Figure 3A This is a schematic diagram of the structure of a speech recognition system provided in an embodiment of this application;

[0016] Figure 3B for Figure 3A The flowchart of a matching method in the embodiment shown is shown.

[0017] Figure 4 A flowchart illustrating the steps of an interaction method for a smart IoT device provided in this application embodiment;

[0018] Figure 5 This is a schematic diagram illustrating one scenario in an embodiment of this application;

[0019] Figure 6 This is a schematic diagram of the structure of a smart IoT device according to an embodiment of this application;

[0020] Figure 7 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0021] To enable those skilled in the art to better understand the technical solutions in the embodiments of this application, the technical solutions in the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art should fall within the protection scope of the embodiments of this application.

[0022] The specific implementation of the embodiments of this application will be further described below with reference to the accompanying drawings.

[0023] Figure 1A A flowchart illustrating a speech recognition method provided in this embodiment is shown in the figure, which includes:

[0024] S101. Based on the speech features of the speech to be recognized, obtain multiple candidate character sequences corresponding to the speech to be recognized and their corresponding first scores.

[0025] In this embodiment, the speech features of the speech to be recognized can be used to characterize the pronunciation characteristics and content of the speech to be recognized.

[0026] Based on the speech features of the speech to be recognized, multiple candidate character sequences corresponding to the unit to be recognized can be determined, and a first score of the candidate character sequence can be determined. The first score is used to characterize the probability of taking the candidate character sequence as the speech input result of the speech to be recognized based on the speech features.

[0027] For specific methods of determining the first score, please refer to the methods of determining the scores of candidate character sequences through acoustic models and language models, which will not be elaborated here.

[0028] S102. Obtain the preferred words of the pronunciation object corresponding to the speech to be recognized.

[0029] Specifically, the speech object to be recognized can be a specific user or a user using an electronic device that collects speech; this embodiment does not limit this.

[0030] The preferred words for the pronunciation target are the words that the pronunciation target hopes to hear. Specifically, preferred words can be words from the pronunciation target's favorites list, landmark names from the pronunciation target's geographical location, or words from currently popular media. The specific preferred words can be set as needed by those skilled in the art, and this embodiment does not limit this.

[0031] S103. Match the multiple candidate character sequences with the preferred words, and determine the second score corresponding to each of the multiple candidate character sequences based on the matching results.

[0032] Specifically, during matching, it can be determined whether the candidate character sequence includes the string corresponding to the preferred word, or whether the pronunciation of the candidate character sequence matches the pronunciation of the preferred word. The specific matching method can be set by those skilled in the art according to their needs, and this embodiment does not limit it in this regard.

[0033] In this embodiment, the second score is used to characterize the probability of selecting the candidate character sequence as the speech recognition result based on the preferred words. The second score can be pre-configured based on the preferred words or calculated based on the matching results; this embodiment does not limit this.

[0034] S104. Based on the first score and the second score corresponding to each of the multiple candidate character sequences, determine the target character sequence as the speech recognition result from the multiple candidate character sequences.

[0035] Specifically, the weighted sum of the first score and the second score can be used to obtain the comprehensive score corresponding to each of the multiple candidate character sequences. Then, based on the comprehensive score, one of the multiple candidate character sequences can be selected as the speech recognition result.

[0036] Of course, the above is only an example, and other solutions for determining the speech recognition result based on the first score and the second score are also within the protection scope of this application.

[0037] Next, an exemplary illustration of the solution of this application will be given through a specific usage scenario.

[0038] Refer to Figure 1B , the user can input the voice "Play Xiaona" to the smart speaker. After the smart speaker collects the voice to be recognized, it can send it to the cloud. The cloud can determine two candidate character sequences according to the voice characteristics of the voice to be recognized, namely "Play Xiaona" and "Tell a joke".

[0039] Moreover, through the acoustic model and language model of the cloud, the first score of the candidate character sequence can be determined. Among them, the first score corresponding to "Play Xiaona" is 0.9 + 0.5, and the first score corresponding to "Tell a joke" is 0.6 + 0.9. Here, AM refers to the partial score output by the acoustic model, and LM refers to the partial score output by the language model.

[0040] In addition, for multiple candidate character sequences, they can be matched with the preferred words. According to the matching results, it can be seen that "Xiaona" in the candidate character sequence "Play Xiaona" matches the preferred word, so the second score bias of this candidate character sequence is 0.5; while the candidate character sequence "Tell a joke" does not include the part that matches the preferred word, so the second score of this candidate character sequence is 0 points (not shown in the figure).

[0041] According to the first score and the second score corresponding to the candidate character sequence, it can be determined that "Play Xiaona" is the speech recognition result. The cloud can search for "Xiaona" and obtain the song "Xiaona", and return the song "Xiaona" as the speech recognition result to the smart speaker. The smart speaker can play "Xiaona" for the pronunciation object.

[0042] In addition, if the preferred words of the pronunciation object do not include "Xiaona", the second scores corresponding to the two candidate character sequences are both 0 points. At this time, according to the first scores corresponding to the two candidate character sequences, it can be determined that "Tell a joke" is the speech recognition result. The cloud can search for "joke" and obtain some popular jokes, and return the obtained jokes as the speech recognition result to the smart speaker. The smart speaker can read the received jokes for the pronunciation object.

[0043] In this embodiment, the first score characterizes the probability of identifying the candidate character sequence as the speech recognition result based on speech features, and the second score characterizes the probability of identifying the candidate character sequence as the speech recognition result based on preferred words. Therefore, by combining the first and second scores, existing experience in speech recognition based on speech features can be comprehensively considered, along with the preferred words that the speaker hopes to receive, to obtain a personalized speech recognition result for the speaker.

[0044] Furthermore, in this embodiment, the second score is determined by matching candidate character sequences with the preferred words, thus decoupling the determination processes of the first and second scores. This allows for flexible adjustment of the preferred words for the pronunciation object as needed, without affecting the determination process of the first score. This makes the speech recognition method provided in this embodiment highly transferable and allows for the addition of a large number of preferred words. Moreover, the solution provided in this embodiment can directly add preferred word matching logic to an existing decoder. If preferred word matching logic is not needed, the second score can be directly set to 0. The solution provided in this embodiment can be obtained with minor modifications to existing language recognition models, making it simple and convenient to implement.

[0045] The speech recognition method of this embodiment can be executed by any suitable electronic device with data processing capabilities, including but not limited to: servers, mobile terminals (such as mobile phones, PADs, etc.) and PCs.

[0046] Figure 2A This embodiment shows a flowchart of a speech recognition method. Figure 2B This application provides a schematic diagram of the structure of a neural network model for speech recognition, as illustrated in an embodiment. Figure 2A As shown, it includes:

[0047] S201. Based on the speech features of the speech to be recognized, obtain multiple modeling units corresponding to the speech to be recognized.

[0048] In this embodiment, the modeling unit is the basic unit for speech recognition. The modeling unit can be pinyin, factors, characters, words, etc. This embodiment does not limit the specific implementation of the modeling unit.

[0049] Specifically, taking phonemes as the modeling unit as an example, a phoneme is the smallest unit of speech defined based on the natural attributes of speech. These natural attributes can include pitch, duration, timbre, and also the articulation location and articulation action. The definitions of these factors can be found in relevant technical documents and will not be elaborated upon here.

[0050] In this embodiment, specifically, according to the speech features of the speech to be recognized, multiple modeling units corresponding to the speech to be recognized can be obtained, and the posterior probabilities respectively corresponding to the multiple modeling units can be determined.

[0051] See Figure 2B , in this embodiment, the neural network model may include an acoustic model AM, and the acoustic model may be a model for calculating the mapping relationship between speech features and modeling units.

[0052] Through the acoustic model, according to the speech features of the speech to be recognized, a phoneme sequence corresponding to the speech features can be determined, and the score of the speech to be recognized corresponding to the phoneme sequence. Each phoneme in the phoneme sequence can be a modeling unit, and the score output by the acoustic model can specifically be the posterior probability of the phoneme sequence.

[0053] In addition, it should be noted that since the speech recognition process is generally performed in units of frames, this step may include:

[0054] Divide the speech to be recognized into multiple frames. For the speech to be recognized in the current frame, obtain the modeling unit corresponding to the speech to be recognized in the current frame according to the speech features of the speech to be recognized in the current frame; add the modeling unit corresponding to the speech to be recognized in the current frame to the end of the path corresponding to the speech to be recognized in the previous frame, and perform beam search to obtain multiple paths corresponding to the speech to be recognized in the current frame and their corresponding first candidate scores.

[0055] Exemplarily, the speech to be recognized can be divided into 10 frames. If the modeling units determined according to the first 5 frames are "come on", then the modeling units determined according to the 6th frame may include "laugh", "xiao", "school". Then, the three modeling units corresponding to the 6th frame can be respectively added to the end of "come on" to obtain three paths. After that, through beam search, the paths "come on laugh", "come on xiao" corresponding to the speech to be recognized in the 6th frame can be obtained; further, the modeling units determined according to the 7th frame may include "na", "talk", "draw". Then, the three modeling units corresponding to the 7th frame can be respectively added to the ends of "come on laugh" and "come on xiao". After that, through beam search, the paths "come on joke", "come on laugh and accept" corresponding to the 7th frame can be obtained.

[0056] S202. Perform beam search on the multiple modeling units corresponding to the speech to be recognized to obtain multiple paths and their corresponding first scores.

[0057] The path is used to represent the candidate character sequence corresponding to the speech to be recognized.

[0058] In this embodiment, multiple modeling units can form a search space and perform beam search in the search space to obtain multiple candidate paths. The token score of each candidate path can be calculated, and the candidate paths with the highest scores are retained as the paths obtained by beam search. The first score corresponding to the path can be determined.

[0059] In this embodiment, the first score corresponding to the path can be determined based on the token score obtained by performing a cluster search and the posterior probability output by the aforementioned sound model.

[0060] See Figure 2B Neural network models used for speech recognition can include decoders and language models.

[0061] The decoder can take the modeling units and their posterior probabilities as input from the acoustic model. A beam search can then be performed based on the modeling units and the decoder, yielding multiple candidate paths. These candidate paths can be scored using a language model to obtain token scores for each path. The candidate paths with the highest scores are retained as the paths obtained from the beam search.

[0062] The decoder can then input the path into the language model, which scores the path. The decoder can then determine the first score corresponding to the path based on the scoring result of the language model (LM). The language model is used to estimate the probability of the existence of a candidate character sequence when it is recognized.

[0063] The decoder can be deployed on a CPU or a GPU. Specifically, the decoder can be an end-to-end decoder or a Beamsearch decoder. This embodiment does not limit this.

[0064] Of course, in practice, decoders can be deployed on both the CPU and the GPU, and the results of multiple decoders can be fused together and output based on the fusion result.

[0065] S203. Obtain the preferred words of the pronunciation object corresponding to the speech to be recognized.

[0066] Optionally, in this embodiment of the application, obtaining the preferred words of the pronunciation object corresponding to the speech to be recognized includes: identifying the pronunciation user corresponding to the speech to be recognized based on the voiceprint features of the speech to be recognized, and obtaining the preferred words corresponding to the pronunciation user; or, determining the device user corresponding to the speech to be recognized based on the device information of the electronic device that acquired the speech to be recognized, and obtaining the preferred words corresponding to the device user. Thus, the pronunciation object can be determined based on the user's voiceprint or the user's device information.

[0067] Specifically, in this embodiment, if the speech object is a speech user, the preferred words are determined based on at least one of the following: the speech user's phone book, the speech user's audio / video playlist, or the speech user's custom name for the IoT device; or, if the speech object is a speech user, the preferred words are determined based on the speech user's user characteristic information; or, if the speech object has a historical record, the preferred words include words extracted from the historical record; or, if the speech object has a historical record, multiple speech objects are clustered based on their respective historical records to obtain cluster groups, and speech objects in the same cluster group share the preferred words extracted from the historical record; or, if the speech object is a device user, the preferred words are determined based on at least one of the following: the phone book or audio / video playlist stored in the used electronic device, or the geographical location information of the used electronic device; or, the preferred words are determined based on media data with high popularity.

[0068] S204. Determine the modeling unit corresponding to the preferred words.

[0069] The modeling unit in this step has the same meaning as the modeling unit mentioned above, and will not be repeated here.

[0070] S205. For any path, match the modeling units included in the path with the modeling units corresponding to the preference words, and determine the second score corresponding to the path based on the matching results.

[0071] See Figure 2B In this embodiment, the decoder can match the modeling unit with the preferred words, thereby obtaining the second score corresponding to the path based on the matching result.

[0072] Optionally, in this embodiment, if the posterior probabilities corresponding to multiple modeling units are determined in the above steps.

[0073] Step S205 may specifically include: determining the modeling units with the highest posterior probabilities from among the multiple modeling units corresponding to the speech to be recognized; for any path, determining the modeling units with the highest posterior probabilities included in the current path; matching the determined modeling units with the modeling units corresponding to the preferred words; and determining the second score corresponding to the current path based on the matching results.

[0074] If the posterior probability is low, it means that the probability of determining the speech recognition result based on it is low. Therefore, in this embodiment, in order to reduce the amount of computation, the modeling units with the highest posterior probabilities are matched with the preferred words.

[0075] Optionally, in this embodiment, if in step S201, speech recognition is performed frame by frame, then in order to reduce the computational complexity, step S205 may specifically include: for any path corresponding to the speech to be recognized in the current frame, matching the modeling units on the current path with the modeling units corresponding to the preferred words; if the modeling units on the current path are partially matched with the preferred words, adding a pending match flag to the current path and recording the preferred words that are partially matched with the current path, so that when matching the path corresponding to the speech to be recognized in the next frame, the path with the pending match flag and the recorded preferred words are preferentially matched, and the second score corresponding to the path with the pending match flag is determined according to the matching result.

[0076] Exemplarily, if the paths corresponding to the speech to be recognized in the current frame include "give a laugh" and "come to a Xiao", the two paths can be matched with the preferred words, and it is determined that "give a laugh" is partially matched with the preferred word "accept with pleasure". At this time, a pending match flag can be added to the path "give a laugh". When obtaining the paths corresponding to the speech to be recognized in the next frame, which include "give a laugh" + "word" and "give a laugh" + "accept", and these two paths have the pending match flag, then these two paths can be matched with the recorded preferred word "accept with pleasure". Among them, the path "give a laugh" + "accept" is matched with the preferred word "accept with pleasure", and it can be determined that the second score of this path is 0.5 points; the path "give a laugh" + "word" has no matching preferred word, so it is determined that the second score of this path is 0 points.

[0077] In addition, since there may be a situation where the posterior probability is small but there may be matching preferred words, in this embodiment, when matching the candidate character sequence with the preferred words, specifically: determining multiple preferred pronunciations corresponding to the preferred words and the occurrence times of each of the preferred pronunciations, and determining the target preferred pronunciation as the preferred pronunciation with the occurrence times in the front row; determining the characters in the candidate character sequence that match the target preferred pronunciation, matching the determined characters with the preferred words, and determining the second scores corresponding to the multiple candidate character sequences according to the matching result.

[0078] For example, according to the preferred pronunciation of the preferred word, if it is determined that the number of occurrences of "zhou" is relatively large, then it can be determined as the target preferred pronunciation. If a character with the pronunciation of "zhou" appears in the candidate character sequence, it can be matched with the preferred word. Specifically, if the preferred word includes "Jay Chou", then its corresponding pronunciation is "zhou", "jie", "lun"; if there is a character with the pronunciation of "zhou" in the candidate character sequence, it is determined whether the pronunciations of the next two characters are "jie" and "lun". If so, it is determined that the candidate character sequence matches the preferred word "Jay Chou", and the second score of the candidate character sequence can be determined to be 0.5; if not, it is determined that the candidate character sequence does not match the preferred word, and the second score of the candidate character sequence can be determined to be 0.

[0079] Furthermore, in order to reduce the computational amount, the preferred pronunciations that appear in the forefront and whose proportion of the total number of occurrences is greater than a preset proportion can be determined as the target preferred pronunciations, where the preset proportion is determined according to the probability that the preferred word appears in the speech to be recognized. If some pronunciations appear less frequently and their probability of appearing in the speech to be recognized is also small, resulting in a small probability of being matched, although they appear in the forefront, in order to reduce the computational amount, they are not taken as the target preferred pronunciations here.

[0080] In addition, in this embodiment, when determining the second score, multiple candidate character sequences can be first matched with the preferred word to determine the preferred word that matches the candidate character sequence; then, according to the preferred word that matches the candidate character sequence, the second score corresponding to the candidate character sequence is determined. Thus, the second score corresponding to the candidate character sequence can be determined according to the possibility that the pronunciation object says the preferred word.

[0081] Furthermore, in this embodiment, determining the second score corresponding to the candidate character sequence according to the preferred word that matches the candidate character sequence may specifically include:

[0082] Pre-determine the preference scores corresponding to each preferred word through a scoring model, and determine the preference score corresponding to the preferred word that matches the candidate character sequence as the second score of the candidate character sequence.

[0083] In this embodiment, the preference scores corresponding to the preferred words can be pre-determined through a trained scoring model. The scoring model can take the historical data of the pronunciation object, the preferred words, etc. as inputs and output the preference scores corresponding to each preferred word.

[0084] In another implementation, determining a second score corresponding to the candidate character sequence based on the preference words that match the candidate character sequence may include: determining a second score of the candidate character sequence based on the length of the preference words that match the candidate character sequence, wherein the value of the second score is proportional to the length of the preference words.

[0085] The longer the preferred word matched by the candidate character sequence, the greater the likelihood that the user will say that preferred word, and the higher the corresponding second score.

[0086] S206. Based on the first score and second score corresponding to each of the multiple candidate character sequences, determine the target character sequence as the speech recognition result from the multiple candidate character sequences.

[0087] The specific implementation method of this step can be referred to the above embodiments, and will not be repeated here.

[0088] The speech recognition method of this embodiment can be executed by any suitable electronic device with data processing capabilities, including but not limited to: servers, mobile terminals (such as mobile phones, PADs, etc.) and PCs.

[0089] See Figure 3A The figure shows a schematic diagram of the structure of a speech recognition system provided in this application, as shown in the figure, which includes:

[0090] Acquisition equipment, used to acquire the speech to be recognized;

[0091] The encoder is used to encode the speech to be recognized, determine the modeling unit corresponding to the speech to be recognized, and determine the posterior probability corresponding to the modeling unit.

[0092] The voiceprint recognition module is used to perform voiceprint recognition based on the speech to be recognized and to determine the speaker corresponding to the speech.

[0093] The preferred word determination module is used to determine the preferred words corresponding to the pronunciation object.

[0094] One data collection device corresponds to one UUID. One data collection device can include multiple users, and each user can have their own preferred keywords. These preferred keywords can be extracted based on the user's historical data. In addition, the phone book, page information, etc. stored in the data collection device can serve as shared preferred keywords for multiple users.

[0095] One or more decoders are used to perform a bundle search based on modeling units to obtain multiple paths, and call the language model to determine the language score corresponding to the path. Based on the language score, the first score corresponding to the path is determined. The modeling units in each path are matched with the preference words, and the second score corresponding to the path is determined based on the matching results.

[0096] See Figure 3B Specifically, when matching the modeling units in each path with preference words and determining the second score corresponding to the path based on the matching results, the following steps may be included:

[0097] 1) Determine whether the posterior probability of the modeling unit corresponding to the path is in the TOP-k range;

[0098] If yes, continue with the next steps; otherwise, return a result of no match.

[0099] 2) Determine whether the pronunciation of the modeling unit corresponding to the path is in the top-n, that is, determine whether the pronunciation of the modeling unit includes the target preference pronunciation.

[0100] If yes, continue with the next steps; otherwise, return a result of no match.

[0101] The top-n here is determined based on the pronunciation of the preferred words. For details on how to determine the pronunciation of the target preferred words, please refer to the above method.

[0102] 3) Determine if the path contains matching preference terms.

[0103] 4) Return the matching results.

[0104] The matching results may include whether the path contains a matching preference term. If so, the returned matching results may also include a second score for the path determined based on the matching results.

[0105] The fusion module is used to fuse the output paths of multiple decoders and the scores corresponding to each path to determine the speech recognition result.

[0106] In this embodiment, the first score characterizes the probability of identifying the candidate character sequence as the speech recognition result based on speech features, and the second score characterizes the probability of identifying the candidate character sequence as the speech recognition result based on preferred words. Therefore, by combining the first and second scores, existing experience in speech recognition based on speech features can be comprehensively considered, along with the preferred words that the speaker hopes to hit, to obtain a personalized speech recognition result for the speaker.

[0107] Furthermore, in this embodiment, the second score is determined by matching candidate character sequences with the preferred words, thus decoupling the determination processes of the first and second scores. This allows for flexible adjustment of the preferred words for the pronunciation object as needed, without affecting the determination process of the first score. This makes the speech recognition method provided in this embodiment highly transferable and allows for the addition of a large number of preferred words. Moreover, the solution provided in this embodiment can directly add preferred word matching logic to an existing decoder. If preferred word matching logic is not needed, the second score can be directly set to 0. The solution provided in this embodiment can be obtained with minor modifications to existing language recognition models, making it simple and convenient to implement.

[0108] The speech recognition method of this embodiment can be executed by any suitable electronic device with data processing capabilities, including but not limited to: servers, mobile terminals (such as mobile phones, PADs, etc.) and PCs.

[0109] Figure 4 A flowchart illustrating the steps of an interaction method for a smart IoT device provided in this application embodiment is shown in the figure, which includes:

[0110] S401. Receive and send the speech to be recognized through a smart IoT device, and determine the preferred words corresponding to the pronunciation object of the speech to be recognized.

[0111] In this embodiment, the smart IoT device can specifically be a device capable of receiving and transmitting the voice to be recognized. For example, the smart IoT device can be a smart speaker, a smart TV with a microphone, an in-vehicle system, etc., but this embodiment does not limit this to a specific type.

[0112] In this embodiment, the preferred words corresponding to the pronunciation object can be determined based on the device identifier of the smart IoT device; or the pronunciation object can be determined by voiceprint recognition of the speech to be recognized, and then the preferred words corresponding to the pronunciation object can be determined.

[0113] Optionally, in this embodiment, in response to a user's setting operation, the preferred words can be determined based on at least one of the following: the phone book stored in the smart IoT device, the audio and video list of the smart IoT device, the historical records corresponding to the smart IoT device, and the custom names of other IoT devices in the network where the smart IoT device is located.

[0114] Specifically, in this embodiment, the user's setting operation can be received through a smart IoT device or other smart devices besides smart IoT devices, and the preferred words can be determined based on the received user operation. In this embodiment, a single word can be determined as the preferred word, or multiple words can be combined into one preferred word. For example, "confession balloon" can be used as a preferred word, or the combination of the two words "Jay Chou + confession balloon" can be used as the preferred word.

[0115] S402. Based on the speech features of the speech to be recognized and the preferred words, perform speech recognition to obtain a first speech recognition result that matches the preferred words and a second speech recognition result that does not match the preferred words.

[0116] The speech features are used to determine candidate character sequences and a first score as candidate speech recognition results, and the preferred words are used to determine a second score based on the matching results between the candidate character sequences and these sequences. The specific methods for determining the first and second scores can be found in the above embodiments and will not be repeated here.

[0117] After determining the first score and the second score corresponding to each candidate character sequence, the first speech recognition result can be determined from the candidate character sequences that match the preferred words based on the first score and the second score, and the second speech recognition result can be determined from the candidate character sequences that do not match the preferred words based on the first score.

[0118] S403. The first speech recognition result and its matched preferred words, as well as the second speech recognition result, are output through the smart IoT device.

[0119] Specifically, the output can be made through the speaker or display device of a smart IoT device.

[0120] S404. If a negative operation is received from the user, the second score corresponding to the output preference word is reduced.

[0121] If a negative operation from the user is received, it indicates that the user is not very satisfied with the speech recognition results determined based on the preferred words. In this case, the second score corresponding to the preferred word can be reduced to make corrections. Alternatively, the second score corresponding to the preferred word can be set to zero if the preferred word is no longer used as the pronunciation object.

[0122] Optionally, in this embodiment, the negative operations of the user include at least one of the following: a selection operation in which the user selects a second speech recognition result that does not match the preference word as the target speech recognition result; a negation operation in which the user does not use either the first speech recognition result or the second speech recognition result as the target speech recognition result; an interruption operation in which the user interrupts the output of the intelligent IOT device; and an input operation in which the user re-enters the speech to be recognized.

[0123] Exemplarily, the intelligent IOT device can output the first speech recognition result and its matching preference word, as well as the second speech recognition result, through a speaker or an interface. If the user inputs the speech "neither" after obtaining the first speech recognition result and its matching preference word, and the second speech recognition result, it can be determined that the user has input a negation operation in which neither the first speech recognition result nor the second speech recognition result is used as the target speech recognition result. The interruption operation can specifically include: the intelligent IOT device receives information that the user has left, the intelligent IOT device receives a pause or return operation, etc.

[0124] See Figure 5 , the user inputs the speech to be recognized "give me a 'xiaona'" to the intelligent IOT device, and the corresponding preference word of the user can be provided. The user can determine the first speech recognition result "give me a 'xiaona'" corresponding to the preference word "xiaona" and the second speech recognition result "tell me a joke" that does not match the preference word based on the speech to be recognized and the preference word.

[0125] The intelligent IOT device can output speech to the user, and the speech content can be: Do you want a joke or the song 'xiaona' that you like.

[0126] If the speech input by the user is "the song 'xiaona'", it can be determined that the user has input a positive operation in which the first speech recognition result is used as the target speech recognition result.

[0127] Alternatively, if the speech input by the user is "joke", it can be determined that the user has input a selection operation in which the second speech recognition result that does not match the preference word is selected as the target speech recognition result. Alternatively, if the speech input by the user is "neither", it can be determined that the user has input a negation operation. Alternatively, if the user inputs other speech, it conforms to the input operation in which the user re-enters the speech to be recognized. When the foregoing situation occurs, it can be determined that the user has input a negative operation, and the second score corresponding to the preference word "xiaona" can be reduced, or "xiaona" is no longer used as a preference word, that is, the second score corresponding to "xiaona" is set to zero.

[0128] Figure 6The figure shows a schematic diagram of a smart IoT device provided in an embodiment of this application. It includes a microphone 601, a processor 602, and a communication device 603.

[0129] The microphone 601 is used to receive the voice to be recognized;

[0130] The processor 602 is used to control the communication device 403 to send the speech to be recognized to the cloud, so that the cloud obtains multiple candidate character sequences corresponding to the speech to be recognized and their corresponding first scores based on the speech features of the speech to be recognized; obtains the preferred words of the pronunciation object corresponding to the speech to be recognized; matches the multiple candidate character sequences with the preferred words, and determines the second scores corresponding to the multiple candidate character sequences respectively based on the matching results; and determines the target character sequence as the speech recognition result from the multiple candidate character sequences based on the first score and the second score corresponding to each of the multiple candidate character sequences.

[0131] The communication device 603 receives the voice recognition results.

[0132] In this embodiment, the receiver can be any device capable of recording sound, such as a microphone; this embodiment is not limited in this regard. The communication device can transmit the voice to be processed to the cloud via cellular networks, Wi-Fi, etc.; this embodiment is not limited in this regard either.

[0133] In this embodiment, the received speech recognition result can be the first speech recognition result and the second speech recognition result in the above embodiment, or the received speech recognition result can be a final speech recognition result. This embodiment does not limit this.

[0134] The smart speaker in this embodiment enables the cloud to implement the corresponding speech recognition methods in the aforementioned multiple method embodiments, and has the beneficial effects of the corresponding method embodiments, which will not be repeated here.

[0135] Reference Figure 7 The diagram shows a structural schematic of an electronic device according to Embodiment 5 of this application. The specific embodiments of this application do not limit the specific implementation of the electronic device.

[0136] like Figure 7 As shown, the electronic device may include: a processor 702, a communications interface 704, a memory 706, and a communications bus 708.

[0137] in:

[0138] The processor 702, communication interface 704, and memory 706 communicate with each other via communication bus 708.

[0139] Communication interface 704 is used to communicate with other electronic devices or servers.

[0140] The processor 702 is used to execute program 710, specifically to perform the relevant steps in the above-described speech recognition method embodiment.

[0141] Specifically, program 710 may include program code that includes computer operation instructions.

[0142] The processor 702 may be a CPU, an Application Specific Integrated Circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of this application. The smart device includes one or more processors, which may be processors of the same type, such as one or more CPUs; or processors of different types, such as one or more CPUs and one or more ASICs.

[0143] Memory 706 is used to store program 710. Memory 706 may include high-speed RAM memory, and may also include non-volatile memory, such as at least one disk storage device.

[0144] Specifically, program 710 can be used to cause processor 702 to execute the operations in the above method embodiments. The specific implementation of each step in program 710 can be found in the corresponding descriptions of the steps and units in the above speech recognition method embodiments, and will not be repeated here. Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working process of the devices and modules described above can be referred to the corresponding process descriptions in the foregoing method embodiments, and will not be repeated here.

[0145] This application also provides a computer program product, including computer instructions that instruct a computing device to perform an operation corresponding to any of the speech recognition methods in the above-described multiple method embodiments.

[0146] It should be noted that, depending on the implementation needs, the various components / steps described in the embodiments of this application can be broken down into more components / steps, or two or more components / steps or parts of the operation of components / steps can be combined into new components / steps to achieve the purpose of the embodiments of this application.

[0147] The methods described in the embodiments of this application can be implemented in hardware, firmware, or as software or computer code that can be stored in a recording medium (such as a CD-ROM, RAM, floppy disk, hard disk, or magneto-optical disk), or as computer code downloaded over a network that is originally stored in a remote recording medium or a non-transitory machine-readable medium and will be stored in a local recording medium. Thus, the methods described herein can be processed by software stored on a recording medium using a general-purpose computer, a dedicated processor, or programmable or dedicated hardware (such as an ASIC or FPGA). It is understood that the computer, processor, microprocessor controller, or programmable hardware includes storage components (e.g., RAM, ROM, flash memory, etc.) capable of storing or receiving software or computer code that, when accessed and executed by the computer, processor, or hardware, implements the speech recognition methods described herein. Furthermore, when a general-purpose computer accesses code used to implement the speech recognition methods shown herein, the execution of the code transforms the general-purpose computer into a dedicated computer for executing the speech recognition methods shown herein.

[0148] Those skilled in the art will recognize that the units and method steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the embodiments of this application.

[0149] The above embodiments are only used to illustrate the embodiments of this application, and are not intended to limit the embodiments of this application. Those skilled in the art can make various changes and modifications without departing from the spirit and scope of the embodiments of this application. Therefore, all equivalent technical solutions also fall within the scope of the embodiments of this application, and the patent protection scope of the embodiments of this application should be defined by the claims.

Claims

1. A speech recognition method, comprising: Based on the speech features of the speech to be recognized, obtain multiple candidate character sequences corresponding to the speech to be recognized and their corresponding first scores; Obtain the preferred words of the pronunciation object corresponding to the speech to be recognized, and the preferred words are obtained based on the personalized data of the pronunciation object; Matching multiple candidate character sequences with the preference words, and determining the second score corresponding to each of the multiple candidate character sequences based on the matching results, includes: for any path corresponding to the speech to be recognized in the current frame, matching the modeling unit on the current path with the modeling unit corresponding to the preference word, wherein the path is used to represent the candidate character sequence corresponding to the speech to be recognized; if the modeling unit on the current path partially matches the preference word, then adding a matching identifier to the current path, and recording the preference word that partially matches the current path, so that when matching the path corresponding to the speech to be recognized in the next frame, the paths with matching identifiers and the recorded preference words are matched first, and the second score corresponding to the paths with matching identifiers is determined based on the matching results; Based on the first score and the second score corresponding to each of the multiple candidate character sequences, the target character sequence as the speech recognition result is determined from the multiple candidate character sequences.

2. The method according to claim 1, wherein, The step of obtaining multiple candidate character sequences and their corresponding first scores for the speech to be recognized based on the speech features includes: Based on the speech features of the speech to be identified, multiple modeling units corresponding to the speech to be identified are obtained. A beam search is performed on the multiple modeling units corresponding to the speech to be identified to obtain multiple paths and their corresponding first scores.

3. The method according to claim 2, wherein, The step of obtaining multiple modeling units corresponding to the speech to be recognized based on the speech features to be recognized includes: Based on the speech features of the speech to be identified, multiple modeling units corresponding to the speech to be identified are obtained, and the posterior probabilities corresponding to the multiple modeling units are determined respectively. The step of matching multiple candidate character sequences with the preferred words and determining the second score corresponding to each of the multiple candidate character sequences based on the matching results includes: Determine the modeling unit corresponding to the preferred words; From the multiple modeling units corresponding to the speech to be recognized, the modeling units with the highest posterior probabilities are determined; For any path, identify the modeling units whose posterior probabilities are among the highest in the current path, match the identified modeling units with the modeling units corresponding to the preferred words, and determine the second score corresponding to the current path based on the matching results.

4. The method according to claim 2, wherein, The process involves obtaining multiple modeling units corresponding to the speech to be recognized based on its speech features, performing a beam search on these modeling units to obtain multiple paths and their corresponding first scores. The paths are used to represent candidate character sequences corresponding to the speech to be recognized, including: The speech to be recognized is divided into multiple frames. For the speech to be recognized in the current frame, a modeling unit corresponding to the speech to be recognized in the current frame is obtained based on the speech features of the speech to be recognized in the current frame. The modeling unit corresponding to the speech to be recognized in the current frame is added to the end of the path corresponding to the speech to be recognized in the previous frame, and a beam search is performed to obtain multiple paths corresponding to the speech to be recognized in the current frame and their corresponding first candidate scores.

5. The method according to claim 1, wherein, The step of matching multiple candidate character sequences with the preferred words and determining the second score corresponding to each of the multiple candidate character sequences based on the matching results includes: Determine multiple preferred pronunciations corresponding to the preferred words and the frequency of occurrence of each preferred pronunciation, and determine the preferred pronunciations with the highest frequency of occurrence as target preferred pronunciations; Identify the characters in the candidate character sequences that match the target preferred pronunciation, match the identified characters with the preferred words, and determine the second scores corresponding to the multiple candidate character sequences based on the matching results.

6. The method according to claim 5, wherein, The step of identifying the most frequently occurring preferred pronunciations as target preferred pronunciations includes: The preferred pronunciations that appear most frequently and whose proportion of occurrences to the total number of occurrences is greater than a preset proportion are identified as target preferred pronunciations, wherein the preset proportion is determined based on the probability of the preferred words appearing in the speech to be recognized.

7. The method according to claim 1, wherein, The step of matching multiple candidate character sequences with the preferred words and determining the second score corresponding to each of the multiple candidate character sequences based on the matching results includes: Match multiple candidate character sequences with the preferred words to determine the preferred words that match the candidate character sequences; A second score is determined based on the preferred words that match the candidate character sequence.

8. The method according to claim 7, wherein, The step of determining the second score corresponding to the candidate character sequence based on the preferred words that match the candidate character sequence includes: The preference score corresponding to each preference word is pre-determined by a scoring model, and the preference score corresponding to the preference word that matches the candidate character sequence is determined as the second score of the candidate character sequence. Alternatively, a second score for the candidate character sequence can be determined based on the length of the preferred word that matches the candidate character sequence, wherein the value of the second score is proportional to the length of the preferred word.

9. The method according to claim 1, wherein, The step of obtaining the preferred words of the pronunciation object corresponding to the speech to be recognized includes: Based on the voiceprint features of the speech to be identified, identify the speaker corresponding to the speech to be identified, and obtain the preferred words of the speaker. Alternatively, based on the device information of the electronic device from which the speech to be recognized is obtained, the device user corresponding to the speech to be recognized can be determined, and the preferred words corresponding to the device user can be obtained.

10. An interaction method for a smart IoT device, comprising: The system receives and sends the speech to be recognized through a smart IoT device, and determines the preferred words corresponding to the pronunciation object of the speech to be recognized. The preferred words are obtained based on the personalized data of the pronunciation object. Speech recognition is performed based on the speech features of the speech to be recognized and the preferred words to obtain a first speech recognition result that matches the preferred words and a second speech recognition result that does not match the preferred words. The speech features are used to determine candidate character sequences and a first score as candidate speech recognition results, and the preferred words are used to determine a second score based on the matching result between the candidate character sequences and the preferred words. The smart IoT device outputs the first speech recognition result and its matched preferred words, as well as the second speech recognition result; If a negative action is received from the user, the second score corresponding to the output preference word is reduced; Determining a second score based on the matching result of the candidate character sequence and the preference word includes: for any path corresponding to the speech to be recognized in the current frame, matching the modeling unit on the current path with the modeling unit corresponding to the preference word, wherein the path is used to represent the candidate character sequence corresponding to the speech to be recognized; if the modeling unit on the current path partially matches the preference word, then adding a matching identifier to the current path and recording the preference word that partially matches the current path, so that when matching the path corresponding to the speech to be recognized in the next frame, the paths with matching identifiers and the recorded preference words are matched first, and the second score corresponding to the paths with matching identifiers is determined according to the matching result.

11. The method according to claim 10, wherein, The user's negative action includes at least one of the following: The user selects a second speech recognition result that does not match the preferred words as the target speech recognition result. The user will not use either the first speech recognition result or the second speech recognition result as a negative operation for the target speech recognition result; The user interrupts the interruption operation output by the smart IoT device; The user re-enters the input operation for the speech to be recognized.

12. The method according to claim 11, wherein, The method further includes: In response to a user's settings action, the preferred words are determined based on at least one of the following: the phonebook stored in the smart IoT device, the audio and video list of the smart IoT device, the historical records corresponding to the smart IoT device, and the custom names of other IoT devices in the network where the smart IoT device is located.

13. A computer storage medium having a computer program stored thereon, which, when executed by a processor, implements the speech recognition method as described in any one of claims 1-9.

14. A computer program product comprising computer instructions that instruct a computing device to perform an operation corresponding to the speech recognition method as described in any one of claims 1-9.

Images