Speech recognition method and device, storage medium and program product
By fusing the text and audio features of speech and hot words, the problem of insufficient accuracy in hot word recognition in existing speech recognition systems is solved, thereby improving the accuracy of hot word recognition and the robustness of the system.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-01-21
- Publication Date
- 2026-03-27
AI Technical Summary
Existing speech recognition systems have low accuracy in identifying hot words, especially in contexts where users expect hot words to appear, they tend to output more common alternatives or spelling errors, affecting user experience and the availability of personalized services.
By encoding the speech to be recognized, speech encoding features are obtained, and combined with the text and audio encoding features of hot words in the target hot word set, a semantic alignment network is used to fuse them, thereby improving the accuracy of hot word recognition.
By integrating the text and audio features of hot words, the accuracy of hot word recognition is improved, and the robustness and sensitivity of the speech recognition system in complex contexts are enhanced.
Smart Images
Figure CN121747541A_ABST
Abstract
Description
TECHNICAL FIELD
[0001] The present application relates to the technical field of artificial intelligence, and in particular to a speech recognition method and device, a storage medium and a program product. BACKGROUND
[0002] With the wide deployment of speech interaction and automatic transcription in scenarios such as intelligent assistants, telephone customer service, meeting minutes, media subtitles, reliable recognition of hot words (such as rare names, place names, brand names, professional terms, or user-defined words, etc.) has become a key problem for speech recognition systems. Current speech recognition systems perform well on common words, but when encountering hot words, the recognition error rate increases significantly, which not only affects user experience, but also restricts the availability of personalized services.
[0003] In order to improve the recognition accuracy of hot words, the research and engineering direction of "hot word enhancement" has emerged. The recognition accuracy of hot words based on the current hot word enhancement speech recognition method still needs to be improved. SUMMARY
[0004] In view of the above problems, the present application provides a speech recognition method, device, storage medium and program product to improve the recognition accuracy of hot words. The specific scheme is as follows:
[0005] The first aspect of the present application provides a speech recognition method, comprising:
[0006] encoding the to-be-recognized speech to obtain first speech encoding features of the to-be-recognized speech;
[0007] encoding the text of each hot word in the target hot word set respectively to obtain text encoding features of each hot word;
[0008] obtaining audio encoding features of each hot word obtained by encoding the audio of each hot word in the target hot word set;
[0009] fusing the text encoding features and the audio encoding features of the same hot word to obtain target encoding features of each hot word, and performing hot word enhanced speech recognition based on the target encoding features of each hot word and the first speech encoding features to obtain a speech recognition result of the to-be-recognized speech.
[0010] In one possible implementation, the to-be-recognized speech is encoded by a speech encoding module, the text of each hot word is encoded by a first text encoding module, the audio of each hot word is encoded by an audio encoding module, the text encoding features and the audio encoding features of the same hot word are fused by an identification module, and hot word enhanced speech recognition is performed based on the target encoding features of each hot word and the first speech encoding features;
[0011] The audio encoding module belongs to a semantic alignment network, and the semantic alignment network further includes a second text encoding module; the semantic alignment network is pre-trained based on a speech-text pair semantic alignment task of speech and text.
[0012] The speech encoding module, the first text encoding module, and the recognition module are jointly trained based on a hotword-enhanced speech recognition task.
[0013] In a possible implementation, the semantic alignment network is trained based on speech-text pairs as training samples; and a training process of the semantic alignment network includes:
[0014] The speech in the training samples is encoded by the audio encoding module of the semantic alignment network to obtain encoded features of the speech in each training sample, and the text in the training samples is encoded by the second text encoding module of the semantic alignment network to obtain encoded features of the text in each training sample.
[0015] A first similarity of the text encoded features and the speech encoded features obtained by the second text encoding module and the audio encoding module encoding the text and the speech in the same training sample, and a second similarity of the text encoded features and the speech encoded features obtained by the second text encoding module and the audio encoding module encoding the text and the speech in different training samples are obtained.
[0016] At least the parameters of the semantic alignment network are updated with the first similarity becoming smaller and the second similarity becoming larger as the target.
[0017] In a possible implementation, the at least updating the parameters of the semantic alignment network includes:
[0018] The parameters of the semantic alignment network and a temperature parameter in a loss function are updated;
[0019] The loss calculated based on the loss function is positively related to the first similarity and negatively related to the second similarity; and the temperature parameter is used to adjust the similarity of the text encoded features and the speech encoded features.
[0020] In a possible implementation, the encoding of the text in the training sample by the second text encoding module includes:
[0021] The text in the training sample is converted into a phoneme sequence or a syllable sequence;
[0022] The phoneme sequence is encoded by the second text encoding module; or the syllable sequence is encoded by the second text encoding module.
[0023] In a possible implementation, the target hotword set is a subset of the preset hotword set, and the target hotword set is obtained in the following manner:
[0024] obtaining an audio encoding feature of each hotword in the preset hotword set, by encoding audio of each hotword in the preset hotword set by using the audio encoding module;
[0025] obtaining a second speech encoding feature of the to-be-recognized speech, by encoding the to-be-recognized speech by using the audio encoding module;
[0026] In the preset hotword set, selecting k hotwords with minimum distances between audio encoding features and the second speech encoding feature as the target hotword set.
[0027] In a possible implementation, the method further includes:
[0028] previously splitting the audio encoding feature of each hotword in the preset hotword set into sub-encoding features of the same length, different sub-encoding features belonging to different subspaces; and clustering the sub-encoding features belonging to the same subspace in each hotword, to determine a plurality of clusters and a cluster center of each subspace;
[0029] Correspondingly, the process of calculating the distance between the audio encoding feature of any hotword and the second speech encoding feature includes:
[0030] splitting the second speech encoding feature into sub-encoding features of the same length;
[0031] calculating the distance between the cluster center of the cluster to which the sub-encoding feature of the any hotword belongs in each subspace and the sub-encoding feature of the second speech encoding feature; and fusing the distances determined in each subspace corresponding to the any hotword, to obtain the distance between the audio encoding feature of the any hotword and the second speech encoding feature.
[0032] The second aspect of the present application provides a computer program product, including computer readable instructions, when the computer readable instructions run on an electronic device, causing the electronic device to implement the voice recognition method of the first aspect or any implementation manner of the first aspect.
[0033] The third aspect of the present application provides an electronic device, including at least one processor and a memory connected with the processor, wherein:
[0034] The memory is configured to store a computer program;
[0035] The processor is configured to execute the computer program, so that the electronic device can implement the voice recognition method of the first aspect or any implementation manner of the first aspect.
[0036] A fourth aspect of this application provides a computer storage medium carrying one or more computer programs that, when executed by an electronic device, enable the electronic device to implement the speech recognition method described in the first aspect or any implementation thereof.
[0037] By employing the above technical solutions, the speech recognition method, device, storage medium, and program product provided in this application encode the speech to be recognized to obtain a first speech coding feature; encode the text of each hot word in the target hot word set to obtain the text coding feature of each hot word; obtain the audio coding feature of each hot word obtained by encoding the audio of each hot word in the target hot word set; fuse the text coding feature and audio coding feature of the same hot word to obtain the target coding feature of each hot word; and perform hot word-enhanced speech recognition based on the target coding feature and the first speech coding feature of each hot word to obtain the speech recognition result of the speech to be recognized. The hot word-enhanced speech recognition method provided in this application integrates the text features and audio features of hot words, thereby improving the accuracy of hot word recognition. Attached Figure Description
[0038] The above and other features, advantages, and aspects of the embodiments of this disclosure will become more apparent from the accompanying drawings and the following detailed description. Throughout the drawings, the same or similar reference numerals denote the same or similar elements. It should be understood that the drawings are schematic, and the originals and elements are not necessarily drawn to scale.
[0039] Figure 1 A flowchart illustrating an implementation of the speech recognition method provided in this application;
[0040] Figure 2 A schematic diagram of the structure of the speech recognition system provided in this application;
[0041] Figure 3 This application provides a flowchart of one implementation method for extracting a target hot word set from a pre-set hot word set.
[0042] Figure 4 A schematic diagram of the structure of the electronic device provided in this application. Detailed Implementation
[0043] The embodiments of this application are described below with reference to the accompanying drawings. The terminology used in the implementation section of this application is for explaining specific embodiments only and is not intended to limit the scope of this application.
[0044] The embodiments of this application will now be described with reference to the accompanying drawings. Those skilled in the art will recognize that, with technological advancements and the emergence of new scenarios, the technical solutions provided in the embodiments of this application are equally applicable to similar technical problems.
[0045] The terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such terms are interchangeable where appropriate; this is merely a way of distinguishing objects with the same attributes in the embodiments of this application. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion, so that a process, method, system, product, or apparatus that comprises a series of elements is not necessarily limited to those elements, but may include other elements not explicitly listed or inherent to those processes, methods, products, or apparatuses.
[0046] The hot word problem stems from two root causes: first, data sparsity, with many proper nouns (i.e., hot words) appearing almost entirely absent from the training corpus; and second, model representation and decoding biased towards end-to-end models (e.g., RNN-T, CTC, Attention-based) or WFST-based systems, whose prior probabilities or subword distributions are not always favorable to certain immediately important words (i.e., hot words). As a result, even in contexts where users expect hot words, the system may still output more "common" alternatives or spelling errors. To improve the accuracy of hot word speech recognition, hot word-enhanced speech recognition methods have emerged.
[0047] The inventors discovered that current hot word enhancement speech recognition methods are all based on hot word text enhancement, failing to fully utilize the acoustic and semantic cues of hot word speech. Speech recognition systems lack sufficient modeling of the correspondence between hot words and speech signals. For hot words with similar semantics but inconsistent pronunciation details, relying solely on text information is insufficient to eliminate ambiguity, resulting in low hot word recognition accuracy. Therefore, this application provides a speech recognition method that enhances hot word speech recognition based on the features of both the text and audio modalities of hot words, thereby improving the accuracy of hot word recognition.
[0048] like Figure 1 The diagram shown is a flowchart of one implementation of the speech recognition method provided in this application, which may include:
[0049] Step S101: Encode the speech to be recognized to obtain the speech coding features of the speech to be recognized (for ease of description and differentiation, it is referred to as the first speech coding feature).
[0050] The acoustic features of each speech frame of the speech to be recognized can be extracted first, and the acoustic features of each speech frame can be encoded to obtain the first speech coding features of the speech to be recognized.
[0051] Step S102: Encode the text of each hot word in the target hot word set to obtain the text encoding features of each hot word.
[0052] The target hotword set can be the preset hotword set, or the target hotword set can be a subset of the preset hotword set.
[0053] As an example, whether to directly use the preset hotword set for hotword enhancement or to use a subset of the preset hotword set for hotword enhancement can be determined according to the size of the preset hotword set. For example, if the number of hotwords in the preset hotword set is less than a preset threshold, the preset hotword set can be used as the target hotword set; if the number of hotwords in the preset hotword set is greater than or equal to the preset threshold, hotwords associated with the to-be-recognized speech are extracted from the preset hotword set as the target hotword set, i.e., the target hotword set is a subset of the preset hotword set.
[0054] If the number of hotwords in the preset hotword set is greater than or equal to the preset threshold, using the preset hotword set for hotword enhancement in speech recognition can easily cause problems such as calculation redundancy and reduced accuracy. To avoid this problem, when the number of hotwords in the preset hotword set is greater than or equal to the preset threshold, a subset of hotwords associated with the to-be-recognized speech is extracted from the preset hotword set for hotword enhancement in speech recognition.
[0055] Step S103: Obtain audio encoding features of each hotword in the target hotword set.
[0056] That is, in the embodiments of the present application, the preset hotword set contains both the text of the hotword and the audio of the hotword.
[0057] The audio encoding features of the hotword can be obtained by encoding the audio of each hotword in the target hotword set in real time. Based on this, step S103 can be to encode the audio of each hotword in the target hotword set to obtain the audio encoding features of each hotword.
[0058] The audio encoding features of the hotword can be obtained by encoding the audio of each hotword in the target hotword set in real time. Based on this, step S103 can be to encode the audio of each hotword in the target hotword set to obtain the audio encoding features of each hotword.
[0059] The execution order of steps S101, S102, and S103 is not specifically limited and can be executed in a preset order or simultaneously.
[0060] Step S104: fuse the text encoding features and the audio encoding features of the same hotword to obtain target encoding features of each hotword, and perform hotword-enhanced speech recognition based on the target encoding features of each hotword and the first speech encoding features to obtain a speech recognition result of the speech to be recognized.
[0061] Optionally, the text encoding features and the audio encoding features of the same hotword can be spliced in the channel dimension to obtain the target encoding features of each hotword. For example, the dimension of the text encoding features of a hotword is HxWxC1, and the dimension of the audio encoding features is HxWxC2, and the dimension of the target encoding features of the hotword is HxWx(C1+C2).
[0062] Optionally, the text encoding features and the audio encoding features of the same hotword can be averaged to obtain the target encoding features of each hotword. For example, the dimension of the text encoding features and the audio encoding features of a hotword is HxWxC1, and the dimension of the target encoding features of the hotword is HxWxC1.
[0063] Optionally, the target encoding features of each hotword can be fused to obtain fusion features based on the attention weights of the target encoding features of each hotword based on the first speech encoding features; the fusion features and the first speech encoding features are spliced to obtain spliced features, and the spliced features are decoded (for example, the spliced features are linearly transformed, and the features after linear transformation are normalized by a softmax function) to obtain a speech recognition result.
[0064] Optionally, the target encoding features of each hotword can be fused to obtain fusion features based on the attention weights of the target encoding features of each hotword based on the first speech encoding features; the fusion features and the first speech encoding features are spliced to obtain spliced features; and the spliced features are autoregressively decoded (i.e., each time a token (such as a phoneme, a syllable, a word, or the like) is decoded, the decoded token is used for the next decoding step) to obtain a speech recognition result.
[0065] In the above two hotword-enhanced speech recognition methods, the process of fusing the target encoding features of each hotword based on the attention weights of the target encoding features of each hotword based on the first speech encoding features to obtain fusion features can include:
[0066] The target encoding features of each hotword are weighted and summed based on the single-head attention weights of the target encoding features of each hotword based on the first speech encoding features to obtain fusion features.
[0067] Or, obtain the multi-head attention weight of the first speech coding feature pair target coding feature of each hot word, for each attention head, based on the attention weight of the first speech coding feature pair target coding feature of each hot word in the attention head, weight sum the target coding feature of each hot word to obtain the fusion feature corresponding to the attention head; concatenate the fusion features corresponding to each attention head and perform linear transformation to obtain the fusion feature.
[0068] The speech recognition method provided by the embodiment of the application fuses the text features and audio features of the hot words when performing hot word enhanced speech recognition, that is, the hot word enhanced speech recognition process introduces hot word text information and acoustic information, thereby improving the recognition accuracy of the hot words.
[0069] In an optional embodiment, the speech recognition method is implemented by a pre-trained speech recognition system, as shown in Figure 2 The speech recognition system provided by the embodiment of the application can include the following components:
[0070] The speech recognition system provided by the embodiment of the application can include the following components:
[0071] The speech coding module 201 is configured to code the to-be-recognized speech to obtain the first speech coding feature of the to-be-recognized speech.
[0072] The first text coding module 202 is configured to code the text of each hot word respectively to obtain the text coding feature of each hot word.
[0073] The audio coding module 203 is configured to code the audio of each hot word respectively to obtain the audio coding feature of each hot word. Since the audio coding module 203 does not necessarily code the hot words in real time, but can code the hot words in advance and store them in a database, the audio coding module 203 can be connected with the recognition module 204 or not.
[0074] The recognition module 204 fuses the text coding feature and the audio coding feature of the same hot word to obtain the target coding feature of each hot word, and performs hot word enhanced speech recognition based on the target coding feature of each hot word and the first speech coding feature to obtain the speech recognition result of the to-be-recognized speech.
[0075] The audio encoding module 203 belongs to a semantic alignment network, the semantic alignment network further includes a text encoding module (denoted as a second text encoding module for the sake of description and distinction), and the semantic alignment network is pre-trained based on a voice-text pair for a voice-text semantic alignment task. As an example, the audio encoding module 203 can be a network of a CLAP (Contrastive Language-Audio Pretraining) structure.
[0076] The voice in the voice-text pair can be audio recorded when the text is read, or can be voice obtained by performing voice synthesis on the text.
[0077] Alternatively, the text in the voice-text pair can be text obtained by performing voice recognition on the voice.
[0078] The semantic alignment task includes that the encoding result of the voice by the audio encoding module is aligned with the encoding result of the text in the same voice-text pair by the first text encoding module at a semantic level. Based on this, the audio encoding module 203 can fully understand the acoustic and semantic information of the voice, and further enhance the recognition ability of the hot word.
[0079] Optionally, the data in the two different modalities of voice and text can be aligned at a semantic level through cross-modal contrastive learning.
[0080] Optionally, the semantic alignment network is trained by taking the voice-text pair as a training sample. In the voice-text pair as the training sample, the text can include a hot word or can not include a hot word. That is, in this application, the training data used to train the semantic alignment network can not be limited to hot words, thereby improving the diversity of the training sample and further improving the robustness of the semantic alignment network. The specific training process can include:
[0081] The voice in the training sample is encoded by the audio encoding module 203 of the semantic alignment network to obtain the encoding features of the voice in each training sample, and the text in the training sample is encoded by the second text encoding module of the semantic alignment network to obtain the encoding features of the text in each training text;
[0082] A first similarity of the text encoding features and the voice encoding features of the text and the voice in the same training sample encoded by the second text encoding module and the audio encoding module 203 is obtained, and a second similarity of the text encoding features and the voice encoding features of the text and the voice in different training samples encoded by the second text encoding module and the audio encoding module 203 is obtained;
[0083] The parameters of the voice alignment network are updated at least with the first similarity becoming smaller and the second similarity becoming larger as the target.
[0084] That is, when training the semantic alignment network, the first similarity of the text encoding feature and the speech encoding feature coded by the second text encoding module and the audio encoding module 203 in the same training sample becomes smaller, the second similarity of the text encoding feature and the speech encoding feature coded by the second text encoding module and the audio encoding module in different training samples becomes larger, and at least the parameters of the semantic alignment network are updated.
[0085] Specifically, corresponding to the i th training sample and the j th (j≠i) training sample, the first similarity of the text encoding feature coded by the second text encoding module in the i th training sample and the speech encoding feature coded by the audio encoding module 203 in the i th training sample becomes smaller, the second similarity of the text encoding feature coded by the second text encoding module in the i th training sample and the speech encoding feature coded by the audio encoding module 203 in the j th training sample becomes larger, and the second similarity of the text encoding feature coded by the second text encoding module in the j th training sample and the speech encoding feature coded by the audio encoding module 203 in the i th training sample becomes larger, and at least the parameters of the second text encoding module and the parameters of the audio encoding module 203 are updated.
[0086] Optionally, the parameters of the semantic alignment network can be updated based on the following loss function:
[0087] (1)
[0088] Wherein, L contrastive is the contrastive learning loss, N is the number of a batch of training samples input into the semantic alignment network; is the encoding feature of the speech in the i th training sample; is the encoding feature of the text in the i th training sample; is the encoding feature of the speech in the j th training sample; is the encoding feature of the text in the j th training sample; is the first similarity of the encoding feature of the speech and the encoding feature of the text in the i th training sample; is the second similarity of the encoding feature of the speech in the i th training sample and the encoding feature of the text in the j th training sample; is the second similarity of the encoding feature of the speech in the j th training sample and the encoding feature of the text in the i th training sample. The similarity of the text encoding feature and the audio encoding feature can be represented by the cosine distance.
[0089] Obviously, the contrastive learning loss is positively related to the first similarity of the text encoding features and the speech encoding features obtained by the second text encoding module and the audio encoding module encoding the text and the speech in the same training sample, and is negatively related to the second similarity of the text encoding features and the speech encoding features obtained by the second text encoding module and the audio encoding module encoding the text and the speech in different training samples.
[0090] Optionally, in order to avoid the contrastive learning loss being too "sharp", leading to unstable training or overfitting, a learnable temperature parameter can be added to the loss function . The temperature parameter is used to adjust the similarity of the text encoding features and the audio encoding features.
[0091] (2)
[0092] Based on this, the updating of the parameters of the semantic alignment network at least includes updating the parameters of the speech alignment network and the temperature parameter in the loss function. Based on this, the stability of the semantic alignment network is improved, the robustness of the speech alignment network is further improved, and the encoding quality of the audio encoding module 203 is improved.
[0093] In an optional embodiment, in order to make the audio encoding module 203 have stronger recognition ability for pronunciation, phonemes or syllables can be used instead of the original text for contrastive learning. Based on this, when the text (text in the training text, or hotword text in the inference stage) is encoded by the second text encoding module, the text can be first converted into a phoneme sequence or a syllable sequence, and then the phoneme sequence is encoded by the second text encoding module to obtain the encoding features of the text; or the syllable sequence is encoded by the second text encoding module to obtain the encoding features of the text.
[0094] The speech encoding module 201, the first text encoding module 202 and the recognition module 204 are jointly trained based on a hotword enhanced speech recognition task. That is, when the speech encoding module 201, the first text encoding module 202 and the recognition module 204 are trained, the parameters of the audio encoding module 203 are frozen.
[0095] The process of jointly training the speech encoding module 201, the first text encoding module 202 and the recognition module 204 can include:
[0096] The speech sample is input into the speech encoding module 201 to obtain the speech encoding features of the speech sample output by the speech encoding module 201.
[0097] The text of the hot word to be enhanced is input into the first text encoding module 202 to obtain text encoding features of each hot word to be enhanced. The hot word to be enhanced can be each hot word in the preset hot word set, or each hot word associated with the speech sample extracted from the preset hot word set.
[0098] The audio encoding features of each hot word to be enhanced obtained by encoding the hot word to be enhanced by the audio encoding module are obtained.
[0099] The speech encoding features of the speech sample, the text encoding features of each hot word to be enhanced, and the audio encoding features of each hot word to be enhanced are input into the recognition module 204. The recognition module 204 fuses the text encoding features and the audio encoding features of the same hot word to be enhanced to obtain target encoding features of each hot word to be enhanced. The speech recognition of the hot word enhancement is performed based on the speech encoding features of the speech sample and the target encoding features of each hot word to be enhanced to obtain the speech recognition result of the speech sample.
[0100] The parameters of the speech encoding module 201, the first text encoding module 202, and the recognition module 204 are updated so that the speech recognition result of the speech sample approaches the text label of the speech sample.
[0101] Through the above training, the speech recognition system not only learns how to use hot words on a literal level, but also learns how to use hot words on a semantic level and an acoustic level, thereby better utilizing the audio details of hot words to improve recognition accuracy and having stronger sensitivity to hot words and stronger robustness in complex contexts.
[0102] In an optional embodiment, when the target hot word set is a subset of the preset hot word set, an implementation flowchart for extracting the target hot word set from the preset hot word set is as shown in Figure 3 , which can include:
[0103] Step S301: Obtain audio encoding features of each hot word obtained by encoding the audio of each hot word in the preset hot word set by the audio encoding module.
[0104] Optionally, the audio encoding features of each hot word obtained by encoding the audio of each hot word in the preset hot word set by the audio encoding module 203 can be pre-stored in a database. When the audio encoding features of each hot word are needed, they can be directly read from the database.
[0105] Optionally, the audio encoding features of each hot word obtained by encoding the audio of each hot word in the preset hot word set by the audio encoding module 203 can be pre-stored in a database. When the audio encoding features of each hot word are needed, they can be directly read from the database.
[0106] Step S302: encode the to-be-recognized speech by the audio encoding module to obtain second speech encoding features of the to-be-recognized speech.
[0107] Step S303: in the preset hotword set, select k hotwords with the smallest distance between the audio encoding features and the second speech encoding features as the target hotword set.
[0108] Optionally, for any hotword, the distance between the audio encoding features and the second speech encoding features of the hotword can be directly calculated.
[0109] The above calculation of the audio encoding features and the second speech encoding features of the hotword has a large amount of calculation and low calculation efficiency. In order to improve the calculation efficiency and further improve the speech recognition efficiency, the audio encoding features of each hotword in the preset hotword set can be pre-split into sub-encoding features of the same length, and different sub-encoding features belong to different subspaces; the sub-encoding features belonging to the same subspace in each hotword are clustered to determine a plurality of clusters and cluster centers of each subspace.
[0110] Suppose the audio features of each hotword are vectors of length N, and the length of the sub-encoding features is M, then the audio features of each hotword are split into N / M sub-features. The first M elements of the audio encoding features of each hotword constitute the first sub-encoding feature, the M+1 to 2M elements constitute the second sub-encoding feature, and so on. The first sub-encoding features of each hotword belong to the same subspace, the second sub-encoding features of each hotword belong to the same subspace, and so on. The number of sub-encoding features in each subspace is the number of hotwords in the preset hotword set. Different sub-encoding features of the audio features of the same hotword belong to different subspaces, and each sub-encoding feature of the audio features of each hotword corresponds to N / M subspaces.
[0111] Correspondingly, the process of calculating the distance between the audio encoding features and the second speech encoding features of any hotword can include:
[0112] Split the second speech encoding features into sub-encoding features of the same length. The second speech encoding features are also split into N / M sub-encoding features of length M.
[0113] Calculate the distance between the cluster center of the cluster to which the sub-encoding features of the any hotword belong in each subspace and the sub-encoding features of the second speech encoding features; fuse the distances corresponding to the any hotword in each subspace to obtain the distance between the audio encoding features and the second speech encoding features of the any hotword.
[0114] For any hot word, determine the cluster to which the sub-code feature of the hot word belongs in each subspace, and the cluster center of the cluster; calculate the distance between the cluster center of the cluster to which the sub-code feature of the hot word belongs in each subspace and the sub-code feature of the second speech code feature, sum the distances corresponding to the hot word determined in the N / M subspaces, and obtain the distance between the audio code feature of the hot word and the second speech code feature.
[0115] Corresponding to the method embodiments, the embodiments of the present application also provide an electronic device. Referring to Figure 4 Fig. 1 shows a structural schematic diagram of an electronic device suitable for implementing the embodiments of the present application. The electronic device in the embodiments of the present application can be a terminal device (such as a car machine, a large-screen device, a mobile phone, a tablet computer, a notebook computer, a desktop computer, etc.), or a server (which can be a single server, a server cluster, or a cloud server, etc.). Figure 4 The electronic device shown is only an example and should not impose any limitation on the functions and use range of the embodiments of the present application.
[0116] As Figure 4 shown, the electronic device can include a processing device (such as a central processing unit, a graphics processing unit, etc.) 401, which can perform various appropriate actions and processes according to programs stored in a read-only memory (ROM) 402 or loaded from a storage device 408 into a random access memory (RAM) 403. In the state that the electronic device is powered on, various programs and data required for the operation of the electronic device are also stored in the RAM 403. The processing device 401, the ROM 402, and the RAM 403 are connected to each other through a bus 404. An input / output (I / O) interface 405 is also connected to the bus 404.
[0117] Generally, the following devices can be connected to the I / O interface 405: input devices 406 including, for example, a touch screen, a touchpad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; output devices 407 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; storage devices 408 including, for example, a memory card, a hard disk, etc.; and communication devices 409. The communication devices 409 can allow the electronic device to communicate with other devices wirelessly or through wires to exchange data. Although Figure 4 The electronic device is shown with various devices, but it should be understood that all the shown devices are not required to be implemented or possessed. More or fewer devices can be alternatively implemented or possessed.
[0118] The embodiments of the present application also provide a computer program product including computer readable instructions, which, when executed on an electronic device, cause the electronic device to implement any of the speech recognition methods provided by the embodiments of the present application.
[0119] The embodiment of the present application further provides a computer readable storage medium, which carries one or more computer programs, and when the one or more computer programs are executed by an electronic device, the electronic device can realize any voice recognition method provided by the embodiment of the present application.
[0120] It should be noted that the apparatus embodiments described above are merely illustrative, and the units described as separate components can or can not be physically separated, and the components displayed as units can or can not be physical units, that is, they can be located in one place, or can be distributed on multiple network units. Part or all of the modules can be selected according to actual needs to achieve the purpose of the embodiment. In addition, the connection relationship between the modules in the apparatus embodiment provided by the present application indicates that there is a communication connection between them, which can be implemented as one or more communication buses or signal lines.
[0121] Through the description of the above embodiments, those skilled in the art can clearly understand that the present application can be realized by means of software and necessary general hardware, and of course, it can also be realized by special hardware including special integrated circuits, special CPUs, special memories, special components and the like. Generally, functions completed by computer programs can be easily realized by corresponding hardware, and the specific hardware structure for realizing the same function can also be various, such as analog circuit, digital circuit or special circuit. However, for the present application, the software program implementation is a better embodiment. Based on this understanding, the technical solutions of the present application can be embodied in the form of a software product, which is stored in a readable storage medium, such as a computer floppy disk, U disk, mobile hard disk, ROM, RAM, magnetic disk or optical disk, etc., including a plurality of instructions to make a computer device (which can be a personal computer, training device or network device, etc.) execute the methods described in each embodiment of the present application.
[0122] In the above embodiments, all or part can be realized by software, hardware, firmware or any combination thereof. When realized by software, it can be realized in the form of a computer program product in whole or in part. Professional technicians can use different methods to realize the described functions for each specific solution, but such implementation should not be considered beyond the scope of the present application.
[0123] The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the processes or functions described in the embodiments of the present application are generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another computer-readable storage medium, for example, the computer instructions can be transmitted from one website, computer, training device or data center to another website, computer, training device or data center through wired (such as coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (such as infrared, wireless, microwave, etc.) mode. The computer-readable storage medium can be any available medium that can be stored by the computer or a data storage device such as a training device, a data center, etc. integrated with one or more available media sets. The available media can be a magnetic medium (for example, a floppy disk, a hard disk, a magnetic tape), an optical medium (for example, a DVD), or a semiconductor medium (for example, a solid state disk (SSD)), etc.
[0124] The various embodiments in the specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referred to each other.
[0125] The above description of the disclosed embodiments enables a person skilled in the art to implement or use the present application. Various modifications to the embodiments will be apparent to those skilled in the art, and the general principles defined herein can be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application will not be limited to the embodiments shown herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.
Claims
1. A speech recognition method, characterized in that, include: The speech to be recognized is encoded to obtain the first speech coding feature of the speech to be recognized; The text of each hot word in the target hot word set is encoded separately to obtain the text encoding features of each hot word; The audio encoding features of each hot word are obtained by encoding the audio of each hot word in the target hot word set; The text encoding features and audio encoding features of the same hot word are fused to obtain the target encoding features of each hot word. Based on the target encoding features of each hot word and the first speech encoding features, hot word-enhanced speech recognition is performed to obtain the speech recognition result of the speech to be recognized.
2. The method according to claim 1, characterized in that, The speech to be recognized is encoded by a speech encoding module, the text of each hot word is encoded by a first text encoding module, the audio of each hot word is encoded by an audio encoding module, the text encoding features and audio encoding features of the same hot word are fused by a recognition module, and speech recognition with hot word enhancement is performed based on the target encoding features of each hot word and the first speech encoding features. The audio encoding module belongs to the semantic alignment network, which further includes a second text encoding module; the semantic alignment network is pre-trained based on a speech-text pair speech and text semantic alignment task. The speech encoding module, the first text encoding module, and the recognition module are jointly trained based on a hot word-enhanced speech recognition task.
3. The method according to claim 2, characterized in that, The semantic alignment network is trained using speech-text pairs as training samples. The training process of the semantic alignment network includes: The audio encoding module of the semantic alignment network encodes the speech in the training samples to obtain the encoding features of the speech in each training sample. The second text encoding module of the semantic alignment network encodes the text in the training samples to obtain the encoding features of the text in each training text. The first similarity between the text encoding features and the speech encoding features obtained by the second text encoding module and the audio encoding module for text and speech encoding in the same training sample, and the second similarity between the text encoding features and the speech encoding features obtained by the second text encoding module and the audio encoding module for text and speech encoding in different training samples are obtained. With the goal of reducing the first similarity and increasing the second similarity, at least the parameters of the semantic alignment network are updated.
4. The method according to claim 3, characterized in that, The update of at least the parameters of the semantic alignment network includes: The parameters of the semantic alignment network and the temperature parameter in the loss function are updated. The loss calculated based on the loss function is positively correlated with the first similarity and negatively correlated with the second similarity; the temperature parameter is used to adjust the similarity between text coding features and speech coding features.
5. The method according to claim 3, characterized in that, The text in the training samples is encoded using the second text encoding module, including: Convert the text in the training samples into phoneme sequences or syllable sequences; The phoneme sequence is encoded using the second text encoding module; or, the syllable sequence is encoded using the second text encoding module.
6. The method according to claim 2, characterized in that, The target hot word set is a subset of the preset hot word set, and the target hot word set is obtained in the following way: The audio encoding features of each hot word are obtained by encoding the audio of each hot word in the preset hot word set through the audio encoding module. The audio encoding module encodes the speech to be recognized to obtain the second speech encoding feature of the speech to be recognized; In the preset hot word set, the k hot words with the smallest distance between the audio coding feature and the second speech coding feature are selected as the target hot word set.
7. The method according to claim 6, characterized in that, Also includes: The audio coding features of each hot word in the preset hot word set are pre-divided into sub-coding features of the same length, and different sub-coding features belong to different subspaces; the sub-coding features of each hot word belonging to the same subspace are clustered to determine multiple clusters and cluster centers in each subspace; Accordingly, the process of calculating the distance between the audio coding features of any hot word and the second speech coding features includes: The second speech coding feature is split into sub-coding features of the same length; Calculate the distance between the cluster center of the cluster to which the sub-coding feature of any hot word in each subspace belongs and the sub-coding feature of the second speech coding feature; fuse the distances corresponding to any hot word in each subspace to obtain the distance between the audio coding feature of any hot word and the second speech coding feature.
8. An electronic device, characterized in that, The electronic device includes at least one processor and a memory connected to the processor; wherein: The memory is used to store computer programs; The processor is used to execute the computer program to enable the electronic device to implement the speech recognition method as described in any one of claims 1 to 7.
9. A computer program product, characterized in that, It includes computer-readable instructions that, when executed on an electronic device, cause the electronic device to implement the speech recognition method as described in any one of claims 1 to 7.
10. A computer storage medium, characterized in that, The storage medium carries one or more computer programs that, when executed by an electronic device, enable the electronic device to implement the speech recognition method as described in any one of claims 1 to 7.