Speech recognition keyword enhancement method and device based on context associated words

Abstract

A speech recognition keyword enhancement method based on context associated words comprises the following steps: acquiring a speech data set and a text label corresponding to the speech data set, extracting keywords and keyword contexts to construct a dynamic word list, and performing feature extraction and data set division on speech data; a speech recognition model is constructed, the speech recognition model comprises a hybrid expert encoder and a double-path attention fusion mechanism, the hybrid expert encoder is used for carrying out a speech recognition task, and feature coding is carried out on keywords and keyword contexts through a keyword expert network and a context expert network which are connected in parallel; the double-path attention fusion mechanism carries out attention interaction on the voice features and the keyword coding features and the context coding features; performing end-to-end training on the model by jointly optimizing the loss function of the speech recognition task and the loss function of the keyword enhancement task; and performing keyword-enhanced real-time recognition on the input voice by using the trained model. According to the method, the recognition accuracy of the keywords in speech recognition can be improved.

Description

Technical Field

[0001] The present invention relates to the technical field of speech signal processing, and in particular to a method and device for enhancing speech recognition keywords based on context-related words. Background Art

[0002] Speech recognition technology is a key enabler of artificial intelligence and is currently widely used in smart cars, mobile navigation, voice translation, and other fields. While end-to-end large-scale continuous speech recognition technology has made significant progress in Mandarin speech recognition, significant deficiencies remain when addressing specific domains or professional scenarios, particularly in keyword recognition. For example, in professional scenarios such as healthcare and customer service, speech recognition models often struggle to accurately identify specialized terminology or specific keywords, resulting in recognition results that do not meet practical requirements.

[0003] Currently, speech recognition methods for keyword enhancement typically construct a WFST graph of keywords within the decoded language model, or add an independent keyword encoding and integrate keyword information into the speech recognition output probability matrix via a cross-attention mechanism. While these methods can dynamically integrate keyword information to a certain extent, they still have limitations. WFST-based methods require the pre-construction of a static graph of keywords. This predefined, fixed structure is difficult to adapt to complex and changing contexts, especially when encountering dynamically updated professional terminology or emerging vocabulary. While methods using a single attention mechanism can dynamically capture keyword information, their modeling capabilities are limited. They are not effective at distinguishing similar-sounding words or low-frequency professional terms, and are prone to confusion errors. Furthermore, these methods often require modifying or retraining the language model, which not only increases system complexity but also deployment costs. In multi-domain and multi-scenario keyword spotting tasks, existing methods struggle to balance general recognition capabilities with the accuracy of professional terminology recognition, resulting in unstable performance when dealing with content from different domains.

[0004] Therefore, how to make full use of contextual information, improve the accuracy of keyword recognition, and construct a keyword enhancement method that can dynamically integrate contextual information in the reasoning stage has become a technical problem that needs to be solved urgently. Summary of the Invention

[0005] In order to solve the problems existing in the prior art, the embodiments of the present application provide a method, apparatus, computing device, computer storage medium and product containing a computer program for speech recognition keyword enhancement based on context-related words, which can improve the recognition accuracy of keywords in speech recognition.

[0006] In the first aspect, an embodiment of the present application provides a method for keyword enhancement in speech recognition based on context-related words, including: obtaining a speech data set and its corresponding text annotations, extracting keywords and keyword contexts to construct a dynamic vocabulary, and performing feature extraction and data set division on the speech data; constructing a speech recognition model, the speech recognition model including a hybrid expert encoder and a two-way attention fusion mechanism, wherein the hybrid expert encoder is used to perform speech recognition tasks, and feature encodes keywords and keyword contexts respectively through a parallel keyword expert network and a context expert network; the two-way attention fusion mechanism is used to perform attention interaction between speech features and keyword encoding features and context encoding features respectively; the model is trained end-to-end by jointly optimizing the loss function of the speech recognition task and the loss function of the keyword enhancement task; and the trained model is used to perform real-time recognition of keyword enhancement on the input speech.

[0007] In some possible implementations, the keyword context is dynamically extended to the speech recognition model through a joint modeling method of context-related keyword enhancement and speech recognition; the loss function of the speech recognition task and the keyword enhancement task are jointly optimized based on end-to-end training to achieve collaborative modeling of keywords and general speech recognition.

[0008] In some possible implementations, the hybrid expert encoder includes: a keyword expert network and a context expert network, wherein the keyword expert network is used to extract the pronunciation and semantic features of the keywords, and the context expert network is used to extract the contextual grammatical and semantic features associated with the keywords, and the two types of features are fused through dynamic weights.

[0009] In some possible implementations, the output weights of the keyword expert network and the context expert network are dynamically adjusted through a gating network enhanced based on context information. The gating network generates adaptive weight values based on the current input speech features and the semantic relevance of the context-related words. The adaptive weight values are used to control the fusion ratio of the two types of features.

[0010] In some possible implementations, the keyword expert network adopts a Transformer architecture, and its feedforward network layer parameters are pre-trained by the keyword recognition task; the context expert network shares the same Transformer architecture, but the feedforward network layer parameters are pre-trained by the context prediction task; the gating network consists of two fully connected layers, the input is the encoder intermediate layer representation, and the output is the expert weight in the range [0,1].

[0011] In some possible implementations, the dual-path attention fusion mechanism includes: a keyword attention path and a context attention path, the keyword attention path is used to calculate the interaction weights of speech features and keyword coding features, and the context attention path is used to calculate the interaction weights of speech features and context coding features; the keyword-enhanced speech coding representation is generated by weighted summation.

[0012] In some possible implementations, the hybrid expert encoder adopts a Transformer architecture, the feedforward neural network is partially replaced by independently initialized keyword expert sub-network and context expert sub-network, and stable gradient propagation is achieved through residual connections and layer normalization.

[0013] In some possible implementations, the joint optimization of the loss function of the speech recognition task and the loss function of the keyword enhancement task is performed using the following formula:

[0014] L ASR =λL CTC +(1-λ)L CE +γL bias ,

[0015] Where, L ASR Represents the total loss of the speech recognition task, λ represents the CTC loss weight coefficient, and 0≤λ≤1, L CTC Characterize the speech recognition CTC loss function, L CE Represents the Attention cross entropy loss function for speech recognition, γ represents the keyword bias loss weight coefficient, L bias Characterizing keyword enhancement loss function.

[0016] In some possible implementations, extracting keywords includes: using NLP word segmentation tools to analyze the grammatical structure of the prompt sentence; extracting N adjacent words before and after the target keyword as basic context-related words (N≥1); retrieving typical collocation words of the keyword from the domain corpus and expanding the context vocabulary.

[0017] In the second aspect, an embodiment of the present application provides a speech recognition keyword enhancement device based on context-related words, including: an acquisition module, used to acquire a speech data set and its corresponding text annotations, extract keywords and keyword contexts to construct a dynamic vocabulary, and perform feature extraction and data set division on the speech data; a processing module, used to construct a speech recognition model, the speech recognition model includes a hybrid expert encoder and a two-way attention fusion mechanism, wherein the hybrid expert encoder is used to perform speech recognition tasks, and feature encode keywords and keyword contexts respectively through a parallel keyword expert network and a context expert network; the two-way attention fusion mechanism is used to perform attention interaction between speech features and keyword encoding features and context encoding features respectively; the processing module is also used to perform end-to-end training on the model by jointly optimizing the loss function of the speech recognition task and the loss function of the keyword enhancement task; the processing module is also used to use the trained model to perform real-time recognition of keyword enhancement on the input speech.

[0018] In a third aspect, an embodiment of the present application provides a computer-readable storage medium comprising computer-readable instructions. When a computer reads and executes the computer-readable instructions, the computer executes the method as described in any one of the first aspects.

[0019] In a fourth aspect, an embodiment of the present application provides a computing device comprising a processor and a memory, wherein the memory stores computer program instructions, and when the computer program instructions are executed by the processor, the method as described in any one of the first aspects is executed.

[0020] In a fifth aspect, an embodiment of the present application provides a product comprising a computer program, which, when the computer program product runs on a processor, enables the processor to execute the method as described in any one of the first aspects. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0022] Figure 1 This is a flow chart of a method for enhancing speech recognition keywords based on context-related words provided in an embodiment of the present application;

[0023] Figure 2 Schematic diagram of a speech recognition keyword enhancement modeling method based on dynamic expansion of context-related words provided by an embodiment of the present application;

[0024] Figure 3 is a schematic diagram of a hybrid expert encoder module provided in an embodiment of the present application;

[0025] Figure 4 Schematic diagram of a dual-path attention keyword bias network provided by an embodiment of the present application;

[0026] Figure 5 1 is a schematic diagram of the structure of a device for a method for enhancing keywords in speech recognition based on context-related words provided in an embodiment of the present application;

[0027] Figure 6 It is a structural diagram of a computing device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0028] To make the objectives, technical solutions, and advantages of the embodiments of the present invention more clear, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts shall fall within the scope of protection of the present invention.

[0029] The term "and / or" as used herein describes an association between related objects, indicating that three possible relationships exist. For example, "A and / or B" can represent: A exists alone, A and B exist simultaneously, or B exists alone. The symbol " / " as used herein indicates that the related objects are in an "or" relationship, for example, A / B means either A or B.

[0030] The terms "first" and "second" in this specification and claims are used to distinguish different objects rather than to describe a specific order of objects. For example, "first response message" and "second response message" are used to distinguish different response messages rather than to describe a specific order of response messages.

[0031] In the embodiments of this application, words such as "exemplary" or "for example" are used to indicate examples, illustrations, or descriptions. Any embodiment or design described as "exemplary" or "for example" in the embodiments of this application should not be interpreted as being preferred or advantageous over other embodiments or designs. Rather, the use of words such as "exemplary" or "for example" is intended to present the relevant concepts in a concrete manner.

[0032] In the description of the embodiments of the present application, unless otherwise specified, "multiple" means two or more, for example, multiple processing units means two or more processing units, etc.; multiple elements means two or more elements, etc.

[0033] To facilitate understanding of the embodiments of the present application, further explanation will be given below with reference to specific embodiments in conjunction with the accompanying drawings. The embodiments do not constitute a limitation on the embodiments of the present invention.

[0034] An embodiment of the present application provides a method for enhancing keywords in speech recognition based on context-related words. The method extracts the context-related words of keywords from prompt sentences, encodes them in combination with a hybrid expert keyword encoder, and dynamically adjusts the recognition weight of keywords, thereby improving the recognition accuracy of keywords in speech recognition.

[0035] For example, Figure 1 A flow chart of a method for enhancing speech recognition keywords based on context-related words provided in an embodiment of the present application is shown. Figure 1 As shown, the method may include the following steps:

[0036] S11: Obtain a speech data set and its corresponding text annotations, extract keywords and keyword contexts to build a dynamic vocabulary, and perform feature extraction and data set division on the speech data.

[0037] In this example, we first need to build a high-quality dataset that can support both speech recognition and keyword enhancement. The construction of the dataset determines whether the model can effectively learn the keyword features and their contextual relevance in professional scenarios.

[0038] Specifically, we first need to collect diverse voice data that covers common scenarios in the target domain. For example, in the medical field, this could include speech recordings such as doctor consultations, medical records, and conversations about medical terminology. Each piece of voice data needs to be annotated with text. The annotations should include the complete speech transcription and additionally include key words. Keyword annotation standards need to be pre-defined based on the application scenario. For example, in a medical setting, terms such as "CT scan," "antibiotics," and "hypertension" should be marked as keywords.

[0039] To construct a context-sensitive word list, complete sentences containing each keyword can be collected as prompts for each keyword. These prompts can be derived from the corresponding text of the speech or extracted from a domain-specific corpus. For example, for the keyword "insulin," the prompt might be "Patients require daily insulin injections to control blood sugar." After collecting the prompts, they are processed using an NLP word segmentation tool to extract the preceding and following words of each keyword as context-sensitive words. For example, in the sentence "Painkillers are needed after surgery to relieve symptoms," the context-sensitive words for the keyword "painkillers" are "take" and "relieve." Next, feature extraction is performed on the speech signal. To enhance the representational power of the features, first-order and second-order difference features can be extracted simultaneously to form a dynamic feature vector. Text data requires standardization, including capitalization, removal of special symbols, and normalization of numerical representations. All text must be converted to a unified character set, such as UTF-8 encoding, to ensure compatibility.

[0040] Divide the processed data into training, validation, and test sets. Ensure that each set covers all keywords and maintains a similar contextual distribution. Use stratified sampling to divide the data based on keyword frequency to avoid missing keywords from any set. For keywords with insufficient data, use oversampling to balance the data distribution.

[0041] S12: Build a speech recognition model with a hybrid expert encoder and a two-way attention fusion mechanism.

[0042] In this embodiment, after obtaining the data of the training set, the validation set, and the test set, the speech recognition model can be constructed. Figure 2 This is a schematic diagram of a speech recognition keyword enhancement modeling method based on dynamic expansion of context-related words provided by an embodiment of the present application. Please refer to Figure 2 The model can be a complex neural network model that organically combines an end-to-end speech recognition model with a keyword enhancement module. This architecture must simultaneously process speech signals and text information and achieve accurate keyword recognition through a multi-layered interaction mechanism. The entire architecture can be divided into three core components: a speech recognition module, a hybrid expert encoder module, and a dual-path attention fusion module. These three components work together to improve speech recognition performance in professional scenarios.

[0043] Specifically, the speech recognition module adopts a hybrid CTC / Attention architecture based on an attention mechanism, where both the encoder and decoder are constructed from stacked Transformer modules. The encoder consists of multiple stacked Transformer layers, each of which incorporates a self-attention mechanism and a feedforward neural network. The encoder input is a preprocessed speech feature sequence, and after multiple layers of transformation, it outputs a high-level speech representation. The encoder output is optimized using the CTC loss function, while the decoder output is trained using the cross-entropy loss function. The keyword and context hybrid expert encoder consists of two Transformer modules, with the feedforward network replaced by two independent hybrid expert layers. The dual-path keyword bias layer is implemented using a multi-head cross-attention mechanism, where the query vector is the output of the speech encoder, and the key-value pairs come from the outputs of the keyword encoder and the context encoder, respectively. The bias output is added to the speech encoder output to generate a keyword-enhanced speech encoding representation. Simultaneously, the bias output is passed through a linearly adaptive keyword decoder to calculate the CTC loss for the keyword label. The final loss function is the weighted sum of the CTC loss and cross entropy loss of speech recognition and the keyword CTC loss. The formula can be expressed as: L ASR =λL CTC +(1-λ)L CE +γL bias , where L ASR Represents the total loss of the speech recognition task, λ represents the CTC loss weight coefficient, and 0≤λ≤1, L CTC Characterize the speech recognition CTC loss function, L CE Represents the Attention cross entropy loss function for speech recognition, γ represents the keyword bias loss weight coefficient, L bias Characterizing keyword enhancement loss function.

[0044] Figure 3 A schematic diagram of a hybrid expert encoder module provided by an embodiment of the present application is shown. Figure 3The hybrid expert encoder module is specifically designed to process keywords and their contextual information. This module comprises two independent expert networks: a keyword expert and a context expert. These networks share the same underlying architecture but focus on different information extraction tasks. The keyword expert is responsible for learning feature representations for the keywords themselves, capturing their pronunciation characteristics, semantic connotations, and specific usage within the domain. The context expert focuses on analyzing contextual information related to the keywords, including grammatical structure, semantic associations, and domain-specific expression conventions. Both expert networks utilize the Transformer architecture, but the feedforward neural network is specially designed, replacing it with specialized sub-networks with different initialization parameters. This structural difference enables the two experts to develop distinct feature extraction capabilities. Connecting the two expert networks is a gated network, a learnable neural network that dynamically assesses the relative importance of keywords and context in the current input. The gated network receives the intermediate representation of the previous layer's output and, through nonlinear transformation, generates a set of weights that determine the contribution of the keyword expert and context expert to the final encoded output. For example, when processing keywords with unique pronunciation but weak context dependence, the gating network will assign higher weights to keyword experts; while for those words whose meaning can only be clarified through context, the weight of context experts will be increased.

[0045] The dual-path attention fusion module is used to integrate keyword enhancement information into the speech recognition process, and achieves accurate keyword enhancement by interacting keyword encoding and context encoding with speech encoding respectively. Figure 4 This is a schematic diagram of a dual-path attention keyword bias network provided by an embodiment of the present application. Please refer to Figure 4 Dual-path attention fusion involves two parallel attention calculation paths: a keyword attention path and a context attention path. The keyword encoding and context encoding are respectively multiplied with the speech encoding by the attention query matrix and the key matrix to obtain the keyword attention score and the context attention score. The keyword attention score is directly normalized using the Softmax function to obtain the keyword weight; the context attention weight is obtained by integrating the context attention score with the keyword attention score to reflect the relevance of the context and the keyword.

[0046] The specific formula is expressed as

[0047] α keyword =Softmax(score keyword )

[0048] α context =Softmax(score context +λ k score keyword )

[0049] Where, α keyword Cross-attention weight representing keywords, score keyword Characterize the keyword score, α context Cross attention weight representing context, score context Represents the contextual attention score, λ k The weight coefficient that represents the keyword score in the context.

[0050] Finally, according to the calculated weights, the weighted summation is performed with the value matrices of the keyword encoding and context encoding to generate the enhanced bias output.

[0051] The model architecture utilizes multiple techniques to ensure effective training. All Transformer layers use residual connections and layer normalization to mitigate the vanishing gradient problem. The attention mechanism employs a multi-head design, allowing the model to learn relevant information in different representation subspaces. Dropout layers are added at key locations in the network to prevent overfitting. A dynamic learning rate adjustment strategy is employed, combined with appropriate weight decay to achieve stable parameter updates. To accelerate convergence, the encoder's underlying parameters can be initialized using a pretrained speech model. While the overall structure is complex, each component can be tailored to specific application scenarios. For example, the expert network structure can be simplified for scenarios with a small number of keywords; the attention mechanism can be optimized for applications with high real-time requirements; and for different specialized domains, only keyword and context settings need to be adjusted without changing the overall architecture. This flexibility enables the solution to adapt to a variety of practical application needs. Notably, the entire model architecture was designed with efficiency in the inference phase in mind. While training requires the calculation of multiple loss functions and a complex attention mechanism, in practice, most computations can be parallelized, and the keyword enhancement module is only activated when a possible keyword is detected. This design allows the model to maintain high performance while also meeting the responsiveness requirements of actual deployments. Furthermore, the architecture supports incremental updates, meaning that when new terminology needs to be added, only specific modules need to be fine-tuned.

[0052] S13: The model is trained end-to-end by jointly optimizing the loss function of the speech recognition task and the loss function of the keyword enhancement task.

[0053] In this example, the model is trained using data from the training set. A specific training strategy is employed during training, enabling the model to simultaneously learn general speech recognition and enhance specialized domain keywords through multi-task learning and dynamic sample generation. The training process coordinates the learning progress of different components and balances the optimization objectives of various loss functions, ensuring that the model maintains overall speech recognition performance while also improving its ability to capture specialized terminology and key information.

[0054] Specifically, model parameters must be initialized before training begins. The encoder portion of the speech recognition backbone network can be initialized using pre-trained speech model parameters. The keyword experts and context experts in the hybrid expert encoder need to be initialized separately to ensure that they develop in different specialized directions at the beginning of training. The parameters of the gated network are typically initialized with small random values to maintain a neutral position at the beginning of training, gradually developing judgment capabilities as data accumulates. The weight matrices of all attention mechanisms are initialized using Xavier. This initialization method, which takes input and output dimensions into consideration, can effectively prevent gradient explosion or vanishing problems.

[0055] First, a batch of speech samples are randomly sampled from the training set. The keywords within these samples are statistically analyzed, and a number of keywords with moderate frequency are selected as key learning targets for the current batch. For each selected keyword, the system extracts its common contextual vocabulary from the associated vocabulary to construct dynamic keyword-context pairs. This allows the model to be exposed to a variety of different keywords and their contexts during training, preventing bias caused by a few high-frequency keywords dominating the training process.

[0056] The forward propagation process involves complex multi-path computation. The speech signal is first converted into a high-level feature representation through an encoder network. Simultaneously, the keywords of the current batch and their contextual information are fed into the two branches of the hybrid expert encoder. The keyword expert network converts these terms into vector representations with rich, specialized semantics, while the context expert captures contextual features. The gating network dynamically calculates weights based on the activations of the intermediate layers, determining the fusion of the outputs of the two expert networks. The fusion result then undergoes a two-way attention interaction with the speech codec. The generated keyword enhancement information not only adjusts the speech feature representation but also generates independent keyword predictions.

[0057] The total loss function during training consists of three components: the traditional CTC loss ensures good alignment of speech features with the output sequence, the attention decoder's cross-entropy loss optimizes text generation quality, and the keyword bias loss strengthens the recognition of target terms. The weights of these three loss components are dynamically adjusted based on the training progress: Initially, the CTC loss is weighted relatively high to establish a basic understanding of speech features. As training progresses, the weight of the cross-entropy loss is gradually increased to optimize the language model. The keyword bias loss's weight is periodically adjusted to effectively guide keyword learning without excessively interfering with overall speech recognition performance.

[0058] The backpropagation process requires special handling of the gradient flow path. Because the model structure contains multiple parallel computation paths, the gradient magnitudes of each component must be controlled to prevent some modules from converging prematurely while others fail to learn adequately. A hierarchical learning rate strategy is employed for the hybrid expert encoder, with lower learning rates for low-level parameters to maintain stable feature extraction capabilities and higher learning rates for high-level parameters to enable rapid adaptation. The parameter update amplitude of the gating network requires additional constraints to prevent drastic weight fluctuations that could destabilize the expert network training. The gradients of the dual-path attention module are normalized to ensure balanced update strength across the two paths.

[0059] At the end of each model training cycle, the training effect is verified using the data in the validation set, and multiple indicators including keyword recognition accuracy, common vocabulary error rate, and context relevance are calculated. These indicators are compared with the loss value of the training set to help developers determine whether the model is overfitting or underfitting. The validation set is also used to adjust the weight coefficients of the loss function, such as λ and γ, and to find the hyperparameter combination that performs best on the validation set through methods such as grid search or Bayesian optimization. Structural parameters such as the sparsity constraint strength of the gated network and the number of attention heads in the hybrid expert model are also ultimately determined based on the performance of the validation set. After the verification results meet expectations, the overall effect of the model is tested on the test set.

[0060] S14: Use the trained model to perform real-time recognition of keyword-enhanced input speech.

[0061] In this embodiment, the trained model is deployed in a real-world application environment to process real-world speech input and output keyword-enhanced recognition results. Unlike the training phase, the inference process must balance computational efficiency and recognition accuracy. During inference, when the user begins speaking, the system captures audio through the microphone array and segments it into segments of appropriate length. A sliding window mechanism is typically used, with each window approximately 2-3 seconds long and a 50% overlap between adjacent windows. Speech signal features are extracted from the audio. Simultaneously, a keyword vocabulary and a related context rule library appropriate for the current scenario are loaded. These resources are typically organized by domain. For example, a medical terminology library might be loaded in a healthcare scenario, while a customer service vocabulary library might be used in a customer service scenario. Speech signal features, keywords, and the keyword context are input into the trained model to determine the final model inference results.

[0062] The above is a speech recognition keyword enhancement method based on context-related words provided in an embodiment of the present application. By converting the speech signal into a corresponding text sequence and introducing a dynamic expansion mechanism of context-related words, the recognition accuracy of keywords in speech recognition is improved. The present invention is particularly aimed at scenarios where high-precision recognition of professional terms or specific keywords is required. Under a unified model framework, the context-related words of keywords are extracted from the prompt sentence and encoded in combination with a hybrid expert keyword encoder to solve the problem of insufficient keyword recognition in traditional speech recognition models. The present invention encodes keywords and contexts separately through a hybrid expert model, and uses a gated network to dynamically evaluate the relevance of context and keywords, so that the model can adaptively fuse keywords and their context information. At the same time, the present invention is based on a two-way attention fusion mechanism to calculate the attention scores of keywords and contexts respectively, and dynamically integrate the weights of the two, so that the model can more accurately adjust the recognition weights of keywords. Under a unified model framework, the present invention realizes the coordinated optimization of keyword recognition and speech recognition tasks, while significantly improving the keyword recognition performance, effectively reducing the complexity and workload of model optimization and system deployment, and significantly enhancing the generalization ability of the model.

[0063] It is understandable that the size of the sequence number of each step in the above-mentioned embodiments does not mean the order of execution, and the execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application. In addition, in some possible implementations, the steps in the above-mentioned embodiments can be selectively executed according to actual conditions, and can be partially executed or fully executed, which is not limited here. All or part of any features of any embodiment of the present application can be freely and arbitrarily combined without contradiction. The combined technical solution is also within the scope of the present application.

[0064] Based on the method in the above embodiment, the embodiment of the present application also provides a speech recognition keyword enhancement device based on context-related words.

[0065] For example, Figure 5 The invention shows a speech recognition keyword enhancement device based on context-related words, which is deployed on a computing device. The speech recognition keyword enhancement device based on context-related words 500 includes: an acquisition module 501 and a processing module 502.

[0066] The acquisition module 501 is used to acquire a speech data set and its corresponding text annotations, extract keywords and keyword contexts to construct a dynamic vocabulary, and perform feature extraction and data set division on the speech data.

[0067] The processing module 502 is used to construct a speech recognition model, which includes a hybrid expert encoder and a two-way attention fusion mechanism. The hybrid expert encoder is used to perform speech recognition tasks and feature encode keywords and keyword contexts through a parallel keyword expert network and a context expert network respectively; the two-way attention fusion mechanism is used to interact the speech features with the keyword encoding features and the context encoding features respectively.

[0068] The processing module 502 is further configured to perform end-to-end training on the model by jointly optimizing the loss function of the speech recognition task and the loss function of the keyword enhancement task.

[0069] The processing module 502 is also used to use the trained model to perform real-time recognition of keyword enhancement on the input speech.

[0070] It should be understood that the above-mentioned device is used to execute the method in the above-mentioned embodiment. The implementation principle and technical effect of the corresponding program module in the device are similar to those described in the above-mentioned method. The working process of the device can refer to the corresponding process in the above-mentioned method and will not be repeated here.

[0071] The present application also provides a computing device 600. Figure 6 As shown, computing device 600 includes a bus 602, a processor 604, a memory 606, and a communication interface 608. Processor 604, memory 606, and communication interface 608 communicate with each other via bus 602. Computing device 600 can be a server or a terminal device. It should be understood that this application does not limit the number of processors and memories in computing device 600.

[0072] The bus 602 may be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus. The bus may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 6 The bus 604 may include a path for transmitting information between various components of the computing device 600 (eg, memory 606, processor 604, communication interface 608).

[0073] The processor 604 may include any one or more processors such as a central processing unit (CPU), a graphics processing unit (GPU), a microprocessor (MP), or a digital signal processor (DSP).

[0074] The memory 606 may include volatile memory, such as random access memory (RAM). The processor 604 may also include non-volatile memory, such as read-only memory (ROM), flash memory, hard disk drive (HDD), or solid state drive (SSD).

[0075] The memory 606 stores executable program code, and the processor 604 executes the executable program code to implement the functions of the acquisition module 501 and the processing module 502, respectively, thereby implementing all or part of the steps of the method in the above embodiment. In other words, the memory 606 stores instructions for executing all or part of the steps of the method in the above embodiment.

[0076] Alternatively, the memory 606 stores executable code, and the processor 604 executes the executable code to respectively implement the functions of the aforementioned context-related word-based speech recognition keyword enhancement device 500, thereby implementing all or part of the steps in the above-mentioned embodiment method. In other words, the memory 606 stores instructions for executing all or part of the steps in the above-mentioned embodiment method.

[0077] The communication interface 608 uses a transceiver module such as, but not limited to, a network interface card or a transceiver to implement communication between the computing device 600 and other devices or a communication network.

[0078] Based on the methods in the above embodiments, an embodiment of the present application provides a computer-readable storage medium, which stores a computer program. When the computer program runs on a processor, the processor executes the methods in the above embodiments.

[0079] Based on the methods in the above embodiments, an embodiment of the present application provides a computer program product. When the computer program product runs on a processor, the processor executes the methods in the above embodiments.

[0080] It is understood that the processor in the embodiments of the present application may be a central processing unit (CPU), or may be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field programmable gate arrays (FPGA), or other programmable logic devices, transistor logic devices, hardware components, or any combination thereof. The general-purpose processor may be a microprocessor or any conventional processor.

[0081] The method steps in the embodiments of the present application can be implemented by hardware or by a processor executing software instructions. The software instructions can be composed of corresponding software modules, which can be stored in random access memory (RAM), flash memory, read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), registers, hard disks, mobile hard disks, CD-ROMs or any other form of storage medium known in the art. An exemplary storage medium is coupled to the processor so that the processor can read information from the storage medium and write information to the storage medium. Of course, the storage medium can also be a component of the processor. The processor and the storage medium can be located in an ASIC.

[0082] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted via the computer-readable storage medium. The computer instructions can be transmitted from one website, computer, server or data center to another website, computer, server or data center via a wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) method. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more available media integrated. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid state drive (SSD)).

[0083] It will be understood that the various numerical numbers involved in the embodiments of the present application are merely distinctions for the convenience of description and are not intended to limit the scope of the embodiments of the present application.

Claims

1. A method for enhancing speech recognition keywords based on context-related words, characterized in that: The method comprises: Obtain speech data sets and their corresponding text annotations, extract keywords and keyword contexts to build a dynamic vocabulary, perform feature extraction and data set partitioning on speech data; Constructing a speech recognition model, the speech recognition model includes a hybrid expert encoder and a two-way attention fusion mechanism, wherein the hybrid expert encoder is used to perform speech recognition tasks and respectively encode features of keywords and keyword contexts through a parallel keyword expert network and a context expert network; the two-way attention fusion mechanism is used to perform attention interaction between speech features and keyword encoding features and context encoding features respectively; The model is trained end-to-end by jointly optimizing the loss function of the speech recognition task and the loss function of the keyword enhancement task; Use the trained model to perform real-time recognition of keyword-enhanced input speech.

2. The method according to claim 1, characterized in that Through the joint modeling method of context-related keyword enhancement and speech recognition, the keyword context is dynamically extended to the speech recognition model; The loss functions of the speech recognition task and the keyword enhancement task are jointly optimized based on end-to-end training to achieve collaborative modeling of keyword and general speech recognition.

3. The method according to claim 1, characterized in that The hybrid expert encoder includes: a keyword expert network and a context expert network, wherein the keyword expert network is used to extract the pronunciation and semantic features of keywords, and the context expert network is used to extract the context grammatical and semantic features associated with the keywords, and the two types of features are fused through dynamic weights.

4. The method according to claim 3, characterized in that The output weights of the keyword expert network and the context expert network are dynamically adjusted through a gating network enhanced based on context information. The gating network generates adaptive weight values based on the semantic relevance of the currently input speech features and the context-related words. The adaptive weight values are used to control the fusion ratio of the two types of features.

5. The method according to claim 1, wherein The keyword expert network adopts the Transformer architecture, and its feedforward network layer parameters are pre-trained by the keyword recognition task; The contextual expert network shares the same Transformer architecture, but the feedforward network layer parameters are pre-trained on the context prediction task; The gating network consists of two fully connected layers, the input is the encoder intermediate layer representation, and the output is the expert weight in the interval [0,1].

6. The method according to claim 1, characterized in that The dual-path attention fusion mechanism includes: a keyword attention path and a context attention path, wherein the keyword attention path is used to calculate the interaction weights of speech features and keyword encoding features, and the context attention path is used to calculate the interaction weights of speech features and context encoding features; The keyword-enhanced speech coding representation is generated by weighted summation.

7. The method according to claim 1, characterized in that The hybrid expert encoder adopts the Transformer architecture, and the feedforward neural network is partially replaced by independently initialized keyword expert sub-network and context expert sub-network, and stable gradient propagation is achieved through residual connection and layer normalization.

8. The method according to claim 1, characterized in that The tool performs the joint optimization of the loss function of the speech recognition task and the loss function of the keyword enhancement task using the following formula: L ASR =λL CTC +(1-λ)L CE +γL bias , Where, L ASR Represents the total loss of the speech recognition task, λ represents the CTC loss weight coefficient, and 0≤λ≤1, L CTC Characterize the speech recognition CTC loss function, L CE Represents the Attention cross entropy loss function for speech recognition, γ represents the keyword bias loss weight coefficient, L bias Characterizing keyword enhancement loss function.

9. The method according to claim 1, characterized in that Extracted keywords include: Use NLP word segmentation tools to analyze the grammatical structure of the prompt sentence; Extract N adjacent words before and after the target keyword as basic context words (N≥1); Retrieve typical collocation words of the keyword from the domain corpus and expand the context word list.

10. A speech recognition keyword enhancement device based on context-related words, characterized in that: The device comprises: The acquisition module is used to obtain the speech data set and its corresponding text annotations, extract keywords and keyword contexts to build a dynamic vocabulary, and perform feature extraction and data set division on the speech data; A processing module is configured to construct a speech recognition model, the speech recognition model comprising a hybrid expert encoder and a two-way attention fusion mechanism, wherein the hybrid expert encoder is configured to perform speech recognition tasks and feature encode keywords and keyword contexts using a parallel keyword expert network and a context expert network; and the two-way attention fusion mechanism is configured to perform attention interaction between speech features, keyword encoding features, and context encoding features. The processing module is further configured to perform end-to-end training on the model by jointly optimizing a loss function of the speech recognition task and a loss function of the keyword enhancement task; The processing module is also used to use the trained model to perform real-time recognition of keyword enhancement on the input speech.

Images