A speech recognition method, apparatus, device, and storage medium

Abstract

The application discloses a speech recognition method and device, equipment and storage medium, and relates to the technical field of artificial intelligence, comprising: constructing a target data set based on a plurality of pairs of homophones and near homophones, and fine-tuning a first automatic speech recognition model based on the target data set to obtain a third automatic speech recognition model; determining an acoustic feature sequence of a target audio, and generating a candidate word sequence corresponding to the acoustic feature sequence, generating an entropy value of each candidate word based on the candidate word sequence and the acoustic feature sequence; determining the candidate word corresponding to the entropy value higher than the preset threshold as a to-be-corrected word, if the error type corresponding to the to-be-corrected word is a terminology error, correcting the to-be-corrected word based on a target domain knowledge base; if the error type corresponding to the to-be-corrected word is an ambiguity error, correcting the to-be-corrected word by using the third automatic speech recognition model, and determining a speech recognition result based on the correction result. The application can improve the accuracy and efficiency of speech recognition.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and in particular to a speech recognition method, apparatus, device, and storage medium. Background Technology

[0002] Automatic Speech Recognition (ASR) technology has made progress over the past decade, but homonym errors—that is, words that are acoustically identical or extremely similar but whose written forms and meanings are completely different are confused, such as recognizing "deployment" as "subordinates"—remain a core bottleneck limiting its reliability in professional fields such as medicine and law.

[0003] Existing technical approaches have significant drawbacks: Decoding constraint methods based on traditional language models are essentially statistical models that rely on the co-occurrence frequency of phrases, which cannot understand long-distance semantic dependencies and deep-level contextual logic, and are difficult to handle homophones that require judgment based on world knowledge or professional background; Post-processing methods based on large language models treat the output of the ASR system as an independent text sequence and hand it over to LLM (Large Language Model) for polishing and correction, completely losing acoustic information. LLM may over-infer based solely on textual context, producing errors that contradict the original audio content, resulting in low accuracy; Although end-to-end unified models deeply integrate acoustic encoders and LLM decoders, improving context awareness, the model's built-in knowledge is static and powerless against domain-specific and dynamically changing knowledge. In addition, the model is huge, with high computational costs for each inference, and cannot allocate resources according to the actual difficulty of the input.

[0004] In conclusion, improving the accuracy and efficiency of speech recognition is a pressing technical problem that needs to be solved. Summary of the Invention

[0005] In view of this, the purpose of this invention is to provide a speech recognition method, apparatus, device, and storage medium that can improve the accuracy and efficiency of speech recognition. The specific solution is as follows:

[0006] In a first aspect, this application provides a speech recognition method, including:

[0007] A first automatic speech recognition model is determined based on a preset encoder and a preset decoder, and a second automatic speech recognition model is obtained by lightweighting the first automatic speech recognition model; wherein, the preset decoder is a decoder determined based on a preset large language model;

[0008] A target dataset is constructed based on several pairs of homophones and near-homophones, and a third automatic speech recognition model is obtained by fine-tuning the preset decoder of the first automatic speech recognition model based on the target dataset using low-rank adaptation technology.

[0009] The first automatic speech recognition model is used to determine the acoustic feature sequence corresponding to the target audio, and the second automatic speech recognition model is used to generate the candidate word sequence corresponding to the acoustic feature sequence based on the autoregressive method. The first automatic speech recognition model is used to generate the entropy value corresponding to each candidate word based on the candidate word sequence and the acoustic feature sequence.

[0010] The candidate word element corresponding to the entropy value higher than the preset threshold is determined as the word element to be corrected. If the error type corresponding to the word element to be corrected is a term error, the word element to be corrected is corrected by the first automatic speech recognition model based on the target domain knowledge base, and the speech recognition result corresponding to the target audio is determined based on the correction result.

[0011] If the error type corresponding to the word to be corrected is an ambiguity error, then the third automatic speech recognition model is used to correct the word to be corrected, and the speech recognition result corresponding to the target audio is determined based on the correction result.

[0012] Optionally, the step of constructing a target dataset based on several pairs of homophones and near-homophones, and using low-rank adaptation technology to fine-tune the preset decoder of the first automatic speech recognition model based on the target dataset to obtain a third automatic speech recognition model includes:

[0013] The first sentence pair corresponding to each pair of homophones is generated using a pre-defined generative large model, and the second sentence pair corresponding to each pair of near-homophones is generated.

[0014] The first audio corresponding to the first sentence pair is generated based on a preset text-to-speech engine, and the second audio corresponding to the second sentence pair is generated. The target dataset is constructed based on the first audio and the second audio.

[0015] Determine the query projection matrix and value projection matrix of the preset decoder of the first automatic speech recognition model;

[0016] Based on the query projection matrix, a first matrix is ​​inserted in parallel into the preset decoder of the first automatic speech recognition model, and based on the value projection matrix, a second matrix is ​​inserted in parallel into the preset decoder of the first automatic speech recognition model.

[0017] The parameters of the query projection matrix and the value projection matrix are frozen, and the parameters of the first matrix and the second matrix are fine-tuned based on the target dataset using the low-rank adaptation technique to obtain the third automatic speech recognition model.

[0018] Optionally, determining the acoustic feature sequence corresponding to the target audio using the first automatic speech recognition model includes:

[0019] The audio waveform corresponding to the target audio is divided into frames, and the framed audio waveform is windowed to obtain the windowed audio waveform.

[0020] Perform a Fast Fourier Transform on the windowed audio waveform to obtain the Mel frequency cepstral coefficients corresponding to the target audio.

[0021] The acoustic feature sequence corresponding to the target audio is generated using the preset encoder of the first automatic speech recognition model based on the Mel frequency cepstral coefficients.

[0022] Optionally, generating the entropy value corresponding to each candidate word based on the candidate word sequence and the acoustic feature sequence using the first automatic speech recognition model includes:

[0023] The first automatic speech recognition model is used to obtain the key-value cache of the second automatic speech recognition model, and the candidate word sequence is determined based on the key-value cache;

[0024] Using the Softmax layer of the preset decoder of the first automatic speech recognition model, a posterior probability distribution is generated based on the candidate word sequence, the acoustic feature sequence, and the preset vocabulary. The entropy value corresponding to each candidate word is generated based on the Shannon entropy formula, the acoustic feature sequence, and the posterior probability distribution.

[0025] Optionally, before correcting the lexical units to be corrected using the first automatic speech recognition model based on the target domain knowledge base, the method further includes:

[0026] The target domain knowledge base is constructed based on several domain terms, and each domain term in the target domain knowledge base is encoded into a corresponding text vector using a preset text embedding model;

[0027] Store each of the text vectors into the target vector database;

[0028] Accordingly, the step of using the first automatic speech recognition model to correct the lexical units to be corrected based on the target domain knowledge base includes:

[0029] The first automatic speech recognition model is used to correct the word to be corrected based on the text vector in the target vector database.

[0030] Optionally, the step of using the first automatic speech recognition model to correct the word to be corrected based on the text vector in the target vector database includes:

[0031] Based on a preset context window, words adjacent to the word to be corrected are determined from the candidate word sequence, and query text is constructed based on the word to be corrected and the words adjacent to the word to be corrected.

[0032] The query text is encoded into a corresponding query vector using the preset text embedding model.

[0033] Based on the query vector, a maximum inner product search is performed on the text vectors in the target vector database, and the target text vector is determined based on the search results and a preset quantity condition;

[0034] Determine the target domain terminology corresponding to the target text vector, and construct target prompt words based on the target domain terminology, the candidate lexical sequence, and the lexical to be corrected;

[0035] The word to be corrected is corrected using the preset decoder of the first automatic speech recognition model based on the target prompt word and the acoustic feature sequence.

[0036] Optionally, after generating the entropy value corresponding to each candidate lemma using the first automatic speech recognition model based on the candidate lemma sequence and the acoustic feature sequence, the method further includes:

[0037] The candidate word elements corresponding to entropy values ​​that are not higher than a preset threshold are determined as target word elements;

[0038] Accordingly, determining the speech recognition result corresponding to the target audio based on the correction result includes:

[0039] The speech recognition result corresponding to the target audio is determined based on the correction result corresponding to the target word and the word to be corrected.

[0040] Secondly, this application provides a voice recognition device, comprising:

[0041] The model determination module is used to determine a first automatic speech recognition model based on a preset encoder and a preset decoder, and to perform lightweight processing on the first automatic speech recognition model to obtain a second automatic speech recognition model; wherein, the preset decoder is a decoder determined based on a preset large language model;

[0042] The dataset construction module is used to construct a target dataset based on several pairs of homophones and near-homophones, and to fine-tune the preset decoder of the first automatic speech recognition model based on the target dataset using low-rank adaptation technology to obtain a third automatic speech recognition model.

[0043] The candidate word sequence generation module is used to determine the acoustic feature sequence corresponding to the target audio using the first automatic speech recognition model, generate the candidate word sequence corresponding to the acoustic feature sequence using the second automatic speech recognition model based on the autoregressive method, and generate the entropy value corresponding to each candidate word using the first automatic speech recognition model based on the candidate word sequence and the acoustic feature sequence.

[0044] The module for determining the word to be corrected is used to determine the candidate word corresponding to the entropy value that is higher than a preset threshold as the word to be corrected. If the error type corresponding to the word to be corrected is a term error, the module uses the first automatic speech recognition model to correct the word to be corrected based on the target domain knowledge base, and determines the speech recognition result corresponding to the target audio based on the correction result.

[0045] The speech recognition result determination module is used to correct the word to be corrected using the third automatic speech recognition model if the error type corresponding to the word to be corrected is an ambiguity error, and to determine the speech recognition result corresponding to the target audio based on the correction result.

[0046] Thirdly, this application provides an electronic device, comprising:

[0047] Memory, used to store computer programs;

[0048] A processor is used to execute the computer program to implement the aforementioned speech recognition method.

[0049] Fourthly, this application provides a computer-readable storage medium for storing a computer program; wherein, when the computer program is executed by a processor, it implements the aforementioned speech recognition method.

[0050] In this application, a first automatic speech recognition model is first determined based on a preset encoder and a preset decoder, and a second automatic speech recognition model is obtained by lightweighting the first automatic speech recognition model; wherein, the preset decoder is a decoder determined based on a preset large language model; then, a target dataset is constructed based on several pairs of homophones and near-homophones, and a third automatic speech recognition model is obtained by fine-tuning the preset decoder of the first automatic speech recognition model based on the target dataset using low-rank adaptation technology; subsequently, the acoustic feature sequence corresponding to the target audio is determined using the first automatic speech recognition model, and the candidate audio sequence corresponding to the acoustic feature sequence is generated using the second automatic speech recognition model based on an autoregressive approach. The process involves selecting a sequence of candidate lexical units and generating entropy values ​​for each candidate lexical unit based on the candidate lexical unit sequence and the acoustic feature sequence using the first automatic speech recognition model. Then, candidate lexical units with entropy values ​​exceeding a preset threshold are identified as lexical units to be corrected. If the error type corresponding to the lexical unit to be corrected is a terminology error, the first automatic speech recognition model corrects the lexical unit to be corrected based on the target domain knowledge base, and the speech recognition result corresponding to the target audio is determined based on the correction result. If the error type corresponding to the lexical unit to be corrected is an ambiguity error, the third automatic speech recognition model corrects the lexical unit to be corrected, and the speech recognition result corresponding to the target audio is determined based on the correction result. As can be seen from the above, in this application, the first automatic speech recognition model first processes the target audio to extract an acoustic feature sequence that reflects key acoustic information of the audio. Then, the second automatic speech recognition model generates a candidate lexical unit sequence corresponding to the acoustic feature sequence in an autoregressive manner, providing a foundation for preliminary recognition. Subsequently, the first automatic speech recognition model combines the candidate lexical unit sequence and the acoustic feature sequence to calculate the entropy value of each candidate lexical unit to obtain the recognition uncertainty of each candidate lexical unit. Subsequently, candidate words with entropy values ​​exceeding a preset threshold are identified as words requiring correction, and their error types are distinguished: if it is a domain terminology error, the first automatic speech recognition model is invoked, and the word to be corrected is corrected using the target domain knowledge base; if it is an ambiguity error, a third automatic speech recognition model, fine-tuned from the first automatic speech recognition model using low-rank adaptation technology, is activated. The third automatic speech recognition model's accurate differentiation of homophones and near-homophones is utilized to complete the correction, and finally, the final speech recognition result of the target audio is determined based on the correction result. It is evident that this application introduces the entropy value of words as an uncertainty quantification indicator, enabling real-time and accurate evaluation of the confidence level of the automatic speech recognition model for each output word, and triggering word correction when the entropy value exceeds a preset threshold. In this way, this application can improve the accuracy and efficiency of speech recognition, providing a flexible and efficient solution for high-quality speech recognition needs in different scenarios. Attached Figure Description

[0051] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0052] Figure 1 A flowchart of a speech recognition method provided in this application;

[0053] Figure 2 A schematic diagram of the structure of a speech recognition device provided in this application;

[0054] Figure 3 This application provides a structural diagram of an electronic device. Detailed Implementation

[0055] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0056] While ASR technology has made progress over the past decade, homophone errors remain a core bottleneck limiting its reliability in professional applications. Existing technologies have significant drawbacks: decoding constraint methods based on traditional language models are essentially statistical models relying on word co-occurrence frequencies, failing to understand long-distance semantic dependencies and deep contextual logic, and struggling to handle homophones requiring the integration of global knowledge or professional background. Post-processing methods based on large language models treat ASR system outputs as independent text sequences for LLM polishing and correction, completely losing acoustic information. LLM, relying solely on textual context, may over-infer, producing errors that contradict the original audio content, resulting in low accuracy. End-to-end unified models, while deeply integrating acoustic encoders and LLM decoders to improve context awareness, have static built-in knowledge, rendering them ineffective for domain-specific and dynamically changing knowledge. Furthermore, their massive size and high computational cost per inference make it impossible to allocate resources according to the actual difficulty of the input. Therefore, this application provides a speech recognition solution that can improve the accuracy and efficiency of speech recognition.

[0057] See Figure 1 As shown, an embodiment of the present invention discloses a speech recognition method, which may include:

[0058] Step S11: Determine a first automatic speech recognition model based on a preset encoder and a preset decoder, and perform lightweight processing on the first automatic speech recognition model to obtain a second automatic speech recognition model; wherein, the preset decoder is a decoder determined based on a preset large language model.

[0059] In this embodiment, a first automatic speech recognition model is determined based on a preset encoder and a preset decoder. The preset encoder is an acoustic encoder built from a deep neural network based on a Conformer architecture; the preset decoder is a decoder determined based on a pre-trained large language model, for example, using Qwen3-7B-Instruct. The decoder has a lightweight adapter network that maps the features output by the acoustic encoder to a dimensionality space that matches the input of the LLM decoder.

[0060] It should be noted that, after obtaining the first automatic speech recognition model Then, the first automatic speech recognition model can be used. Knowledge distillation or pruning is performed to lightweight the model, resulting in a lightweight version of the second automatic speech recognition model. For example, the second automatic speech recognition model It can be a first automatic speech recognition model A model with the same architecture but with half the number of layers or attention heads.

[0061] Step S12: Construct a target dataset based on several pairs of homophones and near-homophones, and use low-rank adaptation technology to fine-tune the preset decoder of the first automatic speech recognition model based on the target dataset to obtain a third automatic speech recognition model.

[0062] In this embodiment, to improve the ASR model's ability to identify common high-frequency homophones, a dictionary containing hundreds of pairs of common homophones and near-homophones can be defined. For example, homophones: "login / login", near-homophones: "originally / originally is". Then, a target dataset can be constructed based on the dictionary. That is, the above-mentioned construction of a target dataset based on several pairs of homophones and near-homophones, and the fine-tuning of the preset decoder of the first automatic speech recognition model based on the target dataset using low-rank adaptation technology to obtain a third automatic speech recognition model, may include: firstly, using a preset generative large model to generate a first sentence pair corresponding to each pair of homophones, and generating a second sentence pair corresponding to each pair of near-homophones; then, using a preset text-to-speech engine to generate a first audio corresponding to the first sentence pair, and generating a second audio corresponding to the second sentence pair, and constructing a target dataset based on the first audio and the second audio; subsequently, determining the query projection matrix and value projection matrix of the preset decoder of the first automatic speech recognition model; then, inserting a first matrix in parallel into the preset decoder of the first automatic speech recognition model based on the query projection matrix, and inserting a second matrix in parallel into the preset decoder of the first automatic speech recognition model based on the value projection matrix; finally, freezing the parameters of the query projection matrix and the value projection matrix, and using low-rank adaptation technology to fine-tune the parameters of the first matrix and the second matrix based on the target dataset to obtain the third automatic speech recognition model.

[0063] Specifically, in this embodiment, a generative large-scale model, such as Qwen2.5-72B-Instruct, can be used to automatically generate corresponding sentence pairs for each pair of homophones and near-homophones, with the sentence pairs being completely different semantically. For example: homophones: deployment and subordinate; generated sentence pairs: He is very concerned about this deployment and he is very concerned about this subordinate. Then, a high-quality TTS (Text-to-Speech) engine, such as Microsoft Azure TTS, can be used to synthesize the generated sentence pairs into audio, thereby constructing a large-scale adversarial dataset focused on training the model to distinguish subtle semantic differences, thus obtaining the target dataset. Subsequently, LoRA (Low-Rank Adaptation) technology can be applied to the weights of the LLM encoder of the first automatic speech recognition model to fine-tune the model. Specifically, the query for determining the Transformer block of the LLM encoder is... Projection matrix and value The model first inserts a low-rank, for example, rank r=8, trainable first matrix A into the query projection matrix and a low-rank, trainable second matrix B into the value projection matrix. Then, using low-rank adaptation techniques, the parameters of the first and second matrices are fine-tuned based on the target dataset. During fine-tuning, the parameters of both the query and value projection matrices of the LLM encoder are frozen to obtain the third automatic speech recognition model. It can be seen that by using the third automatic speech recognition model, "expert knowledge" for distinguishing homophones and near-homophones can be efficiently injected into the model with minimal computational overhead. Furthermore, techniques such as orthogonal initialization can be used to avoid catastrophic forgetting of the general abilities learned during the pre-training phase.

[0064] Step S13: Use the first automatic speech recognition model to determine the acoustic feature sequence corresponding to the target audio, use the second automatic speech recognition model to generate the candidate word sequence corresponding to the acoustic feature sequence based on the autoregressive method, and use the first automatic speech recognition model to generate the entropy value corresponding to each candidate word based on the candidate word sequence and the acoustic feature sequence.

[0065] In this embodiment, a speculative decoding mechanism can be integrated to generate the initial recognition result of the target audio. Specifically, the aforementioned determination of the acoustic feature sequence corresponding to the target audio using the first automatic speech recognition model can include: firstly, segmenting the audio waveform corresponding to the target audio into frames, and then windowing the segmented audio waveform to obtain a windowed audio waveform; subsequently, performing a Fast Fourier Transform (FFT) on the windowed audio waveform to obtain the Mel-frequency cepstral coefficients corresponding to the target audio; and finally, using the preset encoder of the first automatic speech recognition model to generate the acoustic feature sequence corresponding to the target audio based on the Mel-frequency cepstral coefficients. Specifically, the audio waveform corresponding to the target audio first needs to undergo frame segmentation, windowing, and FFT (Fast Fourier Transform) standard preprocessing operations to obtain 80-dimensional logarithmic Mel-frequency cepstral coefficients. The Conformer architecture acoustic encoder combines convolution and self-attention structures, which can efficiently capture local and global dependencies in the audio sequence. Therefore, the acoustic encoder of the first automatic speech recognition model can be used to generate the acoustic feature sequence X corresponding to the target audio based on the Mel-frequency cepstral coefficients.

[0066] It should be noted that after the first automatic speech recognition model generates the acoustic feature sequence X corresponding to the target audio, the second automatic speech recognition model can be used to quickly generate the candidate word sequence corresponding to the acoustic feature sequence X using an autoregressive approach. Then, the first automatic speech recognition model can receive the candidate word sequence and verify the candidate words generated by the second automatic speech recognition model in a one-time, parallel manner. This verification process is achieved by reusing the key-value cache generated by the second automatic speech recognition model, avoiding a large amount of repetitive work in attention calculation by the first automatic speech recognition model. The specific implementation process includes: firstly, using the first automatic speech recognition model to obtain the key-value cache of the second automatic speech recognition model, and determining the candidate word sequence based on the key-value cache; then, using the Softmax layer of the preset decoder of the first automatic speech recognition model, generating a posterior probability distribution based on the candidate word sequence, the acoustic feature sequence, and the preset vocabulary, and generating the entropy value corresponding to each candidate word based on the Shannon entropy formula, the acoustic feature sequence, and the posterior probability distribution. Specifically, in order to generate the entropy value corresponding to each candidate lexical unit, for each time step t, i.e. each candidate lexical unit: using the final Softmax layer of the LLM decoder of the first automatic speech recognition model, based on the acoustic feature sequence X and the verified lexical sequence in the candidate lexical unit sequence. Output a candidate noun for the current word in the entire predefined vocabulary V. posterior probability distribution Then, calculate the Shannon entropy corresponding to the posterior probability distribution. As the model at time step t, i.e., candidate lexical units The quantitative indicator of uncertainty is shown in the following formula:

[0067] ;

[0068] in, is a random variable at time step t, representing the candidate word to be verified; X is an acoustic feature sequence generated by the acoustic encoder, representing the entire target audio. V is the sequence of candidate lexical units that have been validated before time step t; V is the complete vocabulary set used by the model; v is any lexical unit in the vocabulary V. This represents the model's prediction of the probability that the next word in the current context is v. Therefore, when validating the t-th candidate word, it is necessary to combine the validation with the previous t-1 validated candidate words.

[0069] Step S14: The candidate word units corresponding to the entropy values ​​higher than the preset threshold are determined as word units to be corrected. If the error type corresponding to the word unit to be corrected is a terminology error, the first automatic speech recognition model is used to correct the word unit to be corrected based on the target domain knowledge base, and the speech recognition result corresponding to the target audio is determined based on the correction result.

[0070] It can be understood that a high entropy value indicates a relatively flat probability distribution, making it difficult for the model to choose among multiple candidate tokens, that is, the uncertainty is high. A low entropy value indicates that the probability distribution is concentrated in a few tokens, and the model has a high confidence. Therefore, in this embodiment, the entropy values corresponding to each candidate token can be compared with a dynamic preset threshold for comparison. Among them, the preset threshold can be empirically set by analyzing the entropy distribution of the model output on the validation set and combining the expected recall rate and precision rate. If , then the candidate token is marked as a token to be corrected, and the candidate token and its corresponding audio segment are marked as high-uncertainty points, and the subsequent error correction process is triggered. Otherwise, the current speech recognition result is directly accepted. It should be noted that if the first automatic speech recognition model passes the verification of the first j candidate tokens (j < t) of the candidate token sequence, then these j candidate tokens are directly accepted. And error correction starts from the (j + 1)-th position. In this way, in one forward propagation of the first automatic speech recognition model, multiple tokens can be confirmed, thereby reducing the end-to-end inference delay by 2-3 times.

[0071] In this embodiment, error correction can be performed according to the error type corresponding to the token to be corrected. To solve the recognition errors caused by the lack of external knowledge such as domain-specific terms and rare named entities, error correction can be combined with a domain knowledge base. The specific process includes: First, a target domain knowledge base can be constructed based on several domain terms, and each domain term in the target domain knowledge base can be encoded into a corresponding text vector by using a preset text embedding model; then each text vector is stored in the target vector database. Specifically, in this embodiment, one or more proprietary target domain knowledge bases can be constructed offline. For example, a drug dictionary containing medical domain terms such as "oseltamivir" and "amoxicillin", or a corporate domain term dictionary containing several listed company names. After that, a preset text embedding model, such as BGE-large-zh, can be used to encode each domain term in the target domain knowledge base, such as drug names and company names, and their related description texts into high-dimensional text vectors, such as 1024-dimensional text vectors. Finally, each text vector can be stored in a target vector database that supports efficient similarity search, such as FAISS (Facebook AI Similarity Search), and an index is established for each text vector for retrieval.

[0072] In this way, if the error type corresponding to the word to be corrected is a terminology error, the first automatic speech recognition model can be used to correct the word based on the text vector in the target vector database. Specifically, firstly, based on a preset context window, words adjacent to the word to be corrected are determined from the candidate word sequence, and a query text is constructed based on the word to be corrected and its adjacent words. Then, a preset text embedding model is used to encode the query text into a corresponding query vector. Subsequently, a maximum inner product search is performed on the text vector in the target vector database based on the query vector, and the target text vector is determined based on the search results and preset quantity conditions. Then, the target domain terminology corresponding to the target text vector is determined, and target prompt words are constructed based on the target domain terminology, the candidate word sequence, and the word to be corrected. Finally, the preset decoder of the first automatic speech recognition model is used to correct the word based on the target prompt words and the acoustic feature sequence. That is, firstly, based on a preset context window k, for example, k=5, words containing the word to be corrected and its preceding and following adjacent words can be extracted from the candidate word sequence to form the query text. Next, a pre-defined text embedding model, such as the BGE-large-zh model, identical to the target domain knowledge base, can be used to encode the query text into corresponding query vectors. Based on these query vectors, MIPS (Maximum Inner Product Search) is performed on the text vectors in the target vector database to retrieve the top N most relevant target text vectors, for example, the top three, and to determine the target domain terms corresponding to these vectors. Subsequently, to inject the retrieved target domain terms as strong prior information into the speech recognition process, a text prompt containing explicit instructions and target domain terms can be generated, for example: "Please use the following medical knowledge: 1. Oseltamivir, 2. Olanzapine," and combined with audio, correct the recognition result: '...the last prescription for Oseltamivir...'. Finally, the generated target prompt and acoustic feature sequence X can be re-input into a specific adaptation layer of the LLM decoder in the first automatic speech recognition model for further correction. The adaptation layer forces the decoder to generate accurate lexical units within the provided knowledge candidate range, or to perform probability reordering on the complete sentence hypothesis containing these candidate words, and finally outputs the recognition result corrected by external knowledge. The adaptation layer can be understood as an MLP (Multilayer Perceptron), which consists of several fully connected network layers.

[0073] Step S15: If the error type corresponding to the word to be corrected is an ambiguity error, then the third automatic speech recognition model is used to correct the word to be corrected, and the speech recognition result corresponding to the target audio is determined based on the correction result.

[0074] It should be noted that, since the third automatic speech recognition model has the "expert knowledge" ability to distinguish between common homophones and near-homophones, when the error type corresponding to the word to be corrected is an ambiguity error, the third automatic speech recognition model can be used to correct the word to be corrected.

[0075] Understandably, since low entropy values ​​indicate high model confidence, in this embodiment, candidate words with entropy values ​​not exceeding a preset threshold can be identified as target words, and the final speech recognition result corresponding to the target audio can be determined based on the correction results corresponding to the target words and the words to be corrected.

[0076] As can be seen from the above, in this embodiment, a first automatic speech recognition model is first determined based on a preset encoder and a preset decoder, and a second automatic speech recognition model is obtained by lightweighting the first automatic speech recognition model; wherein, the preset decoder is a decoder determined based on a preset large language model; then, a target dataset is constructed based on several pairs of homophones and near-homophones, and a third automatic speech recognition model is obtained by fine-tuning the preset decoder of the first automatic speech recognition model based on the target dataset using low-rank adaptation technology; subsequently, the acoustic feature sequence corresponding to the target audio is determined using the first automatic speech recognition model, and the acoustic feature sequence corresponding to the target audio is generated using the second automatic speech recognition model based on an autoregressive approach. The system first uses a candidate word sequence and a second automatic speech recognition model to generate entropy values ​​for each candidate word based on the candidate word sequence and the acoustic feature sequence. Then, candidate words with entropy values ​​higher than a preset threshold are identified as words to be corrected. If the error type of the word to be corrected is a terminology error, the first automatic speech recognition model corrects the word based on the target domain knowledge base, and the speech recognition result corresponding to the target audio is determined based on the correction result. If the error type of the word to be corrected is an ambiguity error, the third automatic speech recognition model corrects the word, and the speech recognition result corresponding to the target audio is determined based on the correction result. As can be seen from the above, in this embodiment, the first automatic speech recognition model first processes the target audio and extracts an acoustic feature sequence that reflects the key acoustic information of the audio. Then, the second automatic speech recognition model generates a candidate word sequence corresponding to the acoustic feature sequence in an autoregressive manner, providing a foundation for preliminary recognition. Subsequently, the first automatic speech recognition model combines the candidate word sequence and the acoustic feature sequence to calculate the entropy value of each candidate word to obtain the recognition uncertainty of each candidate word. Subsequently, candidate words with entropy values ​​exceeding a preset threshold are identified as words requiring correction, and their error types are distinguished: if it is a domain terminology error, the first automatic speech recognition model is invoked, and the word to be corrected is corrected using the target domain knowledge base; if it is an ambiguity error, a third automatic speech recognition model, fine-tuned from the first automatic speech recognition model using low-rank adaptation technology, is activated. The third automatic speech recognition model's accurate differentiation of homophones and near-homophones is utilized to complete the correction, and finally, the final speech recognition result of the target audio is determined based on the correction result. It can be seen that this embodiment introduces the entropy value of words as an uncertainty quantification indicator, which can evaluate the confidence level of the automatic speech recognition model for each output word in real time and accurately, and trigger word correction when the entropy value exceeds a preset threshold. In this way, this embodiment can improve the accuracy and efficiency of speech recognition, providing a flexible and efficient solution for high-quality speech recognition needs in different scenarios.

[0077] The technical solution of speech recognition method is explained below with specific examples.

[0078] Scenario Description: During a doctor's consultation, the doctor verbally describes the patient's condition and diagnosis using a wearable microphone. The system needs to convert this speech into structured electronic medical record text in real time and accurately. This scenario demands extremely high accuracy, especially for medication names and key symptom descriptions; any error could lead to serious medical consequences.

[0079] System objective: To accurately identify and distinguish near-homophones in general symptom descriptions, such as "dizziness" and "lightheadedness," as well as professional drug names, such as "oseltamivir."

[0080] Implementation process:

[0081] Step A: System Initialization and Configuration

[0082] (1) Basic recognition engine: First automatic speech recognition model An end-to-end ASR model based on a Conformer encoder and a Qwen3-7B-Instruct decoder is adopted.

[0083] (2) Draft model: Second automatic speech recognition model Through the The decoder is pruned layer by layer, for example, from 32 layers to 8 layers, resulting in a lightweight model for speculative decoding.

[0084] (3) Preset threshold : Set a threshold by testing on a medical dialogue validation set. .

[0085] (4) RAG (Retrieval-augmented Generation) domain knowledge base: Construct a medical knowledge base containing tens of thousands of generic names, brand names and indications of drugs, and use the BGE-large-zh model to vectorize the domain terms in the medical knowledge base and store them in the FAISS vector database.

[0086] (5) Fine-tuning model: The third automatic speech recognition model, which uses a dataset containing several pairs of homophones and near-homophones and LoRA technology to fine-tune the speech recognition model. This was achieved through fine-tuning and is used to distinguish between homophones and near-homophones.

[0087] Step B: Real-time Inference and Error Correction Process

[0088] Suppose the doctor dictated the following sentences:

[0089] "The patient reported frequent dizziness over the past week and had finished the last dose of oseltamivir prescribed."

[0090] (1) Basic recognition and uncertainty quantification

[0091] Through a speculative decoding mechanism, the system quickly generates a draft to obtain the corresponding candidate token sequence, and performs parallel verification to obtain the preliminary recognition result and the entropy value of each candidate token.

[0092] Among them, the preliminary recognition result: The patient reported frequent dizziness in the recent week, and the previously prescribed Ostavit has been taken up.

[0093] Uncertainty quantification: 1. At the position of the character "昏", since the pronunciations of "昏" and "晕" are similar and both may appear after the character "头", the probability distribution output by the model is relatively dispersed: P(昏)=0.45, P(晕)=0.40, P(军)=0.05,.... The entropy value at this position is calculated . 2. For the phrase "Ostavit", since this is a rare combination outside the vocabulary, the model shows extremely high uncertainty for each character. For example, the entropy value at the position of the character "斯" .

[0094] (2) Dynamic activation decision

[0095] The system checks the entropy value of each candidate token one by one:

[0096] , marks "头昏" as a token to be corrected, which is a high-uncertainty point, and the error type is an ambiguity error.

[0097] , marks "Ostavit" as a token to be corrected, which is a high-uncertainty point, and the error type is a terminology error.

[0098] The gating mechanism is triggered to perform error correction based on the audio segment containing these two points and the context information.

[0099] (3) Collaborative reasoning

[0100] The system parallelly invokes different error correction modules according to the error types of the tokens to be corrected:

[0101] ] 1. For the token to be corrected "头昏", the error type is an ambiguity error:

[0102] Module routing: The system detects that "头昏" and "头晕" form a known common homophone pair, and invokes the third automatic speech recognition model.

[0103] Fine-tuning model intervention: The context containing "头昏", i.e., "... frequent dizziness in the recent week...", together with the corresponding acoustic features The input is then fed into the decoder of the third automatic speech recognition model.

[0104] Ambiguity cancellation: The "expert knowledge" embedded in the adapter of the third automatic speech recognition model's decoder enables the model to understand that symptoms modified by "frequent" tend to use the more medically precise term "dizziness." The model therefore outputs the corrected result: "dizziness."

[0105] 2. For "Ostave", the error type is terminology error:

[0106] Module routing: The system determines that this is a potential named entity or domain term and calls the RAG domain knowledge base.

[0107] Query construction: Use the context "...the last dose of Ostavi was taken..." as the query text and generate the corresponding query vector.

[0108] Vector retrieval: Based on the query vector, the FAISS vector database is searched, and the top two most relevant entries are returned: [('Oseltamivir', 'For the treatment of influenza A and B...', ('Olanzapine', 'For the treatment of schizophrenia...')].

[0109] Constraint-based decoding: Construct target prompts, such as: "Based on the following medical knowledge: 1. Oseltamivir, an anti-influenza drug; 2. Olanzapine, a psychiatric drug, and combined with audio, correct the recognition result: '...the last prescription for Oseltamivir...'". Then, combine the target prompts with acoustic features. They are input together into the LLM decoder of the first automatic speech recognition model.

[0110] Knowledge Fusion and Adjudication: The LLM decoder combines acoustic information for a more accurate pronunciation. Based on the context, the patient's symptoms were "dizziness," which was more likely related to influenza. Ultimately, the corrected result was determined to be "oseltamivir" with a very high probability.

[0111] Step C: Results Integration and Output

[0112] The system replaces the words to be corrected in the initial recognition results with the corrected output, and presents the user with the final, high-fidelity speech recognition result:

[0113] "The patient reported frequent dizziness over the past week and had finished the last dose of oseltamivir prescribed."

[0114] As can be seen from the above, although the entire process involves multiple stages, the latency perceived by the user is controlled within hundreds of milliseconds due to speculative decoding and parallel processing, achieving a balance between real-time performance and accuracy. Furthermore, the aforementioned speech recognition process can be applied to scenarios such as law enforcement and finance.

[0115] Accordingly, see Figure 2 As shown in the illustration, this application also provides a voice recognition device, which may include:

[0116] The model determination module 11 is used to determine a first automatic speech recognition model based on a preset encoder and a preset decoder, and to perform lightweight processing on the first automatic speech recognition model to obtain a second automatic speech recognition model; wherein, the preset decoder is a decoder determined based on a preset large language model;

[0117] The dataset construction module 12 is used to construct a target dataset based on several pairs of homophones and near-homophones, and to fine-tune the preset decoder of the first automatic speech recognition model based on the target dataset using low-rank adaptation technology to obtain a third automatic speech recognition model.

[0118] The candidate word sequence generation module 13 is used to determine the acoustic feature sequence corresponding to the target audio using the first automatic speech recognition model, generate the candidate word sequence corresponding to the acoustic feature sequence using the second automatic speech recognition model based on the autoregressive method, and generate the entropy value corresponding to each candidate word using the first automatic speech recognition model based on the candidate word sequence and the acoustic feature sequence.

[0119] The lexical unit determination module 14 is used to determine the candidate lexical unit corresponding to the entropy value that is higher than the preset threshold as the lexical unit to be corrected. If the error type corresponding to the lexical unit to be corrected is a terminology error, the first automatic speech recognition model is used to correct the lexical unit to be corrected based on the target domain knowledge base, and the speech recognition result corresponding to the target audio is determined based on the correction result.

[0120] The speech recognition result determination module 15 is used to correct the word to be corrected using the third automatic speech recognition model if the error type corresponding to the word to be corrected is an ambiguity error, and to determine the speech recognition result corresponding to the target audio based on the correction result.

[0121] In some specific embodiments, the dataset construction module 12 may include:

[0122] The sentence pair generation unit is used to generate the first sentence pair corresponding to each pair of homophones and the second sentence pair corresponding to each pair of near-homophones using a pre-defined generative large model.

[0123] The dataset construction unit is used to generate the first audio corresponding to the first sentence pair based on a preset text-to-speech engine, generate the second audio corresponding to the second sentence pair, and construct the target dataset based on the first audio and the second audio.

[0124] The value projection matrix determination unit is used to determine the query projection matrix and value projection matrix of the preset decoder of the first automatic speech recognition model;

[0125] A matrix insertion unit is used to insert a first matrix in parallel in the preset decoder of the first automatic speech recognition model based on the query projection matrix, and to insert a second matrix in parallel in the preset decoder of the first automatic speech recognition model based on the value projection matrix.

[0126] The parameter fine-tuning unit is used to freeze the parameters of the query projection matrix and the parameters of the value projection matrix, and use the low-rank adaptation technique to fine-tune the parameters of the first matrix and the second matrix based on the target dataset to obtain the third automatic speech recognition model.

[0127] In some specific embodiments, the candidate lexical sequence generation module 13 may include:

[0128] An audio waveform framing unit is used to segment the audio waveform corresponding to the target audio into frames, and to perform windowing processing on the segmented audio waveform to obtain a windowed audio waveform.

[0129] The Mel frequency cepstral coefficient determination unit is used to perform a fast Fourier transform on the windowed audio waveform to obtain the Mel frequency cepstral coefficients corresponding to the target audio.

[0130] An acoustic feature sequence generation unit is used to generate the acoustic feature sequence corresponding to the target audio based on the Mel frequency cepstral coefficients using the preset encoder of the first automatic speech recognition model.

[0131] In some specific embodiments, the candidate lexical sequence generation module 13 may include:

[0132] A key-value cache acquisition unit is used to acquire the key-value cache of the second automatic speech recognition model using the first automatic speech recognition model, and to determine the candidate word sequence based on the key-value cache;

[0133] The entropy generation unit is used to generate a posterior probability distribution based on the candidate lexical sequence, the acoustic feature sequence, and the preset vocabulary using the Softmax layer of the preset decoder of the first automatic speech recognition model, and to generate the entropy value corresponding to each candidate lexical based on the Shannon entropy formula, the acoustic feature sequence, and the posterior probability distribution.

[0134] In some specific embodiments, the voice recognition device may further include:

[0135] The domain knowledge base construction module is used to construct the target domain knowledge base based on several domain terms, and to encode each domain term in the target domain knowledge base into a corresponding text vector using a preset text embedding model;

[0136] A text vector storage module is used to store each of the text vectors into a target vector database;

[0137] Accordingly, the module 14 for determining the lexical unit to be corrected may include:

[0138] The lexical correction submodule is used to correct the lexical to be corrected based on the text vector in the target vector database using the first automatic speech recognition model.

[0139] In some specific implementations, the lexical correction submodule may include:

[0140] The query text construction unit is used to determine the word elements adjacent to the word element to be corrected from the candidate word element sequence based on a preset context window, and to construct query text based on the word element to be corrected and the word elements adjacent to the word element to be corrected.

[0141] A query vector determination unit is used to encode the query text into a corresponding query vector using the preset text embedding model;

[0142] The target text vector determination unit is used to perform a maximum inner product search on the text vectors in the target vector database based on the query vector, and determine the target text vector based on the search results and a preset quantity condition;

[0143] The target prompt word construction unit is used to determine the target domain term corresponding to the target text vector, and construct the target prompt word based on the target domain term, the candidate lexical sequence and the lexical to be corrected;

[0144] The lexical correction unit is used to correct the lexical to be corrected based on the target prompt word and the acoustic feature sequence using the preset decoder of the first automatic speech recognition model.

[0145] In some specific embodiments, the voice recognition device may further include:

[0146] The target word determination module is used to determine the candidate word elements corresponding to entropy values ​​that are not higher than a preset threshold as target word elements.

[0147] Accordingly, the speech recognition result determination module 15 may include:

[0148] A speech recognition result determination unit is used to determine the speech recognition result corresponding to the target audio based on the correction result corresponding to the target word and the word to be corrected.

[0149] Furthermore, embodiments of this application also disclose an electronic device, Figure 3 This is a structural diagram of an electronic device 20 according to an exemplary embodiment. The content of the diagram should not be construed as limiting the scope of this application. The electronic device 20 may specifically include: at least one processor 21, at least one memory 22, a power supply 23, a communication interface 24, an input / output interface 25, and a communication bus 26. The memory 22 stores a computer program, which is loaded and executed by the processor 21 to implement the relevant steps in the speech recognition method disclosed in any of the foregoing embodiments. Furthermore, the electronic device 20 in this embodiment may specifically be an electronic computer.

[0150] In this embodiment, the power supply 23 is used to provide operating voltage for each hardware device on the electronic device 20; the communication interface 24 can create a data transmission channel between the electronic device 20 and external devices, and the communication protocol it follows can be any communication protocol applicable to the technical solution of this application, and is not specifically limited here; the input / output interface 25 is used to acquire external input data or output data to the outside world, and its specific interface type can be selected according to specific application needs, and is not specifically limited here.

[0151] In addition, the memory 22, as a carrier for resource storage, can be a read-only memory, random access memory, disk or optical disk, etc. The resources stored thereon can include operating system 221, computer program 222, etc., and the storage method can be temporary storage or permanent storage.

[0152] The operating system 221 is used to manage and control the various hardware devices on the electronic device 20 and the computer program 222, which may be Windows Server, Netware, Unix, Linux, etc. In addition to including a computer program capable of performing the speech recognition method executed by the electronic device 20 as disclosed in any of the foregoing embodiments, the computer program 222 may further include a computer program capable of performing other specific tasks.

[0153] Furthermore, this application also discloses a computer-readable storage medium for storing a computer program; wherein, when the computer program is executed by a processor, it implements the aforementioned disclosed speech recognition method. Specific steps of this method can be found in the corresponding content disclosed in the foregoing embodiments, and will not be repeated here.

[0154] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the apparatus disclosed in the embodiments, since it corresponds to the method disclosed in the embodiments, the description is relatively simple; relevant parts can be referred to in the method section.

[0155] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0156] The steps of the methods or algorithms described in conjunction with the embodiments disclosed herein can be implemented directly by hardware, a software module executed by a processor, or a combination of both. The software module can be located in random access memory (RAM), main memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disk, removable disk, CD-ROM, or any other form of storage medium known in the art.

[0157] Finally, it should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0158] The technical solutions provided in this application have been described in detail above. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the above embodiments are only for the purpose of helping to understand the methods and core ideas of this application. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this application. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. A speech recognition method, characterized in that, include: A first automatic speech recognition model is determined based on a preset encoder and a preset decoder, and a second automatic speech recognition model is obtained by lightweighting the first automatic speech recognition model; wherein, the preset decoder is a decoder determined based on a preset large language model; A target dataset is constructed based on several pairs of homophones and near-homophones, and a third automatic speech recognition model is obtained by fine-tuning the preset decoder of the first automatic speech recognition model based on the target dataset using low-rank adaptation technology. The first automatic speech recognition model is used to determine the acoustic feature sequence corresponding to the target audio, and the second automatic speech recognition model is used to generate the candidate word sequence corresponding to the acoustic feature sequence based on the autoregressive method. The first automatic speech recognition model is used to generate the entropy value corresponding to each candidate word based on the candidate word sequence and the acoustic feature sequence. The candidate word element corresponding to the entropy value higher than the preset threshold is determined as the word element to be corrected. If the error type corresponding to the word element to be corrected is a term error, the word element to be corrected is corrected by the first automatic speech recognition model based on the target domain knowledge base, and the speech recognition result corresponding to the target audio is determined based on the correction result. If the error type corresponding to the word to be corrected is an ambiguity error, then the third automatic speech recognition model is used to correct the word to be corrected, and the speech recognition result corresponding to the target audio is determined based on the correction result.

2. The speech recognition method according to claim 1, characterized in that, The process of constructing a target dataset based on several pairs of homophones and near-homophones, and then using low-rank adaptation technology to fine-tune the preset decoder of the first automatic speech recognition model based on the target dataset to obtain a third automatic speech recognition model, includes: The first sentence pair corresponding to each pair of homophones is generated using a pre-defined generative large model, and the second sentence pair corresponding to each pair of near-homophones is generated. The first audio corresponding to the first sentence pair is generated based on a preset text-to-speech engine, and the second audio corresponding to the second sentence pair is generated. The target dataset is constructed based on the first audio and the second audio. Determine the query projection matrix and value projection matrix of the preset decoder of the first automatic speech recognition model; Based on the query projection matrix, a first matrix is ​​inserted in parallel into the preset decoder of the first automatic speech recognition model, and based on the value projection matrix, a second matrix is ​​inserted in parallel into the preset decoder of the first automatic speech recognition model. The parameters of the query projection matrix and the value projection matrix are frozen, and the parameters of the first matrix and the second matrix are fine-tuned based on the target dataset using the low-rank adaptation technique to obtain the third automatic speech recognition model.

3. The speech recognition method according to claim 1, characterized in that, The step of determining the acoustic feature sequence corresponding to the target audio using the first automatic speech recognition model includes: The audio waveform corresponding to the target audio is divided into frames, and the framed audio waveform is windowed to obtain the windowed audio waveform. Perform a Fast Fourier Transform on the windowed audio waveform to obtain the Mel frequency cepstral coefficients corresponding to the target audio. The acoustic feature sequence corresponding to the target audio is generated using the preset encoder of the first automatic speech recognition model based on the Mel frequency cepstral coefficients.

4. The speech recognition method according to claim 1, characterized in that, The step of generating entropy values ​​for each candidate word based on the candidate word sequence and the acoustic feature sequence using the first automatic speech recognition model includes: The first automatic speech recognition model is used to obtain the key-value cache of the second automatic speech recognition model, and the candidate word sequence is determined based on the key-value cache; Using the Softmax layer of the preset decoder of the first automatic speech recognition model, a posterior probability distribution is generated based on the candidate lexical sequence, the acoustic feature sequence, and the preset vocabulary. The entropy value corresponding to each candidate lexical is then generated based on the Shannon entropy formula, the acoustic feature sequence, and the posterior probability distribution.

5. The speech recognition method according to claim 1, characterized in that, Before using the first automatic speech recognition model to correct the word to be corrected based on the target domain knowledge base, the method further includes: The target domain knowledge base is constructed based on several domain terms, and each domain term in the target domain knowledge base is encoded into a corresponding text vector using a preset text embedding model; Store each of the text vectors into the target vector database; Accordingly, the step of using the first automatic speech recognition model to correct the lexical units to be corrected based on the target domain knowledge base includes: The first automatic speech recognition model is used to correct the word to be corrected based on the text vector in the target vector database.

6. The speech recognition method according to claim 5, characterized in that, The step of using the first automatic speech recognition model to correct the word to be corrected based on the text vector in the target vector database includes: Based on a preset context window, words adjacent to the word to be corrected are determined from the candidate word sequence, and query text is constructed based on the word to be corrected and the words adjacent to the word to be corrected. The query text is encoded into a corresponding query vector using the preset text embedding model. Based on the query vector, a maximum inner product search is performed on the text vectors in the target vector database, and the target text vector is determined based on the search results and a preset quantity condition; Determine the target domain terminology corresponding to the target text vector, and construct target prompt words based on the target domain terminology, the candidate lexical sequence, and the lexical to be corrected; The word to be corrected is corrected using the preset decoder of the first automatic speech recognition model based on the target prompt word and the acoustic feature sequence.

7. The speech recognition method according to any one of claims 1 to 6, characterized in that, After generating the entropy value corresponding to each candidate word based on the candidate word sequence and the acoustic feature sequence using the first automatic speech recognition model, the method further includes: The candidate word elements corresponding to entropy values ​​that are not higher than a preset threshold are determined as target word elements; Accordingly, determining the speech recognition result corresponding to the target audio based on the correction result includes: The speech recognition result corresponding to the target audio is determined based on the correction result corresponding to the target word and the word to be corrected.

8. A voice recognition device, characterized in that, include: The model determination module is used to determine a first automatic speech recognition model based on a preset encoder and a preset decoder, and to perform lightweight processing on the first automatic speech recognition model to obtain a second automatic speech recognition model; wherein, the preset decoder is a decoder determined based on a preset large language model; The dataset construction module is used to construct a target dataset based on several pairs of homophones and near-homophones, and to fine-tune the preset decoder of the first automatic speech recognition model based on the target dataset using low-rank adaptation technology to obtain a third automatic speech recognition model. The candidate word sequence generation module is used to determine the acoustic feature sequence corresponding to the target audio using the first automatic speech recognition model, generate the candidate word sequence corresponding to the acoustic feature sequence using the second automatic speech recognition model based on the autoregressive method, and generate the entropy value corresponding to each candidate word using the first automatic speech recognition model based on the candidate word sequence and the acoustic feature sequence. The module for determining the word to be corrected is used to determine the candidate word corresponding to the entropy value that is higher than a preset threshold as the word to be corrected. If the error type corresponding to the word to be corrected is a term error, the module uses the first automatic speech recognition model to correct the word to be corrected based on the target domain knowledge base, and determines the speech recognition result corresponding to the target audio based on the correction result. The speech recognition result determination module is used to correct the word to be corrected using the third automatic speech recognition model if the error type corresponding to the word to be corrected is an ambiguity error, and to determine the speech recognition result corresponding to the target audio based on the correction result.

9. An electronic device, characterized in that, The electronic device includes a processor and a memory; wherein the memory is used to store a computer program, which is loaded and executed by the processor to implement the speech recognition method as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, Used to store a computer program, which, when executed by a processor, implements the speech recognition method as described in any one of claims 1 to 7.

Images