Speech recognition method and device, electronic equipment and storage medium

Abstract

The application provides a speech recognition method and device, electronic equipment and storage medium, and relates to the technical field of data processing. The method comprises the following steps: performing acoustic feature extraction and encoding processing on to-be-recognized speech data to obtain speech coding features; performing parallel decoding and feature mapping processing on the speech coding features to obtain a connection time sequence classification candidate decoding result and a speech embedding vector; based on the speech embedding vector, performing multi-level retrieval screening from a pre-constructed keyword embedding database to obtain a target keyword set and metadata information; constructing structured prompt information based on the connection time sequence classification candidate decoding result, the target keyword set and the metadata information, and inputting the structured prompt information and the speech embedding vector into a preset large language model for autoregressive reasoning to obtain an initial recognition text; and performing illusion repair processing on the initial recognition text based on acoustic consistency to obtain a final recognition text.

Description

Technical Field

[0001] This invention relates to the field of data processing technology, and in particular to a speech recognition method, device, electronic device, and storage medium. Background Technology

[0002] Speech recognition technology, as an important branch of artificial intelligence, has been widely used in various scenarios such as intelligent assistants, automatic caption generation, voice search, meeting recording, and smart homes. With the development of end-to-end deep learning technology, speech recognition models based on connection-time classification, attention encoding / decoding, and the Transformer architecture have significantly improved the recognition accuracy in general scenarios.

[0003] However, in practical industrial applications, speech recognition systems generally maintain low accuracy when dealing with large-scale keyword datasets, including names of people, places, organizations, technical terms, and newly emerging entities. As the scale of keywords in application scenarios gradually expands to tens of millions, the bottleneck in recognition accuracy caused by the aforementioned issues becomes increasingly prominent. How to improve the keyword recognition accuracy of speech recognition systems in large-scale keyword scenarios has become an urgent technical problem to be solved. Summary of the Invention

[0004] This invention provides a speech recognition method, device, electronic device, and storage medium to solve the problem of how to improve the recognition accuracy of speech recognition systems in large-scale keyword scenarios, and to achieve high accuracy recognition of large-scale keywords and high adaptability to complex contexts.

[0005] This invention provides a speech recognition method, comprising the following steps: Acquire the speech data to be recognized, and perform acoustic feature extraction and encoding processing on the speech data to be recognized to obtain speech coding features; The speech coding features are subjected to parallel decoding and feature mapping processes to obtain the connection time-series classification candidate decoding results and speech embedding vectors; wherein, the speech embedding vectors are mapped to a preset text feature space, which is the feature space of the keyword text embedding vectors output by the preset text encoder; Based on the speech embedding vector, a multi-level search and filtering process is performed from a pre-constructed keyword embedding database to obtain a set of target keywords and metadata information corresponding to the set of target keywords; wherein, the keyword embedding database stores the keyword text embedding vectors output by the preset text encoder; Based on the connection time-series classification candidate decoding results, the target keyword set, and the metadata information, structured prompt information is constructed, and the structured prompt information and the speech embedding vector are jointly input into a preset large language model for autoregressive inference to obtain the initial recognition text. The initial identified text is subjected to an illusion restoration process based on acoustic consistency to obtain the final identified text.

[0006] According to a speech recognition method provided by the present invention, the step of extracting and encoding acoustic features from the speech data to be recognized to obtain speech coding features includes: Acoustic features are extracted from the speech data to be recognized; the acoustic features are input into a preset speech encoder for processing, and the speech encoding features are output.

[0007] According to a speech recognition method provided by the present invention, the step of obtaining a target keyword set by performing multi-level retrieval and filtering from a pre-constructed keyword embedding database based on the speech embedding vector includes: A first keyword candidate set is obtained by filtering based on the similarity between the speech embedding vector and each of the keyword text embedding vectors in the keyword embedding database; Based on a preset pronunciation dictionary, the connection time sequence classification candidate decoding results are converted into a reference pronunciation sequence. Based on the reference pronunciation sequence, the first keyword candidate set is subjected to pronunciation matching and filtering to obtain a second keyword candidate set. Based on the preset text encoder, candidate text embedding vectors corresponding to the connection time-series classification candidate decoding results are extracted, and the target keyword set is obtained by filtering based on the semantic similarity between the candidate text embedding vectors and the text features corresponding to the second keyword candidate set.

[0008] According to a speech recognition method provided by the present invention, a second keyword candidate set is obtained by performing pronunciation matching and filtering on a first keyword candidate set based on the reference pronunciation sequence, including: Based on the preset pronunciation dictionary, each keyword in the first keyword candidate set is converted into a corresponding keyword pronunciation sequence; The keyword pronunciation sequence is matched and compared with the reference pronunciation sequence. Keywords that meet the conditions of complete pronunciation matching in the reference pronunciation sequence and minimum pitch difference are retained to obtain the second keyword candidate set.

[0009] According to a speech recognition method provided by the present invention, the structured prompt information is organized in a structured text format; wherein, in addition to the connection time-series classification candidate decoding results, the target keyword set, and the metadata information, the structured prompt information also includes user historical dialogue information.

[0010] According to a speech recognition method provided by the present invention, the initial recognized text is subjected to hallucination restoration processing based on acoustic consistency to obtain the final recognized text, including: Based on forced alignment processing, the acoustic matching score between the initial recognition text and the speech data to be recognized is calculated; The initial recognized text is converted into a first pronunciation sequence, and each recognition result in the connection time-series classification candidate decoding result is converted into a second pronunciation sequence; Calculate the average pronunciation edit distance between the first pronunciation sequence and each of the second pronunciation sequences; Based on the acoustic matching score and the average pronunciation edit distance, the confidence score of the initial recognized text is calculated; When the confidence score is greater than a preset confidence threshold, the initial identified text is used as the final identified text. When the confidence score is not greater than the preset confidence threshold, the best result among the candidate decoding results of the connection time series classification is taken as the final recognized text.

[0011] According to a speech recognition method provided by the present invention, the method further includes: For each keyword embedded in the keyword database, an initial comprehensive weight is set; wherein, the comprehensive weight is calculated based on the popularity weight and the confidence weight. When a matching keyword is detected in the target keyword set, the popularity weight of the matching keyword is increased by a fixed value, and the popularity weight of all keywords embedded in the database is decayed based on the time decay factor. If the hit keyword is contained in the final recognized text, obtain the output probability of the preset large language model when generating the hit keyword, and perform weighted fusion based on the output probability and the confidence weight of the hit keyword to obtain the updated confidence weight of the hit keyword. If the hit keyword is not included in the final identified text, the confidence weight of the hit keyword is attenuated based on a preset attenuation coefficient to obtain the updated confidence weight of the hit keyword. Based on the updated popularity weight and the updated confidence weight of the hit keywords, the overall weight of the hit keywords is updated. Based on the comprehensive weight, data update and cleanup operations are performed on the keyword embedding database.

[0012] According to a speech recognition method provided by the present invention, the metadata information includes at least one of the following: keyword aliases, scene information, domain information, and summary description.

[0013] The present invention also provides a voice recognition device, comprising the following modules: The acquisition module is used to acquire the speech data to be recognized, and to perform acoustic feature extraction and encoding processing on the speech data to be recognized to obtain speech coding features; The parallel processing module is used to perform parallel decoding and feature mapping on the speech coding features to obtain the connection time-series classification candidate decoding results and speech embedding vectors; wherein, the speech embedding vectors are mapped to a preset text feature space, which is the feature space of the keyword text embedding vectors output by the preset text encoder. The filtering module is used to perform multi-level retrieval and filtering from a pre-constructed keyword embedding database based on the speech embedding vector to obtain a set of target keywords and metadata information corresponding to the set of target keywords; wherein, the keyword embedding database stores the keyword text embedding vectors output by the preset text encoder; The inference module is used to construct structured prompt information based on the connection time-series classification candidate decoding results, the target keyword set, and the metadata information, and input the structured prompt information and the speech embedding vector into a preset large language model for autoregressive inference to obtain the initial recognition text. The repair module is used to perform illusion repair processing on the initial recognized text based on acoustic consistency to obtain the final recognized text.

[0014] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the speech recognition method as described above.

[0015] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the speech recognition method as described above.

[0016] The present invention also provides a computer program product, including a computer program that, when executed by a processor, implements the speech recognition method as described above.

[0017] This invention provides a speech recognition method, apparatus, electronic device, and storage medium. By performing parallel decoding and feature mapping on speech coding features, speech-side information simultaneously participates in subsequent processing in both decoded text and speech embedding vector forms. The speech embedding vector is mapped to a preset text feature space consistent with a keyword embedding database. Within this shared space, multi-level retrieval and filtering are performed from the keyword embedding database based on the speech embedding vector to accurately locate a set of target keywords strongly correlated with the speech to be recognized and their corresponding metadata information. Furthermore, a system is constructed using the connection-time classification candidate decoding results, the target keyword set, and the metadata information. Structured prompts, combined with the speech embedding vectors, are input into a pre-defined large language model for autoregressive inference. Guided by rich prior information, the large language model outputs initial recognition text that better reflects real semantics. Finally, the initial recognition text is constrained and corrected through hallucination restoration processing based on acoustic consistency. This effectively overcomes the recognition bottleneck of traditional speech recognition systems in large-scale keyword scenarios due to limitations in static vocabulary and insufficient dynamic adaptation to context. It significantly improves the accuracy of large-scale keyword speech recognition and its adaptability to complex contexts, and gives the entire system good scalability to meet the needs of industrial-grade large-scale keyword recognition. Attached Figure Description

[0018] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0019] Figure 1 This is a flowchart illustrating the speech recognition method provided in an embodiment of the present invention; Figure 2 This diagram illustrates the overall structure of the speech recognition system provided in an embodiment of the present invention. Figure 3 This is a schematic diagram of the multi-level retrieval and filtering mechanism provided in an embodiment of the present invention; Figure 4 This is a schematic diagram of the training framework provided by the present invention; Figure 5 This is a schematic diagram of the hallucination repair mechanism provided in an embodiment of the present invention; Figure 6 This is a schematic diagram of the keyword embedding database construction process provided in an embodiment of the present invention; Figure 7 This is a schematic diagram of the structure of the speech recognition device provided by the present invention; Figure 8This is a schematic diagram of the structure of the electronic device provided by the present invention. Detailed Implementation

[0020] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0021] Figure 1 This is a flowchart illustrating the speech recognition method provided in an embodiment of the present invention, as shown below. Figure 1 As shown, it includes: Step S101: Obtain the speech data to be recognized, and perform acoustic feature extraction and encoding processing on the speech data to be recognized to obtain speech coding features.

[0022] In this application, the voice data to be recognized can be audio data collected in real time by an electronic device through a microphone or other sound pickup device, offline audio files read by the electronic device from local storage or a remote server, or audio data streams received by the electronic device from a third-party device. This voice data to be recognized covers audio from various scenarios, including meeting recordings, call recordings, smart assistant interaction voices, smart home command voices, in-vehicle voice commands, and multilingual / multi-dialect voices.

[0023] Acoustic feature extraction and encoding refers to the process of first extracting low-level acoustic parameters from the speech data to be recognized, and then generating a high-dimensional feature vector sequence containing temporal context information through an encoding network with temporal modeling capabilities. The speech coding features obtained in this step are a temporal feature sequence, with each time step corresponding to a high-dimensional vector that carries the acoustic information of each time frame in the speech, serving as the common input for subsequent processing of multiple branches.

[0024] Step S102: Perform parallel decoding and feature mapping on the speech coding features to obtain the connection time-series classification candidate decoding results and speech embedding vectors; wherein, the speech embedding vectors are mapped to a preset text feature space, which is the feature space of the keyword text embedding vectors output by the preset text encoder.

[0025] Parallel decoding refers to a probabilistic modeling approach based on connection-time classification, which jointly solves the output probability distribution of speech coding features at each time step to quickly generate multiple candidate recognition results in parallel.

[0026] The connection-time classification candidate decoding result can contain N text candidates (which can be denoted as CTC N-best, where N is a positive integer greater than or equal to 1), and each candidate text corresponds to the possible character sequence output of the current speech data to be recognized.

[0027] To further illustrate the overall synergistic relationship between the parallel decoding and feature mapping processing described above Figure 2 A schematic diagram of the overall structure of the speech recognition system provided in an embodiment of the present invention is shown. Figure 2 As shown, the system mainly consists of a voice input module, a voice encoder, a Connectionist Temporal Classification (CTC) classifier, an adapter, a keyword embedding database, a multi-level matching and filtering module, a structured prompting construction module, a Large Language Model (LLM), an illusion repair module, and a keyword weight dynamic update module.

[0028] The speech coding features output by the speech encoder are sent out in two separate paths: one path is sent to the CTC classifier to perform parallel decoding and obtain the connection-time classification candidate decoding result; the other path is sent to the adapter to perform feature mapping and obtain the speech embedding vector.

[0029] by Figure 2 Taking the system shown as an example, for a piece of speech data to be recognized containing the content "Where is Poyang Lake? What is Jielong-3?", the electronic device can obtain the following candidate texts through parallel decoding: Candidate Text 1: Where is Poyang Lake? What is Relay No. 3? Candidate Text 2: Where is Poyang Lake? What is Relay No. 3? Candidate Text 3: Where is Poyang Lake? What is Relay No. 3? Candidate Text 4: Where is Poyang Lake? What is Relay No. 3?

[0030] Feature mapping processing refers to the process of performing cross-modal transformation on speech coding features through an adapter. The adapter can be implemented using any combination of one or more structures such as multilayer perceptron, projection layer, Transformer layer, and convolutional layer, and the embodiments of the present invention do not impose any restrictions on this.

[0031] The speech embedding vector output after adapter processing is mapped to the same vector space defined by the preset text encoder, so that the speech embedding vector and the keyword text embedding vector in the vector space are consistent in dimension and similar in distribution, and can be directly used for vector similarity calculation.

[0032] The pre-defined text encoder is an independently trained text feature extraction network that can be implemented using network structures with strong semantic modeling capabilities, such as Transformer and Conformer. Its function is to convert keyword string sequences into fixed-dimensional keyword text embedding vectors and define a vector space shared by the keyword embedding database.

[0033] It should be noted that the parallel decoding and feature mapping processing branches can be performed simultaneously based on the same speech coding features, so that candidate decoding and cross-modal vector generation do not block each other in the execution process, thus improving the overall inference efficiency.

[0034] Step S103: Based on the speech embedding vector, perform multi-level retrieval and filtering from the pre-constructed keyword embedding database to obtain the target keyword set and the metadata information corresponding to the target keyword set; wherein, the keyword embedding database stores the keyword text embedding vectors output by the preset text encoder.

[0035] A keyword embedding database is a pre-built offline storage structure designed to support efficient retrieval of tens of millions of keywords. It can be organized using a key-value pair structure, where the keyword string is the key, and the keyword text embedding vector and its metadata information are stored as the value. The metadata information may include one or more of the following auxiliary information: keyword aliases, keyword context, keyword domain, and keyword summary description.

[0036] Multi-level retrieval and filtering refers to the process of using speech embedding vectors and other auxiliary information to narrow down the candidate range from a massive number of keywords in the keyword embedding database through multiple dimensions, and finally filtering out a set of target keywords that are highly relevant to the speech content to be identified.

[0037] The target keyword set obtained in this step typically contains k keywords (k is a preset positive integer that can be configured according to the actual scenario), along with the metadata information of each keyword stored in the database.

[0038] Taking the aforementioned speech to be recognized as an example, the electronic device can filter out several target keywords that are highly relevant to the current speech content from a database of tens of millions of keywords through multi-level retrieval and filtering, such as "Jielong No. 3" and "Poyang Lake".

[0039] Step S104: Based on the candidate decoding results of connection-time classification, the target keyword set, and metadata information, construct structured prompt information, and input the structured prompt information and speech embedding vector into a preset large language model for autoregressive inference to obtain the initial recognition text.

[0040] In this application, structured prompt information refers to prompt data formed by organizing multi-source information in an orderly manner according to a preset field format. It can be carried using any structured data format such as JavaScript Object Notation (JSON), Extensible Markup Language (XML), or a custom template string. The structured prompt information includes at least the connection time-series classification candidate decoding results, the target keyword set, and metadata information for each keyword in the target keyword set.

[0041] The pre-defined large language model can be implemented using any autoregressive generative model that supports multimodal input, such as a causal language model based on the Transformer decoder architecture. The large language model receives two inputs simultaneously: one is structured prompt information input in the form of text tokens, and the other is speech embedding vector input in the form of continuous vectors. The model predicts the output token by token in an autoregressive manner, and finally obtains the initial recognition text corresponding to the current speech data to be recognized.

[0042] For example, the large language model comprehensively considers the various possible spellings (Poyang Lake / Poyang Lake / Poyang Lake / Poyang Lake, Jielong / Jielong / Jielong) in the CTC candidate text, while also referring to the retrieved target keywords "Poyang Lake" and "Jielong No. 3", and combines the pronunciation information contained in the speech embedding vector itself to perform joint reasoning, and finally outputs the initial recognition text "Where is Poyang Lake? What is Jielong No. 3?"

[0043] In an optional embodiment, the electronic device may use JSON format as a structured text format to organize multi-source information according to predefined fields. Within the JSON object, a CTC Nbest field can be set to carry the candidate decoding results for connection time-series classification, a retrieved_keywords field can be set to carry the target keyword set and corresponding metadata information, and a history field can be set to carry the user's historical dialogue information.

[0044] User history dialogue information refers to the dialogue content of several rounds that have occurred in the current session and are earlier in time than the current voice data to be recognized. It can be read by the electronic device from the dialogue management module or the session context cache.

[0045] Using the previous example, the structured prompts generated by electronic devices can adopt the following JSON structure: { “CTC Nbest”:[ "Where is Poyang Lake? What is Jielong No. 3?" "Where is Poyang Lake? What is Jielong No.3", "Where is Poyang Lake? What is Jielong No.3", "Where is Poyang Lake? What is Jielong No.3" , "retrieved_keywords": {"keyword": "Smart Dragon-3", "similarity_score":0.96, "global_weight":0.99, "alias": "SD-3", "domain": "spacecraft", "abstract": "A four-stage solid launch rocket developed under the overall leadership of the China Academy of Launch Vehicle Technology, which is one of the Smart Dragon series solid launch vehicle products launched in 2019 by China Long March Rocket Co., Ltd. affiliated to the 1st Academy of China Aerospace Science and Technology Corporation for the commercial space launch market"}, {"keyword": "Poyang Lake", "similarity_score":0.90, "global_weight":0.92,"alias": "Li Ze, Peng Ze, Guanting Lake, Yang Lan, Danshi Lake", "domain": "scenic spot", "abstract": "Located in the northern part of Jiangxi Province, spanning the three cities of Jiujiang, Nanchang and Shangrao, it is the largest freshwater lake in China and the second largest lake in China, second only to Qinghai Lake. It is one of the main tributaries in the middle and lower reaches of the Yangtze River, and is also an important water-passing, throughput-type, seasonal lake in the Yangtze River basin"} , "history": "I want to go out for a trip recently", "Are there any recommended places", "I like visiting scenic spots and historical sites", "and I am also very interested in aerospace" } In the present application, by organizing multi-source auxiliary information in an orderly manner in a structured text format, and incorporating the user's historical dialogue information into prompt information construction, this embodiment not only ensures the parsing efficiency of the large language model for various types of input information, but also enables the large language model to accurately grasp the current context by explicitly injecting the dialogue context, thereby further improving the contextual consistency and semantic rationality of recognition.

[0046] Step S105: Performing hallucination repair processing on the initially recognized text based on acoustic consistency to obtain the final recognized text.

[0047] ​Hallucination restoration refers to the process of determining and correcting whether there are hallucination phenomena in the initial recognized text that deviate from the actual pronunciation, based on the acoustic information of the speech data to be recognized and the candidate decoding results of the connection time-series classification, from the initial recognized text output by the large language model.

[0048] Acoustic consistency refers to the criterion for hallucination restoration being based on the degree of acoustic agreement between the initial recognized text and the speech data to be recognized. When the initial recognized text passes the acoustic consistency test, it is used as the final recognized text output; otherwise, it reverts to the reliable result in the connection-time classification candidate decoding results as the final recognized text output, thereby avoiding the deviation risk brought about by the free generation of large language models.

[0049] The final recognized text is the speech recognition result output by the electronic device for the speech data to be recognized, which can be further used for downstream tasks such as intelligent question answering, retrieval, dialogue understanding, subtitle generation, and command execution.

[0050] This invention employs parallel decoding and feature mapping of speech coding features, enabling speech-side information to participate in subsequent processing simultaneously in both decoded text and speech embedding vectors. The speech embedding vectors are mapped to a preset text feature space consistent with a keyword embedding database. Within this shared space, multi-level retrieval and filtering are performed from the keyword embedding database based on the speech embedding vectors, accurately identifying a set of target keywords strongly correlated with the speech to be recognized and their corresponding metadata. Then, structured prompts are constructed using the connection-time classification candidate decoding results, the target keyword set, and the metadata information. These prompts are combined with the speech embedding vectors and input into a preset large language model for autoregressive inference. Guided by rich prior information, the large language model outputs initial recognition text that more closely matches the true semantics. Finally, the initial recognition text is constrained and corrected through hallucination repair processing based on acoustic consistency. This effectively overcomes the recognition bottleneck of traditional speech recognition systems in large-scale keyword scenarios due to limitations in static vocabulary and insufficient dynamic adaptation to context. It significantly improves the accuracy of large-scale keyword speech recognition and its adaptability to complex contexts, while also providing the entire system with good scalability to meet the needs of industrial-grade large-scale keyword recognition.

[0051] Optionally, the step of obtaining a target keyword set by performing multi-level retrieval and filtering from a pre-constructed keyword embedding database based on the speech embedding vector includes: A first keyword candidate set is obtained by filtering based on the similarity between the speech embedding vector and each of the keyword text embedding vectors in the keyword embedding database; Based on a preset pronunciation dictionary, the connection time sequence classification candidate decoding results are converted into a reference pronunciation sequence. Based on the reference pronunciation sequence, the first keyword candidate set is subjected to pronunciation matching and filtering to obtain a second keyword candidate set. Based on the preset text encoder, candidate text embedding vectors corresponding to the connection time-series classification candidate decoding results are extracted, and the target keyword set is obtained by filtering based on the semantic similarity between the candidate text embedding vectors and the text features corresponding to the second keyword candidate set.

[0052] In this application, Figure 3 This is a schematic diagram of the multi-level retrieval and filtering mechanism provided in an embodiment of the present invention. For example... Figure 3 As shown, the multi-level search and filtering includes three levels: acoustic dimension filtering, pronunciation dimension filtering, and semantic dimension filtering, which are executed sequentially.

[0053] To facilitate sentence-level retrieval, electronic devices can first perform normalization on the speech embedding vector sequence to obtain a fixed-dimensional sentence-level speech embedding vector. Normalization can employ any method, such as mean normalization, maximum normalization, or weighted average normalization; for example, the arithmetic mean of the vectors at all time points can be used as the sentence-level speech embedding vector.

[0054] This implementation takes average normalization as an example. The keyword-level text embedding vector K_word is obtained by taking the arithmetic mean of the keyword text embedding vector sequence in the sequence dimension. The calculation method is shown in formula (1): (1) in, The length of the keyword string sequence. For the first Keyword text embedding vectors at each time point.

[0055] Sentence-level speech embedding vectors It is obtained by taking the arithmetic mean of the speech embedding vector sequence in the time dimension, and its calculation method is shown in formula (2): (2) Where T is the length of the speech sequence. Let be the speech embedding vector at time i.

[0056] Obtain sentence-level speech embedding vectors With keyword-level text embedding vectors Next, the electronic device further calculates the similarity between the two. Similarity calculation can employ any method such as cosine similarity, Euclidean distance, or dot product similarity. This implementation uses dot product similarity as an example: Let the matrix formed by all keyword-level text embedding vectors in the keyword embedding database be denoted as... ], where n is the total number of keywords in the keyword embedding database; then the similarity calculation between the sentence-level speech embedding vector and all keyword-level text embedding vectors is shown in formula (3): (3) Where d is the dimension of the embedding vector, and the denominator is... This is used to scale the dot product results to prevent the softmax function from saturating due to excessively large dot product values ​​in high-dimensional space.

[0057] During text encoder training, k substrings are randomly selected from the text annotations corresponding to each speech training sample, serving as its k training keywords. Simultaneously, m substrings are randomly selected from each of the other training samples in the same training batch, serving as its m negative keywords. Therefore, the total number of keywords for a training sample is k + batch * m. These keywords all serve as input to the text encoder, outputting the corresponding text embedding vector. The training loss function of the text encoder is the cross-entropy loss function between the k true keywords and the calculated similarity between the k keywords.

[0058] Similarly, the same normalization method can be used to obtain keyword-level text embedding vectors from keyword embedding databases.

[0059] Similarity can be calculated using any method such as cosine similarity, Euclidean distance, or dot product similarity. For example, dot product similarity normalized by a softmax function can be used as the similarity score.

[0060] Electronic devices select the k1 keywords with the highest similarity from the keyword embedding database in descending order of similarity as the first keyword candidate set, where k1 is a preset positive integer (e.g., k1 can be 100, 500, 1000, etc.).

[0061] Taking the aforementioned speech to be recognized as an example, electronic devices can initially recall several acoustically related candidate keywords from tens of millions of keywords through acoustic dimension screening, including: Jielong No. 3, Poyang Lake, chain game, Borneo, etc.

[0062] In this application, the preset pronunciation dictionary is a mapping resource from character sequences to phoneme sequences. For example, in Chinese, pronunciation can be represented by pinyin and tones, while in English, pronunciation can be represented by the International Phonetic Alphabet or the Carnegie Mellon University Pronouncing Dictionary (CMUdict).

[0063] The electronic device iterates through each candidate text in the connection sequence classification candidate decoding results, converting its string into a corresponding reference pronunciation sequence. Based on this reference pronunciation sequence, the electronic device performs pronunciation matching and filtering on the first keyword candidate set, outputting a second keyword candidate set, the number of keywords contained therein denoted as k2, where k2 is less than k1.

[0064] The electronic device inputs candidate text from the temporal classification candidate decoding results into a preset text encoder to obtain an N-best text embedding vector. Simultaneously, it inputs each keyword from the second keyword candidate set into the preset text encoder to obtain a corresponding keyword embedding vector. The electronic device calculates the semantic similarity between each keyword embedding vector and the N-best text embedding vector, and selects the k keywords with the highest semantic similarity as the target keyword set, where k is a positive integer not greater than k².

[0065] It should be added that, Figure 4 This is a schematic diagram of the training framework provided by the present invention, such as... Figure 4 As shown, the preset text encoder can adopt the following during the training phase: Figure 4 The training framework shown is used for joint training.

[0066] Specifically, for each speech training sample's corresponding text annotation, k substrings are randomly selected as positive keywords. Simultaneously, from the text annotations of other training samples in the same training batch, m substrings are randomly selected as negative keywords for each sample. All k+batch×m keywords are fed into the text encoder to obtain the corresponding keyword text embedding vectors. At the same time, the speech training samples are processed by the speech encoder and adapter to obtain their corresponding speech embedding vectors. The training loss function of the text encoder is the cross-entropy loss between the true label distribution of positive keywords and the calculated keyword similarity distribution. Through this training method, the text encoder and adapter jointly learn the alignment relationship between speech and text in the shared feature space.

[0067] This implementation method employs a progressive screening mechanism across three dimensions—acoustics, pronunciation, and semantics—to gradually converge the keyword retrieval process from coarse-grained to fine-grained. This ensures the efficiency of quickly locating relevant candidates from a database of tens of millions of keywords, while effectively suppressing the homophonic false detections and semantic drift issues caused by pure vector retrieval through pronunciation and semantic verification. This results in a final set of target keywords that is highly relevant to the content of the speech to be recognized.

[0068] Optionally, based on the reference pronunciation sequence, the first keyword candidate set is filtered by pronunciation matching to obtain a second keyword candidate set, including: Based on the preset pronunciation dictionary, each keyword in the first keyword candidate set is converted into a corresponding keyword pronunciation sequence; For example, for the keyword "Jielong No. 3" in the first keyword candidate set, the electronic device obtains its keyword pronunciation sequence as "jie2 long2 san1 hao4" through a pronunciation dictionary; for the keyword "Poyang Lake", the corresponding keyword pronunciation sequence is "po2 yang2 hu2".

[0069] The keyword pronunciation sequence is matched and compared with the reference pronunciation sequence. Keywords that meet the conditions of complete pronunciation matching in the reference pronunciation sequence and minimum pitch difference are retained to obtain the second keyword candidate set.

[0070] The matching rules can include: if the keyword pronunciation sequence is a complete match of the reference pronunciation sequence (the pinyin syllable sequence after removing the tone is completely identical), then it is considered a match; If the keyword pronunciation sequence matches the reference pronunciation sequence at the pinyin syllable level, but there are some differences in tone, it is still considered a match, and the number of syllables with different tones is recorded.

[0071] Using the previous example, the candidate text "What is Jielong No. 3?" is converted by a pronunciation dictionary to "jie1long2 san1 hao4 shi4 shen2 me5". Among them, "jie1" is the same as "jie2" in the keyword "Jielong No. 3" in pinyin, only the tone is different. Therefore, it is considered a match and the tone difference is recorded as 1.

[0072] Electronic devices comprehensively analyze the number of tonal differences of keywords, sort the keywords according to the priority strategy of "perfect pronunciation match and minimum tonal difference", and retain the top-ranked keywords as a second keyword candidate set.

[0073] For example, the two candidate keywords "Jielong No. 3" and "Poyang Lake" can be successfully included in the second keyword candidate set because their pronunciation sequences can be found in multiple reference pronunciation sequences and the pitch difference is small. However, candidate keywords such as "chain game" that are only acoustically similar but do not match the pronunciation completely will be eliminated in this stage.

[0074] In this application, by simultaneously considering two conditions—perfect pronunciation matching and minimum pitch difference—this implementation achieves a good balance between strict matching and loose matching. This ensures that even if there are slight pitch misidentifications in the candidate decoding results of the connection-time classification, the corresponding keywords can still be correctly recalled, significantly improving the robustness of pronunciation dimension screening.

[0075] Optionally, the initial recognized text undergoes hallucination restoration processing based on acoustic consistency to obtain the final recognized text, including: Based on forced alignment processing, the acoustic matching score between the initial recognition text and the speech data to be recognized is calculated; The initial recognized text is converted into a first pronunciation sequence, and each recognition result in the connection time-series classification candidate decoding result is converted into a second pronunciation sequence; Calculate the average pronunciation edit distance between the first pronunciation sequence and each of the second pronunciation sequences; Based on the acoustic matching score and the average pronunciation edit distance, the confidence score of the initial recognized text is calculated; When the confidence score is greater than a preset confidence threshold, the initial identified text is used as the final identified text. When the confidence score is not greater than the preset confidence threshold, the best result among the candidate decoding results of the connection time series classification is taken as the final recognized text.

[0076] In this application, Figure 5 This is a schematic diagram of the hallucination repair mechanism provided in an embodiment of the present invention. Figure 5 As shown, the hallucination restoration mechanism takes the initial recognized text output by the large language model as the entry point, and performs acoustic consistency evaluation through two paths: forced alignment and pronunciation editing distance. Finally, it outputs the final recognized text based on the confidence judgment result.

[0077] In this application, forced alignment refers to the process of forcibly mapping each phoneme to a specific time segment in the audio given an audio segment and its corresponding pronunciation sequence, and calculating the overall matching confidence. It can be implemented using any of the following methods: Viterbi aligner based on Hidden Markov Model (HMM), end-to-end aligner based on deep learning, etc.

[0078] The electronic device first converts the initial recognition text into a corresponding pronunciation sequence using a preset pronunciation dictionary, then inputs the speech data to be recognized and the pronunciation sequence into a forced aligner, and finally obtains an acoustic matching score that reflects the degree of acoustic fit between the two.

[0079] A higher acoustic matching score indicates that the pronunciation sequence corresponding to the initial recognition text can find better temporal alignment in the speech data to be recognized, that is, the better the acoustic consistency between the two.

[0080] The first pronunciation sequence refers to the phoneme sequence obtained by converting the initial recognized text through a preset pronunciation dictionary; the second pronunciation sequence refers to the phoneme sequence obtained by converting each candidate text in the connected temporal classification candidate decoding results through a preset pronunciation dictionary.

[0081] For example, the first pronunciation sequence corresponding to the initial recognized text "Where is Poyang Lake? What is Jielong No. 3?" can be "po2 yang2 hu2 zai4 na3 li3 jie2 long2 san1 hao4 shi4 shen2 me5".

[0082] Meanwhile, several candidate texts in the temporal classification candidate decoding results (such as "Where is Poyang Lake? What is Jielong No. 3?", "Where is Poyang Lake? What is Jielong No. 3?", etc.) were also converted into their corresponding second pronunciation sequences.

[0083] In this application, the pronunciation edit distance is used to characterize the degree of difference between two pronunciation sequences at the phoneme level. The classic Levenshtein distance algorithm can be used to calculate the minimum number of edit operations (including insertion, deletion and replacement operations) between the two sequences.

[0084] The electronic device calculates the pronunciation edit distance for the first pronunciation sequence and each of the second pronunciation sequences, and then takes the arithmetic mean of all pronunciation edit distances to obtain the average pronunciation edit distance. The smaller the average pronunciation edit distance, the more consistent the pronunciation of the initial recognized text is with the overall decoding results of the connection temporal classification candidate, that is, the lower the possibility that the output of the large language model deviates from the actual pronunciation.

[0085] The confidence score was positively correlated with the acoustic matching score and negatively correlated with the average pronunciation edit distance.

[0086] The specific calculation method can be a weighted difference approach. The acoustic matching scores are weighted by a weight α, and then the average pronunciation edit distance weighted by a weight β is subtracted to obtain the final confidence score. The calculation formula is as follows: (4) Here, the weights α and β are pre-calibrated positive parameters that can be obtained by optimization based on the validation set (e.g., α is 0.7 and β is 0.3). By jointly measuring the alignment quality at the acoustic level and the consistency at the pronunciation level, the confidence score can simultaneously reflect the degree of agreement between the initial recognized text and the original speech, as well as with the candidate decoding results.

[0087] When the confidence score is greater than the preset confidence threshold, the electronic device will use the initial recognized text as the final recognized text; when the confidence score is not greater than the preset confidence threshold, the electronic device will use the best result among the candidate decoding results of the connection time-series classification as the final recognized text.

[0088] The preset confidence threshold is a critical value pre-set based on the business scenario, which can be flexibly adjusted according to the actual system's trade-off between recognition accuracy and recall. The best result among the candidate decoding results of connection-series classification is generally taken as CTC 1-best, that is, the candidate text with the highest confidence.

[0089] Using the previous example, if the initial recognized text output by the large language model, "Where is Poyang Lake? What is the Jielong 3?", has a confidence score higher than the preset confidence threshold after acoustic matching and pronunciation editing distance evaluation, the electronic device will use it as the final recognized text output. Conversely, if the large language model exhibits severe hallucinations and outputs content that differs significantly from the acoustic data to be recognized (e.g., incorrectly outputting: How wide is the Three Gorges of the Yangtze River? What is the Long March 3?), the confidence score will be significantly lower than the preset confidence threshold. The electronic device will then revert to the CTC 1-best result as the final recognized text output, thereby preventing the hallucination content from being directly exposed.

[0090] In this application, the credibility of the initial recognized text is jointly evaluated by two dimensions: forced alignment and pronunciation edit distance. This implementation method effectively identifies the illusory content that may exist in the output of the large language model that is detached from the actual pronunciation, and ensures the reliability of the speech recognition results in terms of factuality and pronunciation consistency through threshold judgment and backoff strategy.

[0091] Optionally, the method further includes: For each keyword embedded in the keyword database, an initial comprehensive weight is set; wherein, the comprehensive weight is calculated based on the popularity weight and the confidence weight. When a matching keyword is detected in the target keyword set, the popularity weight of the matching keyword is increased by a fixed value, and the popularity weight of all keywords embedded in the database is decayed based on the time decay factor. If the hit keyword is contained in the final recognized text, obtain the output probability of the preset large language model when generating the hit keyword, and perform weighted fusion based on the output probability and the confidence weight of the hit keyword to obtain the updated confidence weight of the hit keyword. If the hit keyword is not included in the final identified text, the confidence weight of the hit keyword is attenuated based on a preset attenuation coefficient to obtain the updated confidence weight of the hit keyword. Based on the updated popularity weight and the updated confidence weight of the hit keywords, the overall weight of the hit keywords is updated. Based on the comprehensive weight, data update and cleanup operations are performed on the keyword embedding database.

[0092] In this application, Figure 6 This is a schematic diagram illustrating the keyword embedding database construction process provided in an embodiment of the present invention. For example... Figure 6 As shown, for each keyword to be added to the database, the electronic device inputs the keyword string into a pre-trained preset text encoder to obtain the corresponding keyword text embedding vector; then, the keyword string is used as the key, and the keyword text embedding vector and the keyword's metadata information are used as the value, and stored in the keyword embedding database in the form of key-value pairs; when all keywords have been added to the database, a keyword embedding database that supports the retrieval of tens of millions of keywords is obtained.

[0093] To further achieve adaptive optimization of keyword embedding in the database, this implementation provides the following dynamic update mechanism based on keyword weights.

[0094] Popularity weight is used to characterize the frequency with which keywords are searched and viewed in a recent period; confidence weight is used to characterize the reliability of keywords being actually hit in the inference process of the large language model in each iteration.

[0095] The overall weight can be obtained by multiplying the popularity weight and the confidence weight, or by weighted summation. During the initial construction of the database, the electronic device assigns a default initial value (e.g., both 1.0) to the popularity weight and confidence weight of each keyword, and calculates the initial value of the overall weight accordingly.

[0096] More specifically, overall weighting Popularity weighting can be used. With confidence weight Obtained by multiplication. .

[0097] Hit keywords refer to keywords that have been included in the target keyword set during the most recent multi-level search and filtering process. Whenever a keyword is hit, the electronic device increases the popularity weight of that hit keyword by a preset fixed value δh (for example, δh is 0.1).

[0098] Meanwhile, the electronic device introduces a time decay factor γ (e.g., γ is 0.99) between 0 and 1, and periodically (e.g., once a day or after every fixed inference batch) multiplies the popularity weight of all keywords embedded in the database by this time decay factor. This leads to a gradual decrease in the popularity weight of keywords that have not received much attention for a long time.

[0099] Through the aforementioned two-way adjustment mechanism of hit bonus and periodic decay, the popularity weight of keyword embedding in the database can dynamically reflect the activity level of each keyword in recent use.

[0100] During the autoregressive decoding process, the pre-defined large language model assigns a corresponding output probability (i.e., the probability value corresponding to the selected token in the softmax distribution) to each output token. The electronic device extracts the output probabilities of several tokens corresponding to the hit keyword from the large language model's inference process and takes the average or minimum value as the overall output probability of the hit keyword.

[0101] Weighted fusion can employ an exponential moving average approach: using α as the historical weight retention coefficient and (1-α) as the acceptance coefficient for the current output probability, the historical confidence weights are linearly fused with the current output probability proportionally. For example, if α=0.9, then when the output probability of the large language model when generating the hit keyword is 0.95, the updated confidence weight = 0.9 × historical confidence weight + 0.1 × 0.95. This fusion method preserves the stability of historical confidence while allowing the latest inference results to have a timely impact on the confidence weights.

[0102] The preset attenuation coefficient β is a constant between 0 and 1 (e.g., β is 0.95). The electronic device directly multiplies the confidence weight of the hit keyword by this preset attenuation coefficient, thereby appropriately reducing the credibility of keywords that are not actually used. This attenuation process can prevent acoustically similar but semantically unrelated keywords from occupying high confidence positions for a long time, thus improving the actual usability of hit keywords in subsequent retrieval stages.

[0103] The electronic device uses the product or weighted sum method described above to combine the updated popularity weight with the updated confidence weight to obtain the latest comprehensive weight of the hit keyword and write it back to the keyword embedding database, overwriting the original comprehensive weight value of the keyword.

[0104] When the storage capacity of the keyword embedding database reaches a preset threshold, the electronic device deletes several keywords with lower overall weights from the keyword embedding database, in ascending order of their overall weights. Simultaneously, for newly added keywords, their overall weights are initialized in the aforementioned manner. Furthermore, the overall weight can also be returned to the large language model as the `global_weight` attribute of the keyword field in the structured prompt information, assisting the large language model in differentially weighting different keywords.

[0105] It should be noted that the above dynamic update mechanism can run in parallel with the main recognition process in an asynchronous thread to avoid affecting the real-time recognition delay.

[0106] In this application, by introducing a keyword comprehensive weight update mechanism driven by both popularity weight and confidence weight, this implementation method endows the keyword embedding database with self-learning and metabolism capabilities. This not only continuously strengthens high-frequency and highly credible keywords, but also reasonably eliminates outdated or inaccurate keywords, thereby ensuring the efficiency and availability of the keyword embedding database in long-term operation.

[0107] Optionally, the metadata information includes at least one of the following: alias of the corresponding keyword, scene information, domain information, and summary description.

[0108] In this application, metadata information refers to auxiliary information stored alongside the keyword string and its text embedding vector, used to describe the additional attributes of the keyword. Metadata information may include at least one of the following: alias of the corresponding keyword, context information, domain information, and summary description.

[0109] Aliases refer to alternative expressions or synonyms of keywords. For example, aliases for the keyword "Poyang Lake" could include: Li Ze, Peng Ze, Guan Ting Lake, Yang Lan, Dan Shi Lake, etc.; aliases for the keyword "Jielong 3" could include SD-3, etc. By introducing alias information, the correct keyword entries can still be accurately retrieved even when users use synonyms.

[0110] Contextual information describes the application scenarios in which keywords typically appear, such as meetings, medical consultations, car navigation, smart homes, and financial customer service. Contextual information helps large language models make appropriate choices based on the current context when polysemous words are ambiguous.

[0111] Domain information describes the professional field or topic category to which keywords belong, such as spacecraft, tourist attractions, finance, medicine, law, and the internet. Domain information enables large language models to identify which keywords belong to specialized terms within structured prompts, thereby improving the accuracy of domain-specific speech recognition.

[0112] The abstract description is used to summarize the core definition or background information of the keywords in a concise text format.

[0113] For example, the abstract description corresponding to the keyword "Jielong-3" can be: "A four-stage solid-propellant launch vehicle developed by the China Academy of Launch Vehicle Technology, it is one of the Jielong series solid-propellant launch vehicle products launched by China Long March Rocket Co., Ltd., a subsidiary of the First Academy of China Aerospace Science and Technology Corporation, for the commercial space launch market in 2019." The abstract description corresponding to the keyword Poyang Lake can be as follows: Located in northern Jiangxi Province, it is situated between the cities of Jiujiang, Nanchang, and Shangrao. It is the largest freshwater lake in China and the second largest lake in China, second only to Qinghai Lake.

[0114] The abstract description provides basic background knowledge of the keywords to the large language model, essentially acting as a built-in encyclopedia.

[0115] The aforementioned metadata information can exist individually or in combination. In practical applications, metadata fields can be flexibly configured according to the needs of the business domain: for example, in a medical consultation scenario, fields such as indications and drug specifications can be added; in an in-vehicle navigation scenario, fields such as latitude and longitude and administrative divisions can be added.

[0116] When a large language model performs reasoning based on structured prompts, metadata can provide the large language model with rich prior knowledge beyond the literal meaning of keywords, thereby helping it to make more accurate judgments in complex scenarios such as polysemous words, homophones, and technical terms.

[0117] For example, regarding the aforementioned speech data to be identified, the large language model can clearly distinguish it from the homophones Jielong 3 or Jielong 3 based on the spacecraft domain information and summary description in the metadata information, and output accurate recognition results.

[0118] By attaching multi-dimensional metadata information to each keyword, this implementation method enables the large language model to obtain rich prior knowledge beyond the literal meaning of the keyword, which significantly improves the recognition accuracy in complex scenarios such as polysemous words, homophones, and technical terms.

[0119] Optionally, the step of performing acoustic feature extraction and encoding processing on the speech data to be recognized to obtain speech coding features includes: Acoustic features are extracted from the speech data to be identified; The acoustic features are input into a preset speech encoder for processing, and the speech coding features are output.

[0120] In this application, acoustic features refer to the low-to-medium dimensional time-series parameter sequences obtained after digital signal processing such as filtering, framing, windowing, and transformation of the original audio waveform data. The acoustic features can be any one of the following: perceptual linear prediction (PLP), Mel-frequency cepstral coefficients (MFCC), or filter bank (FBank) features.

[0121] In a specific example, the electronic device uses FBank features as acoustic features. The specific process is as follows: the original audio is segmented into a series of audio frames with a frame length of 25 milliseconds and a frame shift of 10 milliseconds; a short-time fourier transform (STFT) is performed on each frame after adding a Hamming window to obtain the power spectrum corresponding to that frame; the power spectrum is energy integrated through an 80-dimensional Mel filter bank; the logarithm of the filter bank output is taken to finally obtain the 80-dimensional FBank feature vector of each frame; the FBank feature vectors corresponding to all frames are concatenated in time sequence to obtain the FBank feature sequence corresponding to the speech data to be recognized.

[0122] It should be noted that before extracting acoustic features, electronic devices can selectively perform front-end processing operations such as pre-emphasis and preprocessing (e.g., noise reduction, echo cancellation, automatic gain control) on the speech data to be recognized, in order to improve the stability and noise resistance of the acoustic features.

[0123] The preset speech encoder is a deep neural network used for high-dimensional context modeling of acoustic features. It can be any network structure with speech representation capabilities, such as Hidden-Unit Bidirectional Encoder Representations from Transformers (HuBERT), Wave to Vector 2.0 (Wav2Vec 2.0), Whisper encoder, Conformer encoder, etc. The embodiments of the present invention do not limit this.

[0124] The speech encoder receives a sequence of acoustic features as input, performs temporal modeling on them through a multi-layer self-attention mechanism or convolutional structure, and outputs a high-dimensional vector sequence containing rich contextual and pronunciation information, i.e., speech coding features.

[0125] In a specific example, the speech encoder can employ a 12-layer Transformer coding layer structure, with each layer containing a multi-head self-attention sublayer and a feedforward network sublayer, resulting in an output dimension of 1024. After receiving an 80-dimensional FBank feature sequence, the speech encoder outputs a speech-coded feature sequence with a dimension of T×1024, where T is the number of time frames.

[0126] It should be further explained that during the training of the adapter module, a strategy of freezing the parameters of the speech encoder and the large language model and training only the adapter parameters can be adopted. This will preserve the original speech feature extraction capability of the speech encoder and the original language reasoning capability of the large language model to the greatest extent, and avoid the catastrophic forgetting problem caused by end-to-end joint training.

[0127] This training paradigm of freezing the backbone and training the adapter enables the solution described in this invention to be quickly developed and deployed based on existing open-source large models (such as HuBERT, Whisper, LLaMA, etc.), significantly reducing engineering implementation costs.

[0128] In this application, by introducing mature acoustic feature extraction methods and a speech encoder with strong representational capabilities, this implementation provides a high-quality unified feature foundation for subsequent parallel decoding and cross-modal feature mapping, enhancing the versatility and adaptability of the solution from an engineering implementation perspective. Simultaneously, by freezing the backbone network and training only the adapters, this implementation effectively reduces the training cost and data requirements for model transfer, facilitating the rapid reuse of pre-trained models in different business scenarios.

[0129] The speech recognition device provided by the present invention is described below. The speech recognition device described below can be referred to in correspondence with the speech recognition method described above.

[0130] Figure 7 This is a schematic diagram of the speech recognition device provided by the present invention, as shown below. Figure 7 As shown, it includes: The acquisition module 710 is used to acquire speech data to be recognized, and to perform acoustic feature extraction and encoding processing on the speech data to be recognized to obtain speech coding features; The parallel processing module 720 is used to perform parallel decoding and feature mapping processing on the speech coding features to obtain the connection time-series classification candidate decoding results and speech embedding vectors; wherein, the speech embedding vectors are mapped to a preset text feature space, and the preset text feature space is the feature space of the keyword text embedding vectors output by the preset text encoder. The filtering module 730 is used to perform multi-level retrieval and filtering from a pre-constructed keyword embedding database based on the speech embedding vector to obtain a set of target keywords and metadata information corresponding to the set of target keywords; wherein, the keyword embedding database stores the keyword text embedding vectors output by the preset text encoder; The reasoning module 740 is used to construct structured prompt information based on the connection time-series classification candidate decoding results, the target keyword set, and the metadata information, and input the structured prompt information and the speech embedding vector into a preset large language model for autoregressive reasoning to obtain the initial recognition text; The repair module 750 is used to perform an illusion repair process based on acoustic consistency on the initial recognized text to obtain the final recognized text.

[0131] In this application, by performing parallel decoding and feature mapping on speech coding features, speech-side information participates in subsequent processing simultaneously in two forms: decoded text and speech embedding vectors. The speech embedding vectors are mapped to a preset text feature space consistent with the keyword embedding database. Within this shared space, multi-level retrieval and filtering are performed from the keyword embedding database based on the speech embedding vectors to accurately locate the target keyword set and corresponding metadata information strongly related to the speech to be recognized. Then, structured prompt information is constructed using the connection-time classification candidate decoding results, the target keyword set, and the metadata information. This information, combined with the speech embedding vectors, is input into a preset large language model for autoregressive inference. Guided by rich prior information, the large language model outputs initial recognition text that better fits the true semantics. Finally, the initial recognition text is constrained and corrected through hallucination repair processing based on acoustic consistency. This effectively overcomes the recognition bottleneck of traditional speech recognition systems in large-scale keyword scenarios due to limitations in static vocabulary and insufficient dynamic adaptation to context. It significantly improves the accuracy of large-scale keyword speech recognition and its adaptability to complex contexts, and gives the entire system good scalability to meet the needs of industrial-grade large-scale keyword recognition.

[0132] Figure 8 This is a schematic diagram of the structure of the electronic device provided by the present invention, such as... Figure 8As shown, the electronic device may include a processor 810, a communications interface 820, a memory 830, and a communication bus 840, wherein the processor 810, the communications interface 820, and the memory 830 communicate with each other via the communication bus 840. The processor 810 can call logical instructions in the memory 830 to execute a speech recognition method, which includes: acquiring speech data to be recognized, performing acoustic feature extraction and encoding processing on the speech data to be recognized, and obtaining speech coding features; The speech coding features are subjected to parallel decoding and feature mapping processes to obtain the connection time-series classification candidate decoding results and speech embedding vectors; wherein, the speech embedding vectors are mapped to a preset text feature space, which is the feature space of the keyword text embedding vectors output by the preset text encoder; Based on the speech embedding vector, a multi-level search and filtering process is performed from a pre-constructed keyword embedding database to obtain a set of target keywords and metadata information corresponding to the set of target keywords; wherein, the keyword embedding database stores the keyword text embedding vectors output by the preset text encoder; Based on the connection time-series classification candidate decoding results, the target keyword set, and the metadata information, structured prompt information is constructed, and the structured prompt information and the speech embedding vector are jointly input into a preset large language model for autoregressive inference to obtain the initial recognition text. The initial identified text is subjected to an illusion restoration process based on acoustic consistency to obtain the final identified text.

[0133] Furthermore, the logical instructions in the aforementioned memory 830 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0134] On the other hand, the present invention also provides a computer program product, the computer program product including a computer program, the computer program being able to be stored on a non-transitory computer-readable storage medium, and when the computer program is executed by a processor, the computer is able to execute the speech recognition method provided by the above methods, the method including: acquiring speech data to be recognized, performing acoustic feature extraction and encoding processing on the speech data to be recognized, and obtaining speech encoding features; The speech coding features are subjected to parallel decoding and feature mapping processes to obtain the connection time-series classification candidate decoding results and speech embedding vectors; wherein, the speech embedding vectors are mapped to a preset text feature space, which is the feature space of the keyword text embedding vectors output by the preset text encoder; Based on the speech embedding vector, a multi-level search and filtering process is performed from a pre-constructed keyword embedding database to obtain a set of target keywords and metadata information corresponding to the set of target keywords; wherein, the keyword embedding database stores the keyword text embedding vectors output by the preset text encoder; Based on the connection time-series classification candidate decoding results, the target keyword set, and the metadata information, structured prompt information is constructed, and the structured prompt information and the speech embedding vector are jointly input into a preset large language model for autoregressive inference to obtain the initial recognition text. The initial identified text is subjected to an illusion restoration process based on acoustic consistency to obtain the final identified text.

[0135] In another aspect, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the speech recognition method provided by the above methods, the method comprising: acquiring speech data to be recognized, performing acoustic feature extraction and encoding processing on the speech data to be recognized, and obtaining speech encoding features; The speech coding features are subjected to parallel decoding and feature mapping processes to obtain the connection time-series classification candidate decoding results and speech embedding vectors; wherein, the speech embedding vectors are mapped to a preset text feature space, which is the feature space of the keyword text embedding vectors output by the preset text encoder; Based on the speech embedding vector, a multi-level search and filtering process is performed from a pre-constructed keyword embedding database to obtain a set of target keywords and metadata information corresponding to the set of target keywords; wherein, the keyword embedding database stores the keyword text embedding vectors output by the preset text encoder; Based on the connection time-series classification candidate decoding results, the target keyword set, and the metadata information, structured prompt information is constructed, and the structured prompt information and the speech embedding vector are jointly input into a preset large language model for autoregressive inference to obtain the initial recognition text. The initial identified text is subjected to an illusion restoration process based on acoustic consistency to obtain the final identified text.

[0136] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0137] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0138] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A speech recognition method, characterized in that, include: Acquire speech data to be recognized, and perform acoustic feature extraction and encoding processing on the speech data to be recognized to obtain speech coding features; The speech coding features are subjected to parallel decoding and feature mapping processes to obtain the connection time-series classification candidate decoding results and speech embedding vectors; wherein, the speech embedding vectors are mapped to a preset text feature space, which is the feature space of the keyword text embedding vectors output by the preset text encoder; Based on the speech embedding vector, a multi-level search and filtering process is performed from a pre-constructed keyword embedding database to obtain a set of target keywords and metadata information corresponding to the set of target keywords; wherein, the keyword embedding database stores the keyword text embedding vectors output by the preset text encoder; Based on the connection time-series classification candidate decoding results, the target keyword set, and the metadata information, structured prompt information is constructed, and the structured prompt information and the speech embedding vector are jointly input into a preset large language model for autoregressive inference to obtain the initial recognition text. The initial identified text is subjected to an illusion restoration process based on acoustic consistency to obtain the final identified text.

2. The speech recognition method according to claim 1, characterized in that, The steps for obtaining a set of target keywords by performing multi-level retrieval and filtering from a pre-constructed keyword embedding database based on the speech embedding vector include: A first keyword candidate set is obtained by filtering based on the similarity between the speech embedding vector and each of the keyword text embedding vectors in the keyword embedding database; Based on a preset pronunciation dictionary, the connection time sequence classification candidate decoding results are converted into a reference pronunciation sequence. Based on the reference pronunciation sequence, the first keyword candidate set is subjected to pronunciation matching and filtering to obtain a second keyword candidate set. Based on the preset text encoder, candidate text embedding vectors corresponding to the connection time-series classification candidate decoding results are extracted, and the target keyword set is obtained by filtering based on the semantic similarity between the candidate text embedding vectors and the text features corresponding to the second keyword candidate set.

3. The speech recognition method according to claim 2, characterized in that, Based on the reference pronunciation sequence, the first keyword candidate set is filtered by pronunciation matching to obtain a second keyword candidate set, including: Based on the preset pronunciation dictionary, each keyword in the first keyword candidate set is converted into a corresponding keyword pronunciation sequence; The keyword pronunciation sequence is matched and compared with the reference pronunciation sequence. Keywords that meet the conditions of complete pronunciation matching in the reference pronunciation sequence and minimum pitch difference are retained to obtain the second keyword candidate set.

4. The speech recognition method according to claim 1, characterized in that, The initial recognized text is subjected to hallucination restoration processing based on acoustic consistency to obtain the final recognized text, including: Based on forced alignment processing, the acoustic matching score between the initial recognition text and the speech data to be recognized is calculated; The initial recognized text is converted into a first pronunciation sequence, and each recognition result in the connection time-series classification candidate decoding result is converted into a second pronunciation sequence; Calculate the average pronunciation edit distance between the first pronunciation sequence and each of the second pronunciation sequences; Based on the acoustic matching score and the average pronunciation edit distance, the confidence score of the initial recognized text is calculated; When the confidence score is greater than a preset confidence threshold, the initial identified text is used as the final identified text. When the confidence score is not greater than the preset confidence threshold, the best result among the candidate decoding results of the connection time series classification is taken as the final recognized text.

5. The speech recognition method according to claim 1, characterized in that, The method further includes: For each keyword embedded in the keyword database, an initial comprehensive weight is set; wherein, the comprehensive weight is calculated based on the popularity weight and the confidence weight. When a matching keyword is detected in the target keyword set, the popularity weight of the matching keyword is increased by a fixed value, and the popularity weight of all keywords embedded in the database is decayed based on the time decay factor. If the hit keyword is contained in the final recognized text, obtain the output probability of the preset large language model when generating the hit keyword, and perform weighted fusion based on the output probability and the confidence weight of the hit keyword to obtain the updated confidence weight of the hit keyword. If the hit keyword is not included in the final identified text, the confidence weight of the hit keyword is attenuated based on a preset attenuation coefficient to obtain the updated confidence weight of the hit keyword. Based on the updated popularity weight and the updated confidence weight of the hit keywords, the overall weight of the hit keywords is updated. Based on the comprehensive weight, data update and cleanup operations are performed on the keyword embedding database.

6. The speech recognition method according to any one of claims 1 to 5, characterized in that, The metadata information includes at least one of the following: alias for the corresponding keyword, context information, domain information, and summary description.

7. The speech recognition method according to claim 1, characterized in that, The process of extracting and encoding acoustic features from the speech data to be recognized to obtain speech coding features includes: Acoustic features are extracted from the speech data to be identified; The acoustic features are input into a preset speech encoder for processing, and the speech coding features are output.

8. A voice recognition device, characterized in that, include: The acquisition module is used to acquire the speech data to be recognized, and to perform acoustic feature extraction and encoding processing on the speech data to be recognized to obtain speech coding features; The parallel processing module is used to perform parallel decoding and feature mapping on the speech coding features to obtain the connection time-series classification candidate decoding results and speech embedding vectors; wherein, the speech embedding vectors are mapped to a preset text feature space, which is the feature space of the keyword text embedding vectors output by the preset text encoder. The filtering module is used to perform multi-level retrieval and filtering from a pre-constructed keyword embedding database based on the speech embedding vector to obtain a set of target keywords and metadata information corresponding to the set of target keywords; wherein, the keyword embedding database stores the keyword text embedding vectors output by the preset text encoder; The inference module is used to construct structured prompt information based on the connection time-series classification candidate decoding results, the target keyword set, and the metadata information, and input the structured prompt information and the speech embedding vector into a preset large language model for autoregressive inference to obtain the initial recognition text. The repair module is used to perform illusion repair processing on the initial recognized text based on acoustic consistency to obtain the final recognized text.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that, When the processor executes the computer program, it implements the speech recognition method as described in any one of claims 1 to 7.

10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the speech recognition method as described in any one of claims 1 to 7.

Images