Language recognition system, method and equipment combined with large language model error correction and medium

Abstract

The invention discloses a language recognition system, method and equipment combined with large language model error correction and a medium. A first branch in the system comprises an encoder and a decoder; the second branch comprises an adapter and a large language model; acquiring acoustic features and a hot word list of the to-be-recognized voice signal; sending the acoustic features and the hot word list to an encoder, obtaining acoustic information high-order characterization and hot word bias features, sending the acoustic information high-order characterization and hot word bias features to a decoder for decoding, and obtaining a plurality of candidate results of a first branch; the adapter aligns and splices the acoustic information high-order representation and the hot word offset feature, outputs an acoustic hot word splicing feature, inputs a plurality of candidate results into the large language model, and outputs and obtains a second branch error correction result; and performing error correction verification on the plurality of candidate results of the first branch and the error correction result of the second branch, if the verification is passed, outputting the error correction result of the second branch, otherwise, returning to the final result of the plurality of candidate results in the first branch, and the identification precision can be improved.

Description

Technical Field

[0001] This invention relates to the field of automatic speech recognition technology, specifically to a speech recognition system, method, device, and medium that incorporates large language model error correction. Background Technology

[0002] Automatic speech recognition (ASR), as a core technology of human-computer interaction, has evolved into a key supporting technology for smart terminals, smart homes, and in-vehicle systems since Bell Labs invented the Audrey speech recognition system in 1952. Speech recognition error correction technology can solve recognition errors caused by accents, noise, and technical terms in ASR systems. It is a crucial link in improving speech recognition accuracy, with applications covering multiple fields such as consumer products and enterprise services. Its core purpose is to make the speech-to-text results more closely match actual usage needs. For example, in consumer smart products, if the smart assistant makes ASR recognition errors due to accents, speech speed, or environmental noise (such as recognizing "turn on the air conditioner" as "turn on the desk lamp") when executing user commands, error correction technology can correct these errors, improving the accuracy of command execution and avoiding misoperation. In enterprise meetings, technical terms (such as "asset management plan" in finance and "neural network" in technology) and cross-accented communication can easily lead to ASR recognition deviations. Error correction technology can correct these errors, generating more accurate meeting minutes and reducing the workload of manual transcription.

[0003] Current mainstream end-to-end encoder-attention-decoder (AED) models have achieved high recognition accuracy in specific scenarios, but performance bottlenecks still exist in complex real-world environments (such as noise interference, technical terms, accents, and dialects). In recent years, large language models (LLMs) have made breakthrough progress in the field of natural language understanding and generation, demonstrating powerful context modeling and semantic error correction capabilities.

[0004] The invention patent with patent application publication number CN113035200A discloses a speech recognition error correction method, device and equipment based on human-computer interaction scenarios. Although the patent involves an error correction method, the error correction method in the patent is to obtain the interactive query type, historical interaction information and recognized text, and does not involve multimodal fusion error correction of a large language model.

[0005] Existing edge-side speech recognition systems generally adopt the AED architecture, which mainly consists of three parts: Encoder component: Receives the time-frequency features of the speech signal, such as Mel-frequency cepstral coefficients (MFCC) and Filter Bank, and maps them into an acoustic hidden feature sequence H through deep neural networks, such as the Conformer model or the Long Short-Term Memory (LSTM) neural network model.

[0006] Attention component: Learns the alignment relationship between acoustic features H and target text subunits (phonemes, characters or words) to achieve acoustic-text modality association modeling.

[0007] Decoder component: Based on the contextual features output by the attention mechanism, it generates the probability distribution of the target text sequence, completing the conversion from acoustic features to text.

[0008] Some existing technologies attempt to introduce language models (LMs) to constrain the decoding process, but most of them use traditional n-gram models or lightweight neural network language models. Due to limitations in model capacity and contextual understanding capabilities, their effectiveness in correcting complex grammatical errors and semantic biases is limited.

[0009] Current hot word recognition technologies employ interpolation between an independent language-like model trained on corpora with sentence structures and a general master language model. During the decoding phase, a reserved decoding entry point in the language-like model leads to a WFST network constructed from hot words, thereby enhancing the recognition of key terms. However, current solutions are based on shallow fusion; if no hot words are found in the decoding beam search, no stimulus can be generated. They rely on sentence structures; otherwise, no stimulus can be produced.

[0010] Currently, the shortcomings of existing technologies are: Limitations in Scene Adaptability: While AED models perform well in standard scenarios covered by training data, their error rate increases significantly when faced with unfamiliar domain terminology, dialect variations, or noisy environments. For example, in smart assistant / smart home scenarios, accents / dialects lead to misinterpretation of commands, and commands fail in noisy environments, causing misoperation and safety hazards (such as accidentally turning on gas-related equipment), resulting in frequent user abandonment and a decline in brand reputation. In voice input / note-taking scenarios, errors involving homophones, rare words, and professional terms (such as "deploy" → "deploy") require extensive manual proofreading of transcribed text, reducing efficiency; frequent errors in real-time subtitles for live streaming / short videos affect the viewing experience and content dissemination.

[0011] Insufficient error correction capability: Traditional edge speech recognition systems lack effective post-processing error correction mechanisms and cannot correct grammatical errors (extra words, missing words, word order reversal) and semantic errors (homophone confusion, semantic deviation) that occur in the recognition results. Resource constraints: High-performance language models are difficult to deploy on the edge due to their large number of parameters, which prevents edge systems from using advanced natural language understanding capabilities to improve recognition quality; Risk of blindly correcting errors: Some solutions that combine external models lack error correction and verification mechanisms, which may introduce new errors and reduce the accuracy of identification.

[0012] Hot word effect: Based on shallow fusion, the incentive effect is poor. Summary of the Invention

[0013] The technical problems to be solved by this invention are: in automatic speech recognition, how to use the semantic understanding ability of a large language model to correct the recognition error of an AED model under the condition of limited resources on the edge; how to design an efficient modality fusion mechanism to achieve effective association between acoustic features and text semantics; and how to construct an error correction and verification mechanism to avoid invalid or incorrect error correction and ensure the reliability of system output.

[0014] To solve the above-mentioned technical problems, the present invention provides the following technical solution: A language recognition system that combines edge-side and large language model error correction includes: a first branch and a second branch; wherein the first branch includes an encoder and a decoder; and the second branch includes an adapter and a large language model. Acquire the acoustic features and hot word list of the speech signal to be recognized; input the acoustic features and hot word list into the encoding unit of the encoder to obtain the high-order representation of acoustic information and hot word bias features; The high-order acoustic information representation and hot word bias features are fed into the decoder for decoding to obtain multiple candidate results of the first branch; The adapter aligns and concatenates the high-order representation of acoustic information and the hot word bias features, and outputs the acoustic hot word concatenation features; it inputs the acoustic hot word concatenation features and multiple candidate results into the large language model, and outputs the second branch error correction results; The error correction results of the first branch and the second branch are verified. If the verification is successful, the error correction result of the second branch is output. Otherwise, the final result of the multiple candidate results in the first branch is returned.

[0015] In this embodiment, the encoder adopts a three-level cascaded architecture, which includes a convolution module, a first coding unit, and a second coding unit in sequence; The convolution module is used to capture local time-frequency features in the speech signal to be recognized, obtain acoustic features, and input them into the first coding unit; The first coding unit uses a deep neural network structure composed of multiple Conformer blocks stacked together to obtain high-order representations of acoustic information; The second encoding unit is a hot word retrieval module, which receives high-order acoustic information representation and a list of hot words, and outputs hot word bias features.

[0016] In this embodiment, the hot word retrieval module includes a text encoder, a phoneme-word relevance calculation module, a left-shift module, and a score aggregation module; The text encoder receives a list of hot words as input and encodes the text of the hot words in the list into hot word word embedding vectors. The phono-word correlation calculation module receives the hot word embedding vector and the high-order acoustic information representation. It applies a parameter-free attention mechanism to directly calculate the correlation between the acoustic features of each audio frame in the high-order acoustic information representation and the hot word embedding vector, and obtains the correlation matrix. The left-shift module receives the hot word embedding vector, performs left-shift processing on the hot word embedding, obtains the Value vector, and adjusts the temporal alignment of the hot word features; The score aggregation module receives the Value vector and the relevance matrix, and uses matrix multiplication to multiply the Value vector and the relevance matrix to obtain the hot word bias features.

[0017] In this embodiment, the final result of multiple candidate results in the first branch is: the result with the highest confidence among multiple candidate results is the final result.

[0018] In this embodiment, error correction verification is performed on multiple candidate results of the first branch and the error correction result of the second branch, including: Text edit distance calculation: Calculate the text edit distance between the final result of the first branch and the error correction result of the second branch. If the text edit distance exceeds the distance threshold percentage, it is judged as overcorrection. Confidence verification: If the confidence of the second branch error correction result is lower than the set recognition threshold, it is judged as a low reliability correction.

[0019] In this embodiment, it is verified that: the text editing distance between the final result of the first branch and the error correction result of the second branch is less than the distance threshold, and the confidence level of the error correction result of the second branch is greater than the recognition threshold.

[0020] In this embodiment, preprocessing is performed before the voice signal to be recognized is input to the encoder, including: The real-time input speech stream is windowed to form a continuous sequence of speech feature windows, which is the speech signal to be recognized. The speech signal to be recognized is converted into a Mel spectrum, and a logarithmic operation is performed on the Mel spectrum to obtain the FilterBank feature matrix, which is then input into the encoder.

[0021] This invention also proposes an error correction method for the language recognition system described above, which combines edge-side and large language model error correction, comprising: Acquire the acoustic features and hot word list of the speech signal to be recognized; input the acoustic features and hot word list into the encoding unit of the encoder to obtain the high-order representation of acoustic information and hot word bias features; The high-order acoustic information representation and hot word bias features are fed into the decoder for decoding to obtain multiple candidate results of the first branch; The adapter aligns and concatenates the high-order representation of acoustic information and the hot word bias features, and outputs the acoustic hot word concatenation features; it inputs the acoustic hot word concatenation features and multiple candidate results into the large language model, and outputs the second branch error correction results; The error correction results of the first branch and the second branch are verified. If the verification is successful, the error correction result of the second branch is output. Otherwise, the final result of the multiple candidate results in the first branch is returned.

[0022] The present invention also provides an electronic device, comprising: an executable instruction stored thereon, wherein when the instruction is executed by a processor, the processor executes the error correction method described above.

[0023] The present invention also provides a computer-readable storage medium, comprising: a processor and a memory; wherein the memory stores a computer program adapted for the processor to load and execute the error correction method described above.

[0024] Compared with the prior art, the beneficial effects of the present invention are: Improved Hot Word Retrieval: The encoder proposed in this invention includes a hot word retrieval module and a hot word feature extraction module. The hot word retrieval module takes a list of hot words (hot word text) and audio acoustic features as input, and outputs multiple candidate hot words related to the pronunciation of the currently decoded audio. The hot word feature extraction module takes multiple candidate hot words and audio acoustic features as input, and outputs hot word bias features. The hot word bias features are input to the encoder decoder branch and the large language model error correction branch to improve hot word retrieval capabilities. In voice assistant scenarios, this invention can significantly reduce command misinterpretation, improve interaction success rate, and enhance personalized experience; in enterprise service scenarios, it can accurately identify industry terms and key entities, reduce manual review costs, strengthen quality inspection / compliance and decision credibility, and simultaneously provide double insurance by being compatible with large model error correction.

[0025] Multimodal fusion enhances recognition accuracy: After receiving hot word candidate features, multiple candidate text results, and acoustic representations, the large language model can comprehensively utilize this information for a holistic judgment. In particular, the addition of acoustic representations provides an extra dimension for semantic error correction, enabling the large language model to more accurately capture subtle changes in speech, thereby improving recognition accuracy. This multimodal fusion approach offers higher accuracy and robustness compared to pure text semantic error correction. Integrating hot word candidates, multiple candidate texts, and acoustic representations into the large language model significantly improves recognition accuracy and robustness, effectively addressing pain points such as near-sound confusion, noise distortion, and missed recall of industry terms / hot words, while simultaneously reducing post-processing costs and compliance risks.

[0026] Error Correction Reliability Assurance: An error correction verification mechanism based on text edit distance and confidence level is introduced to achieve accurate correction of identified errors and intelligent rollback of invalid errors. The core advantage of this mechanism is that through quantitative matching of text edit distance and risk assessment of confidence level, it achieves two-way control of "accurate correction" and "invalid rollback" in error correction. This not only solves the problem of "missing to correct critical errors" in ASR error correction, but also avoids the risk of "over-correction introducing new errors," thereby improving the reliability of error correction results and the stability of engineering implementation. Attached Figure Description

[0027] Figure 1 This is a schematic diagram of a language recognition system that combines edge-side and large language model error correction according to an embodiment of the present invention.

[0028] Figure 2 This is a schematic diagram of the first branch of an embodiment of the present invention.

[0029] Figure 3 This is a schematic diagram of the second branch of an embodiment of the present invention. Detailed Implementation

[0030] To facilitate understanding of the technical solution of the present invention by those skilled in the art, the technical solution of the present invention will now be further described in conjunction with the accompanying drawings.

[0031] The terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Therefore, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of this application, "multiple" means two or more, unless otherwise explicitly specified.

[0032] Please see Figure 1As shown, this invention provides a speech recognition system that combines edge-side and large language model error correction, comprising: a first branch and a second branch. The first branch includes an encoder and a decoder, serving as an encoding / decoding branch. The second branch includes an adapter and a large language model, serving as a speech model error correction branch.

[0033] Acoustic features and a hot word list of the speech signal to be recognized are acquired. These are then fed into the encoder unit to obtain high-order acoustic representations and hot word bias features. The high-order acoustic representations and hot word bias features are then fed into the decoder to obtain multiple candidate results for the first branch. The adapter aligns and concatenates the high-order acoustic representations and hot word bias features, outputting concatenated acoustic hot word features. These features and multiple candidate results are then input into the large language model to obtain the error correction result for the second branch. Error correction verification is performed on the multiple candidate results of the first branch and the error correction result of the second branch. If the verification passes, the error correction result of the second branch is output; otherwise, the process reverts to the final result of the multiple candidate results in the first branch.

[0034] In one embodiment of the present invention, before the speech signal to be recognized is input to the encoder, speech preprocessing is performed, including: Windowing: The real-time input speech stream is windowed to form a continuous sequence of speech feature windows, which is the speech signal to be recognized. Specifically, the window length is set to 25ms and the sliding step size is set to 10ms.

[0035] Feature extraction: Extract FilterBank features for each speech feature window sequence, specifically including: The speech signal to be recognized is converted into a Mel spectrum, specifically with a sampling rate of 16kHz and 80 Mel filters. Logarithmic operations are then performed on the Mel spectrum to obtain a FilterBank feature matrix of dimension 80×T, where T is the number of time frames.

[0036] Endpoint detection uses the VAD (Voice Activity Detection) algorithm to separate valid speech segments from silent segments, and only the valid speech segments are processed.

[0037] Please see Figure 1 , 2 As shown, in one embodiment of the present invention, the encoder adopts a three-level cascaded architecture to build a complete processing link from the original audio to the high-dimensional semantic representation, which includes a convolution module, a first coding unit, and a second coding unit in sequence.

[0038] In this embodiment, the first layer is a convolutional module used to capture local time-frequency features in the speech signal to be recognized, obtain acoustic features, and input them into the first coding unit. Specifically, through the sliding window operation of multiple two-dimensional convolutional kernels in the convolutional module, the system can effectively extract edge features and texture information from the Mel spectrogram, providing high-quality input features for subsequent modeling. This feature extraction method based on local receptive fields not only preserves the physical characteristics of the speech signal but also lays a solid foundation for subsequent global modeling.

[0039] In this embodiment, the first encoding unit follows, employing a deep neural network structure composed of multiple stacked Conformer blocks to acquire high-order representations of acoustic information. The Conformer block, a mainstream architecture in current speech recognition, cleverly combines the advantages of Convolutional Neural Networks (CNNs) and Recurrent Neural Networks (RNNs): CNN layers are responsible for capturing long-range dependencies, while RNN layers focus on temporal modeling. The cascaded design of multiple Conformer blocks forms a deep feature extraction network, capable of abstracting high-level semantic features from speech layer by layer. This progressive feature extraction mechanism allows the model to gradually strip away noise interference and focus on the essential features of language.

[0040] In this embodiment, the second encoding unit is a hot word retrieval module, which receives high-order acoustic information representation and a hot word list, and outputs hot word bias features. Hot words are specific words or phrases that need to be prioritized for recognition in Automatic Speech Recognition (ASR) scenarios. Their core function is to improve the system's recognition accuracy for specialized, obscure, or homophonous words. They are commonly found in customer service terminology, brand names, and personal names. Recognition priority can be adjusted by setting weights, and some scenarios support hot word translation. There are two loading methods: static loading is suitable for scenarios with fixed vocabulary, where hot words are embedded in the language model and remain unchanged throughout; dynamic loading updates hot words in real time, adapting to scenarios where vocabulary frequently changes. For example, in customer service scenarios, product names are preset as static hot words, while in live streaming scenarios, guest names are added in real time as dynamic hot words. Generally, the hot word list comes from the user's "contact list," "app name," "song artist," etc. Specifically, the hot word list is obtained by the system from the user's contact list, the names of applications installed on the device, and the user's historical custom hot words when the speech recognition task begins. There is an upper limit to the number of hot words in the hot word list. Hot words that meet the limit are selected according to the priority of hot words (e.g., user-defined hot words have the highest priority) and the order of hot words input time, and then the hot words are formed into the hot word list.

[0041] In this embodiment, the hot word retrieval module includes a text encoder, a phonetic word relevance calculation module, a left shift module, and a score aggregation module.

[0042] The text encoder receives a list of hot words as input and encodes the text of the hot words in the list into hot word embedding vectors.

[0043] The phoneme-word correlation calculation module receives the hot word character embedding vector and the high-level representation of acoustic information, applies a parameter-free attention mechanism, directly calculates the correlation between the acoustic features of each audio frame in the high-level representation of acoustic information and the hot word character embedding vector, and obtains a correlation matrix. Specifically, the phoneme-word correlation calculation module is used to calculate the correlation between each audio frame and each character of the hot word. As a parameter-free attention mechanism, without training parameters, it directly calculates the correlation between the acoustic features of each audio frame and the hot word character embedding vector. For example, if an audio has 100 frames and the hot word "Shushan" is split into 2 characters, a 100×2 correlation matrix will be generated. Then, through "averaging in the word dimension" (weakening the deviation of a single character) and "taking the maximum value in the frame dimension" (capturing the most matching audio frame signal), the sentence-level hot word correlation score is output.

[0044] The left shift module receives the hot word character embedding vector and is used to perform a left shift operation on the hot word character embedding to obtain a Value vector, adjusting the temporal alignment of the hot word features. Specifically, the left shift module is a parameter-free feature adjustment module that performs a left shift operation on the hot word character embedding to obtain a Value vector. The essence of left shift is to adjust the temporal alignment of the hot word features. Since the temporal sequences of audio frames and text characters may not be completely synchronized, left shift can make the hot word features better match the temporal features of the audio, avoiding feature invalidation caused by temporal misalignment during subsequent matrix operations.

[0045] The score aggregation module receives the Value vector and the correlation matrix and is used to multiply the Value vector and the correlation matrix through matrix multiplication to obtain the hot word bias feature. Specifically, this step combines the "correlation between audio and hot word" and the "hot word's own features" to obtain the hot word bias feature. This feature is equivalent to adding a "weight" to the hot word. After subsequent fusion with the acoustic features, it can enable the decoder to preferentially recognize the hot word. The hot word bias feature is added to and fused with the acoustic features and then sent to the decoder for decoding and the large language model for error correction.

[0046] Please refer to Figure 1 、 3 As shown in, in an embodiment of the present invention, the high-level representation of acoustic information, the hot word bias feature, and multiple candidate results of the first branch are transmitted to the large language model. The adapter aligns the concatenated result of the high-level representation of acoustic information and the hot word bias feature to the semantic space of the large language model. The large language model performs semantic error correction based on the acoustical-hot word concatenated feature aligned by the adapter and multiple candidate results of the first branch, and outputs the final recognition result.

[0047] Please refer to Figures 1 to 3 As shown in, in an embodiment of the present invention, the final result of multiple candidate results in the first branch is: the one with the highest confidence among multiple candidate results is the final result.

[0048] In this embodiment, error correction verification is performed on multiple candidate results of the first branch and the error correction result of the second branch, including: Text edit distance calculation: Calculate the text edit distance between the final result of the first branch and the error correction result of the second branch. If the text edit distance exceeds the distance threshold percentage, it is judged as overcorrection.

[0049] Confidence verification: If the confidence of the second branch error correction result is lower than the set recognition threshold, it is judged as a low reliability correction.

[0050] Decision output: If the verification is passed, that is, the text editing distance between the final result of the first branch and the error correction result of the second branch is less than the distance threshold and the confidence of the error correction result of the second branch is greater than the recognition threshold, then the error correction result of the second branch is output; otherwise, the final result of the first branch is reverted to the first branch.

[0051] In this embodiment, the reliability control and intelligent rollback of the error correction results are achieved by using a dual threshold judgment of calculating the text editing distance and generating confidence level verification.

[0052] In another embodiment, the present invention also provides an error correction method for a language recognition system based on the above-described edge-side combined with large language model error correction, comprising: Acquire the acoustic features and hot word list of the speech signal to be recognized; input the acoustic features and hot word list into the encoding unit of the encoder to obtain the high-order representation of acoustic information and hot word bias features.

[0053] The high-order acoustic information representation and hot word bias features are fed into the decoder for decoding to obtain multiple candidate results for the first branch.

[0054] The adapter aligns and concatenates the high-order acoustic information representation and hot word bias features to output acoustic hot word concatenation features; it then inputs the acoustic hot word concatenation features and multiple candidate results into the large language model to output the second branch error correction results.

[0055] The error correction results of the first branch and the second branch are verified. If the verification is successful, the error correction result of the second branch is output. Otherwise, the final result of the multiple candidate results in the first branch is returned.

[0056] In yet another embodiment, the present invention also provides an electronic device, including: an executable instruction stored thereon, wherein when the instruction is executed by a processor, the processor executes the error correction method described above.

[0057] The processor can be a general-purpose processor, including a Central Processing Unit (CPU), a Network Processor (NP), etc.; it can also be a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), a Field-Programmable Gate Array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. The memory may include Random Access Memory (RAM) and may also include Non-Volatile Memory, such as at least one disk storage device. The memory can also be internal memory of the Random Access Memory (RAM) type. The processor and memory can be integrated into one or more independent circuits or hardware, such as an Application Specific Integrated Circuit (ASIC). It should be noted that when the computer program in the memory is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, an electronic device, or a network device, etc.) to execute all or part of the steps of the methods of the various embodiments of the present invention.

[0058] In yet another embodiment, the present invention also provides a computer-readable storage medium, comprising: a processor and a memory; wherein the memory stores a computer program adapted for the processor to load and execute the error correction method described above.

[0059] Computer-readable storage media can be electronic media, magnetic media, optical media, electromagnetic media, infrared media, or semiconductor systems or propagation media. Computer-readable storage media can also include semiconductor or solid-state memory, magnetic tape, removable computer disks, random access memory (RAM), read-only memory (ROM), hard disks, and optical discs. Optical discs can include optical disc-read-only memory (CD-ROM), optical disc-read / write (CD-RW), and DVDs.

[0060] It will be apparent to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above, and that the invention can be implemented in other specific forms without departing from its spirit or essential characteristics. Therefore, the embodiments should be considered illustrative and non-limiting in all respects, and the scope of the invention is defined by the appended claims rather than the foregoing description. Thus, all variations falling within the meaning and scope of equivalents of the claims are intended to be included within the present invention, and no reference numerals in the claims should be construed as limiting the scope of the claims.

[0061] The above-described embodiments are merely examples of implementation methods of the invention. The scope of protection of the present invention is not limited to the above-described embodiments. For those skilled in the art, several modifications and improvements can be made without departing from the concept of the present invention, and these all fall within the scope of protection of the present invention.

Claims

1. A language recognition system that combines edge-side and large language model error correction, characterized in that, include: The first branch and the second branch; wherein the first branch includes the encoder and the decoder; the second branch includes the adapter and the large language model; Acquire the acoustic features and hot word list of the speech signal to be recognized; input the acoustic features and hot word list into the encoding unit of the encoder to obtain the high-order representation of acoustic information and hot word bias features; The high-order acoustic information representation and hot word bias features are fed into the decoder for decoding to obtain multiple candidate results of the first branch; The adapter aligns and concatenates the high-order representation of acoustic information and the hot word bias features, and outputs the acoustic hot word concatenation features; it inputs the acoustic hot word concatenation features and multiple candidate results into the large language model, and outputs the second branch error correction results; The error correction results of the first branch and the second branch are verified. If the verification is successful, the error correction result of the second branch is output. Otherwise, the final result of the multiple candidate results in the first branch is returned.

2. The language recognition system with end-to-end large language model error correction according to claim 1, characterized in that, The encoder adopts a three-level cascaded architecture, which includes a convolutional module, a first coding unit, and a second coding unit in sequence. The convolution module is used to capture local time-frequency features in the speech signal to be recognized, obtain acoustic features, and input them into the first coding unit; The first coding unit uses a deep neural network structure composed of multiple Conformer blocks stacked together to obtain high-order representations of acoustic information; The second encoding unit is a hot word retrieval module, which receives high-order acoustic information representation and a list of hot words, and outputs hot word bias features.

3. The language recognition system with end-to-end large language model error correction according to claim 2, characterized in that, The hot word retrieval module includes a text encoder, a phoneme-word relevance calculation module, a left-shift module, and a score aggregation module; The text encoder receives a list of hot words as input and encodes the text of the hot words in the list into hot word word embedding vectors. The phono-word correlation calculation module receives the hot word embedding vector and the high-order acoustic information representation. It applies a parameter-free attention mechanism to directly calculate the correlation between the acoustic features of each audio frame in the high-order acoustic information representation and the hot word embedding vector, and obtains the correlation matrix. The left-shift module receives the hot word embedding vector, performs left-shift processing on the hot word embedding, obtains the Value vector, and adjusts the temporal alignment of the hot word features; The score aggregation module receives the Value vector and the relevance matrix, and uses matrix multiplication to multiply the Value vector and the relevance matrix to obtain the hot word bias features.

4. The language recognition system with end-to-end combined large language model error correction according to claim 1, characterized in that, The final result of multiple candidate results in the first branch is: the result with the highest confidence among the multiple candidate results is the final result.

5. The language recognition system with end-to-end large language model error correction according to claim 4, characterized in that, Error correction verification is performed on multiple candidate results of the first branch and the error correction results of the second branch, including: Text edit distance calculation: Calculate the text edit distance between the final result of the first branch and the error correction result of the second branch. If the text edit distance exceeds the distance threshold percentage, it is judged as overcorrection. Confidence verification: If the confidence of the second branch error correction result is lower than the set recognition threshold, it is judged as a low reliability correction.

6. The language recognition system with end-to-end combined large language model error correction according to claim 5, characterized in that, The verification results show that the text editing distance between the final result of the first branch and the error correction result of the second branch is less than the distance threshold, and the confidence level of the error correction result of the second branch is greater than the recognition threshold.

7. The language recognition system with end-to-end combined large language model error correction according to claim 1, characterized in that, Before the speech signal to be recognized is input into the encoder, preprocessing is performed, including: The real-time input speech stream is windowed to form a continuous sequence of speech feature windows, which is the speech signal to be recognized. The speech signal to be recognized is converted into a Mel spectrum, and a logarithmic operation is performed on the Mel spectrum to obtain the FilterBank feature matrix, which is then input into the encoder.

8. An error correction method for a language recognition system combining edge-side and large language model error correction according to any one of claims 1-7, characterized in that, include: Acquire the acoustic features and hot word list of the speech signal to be recognized; input the acoustic features and hot word list into the encoding unit of the encoder to obtain the high-order representation of acoustic information and hot word bias features; The high-order acoustic information representation and hot word bias features are fed into the decoder for decoding to obtain multiple candidate results of the first branch; The adapter aligns and concatenates the high-order representation of acoustic information and the hot word bias features, and outputs the acoustic hot word concatenation features; it inputs the acoustic hot word concatenation features and multiple candidate results into the large language model, and outputs the second branch error correction results; The error correction results of the first branch and the second branch are verified. If the verification is successful, the error correction result of the second branch is output. Otherwise, the final result of the multiple candidate results in the first branch is returned.

9. An electronic device, characterized in that, include: It stores executable instructions, which, when executed by a processor, cause the processor to perform the error correction method as described in claim 8.

10. A computer-readable storage medium, characterized in that, include: A processor and a memory; wherein the memory stores a computer program adapted for the processor to load and execute the error correction method of claim 8.

Images