Speech recognition method, intelligent device and storage medium
By employing a phased training and cross-modal mapping approach for speech recognition, the problem of performance degradation in large language models after parameter reduction is addressed. This approach achieves efficient and accurate speech recognition, making it suitable for real-time interactive scenarios and reducing the performance requirements of large language models.
Patent Information
- Application Number
- CN202610458276.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-04-08
- Publication Date
- 2026-05-15
AI Technical Summary
Existing speech recognition methods based on large language models suffer from performance degradation as the parameter scale decreases, making it difficult to balance the requirements of low latency, low computational cost, and high recognition performance. Furthermore, they are susceptible to hallucination risks and misrecognition problems caused by differences in modal representations.
A phased training approach is adopted, including pre-training, modality alignment training, supervised fine-tuning, and joint fine-tuning. By combining a speech feature extraction module, a speech encoder, a speech adapter, and a large language model, the speech adapter is used for cross-modal mapping. Combined with hot word text and phased training, representation differences are reduced and recognition accuracy is improved.
It achieves efficient and accurate speech recognition performance while reducing the parameter size of large language models, making it suitable for real-time interactive scenarios. It reduces the performance requirements of large language models and improves recognition efficiency and accuracy.
Smart Images
Figure CN122050366A_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of speech recognition technology, specifically providing a speech recognition method, a smart device, and a storage medium. Background Technology
[0002] Automatic Speech Recognition (ASR) is a technology that converts speech signals into corresponding text information. In the field of speech recognition technology, large language models have been gradually introduced to perform speech recognition. However, the existing speech recognition methods based on large language models still have at least the following problems: (1) They are highly sensitive to the parameter scale. When the parameter scale of the large language model is reduced, the performance of speech recognition is likely to decline significantly, making it difficult to meet the requirements of low latency, low computing cost and high recognition performance. (2) Large language models themselves have the risk of illusion. The difference in representation between speech modality and text modality will further aggravate this problem, making it easy for large language models to output erroneous recognition content without acoustic basis under conditions such as noise, silence, and unclear pronunciation, making it difficult to meet the strict stability requirements of industrial scenarios.
[0003] Accordingly, a new technical solution is needed in this field to solve the above problems. Summary of the Invention
[0004] This application aims to solve the above-mentioned technical problems, namely, to solve or at least partially solve the following technical problems: how to achieve speech recognition efficiently and accurately based on a large language model.
[0005] In a first aspect, this application provides a speech recognition method, the method comprising:
[0006] Acquire the speech to be recognized and segment the speech into multiple consecutive speech blocks;
[0007] The multiple speech blocks are sequentially input into the speech recognition system for processing to obtain the text information corresponding to the speech to be recognized.
[0008] The speech recognition system includes a speech feature extraction module, a speech encoder, a speech adapter, and a large language model.
[0009] The speech feature extraction module is configured to extract the acoustic features of the current speech block, which is the speech block currently input to the speech recognition system;
[0010] The speech encoder is configured to encode the acoustic features of the current speech block to obtain a phoneme-level speech representation of the current speech block.
[0011] The speech adapter is configured to map the phoneme-level speech representation of the current speech block to the text embedding space of the large language model to obtain the speech embedding of the current speech block.
[0012] The large language model is configured to process the speech embedding of the current speech block based on prompt information, and generate the text information after the speech embedding processing of all speech blocks is completed. The prompt information includes the speech embeddings of multiple consecutive historical speech blocks preceding the current speech block.
[0013] In one technical solution of the above-mentioned speech recognition method, the prompt information further includes hot word text, which is obtained by the speech recognition system in the following way:
[0014] The phoneme acquisition module within the speech recognition system performs phoneme prediction on the phoneme-level speech representation output by the speech encoder to obtain the phoneme sequence of the current speech block.
[0015] The hot word retrieval module in the speech recognition system retrieves hot word texts that match the phoneme sequence of the current speech block from a preset hot word library. The hot word library is configured with multiple hot word texts and matching relationships between each hot word text and at least one phoneme sequence.
[0016] In one technical solution of the above-mentioned speech recognition method, the speech recognition system is trained through the following phased training method:
[0017] The speech encoder is pre-trained, which includes joint training of the speech encoder and the phoneme acquisition module in the speech recognition system, so that the speech encoder learns the mapping relationship between the acoustic features of speech blocks and phoneme sequences; wherein, the phoneme acquisition module is configured to perform phoneme prediction on the phoneme-level speech representation output by the speech encoder to obtain a phoneme sequence.
[0018] During the pre-training process, when the progress of the pre-training reaches a first preset progress, modal alignment training is started in parallel with the pre-training asynchronously. The modal alignment training includes freezing the speech encoder and the large language model and training the speech adapter.
[0019] After the modality alignment training is completed and the pre-training progress reaches the second preset progress, supervised fine-tuning is initiated in parallel with the pre-training, which is asynchronous. The supervised fine-tuning includes freezing the speech encoder and jointly training the speech adapter and the large language model.
[0020] After the supervised fine-tuning and the pre-training are completed, joint fine-tuning is initiated, which includes joint training of the speech encoder, speech adapter and large language model.
[0021] In one technical solution of the above speech recognition method, during the pre-training process, the speech encoder is saved periodically and each saved speech encoder is used as an encoder checkpoint.
[0022] In the supervised fine-tuning, the joint training of the speech adapter and the large language model includes:
[0023] Based on the encoder checkpoints, the reference encoder used for supervised fine-tuning is obtained; based on the reference encoder, the speech adapter and the large language model are jointly trained.
[0024] In one technical solution of the above speech recognition method, obtaining the reference encoder used for supervised fine-tuning based on the encoder checkpoints includes:
[0025] Obtain the encoder checkpoint saved when the pre-training progress reaches the third preset progress, and use the encoder checkpoint as the reference encoder.
[0026] In one technical solution of the above speech recognition method, during the supervised fine-tuning, the reference encoder is further updated periodically in the following manner:
[0027] Periodically retrieve the latest encoder checkpoint saved during the pre-training process and use that encoder checkpoint as the latest checkpoint.
[0028] The similarity between the first phoneme-level speech representation and the second phoneme-level speech representation is obtained, wherein the first phoneme-level speech representation and the second phoneme-level speech representation are respectively obtained by encoding the same acoustic feature by the latest checkpoint and the reference encoder.
[0029] If the similarity is less than a set threshold, the reference encoder is updated to the latest checkpoint; otherwise, the reference encoder is not updated.
[0030] In one technical solution of the above speech recognition method, the phased training method further includes initiating context-enhanced fine-tuning after the joint fine-tuning is completed;
[0031] The context enhancement fine-tuning includes:
[0032] Obtain training samples, which include speech samples and their corresponding ground truth texts;
[0033] Extract keywords from the true text and obtain interference words that are homophones or near-homophones of the keywords. Add the keywords and interference words to the prompt information corresponding to the training samples.
[0034] The speech recognition system is trained based on the training samples and their corresponding prompts.
[0035] In one technical solution of the above speech recognition method, the phased training method further includes: after the context enhancement fine-tuning is completed, reinforcement learning is performed on the speech recognition system.
[0036] In one technical solution of the above speech recognition method, the phased training method further includes: after the pre-training is completed, fine-tuning the phoneme acquisition module based on a hot word retrieval task; wherein, the hot word retrieval task is:
[0037] The hot word retrieval module in the speech recognition system retrieves hot word texts that match the phoneme sequence output by the phoneme acquisition module from a preset hot word library. The hot word library is configured with multiple hot word texts and matching relationships between each hot word text and at least one phoneme sequence.
[0038] In a second aspect, a smart device is provided, the smart device including at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores a computer program, which, when executed by the at least one processor, implements the method described in any of the technical solutions provided in the first aspect.
[0039] In a third aspect, a computer-readable storage medium is provided, wherein a plurality of program codes are stored therein, the program codes being adapted to be loaded and executed by a processor to perform the method described in any of the technical solutions provided in the first aspect above.
[0040] Solution 1. A speech recognition method, characterized in that the method includes:
[0041] Acquire the speech to be recognized and segment the speech into multiple consecutive speech blocks;
[0042] The multiple speech blocks are sequentially input into the speech recognition system for processing to obtain the text information corresponding to the speech to be recognized.
[0043] The speech recognition system includes a speech feature extraction module, a speech encoder, a speech adapter, and a large language model.
[0044] The speech feature extraction module is configured to extract the acoustic features of the current speech block, which is the speech block currently input to the speech recognition system;
[0045] The speech encoder is configured to encode the acoustic features of the current speech block to obtain a phoneme-level speech representation of the current speech block.
[0046] The speech adapter is configured to map the phoneme-level speech representation of the current speech block to the text embedding space of the large language model to obtain the speech embedding of the current speech block.
[0047] The large language model is configured to process the speech embedding of the current speech block based on prompt information, and generate the text information after the speech embedding processing of all speech blocks is completed. The prompt information includes the speech embeddings of multiple consecutive historical speech blocks preceding the current speech block.
[0048] Solution 2. The method according to Solution 1, characterized in that the prompt information further includes hot word text, which is obtained by the speech recognition system in the following way:
[0049] The phoneme acquisition module within the speech recognition system performs phoneme prediction on the phoneme-level speech representation output by the speech encoder to obtain the phoneme sequence of the current speech block.
[0050] The hot word retrieval module in the speech recognition system retrieves hot word texts that match the phoneme sequence of the current speech block from a preset hot word library. The hot word library is configured with multiple hot word texts and matching relationships between each hot word text and at least one phoneme sequence.
[0051] Solution 3. The method according to Solution 1, characterized in that the speech recognition system is trained through the following phased training method:
[0052] The speech encoder is pre-trained, which includes joint training of the speech encoder and the phoneme acquisition module in the speech recognition system, so that the speech encoder learns the mapping relationship between the acoustic features of speech blocks and phoneme sequences; wherein, the phoneme acquisition module is configured to perform phoneme prediction on the phoneme-level speech representation output by the speech encoder to obtain a phoneme sequence.
[0053] During the pre-training process, when the progress of the pre-training reaches a first preset progress, modal alignment training is started in parallel with the pre-training asynchronously. The modal alignment training includes freezing the speech encoder and the large language model and training the speech adapter.
[0054] After the modality alignment training is completed and the pre-training progress reaches the second preset progress, supervised fine-tuning is initiated in parallel with the pre-training, which is asynchronous. The supervised fine-tuning includes freezing the speech encoder and jointly training the speech adapter and the large language model.
[0055] After the supervised fine-tuning and the pre-training are completed, joint fine-tuning is initiated, which includes joint training of the speech encoder, speech adapter and large language model.
[0056] Option 4. The method according to Option 3, characterized in that,
[0057] During the pre-training process, the speech encoder is saved periodically, and each saved speech encoder is used as an encoder checkpoint.
[0058] In the supervised fine-tuning, the joint training of the speech adapter and the large language model includes:
[0059] Based on the encoder checkpoints, the reference encoder used for supervised fine-tuning is obtained; based on the reference encoder, the speech adapter and the large language model are jointly trained.
[0060] Solution 5. The method according to Solution 4, characterized in that, obtaining the reference encoder used for the supervisory fine-tuning based on the encoder checkpoint includes:
[0061] Obtain the encoder checkpoint saved when the pre-training progress reaches the third preset progress, and use the encoder checkpoint as the reference encoder.
[0062] Solution 6. The method according to Solution 5, characterized in that, in the supervised fine-tuning, the reference encoder is further updated periodically in the following manner:
[0063] Periodically retrieve the latest encoder checkpoint saved during the pre-training process and use that encoder checkpoint as the latest checkpoint.
[0064] The similarity between the first phoneme-level speech representation and the second phoneme-level speech representation is obtained, wherein the first phoneme-level speech representation and the second phoneme-level speech representation are respectively obtained by encoding the same acoustic feature by the latest checkpoint and the reference encoder.
[0065] If the similarity is less than a set threshold, the reference encoder is updated to the latest checkpoint; otherwise, the reference encoder is not updated.
[0066] Scheme 7. The method according to Scheme 3, characterized in that the phased training method further includes initiating context-enhanced fine-tuning after the joint fine-tuning is completed;
[0067] The context enhancement fine-tuning includes:
[0068] Obtain training samples, which include speech samples and their corresponding ground truth texts;
[0069] Extract keywords from the true text and obtain interference words that are homophones or near-homophones of the keywords. Add the keywords and interference words to the prompt information corresponding to the training samples.
[0070] The speech recognition system is trained based on the training samples and their corresponding prompts.
[0071] Option 8. The method according to Option 7, characterized in that,
[0072] The phased training method further includes: after the context enhancement fine-tuning is completed, reinforcement learning is performed on the speech recognition system.
[0073] Solution 9. The method according to Solution 3, characterized in that the phased training method further includes: after the pre-training is completed, fine-tuning the phoneme acquisition module based on a hot word retrieval task; wherein, the hot word retrieval task is:
[0074] The hot word retrieval module in the speech recognition system retrieves hot word texts that match the phoneme sequence output by the phoneme acquisition module from a preset hot word library. The hot word library is configured with multiple hot word texts and matching relationships between each hot word text and at least one phoneme sequence.
[0075] Option 10. A smart device, characterized in that it comprises:
[0076] At least one processor;
[0077] And, a memory communicatively connected to the at least one processor;
[0078] The memory stores a computer program, which, when executed by the at least one processor, implements the speech recognition method according to any one of schemes 1 to 9.
[0079] Scheme 11. A computer-readable storage medium storing a plurality of program codes, characterized in that the program codes are adapted to be loaded and run by a processor to perform the speech recognition method according to any one of Schemes 1 to 9.
[0080] The above-described technical solutions of this application have at least one or more of the following beneficial effects:
[0081] In one technical solution of the speech recognition method provided in this application, the method can acquire speech to be recognized and segment it into multiple consecutive speech blocks, and input the multiple speech blocks sequentially into a speech recognition system for processing (i.e., input each speech block into the speech recognition system according to the chronological order of the speech blocks) to obtain the text information corresponding to the speech to be recognized; wherein, the speech recognition system may include a speech feature extraction module, a speech encoder, a speech adapter, and a large language model. Specifically, the speech feature extraction module may be configured to extract the acoustic features of the current speech block, the current speech block being the speech block currently input into the speech recognition system; the speech encoder may be configured to encode the acoustic features of the current speech block to obtain the phoneme-level speech representation of the current speech block; the speech adapter may be configured to map the phoneme-level speech representation of the current speech block to the text embedding space of the large language model to obtain the speech embedding of the current speech block; the large language model may be configured to process the speech embedding of the current speech block based on prompt information, and generate the text information after the speech embedding processing of all speech blocks is completed, the prompt information may include the speech embeddings of multiple consecutive historical speech blocks preceding the current speech block.
[0082] In the above implementation scheme, each speech block of the speech to be recognized is sequentially input into the speech recognition system for processing. After processing the last speech block, the text information corresponding to the entire speech to be recognized can be obtained. This actually forms a streaming incremental reasoning method. When processing each speech block, the speech embedding of the historical speech block can be directly reused as prompt information, without having to process the historical speech block repeatedly (i.e., reducing redundant calculations). This helps to improve the recognition efficiency of the entire speech to be recognized.
[0083] Furthermore, in the above implementation scheme, the speech recognition system actually achieves cross-modal mapping from the speech modality (i.e., the modality of the speech to be recognized) to the text modality (i.e., the modality of the text information output by the large language model). During the mapping process, a speech adapter is used to explicitly partition the functions of the speech encoder and the large language model. The phoneme-level speech representation acts as an intermediate interface connecting the speech encoder and the large language model. This speech recognition system does not directly couple the speech encoder and the large language model together. Therefore, if the speech encoder and the large language model are jointly trained during the training of the speech recognition system, the representational differences between the speech modality and the text modality can be reduced, thereby suppressing the representational drift of the speech encoder caused by these differences. This allows the trained speech recognition system to obtain accurate text information when processing the speech to be recognized. Representational drift can be understood as the output information of the speech encoder being pulled away from the original acoustic representation space by the large language model. In addition, based on the above structure of the speech recognition system, the performance requirements of the large language model can be reduced. Even with a smaller parameter size of the large language model, accurate speech recognition can still be achieved. Attached Figure Description
[0084] The disclosure of this application will become more readily understood with reference to the accompanying drawings. It will be readily understood by those skilled in the art that these drawings are for illustrative purposes only and are not intended to limit the scope of protection of this application. Wherein:
[0085] Figure 1 This is a schematic diagram of the main steps of a speech recognition method in one embodiment of this application;
[0086] Figure 2 This is a schematic diagram of the main structure of a speech recognition system in one embodiment of this application;
[0087] Figure 3 This is a schematic diagram of the main structure of a speech recognition system in another embodiment of this application;
[0088] Figure 4 This is a schematic diagram of a phased training method in one embodiment of this application;
[0089] Figure 5 This is a schematic diagram comparing the performance of the speech recognition system provided in this application and the speech recognition system of the prior art in one embodiment of this application;
[0090] Figure 6 This is a schematic diagram comparing the performance of the speech recognition system provided in this application and the speech recognition system of the prior art in another embodiment of this application;
[0091] Figure 7This is a schematic diagram of the main structure of a smart device in one embodiment of this application. Detailed Implementation
[0092] Some embodiments of this application are described below with reference to the accompanying drawings. Those skilled in the art should understand that these embodiments are merely illustrative of the technical principles of this application and are not intended to limit the scope of protection of this application.
[0093] The relevant user personal information that may be involved in the various embodiments of this application is processed in strict accordance with the requirements of laws and regulations, following the principles of legality, legitimacy, and necessity, based on the reasonable purpose of the business scenario, and includes personal information that users actively provide or that is generated as a result of using the product / service, as well as personal information obtained with user authorization.
[0094] The personal information processed in this application will vary depending on the specific product / service scenario and will be based on the specific scenario in which the user uses the product / service. This may involve the user's account information, device information, driving information, vehicle information, or other related information. This application will treat the user's personal information and its processing with the utmost diligence.
[0095] This application attaches great importance to the security of users' personal information and has taken reasonable and feasible security protection measures that comply with industry standards to protect users' information and prevent unauthorized access, disclosure, use, modification, damage or loss of personal information.
[0096] First, embodiments of the speech recognition method provided in this application will be described.
[0097] See appendix Figure 1 , Figure 1 This is a schematic flowchart illustrating the main steps of a speech recognition method according to an embodiment of this application. Figure 1 As shown, the speech recognition method in this application embodiment mainly includes the following steps S101 to S102.
[0098] Step S101: Acquire the speech to be recognized and segment it into multiple consecutive speech blocks. This application embodiment does not specifically limit the duration of each speech block; in some embodiments, the duration of a speech block can be 640ms.
[0099] Step S102: Input multiple speech blocks sequentially into the speech recognition system for processing to obtain the text information corresponding to the speech to be recognized (i.e., the speech recognition result).
[0100] See appendix Figure 2 , Figure 2 The main structure of the speech recognition system in an embodiment of this application is illustrated by way of example. Figure 2As shown, a speech recognition system may include a speech feature extraction module, a speech encoder, a speech adapter, and a large language model. The following describes these structures.
[0101] 1. Explanation of the speech feature extraction module.
[0102] In this embodiment, the speech feature extraction module can be configured to extract the acoustic features of the current speech block, which is the speech block currently input to the speech recognition system.
[0103] In some embodiments, the acoustic features of a speech block can be extracted in the following manner: the speech block is divided into multiple speech frames, the first feature information of each speech frame is extracted, and the feature information of each speech frame is subjected to global mean-variance normalization to obtain the second feature information of each speech frame. The acoustic features of the speech block are obtained based on the second feature information of all speech frames.
[0104] Specifically, when dividing a speech block into multiple speech frames, a 25ms window and a 10ms frame shift can be used to divide the speech block into multiple speech frames. That is, a speech frame is extracted every 10ms shift on the speech block, and the duration of the speech frame is 25ms.
[0105] When extracting the first feature information of a speech frame, multiple (e.g., 80) Mel filters can be used to extract the first sub-feature information of the speech frame separately. Then, the first sub-feature information is logarithmically processed to obtain the second sub-feature information. All the second sub-feature information is combined to form the first feature information of the speech frame. If the number of Mel filters is 80, then the first feature information is an 80-dimensional feature vector.
[0106] When performing global mean-variance normalization on the feature information of each speech frame, the mean and variance of the feature information of all speech frames can be calculated, and then the mean and variance can be used to normalize the feature information of each speech frame separately. For example, the result of the normalization process could be... , and These represent the feature information of the speech frames before and after normalization processing, respectively. and Let represent the mean and variance mentioned above, respectively.
[0107] 2. Explanation of the voice encoder.
[0108] In this embodiment, the speech encoder can be configured to encode the acoustic features of the current speech block (i.e., the output information of the speech feature extraction module) to obtain a phoneme-level speech representation of the current speech block. The meaning of the current speech block is the same as that of the current speech block in the aforementioned embodiments.
[0109] In some embodiments, the speech encoder may include convolutional downsampling layers and Conformer layers. Additionally, in some embodiments, the speech encoder may employ a Transformer encoder or other network architectures capable of streaming or non-streaming speech modeling.
[0110] 3. Explanation of the voice adapter.
[0111] In this embodiment, the speech adapter can be configured to map the phoneme-level speech representation of the current speech block (i.e., the output information of the speech encoder) to the text embedding space of a large language model to obtain the speech embedding of the current speech block. The meaning of the current speech block is the same as that of the current speech block in the aforementioned embodiments.
[0112] In some embodiments, the speech adapter includes a downsampling layer and a multi-layer mapping network. The downsampling layer can downsample the phoneme-level speech representation (e.g., by a factor of 4), and the multi-layer mapping network (e.g., a two-layer mapping network) can map the downsampled phoneme-level speech representation to the text embedding space of a large language model. By downsampling the phoneme-level speech representation, the length of the input data for the subsequent large language model can be reduced.
[0113] 4. Explain the large language model.
[0114] In this embodiment, the large language model can be configured to process the speech embedding of the current speech block (i.e., the output information of the speech adapter) based on prompt information, and generate the text information corresponding to the speech to be recognized after processing the speech embedding of all speech blocks. The prompt information may include the speech embeddings of multiple (e.g., four) consecutive historical speech blocks preceding the current speech block. The meaning of the current speech block is the same as that in the previous embodiment. After processing the aforementioned multiple speech blocks, the speech text corresponding to the speech to be recognized (i.e., the final speech recognition result) can be obtained.
[0115] Large language models can employ pre-trained text generation models capable of generating text information based on speech embeddings and prompts. In some embodiments, large language models can also employ other autoregressive language models with text generation capabilities.
[0116] In some embodiments, static prompts, incremental speech embeddings, and dynamic prompts can be organized in a fixed order to form input information, which is then fed into a large language model for processing. "Static prompts" refer to pre-fixed prompt content that does not change with speech blocks during a single recognition process, such as system prompts, task instructions, and fixed template tags. This content remains unchanged during each streaming incremental processing, so it can be pre-cached to reduce redundant computation; it can be understood as the fixed content within the prompts. "Incremental speech embeddings" refer to the current speech block; the results of historical speech blocks are cached. Each time, only the incremental speech embedding needs to be calculated, and combined with the cached data, the complete speech-text content can be obtained.
[0117] The technical effects of the speech recognition method in the embodiments of this application will be described below.
[0118] (1) In this embodiment, each speech block of the speech to be recognized is sequentially input into the speech recognition system for processing. After the last speech block is processed, the text information corresponding to the entire speech to be recognized can be obtained. This actually forms a streaming incremental reasoning method. When processing each speech block, the speech embedding of the historical speech block can be directly reused as prompt information. There is no need to repeatedly process the historical speech block (such as repeated encoding and decoding). This helps to improve the recognition efficiency of the entire speech to be recognized, making the speech recognition system suitable for vehicle cockpits and other real-time interactive scenarios.
[0119] (2) The speech recognition system in this embodiment actually realizes cross-modal mapping from speech modality (i.e., the modality of the speech to be recognized) to text modality (i.e., the modality of the text information output by the large language model). In the mapping process, the functions of the speech encoder and the large language model are explicitly divided by the speech adapter. The phoneme-level speech representation is equivalent to the intermediate interface connecting the speech encoder and the large language model. The speech recognition system does not directly couple the speech encoder and the large language model together. In this way, if the speech encoder and the large language model are jointly trained when training the speech recognition system, the representation difference between the speech modality and the text modality can be reduced, thereby suppressing the representation drift of the speech encoder caused by the representation difference. This allows the trained speech recognition system to obtain accurate text information when processing the speech to be recognized. The representation drift can be understood as the output information of the speech encoder being pulled away from the original acoustic representation space by the large language model.
[0120] Furthermore, based on the aforementioned structure of the speech recognition system, the performance requirements of large language models can be reduced, and speech recognition can still be accurately completed even when the parameter size of a large language model is reduced.
[0121] (3) The embodiments of this application utilize the semantic understanding capabilities of large language models to handle open-domain question answering, complex expressions, long-tail knowledge and conversational input, thereby overcoming the problems of weak recognition capabilities and obvious semantic shifts of traditional CTC (Connectionist Temporal Classification) small models in open-domain scenarios.
[0122] The following describes an embodiment of the speech recognition method provided in this application, specifically the processing of speech embedding of speech blocks based on prompt information using a large language model.
[0123] In some embodiments of this application, the prompt information may include not only the speech embeddings of multiple consecutive historical speech blocks, but also hot word text. Hot word text includes, but is not limited to, point of interest names, media names, brand words, personal names, or other target entities. Hot word text may be located at the end or middle of the prompt information, or it may be part of the system prompt information.
[0124] Adding hot word text to the prompts can improve the recognition capabilities of speech recognition systems in scenarios involving long-tail entities and personalized hot words. In this embodiment, hot word text can be obtained through a speech recognition system; the method for obtaining hot word text using a speech recognition system is described below.
[0125] See appendix Figure 3 , Figure 3 The main structure of the speech recognition system in an embodiment of this application is illustrated by way of example. Figure 3 As shown, in addition to the speech feature extraction module, speech encoder, speech adapter and large language model described in the foregoing embodiments, the speech recognition system may also include a phoneme acquisition module and a hot word retrieval module. The following describes these two modules.
[0126] 1. Explanation of the phoneme acquisition module.
[0127] In the embodiments of this application, the phoneme acquisition module can be configured to perform phoneme prediction on the phoneme-level speech representation output by the speech encoder to obtain the phoneme sequence of the current speech block.
[0128] In some embodiments, the phoneme acquisition module may employ a phoneme CTC header.
[0129] 2. Explanation of the hot word search module.
[0130] In this embodiment of the application, the hot word retrieval module can be configured to retrieve hot word texts in a preset hot word library that match the phoneme sequence of the current speech block (i.e., the output information of the phoneme acquisition module).
[0131] The hot word database is configured with multiple hot word texts and matching relationships between each hot word text and at least one phoneme sequence. For numerical expressions, English abbreviations, or entities with multiple pronunciations, multiple phoneme sequence matching relationships can be configured for the same hot word text to improve retrieval recall. In some embodiments, an Aho-Corasick automaton can be used to build a multi-pattern matching index to support low-latency retrieval of large-scale hot word databases.
[0132] In this embodiment, conventional retrieval and matching methods can be used to obtain hot word texts that match the phoneme sequence of the current speech block in a preset hot word database.
[0133] For example, in some embodiments, phoneme exact matching and longest match filtering strategies can be used to reduce interference from erroneous hot words. Phoneme exact matching can be understood as obtaining hot word texts that are consistent with or match the phoneme sequence of the current speech block, while longest match can be understood as selecting the longest hot word text if the phoneme sequence of the current speech block matches multiple hot word texts.
[0134] For example, in some embodiments, approximate matching, edit distance matching, and fuzzy matching can be used to match the phoneme sequence output by the phoneme acquisition module with the phoneme sequence in the hot word library. Alternatively, a combination of exact matching and approximate matching can be used for matching.
[0135] For example, in some embodiments, prefix trees, finite state machines, inverted indexes, vector recall, and rule filtering can be used to retrieve a preset hot word library.
[0136] Based on the aforementioned multi-pattern matching index and retrieval and matching methods, the hot word text acquisition method provided in this application is not only applicable to small-scale hot word libraries, but also meets the application needs of large-scale hot word libraries, frequent updates and strong customization in industrial scenarios.
[0137] Based on the aforementioned phoneme acquisition module and hot word retrieval module, the speech recognition system can obtain the phoneme sequence of the current speech block through the phoneme acquisition module, and then obtain the hot word text that matches the phoneme sequence of the current speech block through the hot word retrieval module.
[0138] The embodiments of the speech recognition method provided in this application will be described below, specifically the training method of the speech recognition system.
[0139] In some embodiments of this application, a phased training method can be used to train the speech recognition system. The following description, in conjunction with the appendix... Figure 4 The phased training method is explained.
[0140] like Figure 4As shown, the phased training method in this embodiment may include phases 1 to 4, which are described below.
[0141] 1. Explanation of Phase 1.
[0142] In this embodiment, the speech encoder can be pre-trained. Pre-training may include jointly training the speech encoder and the phoneme acquisition module so that the speech encoder learns the mapping relationship between the acoustic features of speech blocks and phoneme sequences.
[0143] Specifically, the speech encoder and phoneme acquisition module can be treated as a whole. Speech samples for pre-training are acquired, segmented into multiple consecutive speech blocks, and the acoustic features of each block are obtained. Each block is labeled with its corresponding ground truth phoneme sequence. During pre-training, the acoustic features of each speech block are sequentially input into the speech encoder for training. Taking a single speech block as an example, this block is input into the speech encoder for encoding to obtain a phoneme-level speech representation. The phoneme acquisition module then predicts the phonemes from this representation, obtaining the predicted phoneme sequence for the block. Based on the predicted and ground truth phoneme sequences, the loss value of the loss function used in pre-training is calculated, and the speech encoder parameters are updated according to the loss value. In this way, the speech encoder parameters are iteratively updated for each input speech block, effectively implementing streaming training constraints and enabling the trained speech encoder to adapt to the requirements of streaming incremental inference scenarios.
[0144] In some embodiments, attention masks can be applied to speech blocks during pre-training to restrict the speech blocks that the speech encoder and phoneme acquisition module should focus on or process. These speech blocks are either current or historical, and cannot be future. When processing a speech block, bidirectional modeling (such as bidirectional attention processing) can be performed on the speech frames within that block, enabling the speech encoder and phoneme acquisition module to simultaneously focus on speech frames preceding and following the current speech frame when processing a speech frame within the block.
[0145] The pre-training in stage 1 has the following technical effects: (1) maintaining the speech encoder's ability to model fine-grained acoustic information; (2) preventing the speech encoder from entering semantic modeling too early, thereby reducing the representational differences between the speech modality and the text modality.
[0146] In some embodiments of this application, after the pre-training in stage 1 is completed, the phoneme acquisition module can be fine-tuned based on a hot word retrieval task. The hot word retrieval task involves using the hot word retrieval module to retrieve hot word texts from a preset hot word library that match the phoneme sequences output by the phoneme acquisition module. Based on this implementation, a general phoneme acquisition module can be fine-tuned to adapt to the hot word retrieval task.
[0147] 2. Explanation of Phase 2.
[0148] In this embodiment, during the pre-training process (i.e., during the execution of stage 1), when the pre-training progress reaches a first preset progress, modality alignment training, which is executed asynchronously and in parallel with the pre-training, can be initiated. That is, stages 1 and 2 are executed in parallel without interfering with each other. The first preset progress can be a preset number of steps, a preset number of epochs, or a preset time, etc., and this embodiment does not specifically limit it.
[0149] Modality alignment training can include freezing the speech encoder and the large language model to train the speech adapter. Freezing can be understood as not updating the parameters; that is, the speech encoder and the large language model will still work normally during modality alignment training, but their parameters will not be updated.
[0150] A well-trained speech adapter can map the phoneme-level speech representations of speech blocks output by the speech encoder to the text embedding space of the large language model. Based on this, a basic cross-modal alignment relationship (or cross-modal mapping) can be established between the speech modality (i.e., the modality of the speech to be recognized) and the text modality (i.e., the modality of the text information output by the large language model), thereby reducing the representational differences between the speech modality and the text modality during the joint fine-tuning in the subsequent stage 4.
[0151] 3. Explanation of Phase 3.
[0152] In this embodiment, after modal alignment training is completed and the pre-training progress reaches the second preset progress (i.e., still in the process of stage 1 execution), supervised fine-tuning can be initiated asynchronously and in parallel with pre-training. That is, stages 1 and 3 are executed in parallel without interference. The second preset progress can be a preset number of steps, a preset number of epochs, or a preset time, etc., and this embodiment does not specifically limit it. Furthermore, the second preset progress is greater than the aforementioned first preset progress.
[0153] Supervised fine-tuning can include freezing the speech encoder and jointly training the speech adapter and the large language model. Freezing can be understood as not updating the parameters; that is, the speech encoder will still work normally during supervised fine-tuning, but the parameters will not be updated.
[0154] The large language model that has completed supervised fine-tuning can process the speech embeddings output by the speech adapter to generate text information. The speech embeddings output by the speech adapter are obtained by the speech adapter processing the speech encoder. Therefore, the purpose of supervised fine-tuning can be understood as enabling the large language model to learn how to accurately interpret the phoneme-level speech representations output by the speech encoder, thereby establishing a more stable cross-modal alignment relationship between the speech modality and the text modality, and suppressing the speech encoder from being pulled by the language model and causing representation drift in the subsequent joint fine-tuning in stage 4.
[0155] 4. Explanation of stage 4.
[0156] In this embodiment, joint fine-tuning can be initiated after both supervised fine-tuning and pre-training are completed. Joint fine-tuning may include joint training of the speech encoder, speech adapter, and large language model. After the aforementioned stages, a relatively stable cross-modal alignment relationship has been established between the speech modality and the text modality. By optimizing the speech encoder, speech adapter, and large language model end-to-end through joint fine-tuning, the performance of the entire speech recognition system can be further improved.
[0157] Training the speech recognition system using the phased training method described in the above embodiments enables it to perform speech recognition efficiently and accurately. Furthermore, establishing a stable cross-modal alignment relationship before joint fine-tuning can reduce the performance requirements of large language models, allowing for accurate speech recognition even with a smaller parameter size.
[0158] The following description continues with an embodiment of the speech recognition method provided in this application, specifically focusing on the supervised fine-tuning of stage 3 in the aforementioned phased training method.
[0159] In some embodiments of this application, during the pre-training process of stage 1, the speech encoder can be saved periodically and each saved speech encoder can be used as an encoder checkpoint.
[0160] Based on this, in the supervised fine-tuning of stage 3, the speech adapter and the large language model can be jointly trained in the following way:
[0161] Based on the encoder checkpoint, a reference encoder is obtained for supervised fine-tuning; based on the reference encoder, the speech adapter and the large language model are jointly trained. That is, during supervised fine-tuning, an encoder checkpoint generated during the pre-training process in stage 1 is selected as the speech encoder used for supervised fine-tuning. This speech encoder is frozen and does not undergo parameter updates, but it will still work normally to ensure that supervised fine-tuning can proceed normally.
[0162] In some embodiments of this application, when obtaining the reference encoder used for supervised fine-tuning based on encoder checkpoints, the encoder checkpoints saved when the pre-training progress reaches a third preset progress can be obtained, and these encoder checkpoints can be used as the reference encoder. The third preset progress can be a preset number of steps, a preset number of epochs, or a preset time, etc., and this embodiment does not specifically limit it. Furthermore, the third preset progress is greater than the aforementioned second preset progress.
[0163] In some embodiments of this application, during the supervised fine-tuning in stage 3, the reference encoder can also be updated periodically in the following manner:
[0164] The latest encoder checkpoint saved during pre-training is periodically retrieved and used as the latest checkpoint. The similarity between the first and second phoneme-level speech representations is obtained. The first and second phoneme-level speech representations are obtained by encoding the same acoustic feature by the latest checkpoint and the reference encoder, respectively. If the similarity is less than a set threshold, it indicates that the latest checkpoint and the reference encoder are significantly different, and the reference encoder is updated to the latest checkpoint. Otherwise, the reference encoder is not updated.
[0165] As pre-training progresses, the performance of encoder checkpoints improves, therefore, the latest checkpoint can be considered the optimal encoder checkpoint. If the latest checkpoint differs significantly from the reference encoder, it indicates that the reference encoder is no longer optimal and can be replaced with the optimal encoder checkpoint (i.e., the latest checkpoint). Based on this implementation, the reference encoder can be dynamically updated during supervised fine-tuning, ensuring that the reference encoder is as optimal as possible, and overfitting of a large language model to a single reference encoder can be avoided, which is beneficial for improving the training effect of supervised fine-tuning.
[0166] The following description continues with an embodiment of the speech recognition method provided in this application, which still focuses on the phased training method of the speech recognition system.
[0167] In some embodiments of this application, context-enhanced fine-tuning can be initiated after the joint fine-tuning in stage 4 is completed. This context-enhanced fine-tuning may include the following steps:
[0168] Acquire training samples, which include speech samples and their corresponding ground truth texts; extract keywords from the ground truth texts and obtain interference words that are homophones or near-homophones of the keywords, and add the keywords and interference words to the prompt information corresponding to the training samples; train the speech recognition system based on the training samples and their corresponding prompt information.
[0169] By enhancing the context, large language models can make correct use of hot word text while maintaining their reliance on real acoustic evidence (i.e., truth text), thereby enhancing the speech recognition system's ability to recognize speech in hot word scenarios and reducing the misleading effect of error messages on speech recognition results.
[0170] In some embodiments of this application, after the context enhancement fine-tuning is completed, reinforcement learning can also be performed on the speech recognition system. Reinforcement learning mainly optimizes the speech encoder, speech adapter, and large language model in the speech recognition system. In this embodiment, conventional reinforcement learning methods can be used to perform reinforcement learning on the speech recognition system, such as the GRPO (Group Relative Policy Optimization) reinforcement learning method.
[0171] In some embodiments, the reward items used in reinforcement learning may include recognition accuracy, code-switching consistency, phantom suppression, and hot word hit rate. Additionally, rewards such as latency, stability, multi-turn consistency, or domain word accuracy may be added based on business needs. Specifically, code-switching consistency evaluates the consistency of speech recognition results after changes in the speech type, style, etc.; phantom suppression constrains the speech recognition results from including content not present in the speech to be recognized; hot word hit rate evaluates whether the speech recognition results contain the required hot word text; latency evaluates the time required for speech recognition; stability evaluates the stability of the speech recognition results; multi-turn consistency evaluates the consistency of results obtained from multiple recognitions of the same speech; and domain word accuracy evaluates the accuracy of the speech recognition results in including domain words.
[0172] In some embodiments of this application, reinforcement learning may also be replaced by other policy optimization methods, ranking optimization methods, minimum risk training methods, or alignment methods based on preference data.
[0173] Through the above embodiments, the robustness of the speech recognition system under conditions such as noise, accents, dialects, and code switching can be further improved. The speech recognition method provided in this application unifies speech recognition, hot word enhancement, prompt information utilization, robustness optimization, and streaming inference in a single technical framework. This reduces reliance on external vendor models, joint arbitration modules, and numerous manual rule patches, which helps shorten the optimization cycle of the speech recognition system and improves its consistency and engineering maintainability. Furthermore, because this application adopts a unified large-model speech recognition framework and retains semantic understanding, prompt information enhancement, and end-to-end optimization capabilities, it is easier to extend to advanced capabilities such as multi-turn context consistency, emotion perception, and personalized interaction in subsequent speech recognition system evolution. This overcomes the problems of poor scalability and difficulty in transitioning to next-generation voice interaction architectures in traditional CTC small-model solutions.
[0174] The following description continues with an embodiment of the speech recognition method provided in this application, which still focuses on the phased training method of the speech recognition system.
[0175] In some embodiments of this application, the aforementioned phased training method can be adjusted. For example, encoder pre-training can be completed before modality alignment training, or the context enhancement fine-tuning and the joint fine-tuning part of stage 4 can be merged, or the reinforcement learning stage can be omitted, and basic recognition ability can be obtained solely by the aforementioned phased training.
[0176] The following is in conjunction with the appendix Figure 5 and attached Figure 6 This paper compares and explains the performance of the speech recognition system provided in this application with that of existing speech recognition systems. Figure 5 In the diagram, the horizontal axis "1 to 16" represents the number of layers in the speech encoder of the speech recognition system, the horizontal axis "Emb" represents the speech embedding (large language models can process speech embeddings to generate text information), and the vertical axis represents the CKA (Centered Kernel Alignment) score. For example... Figure 5 As shown, the CKA score of the speech embedding obtained by the speech recognition system provided in this application is better than that obtained by the prior art. This indicates that the speech recognition system provided in this application has a higher cross-modal mapping capability from speech modality to text modality and can complete speech recognition more accurately.
[0177] Please refer to the appendix for further details. Figure 6 , Figure 6 The example illustrates the test results obtained after conducting speech recognition experiments on the speech recognition system provided in this application and six existing speech recognition systems. Figure 6 In this document, Fun-ASR Nano, GLM-ASR Nano, Qwen3-ASR 1.7B, FireRedASR2 LLM, Step-Audio2 Mini, and Qwen3-Omni Instruct represent six existing speech recognition systems. The terms "offline version" and "streaming version" refer to the speech recognition system provided in this application operating in offline and streaming modes, respectively. Offline mode involves acquiring a complete segment of speech to be recognized and then performing speech recognition on this segment using the methods described in steps S101 to S102 of the aforementioned embodiments. Streaming mode involves acquiring speech segments of the speech to be recognized in real time and then performing speech recognition on the acquired speech segments using the methods described in steps S101 to S102 of the aforementioned embodiments. Additionally, Figure 6 The numbers recorded in lines 3 to 19 represent the word error rate of the speech recognition results. The word error rate can be understood as the proportion of misspelled words in the total number of words in the speech recognition results.
[0178] The win-loss statistics between the offline system and the baseline model refer to the win-loss statistics between the offline system and the six existing speech recognition systems mentioned above. Taking Fun-ASR Nano as an example, the win-loss statistics between the offline system and Fun-ASR Nano are 21:02.
[0179] according to Figure 6 The test results shown confirm that the speech recognition system provided in this application has superior recognition performance compared to most existing speech recognition systems.
[0180] It should be noted that although the steps in the above embodiments are described in a specific order, those skilled in the art will understand that in order to achieve the effect of this application, different steps do not necessarily have to be executed in such an order. They can be executed simultaneously (in parallel) or in other orders. These adjusted solutions are equivalent to the technical solutions described in this application and therefore will also fall within the protection scope of this application.
[0181] Those skilled in the art will understand that all or part of the processes in the method of the above-described embodiment can also be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable file, or some intermediate form. The computer-readable storage medium can include any entity or device capable of carrying the computer program code, a medium, a USB flash drive, a portable hard drive, a magnetic disk, an optical disk, a computer memory, a read-only memory, a random access memory, an electrical carrier signal, a telecommunication signal, and a software distribution medium, etc.
[0182] Another aspect of this application provides a computer-readable storage medium.
[0183] In one embodiment of a computer-readable storage medium according to this application, the computer-readable storage medium may be configured to store a program that performs the speech recognition method of the above-described method embodiments. This program may be loaded and run by a processor to implement the speech recognition method. For ease of explanation, only the parts related to the embodiments of this application are shown; for specific technical details not disclosed, please refer to the method section of the embodiments of this application. The computer-readable storage medium may be a storage device comprising various electronic devices. Optionally, in the embodiments of this application, the computer-readable storage medium is a non-transitory computer-readable storage medium.
[0184] Another aspect of this application provides a smart device.
[0185] In one embodiment of a smart device according to this application, the smart device may include at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores a computer program, which, when executed by the at least one processor, implements the method described in any of the above embodiments. The smart device described in this application may include driving equipment, smart vehicles, robots, and other devices. See appendix. Figure 7 , Figure 7 The example illustrates a memory and processor connected via a bus communication connection.
[0186] In some embodiments of this application, the smart device may further include at least one sensor for sensing information. The sensor is communicatively connected to any type of processor mentioned in this application. Optionally, the smart device may further include an autonomous driving system for guiding the smart device to drive autonomously or assisting in driving. The processor communicates with the sensor and / or the autonomous driving system to perform the methods described in any of the above embodiments.
[0187] In the description of this application, "module" and "processor" can include hardware, software, or a combination of both. A module can include hardware circuitry, various suitable sensors, communication ports, memory, and may also include software components, such as program code, or a combination of software and hardware. A processor can be a central processing unit, microprocessor, image processor, digital signal processor, or any other suitable processor. The processor has data and / or signal processing capabilities. The processor can be implemented in software, in hardware, or a combination of both. Computer-readable storage media includes any suitable medium capable of storing program code, such as magnetic disks, hard disks, optical disks, flash memory, read-only memory, random access memory, etc. The term "A and / or B" means all possible combinations of A and B, such as only A, only B, or A and B. The terms "at least one A or B" or "at least one of A and B" have a similar meaning to "A and / or B" and can include only A, only B, or A and B. The singular terms "a" or "this" can also include plural forms.
[0188] The technical solutions of this application have been described above with reference to the preferred embodiments shown in the accompanying drawings. However, it will be readily understood by those skilled in the art that the scope of protection of this application is obviously not limited to these specific embodiments. Without departing from the principles of this application, those skilled in the art can make equivalent changes or substitutions to the relevant technical features, and the technical solutions after these changes or substitutions will all fall within the scope of protection of this application.
Claims
1. A speech recognition method, characterized in that, The method includes: Acquire the speech to be recognized and segment the speech into multiple consecutive speech blocks; The multiple speech blocks are sequentially input into the speech recognition system for processing to obtain the text information corresponding to the speech to be recognized. The speech recognition system includes a speech feature extraction module, a speech encoder, a speech adapter, and a large language model. The speech feature extraction module is configured to extract the acoustic features of the current speech block, which is the speech block currently input to the speech recognition system; The speech encoder is configured to encode the acoustic features of the current speech block to obtain a phoneme-level speech representation of the current speech block. The speech adapter is configured to map the phoneme-level speech representation of the current speech block to the text embedding space of the large language model to obtain the speech embedding of the current speech block. The large language model is configured to process the speech embedding of the current speech block based on prompt information, and generate the text information after the speech embedding processing of all speech blocks is completed. The prompt information includes the speech embeddings of multiple consecutive historical speech blocks preceding the current speech block.
2. The method according to claim 1, characterized in that, The prompt information also includes hot word text, which is obtained by the speech recognition system in the following way: The phoneme acquisition module within the speech recognition system performs phoneme prediction on the phoneme-level speech representation output by the speech encoder to obtain the phoneme sequence of the current speech block. The hot word retrieval module in the speech recognition system retrieves hot word texts that match the phoneme sequence of the current speech block from a preset hot word library. The hot word library is configured with multiple hot word texts and matching relationships between each hot word text and at least one phoneme sequence.
3. The method according to claim 1, characterized in that, The speech recognition system is trained using the following phased training method: The speech encoder is pre-trained, which includes joint training of the speech encoder and the phoneme acquisition module in the speech recognition system, so that the speech encoder learns the mapping relationship between the acoustic features of speech blocks and phoneme sequences; wherein, the phoneme acquisition module is configured to perform phoneme prediction on the phoneme-level speech representation output by the speech encoder to obtain a phoneme sequence. During the pre-training process, when the progress of the pre-training reaches a first preset progress, modal alignment training is started in parallel with the pre-training asynchronously. The modal alignment training includes freezing the speech encoder and the large language model and training the speech adapter. After the modality alignment training is completed and the pre-training progress reaches the second preset progress, supervised fine-tuning is initiated in parallel with the pre-training, which is asynchronous. The supervised fine-tuning includes freezing the speech encoder and jointly training the speech adapter and the large language model. After the supervised fine-tuning and the pre-training are completed, joint fine-tuning is initiated, which includes joint training of the speech encoder, speech adapter and large language model.
4. The method according to claim 3, characterized in that, During the pre-training process, the speech encoder is saved periodically, and each saved speech encoder is used as an encoder checkpoint. In the supervised fine-tuning, the joint training of the speech adapter and the large language model includes: Based on the encoder checkpoints, obtain the reference encoder used for the supervisory fine-tuning; Based on the reference encoder, the speech adapter and the large language model are jointly trained.
5. The method according to claim 4, characterized in that, The step of obtaining the reference encoder used for the supervisory fine-tuning based on the encoder checkpoint includes: Obtain the encoder checkpoint saved when the pre-training progress reaches the third preset progress, and use the encoder checkpoint as the reference encoder.
6. The method according to claim 5, characterized in that, In the supervised fine-tuning, the reference encoder is also periodically updated in the following manner: Periodically retrieve the latest encoder checkpoint saved during the pre-training process and use that encoder checkpoint as the latest checkpoint. The similarity between the first phoneme-level speech representation and the second phoneme-level speech representation is obtained, wherein the first phoneme-level speech representation and the second phoneme-level speech representation are respectively obtained by encoding the same acoustic feature by the latest checkpoint and the reference encoder. If the similarity is less than a set threshold, the reference encoder is updated to the latest checkpoint; Otherwise, the reference encoder will not be updated.
7. The method according to claim 3, characterized in that, The phased training method also includes initiating context-enhanced fine-tuning after the joint fine-tuning is completed; The context enhancement fine-tuning includes: Obtain training samples, which include speech samples and their corresponding ground truth texts; Extract keywords from the true text and obtain interference words that are homophones or near-homophones of the keywords. Add the keywords and interference words to the prompt information corresponding to the training samples. The speech recognition system is trained based on the training samples and their corresponding prompts.
8. The method according to claim 7, characterized in that, The phased training method further includes: after the context enhancement fine-tuning is completed, reinforcement learning is performed on the speech recognition system.
9. The method according to claim 3, characterized in that, The phased training method further includes: after the pre-training is completed, fine-tuning the phoneme acquisition module based on the hot word retrieval task; wherein, the hot word retrieval task is: The hot word retrieval module in the speech recognition system retrieves hot word texts that match the phoneme sequence output by the phoneme acquisition module from a preset hot word library. The hot word library is configured with multiple hot word texts and matching relationships between each hot word text and at least one phoneme sequence.
10. A smart device, characterized in that, include: At least one processor; And, a memory communicatively connected to the at least one processor; The memory stores a computer program, which, when executed by the at least one processor, implements the speech recognition method according to any one of claims 1 to 9.