Speech recognition method and device based on tool invocation, equipment and storage medium

Abstract

The application relates to the technical field of speech recognition, and provides a speech recognition method, device and equipment based on tool calling and a storage medium, which comprises the following steps: performing feature extraction on an audio stream to be recognized to obtain an acoustic representation sequence; inputting the acoustic representation sequence into a decoder to perform autoregressive decoding, and determining whether tool calling is triggered in the decoding process; if it is determined that the calling is triggered, the autoregressive decoding is suspended, the decoding state information is frozen, and a tool calling request is generated; an external tool is called based on the tool calling request, and a returned calling result is obtained; the autoregressive decoding is restored based on the frozen decoding state information, and the calling result is input into the decoder to continue decoding to obtain recognized text. In the application, the decoding state is suspended and frozen in the middle of decoding to call an external tool, the calling result is combined with the frozen state information to seamlessly restore the decoding, early injection and timing alignment of external knowledge in the speech recognition process are realized, error cascading is eliminated, and the recognition accuracy is improved.

Description

Technical Field

[0001] This invention relates to the field of speech recognition technology, and in particular to a speech recognition method, apparatus, device, and storage medium based on tool invocation. Background Technology

[0002] With the popularization of voice interaction technology, Automatic Speech Recognition (ASR) systems have penetrated into the complex business processes of various industries. However, in practical applications, such as smart vehicles and professional dictation scenarios, traditional ASR models trained on fixed corpora are prone to recognition errors due to the presence of a large number of long-tail words, private data, or limitations imposed by user accents.

[0003] To improve speech recognition accuracy in complex scenarios, two main existing technical solutions are currently employed. One solution is retrieval-augmented generation (RAG) or large-scale model error correction based on text post-processing. This solution first uses a traditional end-to-end ASR model to fully decode the input audio to generate initial recognition text, and then inputs this text into a large language model for error correction in conjunction with external retrieval. However, this solution intervenes in error correction only after full decoding is completed, meaning that key acoustic features have already been lost during the decoding process, which can easily lead to severe error cascading, resulting in a low error correction success rate. Furthermore, full decoding followed by error correction significantly increases system response latency.

[0004] Another approach is multi-turn dialogue interaction technology based on large speech models. While this approach combines voice input with tool invocation, the tool invocation occurs during the semantic understanding and task execution phases, not the speech transcription phase. For continuous audio streams with strict temporal characteristics, existing models lack dynamic suspension and state recovery mechanisms during the decoding process. If decoding is paused midway to wait for external data, it can lead to temporal alignment failures and state loss issues in the continuous audio stream. Summary of the Invention

[0005] This invention provides a speech recognition method, apparatus, device, and storage medium based on tool invocation, which addresses the shortcomings of existing speech recognition technologies, such as loss of acoustic features, low recognition accuracy, and failure of timing alignment, due to the fact that post-processing can only be performed after full decoding or the lack of a state management mechanism during continuous audio stream decoding.

[0006] This invention provides a speech recognition method based on tool invocation, comprising: Feature extraction is performed on the audio stream to be identified to obtain an acoustic representation sequence; The acoustic representation sequence is input into the decoder for autoregressive decoding, and it is determined whether a tool call is triggered during the autoregressive decoding process; If a tool call is triggered, the autoregressive decoding is suspended, the current decoding state information of the decoder is frozen, and a tool call request is generated. Based on the tool call request, an external tool is invoked, and the call result returned by the external tool is obtained; The autoregressive decoding is restored based on the frozen decoding state information, and the call result is input into the decoder to continue autoregressive decoding to obtain the recognition text corresponding to the audio stream.

[0007] According to a speech recognition method based on tool invocation provided by the present invention, the step of determining whether to trigger tool invocation during the autoregressive decoding process includes: During the process of generating the next word through autoregressive decoding in the decoder, acoustic uncertainty information is determined based on the acoustic representation sequence; Based on the historical text sequence generated by the decoder, determine the semantic entity requirement information; A joint decision is made based on the acoustic uncertainty information and the semantic entity demand information to determine whether to trigger a tool call.

[0008] According to a speech recognition method based on tool invocation provided by the present invention, the step of determining acoustic uncertainty information based on the acoustic representation sequence includes: The acoustic representation sequence is processed based on the cross-attention mechanism to obtain the predicted probability of each candidate word in the vocabulary as the next word. Based on the predicted probability of each candidate word as the next word, the predicted probability distribution of the next word is obtained, and the entropy of the predicted probability distribution is calculated. If the entropy is greater than a preset threshold, the acoustic uncertainty information is determined to indicate high acoustic uncertainty.

[0009] According to a speech recognition method based on tool invocation provided by the present invention, the step of determining semantic entity demand information based on the historical text sequence generated by the decoder includes: The historical text sequence is used to extract features by employing a causal self-attention mechanism to obtain the semantic context features of the current context; The entity demand vector of the current context is determined based on the semantic context features, and the entity demand vector is used as the semantic entity demand information. The joint decision-making process based on the acoustic uncertainty information and the semantic entity demand information to determine whether to trigger a tool call includes: When the acoustic uncertainty information indicates high acoustic uncertainty and the semantic entity demand information indicates the existence of a target entity demand, the generation probability of regular text lexical units is suppressed, and a tool call control character is output to determine the trigger for tool call.

[0010] According to a speech recognition method based on tool invocation provided by the present invention, suspending the autoregressive decoding and freezing the current decoding state information of the decoder includes: In response to the output of the tool call control character, the reading of subsequent audio frames in the audio stream and the autoregressive calculation of the decoder are paused; Freeze the key-value caches and hidden states of each layer of the decoder, and save the current audio frame alignment timestamp; The decoding status information includes the key-value cache, the hidden state, and the audio frame alignment timestamp.

[0011] According to a speech recognition method based on tool invocation provided by the present invention, the generation of tool invocation request includes: In the suspended state, a structured tool call request is generated based on the fuzzy pronunciation information corresponding to the acoustic representation sequence when the tool is triggered and the entity demand vector. The tool call request carries the tool name and query parameters, and the query parameters are determined according to the fuzzy pronunciation information.

[0012] According to a speech recognition method based on tool invocation provided by the present invention, the method for recovering the autoregressive decoding based on the frozen decoding state information and inputting the invocation result into the decoder to continue autoregressive decoding includes: The call result is encapsulated into a result word sequence delimited by a result identifier, and the result word sequence is input into the decoder as a context prefix; Resume the suspended autoregressive decoding, based on the frozen key-value cache and the hidden state, and continue to predict text words following the result word sequence as a prefix constraint; Based on the saved audio frame alignment timestamps, the reading of subsequent audio frames in the audio stream is resumed, and the subsequent audio frames are aligned and decoded.

[0013] According to a tool-invoking-based speech recognition method provided by the present invention, the step of extracting features from the audio stream to be recognized to obtain an acoustic representation sequence includes: After the audio stream is processed into frames, it is input into the audio encoder to obtain the acoustic hidden state sequence corresponding to the audio frame; The acoustic hidden state sequence is downsampled in the time dimension to obtain the downsampled acoustic hidden state sequence; The downsampled acoustic latent state sequence is mapped to the same dimension as the word embedding space of the decoder to obtain the aligned acoustic representation sequence.

[0014] According to a speech recognition method based on tool invocation provided by the present invention, the step of invoking an external tool based on the tool invocation request and obtaining the invocation result returned by the external tool includes: The tool invocation request is sent to an external tool execution environment, and the tool invocation request is routed to the target external tool through the routing gateway in the external tool execution environment, while simultaneously waiting for the invocation result returned by the target external tool; The external tools include at least one of an address book interface, a map interface, and a knowledge base retrieval engine.

[0015] The present invention also provides a speech recognition device based on tool invocation, comprising: The feature extraction module is used to extract features from the audio stream to be identified, and obtain an acoustic representation sequence. The regression decoding module is used to input the acoustic representation sequence into the decoder for autoregressive decoding, determine whether a tool call is triggered during the autoregressive decoding process, and generate a tool call request if it is determined that a call denial is triggered. The state management module is used to suspend the autoregressive decoding and freeze the current decoding state information of the decoder when it is determined that a tool call has been triggered. The tool invocation module is used to invoke an external tool based on the tool invocation request and obtain the invocation result returned by the external tool; The state management module is also used to restore the autoregressive decoding based on the frozen decoding state information after obtaining the call result; The regression decoding module is also used to input the call result into the decoder to continue autoregressive decoding to obtain the recognition text corresponding to the audio stream.

[0016] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and running on the processor, wherein the processor executes the computer program to implement the speech recognition method based on tool invocation as described above.

[0017] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the speech recognition method based on tool invocation as described above.

[0018] The present invention also provides a computer program product, including a computer program that, when executed by a processor, implements the speech recognition method based on tool invocation as described above.

[0019] The speech recognition method, apparatus, device, and storage medium based on tool invocation provided by this invention avoids the system response delay problem caused by traditional solutions that require waiting for full decoding before processing by determining in real time whether to trigger a tool invocation during the process of inputting the acoustic representation sequence into the decoder for autoregressive decoding. When a tool invocation is triggered, this invention can suspend the current autoregressive decoding process and freeze the decoder's decoding state information. This freezing operation allows the decoder to fully retain the extracted key acoustic features and context state while waiting for external knowledge injection, preventing the decoder from continuing to force autoregressive prediction when faced with uncertain acoustic features, thereby preventing decoding path deviation and error cascading problems caused by early acoustic feature ambiguity. Subsequently, by obtaining accurate external call results based on requesting external tools and inputting these results into the decoder, combined with the previously frozen decoding state information, autoregressive decoding is resumed. This recovery mechanism based on the frozen state can seamlessly splice the externally returned results with the acoustic context before suspension, thereby ensuring the temporal alignment of the continuous audio stream and guaranteeing the continuity and accuracy of the decoding process. Attached Figure Description

[0020] To more clearly illustrate the technical solutions in this invention or related technologies, the accompanying drawings used in the description of the embodiments or related technologies will be briefly introduced below. Obviously, the accompanying drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0021] Figure 1 This is a flowchart illustrating the speech recognition method based on tool invocation provided by the present invention; Figure 2 This is a schematic diagram of the structure of the tool-calling intelligent agent speech recognition system based on acoustic and semantic joint driving provided by the present invention; Figure 3 This is a schematic diagram of the process by which a tool based on acoustic and semantic joint driving calls an intelligent agent speech recognition method, as provided in this invention. Figure 4 This is a schematic diagram of the speech recognition device based on tool invocation provided by the present invention; Figure 5 This is a schematic diagram of the structure of the electronic device provided by the present invention. Detailed Implementation

[0022] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0023] With the popularization of voice interaction technology, Automatic Speech Recognition (ASR) systems have evolved from simple command control to complex business processes across various industries. However, in real-world applications, existing ASR technologies have revealed certain limitations.

[0024] For example, in scenarios involving intelligent in-vehicle systems and local life assistants, users frequently need to use voice commands to operate navigation or access their contacts while driving, such as "navigate to Zhang Ji Braised Chicken on XX Road" or "call Li Mou Xuan." These scenarios contain a large amount of long-tail Points of Interest (POIs) and private contact data. Traditional ASR models are trained on fixed corpora and cannot exhaustively cover all place names and shop names. They are prone to misidentification when encountering uncommon characters or accents, such as misidentifying "Li Mou Xuan" as "Li Mou Xuan," leading to command execution failure and affecting driving safety and user experience.

[0025] In professional fields (such as medicine, law, and finance), dictation scenarios are filled with a large number of technical terms, rare drug names (such as "methylprednisolone sodium succinate"), legal provisions, or financial product codes. These technical terms are not adequately acoustically modeled in general ASR models, resulting in an extremely high error rate. They often require secondary manual proofreading, increasing time costs.

[0026] In scenarios involving dynamic information and multi-turn dialogues (such as intelligent customer service), users often mention dynamic information such as order numbers, flight numbers, and real-time weather. Traditional ASR models can only output text and cannot understand the business meaning of these entities, let alone actively acquire the latest data for self-verification during the recognition process.

[0027] With the development of Large Language Model (LLM) technology, ASR systems are evolving from traditional independent acoustic-language models to end-to-end large speech models. To address the problem of low recognition accuracy for complex long-tail entities (such as contact names, technical terms, geographical locations, etc.), the industry and academia currently mainly adopt the following two existing technical solutions.

[0028] The first solution is a retrieval-augmented generation or large model error correction technology based on text post-processing. This solution adopts a serial pipeline architecture. First, a traditional end-to-end ASR model (such as Whisper) completely decodes the input audio to generate an initial recognized text; then, the initial recognized text is input into a large language model, which queries and corrects suspected erroneous entities in the initial recognized text in combination with an external retrieval engine according to contextual semantics, and finally outputs the corrected text.

[0029] The second solution is a multi-turn dialogue interaction technology based on a large speech model (Speech LLM). This solution directly inputs audio features into the large speech model of the large language model base, enabling it to have multi-turn dialogue and tool calling capabilities. When executing a voice command, the system first fully recognizes the user's voice command. If the large model determines that external information is required (such as querying the weather, querying a database, etc.), it generates a tool calling request at the text level, and replies to the user in voice or text form after obtaining the result.

[0030] However, the above technical solutions still have certain drawbacks in practical applications. Specifically, the essence of the retrieval-augmented generation or large model error correction technology based on text post-processing is "error first, correct later", which has a serious error cascading problem. When ASR outputs an erroneous entity (for example, decoding "Li Mouxuan" as "Li Mouxuan") due to acoustic ambiguity in the early decoding stage, key acoustic features (such as phoneme distribution and confidence) have already been lost during the decoding process. The subsequent text error correction model can only guess based on the wrong text literal and limited context, and cannot trace back to the original audio features, resulting in a low error correction success rate. In addition, decoding the full amount of text before performing error correction greatly increases the overall response latency of the system.

[0031] Although the multi-turn dialogue interaction technology based on the large speech model realizes the combination of speech input and tool calling, its tool calling occurs in the semantic understanding and task execution stage, rather than the speech transcription (ASR) stage. When performing the ASR transcription task, the model still passively and one-off maps the audio stream to a sequence of text tokens (Token, that is, the smallest semantic or character unit in natural language processing, such as a Chinese character or an English subword). Due to the continuous temporal characteristic of audio frames, if the pause-call mechanism of the text agent is directly applied to the ASR decoding process, the external discrete text result cannot be spliced with the continuous acoustic context. Existing large speech models lack a mechanism for dynamic state suspension and multimodal state recovery in the middle of decoding, which easily leads to the problems of temporal alignment failure and state loss of the continuous audio stream.

[0032] To address this, the present invention provides a speech recognition method based on tool invocation, which can dynamically introduce external knowledge in the early stage of speech recognition (ASR) decoding without increasing the system's post-processing latency, and provides a secure and seamless decoding suspension and resumption mechanism to achieve high-accuracy end-to-end agent speech recognition, thereby overcoming the above-mentioned defects.

[0033] It should be noted that the execution entity of the method provided by this invention can be a cloud server, an edge computing node, or a local smart terminal device, such as a smart vehicle terminal (in-vehicle system), a smartphone, a smart wearable device, or a dedicated voice dictation device. In some complex application scenarios (such as professional field dictation scenarios or smart vehicle assistance scenarios), the execution entity typically includes a processor and a memory. The processor executes pre-configured computer program instructions in the memory to call the underlying large language model base and the corresponding acoustic processing module, thereby realizing end-to-end intelligent agent speech recognition and dynamic interaction with external tools.

[0034] Figure 1 This is a flowchart illustrating the speech recognition method based on tool invocation provided by the present invention, as shown below. Figure 1 As shown, the method includes: Step S10: Extract features from the audio stream to be identified to obtain an acoustic representation sequence.

[0035] Specifically, the audio stream to be identified can be a continuous user speech stream captured in real time by a microphone, or a pre-recorded and input audio file, such as voice commands containing long-tail points of interest, private contacts, or technical terms. After acquiring the audio stream, the system uses the underlying audio encoder to extract physical features. The specific feature extraction process typically involves preprocessing the continuous audio stream, such as framing and windowing, and then mapping it into a high-dimensional acoustic latent state sequence, thereby forming an acoustic representation sequence that can be understood by subsequent models.

[0036] Here, the acoustic representation sequence refers to the high-dimensional feature vector sequence generated after feature extraction and modality alignment of the original continuous audio stream, which matches the word embedding space dimension of the large language model decoder. Specifically, it is the final result obtained after further downsampling in the time dimension (to preserve key acoustic features and significantly shorten the sequence length) and linear mapping in the feature dimension (to eliminate the dimensional differences between the acoustic modality and the text semantic modality) from the high frame rate acoustic latent state sequence initially extracted by the audio encoder. This acoustic representation sequence not only fully preserves the underlying acoustic features such as phoneme distribution and pronunciation details in the original speech input, but also has been transformed into a form that the decoder can directly read and understand in terms of data structure and representation dimension. This provides a cross-modal data foundation for subsequent model decoding and tool-triggered judgment by combining acoustic features with text semantics.

[0037] Step S20: Input the acoustic representation sequence into the decoder for autoregressive decoding, and determine whether to trigger a tool call during the autoregressive decoding process.

[0038] Specifically, a decoder typically refers to an intelligent agent decoder built on a large language model foundation, possessing text generation and logical reasoning capabilities. Autoregressive decoding refers to the process by which a decoder predicts and generates a text sequence word by word based on the input acoustic representation sequence and historically generated contextual information.

[0039] Unlike traditional speech recognition models that passively and one-time map the entire audio stream to text, the decoder in this embodiment dynamically evaluates its state each time it predicts and generates the next token. Specifically, the model comprehensively considers whether the acoustic features of the current location are ambiguous, such as physical incomprehensibility caused by accents or uncommon characters, and whether the semantic logic of the current context points to a specific entity. For example, if the preceding semantics are "make a phone call," a proper noun (i.e., the contact's name) is logically required. When the model finds that it cannot accurately determine the specific word corresponding to the current pronunciation based solely on internal knowledge during autoregressive decoding, it can promptly trigger tool calls in the early stages of decoding.

[0040] Step S30: If it is determined that a tool call has been triggered, the autoregressive decoding is suspended, the current decoding state information of the decoder is frozen, and a tool call request is generated.

[0041] Specifically, when it is determined that external knowledge is needed, i.e., when a tool call is triggered, the system will immediately intervene in the current process to avoid timing alignment failures and state loss in continuous audio streams. Specifically, the system will suspend the current autoregressive decoding process and freeze the decoder's current decoding state information.

[0042] Here, suspending autoregressive decoding means pausing the decoder's regular autoregressive text generation calculations and temporarily stopping the reading of subsequent audio frame data. Decoding state information refers to the internal context data of the decoder neural network at the current moment, mainly including the hidden states of each layer of the model, key-value caches (KV-Cache), etc., thereby achieving the freezing and preservation of continuous acoustic context.

[0043] Without losing state, the decoder generates a structured tool call request based on the current ambiguous pronunciation features and historical semantic intent. This request is intended to express specific clues to be queried to external systems, such as suspected pinyin or query range.

[0044] Step S40: Invoke an external tool based on the tool invocation request and obtain the invocation result returned by the external tool.

[0045] Specifically, the generated tool invocation request will be sent to an external tool execution environment. This external tool can be a locally stored database interface or a cloud-based application programming interface (API). For example, in the scenario of intelligent in-vehicle and local life assistant, the external tool may include a local address book API or a real-time map search API; in the scenario of professional domain dictation, the external tool may be an internal enterprise knowledge base search engine or a medical drug name database, etc. After receiving the request, the external tool executes the corresponding search or query logic and returns accurate information that exists in the real world (i.e., the invocation result) to the system.

[0046] Step S50: Recover the autoregressive decoding based on the frozen decoding state information, and input the call result into the decoder to continue autoregressive decoding to obtain the recognition text corresponding to the audio stream.

[0047] Specifically, after obtaining the call result returned from the external source, the system needs to append it to the suspended decoding process. At this point, the system uses the accurate call result as a priori context constraint input to the decoder and wakes up the previously frozen decoding state information (i.e., restores the original hidden state and key-value cache).

[0048] Since the decoder did not lose the key acoustic alignment features before the suspension, after resuming autoregressive decoding, the decoder can use the exact text result given externally (i.e. the externally returned call result) to correct the deviation path that may have been caused by acoustic ambiguity, and naturally continue to process the remaining unread audio frames until the alignment decoding of the full audio is completed, and finally output complete and accurate recognized text.

[0049] For example, when a user's intention is "call Li Mou Xuan" but the accent is unclear, the decoder stops when it encounters "Li Mou Xuan" and checks the address book. After obtaining the accurate result "Li Mou Xuan", it resumes decoding and outputs the correct full sentence, rather than the incorrect text caused by blind guessing.

[0050] The method provided in this invention avoids the system response delay problem caused by traditional solutions that require waiting for full decoding before processing by determining in real time whether a tool call is triggered during the process of inputting the acoustic representation sequence into the decoder for autoregressive decoding. When a tool call is triggered, this invention can suspend the current autoregressive decoding process and freeze the decoder's decoding state information. This freezing operation allows the decoder to fully retain the extracted key acoustic features and context state while waiting for external knowledge injection, preventing the decoder from continuing to force autoregressive prediction when facing uncertain acoustic features, thereby preventing decoding path offset and error cascading problems caused by early acoustic feature ambiguity. Subsequently, by obtaining accurate external call results based on the request to call external tools and inputting these results into the decoder, combined with the previously frozen decoding state information, autoregressive decoding is restored. This recovery mechanism based on the frozen state can seamlessly stitch the externally returned results with the acoustic context before suspension, thereby ensuring the temporal alignment of the continuous audio stream and guaranteeing the continuity and accuracy of the decoding process.

[0051] Based on the above embodiments, step S10, which involves extracting features from the audio stream to be identified to obtain an acoustic representation sequence, includes: Step S11: After the audio stream is processed into frames, it is input into the audio encoder to obtain the acoustic latent state sequence corresponding to the audio frame.

[0052] Specifically, after acquiring a continuous audio stream, the system first performs acoustic preprocessing operations such as frame segmentation and windowing. The processed audio frame data is then input into the audio encoder. This audio encoder can employ a structure such as WhisperEncoder or a multi-layer Transformer network model to extract the deep acoustic features of each audio frame. After computation by the audio encoder, a high-dimensional acoustic hidden state sequence corresponding to each audio frame is finally extracted. This sequence can be represented as... ,in Indicates the first The acoustic latent state vector corresponding to the frame audio.

[0053] Step S12: Downsample the acoustic latent state sequence in the time dimension to obtain the downsampled acoustic latent state sequence.

[0054] Specifically, since the acoustic latent state sequence output by the audio encoder usually has a high frame rate, directly inputting it into the decoder would result in excessive computational complexity and a large amount of redundant information. Therefore, a modal alignment adapter is needed to process it.

[0055] The modal alignment adapter internally contains a temporal downsampling unit, which typically employs a one-dimensional convolutional layer (i.e., 1D convolution, such as a convolutional kernel with a stride of 2) or a pooling layer. This unit processes high-frame-rate acoustic latent state sequences. Perform time-dimension downsampling, for example, downsampling the original 50Hz frame rate to 25Hz. This operation can significantly shorten the sequence length while preserving key acoustic features (such as phoneme boundaries and articulation contours).

[0056] Step S13: Map the downsampled acoustic latent state sequence to the same dimension as the word embedding space of the decoder to obtain the aligned acoustic representation sequence.

[0057] Specifically, since the dimension of acoustic feature vectors is usually different from the word embedding dimension used by large language models (i.e., agent decoders), the modality alignment adapter usually also contains a dimension mapping unit, which can be composed of a multilayer perceptron (MLP) or a linear transform layer.

[0058] The dimension mapping unit uses an MLP or linear transformation layer to spatially project the downsampled acoustic feature vectors. Through linear or nonlinear mapping, the dimension of the acoustic features is transformed to the same dimension as the word embedding space of the agent decoder, thereby outputting the final aligned acoustic representation sequence.

[0059] The method provided in this invention effectively eliminates the physical dimension difference between continuous acoustic modalities and discrete text modalities by introducing a modal alignment adapter that includes a temporal downsampling unit and a dimension mapping unit. This not only reduces the computational complexity caused by long speech sequences in the time dimension, but also achieves the adaptation and connection between acoustic information and the large language model base in the feature dimension.

[0060] Based on any of the above embodiments, step S20, determining whether to trigger a tool call during the autoregressive decoding process, includes: Step S21: During the process of the decoder autoregressive decoding to generate the next lexical, acoustic uncertainty information is determined based on the acoustic representation sequence.

[0061] Specifically, in autoregressive decoding, the decoder generates text token by token. In generating the first... When dealing with a single word, the decoder needs to read the acoustic features of the underlying input. Due to accents, environmental noise, or obscure words in real-world scenarios, the acoustic features may be ambiguous at the physical level. This step aims to evaluate the clarity and certainty of the current pronunciation segment by analyzing the acoustic representation sequence, thereby extracting acoustic uncertainty information to characterize the degree of ambiguity of the current physical speech signal.

[0062] Step S22: Determine semantic entity requirement information based on the historical text sequence generated by the decoder.

[0063] Specifically, in addition to physical acoustic features, the decoder also needs to combine linguistic logic for analysis. The system extracts the historical text sequence consisting of all lexical units generated by the decoder before the current moment and analyzes the current context. Semantic entity demand information refers to whether the current context logically points to a proper noun or entity category that requires specific external knowledge supplementation, such as personal names, place names, or technical terms.

[0064] Step S23: Based on the acoustic uncertainty information and the semantic entity demand information, a joint decision is made to determine whether to trigger a tool call.

[0065] Specifically, in continuous speech stream decoding, not all acoustically ambiguous segments require the invocation of external tools (e.g., unclear pronunciation of ordinary function words does not require tool invocation), and excessive invocation will lead to increased system latency. Therefore, the joint decision-making routing module inside the agent decoder will jointly determine the acoustic uncertainty information at the physical level and the semantic entity requirement information at the logical level. Only when both of them meet the triggering conditions will the system make a decision to trigger tool invocation.

[0066] The method provided in this invention solves the problem of lacking an accurate tool call timing judgment mechanism in the continuous speech stream decoding process by constructing a joint decision-making mechanism based on the uncertainty of acoustic features and the contextual semantic intent. This enables precise and on-demand tool triggering and effectively avoids system delay problems caused by invalid calls.

[0067] Based on any of the above embodiments, in step S21, determining the acoustic uncertainty information according to the acoustic characterization sequence includes: Step S211: Based on the cross-attention mechanism, feature processing is performed on the acoustic representation sequence to obtain the predicted probability of each candidate word in the vocabulary as the next word. Step S212: Based on the predicted probability of each candidate word as the next word, obtain the predicted probability distribution of the next word and calculate the entropy of the predicted probability distribution; In step S213, when the entropy is greater than a preset threshold, it is determined that the acoustic uncertainty information indicates high acoustic uncertainty.

[0068] Specifically, a cross-attention mechanism is adopted inside the decoder, and the mechanism is configured to establish an alignment relationship between the text position currently being generated and the acoustic features at the input end. Specifically, in the autoregressive generation of the th token by the decoder , the decoder first takes the text context hidden state at the current moment as a query vector, and takes the input acoustic representation sequence (that is, the sequence containing the acoustic hidden state at the current alignment moment ) as a key vector and a value vector; then, the dot product between the query vector and each key vector is calculated through the cross-attention mechanism, and normalization processing is performed to obtain the attention weight for the acoustic feature at the current decoding step; subsequently, the attention weight is used to perform weighted summation on the value vectors to obtain the corresponding acoustic context feature vector; finally, the acoustic context feature vector (or its concatenated vector with the text hidden state) is input to the linear mapping layer (such as a fully connected layer) at the end of the decoder, and processed through the softmax activation function to map it to the dimension space identical to the vocabulary of the model, so as to calculate the predicted probability value of each candidate token in the vocabulary as the next output token.

[0069] The system aggregates the calculated predicted probability values of all candidate tokens in the vocabulary to construct the predicted probability distribution corresponding to the next token . Then, for the predicted probability distribution, the system can calculate the entropy of the predicted probability distribution by using the entropy calculation formula in information theory . Here, the entropy is used to measure the uncertainty of the current decoding system at the acoustic level. When the audio pronunciation is clear and the model is confident of the next token, the predicted probability of a certain candidate token will be extremely high, and the overall probability distribution is concentrated, and the entropy calculated at this time is small; conversely, when the pronunciation is blurred due to accents or rare characters, and the model cannot make an accurate choice among a plurality of candidate tokens (such as "xuan" in different Chinese characters), the probability values of the respective candidate tokens are relatively average and not high, the probability distribution tends to be flat, and the calculated entropy becomes larger.

[0070] The system compares the calculated entropy of the predicted probability distribution with a preset threshold (such as ). If it is greater than the preset threshold , it indicates that the current model cannot accurately predict the next token based on the physical acoustic features, and the system accordingly determines that the current acoustic uncertainty information indicates high acoustic uncertainty, that is, the model cannot hear clearly.

[0071] Based on any of the above embodiments, step S22 specifically includes: Step S221: Use a causal self-attention mechanism to extract features from the historical text sequence to obtain the semantic context features of the current context; Step S222: Determine the entity demand vector of the current context based on the semantic context features, and use the entity demand vector as the semantic entity demand information.

[0072] Specifically, the decoder not only focuses on the sound but also on the generated text. In generating the first... each word element During the process, the decoder reads the generated historical text sequence through a causal self-attention mechanism (an attention mechanism that only allows the current position to pay attention to information about its previous positions). This allows us to extract semantic context features of the current context that contain deep grammar and semantic relationships.

[0073] Based on the semantic context features extracted above, the system extracts the entity demand vector for the current context through an internal mapping layer or classification head. This vector is used to characterize the degree of demand for a specific knowledge entity at the current location. For example, if the historical text sequence is "make a phone call", after the system analyzes the context, the extracted entity demand vector will strongly indicate that a "person's name" entity is needed subsequently. This vector serves as the semantic entity demand information for subsequent joint decision-making.

[0074] Based on any of the above embodiments, step S23 specifically includes: When the acoustic uncertainty information indicates high acoustic uncertainty and the semantic entity demand information indicates the existence of a target entity demand, the generation probability of regular text lexical units is suppressed, and a tool call control character is output to determine the trigger for tool call.

[0075] Specifically, in this step, the joint decision routing module executes specific judgment logic, when the entropy calculated by the cross-attention side... Greater than (i.e., there is high acoustic uncertainty), and the entity demand vector extracted from the attention side When a system indicates a strong current need for a specific target entity (such as a contact's name or technical terminology), it determines that the joint triggering condition is met. At this point, the model actively intervenes in its generation process, suppressing the generation probability of regular text terms from the original vocabulary (such as specific candidate words from a Chinese character database), and forcing the model to output a specific tool invocation control character, such as...<tool_call> This control character is a special marker independent of the regular language system, specifically used to send a signal to an external state manager to pause and invoke tools.

[0076] Based on any of the above embodiments, step S30, which involves suspending the autoregressive decoding and freezing the current decoding state information of the decoder, includes: Step S31: In response to the output of the tool call control character, pause the reading of subsequent audio frames in the audio stream and the autoregressive calculation of the decoder; Step S32: Freeze the key-value cache and hidden state of each layer of the decoder, and save the current audio frame alignment timestamp; The decoding status information includes the key-value cache, the hidden state, and the audio frame alignment timestamp.

[0077] Specifically, when the decoder's joint decision routing module forces the model to output control symbols<tool_call> Subsequently, the system's state manager immediately intervenes and captures the control character. In response to this control character, the state manager issues a blocking command to the front-end audio processing module, suspending the input of subsequent audio stream data frames to the decoder, and simultaneously suspending the decoder's regular computation process for the plain text autoregressive task. This operation ensures that, while waiting for external knowledge to be returned, the model will not continue to consume and miscode subsequent consecutive audio frames based on uncertain and ambiguous pronunciation.

[0078] While pausing reading and computation, the state manager simultaneously executes a state protection mechanism. The system freezes the key-value cache (KV-Cache) and hidden states in all network layers of the current decoder. The key-value cache and hidden states retain all acoustic feature context and generated text semantic context from the audio start position to the current trigger position. Freezing them ensures that critical acoustic alignment features are not released or overwritten during the interruption. Furthermore, the system saves the current audio frame alignment timestamp, which records the specific physical time point or frame index position corresponding to when the model stopped decoding, providing accurate temporal alignment coordinates for subsequent decoding resumption.

[0079] The method provided in this invention solves the engineering problem that the tool calling mechanism of large model agents is difficult to apply directly to continuous audio streams by pausing the calculation and saving the key value cache, hidden state and audio frame alignment timestamp, and ensures the continuity of the decoding process and the integrity of the acoustic context after being interrupted by external calls.

[0080] Based on any of the above embodiments, in step S30, the tool invocation request includes: In the suspended state, a structured tool call request is generated based on the fuzzy pronunciation information corresponding to the acoustic representation sequence when the tool is triggered and the entity demand vector. The tool call request carries the tool name and query parameters, and the query parameters are determined according to the fuzzy pronunciation information.

[0081] Specifically, while the decoding process is suspended and regular autoregressive computation is paused, the model uses the context information obtained from the last computation to perform a one-time request generation operation. Specifically, the model extracts the fuzzy pronunciation information (such as the corresponding pinyin sequence or phoneme distribution) captured by the cross-attention layer when the tool call is triggered, and combines it with the entity demand vector extracted by the causal self-attention layer to output a structured tool call request in a preset format (such as JSON). This request explicitly includes the target tool's tool name (tool_name) and the specific query parameters (args).

[0082] Taking an in-vehicle voice assistant scenario as an example, when a user's intention is "to call Li Mou-xuan in the R&D department," but the pronunciation of "Xuan" and "Xuan" is unclear due to accent, the model outputs a control character when it encounters this point of ambiguity.<tool_call> Then it enters a suspended state. Subsequently, based on the preceding semantics (R&D Department) and the current fuzzy pinyin, the model generates a structured tool call request, for example, generating request data in the following format: {"tool_name": "SearchContact", "args": {"pinyin": "li mou x / xuan", "dept": "R&D Department"}}. In this request, the pinyin range in the query parameters is determined based on the fuzzy pronunciation information at the acoustic level.

[0083] The method provided in this invention, in the decoder suspension state, effectively transforms the underlying acoustic fuzzy features with the high-level semantic context logic to generate a structured tool call request with clear directionality. This not only realizes cross-modal parameter transmission between physical audio signals and external tools, but also improves the recall accuracy of external retrieval tools when faced with accents or rare characters by providing query parameters containing fuzzy pronunciations.

[0084] Based on any of the above embodiments, step S40 specifically includes: The tool invocation request is sent to an external tool execution environment, and the tool invocation request is routed to the target external tool through the routing gateway in the external tool execution environment, while simultaneously waiting for the invocation result returned by the target external tool; The external tools include at least one of an address book interface, a map interface, and a knowledge base retrieval engine.

[0085] Specifically, after generating a structured tool invocation request during the decoding suspension period, the system sends it to an external tool execution environment independent of the decoder. This execution environment deploys a routing gateway responsible for parsing the tool name carried in the request and accurately routing the query parameters to the corresponding target external tool based on that name. After the request is sent, the entire speech recognition system enters a synchronous waiting state, maintaining the currently frozen decoder context until the target external tool completes execution and returns the final query data (i.e., the invocation result).

[0086] To address the challenges of long-tail entity recognition across various industries and scenarios, external tool execution environments integrate multiple types of APIs or search engines. For example, in smart vehicle and local life assistant scenarios, external tools may include a contact API for querying private contact data, and a map API for querying long-tail points of interest (POIs) or shop names. In dictation scenarios in professional fields such as healthcare, law, or finance, external tools may include a knowledge base search engine for real-time retrieval of rare drug names, legal provisions, or dynamic financial product codes.

[0087] Based on any of the above embodiments, step S50 specifically includes: Step S51: Encapsulate the call result into a result word sequence defined by a result identifier, and input the result word sequence as a context prefix into the decoder.

[0088] Specifically, after the external tool returns the exact result of the call (e.g., the retrieved contact name "Li Mou Xuan"), the system first performs a result transformation operation. Specifically, the system encapsulates the plain text result using specific start and end control tags (i.e., result identifiers), encapsulating it into a sequence of result tokens defined by the result identifiers, for example, formatted as...<tool_result> Li Mou Xuan< / tool_result> The token sequence is then used by the state manager as a correct context prefix, which is directly injected into the suspended decoder as a basis for the historical text generated by the model.

[0089] Step S52: Resume the suspended autoregressive decoding, and based on the frozen key-value cache and the hidden state, continue to predict the text words following the result word sequence using the result word sequence as a prefix constraint.

[0090] Specifically, in this step, the state manager performs a system wake-up operation. After receiving the aforementioned result token sequence, the decoder reactivates the previously frozen and saved key-value cache and hidden states. At this point, the decoder no longer relies on its own ambiguous acoustic predictions, but uses the externally given exact result token sequence as a prefix constraint. Through the cross-attention mechanism, the decoder realigns and fuses this forcibly injected text constraint with the previously frozen underlying acoustic hidden states, corrects the acoustic search path at positions of ambiguous pronunciation (such as the ambiguity between the Chinese characters "Xuan"), and on this basis continues to perform normal autoregressive calculation to predict normal text tokens after the result identifier.

[0091] Step S53, according to the saved audio frame alignment timestamp, resume reading subsequent audio frames in the audio stream, and perform alignment decoding on the subsequent audio frames.

[0092] Specifically, while the text generation calculation is resumed, the audio processing module at the front end of the system also resumes operation synchronously. The system finds the physical truncation position where the audio stream is blocked according to the audio frame alignment timestamp saved at the time of suspension, and restarts reading of subsequent unprocessed continuous audio frames. Since the timestamp records the accurate timing position, the external discrete text result can achieve timing alignment with the re-input continuous acoustic features. The decoder then processes the remaining audio context until the entire audio stream is decoded, and finally outputs the complete and correct recognized text.

[0093] The method provided by the embodiments of the present invention solves the problems of seamless wake-up and timing alignment after the return of external discrete text results through a specific result encapsulation and prefix constraint decoding mechanism, combined with the frozen neural network state and timestamp.

[0094] Based on any of the above embodiments, the embodiments of the present invention provide a speech recognition method for a tool-calling agent driven by the combination of acoustics and semantics. The method can be applied to a speech recognition system for a tool-calling agent driven by the combination of acoustics and semantics. The system architecture and method execution flow of the embodiments of the present invention will be described in detail below.

[0095] Figure 2 is a schematic structural diagram of the speech recognition system for a tool-calling agent driven by the combination of acoustics and semantics provided by the present invention, as shown in Figure 2 the figure, the system mainly includes five core modules, namely an audio encoder, a modal alignment adapter, an agent decoder, a state manager, and an external tool execution environment. These five modules cooperate with each other to jointly complete the complete closed loop from continuous speech input to dynamic tool call and then to high-precision text output.

[0096] The audio encoder converts the input continuous audio stream into a high-dimensional acoustic latent state representation. The modality alignment adapter is positioned between the audio encoder and the agent decoder to eliminate dimensional differences between modalities and reduce computational complexity. The agent decoder, based on a large language model, combines acoustic features with historical text semantics to autoregressively generate recognition text or tool invocation instructions, and includes a joint decision routing module. The state manager is responsible for freezing the decoder's KV-Cache and hidden state during tool invocation and waking up the decoding process after the result is returned. The external tool execution environment includes a routing gateway and multiple external tools, such as an address book API, a map API, and an enterprise knowledge base retrieval engine.

[0097] Figure 3 This is a flowchart illustrating the process of a tool calling an intelligent agent speech recognition method based on joint acoustic and semantic driving, as provided in this invention. Figure 3 As shown, the core decoding process of this method is essentially a state machine loop, which specifically includes the following steps: Step S1: Audio input and feature extraction.

[0098] The system first acquires the user's continuous audio stream, which serves as the initial input to the entire system. After frame-segmentation and windowing, the audio stream is input to the front-end audio encoder. The audio encoder extracts acoustic features at the physical level, converting the original continuous analog signal or discrete digital audio stream into a high-dimensional acoustic latent state sequence containing rich acoustic feature information.

[0099] Step S2 involves downsampling and dimension mapping of the acoustic hidden state sequence.

[0100] Subsequently, the data flows into the modality alignment adapter, which downsamples and maps the acoustic latent state sequence to dimensions. Since the high-dimensional acoustic latent state sequence has a high frame rate and its dimensions do not match the base of the large language model, the modality alignment adapter shortens the sequence length by downsampling in the temporal dimension (e.g., using convolutional layers) and projects it into the feature space using structures such as multilayer perceptrons. This aligns the acoustic feature sequence with the word embedding space of the subsequent agent decoder in terms of dimensions, resulting in an aligned acoustic representation sequence.

[0101] Step S3: Decoding and triggering judgment driven by both acoustics and semantics.

[0102] The aligned acoustic representation sequence enters the agent decoder. During the autoregressive decoding process to generate text, the decoder calculates the acoustic probability prediction entropy in real time and extracts semantic entity requirements. That is, on the one hand, it calculates the entropy value of the probability distribution of the next predicted word through a cross-attention mechanism to assess the ambiguity of the acoustic features, and on the other hand, it reads historical text through a causal self-attention mechanism to extract entity requirements from the context.

[0103] Next, the joint decision-making routing module within the agent decoder enters the decision-making branch, which determines whether both high acoustic uncertainty and a specific entity requirement are simultaneously satisfied. If the result is negative (e.g., the pronunciation is clear, or the pronunciation is unclear but the context does not require a specific entity), the system outputs the recognized text token along the normal decoding path and continues with subsequent decoding tasks. If the result is positive (i.e., both high acoustic uncertainty and a specific entity requirement are satisfied), the system executes the operation of forcibly outputting a specific control character along the intervention path.

[0104] Step S4: Decode the suspended and structured request generation.

[0105] When a specific control character (such as<tool_call> Upon output, the state manager immediately responds and takes over the process, suspending decoding, freezing the KV-Cache and hidden states, and generating a tool invocation request. Specifically, the state manager pauses the reading of the current audio stream and the autoregressive computation of the decoder, freezes the KV-Cache and hidden states of all layers of the current decoder, and saves the current audio frame alignment timestamp. Simultaneously, the agent decoder, based on the previous acoustic ambiguity and semantic intent, generates a structured tool invocation request containing query parameters and sends it to the backend external tool execution environment.

[0106] Step S5: External tools are executed synchronously.

[0107] Subsequently, the external tool is triggered to execute and returns the call result. The generated structured tool call request is sent to the external tool execution environment. For example, the pinyin and fuzzy pronunciation parameters are sent to the address book retrieval API. The system synchronously waits for the query results from the external tool.

[0108] Step S6: Result injection and decoding recovery.

[0109] After obtaining the result, the state manager and the intelligent agent decoder work together to convert the result into a token sequence as a prefix injection, waking up and resuming the decoding process. Specifically, the text result returned from the outside is encapsulated into a result token sequence after being added with a specific identifier, and injected into the decoder as the correct prefix constraint. At the same time, the state manager wakes up the suspended decoding process.

[0110] Finally, the agent decoder combines the contextual hidden state and the result to perform the next decoding. The decoder uses the injected accurate result to correct previous acoustic ambiguities and continues to read subsequent audio features based on the recovered temporal position until the alignment decoding of the entire audio stream is completed.

[0111] To facilitate a clear understanding of the overall mechanism of the above method, a specific example is provided below. Taking contact recognition in the context of an in-vehicle voice assistant as an example, the user's true intention is to say to the car's system, "Call Li Mou-xuan in the R&D department" (the user has a regional accent, and the pronunciation of "Xuan" and "Xuan" is unclear).

[0112] In traditional ASR (Automatic Sentence Recognition), the acoustic features are blurred due to accents, and "Li Mouxuan" is also a legitimate name. Traditional ASR might directly decode and output "Call Li Mouxuan in the R&D department," then execute the dialing, calling the wrong person. Even with subsequent text correction, seeing the three characters "Li Mouxuan" would not be enough to determine the correctness of the message because of the lack of acoustic cues.

[0113] In the execution of the method of this invention, the model first reads the audio and successfully decodes the text "make a phone call". When the audio reaches "Li Mou Xuan", due to the similarity between the accent and pronunciation, the feature output by the acoustic encoder oscillates between "Xuan" and "Xuan", and the calculated prediction probability entropy is obtained. Suddenly increased (exceeding the threshold) At this point, the joint decision-making routing module discovers that, on the one hand, the acoustic uncertainty is high (unclear sound), and on the other hand, the semantic context... A strong indication is that a "contact entity" is needed subsequently, as the preceding semantics are "make a phone call." Therefore, the model needs to utilize an external address book for its judgment, and thus stops outputting "Li Mouxuan," instead outputting a specific control character.<tool_call> The state manager immediately freezes the current neural network cache, and the model then generates a call request: {"tool_name": "SearchContact", "args": {"pinyin": "li mou x / xuan", "dept": "R&D Department"}} (the model generates query parameters based on the fuzzy sound).

[0114] Subsequently, the vehicle system queried the local address book API and found that the address book only contained "Li Mou Xuan (R&D Department)" and not "Li Mou Xuan". Therefore, it returned the string "Li Mou Xuan". The system then packaged "Li Mou Xuan" into...<tool_result> Li Mou Xuan< / tool_result> The text is sent back to the decoder, which is then awakened. Using the externally provided text constraint "Li Mou Xuan", the decoder corrects the originally ambiguous acoustic path and then outputs the subsequent text (if there is any). Finally, it outputs the complete and correct transcribed text, which is "Call Li Mou Xuan from the R&D department".

[0115] In this embodiment of the invention, the model actively pauses decoding to query externally the moment it encounters a physical acoustic obstacle during decoding, and uses the underlying state machine to maintain the original audio context without loss. This achieves true listening, querying and correction at the same time, avoiding the predicament of correcting errors after they occur in traditional technologies.

[0116] The speech recognition device based on tool invocation provided by the present invention will be described below. The speech recognition device based on tool invocation described below can be referred to in correspondence with the speech recognition method based on tool invocation described above.

[0117] Based on any of the above embodiments Figure 4 This is a schematic diagram of the structure of the speech recognition device based on tool invocation provided by the present invention, as shown below. Figure 4 As shown, the device includes: The feature extraction module 410 is used to extract features from the audio stream to be identified, and obtain an acoustic representation sequence. The regression decoding module 420 is used to input the acoustic representation sequence into the decoder for autoregressive decoding, determine whether a tool call is triggered during the autoregressive decoding process, and generate a tool call request if it is determined that a call rejection is triggered. The state management module 430 is used to suspend the autoregressive decoding and freeze the current decoding state information of the decoder when it is determined that a tool call has been triggered. The tool invocation module 440 is used to invoke an external tool based on the tool invocation request and obtain the invocation result returned by the external tool; The state management module 430 is also used to restore the autoregressive decoding based on the frozen decoding state information after obtaining the call result; The regression decoding module 420 is also used to input the call result into the decoder to continue autoregressive decoding and obtain the recognition text corresponding to the audio stream.

[0118] The apparatus provided in this invention, during the autoregressive decoding process of inputting acoustic representation sequences into the decoder, avoids the system response delay problem caused by the traditional solution requiring full decoding before processing by determining in real time whether a tool call is triggered. When a tool call is triggered, this invention can suspend the current autoregressive decoding process and freeze the decoder's decoding state information. This freezing operation allows the decoder to fully retain the extracted key acoustic features and context state while waiting for external knowledge injection, preventing the decoder from continuing to force autoregressive prediction when facing uncertain acoustic features, thereby preventing decoding path offset and error cascading problems caused by early acoustic feature ambiguity. Subsequently, by obtaining accurate external call results based on the request to call external tools and inputting these results into the decoder, combined with the previously frozen decoding state information, autoregressive decoding is resumed. This recovery mechanism based on the frozen state can seamlessly stitch the externally returned results with the acoustic context before suspension, thereby ensuring the temporal alignment of the continuous audio stream and guaranteeing the continuity and accuracy of the decoding process.

[0119] Based on any of the above embodiments, the regression decoding module includes: An acoustic information determination unit is used to determine acoustic uncertainty information based on the acoustic representation sequence during the process of the decoder generating the next word through autoregressive decoding. An entity requirement determination unit is used to determine semantic entity requirement information based on the historical text sequence generated by the decoder. The joint decision-making unit is used to make joint decisions based on the acoustic uncertainty information and the semantic entity demand information to determine whether to trigger tool invocation.

[0120] Based on any of the above embodiments, the acoustic information determination unit is specifically used for: The acoustic representation sequence is processed based on the cross-attention mechanism to obtain the predicted probability of each candidate word in the vocabulary as the next word. Based on the predicted probability of each candidate word as the next word, the predicted probability distribution of the next word is obtained, and the entropy of the predicted probability distribution is calculated. If the entropy is greater than a preset threshold, the acoustic uncertainty information is determined to indicate high acoustic uncertainty.

[0121] Based on any of the above embodiments, the entity requirement determination unit is specifically used for: The historical text sequence is used to extract features by employing a causal self-attention mechanism to obtain the semantic context features of the current context; The entity demand vector of the current context is determined based on the semantic context features, and the entity demand vector is used as the semantic entity demand information. The joint decision-making unit is specifically used for: When the acoustic uncertainty information indicates high acoustic uncertainty and the semantic entity demand information indicates the existence of a target entity demand, the generation probability of regular text lexical units is suppressed, and a tool call control character is output to determine the trigger for tool call.

[0122] Based on any of the above embodiments, the state management module is specifically used for: In response to the output of the tool call control character, the reading of subsequent audio frames in the audio stream and the autoregressive calculation of the decoder are paused; Freeze the key-value caches and hidden states of each layer of the decoder, and save the current audio frame alignment timestamp; The decoding status information includes the key-value cache, the hidden state, and the audio frame alignment timestamp.

[0123] Based on any of the above embodiments, the regression decoding module further includes a request generation unit, which is used for: In the suspended state, a structured tool call request is generated based on the fuzzy pronunciation information corresponding to the acoustic representation sequence when the tool is triggered and the entity demand vector. The tool call request carries the tool name and query parameters, and the query parameters are determined according to the fuzzy pronunciation information.

[0124] Based on any of the above embodiments, the regression decoding module is further configured to encapsulate the call result into a result word sequence defined by a result identifier, and input the result word sequence as a context prefix into the decoder; The state management module is also used to resume the suspended autoregressive decoding; The regression decoding module is also used to predict text words following the result word sequence based on the frozen key-value cache and the hidden state, using the result word sequence as a prefix constraint; and to restore the reading of subsequent audio frames in the audio stream according to the saved audio frame alignment timestamp, and to perform alignment decoding on the subsequent audio frames.

[0125] Based on any of the above embodiments, the feature extraction module is specifically used for: After the audio stream is processed into frames, it is input into the audio encoder to obtain the acoustic hidden state sequence corresponding to the audio frame; The acoustic hidden state sequence is downsampled in the time dimension to obtain the downsampled acoustic hidden state sequence; The downsampled acoustic latent state sequence is mapped to the same dimension as the word embedding space of the decoder to obtain the aligned acoustic representation sequence.

[0126] Based on any of the above embodiments, the tool invocation module is specifically used for: The tool invocation request is sent to an external tool execution environment, and the tool invocation request is routed to the target external tool through the routing gateway in the external tool execution environment, while simultaneously waiting for the invocation result returned by the target external tool; The external tools include at least one of an address book interface, a map interface, and a knowledge base retrieval engine.

[0127] Figure 5 This is a schematic diagram of the structure of the electronic device provided by the present invention, such as... Figure 5 As shown, the electronic device may include a processor 510, a communication interface 520, a memory 530, and a communication bus 540, wherein the processor 510, the communication interface 520, and the memory 530 communicate with each other via the communication bus 540. The processor 510 can call logical instructions in the memory 530 to execute a tool-invoking-based speech recognition method. This method includes: extracting features from the audio stream to be recognized to obtain an acoustic representation sequence; inputting the acoustic representation sequence into a decoder for autoregressive decoding, and determining whether a tool invoking is triggered during the autoregressive decoding process; if a tool invoking is triggered, suspending the autoregressive decoding, freezing the current decoding state information of the decoder, and generating a tool invoking request; invoking an external tool based on the tool invoking request, and obtaining the invoking result returned by the external tool; resuming the autoregressive decoding based on the frozen decoding state information, and inputting the invoking result into the decoder to continue autoregressive decoding to obtain the recognized text corresponding to the audio stream.

[0128] Furthermore, the logical instructions in the aforementioned memory 530 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to related technologies, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0129] On the other hand, the present invention also provides a computer program product, which includes a computer program that can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer is able to execute the speech recognition method based on tool invocation provided by the above methods. The method includes: extracting features from the audio stream to be recognized to obtain an acoustic representation sequence; inputting the acoustic representation sequence into a decoder for autoregressive decoding, and determining whether a tool invocation is triggered during the autoregressive decoding process; if a tool invocation is triggered, suspending the autoregressive decoding, freezing the current decoding state information of the decoder, and generating a tool invocation request; invoking an external tool based on the tool invocation request and obtaining the invocation result returned by the external tool; resuming the autoregressive decoding based on the frozen decoding state information, and inputting the invocation result into the decoder to continue autoregressive decoding to obtain the recognized text corresponding to the audio stream.

[0130] In another aspect, the present invention also provides a non-transitory computer-readable storage medium storing a computer program thereon. When executed by a processor, the computer program implements the speech recognition method based on tool invocation provided by the above methods. The method includes: extracting features from an audio stream to be recognized to obtain an acoustic representation sequence; inputting the acoustic representation sequence into a decoder for autoregressive decoding, and determining whether a tool invocation is triggered during the autoregressive decoding process; if a tool invocation is triggered, suspending the autoregressive decoding, freezing the current decoding state information of the decoder, and generating a tool invocation request; invoking an external tool based on the tool invocation request, and obtaining the invocation result returned by the external tool; resuming the autoregressive decoding based on the frozen decoding state information, and inputting the invocation result into the decoder to continue autoregressive decoding to obtain the recognized text corresponding to the audio stream.

[0131] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0132] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the parts that contribute to the related technology, can be embodied in the form of software products. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0133] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A speech recognition method based on tool invocation, characterized in that, include: Feature extraction is performed on the audio stream to be identified to obtain an acoustic representation sequence; The acoustic representation sequence is input into the decoder for autoregressive decoding, and it is determined whether a tool call is triggered during the autoregressive decoding process; If a tool call is triggered, the autoregressive decoding is suspended, the current decoding state information of the decoder is frozen, and a tool call request is generated. Based on the tool call request, an external tool is invoked, and the call result returned by the external tool is obtained; The autoregressive decoding is restored based on the frozen decoding state information, and the call result is input into the decoder to continue autoregressive decoding to obtain the recognition text corresponding to the audio stream.

2. The speech recognition method based on tool invocation according to claim 1, characterized in that, Determining whether to trigger a tool call during the autoregressive decoding process includes: During the process of generating the next word through autoregressive decoding in the decoder, acoustic uncertainty information is determined based on the acoustic representation sequence; Based on the historical text sequence generated by the decoder, determine the semantic entity requirement information; A joint decision is made based on the acoustic uncertainty information and the semantic entity demand information to determine whether to trigger a tool call.

3. The speech recognition method based on tool invocation according to claim 2, characterized in that, The step of determining acoustic uncertainty information based on the acoustic characterization sequence includes: The acoustic representation sequence is processed based on the cross-attention mechanism to obtain the predicted probability of each candidate word in the vocabulary as the next word. Based on the predicted probability of each candidate word as the next word, the predicted probability distribution of the next word is obtained, and the entropy of the predicted probability distribution is calculated. If the entropy is greater than a preset threshold, the acoustic uncertainty information is determined to indicate high acoustic uncertainty.

4. The speech recognition method based on tool invocation according to claim 2, characterized in that, The step of determining semantic entity requirement information based on the historical text sequence generated by the decoder includes: The historical text sequence is used to extract features by employing a causal self-attention mechanism to obtain the semantic context features of the current context; The entity demand vector of the current context is determined based on the semantic context features, and the entity demand vector is used as the semantic entity demand information. The joint decision-making process based on the acoustic uncertainty information and the semantic entity demand information to determine whether to trigger a tool call includes: When the acoustic uncertainty information indicates high acoustic uncertainty and the semantic entity demand information indicates the existence of a target entity demand, the generation probability of regular text lexical units is suppressed, and a tool call control character is output to determine the trigger for tool call.

5. The speech recognition method based on tool invocation according to claim 4, characterized in that, The step of suspending the autoregressive decoding and freezing the current decoding state information of the decoder includes: In response to the output of the tool call control character, the reading of subsequent audio frames in the audio stream and the autoregressive calculation of the decoder are paused; Freeze the key-value caches and hidden states of each layer of the decoder, and save the current audio frame alignment timestamp; The decoding status information includes the key-value cache, the hidden state, and the audio frame alignment timestamp.

6. The speech recognition method based on tool invocation according to claim 4, characterized in that, The generation tool invocation request includes: In the suspended state, a structured tool call request is generated based on the fuzzy pronunciation information corresponding to the acoustic representation sequence when the tool is triggered and the entity demand vector. The tool call request carries the tool name and query parameters, and the query parameters are determined according to the fuzzy pronunciation information.

7. The speech recognition method based on tool invocation according to claim 5, characterized in that, The process of restoring the autoregressive decoding based on the frozen decoding state information and inputting the call result into the decoder to continue autoregressive decoding includes: The call result is encapsulated into a result word sequence delimited by a result identifier, and the result word sequence is input into the decoder as a context prefix; Resume the suspended autoregressive decoding, based on the frozen key-value cache and the hidden state, and continue to predict text words following the result word sequence as a prefix constraint; Based on the saved audio frame alignment timestamps, the reading of subsequent audio frames in the audio stream is resumed, and the subsequent audio frames are aligned and decoded.

8. The speech recognition method based on tool invocation according to any one of claims 1 to 7, characterized in that, The process of extracting features from the audio stream to be identified to obtain an acoustic representation sequence includes: After the audio stream is processed into frames, it is input into the audio encoder to obtain the acoustic hidden state sequence corresponding to the audio frame; The acoustic hidden state sequence is downsampled in the time dimension to obtain the downsampled acoustic hidden state sequence; The downsampled acoustic latent state sequence is mapped to the same dimension as the word embedding space of the decoder to obtain the aligned acoustic representation sequence.

9. The speech recognition method based on tool invocation according to any one of claims 1 to 7, characterized in that, The step of invoking an external tool based on the tool invocation request and obtaining the invocation result returned by the external tool includes: The tool invocation request is sent to an external tool execution environment, and the tool invocation request is routed to the target external tool through the routing gateway in the external tool execution environment, while simultaneously waiting for the invocation result returned by the target external tool; The external tools include at least one of an address book interface, a map interface, and a knowledge base retrieval engine.

10. A speech recognition device based on tool invocation, characterized in that, include: The feature extraction module is used to extract features from the audio stream to be identified, and obtain an acoustic representation sequence. The regression decoding module is used to input the acoustic representation sequence into the decoder for autoregressive decoding, determine whether a tool call is triggered during the autoregressive decoding process, and generate a tool call request if it is determined that a call denial is triggered. The state management module is used to suspend the autoregressive decoding and freeze the current decoding state information of the decoder when it is determined that a tool call has been triggered. The tool invocation module is used to invoke an external tool based on the tool invocation request and obtain the invocation result returned by the external tool; The state management module is also used to restore the autoregressive decoding based on the frozen decoding state information after obtaining the call result; The regression decoding module is also used to input the call result into the decoder to continue autoregressive decoding to obtain the recognition text corresponding to the audio stream.

11. An electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that, When the processor executes the computer program, it implements the speech recognition method based on tool invocation as described in any one of claims 1 to 9.

12. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the speech recognition method based on tool invocation as described in any one of claims 1 to 9.

13. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by the processor, it implements the speech recognition method based on tool invocation as described in any one of claims 1 to 9.

Images