A speech understanding shunt method based on speech necessity judgment

Abstract

This invention discloses a speech understanding triage method based on speech necessity judgment, belonging to the field of speech recognition technology. To address the problems of existing technologies lacking pre-judgment of the necessity of preserving the original speech and struggling to balance speech understanding accuracy and processing cost, this invention performs automatic speech recognition on the input speech and extracts prosodic acoustic features. Based on recognition confidence, prosodic acoustic features, and text uncertainty, it determines whether the current input speech must rely on the prosodic and acoustic information in the original speech, thereby determining whether to use a text transcription model path or an end-to-end speech model path, and outputs the speech understanding result based on the corresponding path result. This invention can control processing costs while ensuring speech understanding accuracy, achieving a beneficial effect that balances understanding accuracy and efficiency. It is applicable to fields such as intelligent terminal voice interaction, speech recognition, natural language understanding, and intelligent human-computer interaction.

Description

Technical Field

[0001] This invention belongs to the field of speech recognition technology, specifically involving a speech understanding and diversion technology based on speech necessity judgment. Background Technology

[0002] With the development of voice interaction on smart terminals, existing technologies typically convert speech to text first, and then complete semantic understanding and response generation based on the text. This type of solution can improve the efficiency of voice interaction and is suitable for scenarios such as conventional command recognition and question answering. However, its processing focus is mainly on transcribing the text. For the problem that the same transcribed text may express different meanings under different intonations, pauses, stresses, and speech rates, it often lacks a pre-judgment mechanism to determine whether the original speech prosodic information must be retained.

[0003] On the other hand, existing technologies also include optimization schemes based on the quality of the recognized text, such as improving processing performance through recognition confidence correction, recognition result filtering, or text classification. These schemes can improve the reliability of recognition results and the accuracy of text processing to a certain extent, but their technical focus remains on recognizing the text itself. They fail to address the problem of how to determine whether the current input still relies on the prosody and acoustic information in the original speech for correct understanding during the speech comprehension stage, nor do they provide technical means to differentiate between different comprehension paths based on this judgment result.

[0004] Furthermore, existing technologies also include solutions that directly utilize raw speech for sentiment analysis, attitude recognition, or multi-turn contextual understanding. While these solutions can enhance the understanding of complex speech, they typically assume a relatively complete speech analysis or contextual processing chain for the input speech, resulting in high processing costs, and the same high-cost processing is applied to simple samples. Therefore, a speech understanding offloading technology is still needed that can first determine the necessity of the speech before deciding whether to use a text transcription model path or an end-to-end speech model path. Summary of the Invention

[0005] To address the problems in existing technologies, such as the lack of pre-judgment regarding the necessity of preserving original speech and the difficulty in balancing speech understanding accuracy and processing cost, this invention proposes the following solution: A speech understanding triage method based on speech necessity judgment, the method comprising: S1. Receive input speech for speech understanding processing in a smart terminal scenario, perform automatic speech recognition on the input speech to obtain transcribed text and recognition confidence, and extract prosodic acoustic features from the input speech; S2. Input the transcribed text into a text agent for initial text judgment to obtain text uncertainty; S3. Based on recognition confidence, prosodic acoustic features and text uncertainty, determine whether the current input speech must rely on the prosodic and acoustic information in the original speech to make a correct judgment, and obtain the speech necessity judgment result. S4. Based on the speech necessity judgment result, determine whether to use a transcription text model path or an end-to-end speech model path for the input speech; S5. When the input speech is processed using a transcription text model path, the transcription text model path result obtained based on the transcribed text is output; when the input speech is processed using an end-to-end speech model path, the end-to-end speech model is invoked, and the end-to-end speech model path result is generated based on the input speech. S6. Output the speech understanding result based on the transcribed text model path result or the end-to-end speech model path result.

[0006] Furthermore, the prosodic acoustic features described in S1 include acoustic representations of pitch, energy, duration, pauses, and depth.

[0007] Furthermore, the speech necessity judgment result described in S3 includes a speech necessity score, a speech necessity level, or a classification result indicating whether or not the original speech is required.

[0008] Furthermore, the method described in S3 for obtaining the speech necessity judgment result is as follows: combining the confidence and uncertainty of the text agent, ASR recognition quality, pause position, stress distribution, speech rate change, pitch trend, energy change, and the degree of divergence between different agents, to determine whether the current input speech must rely on the prosody and acoustic information in the original speech to make a correct judgment.

[0009] Further, S4, determining whether to use a transcription text model path or an end-to-end speech model path for the input speech, includes: when the speech necessity judgment result indicates that it does not need to rely on the original speech, determining to use a transcription text model path for the input speech; when the speech necessity judgment result indicates that it must rely on the original speech, determining to use an end-to-end speech model path for the input speech.

[0010] Furthermore, in S4, when the speech necessity judgment result indicates that it must rely on the original speech, or the text uncertainty indicates that there are multiple interpretations, or there is a conflict between different pieces of evidence, an end-to-end speech model path is determined for the input speech.

[0011] Furthermore, the method also includes: after adopting the end-to-end speech model path, if the result of the transcribing text model path is consistent with the result of the end-to-end speech model path, then the processing ends; if the two path results still have obvious conflicts, then the context agent and the verification agent are called to complete the final judgment.

[0012] Furthermore, the method determines whether to use a transcription text model path or an end-to-end speech model path for the input speech, provided that cost control requirements are met. The cost control requirements are manifested in token cost, inference steps, number of interface calls, processing latency, or a combination of these quantities.

[0013] Furthermore, the method also includes early termination and cost checks; when the results of a certain stage have reached the preset reliability requirements, or when different agents have formed a stable and consistent judgment, subsequent processing is stopped; when the remaining cost space is insufficient to support the next round of analysis, processing is terminated.

[0014] Based on the same inventive concept, the present invention also proposes a computer storage medium on which a computer program is stored, wherein the computer program executes the above-described method when running on a processor.

[0015] Compared with the prior art, the present invention has the following beneficial effects: This invention performs automatic speech recognition on input speech and extracts prosodic acoustic features, enabling speech understanding processing to simultaneously incorporate transcribed text information and original speech information, thus providing a foundation for subsequent speech necessity determination. Compared with existing technologies, it addresses the problem of processing only the transcribed text while neglecting the prosodic and acoustic information in the original speech.

[0016] This invention determines the necessity of speech based on recognition confidence, prosodic acoustic features, and textual uncertainty, thus explicitly representing the need to preserve the original speech and enabling preliminary judgment before speech comprehension. Compared to existing technologies, this invention addresses the problem of lacking preliminary judgment on the necessity of preserving the original speech.

[0017] This invention determines whether to use a transcription model or an end-to-end speech model for the input speech based on the speech necessity judgment result, allowing different input speech to enter the corresponding understanding path. This ensures the accuracy of speech understanding, which relies on prosody and acoustic information, while controlling processing costs. Compared with existing technologies, it improves upon the problem of uniformly using the same processing path for both simple and complex samples.

[0018] This invention improves the reliability of speech understanding results and reduces unnecessary processing overhead by invoking the context agent and the verification agent to make the final judgment when there is a conflict in the path results. Through cost control, early termination and cost checking, the processing in complex speech understanding scenarios can be further corrected and terminated in a timely manner.

[0019] This invention features speech understanding triage based on speech necessity judgment, which can improve the processing capability of speech understanding tasks that rely on prosody and acoustic information while taking into account both the accuracy of speech understanding and processing cost. It is applicable to the fields of intelligent terminal voice interaction, speech recognition, natural language understanding and intelligent human-computer interaction. Attached Figure Description

[0020] Figure 1 This is a flowchart of the speech understanding triage method based on speech necessity judgment as described in the implementation method. Detailed Implementation

[0021] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present invention, and not all of them. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0022] Implementation Method 1 like Figure 1 As shown, this embodiment proposes a speech understanding triage method based on speech necessity judgment, including receiving input speech and performing automatic speech recognition and prosodic acoustic feature extraction, performing initial text judgment on the transcribed text and obtaining text uncertainty, judging speech necessity based on recognition confidence, prosodic acoustic features and text uncertainty, determining the processing path based on the speech necessity judgment result, generating the corresponding path result according to the determined path, and outputting the speech understanding result, as detailed below: S1. Receive input speech for speech understanding processing in a smart terminal scenario, perform automatic speech recognition on the input speech to obtain transcribed text and recognition confidence, and extract prosodic acoustic features from the input speech; S2. Input the transcribed text into a text agent for initial text judgment to obtain text uncertainty; S3. Based on recognition confidence, prosodic acoustic features and text uncertainty, determine whether the current input speech must rely on the prosodic and acoustic information in the original speech to make a correct judgment, and obtain the speech necessity judgment result. S4. Based on the speech necessity judgment result, determine whether to use a transcription text model path or an end-to-end speech model path for the input speech; S5. When the input speech is processed using a transcription text model path, the transcription text model path result obtained based on the transcribed text is output; when the input speech is processed using an end-to-end speech model path, the end-to-end speech model is invoked, and the end-to-end speech model path result is generated based on the input speech. S6. Output the speech understanding result based on the transcribed text model path result or the end-to-end speech model path result.

[0023] Preferably, the initial text judgment is used to characterize the degree of ambiguity when performing speech understanding based on transcribed text, the speech necessity judgment is used to characterize whether the current input speech still needs to rely on the prosody and acoustic information in the original speech to complete the correct understanding, and the transcribed text model path and the end-to-end speech model path are used to correspond to the processing path where the text information is sufficient to support understanding and the processing path that relies on the original speech information to complete the understanding, respectively.

[0024] By performing automatic speech recognition on the input speech and extracting prosodic acoustic features, the technical effect of simultaneously acquiring transcribed text information and original speech information is achieved. By judging the necessity of speech based on recognition confidence, prosodic acoustic features and text uncertainty, and further determining whether to use a transcription text model path or an end-to-end speech model path for the input speech, the technical effect of completing the pre-judgment and corresponding path diversion in the speech understanding stage is achieved. By outputting the corresponding path results as the speech understanding results, the technical effect of balancing the accuracy of speech understanding and processing cost is achieved.

[0025] Furthermore, the prosodic acoustic features described in S1 include acoustic representations of pitch, energy, duration, pauses, and depth.

[0026] Preferably, the pitch is used to characterize the pitch trend of the input speech, the energy is used to characterize the energy change of the input speech, the duration and pauses are used to characterize the rhythm and pause information in the input speech, and the deep acoustic representation is used to characterize the deep acoustic features of the input speech.

[0027] By using pitch, energy, duration, pauses, and depth acoustic representations as prosodic acoustic features, a technical effect of jointly representing prosodic and acoustic information in input speech is achieved, thus providing a more complete feature basis for subsequent speech necessity judgment.

[0028] Furthermore, the speech necessity judgment result described in S3 includes a speech necessity score, a speech necessity level, or a classification result indicating whether or not the original speech is required.

[0029] Preferably, the speech necessity score is used to characterize the degree of dependence of the current input speech on the original speech information, the speech necessity level is used to characterize the hierarchy of dependence of the current input speech on the original speech information, and the classification result of whether or not it needs to rely on the original speech is used to characterize whether the current input speech still needs to rely on the prosody and acoustic information in the original speech to complete the understanding.

[0030] By representing the speech necessity judgment result as a speech necessity score, speech necessity level, or classification result of whether it is necessary to rely on the original speech, the technical effect of explicitly representing the need to preserve the original speech is achieved, which facilitates the subsequent path determination process to call the corresponding judgment result.

[0031] Furthermore, the method described in S3 for obtaining the speech necessity judgment result is as follows: combining the confidence and uncertainty of the text agent, ASR recognition quality, pause position, stress distribution, speech rate change, pitch trend, energy change, and the degree of divergence between different agents, to determine whether the current input speech must rely on the prosody and acoustic information in the original speech to make a correct judgment.

[0032] Preferably, the confidence and uncertainty of the text agent are used to characterize the degree of grasp and ambiguity when understanding based on transcribed text; the ASR recognition quality is used to characterize the credibility of the automatic speech recognition result; the pause position, stress distribution, speech rate change, pitch trend and energy change are used to characterize the prosodic change information in the input speech; and the degree of divergence between different agents is used to characterize the consistency state between different processing results.

[0033] By combining the confidence and uncertainty of the text agent, ASR recognition quality, pause position, stress distribution, speech rate change, pitch trend, energy change, and the degree of divergence between different agents, the technical effect of judging the necessity of speech from multiple dimensions such as text-side information, recognition quality information, and original speech information is achieved, thereby improving the reliability of the speech necessity judgment results.

[0034] Further, S4, determining whether to use a transcription text model path or an end-to-end speech model path for the input speech, includes: when the speech necessity judgment result indicates that it does not need to rely on the original speech, determining to use a transcription text model path for the input speech; when the speech necessity judgment result indicates that it must rely on the original speech, determining to use an end-to-end speech model path for the input speech.

[0035] Preferably, the transcribed text model path is used to complete subsequent processing when the transcribed text is sufficient to support speech understanding, and the end-to-end speech model path is used to complete subsequent processing when it is necessary to rely on prosodic and acoustic information in the original speech.

[0036] By using a transcription model path when the speech necessity judgment result indicates that it does not depend on the original speech, and using an end-to-end speech model path when the speech necessity judgment result indicates that it must depend on the original speech, the technical effect of targeted diversion of input speech based on the speech necessity judgment result is achieved, thereby avoiding the uniform use of the same processing link for input speech of different complexity.

[0037] Furthermore, in S4, when the speech necessity judgment result indicates that it must rely on the original speech, or the text uncertainty indicates that there are multiple interpretations, or there is a conflict between different pieces of evidence, an end-to-end speech model path is determined for the input speech.

[0038] Preferably, the conflict between different pieces of evidence is used to characterize an inconsistency between the understanding information obtained from the transcribed text and the understanding information obtained from the original speech.

[0039] By determining the end-to-end speech model path for the input speech when the speech necessity judgment result is characterized as a necessity to rely on the original speech and the text uncertainty is characterized as multiple interpretations or conflicts between different pieces of evidence, the technical effect of further introducing the original speech information to participate in the understanding when there is ambiguity in text understanding or inconsistency in existing evidence is achieved, thereby improving the processing accuracy in complex speech understanding tasks.

[0040] Furthermore, the method also includes: after adopting the end-to-end speech model path, if the result of the transcribing text model path is consistent with the result of the end-to-end speech model path, then the processing ends; if the two path results still have obvious conflicts, then the context agent and the verification agent are called to complete the final judgment.

[0041] Preferably, the context agent is used to process speech understanding information related to the preceding text, and the verification agent is used to perform consistency checks on the processing results of different paths.

[0042] By using an end-to-end speech model path to determine the consistency between the results of the transcription text model path and the end-to-end speech model path, and calling the context agent and the verification agent to complete the final judgment when there is still a significant conflict between the two path results, the technical effect of further checking and correcting the results of complex speech understanding is achieved, thereby improving the reliability of the final speech understanding results.

[0043] Furthermore, the method determines whether to use a transcription text model path or an end-to-end speech model path for the input speech, provided that cost control requirements are met. The cost control requirements are manifested in token cost, inference steps, number of interface calls, processing latency, or a combination of these quantities.

[0044] Preferably, the cost control requirements are used to characterize the resource consumption status during the speech understanding processing process.

[0045] By determining whether to use a transcription text model or an end-to-end speech model for the input speech while meeting cost control requirements, the technical effect of balancing speech understanding processing resource consumption during path splitting is achieved, thereby reducing unnecessary high-cost processing.

[0046] Furthermore, the method also includes early termination and cost checks; when the results of a certain stage have reached the preset reliability requirements, or when different agents have formed a stable and consistent judgment, subsequent processing is stopped; when the remaining cost space is insufficient to support the next round of analysis, processing is terminated.

[0047] Preferably, the early termination is used to stop calling subsequent processing paths when the current processing result has met the subsequent output conditions, and the cost check is used to restrict the continued execution of high-cost processing when the remaining cost space is insufficient.

[0048] By setting early termination and cost checks, the technical effect of ending processing in a timely manner when the current processing result has met the output conditions or the remaining cost space is insufficient is achieved, thereby further controlling unnecessary overhead in the speech understanding processing process.

[0049] The method described in this embodiment can be executed by a processor calling a computer program, which can be stored in a computer storage medium. When the computer program is executed by the processor, the above-described speech understanding triage method based on speech necessity judgment can be implemented.

[0050] Implementation Method 2 In this embodiment, a speech understanding triage method based on speech necessity judgment is applied to speech understanding processing in smart terminal scenarios. The method adopts a text-first, on-demand triage approach. First, it receives the input speech and available context, and completes automatic speech recognition, initial text judgment, and prosodic acoustic feature extraction to provide textual and audio evidence for subsequent judgments. Then, based on recognition confidence, prosodic acoustic features, and text uncertainty, it determines whether the current input speech must rely on the prosodic and acoustic information in the original speech to be correctly judged, obtaining a speech necessity judgment result. Next, based on the speech necessity judgment result, it determines whether to use a text transcription model path or an end-to-end speech model path for the input speech. After obtaining the path result, it calls the context agent and verification agent as needed to complete result fusion, consistency judgment, early termination, and cost checks, finally outputting the speech understanding result.

[0051] After the input voice is received, automatic speech recognition, initial text judgment, and prosodic acoustic feature extraction are first completed to form textual and speech evidence. Then, through the speech necessity judgment and path determination process, it is decided whether to use the transcription text model path or the end-to-end speech model path for the input voice. After obtaining the path result, the context agent and the verification agent are called as needed to complete the result fusion, consistency judgment, early termination and cost check, and finally output the speech understanding result.

[0052] In this implementation, the main inputs to the text agent are ASR (Automatic Speech Recognition) transcribed text, task description, speaking turns, and other textual information. Its role is to provide a low-cost preliminary understanding result and output textual evidence and uncertainty. The main input to the speech agent is the original speech or acoustic feature sequence. Its role is to supplement the text with pauses, stress, speech rate, intonation, and emotion evidence that cannot be directly reflected in the text. The main inputs to the context agent are multi-turn dialogue history, role relationships, and preceding context. Its role is to handle issues that are difficult to judge in a single sentence but can be clarified by combining the preceding and following context. The main inputs to the verification agent are the candidate conclusions of each agent and their evidence. Its role is to perform consistency checks and final gatekeeping when there are obvious differences in different paths. The main inputs to the diversion processing part are speech necessity judgment, text confidence, remaining cost space, and current divergence state. Its role is to determine whether the current input speech enters the transcribed text model path, the end-to-end speech model path, or continues to call other processing links.

[0053] In the process of determining the necessity of speech, the necessity of speech refers to whether, given the transcribed text, the current input speech still requires the prosodic and acoustic information of the original speech to be correctly judged. The result of the speech necessity determination can be expressed as a speech necessity score, a speech necessity level, or a classification result indicating whether the original speech is needed. The criteria used to determine the necessity of speech include the confidence and uncertainty of the text agent, ASR recognition quality, pause positions, stress distribution, speech rate changes, pitch trends, energy changes, and the degree of divergence between different agents. This part transforms the speech call requirements, which originally relied on experience-based judgment, into decision signals that the system can directly utilize.

[0054] In this implementation, the speech necessity score indicates whether the current input speech must rely on speech prosody information for judgment; the higher the score, the more inclined to adopt the end-to-end speech model path. The text confidence score indicates the degree of confidence of the text agent in the current answer; a high confidence score allows the process to end directly, while a low confidence score is more likely to escalate the analysis. The text uncertainty score indicates whether the candidate answers in the text are scattered and whether there are multiple interpretations; a high uncertainty score indicates that simply looking at the transcribed text may not be sufficient. The cost margin indicates the proportion of remaining available cost to the total cost limit, which determines whether the system can continue to call more expensive understanding paths. The agent divergence score indicates whether there is a significant conflict between the text, speech, and contextual conclusions; the greater the divergence, the more necessary it is to continue analysis or verification.

[0055] In the path determination process, a judgment-then-triage approach is adopted. The system first performs an initial text judgment; when the text agent has a high degree of confidence and relevant evidence points to a consistent conclusion, the result of the transcription text model path can be directly adopted; when the necessity of speech is high, the confidence of text is low, the text result has obvious ambiguity, or there is a conflict between different pieces of evidence, the end-to-end speech model path is further adopted. If the problem is still strongly related to the preceding text, the context agent is called again; if multiple results are still difficult to unify, the verification agent is called for final verification. The cost control requirements can be expressed as token cost, inference steps, number of interface calls, processing latency, or a combination of these quantities. Under the premise of meeting the cost control requirements, the system determines which understanding path is more suitable for the current input speech, rather than sending all input speech into the high-cost model from the beginning.

[0056] In a typical execution process, automatic speech recognition is first performed on the input speech to obtain the transcribed text and the corresponding recognition confidence score. The text agent provides a preliminary answer based on the transcribed text and task prompts, and generates text-level uncertainty. Simultaneously, pitch, energy, duration, pauses, and depth acoustic representations are extracted from the original speech. Subsequently, the speech necessity score of the current input speech is obtained, and based on this score and the cost state, a decision is made: whether to directly input the transcribed text into the text model and adopt the result of the transcribed text model path, or to call the end-to-end speech model path. If the end-to-end speech model path is adopted, it generates supplementary conclusions based on the acoustic and prosodic information in the original speech. If the result of the transcribed text model path is consistent with the result of the end-to-end speech model path, the system can terminate. If there is still a significant conflict between the two, the context agent and the verification agent are called to complete the final judgment.

[0057] In the process of multi-agent collaboration and early termination, the method does not require all processing steps to run simultaneously. For simple input speech, the task can be completed by transcribing the text model path; for moderately difficult input speech, only an additional end-to-end speech model path may be needed; the context agent and verification agent are only activated in cases of high ambiguity, multi-turn context dependence, or significant conflicting conclusions. To avoid invalid reasoning, an early termination mechanism is implemented. When the results of a certain stage have reached a preset reliability requirement, or when different agents have formed a stable and consistent judgment, the system can stop immediately; conversely, if the remaining cost space is insufficient to support the next round of analysis, the system also terminates promptly, rather than continuing to perform costly but low-return calculations.

[0058] In terms of training implementation, supervised learning, reinforcement learning, or a combination of both are allowed to be used to train the speech necessity judgment and path determination process. A relatively stable implementation approach is to first construct initial labels based on whether the input speech ultimately requires calling the end-to-end speech model, and then perform warm-start training on the path determination process; then, continue to optimize the path determination strategy with the joint optimization objectives of improving understanding accuracy, model call cost, and processing latency, so that the system gradually acquires the ability to determine whether the end-to-end speech model must be called based on the features of the input speech. This ability can be acquired through rule construction or through data-driven learning.

[0059] In terms of evaluation implementation, the processing effect can be assessed by combining real anchor data and controllable counterfactual data. Real anchor data is used to illustrate the actual effect of the system in real-world scenarios; controllable counterfactual data is used to construct sample pairs with the same text but different speaking styles, thereby observing whether the system triggers the end-to-end speech model path when speech is important. The corresponding evaluation metrics can include accuracy, average cost, average latency, splitting efficiency, and counterfactual flip correctness. Among them, splitting efficiency is used to measure whether the system achieves higher results with lower costs, and counterfactual flip correctness is used to verify whether the system truly utilizes the prosodic information in the original speech, rather than simply mechanically taking an extra path.

[0060] The method described in this embodiment is applicable to emotion recognition, attitude recognition, certainty and hesitation recognition, ambiguity judgment of interrogative and declarative sentences, stress focus recognition, pause boundary ambiguity resolution, dialogue intent judgment, and other speech understanding tasks that are significantly dependent on prosody, pauses, speech rate, stress, and context. In engineering implementation, the text agent can be handled by a large language model, a small language model, or a rule system; the end-to-end speech model path can employ a large end-to-end speech model or a speech encoder combined with a classification head; the context agent can be used to process multi-turn dialogue history; and the verification agent can employ a consistency scoring model or a rule checker.

Claims

1. A voice understanding shunt method based on voice necessity determination, characterized by, The method includes: S1. Receive input speech for speech understanding processing in a smart terminal scenario, perform automatic speech recognition on the input speech to obtain transcribed text and recognition confidence, and extract prosodic acoustic features from the input speech; S2. Input the transcribed text into a text agent for initial text judgment to obtain text uncertainty; S3. Based on recognition confidence, prosodic acoustic features and text uncertainty, determine whether the current input speech must rely on the prosodic and acoustic information in the original speech to make a correct judgment, and obtain the speech necessity judgment result. S4. Based on the speech necessity judgment result, determine whether to use a transcription text model path or an end-to-end speech model path for the input speech; S5. When the input speech is processed using a transcription text model path, the transcription text model path result obtained based on the transcribed text is output; when the input speech is processed using an end-to-end speech model path, the end-to-end speech model is invoked, and the end-to-end speech model path result is generated based on the input speech. S6. Output the speech understanding result based on the transcribed text model path result or the end-to-end speech model path result.

2. The method according to claim 1, characterized in that, The prosodic acoustic features described in S1 include acoustic representations of pitch, energy, duration, pauses, and depth.

3. The method according to claim 1, characterized in that, The speech necessity judgment result mentioned in S3 includes a speech necessity score, a speech necessity level, or a classification result indicating whether or not the original speech is required.

4. The method according to claim 1, characterized in that, The method described in S3 for obtaining the speech necessity judgment result is as follows: combining the confidence and uncertainty of the text agent, ASR recognition quality, pause position, stress distribution, speech rate change, pitch trend, energy change, and the degree of divergence between different agents, to determine whether the current input speech must rely on the prosody and acoustic information in the original speech to make a correct judgment.

5. The method according to claim 1, characterized in that, S4, determining whether to use a transcription text model path or an end-to-end speech model path for the input speech, includes: when the speech necessity judgment result indicates that it does not need to rely on the original speech, determining to use a transcription text model path for the input speech; when the speech necessity judgment result indicates that it must rely on the original speech, determining to use an end-to-end speech model path for the input speech.

6. The method according to claim 1, characterized in that, In step S4, when the speech necessity judgment result indicates that it must rely on the original speech, or the text uncertainty indicates that there are multiple interpretations, or there is a conflict between different pieces of evidence, an end-to-end speech model path is determined for the input speech.

7. The method according to claim 1, characterized in that, The method further includes: after adopting the end-to-end speech model path, if the result of the transcribed text model path is consistent with the result of the end-to-end speech model path, then the processing ends; if the two path results still have obvious conflicts, then the context agent and the verification agent are called to complete the final judgment.

8. The method according to claim 1, characterized in that, The method determines whether to use a text transcription model path or an end-to-end speech model path for the input speech while meeting cost control requirements. The cost control requirements are manifested in token cost, inference steps, number of interface calls, processing latency, or a combination of these quantities.

9. The method according to claim 1, characterized in that, The method also includes early termination and cost checks; when the results of a certain stage have reached the preset reliability requirements, or when different agents have formed a stable and consistent judgment, the subsequent processing is stopped; when the remaining cost space is insufficient to support the next round of analysis, the processing is terminated.

10. A computer storage medium having a computer program stored thereon, characterized in that, The computer program, when running on a processor, performs the method according to any one of claims 1 to 9.

Images