Voice interaction processing method and related apparatus

Abstract

The application discloses a voice interaction processing method and related device, and relates to the technical fields of artificial intelligence and human-computer interaction, and comprises the following steps: acquiring multi-modal fact information and voice text in a current voice input time window; inputting the voice text and historical dialogue context into a language processing model to acquire a semantic evaluation result and candidate reply content through single inference; performing multi-dimensional confidence fusion calculation based on the multi-modal fact information, the historical dialogue context, the semantic evaluation result and the voice text to obtain a comprehensive confidence evaluation index; determining an acceptance or rejection decision based on a comparison result of the comprehensive confidence evaluation index and a preset threshold; and generating a structured processing result based on the decision and the candidate reply content and returning the device end to perform a response operation. The application utilizes single inference to cooperate with multi-modal evidence fusion, thereby guaranteeing recognition accuracy while reducing system time delay.

Description

Technical Field

[0001] This application relates to the fields of artificial intelligence and human-computer interaction technology, and in particular to a voice interaction processing method and related apparatus. Background Technology

[0002] In open-ended voice interaction scenarios such as child companionship, picture book reading, and family services, smart devices typically need to generate corresponding responses after recognizing the user's voice to maintain a natural and smooth human-computer dialogue experience. However, in real-world usage environments, there are often adult conversations, discussions among bystanders, television sounds, or other background noise. While voice input from these non-directional devices can be transcribed into text by the voice recognition system, it is not an effective interactive command.

[0003] Existing technologies typically rely on wake word detection, single voice rule matching, or single text intent recognition to determine the validity of voice messages. These solutions struggle to cover complex scenarios such as short sentence responses, quick responses, face-gazing interactions, page-turning triggers, and user replies after device-generated questions. This can easily lead to devices misresponding to conversations with others or incorrectly filtering out brief, valid commands from real users, thereby disrupting normal interaction flows and degrading the user experience.

[0004] To improve the ability to understand complex contexts, some solutions attempt to introduce large language models for deep semantic judgment. However, adding a large model call step outside the original speech recognition chain, or using multiple model calls for intent judgment and response generation separately, increases system response latency and computational resource consumption. Therefore, there is an urgent need for a speech interaction processing solution that can balance the accuracy of intent recognition and system response efficiency without changing the main chain of front-end speech acquisition and recognition. Summary of the Invention

[0005] In view of the problems of low accuracy in recognizing voice interaction intent in open environments and high response latency caused by the introduction of large models in existing technologies, this application provides a voice interaction processing method and related apparatus to achieve multimodal evidence fusion decision-making and single-inference collaboration of language models, thereby reducing system latency while ensuring recognition accuracy. The specific solution is as follows:

[0006] The first aspect of this application provides a voice interaction processing method, including:

[0007] Obtain multimodal factual information within the current voice input time window, as well as the voice text corresponding to the current voice input;

[0008] The speech text and historical dialogue context are input into the language processing model, and semantic evaluation results and candidate response content are obtained through a single inference.

[0009] Based on the multimodal factual information, the historical dialogue context, the semantic evaluation results, and the speech text, a multi-dimensional confidence fusion calculation is performed to obtain a comprehensive credibility evaluation index.

[0010] Based on the comparison results between the comprehensive credibility assessment index and the preset threshold, a decision to accept or reject is made.

[0011] Based on the decision and the candidate response content, a structured processing result is generated and returned to the device, so that the device can perform the corresponding response operation based on the structured processing result.

[0012] In one possible implementation, the multimodal factual information includes visual detection information and temporal state information; the semantic evaluation result includes a semantic orientation score and a non-interactive dialogue determination suggestion; wherein, the semantic orientation score is used to characterize the probability assessment of the voice input for this device, and the non-interactive dialogue determination suggestion is used to characterize the determination result that the voice input belongs to a non-interactive dialogue; the candidate response content includes the response text and an identifier indicating whether the response text expects further response.

[0013] The multi-dimensional confidence fusion calculation based on the multimodal factual information, the historical dialogue context, the semantic evaluation result, and the speech text includes:

[0014] Based on the visual detection information in the multimodal factual information, a visual evidence score is calculated;

[0015] The semantic orientation score and the visual evidence score are weighted and calculated respectively to obtain a semantic weighted score and a visual weighted score;

[0016] The semantic weighted score and the visual weighted score are corrected based on the temporal state information in the multimodal factual information, the historical dialogue context, and the voice text to obtain the comprehensive credibility evaluation index.

[0017] In one possible implementation, the visual detection information includes visual detection statistics obtained based on image detection and visual detection service availability information;

[0018] The methods for determining the visual evidence score include:

[0019] When it is determined that the visual detection service is available based on the visual detection service availability information, the proportion of facial orientation, facial contour, and human body contour detected within the current voice input time window is obtained based on the visual detection statistics.

[0020] Based on the facial orientation ratio, the facial contour ratio, and the human body contour ratio, the first evidence score, the second evidence score, and the third evidence score are calculated respectively.

[0021] The maximum value among the first evidence score, the second evidence score, and the third evidence score is determined as the visual evidence score;

[0022] When it is determined that the visual inspection service is unavailable based on the availability information of the visual inspection service, the preset neutral evidence score is determined as the visual evidence score.

[0023] In one possible implementation, the step of correcting the semantic weighted score and the visual weighted score based on the temporal state information in the multimodal factual information, the historical dialogue context, and the speech text to obtain the comprehensive credibility evaluation index includes:

[0024] Based on the temporal state information, the historical dialogue context, and the voice text, the wake-up source score, the response context score, and the sentence integrity correction score are calculated respectively.

[0025] The semantic weighted score, the visual weighted score, the wake-up source score, the conversation context score, and the sentence integrity correction score are fused together to obtain the comprehensive credibility evaluation index.

[0026] In one possible implementation, the timing state information includes voice time window information, device playback state information, wake-up source information, and session identifier information;

[0027] The calculation method for the wake-up source bonus includes:

[0028] When the current voice input is the first voice text after wake-up, the corresponding bonus value is determined according to the wake-up source information in the timing status information; wherein, different wake-up source types correspond to different bonus values, and the wake-up source types include voice wake-up, face wake-up, and page-turning wake-up;

[0029] When the current voice input is not the first voice text after wake-up, the wake-up source score is zero.

[0030] In one possible implementation, the historical dialogue context includes the previous round of device dialogue, an indicator of whether the previous round of device dialogue expects a response, the previous round of decision results, and the number of consecutive rejections within the same session;

[0031] The calculation method for the context score includes:

[0032] Obtain an identifier from the historical dialogue context indicating whether the previous round of device dialogue expects a response;

[0033] When the indicator indicating whether the previous device dialogue is expected to receive a response indicates that a response is expected, and the current voice input occurs during or after the device plays the previous device dialogue, within a preset time period, and the voice text constitutes a response to the previous device dialogue, the context score for the conversation is increased.

[0034] In one possible implementation, the sentence integrity correction score is calculated as follows:

[0035] Based on the text length, grammatical structure, semantic completeness, and contextual relationships of the spoken text, determine whether the spoken text is a complete sentence;

[0036] When the spoken text is determined not to be a complete sentence, the sentence integrity correction score is negatively corrected.

[0037] In one possible implementation, determining whether the spoken text is a complete sentence includes:

[0038] When the length of the voice text exceeds a preset length threshold, and the semantics of the voice text are interrupted or lack core grammatical components, and the voice text has no connection with the previous round of device dialogue, the voice text is determined to be not a complete sentence.

[0039] When the length of the voice text does not exceed the preset length threshold, the voice text is judged to be a complete sentence in combination with the previous round of device dialogue.

[0040] In one possible implementation, the method further includes:

[0041] When the non-interactive dialogue determination suggestion meets the preset non-interactive dialogue rejection conditions, and the current voice input is not the first voice text after the voice wake-up operation, the current voice input is not allowed to be accepted based solely on at least one of the visual weighted score, the wake-up source score, and the conversation context score.

[0042] In one possible implementation, the method further includes:

[0043] Within the same session, when a preset number of rejection decisions are made consecutively, an instruction to end the current voice monitoring is generated;

[0044] The statistics of the number of consecutive rejection decisions do not span different sessions.

[0045] In one possible implementation, the method further includes:

[0046] After acquiring the multimodal factual information and the voice text, when the voice text matches a preset special business scenario, it is processed according to the preset business logic corresponding to the preset special business scenario, and the interactive response corresponding to the preset business logic is directly executed.

[0047] In one possible implementation, the content of the structured processing result varies depending on the decision made:

[0048] When the decision is to accept, the structured processing result includes at least the statement identifier, the acceptance decision, the comprehensive credibility assessment index, the reason for the decision, the response text, and the instruction to continue listening;

[0049] When the decision is to reject, the structured processing result includes at least the statement identifier, the rejection decision, the comprehensive credibility assessment index, and the decision reason, but does not include the response text.

[0050] A second aspect of this application provides a voice interaction processing device, comprising:

[0051] The acquisition unit is used to acquire multimodal factual information within the current voice input time window, as well as the voice text corresponding to the current voice input;

[0052] The reasoning unit is used to input the speech text and historical dialogue context into the language processing model, and obtain semantic evaluation results and candidate response content through a single reasoning;

[0053] The multi-dimensional confidence fusion calculation unit is used to perform multi-dimensional confidence fusion calculation based on the multimodal factual information, the historical dialogue context, the semantic evaluation result and the speech text to obtain a comprehensive credibility evaluation index.

[0054] The decision-making unit is used to determine whether to accept or reject the recognition based on the comparison result between the comprehensive credibility evaluation index and the preset threshold.

[0055] The processing result generation and feedback unit is used to generate a structured processing result based on the decision and the candidate response content and return it to the device, so that the device can perform the corresponding response operation based on the structured processing result.

[0056] A third aspect of this application provides a computer program product including computer-readable instructions that, when executed on an electronic device, cause the electronic device to implement the voice interaction processing method of the first aspect or any implementation thereof.

[0057] A fourth aspect of this application provides an electronic device, including at least one processor and a memory connected to the processor, wherein:

[0058] The memory is used to store computer programs;

[0059] The processor is used to execute the computer program so that the electronic device can implement the voice interaction processing method of the first aspect or any implementation thereof.

[0060] The fifth aspect of this application provides a computer-readable storage medium carrying one or more computer programs that, when executed by an electronic device, enable the electronic device to implement the voice interaction processing method of the first aspect or any implementation thereof.

[0061] By employing the aforementioned technical solutions, the voice interaction processing method and related apparatus provided in this application collect multimodal factual information within the voice input time window, input the voice text and historical dialogue context into a language processing model for single-pass reasoning, and simultaneously obtain semantic evaluation results and candidate response content. Then, based on multimodal factual information, historical dialogue context, semantic evaluation results, and voice text, multi-dimensional confidence fusion calculation is performed to determine the final decision. This solution utilizes the single-pass reasoning mechanism of the language processing model to merge semantic orientation judgment and response generation tasks, avoiding the multiple network requests and reasoning overhead caused by separately calling the intent recognition model and the response generation model in traditional solutions. This reduces system response latency while ensuring complex semantic understanding capabilities. Furthermore, by performing multi-dimensional fusion calculations of objective factual evidence such as visual detection information and temporal state information collected on the edge with the semantic evaluation results from the server, a cross-validation mechanism is constructed. This compensates for the uncertainty of single text semantic judgment under short sentences, rapid responses, or background noise interference, improving robustness to the recognition of genuine interactive intent. This allows the device to distinguish between effective interactions and conversations with others without changing the main parameters of the front-end voice recognition link, enhancing the overall interactive experience in open environments. Attached Figure Description

[0062] The above and other features, advantages, and aspects of the embodiments of this disclosure will become more apparent from the accompanying drawings and the following detailed description. Throughout the drawings, the same or similar reference numerals denote the same or similar elements. It should be understood that the drawings are schematic, and the originals and elements are not necessarily drawn to scale.

[0063] Figure 1 A flowchart illustrating a voice interaction processing method provided in an embodiment of this application;

[0064] Figure 2 This is a schematic diagram of the structure of a voice interaction processing device provided in an embodiment of this application;

[0065] Figure 3This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0066] The embodiments of this application are described below with reference to the accompanying drawings. The terminology used in the implementation section of this application is for explaining specific embodiments only and is not intended to limit the scope of this application.

[0067] The embodiments of this application will now be described with reference to the accompanying drawings. Those skilled in the art will recognize that, with technological advancements and the emergence of new scenarios, the technical solutions provided in the embodiments of this application are equally applicable to similar technical problems.

[0068] The terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such terms are interchangeable where appropriate; this is merely a way of distinguishing objects with the same attributes in the embodiments of this application. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion, so that a process, method, system, product, or apparatus that comprises a series of elements is not necessarily limited to those elements, but may include other elements not explicitly listed or inherent to those processes, methods, products, or apparatuses.

[0069] To address the aforementioned problems, this application provides a voice interaction processing method. The voice interaction processing method of this application embodiment will be described in detail below with reference to the accompanying drawings.

[0070] Example 1

[0071] Reference Figure 1 , Figure 1 This is a flowchart illustrating a voice interaction processing method provided in an embodiment of this application. This method can be applied to a server or a voice interaction system that includes a server. Figure 1 As shown in the figure, the voice interaction processing method provided in this application embodiment may include steps S100 to S500, which are described in detail below.

[0072] Step S100: Obtain multimodal factual information within the current voice input time window, as well as the voice text corresponding to the current voice input.

[0073] In this embodiment, the current voice input time window refers to the period from the start of voice activity detected on the device to the end of the final text generated by voice recognition. Within this time window, the device continuously collects objective environmental state data related to the current voice input, i.e., multimodal factual information. This information includes, but is not limited to, visual detection information and temporal state information. Visual detection information reflects physical characteristics such as the user's limbs or facial orientation during speaking, while temporal state information reflects the temporal relationship between voice input and device playback status and wake-up events. It should be understood that multimodal factual information is collected and transmitted only as objective evidence. The device does not directly determine whether the current voice is directed at its own device based on this information locally, but rather sends it along with the voice text to the server for unified processing. This architecture of edge-side collection and cloud-based decision-making avoids the problem of simplified judgment logic caused by the limited computing power of the device, while also ensuring that subsequent fusion calculations can be based on complete and original objective evidence, laying a data foundation for improving the accuracy of intent recognition.

[0074] Step S200: Input the speech text and historical dialogue context into the language processing model, and obtain the semantic evaluation results and candidate response content through a single inference.

[0075] In this embodiment, the language processing model can be a large language model deployed on the server or other generative models with natural language understanding and generation capabilities. Historical dialogue context includes information such as the previous device dialogue, whether the previous dialogue expected a response, and the conversation state, providing coherent contextual support for the model. In this embodiment, single-inference refers to the language processing model executing semantic orientation judgment and response content generation in parallel or sequentially within the same model call request, and outputting a complete response containing the semantic evaluation result and candidate response content in one go. The semantic evaluation result characterizes the model's confidence in whether the current voice input is targeted at the device, while the candidate response content is the response text generated when the interaction is deemed valid. Compared to the traditional approach of first calling the intent recognition model to determine validity and then calling the response generation model to generate the response in two sequential calls, this embodiment achieves computational power reuse and network request merging through a single-inference mechanism, reducing system response latency. Meanwhile, since semantic judgment and response generation share the same context encoding process, the model implicitly considers the intermediate features of intent judgment when generating responses, which makes the semantic evaluation results and candidate response content logically consistent, avoiding the problem of judgment and response being disconnected that may be caused by separate calls.

[0076] Step S300: Based on multimodal factual information, historical dialogue context, semantic evaluation results and voice text, multi-dimensional confidence fusion calculation is performed to obtain a comprehensive credibility evaluation index.

[0077] Furthermore, the multi-dimensional confidence fusion calculation is a process of cross-validating the subjective semantic assessment output by the language processing model with objective multimodal factual information collected from the edge. While the semantic assessment result can capture deep semantic intent, it may be uncertain when faced with short sentences, pre-emptive responses, background noise, or snippets of casual conversation. Multimodal factual information, such as facial gaze, the first speech after wake-up, and the response time window, is physical evidence independent of textual semantics and has strong anti-interference capabilities. The fusion calculation dynamically adjusts the contribution weight of each dimension of evidence to the final decision based on its reliability and relevance in the current context, rather than simply adding up the values. For example, when the semantic assessment result is in an ambiguous range, strong visual evidence or a clear wake-up source can improve the overall confidence; conversely, if the semantics clearly indicate a non-interactive dialogue, even weak visual evidence may be suppressed. Through this multi-dimensional fusion mechanism, the system can effectively compensate for the limitations of single semantic judgment in complex open environments and improve the robustness to recognizing genuine interactive intent.

[0078] Step S400: Based on the comparison results between the comprehensive credibility assessment index and the preset threshold, determine the decision to accept or reject the recognition.

[0079] In this embodiment, the preset threshold is the decision boundary that distinguishes between valid and invalid interactions. Its value can be calibrated based on the tolerance for false responses and the missed response rate in the actual application scenario. When the comprehensive credibility assessment index reaches or exceeds the preset threshold, an acceptance decision is made, indicating that the current voice input is recognized as a valid command directed to this device. When the comprehensive credibility assessment index is below the preset threshold, a rejection decision is made, indicating that the current voice input is judged as a conversation with a stranger, background noise, or non-directional voice. This step transforms continuous credibility scores into discrete decision results, providing a clear basis for differentiated responses from downstream devices. It should be understood that the preset threshold is not fixed and can be dynamically adjusted according to business needs in different embodiments to balance interaction sensitivity and anti-interference capability.

[0080] Step S500: Based on the decision and candidate response content, generate a structured processing result and return it to the device so that the device can execute the corresponding response operation based on the structured processing result.

[0081] In this embodiment, the structured processing result is a data object containing multiple standardized fields, including at least metadata such as statement identifier, decision type, comprehensive credibility assessment index, and decision reason. When the decision is "accept," the structured processing result also includes the response text from the candidate response content and a "continue listening" instruction; when the decision is "reject," the structured processing result does not include the response text to avoid the device mistakenly playing unexpected content. After receiving the structured processing result, the device does not need to perform secondary parsing or intent inference and can directly execute the corresponding response operation based on the decision type field, such as playing the response voice, remaining silent, updating the dialogue context, or ending listening. This structured return mechanism not only ensures the determinism and consistency of the terminal response but also provides a traceable decision basis for subsequent debugging analysis, user feedback, and strategy optimization.

[0082] Through the above steps S100 to S500, this embodiment achieves multimodal evidence fusion decision-making and language model single-inference collaboration without changing the main link of front-end voice acquisition and recognition. This not only ensures the accuracy of intent recognition in complex scenarios, but also effectively controls system response delay.

[0083] Example 2

[0084] Based on the voice interaction processing method provided in Embodiment 1 above, this embodiment further details the specific implementation of multi-dimensional confidence fusion calculation and the hierarchical calculation logic of visual evidence score.

[0085] In this embodiment, the multimodal factual information includes visual detection information and temporal state information. The semantic evaluation results include a semantic orientation score and a non-interactive dialogue determination suggestion. The semantic orientation score is used to characterize the probability assessment of the voice input for this device, and the non-interactive dialogue determination suggestion is used to characterize the determination result that the voice input belongs to a non-interactive dialogue. The candidate response content includes the response text and an indicator indicating whether the response text expects further response. This multi-dimensional information definition method structurally separates subjective semantic understanding from objective physical environment state, providing a standardized data foundation for subsequent refined fusion calculations.

[0086] Furthermore, the process of multi-dimensional confidence fusion calculation based on multimodal factual information, historical dialogue context, semantic evaluation results, and speech text includes: calculating visual evidence score based on visual detection information in multimodal factual information; calculating semantic weighted score and visual weighted score by weighting the semantic orientation score and visual evidence score respectively; and correcting the semantic weighted score and visual weighted score based on temporal state information, historical dialogue context, and speech text in multimodal factual information, and finally obtaining a comprehensive credibility evaluation index.

[0087] The semantic weighted score is obtained by multiplying the semantic orientation score by a preset semantic weight, and the visual weighted score is obtained by multiplying the visual evidence score by a preset visual weight. For example, the semantic weight can be set to 70%, and the visual weight can be set to 30%. This means that in the basic scoring stage, semantic understanding plays a dominant role, while the contribution of visual evidence, as an important auxiliary verification method, is limited to a certain range to prevent excessive interference from visual noise. The correction score, as a dynamic adjustment item, can provide positive incentives or negative penalties on top of the basic score, thereby flexibly responding to special contexts such as wake-up, answering, and truncation. It should be understood that the above weight ratios are only illustrative examples, and in actual applications, they can be dynamically adjusted according to device type, environmental noise level, and user habits. This application does not limit this.

[0088] To determine the visual evidence score, this embodiment employs a hierarchical calculation strategy to balance detection accuracy and scene adaptability. Specifically, the visual detection information includes visual detection statistics obtained based on image detection and visual detection service availability information. When the visual detection service is determined to be available based on the visual detection service availability information, the system obtains the proportion of facial orientation, facial contour, and human body contour detected within the current voice input time window based on the visual detection statistics. These three levels correspond to interaction intent evidence from strongest to weakest: the facial orientation proportion reflects whether the user is looking directly at the device, representing the strongest visual signal of interaction intent; the facial contour proportion reflects the presence of a face within the field of vision, indicating that even if the face is not directly facing the device, the user may still be in an interactive posture; the human body contour proportion reflects a broader sense of presence, applicable to long-distance or sideways interaction scenarios. Based on these three proportions, the system calculates the first evidence score, the second evidence score, and the third evidence score respectively, and determines the maximum value among the three as the final visual evidence score.

[0089] As a preferred implementation, the calculation of evidence scores at each level can employ differentiated coefficient mappings. For example, the first evidence score (corresponding to facial orientation) can be obtained by multiplying the proportion of facial orientation by 100, the second evidence score (corresponding to facial contour) can be obtained by multiplying the proportion of facial contour by 65, and the third evidence score (corresponding to human body contour) can be obtained by multiplying the proportion of human body contour by 30. Using the maximum value method instead of weighted summation implies adhering to the "strong evidence priority principle." That is, within the same voice input time window, as long as a high-confidence visual signal (such as a frontal gaze) is captured, this strong signal is directly adopted as visual evidence, without being diluted by weak signals or detection fluctuations that may exist within the same time period. This mechanism effectively avoids the situation where the number of frontal frames decreases due to slight head movements during user speech, thus lowering the overall visual score and improving the capture rate of real interactive behavior. It should be noted that the coefficients 100, 65, 30, etc. mentioned above are only specific examples listed for ease of understanding. In actual deployment, these coefficients can be calibrated according to the camera's field of view, installation height, and the average height distribution of the target users, as long as the relative intensity relationship between different levels is maintained.

[0090] Furthermore, to enhance system robustness, this embodiment also incorporates a fault-tolerant handling mechanism for visual service anomalies. When the visual inspection service is determined to be unavailable based on availability information, the system assigns a preset neutral evidence score as the visual evidence score. "Unavailable status" here includes, but is not limited to, camera hardware failure, visual algorithm service crashes, data transmission interruptions, or privacy occlusion mode activation. The neutral evidence score is typically set to the midpoint of the visual scoring range, for example, 50 points in a 0-100 scoring system. The design consideration of using a neutral score instead of zero or full marks is that a zero score would cause the system to tend to reject all speech when visual failure occurs, resulting in severe missed responses; a full mark would cause the system to lose its visual dimension discrimination ability, increasing the risk of false responses. As a form of "no prior information," the neutral score neither rewards nor punishes the current speech input, allowing the decision-making logic to smoothly degenerate into a mode that relies solely on semantic and temporal information. This ensures that the core functions of voice interaction remain available even when the visual module fails, avoiding systemic paralysis caused by a single point of failure.

[0091] Example 3

[0092] Based on the multi-dimensional confidence fusion calculation framework provided in the foregoing embodiments, this embodiment further details the specific implementation method for correcting semantic weighting and visual weighting based on temporal state information, historical dialogue context, and voice text in multimodal factual information.

[0093] In this step, the correction process includes: calculating wake-up source scores, response context scores, and sentence integrity correction scores based on temporal state information, historical dialogue context, and voice text; and fusing the semantic weighted score, visual weighted score, wake-up source score, response context score, and sentence integrity correction score to obtain a comprehensive credibility assessment index. The initial design intention of this three-dimensional correction mechanism is that while the basic semantic weighted score and visual weighted score can reflect the current interaction probability, they are insufficient to cover the complex temporal dependencies and contextual coherence features in open environments. By introducing dynamic correction factors from the three dimensions of wake-up source, response context, and sentence integrity, the system can, on top of the basic score, provide positive incentives for inputs with clear interaction intent signals (such as proactive wake-up or responding to questions), while negatively suppressing inputs that are suspected of being misidentified or truncated segments, thereby improving the discriminative power and robustness of the comprehensive credibility assessment index in complex scenarios.

[0094] For calculating the wake-up source score, the temporal state information includes voice time window information, device playback status information, wake-up source information, and session identifier information. The calculation method for the wake-up source score specifically includes: when the current voice input is the first voice text after wake-up, the corresponding score is determined based on the wake-up source information in the temporal state information; different wake-up source types correspond to different score values, including voice wake-up, face wake-up, and page-turning wake-up; when the current voice input is not the first voice text after wake-up, the wake-up source score is zero. This design reflects the interaction logic of "trigger-based authorization, subsequent actions return to normal." The wake-up behavior itself is the strongest explicit signal of the user's interactive intent; therefore, giving a high positive correction to the first voice text after wake-up can effectively offset the semantic confidence fluctuations caused by environmental noise or voice recognition errors. For example, in one specific implementation, voice wake-up can correspond to a score of 50, face wake-up to a score of 25, and page-turning wake-up to a score of 15. These differentiated score settings reflect the varying intensity of interaction intent implied by different triggering methods: voice wake-up typically indicates a clear verbal command from the user, representing the strongest intent; face wake-up indicates the user is looking at the device and triggering an interaction, representing a secondary intent; page-turning wake-up may be accompanied by reading behavior, representing a relatively weaker intent. It is important to emphasize that this scoring mechanism is strictly limited to the first voice-text message after wake-up; subsequent ordinary chat voice messages no longer carry wake-up source information or receive zero wake-up source points. This restriction avoids a single wake-up action from continuously positively interfering with subsequent multiple rounds of dialogue, prevents the system from remaining in a highly sensitive state for an extended period after user wake-up and mistakenly responding to conversations from others, and ensures timely reset of the interaction state and independence of decision-making.

[0095] For calculating the context-based scoring, the historical dialogue context includes the previous device script, an indicator indicating whether the previous device script expected a response, the previous decision result, and the number of consecutive rejections within the same session. The calculation method for context-based scoring includes: obtaining the indicator indicating whether the previous device script expected a response from the historical dialogue context; when the indicator indicates that a response is expected, and the current voice input occurs during or after the device plays the previous device script within a preset time period, and the voice text constitutes a response to the previous device script, a context-based score is added. This mechanism aims to solve the problem of short-phrase responses being easily misjudged in traditional solutions. In human-computer dialogue, user responses to device questions are often brief, such as "okay," "no," or "this one." These short texts are semantically ambiguous when removed from context, making them easily judged as invalid input or idle chatter by a single semantic model. This embodiment constructs a precise conversation protection logic through joint verification of three conditions: "expected response indicator + time window + content constituting the response." Bonus points are only awarded when the device has just issued a clear question or invitation (with the expected response flag set to true), the user's voice input is closely connected in time (during or shortly after playback), and the text content semantically responds to the device's question. For example, when the device asks, "Which book do you want to listen to?" and the user answers "This one," even though "This one" is only three characters long and lacks a subject-verb-object structure, the system can accurately identify it as a valid response and provide positive correction based on the aforementioned three conditions. This avoids missed responses due to insufficient text and ensures the natural flow of the dialogue.

[0096] The core of calculating the sentence integrity correction score lies in distinguishing between valid short instructions and invalid truncated segments. The calculation method includes: determining whether the speech text is a complete sentence based on its length, grammatical structure, semantic completeness, and contextual relationships; when the speech text is determined not to be a complete sentence, a negative correction is applied to the sentence integrity correction score. To avoid indiscriminately penalizing short texts, the logic for determining whether a speech text is a complete sentence is refined: when the length of the speech text exceeds a preset length threshold, and the speech text has semantic breaks or lacks core grammatical components, and the speech text has no connection to the previous round of device prompts, the speech text is determined not to be a complete sentence; when the length of the speech text does not exceed the preset length threshold, the previous round of device prompts is considered when determining whether the speech text is a complete sentence. The technical consideration behind this differentiated processing is as follows: short texts (such as "open" and "stop"), although few in number, may be complete imperative sentences or responses, and their validity must be determined in context. They cannot be negatively corrected simply because of their length. Conversely, long texts (such as those exceeding 10 Chinese characters) with obvious semantic breaks, missing subjects or predicates, and not responding to a device's question, are highly likely to be snippets of conversation or background television sound captured by the speech recognition system. For example, "The weather is really nice today, let's go together" is a semantically complete long sentence and should not be negatively corrected; while "That's what I think, and then," although longer, is semantically fragmented, logically interrupted, and not a response, and should be judged as an incomplete sentence and negatively corrected. Through this negative correction mechanism, the system can effectively reduce the false response rate to long background noise or irrelevant chatter while maintaining sensitivity to short, effective commands from real users, achieving a dynamic balance between anti-interference capability and interactive sensitivity.

[0097] Example 4

[0098] Based on the multi-dimensional confidence fusion calculation framework provided in the aforementioned embodiments, this embodiment further introduces a protection rule for non-interactive dialogues to prevent non-semantic evidence from excessively interfering with the decision-making results in specific situations. Specifically, when the non-interactive dialogue determination suggestion meets the preset non-interactive dialogue rejection conditions, and the current voice input is not the first voice text after the voice wake-up operation, the current voice input is not allowed to be accepted based solely on at least one of the visual weighting score, wake-up source score, or context score.

[0099] This protection rule essentially constructs a security constraint mechanism at the decision-making level. Specifically, the non-interactive dialogue judgment suggestion meets the preset non-interactive dialogue rejection condition. This means that the semantic evaluation result output by the language processing model during a single inference process clearly indicates that the probability of the current voice input belonging to a conversation with someone else, background noise, or non-directional content exceeds a preset security threshold. This condition represents a strong negative signal in the semantic dimension, indicating that from the perspective of text content and contextual understanding, the current voice is highly unlikely to be a valid instruction for this device. Under this premise, the system imposes strict restrictions on the acceptance of non-semantic evidence, that is, it does not allow the system to forcibly reverse the rejection decision solely based on any one or more combinations of visual weighting scores, wake-up source scores, or conversation context scores. For example, even if a high proportion of frontal facial orientation is detected within the current voice input time window, thus obtaining a high visual weighting score, or if the current voice immediately follows the device's question in time sequence, thus obtaining a conversation context score, as long as the semantic level has already given a clear non-interactive dialogue judgment, and the current voice is not the first voice after wake-up, the system will still maintain the rejection decision. This design effectively avoids misleading single-modal evidence due to environmental coincidences (such as someone else facing the device while asking a question), ensuring the dominance of semantic understanding in complex and open environments and reducing the risk of false responses.

[0100] The protection rule sets a clear exception: when the current voice input is the first voice text after a voice wake-up operation, the restriction does not take effect. The rationale for this exception is that users actively triggering the device through voice wake-up words, facial gaze, or page turning is itself the strongest explicit expression of their interactive intent. In the first round of interaction after wake-up, even if the user's expression is brief, colloquial, or even grammatically flawed, causing the language processing model to misinterpret it as non-interactive dialogue, the system should prioritize respecting the user's active triggering behavior and giving this round of voice a higher degree of trust to avoid affecting the user's interactive experience. In other words, the protection rule aims to filter out environmental interference without explicit interactive triggers, rather than suppressing genuine user-initiated interactions. Only when voice input occurs in subsequent rounds after wake-up, or when no wake-up operation is performed at all, will the protection rule activate its constraint on non-semantic evidence, thereby achieving a dynamic balance between interactive sensitivity and anti-interference capability.

[0101] To more clearly illustrate the technical effects of this embodiment, a specific application scenario is used for comparison. Assume a child companion robot is in picture book reading mode, and two adults are conversing nearby. One adult happens to be facing the robot while speaking, and their conversation happens to overlap with a brief silence period after the robot's previous reading. In a traditional solution without the protection rules of this embodiment, the detection of a facing direction generates a high visual weighting score. Combined with the coincidence of the timing window potentially adding contextual bonuses, the superposition of these non-semantic evidences could cause the overall credibility assessment index to exceed a preset threshold. This could lead the robot to mistakenly identify the conversation as a valid instruction and generate a response, resulting in a false response and interrupting the real user's reading experience. However, in the solution applying the protection rules of this embodiment, the language processing model can accurately identify the speech content as casual conversation between adults rather than an instruction to the robot through a single inference. The output non-interactive dialogue judgment suggestion meets the rejection condition. Simultaneously, since this speech input is not the first speech after the user actively wakes it up, the protection rules immediately take effect, prohibiting acceptance of the input based solely on visual weighting and contextual bonuses. Ultimately, the system correctly executed the rejection decision and did not generate any response, thus effectively avoiding the risk of false responses. Through the above mechanism, this embodiment provides a crucial semantic protection mechanism for voice interaction systems in open environments without sacrificing the smoothness of normal interaction.

[0102] Example 5

[0103] Based on the voice interaction processing method provided in the foregoing embodiments, this embodiment further describes the edge control mechanism outside the main process, specifically including special scenario bypass processing and continuous denial-of-identity listening termination strategy. These two control nodes are independent of the multi-dimensional confidence fusion calculation process, improving the overall robustness of the system from the two dimensions of business compatibility and resource effectiveness, respectively.

[0104] In one possible implementation, after acquiring multimodal factual information and speech text, when the speech text matches a preset special business scenario, it is processed according to the preset business logic corresponding to the preset special business scenario, and the corresponding interactive response is directly executed. This step typically serves as a pre-judgment stage before proceeding to the aforementioned multi-dimensional confidence fusion calculation. Preset special business scenarios refer to business types with clear instruction characteristics that can determine intent without relying on complex semantic reasoning or multimodal evidence verification, such as keyword matching, specific scenario confirmation, and picture book reading confirmation. When speech text matches these preset scenarios, the system will directly route to the existing dedicated business logic module for processing, instead of executing the semantic evaluation, visual evidence fusion, and comprehensive confidence calculation intent recognition processes described in this application embodiment. The technical consideration behind this bypass design is that existing dedicated business logic has often undergone long-term optimization, ensuring sufficient response speed and accuracy for specific instructions. Forcing it into a general multimodal fusion judgment framework could not only introduce unnecessary computational delays but also lead to misjudgments of precise instructions due to the generalization characteristics of the general model. By retaining this bypass mechanism, this application achieves compatibility with existing services while introducing new intent recognition capabilities, ensuring the stability and response efficiency of the system in mixed business scenarios.

[0105] In another possible implementation, within the same session, when a preset number of rejection decisions are made consecutively, a command to end the current voice monitoring is generated; the statistics of consecutive rejection decisions do not span different sessions. This mechanism aims to solve the problem of ineffective resource consumption caused by continuous false audio reception in open environments. In practical applications, environmental noise or idle chatter may cause the system to make consecutive rejection decisions in a short period of time. If the device remains in a monitoring state, it not only wastes power and computing power but may also raise user privacy concerns due to continuous uploading of audio data. This embodiment sets a threshold for the number of consecutive rejections (e.g., two). When the cumulative number of rejection decisions within the same session reaches this threshold, the server sends an end-of-monitoring command to the device, causing the device to actively exit the current voice recognition session. This statistics are strictly limited to the same session identifier; the number of rejections between different sessions does not affect each other and is not accumulated. This is because different sessions typically represent different interaction intentions or time segments. The end of a previous session often means a reset of the interaction context. If the rejection records of historical sessions are carried over to a new session, the user may be prematurely terminated from listening when initiating a valid interaction due to exceeding the historical rejection limit, thus impairing the normal interaction experience. Through this session-level isolation statistics and automatic termination mechanism, the system can ensure the continuity of normal multi-turn dialogues while promptly cutting off invalid environmental noise listening links, achieving a dynamic balance between interaction sensitivity and system efficiency.

[0106] Example 6

[0107] Based on the voice interaction processing method provided in the foregoing embodiments, this embodiment further details the specific content of the structured processing results returned by the server to the device. The content of the structured processing results varies depending on the decision made. This differentiated data structure design aims to decouple the server's intent recognition logic from the device's response execution logic, so that the device does not need to parse semantics or guess intent again, but can accurately execute the corresponding interactive operation based solely on standardized fields.

[0108] In this step, when the decision is to accept, the structured processing result includes at least a statement identifier, an acceptance decision, a comprehensive credibility assessment index, a reason for the decision, a response text, and a command to continue listening. The statement identifier uniquely associates the request and response, ensuring data consistency in multi-concurrency scenarios; the acceptance decision explicitly indicates that the current voice interaction on the device is valid; the comprehensive credibility assessment index provides a quantitative reference for the confidence level of this decision; the response text is the candidate response content generated by the language processing model, which the device uses for speech synthesis and playback; and the command to continue listening controls whether the device automatically starts the next round of voice acquisition after playing the response, maintaining the continuity of multi-round dialogues. For example, in a picture book reading Q&A scenario, when the robot asks "Which book do you want to listen to?" and receives the user's answer "This one" and determines it as accepted, the returned structured result includes not only the response text "Okay, let's read this one," but also a command to continue listening, so that after the robot reads the book title, it can continue waiting for the user's next page turn or confirmation command.

[0109] When the decision is rejection, the structured processing result includes at least the statement identifier, rejection decision, comprehensive credibility assessment index, and decision reason, but excludes the response text. This field definition is one of the key technical means of this application to prevent false responses. In traditional solutions, even if the system determines that a certain voice segment may not be a valid instruction, it may still return a default response (such as "I didn't hear you" or "Please say it again"), which can easily interrupt conversations or cause unnecessary noise interference in open environments. This embodiment explicitly stipulates at the protocol level that no response text is carried under rejection decision, forcing the device to remain silent when executing the rejection decision, without performing any voice broadcast or writing the current voice input into the formal dialogue history context. This mechanism of "not returning response text when rejecting" cuts off the execution path of false responses from the source of data transmission, ensuring absolute quietness and imperceptibility when the system filters non-interactive dialogues, and improving the user experience.

[0110] Furthermore, regardless of whether the decision is accepted or rejected, the structured processing results include a "decision reason" field. This field records key factors or triggering rules that lead to the current decision result, such as "low visual evidence score," "semantic judgment as non-interactive dialogue," "bonus points for hitting the context of the response," or "negative correction for sentence completeness." The technical value of the decision reason field lies in providing a data foundation for the system's observability and continuous optimization. For the device side, the decision reason can be written to local logs or uploaded to a cloud analysis platform to help developers quickly locate the root cause of false rejections or false responses. For the user feedback mechanism, when a user is dissatisfied with the robot's response and provides feedback, the backend can directly associate it with the decision reason at that time, thereby accurately determining whether it is caused by model capability issues, sensor malfunctions, or environmental interference, rather than relying solely on vague user descriptions for troubleshooting. It should be understood that the fields listed above are only illustrative examples. In practical applications, the structured processing results can also be extended to include session state markers, sentiment tags, or other metadata according to business needs. As long as the core fields are differentiated between the acceptance and rejection states, they all fall within the scope of protection of this application.

[0111] Example 7

[0112] To more clearly illustrate the technical effects of the voice interaction processing method provided in this application in practical applications, a comprehensive demonstration is given below using a child companion robot in an open home environment for picture book reading as an example. It should be understood that this scenario is merely an illustrative example and is not intended to limit the scope of protection of this application. The method of this application is also applicable to other open voice interaction devices with background noise or interference from others, such as home service robots, smart speakers, and in-vehicle voice assistants.

[0113] In this example scenario, the child companion robot is in picture book reading mode, reading a story to a child. During the brief silence period after the robot finishes reading, two adults are conversing nearby. A segment of one adult's conversation is picked up by the robot's microphone array and transcribed into text by the speech recognition system. Simultaneously, the adult is facing the robot sideways while speaking, causing the device's camera to detect a human silhouette within the current voice input window, but not a clear facial orientation or outline. At this point, the device, following the description of step S100 in Embodiment 1 above, acquires multimodal factual information including the adult's conversation text, human silhouette proportion statistics, and temporal state information, and sends it to the server. It should be noted that since this voice input is not initiated by the user actively triggering a wake word or facial gaze, the wake-up source field in the temporal state information is empty or marked as non-wakeful.

[0114] After receiving the above information, the server executes the single-inference mechanism in step S200 of the aforementioned embodiment 1. The language processing model completes the semantic understanding and response generation preparation of the conversation text in parallel within the same call request. The semantic evaluation results output by the model show that the semantic orientation score of the text content is low, and the non-interactive dialogue judgment suggestion clearly indicates that the probability of the content belonging to a conversation between strangers exceeds the preset safety threshold, thus satisfying the non-interactive dialogue rejection condition. Although the model also generates candidate response content for the text, since the subsequent fusion decision will determine whether the content is adopted, the response generation at this time is only a preparatory action under the computing power reuse mechanism and does not generate additional network request overhead. Compared to the traditional serial processing mode that requires first calling the intent recognition model to judge validity and then calling the response generation model after confirming validity, this embodiment reduces the end-to-end response latency from voice input to decision output through single-inference, avoiding the interactive stuttering caused by multiple model calls.

[0115] Subsequently, the server proceeds to the multi-dimensional confidence fusion calculation process described in Embodiments 2 and 3. In the visual evidence score calculation stage, although no frontal or human face is detected within the current time window, based on the proportion of detected human body contours, the system calculates a non-zero visual evidence score according to the principle of prioritizing strong evidence, and obtains a certain visual weighted score after weighting. However, when executing the non-interactive dialogue protection rule described in Embodiment 4, the system identifies two key conditions simultaneously: first, the non-interactive dialogue judgment suggestion output by the language processing model meets the rejection condition; second, the current voice input is not the first voice text after the voice wake-up operation. Based on this, the protection rule immediately takes effect, prohibiting the forced reversal of semantic-level negative judgments solely based on visual weighted scores or other corrections. This means that even if there are coincidental visual signals or temporal connections in the environment, as long as the semantic core clearly points to a non-interactive object and there is a lack of explicit wake-up confirmation, the system will still maintain respect for semantic dominance. This mechanism effectively solves the common "visual misleading" problem in traditional multimodal fusion schemes, i.e., preventing high-confidence misjudgments caused by someone accidentally facing the device or speaking during a questioning interval.

[0116] After being constrained by fusion calculation and protection rules, the final comprehensive credibility assessment index is lower than the preset threshold. Based on this, the server determines the rejection decision and generates a structured processing result without reply text, as defined in Embodiment 6 above, and returns it to the device. After parsing the result, the device performs a silent operation, neither playing any voice reply nor adding the adult conversation to the formal dialogue history context, thus completely avoiding interruption of the real user's reading experience and rude interference with other people's conversations. Furthermore, if a second similar mis-recorded conversation occurs in the same session after this rejection and is also judged as a rejection, the server will generate a command to end the current voice monitoring and send it to the device, according to the continuous rejection monitoring termination strategy described in Embodiment 5 above. Upon receiving the command, the device will actively exit the current voice recognition session and stop audio acquisition and uploading. This session-level automatic termination mechanism not only promptly cuts off the continuous consumption of system resources by invalid environmental noise, but also reduces the retention of user audio data during unnecessary periods from the perspective of privacy protection. It demonstrates the comprehensive robustness of the system in an open environment, taking into account interactive sensitivity, anti-interference ability and resource efficiency.

[0117] The above describes a voice interaction processing method provided by an embodiment of this application. The following describes the apparatus for performing the above-described voice interaction processing method.

[0118] Please see Figure 2 , Figure 2 This is a schematic diagram of the structure of a voice interaction processing device provided in an embodiment of this application. Figure 2 As shown, the voice interaction processing device includes:

[0119] The acquisition unit 11 is used to acquire multimodal factual information within the current voice input time window, as well as the voice text corresponding to the current voice input;

[0120] Inference unit 12 is used to input the speech text and historical dialogue context into the language processing model, and obtain semantic evaluation results and candidate response content through a single inference;

[0121] The multi-dimensional confidence fusion calculation unit 13 is used to perform multi-dimensional confidence fusion calculation based on the multimodal factual information, the historical dialogue context, the semantic evaluation result and the speech text to obtain a comprehensive credibility evaluation index.

[0122] The decision-making unit 14 is used to determine the acceptance or rejection decision based on the comparison result between the comprehensive credibility evaluation index and the preset threshold.

[0123] The processing result generation and feedback unit 15 is used to generate a structured processing result based on the decision and the candidate response content and return it to the device, so that the device can perform the corresponding response operation based on the structured processing result.

[0124] Each unit in the aforementioned voice interaction processing device can be implemented entirely or partially through software, hardware, or a combination thereof. These units can be embedded in or independent of the processor in a computer device, or stored in the computer device's memory as software, so that the processor can call and execute the corresponding operations of each unit.

[0125] This application also provides an electronic device in its embodiments. (See reference...) Figure 3 The diagram illustrates a structural schematic suitable for implementing the electronic device in the embodiments of this application. The electronic device in the embodiments of this application may include, but is not limited to, fixed terminals such as mobile phones, laptops, PDAs (personal digital assistants), PADs (tablet computers), desktop computers, etc. Figure 3 The electronic device shown is merely an example and should not impose any limitation on the functionality and scope of use of the embodiments of this application.

[0126] like Figure 3 As shown, the electronic device may include a processing unit (e.g., a central processing unit, a graphics processing unit, etc.) 601, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 602 or a program loaded from a storage device 608 into a random access memory (RAM) 603. When the electronic device is powered on, the RAM 603 also stores various programs and data required for the operation of the electronic device. The processing unit 601, ROM 602, and RAM 603 are interconnected via a bus 604. An input / output (I / O) interface 605 is also connected to the bus 604.

[0127] Typically, the following devices can be connected to I / O interface 605: input devices 606 including, for example, touchscreens, touchpads, keyboards, mice, cameras, microphones, accelerometers, gyroscopes, etc.; output devices 607 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; storage devices 608 including, for example, memory cards, hard drives, etc.; and communication devices 609. Communication device 609 allows electronic devices to communicate wirelessly or wiredly with other devices to exchange data. Although Figure 3 Electronic devices with various devices are shown, but it should be understood that it is not required to implement or have all of the devices shown. More or fewer devices may be implemented or have instead.

[0128] This application also provides a computer program product including computer-readable instructions, which, when executed on an electronic device, cause the electronic device to implement any of the voice interaction processing methods provided in this application.

[0129] This application also provides a computer-readable storage medium that carries one or more computer programs. When the one or more computer programs are executed by an electronic device, the electronic device can implement any of the voice interaction processing methods provided in this application.

[0130] It should also be noted that the device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. In addition, in the device embodiment drawings provided in this application, the connection relationship between modules indicates that they have a communication connection, which can be implemented as one or more communication buses or signal lines.

[0131] Through the above description of the embodiments, those skilled in the art can clearly understand that this application can be implemented by means of software plus necessary general-purpose hardware, or it can be implemented by special-purpose hardware including application-specific integrated circuits, special-purpose CPUs, special-purpose memory, special-purpose components, etc. Generally, any function performed by a computer program can be easily implemented by corresponding hardware, and the specific hardware structure used to implement the same function can also be diverse, such as analog circuits, digital circuits, or special-purpose circuits. However, for this application, software program implementation is more often the preferred implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a readable storage medium, such as a computer floppy disk, USB flash drive, mobile hard disk, ROM, RAM, magnetic disk, or optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, training equipment, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0132] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, as a computer program product.

[0133] The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions may be transmitted from one website, computer, training device, or data center to another website, computer, training device, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium may be any available medium that a computer can store or a data storage device such as a training device or data center that integrates one or more available media. The available media may be magnetic media (e.g., floppy disks, hard disks, magnetic tapes), optical media (e.g., DVDs), or semiconductor media (e.g., solid-state drives (SSDs)).

Claims

1. A voice interaction processing method, characterized in that, The method includes: Obtain multimodal factual information within the current voice input time window, as well as the voice text corresponding to the current voice input; The speech text and historical dialogue context are input into the language processing model, and semantic evaluation results and candidate response content are obtained through a single inference. Based on the multimodal factual information, the historical dialogue context, the semantic evaluation results, and the speech text, a multi-dimensional confidence fusion calculation is performed to obtain a comprehensive credibility evaluation index. Based on the comparison results between the comprehensive credibility assessment index and the preset threshold, a decision to accept or reject is made. Based on the decision and the candidate response content, a structured processing result is generated and returned to the device, so that the device can perform the corresponding response operation based on the structured processing result.

2. The method according to claim 1, characterized in that, The multimodal factual information includes visual detection information and temporal state information; the semantic evaluation result includes a semantic orientation score and a non-interactive dialogue determination suggestion; wherein, the semantic orientation score is used to characterize the probability evaluation of the voice input for this device, and the non-interactive dialogue determination suggestion is used to characterize the determination result that the voice input belongs to a non-interactive dialogue; the candidate response content includes the response text and an indicator of whether the response text expects further response. The multi-dimensional confidence fusion calculation based on the multimodal factual information, the historical dialogue context, the semantic evaluation result, and the speech text includes: Based on the visual detection information in the multimodal factual information, a visual evidence score is calculated; The semantic orientation score and the visual evidence score are weighted and calculated respectively to obtain a semantic weighted score and a visual weighted score; The semantic weighted score and the visual weighted score are corrected based on the temporal state information in the multimodal factual information, the historical dialogue context, and the voice text to obtain the comprehensive credibility evaluation index.

3. The method according to claim 2, characterized in that, The visual inspection information includes visual inspection statistics obtained based on image detection and visual inspection service availability information. The methods for determining the visual evidence score include: When it is determined that the visual detection service is available based on the visual detection service availability information, the proportion of facial orientation, facial contour, and human body contour detected within the current voice input time window is obtained based on the visual detection statistics. Based on the facial orientation ratio, the facial contour ratio, and the human body contour ratio, the first evidence score, the second evidence score, and the third evidence score are calculated respectively. The maximum value among the first evidence score, the second evidence score, and the third evidence score is determined as the visual evidence score; When it is determined that the visual inspection service is unavailable based on the availability information of the visual inspection service, the preset neutral evidence score is determined as the visual evidence score.

4. The method according to claim 2, characterized in that, The semantic weighted score and the visual weighted score are corrected based on the temporal state information in the multimodal factual information, the historical dialogue context, and the speech text to obtain the comprehensive credibility evaluation index, including: Based on the temporal state information, the historical dialogue context, and the voice text, the wake-up source score, the response context score, and the sentence integrity correction score are calculated respectively. The semantic weighted score, the visual weighted score, the wake-up source score, the conversation context score, and the sentence integrity correction score are fused together to obtain the comprehensive credibility evaluation index.

5. The method according to claim 4, characterized in that, The timing status information includes voice time window information, device playback status information, wake-up source information, and session identifier information; The calculation method for the wake-up source bonus includes: When the current voice input is the first voice text after wake-up, the corresponding bonus value is determined according to the wake-up source information in the timing status information; wherein, different wake-up source types correspond to different bonus values, and the wake-up source types include voice wake-up, face wake-up, and page-turning wake-up; When the current voice input is not the first voice text after wake-up, the wake-up source score is zero.

6. The method according to claim 4, characterized in that, The historical dialogue context includes the previous round of device dialogue, an indicator of whether the previous round of device dialogue expects a response, the previous round of decision results, and the number of consecutive rejections within the same session; The calculation method for the context score includes: Obtain an identifier from the historical dialogue context indicating whether the previous round of device dialogue expects a response; When the indicator indicating whether the previous device dialogue is expected to receive a response indicates that a response is expected, and the current voice input occurs during or after the device plays the previous device dialogue, within a preset time period, and the voice text constitutes a response to the previous device dialogue, the context score for the conversation is increased.

7. The method according to claim 4, characterized in that, The calculation method for the sentence integrity correction score includes: Based on the text length, grammatical structure, semantic completeness, and contextual relationships of the spoken text, determine whether the spoken text is a complete sentence; When the spoken text is determined not to be a complete sentence, the sentence integrity correction score is negatively corrected.

8. The method according to claim 7, characterized in that, The step of determining whether the spoken text is a complete sentence includes: When the length of the voice text exceeds a preset length threshold, and the semantics of the voice text are interrupted or lack core grammatical components, and the voice text has no connection with the previous round of device dialogue, the voice text is determined to be not a complete sentence. When the length of the voice text does not exceed the preset length threshold, the voice text is judged to be a complete sentence in combination with the previous round of device dialogue.

9. The method according to claim 4, characterized in that, The method further includes: When the non-interactive dialogue determination suggestion meets the preset non-interactive dialogue rejection conditions, and the current voice input is not the first voice text after the voice wake-up operation, the current voice input is not allowed to be accepted based solely on at least one of the visual weighted score, the wake-up source score, and the conversation context score.

10. The method according to claim 1, characterized in that, The method further includes: Within the same session, when a preset number of rejection decisions are made consecutively, an instruction to end the current voice monitoring is generated; The statistics of the number of consecutive rejection decisions do not span different sessions.

11. The method according to claim 1, characterized in that, The method further includes: After acquiring the multimodal factual information and the voice text, when the voice text matches a preset special business scenario, it is processed according to the preset business logic corresponding to the preset special business scenario, and the interactive response corresponding to the preset business logic is directly executed.

12. The method according to claim 1, characterized in that, The content of the structured processing result varies depending on the decision made: When the decision is to accept, the structured processing result includes at least the statement identifier, the acceptance decision, the comprehensive credibility assessment index, the reason for the decision, the response text, and the instruction to continue listening; When the decision is to reject, the structured processing result includes at least the statement identifier, the rejection decision, the comprehensive credibility assessment index, and the decision reason, but does not include the response text.

13. A computer program product, characterized in that, It includes computer-readable instructions that, when executed on an electronic device, cause the electronic device to implement the voice interaction processing method as described in any one of claims 1 to 12.

14. An electronic device, characterized in that, It includes at least one processor and a memory connected to the processor, wherein: The memory is used to store computer programs; The processor is used to execute the computer program to enable the electronic device to implement the voice interaction processing method as described in any one of claims 1 to 12.

15. A computer-readable storage medium, characterized in that, The storage medium carries one or more computer programs that, when executed by an electronic device, enable the electronic device to implement the voice interaction processing method as described in any one of claims 1 to 12.

Images