Voice response method and device and electronic equipment

Abstract

The invention discloses a voice response method and device and electronic equipment. The voice response method comprises the following steps: collecting a current input voice and multi-modal information associated with the input voice; inputting the input voice and the multi-mode information into a preset voice instruction arbitration model, and judging whether the input voice is instruction voice or not through the voice instruction arbitration model; and if it is determined that the input voice is the commanding voice, executing a voice instruction corresponding to the input voice. According to the method and the device, by setting the corresponding voice instruction arbitration model, the instruction voice and the daily dialogue or the environment interference can be accurately recognized, and the response is performed in a targeted manner, so that the dependence of a traditional voice assistant on a specific wake-up word is eliminated, and the user experience is improved.

Description

Technical Field

[0001] This invention relates to the field of speech processing technology, and more particularly to speech response methods and apparatus, and electronic devices. Background Technology

[0002] Currently, intelligent voice assistants (such as Amazon Alexa, Google Assistant, and Huawei Xiaoyi) are widely used in various devices such as smart homes, smart cockpits, and smart speakers. However, these voice assistants all rely on specific wake words (such as "Xiaoyi Xiaoyi" or "Hey Google") to activate their voice command reception function. In practical applications, while the wake word mechanism can reduce false triggers to some extent, it forces users to repeatedly say the wake word when interacting with the device, resulting in cumbersome operations and a redundant experience. For example, when executing multiple commands, users must say the wake word to wake up the device each time, which not only reduces interaction efficiency but also affects the user experience. Summary of the Invention

[0003] This invention provides a voice response method, apparatus, and electronic device to solve the problem of cumbersome operation caused by existing voice assistants relying on wake words.

[0004] In a first aspect, the present invention provides a voice response method. The voice response method includes: acquiring current input speech and multimodal information associated with the input speech; inputting the input speech and the multimodal information into a preset voice command arbitration model to determine whether the input speech is a command speech through the voice command arbitration model; and if the input speech is determined to be the command speech, executing a voice command corresponding to the input speech.

[0005] In one implementation of the first aspect, the voice response method further includes: if it is determined that the input voice is not the instruction voice, then the input voice is ignored.

[0006] In one implementation of the first aspect, determining whether the input speech is a directive speech through the voice command arbitration model includes: determining whether the input speech is a directive speech by using the voice command arbitration model, which uses the identity confidence, tone reasonableness, context reasonableness, environment matching degree, and habit matching degree associated with the input speech as the determination criteria.

[0007] In one implementation of the first aspect, the voice response method further includes: acquiring training data and training a large language model based on the training data to obtain the voice command arbitration model.

[0008] In one implementation of the first aspect, the voice response method further includes: collecting user interaction feedback data and constructing incremental training data based on the user interaction feedback data; and incrementally fine-tuning the voice command arbitration model based on the incremental training data to achieve continuous optimization of the voice command arbitration model.

[0009] In one implementation of the first aspect, acquiring training data includes: collecting input speech and multimodal information associated with the input speech; preprocessing the input speech and the multimodal information respectively to obtain corresponding multidimensional perception scores; acquiring a preset arbitration formula and determining an execution strategy based on the arbitration formula and the multidimensional perception scores; and constructing labeled data based on the input speech, the multimodal information, the multidimensional perception scores, the arbitration formula, and the execution strategy to serve as the training data.

[0010] In one implementation of the first aspect, preprocessing the input speech and the multimodal information to obtain a corresponding multidimensional perceptual score includes: performing timbre recognition on the input speech to obtain an identity confidence score; performing tone analysis on the input speech to obtain a tone reasonableness score; obtaining contextual information of the input speech and performing semantic analysis on the input speech in conjunction with the contextual information to obtain a contextual reasonableness score; obtaining an environment matching score based on environmental state information in the multimodal information; obtaining a habit matching score based on user habit information in the multimodal information; and associating the identity confidence score, the tone reasonableness score, the contextual reasonableness score, the environment matching score, and the habit matching score as a multidimensional perceptual score for the input speech.

[0011] In one implementation of the first aspect, determining the execution strategy based on the arbitration formula and the multi-dimensional perception score includes: calculating the comprehensive confidence score of the input speech based on the multi-dimensional perception score and the preset arbitration formula; and determining the execution strategy of the input speech based on the comprehensive confidence score.

[0012] In one implementation of the first aspect, determining the execution strategy for the input speech based on the comprehensive confidence score includes: comparing the comprehensive confidence score with a preset first threshold and a second threshold; if the comprehensive confidence score is greater than or equal to the first threshold, then setting the execution strategy as an execution instruction; if the comprehensive confidence score is less than the first threshold but greater than or equal to the second threshold, then sending a confirmation request to the user and using the feedback result of the confirmation request as the execution strategy; if the comprehensive confidence score is less than the second threshold, then setting the execution strategy as not responding to the input speech.

[0013] In one implementation of the first aspect, training the large language model based on the training data to obtain the voice command arbitration model includes: supervising the fine-tuning of the large language model based on the training data until the model evaluation of the large language model meets the standard; and using the large language model that meets the model evaluation standard as the voice command arbitration model.

[0014] In one implementation of the first aspect, the voice response method further includes: deploying the voice command arbitration model on the local device where the user terminal is located.

[0015] Secondly, the present invention provides a voice response device. The voice response device includes: an information acquisition module configured to acquire current input speech and multimodal information associated with the input speech; a model inference module configured to input the input speech and the multimodal information into a preset voice command arbitration model to determine whether the input speech is a command speech through the voice command arbitration model; and an execution module configured to execute a voice command corresponding to the input speech if the input speech is determined to be the command speech.

[0016] Thirdly, the present invention provides an electronic device. The electronic device includes: a memory configured to store an executable program; and a processor configured to execute the program to perform the method described in any of the preceding embodiments.

[0017] According to embodiments of this disclosure, when corresponding input speech is collected, the input speech and its associated multimodal information are directly acquired, and both are input into a dedicated voice command arbitration model for reasoning. This determines whether the input speech is a genuine and valid command, achieving a comprehensive and in-depth judgment of the user's intent. Based on this, if it is a command, the corresponding voice command is executed to respond, thereby waking up the intelligent voice assistant to execute the corresponding command. In summary, by setting a corresponding voice command arbitration model, this invention eliminates the dependence of traditional voice assistants on specific wake words, accurately identifies command speech from daily conversations or environmental interference, and responds accordingly. This fundamentally reduces the false trigger rate, upgrades the interaction method from passive wake-up to active perception, and improves the system's intelligence level and user experience. Attached Figure Description

[0018] Figure 1 This is a flowchart illustrating a voice response method according to an embodiment of the present disclosure.

[0019] Figure 2 This is a flowchart illustrating an application example of a voice response method according to an embodiment of the present disclosure.

[0020] Figure 3 This is a block diagram illustrating a voice response device according to an embodiment of the present disclosure. Detailed Implementation

[0021] To explain in detail the technical content, objectives, and effects of the present invention, the following description is provided in conjunction with the embodiments and accompanying drawings.

[0022] Currently, mainstream voice assistant systems (such as Amazon Alexa, Google Assistant, Apple Siri, Baidu Xiaodu, and Huawei Xiaoyi) are widely used in smart speakers, smart homes, and smart cockpits. However, these voice assistant systems all share a common limitation in their interaction design: they must rely on specific wake words to activate the device's command receiving function. It's worth noting that the main purpose of setting up a wake word mechanism is to reduce misinterpretations and misprocessing by smart devices, thereby improving accuracy, privacy, security, and user experience. This wake word mechanism ensures that the device only responds to specific wake words, effectively distinguishing between commands and irrelevant conversations, avoiding accidental operations, protecting user privacy, and improving reliability. For example, if a user is talking to family members at home, the device might mistakenly interpret casual conversation like "turn on the heater" or "set the alarm" as commands, leading to wasted resources and undesirable consequences for the user.

[0023] However, in practical use, the forced wake-up mechanism of the aforementioned wake words makes the entire interaction process extremely cumbersome and redundant, resulting in a poor user experience. Take a typical smart home scenario as an example: if a user wants to play music before opening the curtains, they must say the complete commands "Hey Celia, play a song" and "Hey Celia, open the curtains" twice. Similarly, in a car environment, executing two separate commands, "open the car window" and "check today's weather," also requires two separate wake-up calls. This frequent and repetitive wake-up process disrupts the natural flow of the interaction, significantly reduces the user experience, and causes many users to abandon voice assistants, opting instead for manual operation or more direct methods like using their phones.

[0024] The above-mentioned mechanically repetitive interaction mode exposes the deep defects in the underlying architecture of the current voice assistant: (1) Over-reliance on wake words: The system requires the user to actively say a specific wake word to be activated. (2) Adoption of single-point decision mechanism: Intent recognition only relies on the text content after speech recognition, lacking a comprehensive evaluation of multi-dimensional information such as the identity of the instruction issuer, environmental context, and dialogue history, resulting in high false trigger rate (such as idle chat being mistakenly executed) and missed trigger rate (such as euphemistic instructions being ignored). (3) Separation of identity and intent recognition: Speaker recognition technology is usually only used for voiceprint unlocking or personalized recommendations, and fails to be dynamically bound to the execution permission of the instruction. (4) Environmental data is not involved in decision-making: Environmental information such as weather, time, temperature and humidity is mostly used for information broadcasting, but not for evaluating the rationality of the instruction in the current scenario. (5) Dialogue history is not effectively utilized: Historical dialogue records are only used for reference resolution or idle chat continuation, and are not weighted for enhancing the calculation of the confidence of the current statement intent. (6) Large models are used only as text classifiers: The capabilities of large language models are simplified and used only for text intent classification, failing to fully realize their potential in multimodal reasoning and decision interpretation. (7) Static fixation after model deployment: Once deployed, the system cannot be optimized based on real user interaction behavior, leading to long-term misjudgment problems. (8) Lack of user calibration mechanism: The system lacks uncertainty assessment and active confirmation links when executing instructions, and users lack effective correction opportunities and interaction channels. (9) Lack of data closure loop: User feedback behavior is not collected and utilized in a structured manner, wasting valuable labeled data resources, and the system cannot form a closed loop for continuous optimization.

[0025] To address at least the aforementioned technical problems, this disclosure provides a voice response method. When corresponding input voice is acquired, the method directly obtains the input voice and its associated multimodal information, and inputs both into a dedicated voice command arbitration model for reasoning. This determines whether the input voice is a genuine and valid command, achieving a comprehensive and in-depth judgment of the user's intent. Based on this, if it is a command, the method responds by executing the corresponding voice command, thereby waking up the intelligent voice assistant to execute the corresponding command. In summary, by setting up a corresponding voice command arbitration model, this invention eliminates the dependence of traditional voice assistants on specific wake words, accurately identifies command voices from daily conversations or environmental interference, and responds accordingly. This fundamentally reduces the false trigger rate, upgrades the interaction method from passive wake-up to active perception, and improves the system's intelligence level and user experience.

[0026] In the following, the technical solutions according to this disclosure will be described with reference to specific embodiments and in conjunction with the accompanying drawings.

[0027] Figure 1 This is a flowchart illustrating a voice response method 100 according to an embodiment of the present disclosure. (Refer to...) Figure 1 The voice response method 100 includes the following steps 102 to 106.

[0028] Step 102: Collect the current input speech and the multimodal information associated with the input speech.

[0029] The input speech refers to the audio signal emitted by the user and captured by audio acquisition devices such as microphones. The multimodal information refers to auxiliary information acquired synchronously with the input speech, such as current environmental status information, user habit information, and dialogue context information. The environmental status information may include environmental information such as time, geographical location, weather, temperature, humidity, and light intensity, which is typically obtained through device clocks, GPS, sensors, or network APIs. The user habit information refers to the user's historical behavioral patterns, such as sleep schedules and device usage habits. Furthermore, the dialogue context information refers to information such as records of previous rounds of human-machine or human-to-human dialogue.

[0030] Step 104: Input the input speech and the multimodal information into a preset speech command arbitration model, so as to determine whether the input speech is a command speech through the speech command arbitration model.

[0031] The commanding voice refers to voice content actively issued by an authorized user based on a clear intention, aimed at triggering the smart device to perform a specific operation, rather than interfering voices such as idle chatter, mishearing, environmental noise, or unauthorized human voices.

[0032] The voice command arbitration model is a specially trained decision-making model capable of fusing and reasoning about the current input voice and its multimodal information, and outputting a corresponding judgment result. The judgment result can be that the input voice is a command voice, or that the input voice is not a command voice (i.e., non-command voice).

[0033] Step 106: If the input voice is determined to be the instruction voice, then execute the voice instruction corresponding to the input voice.

[0034] In some embodiments, executing a corresponding voice command means that the control system executes the command operation corresponding to the input voice, such as turning on the air conditioner or playing music.

[0035] In some embodiments, the voice response method may further include: if it is determined that the input voice is not the command voice, then ignoring the input voice. It should be noted that ignoring the input voice means not responding to the input voice, to avoid accidental operation.

[0036] As described above, for non-command voice input, the input voice is directly ignored, thereby effectively reducing false responses to irrelevant voice input, minimizing interference to the user caused by responding to non-command voice input, and improving the system's intelligence level and user experience.

[0037] In some embodiments, determining whether the input speech is a directive through the voice command arbitration model includes: determining whether the input speech is a directive by using the voice command arbitration model, which uses the identity confidence, tone reasonableness, context reasonableness, environment matching degree, and habit matching degree associated with the input speech as the determination criteria.

[0038] As described above, by using identity confidence, tone reasonableness, context reasonableness, environment matching degree, and habit matching degree as the judgment criteria of the voice command arbitration model, the voice command arbitration model can comprehensively evaluate the credibility and scenario reasonableness of the input voice, significantly improving the robustness and accuracy of intent recognition in complex scenarios.

[0039] It should be noted that the above-mentioned voice command arbitration model is trained based on a large language model. The following describes in detail how the voice command arbitration model is trained based on a large language model.

[0040] In some embodiments, the voice response method may further include: acquiring training data and training a preset large language model based on the training data to obtain the voice command arbitration model.

[0041] The large language model is a basic artificial intelligence model that is trained on massive amounts of data, can understand and generate natural language, and can be fine-tuned to adapt to specific tasks. For example, it can be the Qwen large model.

[0042] As described above, by leveraging the powerful contextual understanding and reasoning capabilities of large language models and training them with specific training data, a highly optimized model specifically designed for voice command arbitration can be obtained for subsequent voice command arbitration.

[0043] In some embodiments, acquiring training data includes: collecting input speech and multimodal information associated with the input speech; preprocessing the input speech and the multimodal information respectively to obtain corresponding multidimensional perception scores; acquiring a preset arbitration formula and determining an execution strategy based on the arbitration formula and the multidimensional perception scores; and constructing labeled data based on the input speech, the multimodal information, the multidimensional perception scores, the arbitration formula, and the execution strategy to serve as the training data.

[0044] The labeled data refers to a standardized dataset automatically generated by a rule-driven system for training a speech command arbitration model. Its core component is to associate the original input speech and corresponding multimodal information with the obtained multidimensional perceptual scores, comprehensive confidence scores, and execution strategies as standard answers, thereby forming input-output paired supervised learning samples. In essence, it provides standard answers generated by a controllable rule system for large language models, so as to realize the model's mapping learning from perceptual information to intelligent decision-making.

[0045] The preprocessing refers to the process of extracting and quantifying features from the input speech and multimodal information from different dimensions to generate perceptual scores of different dimensions. The multidimensional perceptual score refers to a set of scores from multiple dimensions that quantify the credibility of the input speech. It may include the identity confidence score (i.e., Score_ID) obtained through timbre recognition, the tone reasonableness score (i.e., Score_Tone) obtained through tone analysis, the context reasonableness score (i.e., Score_Dialog) obtained through semantic analysis combined with context, the environment matching score (i.e., Score_Env) calculated based on environmental state information, and the habit matching score (i.e., Score_Habit) derived from user habit information. Each score is a continuous value in the interval [0, 1], which is used to calculate the comprehensive confidence score in the subsequent arbitration formula.

[0046] The arbitration formula is a weighted summation formula used to calculate the overall confidence score, which can be expressed as: Overall Confidence Score = α×Score_ID + β×Score_Tone + γ×Score_Dialog + δ×Score_Env + ε×Score_Habit, where α, β, γ, δ, and ε are preset weight coefficients.

[0047] As described above, by associating input speech, multimodal information, multidimensional perceptual scores, arbitration formulas, and execution strategies as a single labeled data set, each labeled data set contains input features, reasoning basis, and standard answer. This provides high-quality training materials for supervised fine-tuning, fundamentally improving the model's learning effect and decision-making accuracy.

[0048] In some embodiments, preprocessing the input speech and the multimodal information to obtain a multidimensional perceptual score includes: performing timbre recognition on the input speech to obtain an identity confidence score; performing tone analysis on the input speech and obtaining tone reasonableness based on the tone analysis results; obtaining contextual information of the input speech and performing semantic analysis on the input speech in conjunction with the contextual information to obtain contextual reasonableness; obtaining an environment matching degree based on environmental state information in the multimodal information; obtaining a habit matching degree based on user habit information in the multimodal information; and associating the identity confidence score, tone reasonableness, contextual reasonableness, environment matching degree, and habit matching degree as a multidimensional perceptual score for the input speech.

[0049] In practical applications, to achieve accurate judgment of user intent, this invention introduces a multi-layer self-optimizing arbitration mechanism. This mechanism is based on deep learning algorithms and uses multi-dimensional information such as speaking time, location, tone, and dialogue context to collaboratively analyze and comprehensively judge the user's true intent.

[0050] (1) Voice Recognition: Voice characteristics are collected by pre-recording user audio. When a command such as "play music" is received, the speaker recognition function is activated first to compare the current input speech with the pre-recorded voiceprint to verify whether it comes from an authorized user. This effectively eliminates interference from television, other devices, or visitor voices. It should be noted that the corresponding function module of voice recognition takes the original input speech as input and outputs a quantified identity confidence score Score_ID∈[0,1]. This identity confidence score is used to represent the probability that the current input speech comes from an authorized user. In practical applications, open-source speaker recognition technology (such as voiceprint embedding model) can be used for voice recognition to output a quantified identity confidence score.

[0051] (2) Tone / Context Analysis: By analyzing the emotions, attitudes, and intonations (such as anger and questioning) in the input speech, the system identifies whether the speaker's tone is consistent with the current context, thus obtaining a tone reasonableness score. For example, when a user says angrily, "Why did you play music yesterday? It was so noisy!", the system can determine from the tone analysis that "playing music" is a complaint rather than an instruction. It should be noted that the main function of tone / context analysis is to evaluate the reasonableness of tone. Its input is the original input speech, and the output is the tone reasonableness score Score_Tone ∈ [0, 1]. In practical applications, when performing tone analysis, an open-source tone recognition model can be used, with the focus on excluding abnormal tones / abnormal instructions.

[0052] (3) Understanding complete sentences and their context: Semantic coherence is determined by analyzing complete sentences and dialogue history to obtain a contextual reasonableness score, which is used to evaluate whether the current input speech constitutes a valid instruction. For example, when a user says, "The music you played yesterday was really good, can you send me the song title?", although it contains the field "music played", the context indicates that this is a dialogue rather than an instruction. The input is the most recent N rounds of dialogue records, and the output contextual reasonableness score is Score_Dialog∈[0,1]. For example, if the previous sentence is "It's so hot", and the current input speech is "Turn on the air conditioner?", the contextual reasonableness score can be 0.9; if the previous sentence is "Tell a joke", and the current input speech is "Turn on the air conditioner?", the contextual reasonableness score can be 0.2. In practical applications, the input speech can be converted into text through a speech-to-text module, and then handed over to the corresponding large language model for semantic analysis to identify the intent, thereby obtaining the corresponding contextual reasonableness score.

[0053] (4) Environmental Perception: The rationality of the instruction is evaluated by comprehensively considering environmental factors such as time, location, and season, thereby obtaining an environmental matching degree. For example, when receiving "open the curtains", the system will first determine the current time. For example, if it is 11 pm, which is bedtime, the system can determine with a high probability that it is not an instruction issued by the user and is a distractor. When receiving "open the car window", the system will obtain the current location. If it detects that you are at home and there is no car window, it can be clearly determined that this sentence is a distractor and not an instruction. When receiving "turn on the heater", the system will obtain the current time and location. If it is summer and you are in a southern city, it can be clearly determined that this sentence is a distractor and not an instruction.

[0054] In other words, through environmental perception, the matching degree between the input speech and the physical environment can be evaluated. The input includes sensor data such as time, geographical location, and weather, and the output is the environmental matching degree Score_Env∈[0,1]. This environmental matching degree is used to characterize the degree of matching between the input speech and the environmental state information. For example, the environmental matching degree score of "turn on the air conditioner" in a 35°C environment is 0.95; the environmental matching degree score of "turn on the heater" in a 30°C summer environment is 0.1; and the environmental matching degree score of "turn on the lights" at 8 pm is 0.9.

[0055] (5) Habit perception: By learning users' lifestyle habits (such as entertainment and bedtime at fixed times), it determines whether the current input speech conforms to the individual's behavioral pattern, thereby obtaining a habit matching degree (i.e., Score_Habit). Through this habit perception, the matching degree between the input speech and the user's habits can be evaluated. The input is the text after speech-to-text conversion, and the output habit matching degree Score_Habit∈[0,1]. The core is to identify whether the input speech conflicts with the established habit pattern.

[0056] As described above, the input speech was comprehensively and quantitatively evaluated from five key dimensions: timbre, tone, context, environment, and habits. This allowed for the comprehensive filtering of interference factors from multiple levels, significantly improving the system's ability to distinguish between command speech (i.e., real commands) and non-command speech, and providing a comprehensive and reliable basis for subsequent arbitration decisions.

[0057] In some embodiments, determining the execution strategy based on the arbitration formula and the multi-dimensional perception score includes: calculating the comprehensive confidence score of the input speech based on the multi-dimensional perception score and a preset arbitration formula; and determining the execution strategy of the input speech based on the comprehensive confidence score.

[0058] The overall confidence score refers to the overall score calculated using the arbitration formula, which reflects the probability that the input speech is a command. The execution strategy refers to the final handling plan made by the speech command arbitration model after fusion inference on the current input speech, which may include execution, ignoring, requesting further instructions from the user, etc., and is not limited here.

[0059] In the arbitration formula, each dimension of the multi-dimensional perception score is assigned a fixed weight. For example: speaker identity confidence is 0.9, with a pre-defined weight coefficient α of 0.4; tone reasonableness is 0.9, with a pre-defined weight coefficient β of 0.1; context reasonableness is 0.9, with a pre-defined weight coefficient γ of 0.3; environment matching is 0.9, with a pre-defined weight coefficient δ of 0.2; and habit matching is 0.1, with a pre-defined weight coefficient ε of 0.1. Based on this, taking the input voice "In summer, it's over 30 degrees Celsius in the user's home. The user says: It's so hot, turn on the air conditioner?" as an example, the overall confidence score of the input voice is 0.4×0.9 + 0.1×0.9 + 0.3×0.9 + 0.2×0.9 + 0.1×0.1 = 0.91.

[0060] As described above, the originally ambiguous problem of intent judgment is transformed into a calculable and comparable comprehensive confidence score, and the execution strategy is determined based on a preset threshold range. This establishes a clear quantitative decision-making standard, thereby enabling differentiated response strategies to be adopted according to different confidence levels.

[0061] In some embodiments, determining the execution strategy for the input speech based on the comprehensive confidence score includes: comparing the comprehensive confidence score with a preset first threshold and a second threshold; if the comprehensive confidence score is greater than or equal to the first threshold, then setting the execution strategy to execute an instruction; if the comprehensive confidence score is less than the first threshold but greater than or equal to the second threshold, then sending a confirmation request to the user and using the feedback result of the confirmation request as the execution strategy; if the comprehensive confidence score is less than the second threshold, then setting the execution strategy to not respond to the input speech.

[0062] The first threshold and the second threshold refer to the critical values ​​used to divide the execution strategy, with the first threshold being greater than the second threshold. It should be noted that the values ​​of the first threshold and the second threshold can be set according to the actual situation, such as the first threshold being 0.8 and the second threshold being 0.5, without any restrictions here.

[0063] The confirmation request refers to sending a prompt (such as a voice or interface prompt) to the user, asking whether to execute the corresponding voice command, such as "Do you want to turn on the air conditioner?". Based on this, a user feedback result can be obtained, which can refer to the user's response to the confirmation request, such as "OK to turn on the air conditioner" or "No need to turn on the air conditioner".

[0064] It is important to note that the aforementioned user feedback-driven closed-loop optimization mechanism is the core of this invention's zero-cost self-optimization. This mechanism treats user feedback on confirmation requests as high-quality labeled data, which is used to continuously fine-tune the large-scale model on the edge and to optimize the parameter configuration of the arbitration layer. Under this mechanism, users do not need to perform any additional operations; their daily interactions are equivalent to providing the system with free, labeled training data. Specifically, when the system's overall confidence score is in the middle range, by actively initiating a confirmation request to the user and obtaining the corresponding feedback, the user can obtain low-cost labeled data. This labeled data includes the original input speech, multimodal information, multi-dimensional perceptual scores, arbitration formulas, and execution strategies (i.e., feedback results). This labeled data requires no manual annotation and can be continuously accumulated at zero cost through natural interaction. Its quantity and scope of application far exceed traditional data collection methods.

[0065] In practical applications, as the aforementioned high-quality, gold-standard data is continuously input into the system, it can continuously fine-tune the voice command arbitration model through incremental learning techniques. During this process, the model not only learns user intent patterns but also adapts to individual user habits. For example, when the system discovers through data that a user's confirmation rate for euphemistic expressions like "It's so hot!" consistently exceeds 90% in summer, it will automatically increase the confidence score threshold for such scenarios, reducing unnecessary confirmation interruptions. As the data wheel spins, the user's confirmation rate can rapidly decrease, achieving a self-evolving effect of becoming more accurate and having a lower confirmation rate with increasing usage.

[0066] As described above, on the one hand, setting three levels of decision-making rules forms a tiered response mechanism. Specifically, at high confidence levels, the system executes automatically to ensure smoothness; at medium confidence levels, user confirmation is introduced to mitigate risks and collect feedback; and at low confidence levels, the system actively ignores the decision to avoid disruption. On the other hand, because user feedback is incorporated into the decision-making process in real time at medium confidence levels, not only is the accuracy of each interaction improved, but high-quality, labeled data is also accumulated, creating a closed-loop condition for continuous model optimization and significantly reducing the confirmation rate over long-term use.

[0067] In some embodiments, training the large language model based on the training data to obtain the voice command arbitration model includes: supervising the fine-tuning of the large language model based on the training data until the model evaluation of the large language model meets the standard; and using the large language model that meets the model evaluation standard as the voice command arbitration model.

[0068] SFT (Supervised Fine-Tuning) refers to using training data containing standard answers (i.e. labeled execution policies) to fine-tune the parameters of a large language model through backpropagation, enabling it to learn the mapping relationship from data input to execution policy.

[0069] The model evaluation refers to the systematic verification of the performance of the trained voice command arbitration model in real-world scenarios through standardized test sets and quantitative indicator systems. In actual evaluation, three core indicators need to be calculated: accuracy (the proportion of correct judgments made by the model), false trigger rate (the proportion of incorrect command executions by the model), and user confirmation rate (the proportion of cases where the model cannot determine the correct answer and requires user confirmation). A well-trained voice command arbitration model requires all three indicators to simultaneously meet preset thresholds, for example: accuracy > 95%, false trigger rate < 5%, and user confirmation rate < 5%. Furthermore, this evaluation is valid only if it is based on more than 5000 real-world scenario voice test data points (covering different temperatures, times, locations, user types, etc.).

[0070] As described above, through supervised fine-tuning and continuous iterative optimization until the model evaluation meets the standards, the large language model can effectively learn and internalize the rule-based arbitration logic in the early stage, realizing a smooth and safe transition from rule-driven to data and model-driven, and enabling the model to gradually master complex multimodal fusion reasoning capabilities.

[0071] To further achieve a personalized and intelligent user experience that becomes more accurate with use, relevant user interaction feedback data can be collected during the actual service process of the voice command arbitration model to optimize it, thereby introducing an incremental learning mechanism driven by user feedback. The specific implementation steps are as follows.

[0072] In some embodiments, the voice response method may further include: collecting user interaction feedback data, constructing incremental training data based on the user interaction feedback data, and incrementally fine-tuning the voice command arbitration model based on the incremental training data to achieve continuous optimization of the voice command arbitration model.

[0073] The user interaction feedback data refers to the natural interactive information expressed by the user through language, operation, or emotion during the voice interaction process, which reflects the user's true intention and whether they approve of the execution strategy when the system makes an incorrect response to the intention of the input voice. For example, the user's repetition, correction, or emotional expression of unresponsive instructions. It is a high-quality labeled sample (i.e. incremental training data) used to optimize the voice command arbitration model.

[0074] For example, in a real-world application scenario, suppose a user recently purchases a robot vacuum cleaner. One day, seeing the dirty floor, the user might casually complain, "There's so much dirt on the floor, it's so annoying!" At this point, if the voice command arbitration model, based on its current knowledge, fails to associate this statement with the user's intention to control the cleaning device, it may not respond. The user might then further clarify their intention: "Mop the floor!" If the voice command arbitration model still fails to understand accurately, the user might express disappointment: "Fine, you're so stupid." This complete interaction sequence, from the initial complaint to the subsequent clear instruction and the final negative feedback, will be structured and recorded, forming valuable incremental training data. This incremental training data can include key information from such scenarios, such as the original input voice ("There's so much dirt on the floor, it's so annoying!"), its corresponding multimodal information, and the user's final implicit desired action ("Start the robot vacuum cleaner"). Subsequently, this incremental training data generated in real-world scenarios, containing implicit or explicit user feedback, will be used for a new round of incremental fine-tuning of the voice command arbitration model.

[0075] As described above, based on the closed-loop mechanism of use-feedback-optimization, high-quality labeled samples can be constructed with the help of user interaction feedback data in real scenarios, and the model can be continuously trained and optimized. This enables the voice command arbitration model to achieve personalized adaptation that becomes more accurate with use, effectively improving the accuracy of system intent recognition, scenario adaptability and user interaction satisfaction.

[0076] In some embodiments, the voice response method may further include: deploying the voice command arbitration model on a local device where the user terminal is located, so that the voice command arbitration model is an edge-side model. In practical applications, the voice command arbitration model can be deployed on terminal devices (such as smart speakers, in-vehicle systems, and mobile phones).

[0077] As described above, compared to cloud deployment, this invention deploys the voice command arbitration model on the user's local device, making the model independent of cloud services. This effectively mitigates risks such as privacy leaks, network latency uncertainties, and ongoing usage costs. Specifically, firstly, since the application scenarios on the device side are relatively fixed, there is no need to rely on a model with a large number of parameters; a smaller model is sufficient to meet specific needs, effectively reducing computational resource consumption. Secondly, regarding privacy protection, all voice and contextual data are processed locally, avoiding the uploading of sensitive information to the cloud and fundamentally preventing the risk of user privacy leaks. Furthermore, in terms of cost, device-side deployment does not require additional fees for cloud API calls or computing power leasing, achieving long-term economic efficiency. Finally, in terms of response performance, compared to the uncertain latency caused by unstable network conditions in cloud services, device-side processing can provide controllable and stable low-latency responses, thus ensuring the real-time performance and smoothness of the interaction.

[0078] Figure 2 This is a flowchart illustrating an application example of a voice response method according to an embodiment of the present disclosure. (Refer to...) Figure 2 The speech response method can be divided into three stages to achieve a smooth transition and continuous evolution from rule-driven to model-driven autonomous decision-making.

[0079] The first phase is the tuning period. During this phase, the system acquires input speech and corresponding multimodal information through various information collection modules (such as voice acquisition, environmental sensors, and user habit records). This multimodal information includes current environmental status information and user habit information. The data processing module then preprocesses this data to generate a multi-dimensional perception score, including identity confidence, tone reasonableness, context reasonableness, environment matching degree, and habit matching degree. This multi-dimensional perception score is input into the arbitration module and weighted according to preset weight coefficients, calculated using the arbitration formula: Comprehensive Confidence Score = α×Score_ID + β×Score_Tone + γ×Score_Dialog + δ×Score_Env + ε×Score_Habit. Finally, the system implements a preset execution strategy based on the range of this comprehensive confidence score. For example, when the overall confidence score is greater than or equal to 0.8, the system directly executes the voice command corresponding to the input speech; when it is less than 0.5, the input speech is ignored; and when it is in the intermediate range, a confirmation request is sent to the user to obtain the corresponding feedback result. The input speech, multimodal information, multi-dimensional perceptual scores, arbitration formulas, and execution strategies as standard answers involved in this stage will be associated to form labeled data. By collecting labeled data from preset data sets, a high-quality labeled data set can be constructed for fine-tuning the large language model.

[0080] The second stage is the evolutionary phase, whose core objective is to achieve intelligent decision-making through fine-tuning the large speech model. The system uses the labeled data set generated in the first stage to supervise the fine-tuning of the large language model. During this process, the large language model learns and internalizes the rules of comprehensive arbitration performed in the first stage. The resulting speech command arbitration model possesses strong generalization capabilities, capable of directly inferring execution strategies from input data. Finally, before deploying the model, a comprehensive evaluation of key indicators is required, including accuracy, false trigger rate, decision consistency, and user confirmation rate, to ensure its performance meets application requirements.

[0081] In this second phase, the entire data, decision formulas, and processes from the first phase are used as high-quality labeled data to fine-tune the large language model, allowing it to learn the entire decision-making process and improve its generalization ability. Then, its performance is evaluated, including accuracy, false trigger rate, consistency, etc., to confirm whether it meets the requirements.

[0082] The third stage is the usage period, marking the system's entry into a fully autonomous phase. In this stage, the system architecture is simplified: the multimodal information acquired by the data acquisition module, including input voice, environmental status information, and user habit information, no longer requires cumbersome intermediate calculations and scoring, but is directly input into the voice command arbitration model. The voice command arbitration model outputs the final execution strategy (execute or ignore) end-to-end, greatly improving response efficiency. In other words, in this third stage, various types of data collected by the data acquisition module can be directly input into the voice command arbitration model, outputting results directly without complex calculations. Moreover, even if the user does not actively utter voice commands such as "turn the air conditioner on or off," the voice command arbitration model will still comprehensively judge whether to execute the corresponding command based on the input data, improving the system's intelligence.

[0083] Meanwhile, the system continuously collects user interaction data and feedback during actual operation, especially user confirmation or correction behaviors regarding execution strategies. This user interaction feedback data is used for continuous incremental learning and optimization of the voice command arbitration model, forming a data-driven model that ultimately becomes increasingly intelligent with use. Specifically, in the third stage of use, user interaction feedback data generated during usage can be collected to construct incremental training data. This incremental training data can then be used to further fine-tune the voice command arbitration model, achieving continuous optimization and gradually adapting it to specific user language habits and intent expression methods. This personalized adaptation effectively improves the accuracy of the system's intent recognition.

[0084] In summary, this invention eliminates the wake word and integrates multi-dimensional perceptual scoring based on timbre, tone, context, time, location, season, dialogue context, and user habits to comprehensively determine whether a user's voice is a valid command or environmental interference. Based on these parameters, the system utilizes deep learning technology to train a dedicated voice command arbitration model to more intelligently identify user intentions. In terms of architectural design, this invention adopts a large-model progressive takeover mechanism, divided into three stages: a training period, an evolution period, and a usage period. This achieves a smooth transition from rule-driven to model-autonomous decision-making, ensuring both the controllability and security of the system in the initial stage while gradually releasing intelligent decision-making capabilities in later stages.

[0085] Figure 3 This is a block diagram illustrating a voice response device 200 according to an embodiment of the present invention. (Refer to...) Figure 3 The voice response device 200 includes an information acquisition module 210, a model reasoning module 220, and an execution module 230.

[0086] The information acquisition module 210 is configured to acquire the current input speech and multimodal information associated with the input speech.

[0087] The model reasoning module 220 is configured to input the input speech and the multimodal information into a preset speech command arbitration model, so as to determine whether the input speech is a command speech through the speech command arbitration model.

[0088] The execution module 230 is configured to execute a voice command corresponding to the input voice if it is determined that the input voice is the instruction voice.

[0089] In some embodiments, the voice response device 200 may further include a model training module, which is configured to acquire training data and train a preset large language model based on the training data to obtain the voice command arbitration model.

[0090] In some embodiments, the voice response device 200 may further include a model incremental fine-tuning module, which is configured to collect user interaction feedback data, construct incremental training data based on the user interaction feedback data, and perform incremental fine-tuning on the voice command arbitration model based on the incremental training data, so as to achieve continuous optimization of the voice command arbitration model.

[0091] It should be understood that the information acquisition module 210, model inference module 220, execution module 230, model training module, and model incremental fine-tuning module can be further configured to execute the corresponding steps or actions in the speech response method described in the above embodiments, which will not be elaborated here.

[0092] According to another aspect of the invention, this disclosure also provides an electronic device. The electronic device includes a memory and a processor. The memory is configured to store an executable program. The processor is communicatively connected to the memory and configured to execute the program to perform the voice response method as described above.

[0093] In summary, based on the voice response method, apparatus, and electronic device provided by this invention, the present invention aims to provide a voice command execution control system that does not require a wake word, does not rely on single-point model accuracy, but achieves high accuracy, low false trigger rate, strong scene adaptability, and continuous evolution through multi-dimensional context fusion, dynamic arbitration, user feedback closed loop, and incremental learning mechanism. This invention only considers triggering execution when the comprehensive confidence level across the three dimensions of identity credibility, environment rationality, and contextual coherence meets the standard. When uncertain, it actively solicits the user, using genuine user feedback as golden annotation data to drive continuous model optimization, forming a closed loop of perception—arbitration—confirmation—feedback—learning—optimization.

[0094] Furthermore, this invention achieves significant overall performance improvements by introducing an intelligent decision-making mechanism based on multi-dimensional context awareness and dynamic arbitration. Specifically, the system can effectively distinguish between genuine instructions and casual greetings, reducing the false trigger rate by 50% to 80%. Simultaneously, it possesses a deep understanding of euphemisms, for example, reasonably associating "It's so hot!" with the voice command "Turn on the air conditioner?", and exhibits strong scene adaptability, automatically adjusting decision-making strategies according to different cities, seasons, and user habits. In terms of security, it ensures that only authorized users can trigger sensitive operations in reasonable environments and implements mandatory confirmation for high-risk operations, thereby significantly reducing the risk of misoperation, enhancing user security and final decision-making power, and increasing user trust. In addition, this invention has excellent interpretability, capable of outputting arbitration reasons to assist in understanding and debugging, and has high technical compatibility, not relying on specific automatic speech recognition, speaker verification, or large language models, facilitating integration and subsequent upgrades. Simultaneously, this invention constructs a closed-loop optimization mechanism centered on user feedback, enabling the model to continuously evolve over time and improve accuracy. Furthermore, during the model's adaptive evolution, effective user feedback itself becomes high-quality, zero-cost training samples, driving the model to gradually adapt to users' personalized language habits and intent expressions. Ultimately, this positive cycle of "more users—more feedback data—more accurate model—better user experience—attracting more users" forms a robust data flywheel barrier. In summary, based on the synergistic effects of these multiple dimensions, this invention ultimately achieves an intelligent system that does not rely on wake words or keyword matching, can naturally understand everyday human conversations, and accurately identify user intent, providing an experience akin to a personal assistant constantly aware of user needs.

[0095] Furthermore, this invention achieves intelligent wake-up functionality without a wake word, proactively understanding and judging the true intent of user statements, accurately distinguishing between valid instructions and unintentional dialogue or environmental interference. This allows for accurate execution of instructions while automatically filtering out irrelevant information, much like a 24 / 7 online personal assistant. Its core advantage lies in its continuous evolution capability: the system continuously fine-tunes the model based on user feedback, improving recognition accuracy over time; the user confirmation rate has significantly decreased from approximately 20% initially to below 5%, drastically reducing confirmation interference; and the model can gradually adapt to users' personal habits, achieving highly personalized service. The entire process requires no manual parameter tuning, with the large model automatically optimizing arbitration parameters, effectively reducing maintenance costs. In terms of security mechanisms, the system can force user confirmation for high-risk operations such as unlocking, while intelligently skipping confirmation for low-risk operations such as turning on lights, balancing convenience and security.

[0096] The above description is merely an embodiment of the present invention and does not limit the patent scope of the present invention. Any equivalent modifications made based on the content of the present invention specification and drawings, or direct or indirect applications in related technical fields, are similarly included within the patent protection scope of the present invention.

Claims

1. A voice response method, characterized in that, include: Collect the current input speech and the multimodal information associated with the input speech; The input speech and the multimodal information are input into a preset speech command arbitration model to determine whether the input speech is a command speech. as well as If the input speech is determined to be the instruction speech, then the speech instruction corresponding to the input speech is executed.

2. The voice response method according to claim 1, characterized in that, Also includes: If the input speech is determined not to be the instruction speech, then the input speech is ignored.

3. The voice response method according to claim 1, characterized in that, Determining whether the input speech is a command speech through the speech command arbitration model includes: The voice command arbitration model, which uses the identity confidence, tone reasonableness, context reasonableness, environment matching degree, and habit matching degree associated with the input voice as the judgment criteria, determines whether the input voice is a command voice.

4. The voice response method according to claim 1, characterized in that, Also includes: Acquire training data and train a large language model based on the training data to obtain the speech command arbitration model.

5. The voice response method according to claim 1, characterized in that, Also includes: Collect user interaction feedback data, and construct incremental training data based on the user interaction feedback data; as well as The voice command arbitration model is incrementally fine-tuned based on the incremental training data to achieve continuous optimization of the voice command arbitration model.

6. The voice response method according to claim 4, characterized in that, Obtaining training data includes: Collect input speech and multimodal information associated with the input speech; The input speech and the multimodal information are preprocessed respectively to obtain the corresponding multidimensional perception score; Obtain a preset arbitration formula, and determine an execution strategy based on the arbitration formula and the multi-dimensional perception score; and Annotated data is constructed based on the input speech, the multimodal information, the multidimensional perception score, the arbitration formula, and the execution strategy, and used as the training data.

7. The voice response method according to claim 6, characterized in that, The input speech and the multimodal information are preprocessed separately to obtain the corresponding multidimensional perception score, including: The input speech is subjected to timbre recognition to obtain the identity confidence level; The input speech is analyzed for tone of voice to obtain the tone appropriateness. Obtain the context information of the input speech, and perform semantic analysis on the input speech in combination with the context information to obtain the contextual reasonableness. Based on the environmental state information in the multimodal information, the environmental matching degree is obtained; Based on user habit information in the multimodal information, a habit matching degree is obtained; and The identity confidence level, tone reasonableness, context reasonableness, environment matching degree, and habit matching degree are correlated as a multi-dimensional perceptual score for the input speech.

8. The voice response method according to claim 6, characterized in that, Determining the execution strategy based on the arbitration formula and the multi-dimensional perception score includes: Based on the multi-dimensional perceptual scoring and the preset arbitration formula, the comprehensive confidence score of the input speech is calculated; and The execution strategy for the input speech is determined based on the comprehensive confidence score.

9. The voice response method according to claim 8, characterized in that, Determining the execution strategy for the input speech based on the comprehensive confidence score includes: The overall confidence score is compared with a preset first threshold and a second threshold. If the overall confidence score is greater than or equal to the first threshold, then the execution strategy is set to execute the instruction; If the overall confidence score is less than the first threshold but greater than or equal to the second threshold, a confirmation request is sent to the user, and the feedback result of the confirmation request is used as the execution strategy. If the overall confidence score is less than the second threshold, the execution strategy is set to not respond to the input speech.

10. The voice response method according to claim 4, characterized in that, Training the large language model based on the training data to obtain the voice command arbitration model includes: The large language model is then supervised and fine-tuned based on the training data until the model evaluation meets the requirements; and The large language model that meets the evaluation criteria will be used as the speech command arbitration model.

11. The voice response method according to claim 1, characterized in that, Also includes: The voice command arbitration model is deployed on the local device where the user's terminal is located.

12. A voice response device, characterized in that, include: The information acquisition module is configured to acquire the current input speech and multimodal information associated with the input speech; The model reasoning module is configured to input the input speech and the multimodal information into a preset speech command arbitration model, so as to determine whether the input speech is a command speech through the speech command arbitration model; as well as The execution module is configured to execute a voice command corresponding to the input voice if it is determined that the input voice is the instructional voice.

13. An electronic device, characterized in that, include: The memory is configured to store executable programs; as well as A processor is configured to execute the program to cause the electronic device to perform the method according to any one of claims 1 to 11.

Images