Artificial Intelligence-Based Intelligent Robot Voice Interaction Method, System, and Device

Abstract

This application provides a voice interaction method, system, and device for intelligent robots based on artificial intelligence, relating to the fields of artificial intelligence and intelligent robot technology. It solves the technical problems in existing technologies where robots struggle to understand ambiguous instructions with unclear referents or missing conditions, lack proactive clarification and causal reasoning capabilities, and are unable to update their cognitive models online. The method includes: constructing a causal graph containing environmental states, user intentions, actions, and outcome variables; mapping natural language instructions to nodes in the causal graph and tracking the posterior probability distribution; determining the executability of the instruction based on preset preconditions; otherwise, selecting a clarification question with the highest information gain and updating the posterior distribution; executing the action and updating the causal graph structure and conditional probability table online based on the result differences. This application, used in human-machine interaction processes for service robots, industrial robotic arms, and autonomous driving, can improve instruction understanding accuracy, human-machine collaboration efficiency, and system adaptability.

Description

Technical Field

[0001] This invention belongs to the field of artificial intelligence, specifically to an intelligent robot voice interaction method, system, and device based on artificial intelligence. Background Technology

[0002] Currently, human-robot voice interaction in intelligent robots primarily relies on end-to-end deep learning models or rule-based task dialogue systems. End-to-end models typically employ large-scale pre-trained language models combined with vision-language models to map user voice commands into sequences of actions that the robot can execute; rule-based systems manage the interaction process through finite state machines or dialogue trees. However, the aforementioned existing technologies have the following significant shortcomings: First, end-to-end models essentially learn the statistical correlations between multimodal data rather than the true causal structure. When user instructions contain ambiguities such as unclear referents (e.g., "it" in "bring it over"), missing conditions (e.g., the implicit triggering condition in "wait until everyone finishes eating before clearing the plates"), or spatiotemporal ambiguity (e.g., "nearby" in "put it nearby"), the model often fails to actively identify uncertainties and instead directly outputs the action with the highest probability, leading to task execution errors or failures.

[0003] Second, although traditional dialogue systems can preset clarification questions, their clarification strategies are fixed and cannot dynamically select the question with the highest information gain based on the current environmental state and the robot's uncertainty about its own cognition. Therefore, multiple rounds of ineffective communication are often required, which seriously reduces the efficiency of human-computer interaction and user experience.

[0004] Third, most existing methods use static models, meaning that once deployed, robots cannot use success or failure feedback during execution to correct their understanding. In particular, when the actual execution results deviate from their internal predictions, the system cannot deduce the error in the causal model and therefore cannot adjust its understanding of causal relationships in the world online. This leads to its rigid performance in complex and ever-changing environments, making it impossible to achieve true continuous learning and adaptation.

[0005] In summary, how to enable intelligent robots to possess proactive causal reasoning, efficient ambiguity resolution, and online self-evolution capabilities has become a key technical problem that urgently needs to be solved in this field. Summary of the Invention

[0006] This application provides a voice interaction method, system and device for intelligent robots based on artificial intelligence. In the prior art, intelligent robots have difficulty in effectively understanding natural language instructions with ambiguity such as unclear referents, missing conditions or spatiotemporal ambiguity. They lack the ability to actively clarify and reason causally, resulting in low interaction efficiency. They also have the technical problem of not being able to dynamically update their own cognitive model according to the actual execution results to adapt to complex and ever-changing environments.

[0007] To achieve the above objectives, this application adopts the following technical solution: Firstly, it provides an AI-based intelligent robot voice interaction method, including: A causal graph containing state variables and causal dependencies is constructed based on historical interaction data; the causal graph is used to represent the causal dependencies between the robot's perceived environmental state variables, user intention variables, robot action variables, and task result variables. The system acquires natural language commands input by the user, maps the entities in the natural language commands to causal variable nodes in the causal graph, and uses the causal graph to track the posterior probability distribution of the causal variable nodes based on the multimodal data of the current environment. Based on the posterior probability distribution and the preset task preconditions, it is determined whether the natural language instruction meets the instruction execution conditions; if not, the candidate clarification question with the highest information gain is selected to ask the user, and the user's clarification response is received; according to the user's clarification response, the posterior probability distribution of the causal variable node is updated using causal inference. Execute the action sequence corresponding to the natural language instruction based on the updated posterior probability distribution, and update the structure of the causal graph online according to the difference between the execution result and the prediction result.

[0008] Based on the above technical solutions, the AI-based intelligent robot voice interaction method provided in this application introduces a causal graph to explicitly model the causal dependencies between environmental state, user intent, robot actions, and task results. This enables the robot to possess interpretable causal reasoning capabilities, allowing it to proactively identify ambiguities in instructions and generate clarifying questions with the highest information gain. This significantly improves the robustness of handling ambiguous natural language instructions with unclear referents or missing conditions. By utilizing Bayesian filtering to track the posterior probability distribution of causal variable nodes and combining it with user clarification responses for causal intervention updates, the quantification and dynamic resolution of uncertainty are achieved, avoiding blind execution or repeated trial and error. Furthermore, by comparing the execution results with the prediction results and updating the causal graph structure online, the robot can continuously correct its causal cognitive model from interactive feedback, enhancing the system's adaptability to dynamic environments and personalized user habits. In summary, the method proposed in this application has significant advantages in terms of instruction understanding accuracy, human-machine collaborative interaction efficiency, and the system's self-evolution capability.

[0009] Furthermore, the construction of a causal graph containing state variables and causal dependencies based on historical interaction data specifically includes: Acquire multimodal time-series data collected when the robot performs historical tasks; the multimodal time-series data includes visual image sequences, voice command text, action execution logs, and task success feedback signals; A variational encoder is used to decouple the latent variables in the multimodal data into independent causal factors, and a causal regularization term is used to constrain each causal factor to be determined only by its corresponding causal parent node; wherein, the latent variables are abstract features extracted from the multimodal data, including but not limited to object position, user intent, and task stage; Conditional independence tests are performed on the decoupled causal factor time series, and a preliminary causal graph structure is constructed using a differentiable directed acyclic graph learning algorithm. The nodes of the preliminary causal graph structure include environmental state variables, user intention variables, robot action variables, and task result variables. Each node represents the state variable of a causal factor, and the directed edges between nodes represent direct causal dependencies. The initial causal graph structure is subjected to sparsity penalty and causal sufficiency constraint to obtain the final causal graph, and the value space and prior probability distribution of each state variable are stored; wherein, the state variables include environmental state variables and user intent variables.

[0010] Furthermore, the step of using a variational encoder to decouple the latent variables in the multimodal data into independent causal factors includes: Construct a variational autoencoder, the encoder network of which maps multimodal time series data into a low-dimensional latent vector representing latent variables, each dimension of the latent vector corresponding to a causal factor; A causal regularization term is added to the loss function of the variational autoencoder. The causal regularization term is the sum of the mutual information between each dimension of each latent vector. The mutual information between each dimension of the latent vector is minimized through adversarial training, so that the different dimensions are statistically independent. The decoder network of the variational autoencoder reconstructs the original multimodal time series data based on the independent causal factors obtained from sampling, and uses the impounding difference and KL divergence to jointly optimize the network parameters. The value of each dimension of the latent vector is used as the value of the corresponding causal factor to form a decoupled causal factor time series.

[0011] Furthermore, a sparsity penalty and a causal sufficiency constraint are imposed on the preliminary causal graph structure, including: Construct a structural scoring function, which uses the Bayesian information criterion for scoring, and the calculation formula is as follows: Wherein, G is the causal graph structure, and D is the dataset composed of the time-series sequences of the causal factors. For the maximum likelihood estimation of parameters under graph G, Let N be the number of independent parameters in graph G, and N be the data sample size. Score the structure; By adding a sparsity penalty coefficient to the structural scoring function, the overall scoring function is obtained: ;in, Let be the total number of edges in the causal graph, and λ be the preset regularization coefficient. For the overall score; In the process of searching the graph structure using the total score function, a confounding variable detector is introduced. The confounding variable detector judges whether there is an unobserved confounding variable between any two nodes in the causal graph structure based on the residual independence test. If so, it prohibits the addition of a direct edge between the two nodes, thus forming a causal sufficiency constraint. By using a differentiable directed acyclic graph (DAG) constraint optimization method, the total score function is minimized while maintaining the graph structure as a DAG, to obtain the final causal graph.

[0012] Further, mapping entities in the natural language instructions to causal variable nodes in the causal graph includes: Dependency parsing and semantic role labeling are performed on the natural language instructions to extract action verbs, entity nouns, locative words and tense modifiers from the instructions; Using a pre-trained big oracle model, the vector cosine similarity between each extracted entity noun and each state variable name in the causal graph is calculated, and the state variable nodes with similarity exceeding a preset threshold are used as the mapping targets of the entity nouns, thus obtaining the entity noun mapping method. Using the same method as the entity noun mapping method, action verbs are mapped to action variable nodes, locative words are mapped to spatial relation variable nodes, and tense modifiers are parsed into time window parameters and associated with the shortest duration requirement in the preconditions; After completing the node mapping, a mapping relationship set is established, which includes state variable nodes, action variable nodes, and spatial relationship variable nodes; Based on the directed edge information pre-stored in the causal graph, determine the action variable nodes and spatial relation variable nodes associated with each state variable node.

[0013] Furthermore, determining the action variable nodes and spatial relationship variable nodes associated with each state variable node includes: For any state variable node, if there is a directed edge in the cause-effect graph pointing from the state variable node to an action variable node, then the action variable node needs to check the state variable of the state variable node when performing condition judgment. If there exists a directed edge from a state variable node to a spatial relation variable node, then the spatial relation variable node participates in the subsequent numerical condition comparison as an attribute node of the state variable. Once an entity noun is mapped to a state variable node, all action variable nodes and spatial relationship variable nodes associated with that state variable node are automatically recorded as the associated check objects when making subsequent judgments on preconditions.

[0014] Furthermore, the preset task prerequisites include: For each action variable node, the necessary condition set in the causal graph consists of all the parent nodes of the action variable node. The necessary condition set is represented in logical conjunction normal form and is in the form of the union of condition 1, condition 2 and condition 3. Each necessary condition includes the target range of values ​​for the state variable or discrete state values, and the shortest duration for which the state variable is maintained. The preconditions also include numerical inequality constraints on continuous environmental variables; these continuous environmental variables include the distance between objects, the torque value of the robot joints, and the temperature or light intensity of the environment; the numerical inequality constraints are used to limit the set of or physical conditions that the environment must satisfy before the robot performs an action. For tasks involving multi-party collaboration, the preconditions also include intent confirmation flags from other parties; these other parties include users or another robot, and the intent confirmation is represented as a Boolean variable indicating that the other parties have agreed or completed the sub-task.

[0015] Furthermore, the posterior probability distribution is obtained as follows: The causal graph is viewed as a dynamic Bayesian network, and the value of the state variable node at each time step in the dynamic Bayesian network depends on the node value at the previous time step and the current observation data. Define the prior probability distribution ;in It is the vector of values ​​of all state variables at the initial time, and the joint prior distribution of all state variables at the initial time is obtained from the prior probability distribution of each state variable stored in the causal graph; For each time step t, the state transition probability is calculated based on the structure of the cause-effect graph. ;in, Let be the vector of values ​​for all state variables at time t. Let be the action variable executed by the robot at time t-1; the state transition probability is given by the conditional probability table of each node in the causal graph with its parent node as the condition; Acquire multimodal observation data at the current moment And calculate the observation probability. The observation probability is obtained by mapping the state variables to an observation model in the sensor space. Based on the state transition probability and the observation probability, the posterior probability distribution of the causal node at the current time is updated using the Bayesian filtering recursive formula. The formula is: ;in, Let be the posterior probability distribution of the previous time step. This indicates that in the given historical observation sequence Under the condition of current observation data Marginal likelihood of spending money.

[0016] Further, determining whether the natural language instruction meets the instruction execution conditions based on the posterior probability distribution and preset task preconditions includes: Read the set of preconditions corresponding to the action variable node of the current instruction from the cause-effect graph; the set of preconditions contains multiple subconditions, each of which is associated with a state variable. For each sub-condition, the marginal probability value of the associated state variable is extracted from the posterior probability distribution at the current time. This marginal probability value is obtained by reading the probability of the target interval specified by the sub-condition from the posterior probability distribution, and the calculation formula is as follows: ;in, Let represent the marginal probability value of the i-th sub-condition. This represents the state variable associated with the i-th sub-condition. Represents the state variables under all observed data up to time t. The posterior probability distribution; The marginal probability value is compared with a preset confidence threshold. If the marginal probability value is greater than or equal to the confidence threshold, the sub-condition is determined to be satisfied; otherwise, it is determined to be unsatisfied. When all sub-conditions are met, the output shows that the instruction is executable; when at least one sub-condition is not met, the output shows that the instruction is not executable, and the state variable associated with the first sub-condition that is not met is recorded as the target variable that needs clarification. Clarification indicates that the robot's current uncertainty about the value of the state variable exceeds the allowable range, and additional information needs to be obtained by asking the user to eliminate the uncertainty and thus meet the execution conditions.

[0017] Furthermore, the information gain The calculation formula is: Where Q represents the candidate clarification question. For the space of possible values ​​for the answer to the question, for One of the specific answers, Y represents the prior probability of the answer to the question, obtained from the prior distribution of the state variables stored in the causal graph; Y is the random variable representing the probability of task success. The entropy is the probability of task success in the current state. The success probability of the task is calculated based on the current posterior probability distribution, which is obtained through causal reasoning; To obtain the answer by assumption Then, the conditional entropy of the task success probability under the updated posterior probability distribution.

[0018] Furthermore, the candidate clarification problem is used to eliminate cases of ambiguous references, missing conditions, or spatiotemporal ambiguity in instructions, specifically including: In cases where the referent is unclear, the candidate clarification question asks the user for the specific identity of the object being referred to; the list of candidate objects is composed of all object names in the current visual detection results that match the semantic category of the pronoun in the user's command. In the case of missing conditions, the candidate clarification question is to ask whether the missing conditions have been met or whether it is necessary to wait. In cases where spatiotemporal ambiguity exists, the candidate clarification question is to inquire about the specific target location or the waiting time. All candidate clarifications are generated by filling the names and values ​​of causal variable nodes into a predefined text template. The generated original text is then rewritten into fluent sentences that conform to human conversational habits by a natural language model.

[0019] Further, updating the posterior probability distribution of the causal variable nodes using causal inference based on the user clarification response includes: The user's clarification response is parsed as an assignment to a specific causal variable; the assignment is divided into hard assignment and soft assignment. Hard assignment means that the probability distribution of the causal variable is completely focused on the value of the user's response, while soft assignment means that the probability distribution of the causal variable is adjusted according to the confidence level of the user's response. The confidence level of the user's response is determined by the confidence score of speech recognition or the clarity of the user's answer. Perform a do operation on the causal graph, wherein the do operation deletes all edges of the assigned causal variable and sets the value of the causal variable to the value indicated by the user clarification response; Run the Bayesian network inference algorithm on the causal graph after the do operation is completed, and recalculate the posterior probability distribution of all other causal variable nodes.

[0020] Furthermore, updating the structure of the causal graph online based on the difference between the execution result and the prediction result includes: After executing the action sequence, the actual task result and the intermediate state change sequence during the execution process are obtained; the actual task result includes a success flag or a failure flag, and the intermediate state change sequence is the actual observed value of each state variable at each time step. The actual task result is compared with the task result predicted based on the cause-effect graph before execution, and the intermediate state change sequence is compared with the state change trajectory predicted before execution. If the actual task result is the same as the predicted task result and all intermediate state deviations are within the preset tolerance range, it is determined that there is no need to update the cause-effect graph structure. If the actual task result differs from the predicted result, or if at least one intermediate state deviation exceeds the preset tolerance range, a discrepancy is determined, and the following sub-steps are executed: Record the observation sequence and action sequence of the prediction bias to construct a counterfactual sample; the counterfactual sample is the state sequence that appears after changing a certain action or intervening in a certain variable in the actual history; Using the online greedy search algorithm in the structural causal model, while maintaining the constraints of the directed acyclic graph, a new edge is added, an existing edge is deleted, or the direction of an edge is reversed, so that the new causal graph can explain the counterfactual sample, thus obtaining a candidate new causal graph; For each candidate new causal graph, calculate the Bayesian information criterion score. If the difference between the new score and the original score is greater than the preset update threshold, then replace the cause-effect graph structure with the new causal graph. Simultaneously, the conditional probability table of the corresponding state variable in the causal graph is updated. The update of the conditional probability table adopts maximum likelihood estimation, and the conditional probability value of each node under the combination of parent nodes is recalculated based on the latest collected interaction data.

[0021] Secondly, this application provides an intelligent robot voice interaction device, including: a communication unit and a processing unit; The communication unit is used to acquire natural language commands input by the user and multimodal data of the current environment, and to receive the user's clarification response to the clarification question; The processing unit is configured to construct a causal graph containing state variables and causal dependencies based on historical interaction data, map entities in the natural language instructions to causal variable nodes in the causal graph, and use the causal graph to track the posterior probability distribution of the causal variable nodes based on the multimodal data. Based on the posterior probability distribution and the preset task preconditions, it is determined whether the natural language instruction meets the instruction execution conditions. If not, the candidate clarification question with the highest information gain is selected and asked to the user through the communication unit. Based on the received user clarification response, the posterior probability distribution of the causal variable node is updated using causal reasoning. Execute the action sequence corresponding to the natural language instruction based on the updated posterior probability distribution, and update the structure of the causal graph online according to the difference between the execution result and the prediction result.

[0022] Thirdly, this application provides an intelligent robot voice interaction device, including: a processor and a storage medium; The storage medium includes instructions, and the processor is configured to execute the instructions to implement the AI-based intelligent robot voice interaction method as described in the first aspect or any possible implementation thereof.

[0023] Fourthly, this application provides an artificial intelligence-based intelligent robot voice interaction system, including: a cause-effect graph construction module, an instruction understanding module, an interaction clarification module, and an execution update module; wherein, The causal graph construction module is used to construct a causal graph containing state variables and causal dependencies based on historical interaction data; the causal graph is used to represent the causal dependencies between the robot's perceived environmental state variables, user intention variables, robot action variables, and task result variables; The instruction understanding module is used to acquire natural language instructions input by the user, map the entities in the natural language instructions to causal variable nodes in the causal graph, and use the causal graph to track the posterior probability distribution of the causal variable nodes based on the multimodal data of the current environment. The interactive clarification module is used to determine whether the natural language instruction meets the instruction execution conditions based on the posterior probability distribution and preset task preconditions; if not, it selects the candidate clarification question with the highest information gain to ask the user and receives the user's clarification response; and updates the posterior probability distribution of the causal variable node using causal inference based on the user's clarification response. The execution update module is used to execute the action sequence corresponding to the natural language instruction based on the updated posterior probability distribution, and update the structure of the causal graph online according to the difference between the execution result and the prediction result.

[0024] Compared with the prior art, the beneficial effects of this application are: First, by introducing a causal factor decoupling technique based on variational autoencoders, this application can automatically learn independent and interpretable causal representations from multimodal time-series data. Simultaneously, sparsity penalties and causal sufficiency constraints are incorporated into the causal graph construction process, effectively eliminating redundant causal edges and reducing the risk of model overfitting. Furthermore, confounding variable detection eliminates interference from unobserved common causes, resulting in a more concise, accurate, and interpretable causal graph. In addition, this causal graph explicitly models the deep dependencies between environmental states, user intentions, robot actions, and task outcomes, laying a solid foundation for subsequent proactive clarification, conditional judgment, and adaptive updates.

[0025] Second, this application achieves highly robust fuzzy processing capabilities in the natural language instruction understanding and execution stage. By semantically matching entity nouns, action verbs, locative words, etc., with causal graph nodes and combining this with temporal parsing, user instructions can be accurately mapped to the causal model. Furthermore, a dynamic Bayesian network is used to track the posterior probability distribution of state variables in real time, and the executability of instructions is judged based on preset logical conjunction preconditions. When the conditions are not met, the system actively calculates the clarifying question with the maximum information gain and asks the user a question through natural language generation, thereby eliminating ambiguous references, missing conditions, or spatiotemporal ambiguity with the fewest interaction rounds. This method can improve the robot's instruction understanding success rate in complex and uncertain environments, while reducing the user's communication burden and making human-machine collaboration smoother and more natural.

[0026] Third, this application possesses continuous learning and self-evolution capabilities. Upon receiving clarification from the user, the system rapidly updates the posterior probability distribution through causal intervention, enabling the robot to instantly adjust its understanding of the current state using feedback information. More importantly, when the actual execution result deviates from the causal graph prediction, the system can automatically add, delete, or reverse causal edges based on counterfactual samples and an online greedy search algorithm, and evaluate the degree of improvement in the graph structure using the Bayesian information criterion, thereby achieving dynamic correction of the causal graph and incremental updates to the conditional probability table. This mechanism allows the robot to learn from each interaction, gradually correct its causal cognitive model, adapt to dynamic environmental changes and the personalized habits of different users, thus freeing it from dependence on static preset rules and enhancing the system's versatility and long-term stability. Attached Figure Description

[0027] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0028] Figure 1 A system architecture diagram of an AI-based intelligent robot voice interaction system provided for embodiments of this application; Figure 2 A flowchart illustrating the AI-based intelligent robot voice interaction method provided in this application embodiment; Figure 3 A flowchart illustrating another AI-based intelligent robot voice interaction method provided in this application embodiment; Figure 4 A flowchart illustrating another AI-based intelligent robot voice interaction method provided in this application embodiment; Figure 5 A schematic diagram of the structure of an artificial intelligence-based intelligent robot voice interaction device provided in an embodiment of this application; Figure 6 A schematic diagram of the hardware structure of an AI-based intelligent robot voice interaction device provided in an embodiment of this application. Detailed Implementation

[0029] The AI-based intelligent robot voice interaction method provided in this application can be applied to, for example... Figure 1 In the AI-based intelligent robot voice interaction system shown, such as Figure 1 As shown, the system includes: a cause-effect graph construction module, an instruction understanding module, an interaction clarification module, and an execution update module; among which, The causal graph construction module is used to construct a causal graph containing state variables and causal dependencies based on historical interaction data. The causal graph is used to represent the causal dependencies between the robot's perceived environmental state variables, user intention variables, robot action variables, and task result variables. The instruction understanding module is used to acquire natural language instructions input by the user, map the entities in the natural language instructions to causal variable nodes in the causal graph, and use the causal graph to track the posterior probability distribution of the causal variable nodes based on the multimodal data of the current environment. The interactive clarification module is used to determine whether a natural language instruction meets the instruction execution conditions based on the posterior probability distribution and preset task preconditions. If not, it selects the candidate clarification question with the highest information gain to ask the user and receives the user's clarification response. Based on the user's clarification response, it uses causal inference to update the posterior probability distribution of the causal variable nodes. The execution update module is used to execute the action sequence corresponding to the natural language instruction based on the updated posterior probability distribution, and update the structure of the causal graph online according to the difference between the execution result and the prediction result.

[0030] To address the technical problems of existing intelligent robot voice interaction technologies, such as lack of causal reasoning ability, inability to handle ambiguous commands, and poor adaptability, embodiments of this application provide an artificial intelligence-based intelligent robot voice interaction method, which includes: A causal graph containing state variables and causal dependencies is constructed based on historical interaction data; the causal graph is used to represent the causal dependencies between the robot's perceived environmental state variables, user intention variables, robot action variables, and task result variables. The system acquires natural language commands input by the user, maps entities in the natural language commands to causal variable nodes in the causal graph, and uses the causal graph to track the posterior probability distribution of the causal variable nodes based on the multimodal data of the current environment. Based on the posterior probability distribution and the preset task preconditions, it is determined whether the natural language instruction meets the instruction execution conditions; if not, the candidate clarification question with the highest information gain is selected to ask the user and the user's clarification response is received; based on the user's clarification response, the posterior probability distribution of the causal variable node is updated using causal inference. Execute the action sequence corresponding to the natural language instruction based on the updated posterior probability distribution, and update the structure of the causal graph online according to the difference between the execution result and the prediction result.

[0031] Based on this, this application achieves deep semantic understanding and uncertain reasoning of user commands by explicitly modeling the causal dependencies between variables through causal graphs; it proactively initiates clarification interactions when command conditions are not met and selects the optimal problem based on information gain, reducing the number of interaction rounds; at the same time, through online causal graph structure updates, the robot has the ability to continuously learn and adapt to environmental changes. Compared with traditional end-to-end black box models or association rule-based methods, it has higher interpretability, sample efficiency and generalization performance.

[0032] like Figure 2 As shown in the embodiments of this application, the intelligent robot voice interaction method based on artificial intelligence includes: S1. Construct a causal graph containing state variables and causal dependencies based on historical interaction data.

[0033] In this context, causal graphs are used to represent the causal dependencies between environmental state variables perceived by the robot, user intent variables, robot action variables, and task outcome variables. Environmental state variables include, but are not limited to, physical environmental parameters such as object position, obstacle distance, light intensity, temperature, and humidity, as well as user-related states such as user position, user posture, and user facial expressions. User intent variables represent the potential goals expressed by the user through natural language commands, such as picking up objects, navigating, asking for information, and operating equipment. Robot action variables include executable actions such as moving, grasping, rotating, speaking, and adjusting joint angles. Task outcome variables represent the effects achieved after performing actions, such as task success or failure, whether the object was moved to the target position, and user satisfaction. Causal graphs use directed acyclic graph structures or directed graph structures with feedback edges to depict the direct causal relationships between the above variables, such as "user intent variables lead to robot action variable selection" or "environmental state variables and robot action variables jointly determine task outcome variables."

[0034] By constructing a causal graph, semantic understanding, decision-making reasoning, and execution feedback in robot voice interaction are unified within a causal framework. This enables the robot to distinguish between correlations or spurious associations and genuine causal drivers, allowing it to make reasonable inferences based on causal invariance even when faced with unseen scenarios or out-of-distribution commands. Compared to traditional interaction methods based on end-to-end mapping of deep neural networks, this method does not rely on large-scale labeled command-action pairs to cover all possible situations. Instead, it achieves combinatorial generalization and counterfactual reasoning through the structured knowledge of the causal graph. Furthermore, the visualization characteristics of the causal graph facilitate developers in debugging, interpreting, and correcting the robot's behavior strategies, avoiding the "black box" limitation of traditional methods that makes it difficult to diagnose the source of errors.

[0035] In some implementations, the process of constructing a cause-effect graph includes the following steps: First, time-series variable observations are extracted from historical interaction data, which includes user voice command text, environmental data recorded by robot sensors, robot action logs, and labels indicating task success or failure.

[0036] Then, causal structure learning algorithms are used, such as constraint-based conditional independence testing algorithms like the Peter-Clark algorithm, score-based greedy equivalence search algorithms, or nonlinear independent component estimation methods based on function causal models, to automatically learn the directions of directed edges between variables from the data. To improve learning accuracy, prior knowledge from domain experts can also be incorporated, adding known causal edges or prohibited edges as hard constraints to the learning process.

[0037] In addition, a hybrid approach can be used: first, time-series Granger causality tests are used to screen candidate causal pairs, and then structural equation modeling is used to verify the directionality. When the amount of data is insufficient, online incremental structure learning, such as the Markov Chain Monte Carlo sampling method based on Bayesian structure updates, can be used to gradually improve the causal graph.

[0038] The completed causal graph has one causal variable for each node and a causal strength parameter for each directed edge, such as coefficients in a linear structural equation or kernel function parameters in a nonlinear Gaussian process.

[0039] It should be noted that state variables in a causal graph can be discrete, such as "door state: open or closed," or continuous, such as "distance from robot to target: 2.5 meters." Causal dependencies can be linear or nonlinear, and unobserved confounding variables are allowed. These can be handled by introducing latent variables or using instrumental variables. Constructing a causal graph does not require all variables to be fully observed in a single interaction; some variables can participate in inference as latent variables. Furthermore, the granularity of the causal graph can be dynamically adjusted according to the application scenario: for simple dialogue tasks, state variables can be aggregated into high-level semantic concepts such as "is the room clean?"; for precision manipulation tasks, state variables need to be refined to low-level physical quantities such as joint angles and force feedback.

[0040] For example, in a home service robot scenario, historical interaction data records that after a user says, "Give me the cup on the table," the robot visually detects the cup's position, plans a path, grasps the cup, moves it to the user, and delivers it. The learned causal graph contains the following nodes: environmental state variables are "cup position," "table position," "user position," and "whether the cup is graspable"; user intention variable is "the user wants to obtain the cup"; robot action variables are "move to the table," "grab the cup," "move to the user's position," and "deliver the cup"; and task outcome variable is "the user successfully obtains the cup." Causal edges include: "the user wants to obtain the cup" causally leads to "the robot performing the action of moving to the table"; "cup position" and "grabbing the cup" jointly determine "whether the grasping is successful"; and "successful grasping" further causally affects "the user successfully obtains the cup." This causal graph clearly reveals the key causal relationship between the success or failure of an action and the environmental state.

[0041] S2. Obtain the natural language instructions input by the user, map the entities in the natural language instructions to the causal variable nodes in the causal graph, and use the causal graph to track the posterior probability distribution of the causal variable nodes based on the multimodal data of the current environment.

[0042] The causal graph itself defines the joint probability distribution decomposition form among variables, which is the product of the conditional probability distributions of each variable given its causal parent node. Multimodal data from the current environment, such as camera images, LiDAR point clouds, microphone array audio, and tactile sensor signals, provide observations of some causal variable nodes. After inputting these observations as evidential variables into the causal graph, the probability distributions of all unobserved causal variable nodes, i.e., the posterior probability distributions, can be updated using Bayesian or variational inference methods. Since the directed edges of the causal graph represent the generation mechanism, the posterior probability distribution reflects the likelihood of each causal variable taking different states under the current observed evidence. For example, observing the environmental state "there is an obstacle in front of the robot" suggests that the posterior distribution of "the probability of the robot's movement being successful" will decrease.

[0043] In some implementations, entity mapping includes the following steps: First, semantic parsing is performed on the natural language instructions: named entity recognition models, such as transformer-based bidirectional encoder representation models, are used to identify entity words such as target objects, locations, and actions in the instructions, while dependency parsing is used to determine the semantic relationships between entities.

[0044] Then, an entity-node mapping table is established. This mapping table can be pre-defined manually, mapping common words to causal variable nodes, or it can use semantic similarity matching. For example, verb phrases in the instructions are embedded into a vector space, and cosine similarity is calculated between them and the action variable names or descriptions in the causal graph. The node with the highest similarity exceeding a threshold is taken as the mapping result. For new words not appearing in the mapping table, new entities in the user's instructions can be recorded using online learning methods, prompting the user to confirm or automatically associating the entity with the closest variable type, such as "unknown objects belong to object instance nodes". After mapping, the entity values ​​extracted from the instructions are used as partial observational evidence for the corresponding causal variable nodes.

[0045] Subsequently, based on the multimodal data of the current environment, the specific implementation method of using causal graphs to track the posterior probability distribution of causal variable nodes is as follows: First, the multimodal data is converted into observed values ​​or observed likelihoods of causal variable nodes through the corresponding perceptual model. For example, camera images are input into an object detection network to obtain the confidence distribution of object category and location, and this distribution is used as the observed likelihood of the "object location" variable.

[0046] Then, a causal inference algorithm is executed: if the number of variable nodes in the causal graph is small and they are discrete, the exact variable elimination algorithm is used to calculate the posterior marginal probabilities. If the number of variables is large or there are continuous variables, an approximate inference method is used, such as likelihood weighting based on importance sampling, Markov chain Monte Carlo sampling, or automatic differential variational inference based on variational inference. For linear Gaussian structure equation models, Kalman filtering or belief propagation algorithms can be used to efficiently track the posterior distribution.

[0047] During the inference process, the entity observations obtained by mapping natural language instructions and the multimodal data observations are used together as the evidence set to calculate the posterior probability distribution of all causal variable nodes. The posterior probability distribution is stored in the form of a probability mass function, a probability density function, or a parameterized distribution such as a Gaussian distribution.

[0048] It's important to note that the mapping process doesn't require every entity in the natural language command to precisely correspond to a causal variable node. Unmapped entities are allowed; in this case, they are considered supplementary descriptions of the user's intent but do not participate in causal inference. If the command lacks a key entity, such as only saying "bring that thing over" without specifying the object's name, the "object instance" variable node in the mapping result will be a missing observation. In this case, the posterior probability distribution will exhibit high entropy, i.e., high uncertainty, thus triggering clarification questions in subsequent steps. When tracking the posterior probability distribution, a time-expanded version of the causal graph can be used. This involves copying the causal graph by time slices and adding time edges to form a dynamic causal graph. In this case, tracking the posterior probability distribution transforms into a state estimation problem, which can be solved using particle filtering or recursive Bayesian estimation.

[0049] S3. Based on the posterior probability distribution and the preset task preconditions, determine whether the natural language instruction meets the instruction execution conditions; if not, select the candidate clarification question with the highest information gain to ask the user and receive the user's clarification response; based on the user's clarification response, use causal inference to update the posterior probability distribution of the causal variable nodes.

[0050] The role of step S3 is to proactively detect and resolve situations where instructions cannot be reliably executed due to environmental uncertainties, ambiguous user commands, or missing preconditions, thereby preventing blind robot actions that could lead to task failure or safety risks. By selecting the clarification question with the highest information gain, the robot can acquire the most critical information with the fewest interaction rounds, improving user experience and task completion efficiency. Simultaneously, by updating the posterior probability distribution using the user's clarification response, the robot's belief state gradually converges to the real-world state, providing a high-confidence basis for subsequent action sequences. This step transforms passive instruction understanding into proactive interactive understanding, demonstrating the intelligent robot's leap from "hearing instructions" to "understanding intentions."

[0051] In some implementations, preconditions for the task are first preset. These preconditions are stored as logical formulas on a cause-effect graph. For example, for the "execute grasp" action, the preconditions are "the target object's position is known," "the mean posterior probability of grasp success is greater than 0.8," and "the robot gripper is fault-free." Preconditions can be defined separately for different action variables or for the target variable corresponding to the entire instruction.

[0052] Then, based on the posterior probability distribution of all current causal variable nodes, the probability of each precondition being true is calculated. If the probability of all preconditions being true is higher than the corresponding threshold, such as 0.9, then the execution condition is deemed met; otherwise, it is not.

[0053] When the execution conditions are not met, the system enters the clarification phase. First, it identifies the set of key variables that cause the conditions to be unmet—variables associated with preconditions that have high posterior probability entropy or whose posterior probabilities do not meet the threshold. For each key variable, the system pre-generates or dynamically generates candidate clarification questions. The generation methods for candidate clarification questions include: template-based generation, for example, for the variable "target object," the question template is "Which object are you referring to? Options are a list of currently detected objects"; and free-text generation, using a large language model to generate natural-sounding questions based on the variable name and the current context.

[0054] Then, the information gain of each candidate clarification question is calculated. Information gain is defined as the expected reduction in information entropy of the updated posterior probability distribution relative to the current posterior probability distribution after obtaining the answer to the question. The candidate clarification question with the highest information gain is selected and converted into natural language output to the user. The user responds with a clarification response via voice or text, and the system again uses named entity recognition and semantic parsing to extract information from the response as new observational evidence.

[0055] Next, the entity values ​​extracted from the clarifying response are used as deterministic or soft observations of the corresponding variables, and causal inference algorithms such as variable elimination or sampling methods are run again to obtain the updated posterior probability distribution.

[0056] It should be noted that information gain can be calculated based on Monte Carlo simulation: for each candidate question, the various possible answers the user might give and their probabilities are simulated, which can be obtained from the current posterior marginal distribution or from the distribution predicted by the language model. When the user clarifies that the response is inconsistent with the robot's expectations, the system should remain humble, accept the information provided by the user, and override previous beliefs.

[0057] For example, the user instruction is "Put the water glass on the table." The current posterior probability distribution shows a high entropy for the "water glass location" variable because the system visually detects two similar water glasses, one on the coffee table and one on the bookshelf, each with a probability of approximately 0.5. The precondition requires that "the target water glass location is unique and definite." Since this condition is not met, the system generates two candidate clarification questions: Question A, "Which water glass are you referring to? The one on the coffee table or the one on the bookshelf?", and Question B, "Is there water in the water glass?". Information gain calculation: Question A directly locates the water glass's location; if the entropy of the water glass location variable drops to 0 after obtaining the answer, the information gain is high. Question B is irrelevant to the water glass's location, resulting in low information gain. Therefore, Question A is chosen to ask the user. The user clarifies with the response "The one on the coffee table." The system maps this response to the "water glass location" variable as "coffee table," re-runs the causal inference, and the updated posterior probability shows the probability of the water glass on the coffee table is 0.99. At this point, the execution condition is met, and subsequent steps continue.

[0058] S4. Execute the action sequence corresponding to the natural language instruction based on the updated posterior probability distribution, and update the structure of the causal graph online according to the difference between the execution result and the prediction result.

[0059] Executing action sequences based on the updated posterior probability distribution means that the robot assigns the variable with the highest probability in its current belief state as the basis for decision-making, planning and executing a series of actions. The generation of action sequences can employ causal path planning from the current state to the target state in a causal graph. For example, using the interferometric quantifier in a structural causal model, the expected value of the task outcome variable after executing different actions can be calculated, and the action sequence that maximizes the probability of success can be selected. The difference between the execution result and the predicted result reflects the degree of mismatch between the causal graph model and the actual environment. Online updates to the causal graph structure allow the robot to continuously correct model errors, adapt to environmental changes or drift during long-term use, and achieve continuous learning.

[0060] In some implementations, executing the action sequence corresponding to the natural language instruction includes the following sub-steps: The updated posterior probability distribution is input into the decision module, which performs causal programming based on the causal graph. One implementation of causal programming is using causal Bayesian reinforcement learning, where the causal graph serves as background knowledge for state transitions, and the optimal action sequence is obtained through policy search or Monte Carlo tree search. Another implementation is to substitute the point estimates from the posterior distribution into the structural equations of the causal graph, forward simulating the computation from the current intervention action to the task outcome, and selecting the set of actions that enable the task outcome variable to reach the expected value. Each basic action in the action sequence is executed by the robot's underlying controller; for example, the movement path is executed by the navigation module, and grasping is executed by the robotic arm control module. During execution, the environmental state changes and the success or failure of the execution result are recorded in real time after each action.

[0061] Next, before executing the action sequence, a causal graph is used to predict the distribution of the task outcome variable and other intermediate variables based on the current posterior probability distribution and the planned action sequence. For example, the "success rate of grabbing" is predicted to be 0.9. After execution, the actual observed value of the task outcome variable, "grabbing failed," is obtained. The discrepancy is calculated, such as a predicted success but actual failure, or the residual between the predicted continuous value and the actual value.

[0062] Then, the causal structure is updated using the difference information. Online update methods can employ score-based local search: starting with the current causal graph, evaluate graph structures in the neighborhood graph space that better fit the differences in the new data, adding, deleting, or reversing only one directed edge at a time, and using the Bayesian information criterion or the minimum description length criterion to determine whether to accept the new structure. Alternatively, constraint-based online conditional independence testing can be used: after collecting new observation data, retest the conditional independence between variable pairs; if it conflicts with the current graph, adjust the edge direction. For nonlinear causal models, causal gradient descent can be used to parameterize the structure of the causal graph into a differentiable form, updating the adjacency matrix by minimizing the loss function of the prediction error.

[0063] It should be noted that stable causal edges that have been confirmed through multiple interactions should be protected during the update process. This can be achieved by setting a confidence counter for each edge; edges can only be deleted when the confidence level falls below a threshold. When a significant change in the environment is detected, such as seasonal changes causing alterations in object location distribution, the system can automatically reset the confidence levels of some edges to accelerate adaptation. Furthermore, online updates not only include the addition and deletion of directed edges but also the adaptive adjustment of causal strength parameters such as structural equation coefficients. This can be accomplished using recursive least squares or Bayesian parameter updates.

[0064] For example, in the action sequence of "placing the water cup on the table," the robot predicted a success rate of 0.99 for grasping the cup. However, during actual execution, the grasping failed twice due to the slippery surface of the cup. The execution result differed from the prediction. After collecting the discrepancy data, the system initiated an online update of the causal graph: the causal graph originally contained the edge "object surface friction" -> "grasping success rate," but the friction variable had never been observed. Through discrepancy analysis, the system inferred that there might be an unobserved confounding factor, "the cup is slippery." Therefore, the system added a latent variable node "surface wetness" to the current causal graph, and added the edge "surface wetness" -> "grasping success rate," as well as an edge from the environmental state variable "whether the cup has just been washed" to "surface wetness." After the update, the new causal graph can explain the reason for this execution failure and provide more accurate predictions for similar scenarios in the future.

[0065] Based on the above technical solutions, the AI-based intelligent robot voice interaction method provided in this application introduces a causal graph as the core knowledge representation and reasoning engine, realizing a paradigm shift from passive command response to active causal reasoning. This method effectively handles the uncertainty of perception and language by utilizing causal posterior probability distribution in the semantic understanding stage; employs an information gain maximization strategy for clarifying questions in the interaction stage, reducing unnecessary dialogue rounds; utilizes causal programming to generate interpretable action sequences in the decision-making and execution stage; and continuously refines the causal model through online structural updates in the adaptive learning stage, enabling the robot to continuously improve its performance. Compared to traditional methods, this application possesses stronger robustness, interpretability, and data efficiency, making it particularly suitable for intelligent robot voice interaction scenarios in complex and open environments such as home services, medical assistance, and industrial collaboration.

[0066] In one possible implementation of the embodiments of this application, combined with Figure 2 ,like Figure 3 As shown, the above S1 can be implemented through the following S101, S102 and S103, which are explained in detail below: S101. Acquire multimodal time-series data collected when the robot performs historical tasks, and use a variational encoder to decouple the latent variables in the multimodal data into independent causal factors.

[0067] The multimodal time-series data includes visual image sequences, voice command text, action execution logs, and task success feedback signals. A variational encoder decomposes the implicit, non-observable abstract features in the data, such as object location, user intent, and task stage, into statistically independent causal factors, laying the foundation for constructing an accurate causal graph.

[0068] In some implementations, a variational autoencoder is used to achieve decoupling, specifically including the following operations: Step 1: Construct the network structure of the variational autoencoder. A variational autoencoder is constructed, which contains three sub-networks: the encoder network... Decoder network and a discriminator network (For adversarial training). The input x is multimodal time-series data, and z is a low-dimensional latent vector with dimension d equal to the preset number of causal factors. The encoder maps the input to the mean and variance of the latent vector: Then, sampling is performed using reparameterization techniques to obtain... ,in The decoder reconstructs the input based on the latent vector: .

[0069] Step 2: Define the basic loss function.

[0070] The standard loss function of a variational autoencoder consists of the reconstruction loss and the KL divergence: ;in, The prior is a standard normal distribution. The reconstruction loss uses mean squared error or cross-entropy. This is the balance coefficient (usually taken as 1).

[0071] Step 3: Add a causal regularization term to achieve decoupling. To achieve statistical independence between the dimensions of the latent vector, a causal regularization term is added to the loss function. This term is the sum of the mutual information between the latent dimensions: Among them, mutual information Since direct computation is difficult, adversarial training can be used for approximate minimization. Specifically, a discriminator network D is introduced, whose input is the two dimensions of the latent vector. The output indicates whether the two dimensions come from joint samples of the same distribution (binary classification). The encoder's goal is to make the discriminator indistinguishable, i.e., to minimize the discriminator's classification accuracy. A differentiable upper bound of mutual information is: ; Therefore, an adversarial approach is used in actual training: With a fixed encoder, the discriminator is trained to maximize the above upper bound, that is, to correctly distinguish joint samples from edge product samples.

[0072] With the discriminator fixed, the encoder is updated to minimize the upper bound of mutual information, i.e., let and Close to independence.

[0073] The final loss function is: ;in, To counteract the loss (the encoder part takes a negative sign). This is a hyperparameter.

[0074] Step 4: Jointly optimize network parameters. Mini-batch stochastic gradient descent is used to simultaneously optimize the encoder, decoder, and discriminator. During training, for each batch of multimodal temporal data x, the encoder output... , And sample z to obtain z, calculate reconstruction loss and KL divergence; at the same time, for each pair of dimensions of z And independently sampled after shuffling Input the discriminator and calculate the adversarial loss. Update the encoder and decoder based on the total loss (the discriminator is updated separately). Training convergence is indicated by the discriminator accuracy approaching 0.5 (indistinguishable) and the reconstruction loss stabilizing at a small value.

[0075] Step 5: Output the decoupled causal factor time series. After training, use the encoder as a feature extractor. This applies to any input multimodal time series data. (Time step t), obtained through the encoder (or after sampling) ),Pick Each dimension is used as the value of the corresponding causal factor. Then, the causal factor vectors of all time steps are arranged in chronological order to obtain the decoupled causal factor time series sequence. Each of them The dimensions are statistically independent, thus eliminating the confounding correlations in the original multimodal data.

[0076] It should be noted that minimizing mutual information in the causal regularization term can also be achieved using other methods, such as minimizing the Hilbert-Schmidt independence criterion or using the total correlation coefficient, but adversarial training is more stable in practice. The dimension d of the latent vector can be determined through cross-validation or the elbow rule based on the reconstruction error.

[0077] For example, suppose the input data consists of 10 consecutive seconds of visual images, voice command text, and joint torque sequences from the robot. Set d=5. After training, the encoder outputs a 5-dimensional latent vector where: the first dimension primarily changes with the object's position and is independent of other dimensions; the second dimension changes with the verbs "take" or "put" in the user's command; the third dimension changes with the task stage (start, in progress, end); the fourth dimension changes with ambient light intensity; and the fifth dimension is random noise. Thus, the first dimension represents the causal factor of "object position," the second dimension represents the factor of "user's action intent," and the third dimension represents the factor of "task stage," achieving effective decoupling.

[0078] S102. Perform conditional independence tests on the decoupled causal factor time series and construct a preliminary causal graph structure using a differentiable directed acyclic graph learning algorithm.

[0079] The nodes in the preliminary causal graph structure include environmental state variables, user intent variables, robot action variables, and task result variables. Each node represents a causal factor, and the directed edges between nodes represent direct causal dependencies.

[0080] In some implementations, the specific operations are as follows: First, the decoupled causal factor time series obtained from S101 is... As input, each of Let be a d-dimensional vector, with each dimension corresponding to a causal factor. Define the node set. Each node This represents a causal factor. Based on the physical meaning of the causal factor, nodes are classified into environmental state variables, user intention variables, robot action variables, and task outcome variables, but the classification does not affect the structure learning algorithm itself.

[0081] Then, for each pair of nodes Calculate given all other nodes Conditional independence under given conditions; where the backslash "\" represents the difference operation. Specifically, the kernel conditional independence test method can be used: define a kernel function k to map the variables to the reproducing kernel Hilbert space, and calculate the estimator of the partial cross-covariance operator norm, the formula is: Where n is the sample size. Given other variables and Empirical estimates of the partial cross covariance operator, Let represent the Frobenius norm. Under the null hypothesis of "conditional independence", It asymptotically follows a chi-square distribution with 1 degree of freedom; Given significance level (Usually taken as 0.01 or 0.05), if The p-value is greater than Then the conditional independence is accepted, that is... and If there is no edge between them, then the condition of independence is rejected, and an edge is added to connect them in the undirected skeleton graph. and After performing the above checks on all node pairs, an undirected skeleton graph is obtained. .

[0082] Next, each undirected edge in the undirected skeleton graph is treated as a potential directed edge to be oriented, and the NOTEARS (Nonlinear Optimization for DAG) algorithm is used for orientation. This algorithm encodes the graph structure as an adjacency matrix. ,in This indicates whether there exists a directed edge from node i to node j. Then, through continuous optimization, the sum of the data fitting loss and the sparse regularization term is minimized, while simultaneously satisfying the directed acyclic graph constraint h(W)=0, thus obtaining the optimal adjacency matrix. .

[0083] Solving for the optimal adjacency matrix yields the solution. Then, for each non-zero element ,like > ( If a preset threshold (e.g., 0.01) is set, a directed edge is added from node i to node j. Simultaneously, the graph is checked for directed cycles; if one exists, the cycle is broken by flipping the edge with the minimum weight. The resulting directed acyclic graph is the preliminary causal graph structure. The nodes in this graph correspond one-to-one with the input causal factors, and the directed edges represent direct causal dependencies.

[0084] It should be noted that the reliability of conditional independence tests depends on the assumption of independence between causal factors, while the decoupling step of S101 has eliminated confounding between factors to the greatest extent, thus making the test results more credible.

[0085] S103. Apply sparsity penalty and causal sufficiency constraint to the initial causal graph structure to obtain the final causal graph, and store the value space of each state variable and the prior probability distribution.

[0086] The state variables include environmental state variables and user intent variables. Sparsity penalties are used to eliminate redundant causal edges, and causal sufficiency constraints are used to exclude the influence of unobserved confounding variables, resulting in a simpler and more realistic causal graph.

[0087] In some implementations, the specific operations are as follows: Step S103-1: Define the structure scoring function and add a sparsity penalty.

[0088] Let the initial cause-effect graph be... Its node set For each of the d causal factors, the edge set is E. The Bayesian information criterion is used as the basic score to measure the goodness of fit of the graph structure to the data D (i.e., the time series of causal factors): ;in, This is the maximum likelihood estimate of the parameters (i.e., the conditional probability table for each node) under graph G. Let G be the number of independent parameters in graph G (equal to the number of parent node combinations of all nodes multiplied by the degrees of freedom of each conditional probability), and N be the sample size.

[0089] To eliminate redundant causal edges, a sparsity penalty term is added to the BIC score, resulting in the overall score function: Where |E| is the total number of edges in the causal graph. This is a preset regularization coefficient (the value range is usually 0.01 to 0.1, and can be adjusted according to the complexity of the task).

[0090] Step S103-2: Apply causal sufficiency constraints.

[0091] The causal sufficiency assumption requires that for any two nodes in the graph, all common causes should be explicitly modeled, i.e., there should be no unobserved confounding variables. To test and constrain whether the initial causal graph satisfies this assumption, a confounding variable detector is introduced. Specifically, for each potential edge in the graph (including existing edges and edges to be added), the correlation between the residuals of two nodes under the current graph structure is calculated.

[0092] Set nodes and In the current graph structure G, respectively, and Perform linear regression (or logistic regression if the variables are discrete) on all its parent nodes to obtain the residuals. and If G already contains and All the common causes, then the residual and The nodes should be statistically independent. Then, use a kernel independence test or a partial correlation coefficient test to calculate the p-value. If the p-value is less than the significance level (e.g., 0.05), it indicates the presence of an unobserved confounding variable, and in this case, adding a direct edge between the two nodes is prohibited.

[0093] During the optimization process, this constraint is achieved by modifying the generation method of the candidate graph structure: only when the residual independence test passes ( Only under the premise of ) is it allowed to and Add a directed edge between them; otherwise, force no edge.

[0094] Step S1=3-3: Minimize the total score using the constraints of a differentiable directed acyclic graph.

[0095] Encode the causal graph structure G into an adjacency matrix. ,in This indicates whether an edge exists that points from node i to node j. Total score function. It can be viewed as a function of W. The optimization objective is: And the constraint of causal sufficiency.

[0096] Where h(W)=0 ensures the graph structure is a directed acyclic graph, and ⊙ represents element-wise multiplication. Since The problem includes the log-likelihood term related to the graph structure in the BIC score. This optimization problem can be transformed into an unconstrained problem for iterative solution using the augmented Lagrangian method.

[0097] Step S103-4: Obtain the final cause-effect graph and store the information of each state variable.

[0098] After iterative convergence, the optimal adjacency matrix is ​​obtained. Set a threshold =0.01, will Delete the edges that are not explicitly defined, and keep the rest, thus obtaining the final directed acyclic graph. The nodes in this graph are state variables (including environmental state variables and user intent variables, etc.), and the directed edges represent the final causal dependencies.

[0099] Then, for each state variable Based on historical data, the value space and prior probability distribution of D are statistically analyzed: like For discrete variables, enumerate all possible values ​​and calculate the frequency of each value as the prior probability. ; like For continuous variables, fit a Gaussian mixture model or kernel density estimation, and store the parameters of its probability density function (such as mean and variance), or discretize the continuous variable into equal frequency intervals and store the prior probability of each interval.

[0100] This information is stored in a causal graph data structure for subsequent tracking of the posterior probability distribution.

[0101] It should be noted that the choice of regularization coefficient λ affects the sparsity of the graph. λ can be selected through cross-validation to maximize the prediction likelihood on the validation set, or it can be set based on the expected number of edges.

[0102] Based on the above technical solution, through steps S101 to S103, this application can automatically learn a clean, interpretable, and causally sufficient causal graph from the original multimodal historical data, providing a solid and reliable knowledge foundation for subsequent instruction understanding, proactive clarification, and online updates. This causal graph not only explicitly reveals the deep driving relationships between the environment, user, action, and result, but also avoids overfitting and confusion bias through sparsity and sufficiency constraints, improving the robot's robustness in handling fuzzy instructions and its long-term adaptive capabilities.

[0103] In one possible implementation of the embodiments of this application, combined with Figure 2 ,like Figure 4 As shown, the above S2 can be implemented through the following S201, S202 and S203, which are explained in detail below: S201. Obtain the natural language instructions input by the user, perform dependency parsing and semantic role labeling on the natural language instructions, and extract action verbs, entity nouns, locative words and tense modifiers from the instructions.

[0104] Among them, dependency parsing is used to identify the modification relationship between words, and semantic role labeling is used to determine the agent, patient, instrument, and other roles of an action.

[0105] In some implementations, readily available natural language processing (NLP) toolkits, such as the Stanford Core NLP toolkit or the Harbin Institute of Technology Language Technology Platform, can be used to perform word segmentation, part-of-speech tagging, dependency parsing, and semantic role labeling on user commands. Specific operations include: first, segmenting the command text into sentences and words; then, using a pre-trained syntactic analysis model to generate a dependency tree, from which the core predicate (i.e., the action verb) and its subordinate entity nouns (such as the head of a noun phrase), locative words (such as "left side," "above," "inside"), and tense modifiers (such as "already," "currently," "will"). For complex commands containing multiple nested clauses, the above components of each clause can be extracted recursively. The extraction results are stored in key-value pairs, for example, the action verb is "take," the entity noun is "cup," the locative word is "over there," and the tense modifier is "now."

[0106] It should be noted that if the user command contains pronouns (such as "it" or "that"), the entity noun may not be able to directly obtain the specific object name. In this case, the pronoun itself is temporarily stored as an entity noun and resolved in subsequent mapping steps through semantic matching and visual detection. The parsing results of tense modifiers will be used to determine the time window in the preconditions. For example, "already" indicates that the action needs to be executed after the state is completed, and "is in progress" indicates that intervention is needed during the action.

[0107] S202. Using a pre-trained large language model, the extracted action verbs, entity nouns, locative words, and tense modifiers are mapped to the corresponding nodes in the causal graph, and a set of mapping relationships is established to determine the action variable nodes and spatial relationship variable nodes associated with each state variable node.

[0108] The purpose of mapping is to align semantic units in natural language with predefined variable nodes in the causal graph, so that subsequent conditional judgments and inferences can be performed based on the graph structure.

[0109] In some implementations, the following operations are specifically included: First, prepare the pre-trained large language model. Choose a pre-trained model suitable for semantic similarity calculation, such as Sentence-BERT or RoBERTa, and input all node names (including state variable node names, action variable node names, and spatial relation variable node names) from the causal graph into the model to obtain the semantic vector representation of each node name. Let the total number of nodes in the causal graph be M, and the name of the j-th node be... Its semantic vector is: ; where d is the dimension of the model output vector.

[0110] Then, each semantic unit (action verb, entity noun, locative word, tense modifier) ​​extracted from the natural language instruction is mapped separately: (1) Mapping of entity nouns: For each extracted entity noun w, its semantic vector is calculated using the same pre-trained large language model. Then calculate. semantic vectors of all state variable nodes Cosine similarity between ; Set similarity threshold (For example, 0.7). All Node j is used as a candidate mapping target. If there are multiple candidates, the node with the highest similarity is selected as the final mapping result; for pronouns (such as "it" or "that"), they are not directly mapped, but their referential requirements are recorded.

[0111] (2) Mapping of action verbs: Using the same method as with entity nouns, the extracted action verbs are compared with the action variable node names in the causal graph using cosine similarity calculation. The action variable node with the highest similarity exceeding the threshold is selected as the mapping result. If the verb is a compound word (such as "pick up"), it can be processed by stemming or synonym normalization before matching.

[0112] (3) Mapping of locative words: Match locative words (such as "left side", "above", "inside") with the names of spatial relationship variable nodes in the causal graph. Since the number of locative words is limited, exact matching, thesaurus mapping, or semantic similarity calculation can be used. The matching results are directly associated with the corresponding spatial relationship variable nodes.

[0113] (4) Mapping of tense modifiers: Tense modifiers (such as "already", "currently", and "will") are not directly mapped to nodes, but are parsed as time window parameters. For example, "already" indicates that the action needs to be executed after the state is completed, corresponding to the minimum duration requirement in the precondition being set to "completed"; "currently" indicates that intervention is needed while the action is in progress; and "will" indicates a future point in time. The parsing results are stored in key-value pairs.

[0114] Next, establish a set of mapping relationships. Summarize all the successful mapping results into a single set of mapping relationships. Each element is a quadruple: The confidence level is the corresponding cosine similarity value. Simultaneously, for unmapped entity nouns, their original text and extracted features are recorded for subsequent generation of clarification questions.

[0115] Then, determine the action variable nodes and spatial relationship variable nodes associated with each state variable node. The specific approach is as follows: Traverse all the directed edge information pre-stored in the causal graph. For any state variable node : If there is a directed edge in the causal graph from to a certain action variable node , then record as the associated action node of . This means that before executing the action corresponding to , it is necessary to check whether the value of satisfies the preconditions.

[0116] If there is a directed edge in the causal graph from to a certain spatial relationship variable node , then record as the spatial attribute node of . When performing numerical condition comparison on subsequently, it is necessary to refer to the value of simultaneously (for example, comparing whether the actual coordinates of an object are within the "left area").

[0117] Finally, integrate the mapping results and the association relationships. For the current natural language instruction, find its corresponding state variable node, action variable node, and spatial relationship variable node according to the mapping relationship set

[0118] . If the action verb mapping is successful, use this action node as the target action of the current instruction. If the entity noun mapping is successful, use this state variable node as the affected object of the instruction. At the same time, automatically obtain the action node (for precondition checking) and spatial relationship node (for numerical comparison) associated with this state variable node by using the association relationship. For example, if the entity noun "cup" is mapped to the state variable node "cup position", and there are edges "cup position → grab" and "cup position → left area" in the causal graph, then the system automatically records that "grab" is the precondition action to be checked, and "left area" is the spatial constraint of "cup position". This information will be passed to the subsequent condition judgment module.

[0119]

[0120] It should be noted that the use of pre-trained large language models can improve the robustness of matching. For example, synonyms such as "take", "fetch", and "grab" can be correctly matched to the same action variable node "grab". If there is no corresponding node in the causal graph, the model extension mechanism can be triggered to add new nodes in the subsequent online update step. In addition, when mapping orientation words, it is necessary to convert them in combination with the current orientation of the robot and the coordinate system. For example, "left" needs to be converted to a relative angle in the robot coordinate system.

[0121] S203. Treat the causal graph as a dynamic Bayesian network and track the posterior probability distribution of causal variable nodes based on the multimodal data of the current environment and the established mapping relationship.

[0120] The posterior probability distribution represents the probability of each causal variable taking on its various possible values, given current and historical multimodal observation data. By tracking these distributions, the robot can quantify its uncertainty regarding environmental conditions and user intentions, providing a basis for proactive clarification.

[0121] In some implementations, the specific operations are as follows: First, the causal graph constructed in S1 is transformed into a dynamic Bayesian network. The dynamic Bayesian network is an extension of the Bayesian network in the time domain, which contains two time slices: the values ​​of the state variable nodes at each time step t depend on the values ​​of the nodes at the previous time step and the action variables performed by the robot at the previous time step.

[0122] Specifically, the causal graph structure G within a time slice is defined to be the same as the final causal graph obtained from S1, and a transition graph from S1 is added between adjacent time slices. arrive Directed edges, with the direction being a variable self-loop with the same node name (i.e., (), to represent the duration of a state. Simultaneously, action variables... As an external intervention node, it points to the node affected by it. The variables in the causal graph (the influence relationship is determined by the edges from the action variables to the state variables in the causal graph). Therefore, the dynamic Bayesian network defines the joint probability distribution: ;in, Let be the vector of values ​​for all state variables at time t. Let be the action variable performed by the robot at time t-1.

[0123] Then, set the initial prior probability distribution. . This is the vector of values ​​for all state variables at the initial time. The joint prior distribution at the initial time is obtained from the prior probability distribution of each state variable stored in the causal graph, assuming that the variables are independent at the initial time: ;in The marginal prior distribution of the i-th state variable is read from the prior probability distribution stored in step S1 (the probability mass function for discrete type and the probability density function for continuous type).

[0124] Next, for each time step t (from 1 to the current time), a Bayesian filtering recursion is performed to calculate the posterior probability distribution at the current time. ,in This involves all multimodal observation data from the initial time to the current time. The recursion is divided into two steps: 1. Prediction Steps: Using the posterior probability distribution and state transition probability from the previous time step, calculate the prior prediction distribution of the current state: ; Among them, the state transition probability This is given by the conditional probability table in the cause-effect graph. Specifically, for each state variable... Its value depends only on its parent node (in a dynamic Bayesian network, the parent node may include...). and (some variables in the text), therefore: ;in yes In the dynamic Bayesian network, all parent node variables.

[0125] 2. Update steps: Obtain the multimodal observation data at the current moment. (e.g., visual images, speech signals, tactile feedback), and use observation models to calculate observation probabilities. The observation model maps state variables to sensor space, typically assuming that each sensor is independent; therefore: Where K represents the number of sensors. For example, for visual observation, a Gaussian observation model can be established: ,in Let be the mapping function from state to image features. Then, use Bayes' theorem to calculate the posterior probability distribution at the current time step: ; Normalized denominator for: .

[0126] For discrete state variables, the above summation is calculated directly; for continuous state variables, particle filtering or extended Kalman filtering is typically used for approximate inference. This application employs the particle filtering method: using... A weighted particle To approximate the posterior distribution, the recursive formula correspondingly becomes the resampling and weighting of particles.

[0127] Finally, the marginal posterior probability distribution of each causal variable node is extracted from the obtained posterior probability distribution. For discrete variables, the marginal distribution is the probability of each value; for continuous variables, the marginal distribution can be the mean and variance of a Gaussian distribution, or the probability density function obtained through kernel density estimation. Specifically, for the state variable nodes involved in establishing the mapping relationship in S202 (such as "cup position"), their marginal posterior probabilities need to be output for subsequent precondition judgments and clarification of question selection. If multiple state variables are mapped in the user instruction, the marginal distributions of these variables are tracked simultaneously.

[0128] It should be noted that the state transition probability Action variables in This can be either an actual action performed by the robot (known) or a planned action (used during planning). When tracking current beliefs, Retrieved from historical execution logs. When multimodal data contains missing or noisy data, the conditional independence of causal graphs can help robots reason using only available observed variables, without requiring complete data. Furthermore, if a state variable has no direct observation, its priors and state transitions can still provide estimates; uncertainty accumulates over time, triggering clarification.

[0129] Based on the above technical solution, through steps S201 to S203, this application can accurately and robustly map the user's natural language commands to causal graph nodes, and utilize a dynamic Bayesian network to track the posterior probability distribution of each state variable in real time. This process not only considers the flexibility of semantic matching but also integrates multimodal observation data, enabling the robot to quantify its own cognitive state in uncertain environments, thereby improving its ability to understand fuzzy commands and its interaction efficiency.

[0130] In one possible implementation of this application embodiment, the above-mentioned S3 can be specifically implemented by the following S301, S302 and S303, which are described in detail below: S301. Based on the posterior probability distribution obtained in S2 and the preset task preconditions, determine whether the natural language instruction meets the instruction execution conditions.

[0131] This step aims to determine whether the robot can safely and effectively execute user commands given its current beliefs about the environmental state and the user's intentions. If all necessary conditions are met, the robot can directly execute the action sequence; if any conditions are not met, the specific missing condition variables need to be located to provide a target for subsequent proactive clarification.

[0132] In some implementations, the specific operations are as follows: First, the set of preconditions corresponding to the action variable node mapped to the current instruction is read from the causal graph. This set of preconditions is the set of necessary conditions consisting of all parent nodes of each action variable node stored when S1 constructs the causal graph, represented in logical conjunction normal form, such as "condition 1, condition 2, and condition 3". Each necessary condition contains three elements: the associated state variable, the target value range or discrete state value of the state variable, and the minimum duration for which the state must be maintained. In addition, the preconditions also include numerical inequality constraints on continuous environmental variables (such as distances between objects being less than 0.05 meters) and intent confirmation flags for multi-agent collaboration.

[0133] Then, for each sub-condition, the marginal probability value of the associated state variable is extracted from the posterior probability distribution obtained from S2 at the current time. The specific calculation method for the marginal probability value is as follows: if the state variable is discrete, the probability that the state variable takes the value specified by the sub-condition is directly read out; if it is continuous, the posterior probability density function is integrated over the target interval, as shown in the formula: ;in, Let represent the marginal probability value of the i-th sub-condition. This represents the state variable associated with the i-th sub-condition. Represents the state variables under all observed data up to time t. The posterior probability density function (continuous type) or probability mass function (discrete type).

[0134] Next, the calculated marginal probability value The confidence threshold preset for this sub-condition (The value typically ranges from 0.7 to 0.95 and can be dynamically adjusted according to mission safety requirements.) Compare. If If the condition is met, then the sub-condition is deemed satisfied; otherwise, it is deemed not satisfied.

[0135] When all sub-conditions are met, output the judgment result "Instruction Executable" and proceed to execution step S4. When at least one sub-condition is not met, output the judgment result "Instruction Not Executable" and record the state variable associated with the first sub-condition that is not met as the target variable that needs clarification. Here, "needs clarification" means that the robot's current uncertainty about the value of this state variable exceeds the allowable range, that is, its posterior probability distribution is not concentrated in the value range required by the sub-condition. Additional information must be obtained by asking the user to eliminate this uncertainty and thus meet the execution conditions.

[0136] It should be noted that the confidence threshold can be dynamically set according to the risk level of the task: for high-risk tasks (such as moving fragile items or interacting with the human body), the threshold should be increased to above 0.95; for low-risk routine tasks, the threshold can be appropriately reduced to 0.7 to improve the smoothness of the interaction.

[0137] S302. When S301 determines that the instruction does not meet the execution conditions, select the candidate clarification question with the highest information gain to ask the user, and receive the user's clarification response.

[0138] This step aims to enable the robot to proactively identify the information that best eliminates current uncertainty, obtaining the key information needed to execute instructions with the fewest possible interaction rounds, thereby improving the efficiency of human-robot collaboration. Information gain measures the reduction in uncertainty regarding the probability of task success after obtaining a given answer.

[0139] In some implementations, the specific operations are as follows: First, for the target variable requiring clarification recorded in S301, one or more candidate clarification questions are generated. Each candidate clarification question corresponds to a different aspect of the variable. For example, for the discrete variable "cup position," one could ask, "Are you pointing to the cup on the left or right?"; for the continuous variable "water temperature," one could ask, "What is the current water temperature?". Then, for each candidate clarification question Q, its expected information gain EIG(Q) is calculated using the following formula: ;in, Let x be the space of possible values ​​for the answer to the question. A specific answer is given by P(x), which is the prior probability of the answer obtained from the prior distribution of the state variables stored in the causal graph. Y is a random variable representing the probability of task success, with a value between 0 and 1. The entropy is the probability of task success in the current state, calculated using the following formula: ,in The success probability of a task is obtained by inferring the probability of the task by inputting the current posterior probability distribution as evidence into the causal graph. The conditional entropy of the success probability of the new task is calculated by using the answer x to intervene in the causal graph and update the posterior probability distribution after obtaining the answer x.

[0140] In the specific calculation, for each possible answer x, a causal intervention operation similar to S303 is first performed to obtain the updated posterior distribution, and then the task success probability under this distribution is calculated. Then substitute the values ​​into the entropy formula. Calculate the weighted average of all answers to obtain EIG(Q).

[0141] The candidate clarification question that maximizes EIG(Q) is selected as the actual question asked to the user. For continuous variables, the continuous range can be discretized into equal-frequency intervals or an adaptive interval based on maximizing information gain. When asking a question, the text generation module converts the question into a natural language sentence, which is then output to the user via speech synthesis or screen display. Subsequently, the user's clarification response is received via speech recognition or a graphical interface, and semantic parsing is performed on the response to extract the assignment information to the target variable.

[0142] In some implementations, candidate clarification questions include the following specific content or structural forms: Each candidate clarification question is represented as a structured object, noting that it contains three fields: Target variable : That is, the state variable nodes that need to be clarified as determined in S301.

[0143] Question Template : A string with placeholders, where each placeholder corresponds to a variable name, a description of its value space, or a constraint.

[0144] Answer Space The set of all possible answers to the question, which can be a discrete list of options or a continuous range of values.

[0145] Based on the type of the target variable, the following three basic templates are preset: (1) Discrete referential resolution problems (used for problems with unclear referents).

[0146] When the target variable is a discrete category such as object identity or location, the question template is: "Which of the following do you mean by '[pronoun]': {option list}?"; where [pronoun] is a pronoun extracted from the user command (such as "it", "that") or an implicit reference (such as "that thing"); {option list} consists of all object names in the current visual detection that match the semantic category of the pronoun, and each option corresponds to a discrete value of the target variable; answer space. =List of options + optional "None" or "Other".

[0147] (2) Problems with missing conditions (used for problems with missing conditions).

[0148] When the target variable is whether a certain state condition is met, the question template is: "Should I wait / check [condition description] before executing?" or "The current value of [state variable name] is [the value with the highest probability at present], does this meet your requirements?"; the condition description reads the semantic label of the variable from the cause-effect graph (e.g., "water temperature reaches 100 degrees Celsius"); answer space. ={Yes,No} or {Compliant,Disagree}, and for continuous variables, it may also be a "specific numerical value".

[0149] (3) Spatiotemporal fuzzy problems (used for spatiotemporal fuzziness).

[0150] When the target variable is spatial location coordinates or time duration, the problem template is as follows: Space: "Please specify whether the target location of the [action] is [candidate location A] or [candidate location B]?" or "Please specify where the [object] should be placed?"; Time: "Please specify the waiting time in seconds?" or "How many seconds do you mean by 'a while'?"

[0151] Candidate locations are generated using prominent landmarks in the current environment (such as "left side of the desktop" or "inside the dishwasher"); the answer space... It can be a discrete option or a continuous interval (such as [0,60] seconds).

[0152] Then, the structured question template is transformed into fluent natural language through the following two sub-steps: Template filling: The name of the target variable, the current candidate option list, action verbs, etc., are filled into the template to generate the basic text. For example, the template "What [pronoun] are you referring to {option list}?" becomes "What 'it' are you referring to, the red cup or the blue cup?".

[0153] Language model rewriting: Input the base text into a lightweight pre-trained language model (such as T5-small), and require the model to output a variant that better conforms to human conversational habits, such as "Do you mean the red cup or the blue cup?". During the rewriting process, the semantics remain unchanged; only the tone, polite words are adjusted, and word order is optimized.

[0154] It should be noted that a nonverbal instruction detection step can be added before generating and selecting candidate clarification questions. This involves using a multimodal interaction module to detect the user's nonverbal gestures, head posture, and gaze direction in real time. Combined with object detection results in the current environment, it determines whether the user has explicitly referred to the target variable nonverbally. If a valid nonverbal instruction is detected and the confidence level exceeds a preset threshold, the instruction is directly parsed as an assignment to the target variable, causal intervention is performed to update the posterior probability distribution, and the subsequent candidate clarification question generation and questioning process is skipped, directly proceeding to the action execution step. If no valid instruction is detected or the confidence level is insufficient, clarification questions are generated and asked according to the principle of maximizing information gain. This nonverbal instruction detection step can significantly reduce unnecessary interaction rounds and improve the naturalness and efficiency of human-computer collaboration.

[0155] S303. Based on the user's clarification response, update the posterior probability distribution of the causal variable nodes using causal inference.

[0156] This step utilizes new information provided by the user as evidence for intervention, forcibly altering the values ​​of relevant variables in the causal graph. This change is then propagated to all other causally related variables through causal inference, resulting in a more accurate state of belief and providing a reliable basis for subsequent implementation. Unlike simple conditional probability updates, causal intervention severs the dependency between the intervened variable and the cause, simulating the effect of humans actively changing their environment or intentions.

[0157] In some implementations, the specific operations are as follows: Let the clarification target variable determined in S302 be... Its value space is The clarifying response R returned by the user via voice or text is first processed by the semantic parsing module. The parsing results are divided into two types: Hard assignment: The user's response is explicit and affirmative, such as "left side," "red cup," or "yes." In this case, a definite value is generated. And set the confidence level to c=1.0.

[0158] Soft assignment: When the user response is uncertain, such as "it seems to be to the left," "about 90 degrees," or the speech recognition confidence is low (e.g., 0.7), a value is generated. and a confidence level .

[0159] The parsed assignment information is represented as The type is either "hard" or "soft".

[0160] In order to incorporate the user's clarified information into the system as a proactive intervention, a do operation needs to be performed, which means cutting off all directed edges between the variable being intervened and its cause.

[0161] For hard assignments, remove all pointers from the cause-effect graph. The incoming edge, and will The posterior probability distribution is forcibly set to the Dirac distribution: , ; For soft assignment: also delete all pointers. The incoming edges are defined, but their posterior probability distribution is set as follows: , That is, the probability mass of 1-c is retained and distributed to other possible values ​​according to the prior proportion. This preserves uncertainty and avoids completely trusting answers with low confidence.

[0162] After completing the do operation, a modified causal graph G' is obtained (its structure is the same as the original graph, but...). (No longer has a parent node). The marginal distribution after intervention. As evidence, all other variables were calculated using a Bayesian network inference algorithm. The posterior probability distribution. The inference formula is: ;in, For the set of all nodes, the product traversal divides All nodes outside the scope, the conditional probability of each node. The conditional probability table is stored in the causal graph G (note: in G', ...). There is no parent node, but its conditional probability has been replaced by the intervention distribution. The denominator is a normalization constant.

[0163] For continuous variables, particle filtering or analytical formulas based on linear Gaussian models can be used for inference. If the graph is large, only a subset of variables relevant to the execution of subsequent tasks (e.g., variables associated with other preconditions not satisfied in S301) can be computed to improve efficiency.

[0164] Next, each causal variable obtained through reasoning will be... posterior probability distribution The belief state is stored as it is at the current moment. The distributions of variables associated with other unmet conditions in S301 may change due to intervention; for example, knowing the cup's location increases the probability of a successful grab. These updated distributions will be used to reassess the instruction execution conditions.

[0165] If the assignment obtained from parsing Compared with current multimodal observation If there is a clear contradiction (e.g., the user says "the cup is on the left," but the visual system confirms there is no cup on the left), then the assignment should not be adopted immediately. In this case, the system can: Mark the conflict as "suspicious", temporarily use soft assignment and set the confidence level to a low value (such as 0.3), and continue reasoning; Initiating a second confirmation: "Are you referring to the cup on the left? I don't see the cup." If there are multiple conflicts, the conflict records will be used as counterfactual samples in the subsequent online update step S4 to correct the causal graph structure (e.g., by adding unobserved confounding variables).

[0166] It should be noted that the confidence level 'c' in the soft assignment can be dynamically adjusted based on the confidence score of the speech recognition system, the degree of hesitation in the user's response (e.g., speech pauses, ambiguous words), and the user's accuracy in historical interactions.

[0167] Based on the above technical solution, through steps S301 to S303, this application enables the robot to automatically determine the executability of instructions, proactively select the clarifying question with the highest information gain, and accurately update its belief state through causal intervention. This process not only minimizes the user's interaction burden but also ensures that the robot always acquires necessary information with the optimal strategy in uncertain environments, thereby improving the naturalness, efficiency, and success rate of human-machine collaboration.

[0168] In one possible implementation of this application embodiment, the above-mentioned S4 can be specifically implemented by the following S401, S402 and S403, which are described in detail below: S401. Based on the updated posterior probability distribution of S303, execute the action sequence corresponding to the natural language instruction, and obtain the actual task result and the intermediate state change sequence during the execution process.

[0169] This step translates the clarified and belief-updated decision into physical actions, while recording real feedback during execution to provide a data foundation for subsequent online updates of the causal graph. The intermediate state change sequence includes the actual observations of each state variable at each time step, such as the trajectory of the object's position over time, force sensor readings, and indicators of task success or failure.

[0170] In some implementations, the specific operations are as follows: First, based on the updated posterior probability distribution of S303, the current optimal action plan is extracted from the causal graph. The action plan can be generated by calling a planner, such as a causal graph-based plan recognition or a model-based reinforcement learning strategy, which decomposes high-level instructions into a sequence of low-level motor control instructions.

[0171] Then, these motion commands are executed through the robot's underlying controller (such as position control, force control, or impedance control), while multimodal data during the execution process is recorded at a fixed sampling frequency, including joint angles, end effector pose, object detection results, contact force, task progress, etc.

[0172] After execution, the actual task result is obtained, which is a Boolean value or scalar score indicating whether the task succeeded or failed, and the specific reason for failure (such as slippage during capture, collision, timeout, etc.). Simultaneously, the intermediate state change sequence is compiled as a list of time-stamped state variable observations. This data, along with the expected state trajectory predicted based on the causal graph before execution, is stored in a temporary buffer.

[0173] S402. Compare the actual execution results with the predicted results to determine whether there is a significant difference. If there is a difference, construct a counterfactual sample.

[0174] This step serves to check whether the current causal graph accurately predicts the execution process. If the prediction does not match reality, it indicates that the causal graph contains errors (such as missing causal edges, incorrect directions, or conditional probability deviations) and needs correction. Counterfactual samples are constructed hypothetical scenarios of "what would have happened if different actions or conditions had been taken," used to guide adjustments to the causal graph structure.

[0175] In some implementations, the specific operations are as follows: After executing the action sequence, the system records two types of data: Actual results: including the final task results (Success is 1, failure is 0) and intermediate state change sequence ,in Let be the vector of observed values ​​for all state variables at time step t.

[0176] Prediction results: Before execution, the current causal graph G and the posterior probability distribution after intervention are used. The predicted task results are obtained through forward simulation. and the predicted intermediate state sequence The prediction method is as follows: starting from the initial distribution, each step is based on the state transition probability. Sample or obtain the desired value.

[0177] Then, define the deviation vector. (For continuous variables) or indicator functions (For discrete variables). For each time step t and each state variable Calculate the absolute value of the deviation If there exists a variable i and a time step t such that... ( (for preset tolerance thresholds), or If the difference is significant, then a significant difference is determined. Otherwise, there is no need to update the causal graph.

[0178] After determining that a significant difference exists, it is necessary to identify the variables and time points most likely to cause the bias. Counterfactual anomaly localization methods can be used: calculate the anomaly location for each variable. The cumulative deviation or root mean square error over the entire time series is used; then the variable with the largest cumulative deviation and the moment when the deviation first exceeds the threshold are selected as the focus.

[0179] Counterfactual samples describe how changing the value of a certain action or variable will produce different results, while keeping other conditions constant. A procedural counterfactual reasoning algorithm is constructed based on the structural causal model: Based on the bias analysis, suppose that if at time 10:00... For variables Different values ​​were adopted or the action If different choices are made, the subsequent state sequence will not deviate. Typically, The value is taken as the actual expected value but not achieved. For example, if the actual sliding amount is too large, the sliding amount in the counterfactual case is 0; if oil stains are not actually detected, the oil stains are detected and wiped in the counterfactual case.

[0180] Then, the existing structure in the current causal graph G is used as the model. Counterfactual reasoning consists of three steps: 1. **Abduction:** Updating the posterior distribution of unobserved variables using actual observed data. That is, given an actual observed sequence... and action sequence By estimating the posterior distribution of all exogenous variables through Bayesian inference, a complete probabilistic description of the world under factual conditions is obtained.

[0181] 2. Intervention (Action): This involves applying the target variable... At any moment The value is forcibly set to And delete all its incoming edges (do operation).

[0182] 3. Prediction: On the modified graphical model, using the posterior distribution of the exogenous variables obtained through backtracking, re-simulate from time point [time]. The subsequent state trajectory yields a counterfactual state sequence. .

[0183] Finally, the constructed counterfactual sample is represented as a quadruple: Simultaneously record the original values ​​in the actual trajectory that caused the deviation. The sample will be fed into S403 for online updating of the causal graph structure.

[0184] It should be noted that if the key variable causing the bias cannot be uniquely identified (e.g., multiple variables exhibit bias simultaneously), multiple candidate counterfactual samples can be generated, each corresponding to a different intervention variable and target value. In subsequent step S403, each candidate sample will be tested to determine whether the modified graph structure can explain the counterfactual, and the modification that can explain the most samples will be selected.

[0185] S403. Update the structure and conditional probability table of the causal graph online based on counterfactual samples.

[0186] This step utilizes counterfactual samples to dynamically correct the causal graph, gradually bringing the robot's causal cognitive model closer to the real world. By adding, deleting, or reversing causal edges and updating the conditional probability table, the robot can learn from mistakes and adapt to environmental changes and user habits.

[0187] In some implementations, the specific operations are as follows: First, for each counterfactual sample constructed by S402 The system generates three types of candidate modification operations: Adding an edge: Adding an edge from a variable in a cause-effect graph. point to The directed edge. The selection criteria are based on the type of variable involved in the counterfactual intervention: if This is achieved by changing action variable A, so the candidate edge to add is... If this is achieved by changing the state variable u, then the candidate edge to add is... .

[0188] Delete edge: Delete an existing edge pointing to another edge in the graph. Edges, for example, existing in the current graph However, counterfactual evidence suggests that this edge may be redundant.

[0189] Reverse edge: Reverse existing edge Reverse to Alternatively, reverse other related edges.

[0190] All candidate modifications must ensure that the modified graph remains a directed acyclic graph (DAG), which can be verified by checking whether the trace of the power of the adjacency matrix is ​​zero.

[0191] Then, let the cause-effect diagram be... Candidate image is Using the Bayesian Information Criterion (BIC) as the scoring function, the two methods were compared for all data. The degree of fit (including historical data and newly collected execution data). BIC is defined as: ;in, For the maximum likelihood estimation of parameters under graph G, The number of independent parameters. This represents the total sample size.

[0192] Calculate the score difference: ; like ( If a preset update threshold (e.g., 2.0) is set, the modification is accepted; otherwise, it is rejected. If multiple candidates meet the requirements, the modification is selected. The biggest candidate.

[0193] Once the graph structure is determined and updated, each node needs to be re-estimated. Regarding its parent node The conditional probability distribution is determined using maximum likelihood estimation based on all available data. Recalculate: For discrete variables: ;in, When the parent node of the j-th node is selected from the k-th combination, the sample count of the m-th value of that node is taken. The total number of samples for the k-th combination for the parent node. To smooth out small samples, Laplace smoothing can be added: ;in Let j be the number of possible values ​​for node j. This is the smoothing parameter (usually set to 1).

[0194] For continuous variables: Assume a Gaussian distribution. Then update the regression parameters. Refitting can be achieved using least squares or Bayesian linear regression.

[0195] The updated conditional probability table covers all nodes and is stored in a graph structure for subsequent posterior probability tracking.

[0196] If a candidate is accepted Then the current cause-effect graph will be replaced with The old table was then replaced with the newly estimated conditional probability table. Subsequently, counterfactual reasoning was performed again on the new graph (using the same counterfactual intervention) to verify the predicted state sequence. Is it consistent with counterfactual objectives? If consistency is maintained, continue to the next round of modifications until the maximum number of iterations is reached or the deviation is acceptable.

[0197] Finally, the finalized graph structure and its conditional probability table are serialized, stored, and synchronized to the robot's long-term memory module.

[0198] Based on the above technical solution, through steps S401 to S403, this application enables the robot to automatically evaluate the predictive accuracy of the causal graph after actual execution and to correct the causal structure and parameters online using counterfactual samples. This mechanism endows the robot with the ability to continuously learn and self-evolve, enabling it to adapt to dynamically changing environments, personalized user habits, and unforeseen task scenarios, thereby improving the system's robustness, versatility, and long-term autonomy, and making up for the shortcomings of traditional static models that cannot learn online.

[0199] This application embodiment can divide the artificial intelligence-based intelligent robot voice interaction device into functional units according to the above method example. For example, each function can be divided into a separate functional unit, or two or more functions can be integrated into one processing unit. The integrated unit can be implemented in hardware or as a software functional unit. It should be noted that the unit division in this application embodiment is illustrative and only represents one logical functional division; other division methods may be used in actual implementation.

[0200] When using integrated units Figure 5 A possible structure of an AI-based intelligent robot voice interaction device 50 is shown, which includes a processing unit 501, a communication unit 502, and an optional storage unit 503. The processing unit 501 is used to control the operation of the management device, the communication unit 502 is used to communicate with other devices, and the storage unit 503 is used to store program code and data.

[0201] Specifically, the communication unit 502 is used to acquire the user's input natural language instructions, multimodal data of the current environment, and the user's response to the clarification question. The processing unit 501 is used to construct a causal graph based on historical interaction data, map entities in the instructions to nodes in the causal graph, track the posterior probability distribution of causal variables using multimodal data, determine whether the instruction is executable based on the posterior probability and preconditions, and if not, select the clarification question with the highest information gain and ask it through the communication unit, update the posterior distribution based on the user's clarification response, execute the action sequence, and update the causal graph online based on the difference between the execution result and the prediction result. The processing unit 501 is also used to treat the causal graph as a dynamic Bayesian network and perform Bayesian filtering recursion, and construct counterfactual samples and update the graph structure and conditional probability table when there is a difference in prediction. The communication unit 502 is also used to output the clarification question and parse the user response into hard or soft assignments to the causal variables; the processing unit 501 performs do operations accordingly, deletes the incoming edges of the assigned variables and sets their values, and recalculates the posterior probability distribution of other nodes.

[0202] The processing unit 501 can be a processor or a controller, the communication unit 502 can be a communication interface, transceiver, input / output interface, or pins, and the storage unit 503 can be a memory. When the device is a chip, the communication unit 502 corresponds to an input / output interface or pins, and the storage unit 503 can be on-chip or off-chip memory. The communication unit can also be called a transceiver unit, where the part used for receiving can be regarded as a receiver or receiving circuit, and the part used for transmitting can be regarded as a transmitter or transmitting circuit.

[0203] like Figure 5 The units in this application are implemented as software functional modules and sold or used independently, and can be stored in a computer-readable storage medium. The technical solutions of this application can be embodied in the form of software products, which are stored in media such as USB flash drives, mobile hard drives, ROM, RAM, magnetic disks or optical disks, and contain several instructions to cause computer devices or processors to execute all or part of the steps of the embodiments of this application. Figure 5 The units in the process can also be called modules, such as processing modules.

[0204] This application embodiment also provides the hardware structure of an artificial intelligence-based intelligent robot voice interaction device 60, such as... Figure 6 As shown, the device includes a processor 601, and optionally a memory 602 connected to the processor 601. In one implementation, the device 60 also includes a transceiver 603, and the three are connected via a bus. The transceiver 603 is used to communicate with other devices or communication networks, and its receiver is used to perform a receiving step, while the transmitter is used to perform a transmitting step.

[0205] Although this application has been described herein in conjunction with various embodiments, those skilled in the art, by reviewing the accompanying drawings, disclosure, and appended claims, will understand and implement other variations of the disclosed embodiments in carrying out the claimed application. In the claims, the word "comprising" does not exclude other components or steps, and "a" or "an" does not exclude multiple instances. A single processor or other unit can implement several functions listed in the claims. While different dependent claims may recite certain measures, this does not mean that these measures cannot be combined to produce good results.

[0206] Although this application has been described in conjunction with specific features and embodiments, it is obvious that various modifications and combinations can be made thereto without departing from the spirit and scope of this application. Accordingly, this specification and drawings are merely exemplary illustrations of this application as defined by the appended claims, and are considered to cover any and all modifications, variations, combinations, or equivalents within the scope of this application. Clearly, those skilled in the art can make various alterations and modifications to this application without departing from the spirit and scope of this application. Thus, if such modifications and modifications of this application fall within the scope of the claims of this application and their equivalents, this application also intends to include such modifications and modifications.

Claims

1. An AI-based intelligent robot voice interaction method, characterized in that, include: A causal graph containing state variables and causal dependencies is constructed based on historical interaction data; the causal graph is used to represent the causal dependencies between the robot's perceived environmental state variables, user intention variables, robot action variables, and task result variables. The system acquires natural language commands input by the user, maps the entities in the natural language commands to causal variable nodes in the causal graph, and uses the causal graph to track the posterior probability distribution of the causal variable nodes based on the multimodal data of the current environment. Based on the posterior probability distribution and the preset task preconditions, it is determined whether the natural language instruction meets the instruction execution conditions; If the conditions are not met, the candidate clarification question with the highest information gain is selected to ask the user, and the user's clarification response is received; based on the user's clarification response, the posterior probability distribution of the causal variable node is updated using causal inference; Execute the action sequence corresponding to the natural language instruction based on the updated posterior probability distribution, and update the structure of the causal graph online according to the difference between the execution result and the prediction result.

2. The AI-based intelligent robot voice interaction method according to claim 1, characterized in that, The construction of a causal graph containing state variables and causal dependencies based on historical interaction data specifically includes: Acquire multimodal time-series data collected when the robot performs historical tasks; the multimodal time-series data includes visual image sequences, voice command text, action execution logs, and task success feedback signals; A variational encoder is used to decouple the latent variables in the multimodal data into independent causal factors, and a causal regularization term is used to constrain each causal factor to be determined only by its corresponding causal parent node; wherein, the latent variables are abstract features extracted from the multimodal data, including object position, user intent, and task stage; Conditional independence tests are performed on the decoupled causal factor time series, and a preliminary causal graph structure is constructed using a differentiable directed acyclic graph learning algorithm. The nodes of the preliminary causal graph structure include environmental state variables, user intention variables, robot action variables, and task result variables. Each node represents the state variable of a causal factor, and the directed edges between nodes represent direct causal dependencies. The initial causal graph structure is subjected to sparsity penalty and causal sufficiency constraint to obtain the final causal graph, and the value space and prior probability distribution of each state variable are stored; wherein, the state variables include environmental state variables and user intent variables.

3. The AI-based intelligent robot voice interaction method according to claim 1, characterized in that, The step of mapping entities in the natural language instructions to causal variable nodes in the causal graph includes: Dependency parsing and semantic role labeling are performed on the natural language instructions to extract action verbs, entity nouns, locative words and tense modifiers from the instructions; Using a pre-trained big oracle model, the vector cosine similarity between each extracted entity noun and each state variable name in the causal graph is calculated, and the state variable nodes with similarity exceeding a preset threshold are used as the mapping targets of the entity nouns, thus obtaining the entity noun mapping method. Using the same method as the entity noun mapping method, action verbs are mapped to action variable nodes, locative words are mapped to spatial relation variable nodes, and tense modifiers are parsed into time window parameters and associated with the shortest duration requirement in the preconditions; After completing the node mapping, a mapping relationship set is established, which includes state variable nodes, action variable nodes, and spatial relationship variable nodes; Based on the directed edge information pre-stored in the causal graph, determine the action variable nodes and spatial relation variable nodes associated with each state variable node.

4. The AI-based intelligent robot voice interaction method according to claim 1, characterized in that, The preset task prerequisites include: For each action variable node, the necessary condition set in the causal graph consists of all parent nodes of the action variable node, and the necessary condition set is represented in logical conjunction normal form. Each necessary condition includes the target range of values ​​for the state variable or discrete state values, and the shortest duration for which the state variable is maintained. The preconditions also include numerical inequality constraints on continuous environmental variables; these continuous environmental variables include the distance between objects, the torque value of the robot joints, and the temperature or light intensity of the environment; the numerical inequality constraints are used to limit the set of or physical conditions that the environment must satisfy before the robot performs an action. For tasks involving multi-party collaboration, the preconditions also include intent confirmation flags from other parties; these other parties include users or another robot, and the intent confirmation is represented as a Boolean variable indicating that the other parties have agreed or completed the sub-task.

5. The AI-based intelligent robot voice interaction method according to claim 1, characterized in that, The step of determining whether the natural language instruction meets the instruction execution conditions based on the posterior probability distribution and preset task preconditions includes: Read the set of preconditions corresponding to the action variable node of the current instruction from the cause-effect graph; the set of preconditions contains multiple subconditions, each of which is associated with a state variable. For each sub-condition, the marginal probability value of the associated state variable is extracted from the posterior probability distribution at the current time. This marginal probability value is obtained by reading the probability of the target interval specified by the sub-condition from the posterior probability distribution, and the calculation formula is as follows: ;in, Let represent the marginal probability value of the i-th sub-condition. This represents the state variable associated with the i-th sub-condition. Represents the state variables under all observed data up to time t. The posterior probability distribution; The marginal probability value is compared with a preset confidence threshold. If the marginal probability value is greater than or equal to the confidence threshold, the sub-condition is determined to be satisfied; otherwise, it is determined to be unsatisfied. When all sub-conditions are met, the output shows that the instruction is executable; when at least one sub-condition is not met, the output shows that the instruction is not executable, and the state variable associated with the first sub-condition that is not met is recorded as the target variable that needs clarification. Clarification indicates that the robot's current uncertainty about the value of the state variable exceeds the allowable range, and additional information needs to be obtained by asking the user to eliminate the uncertainty and thus meet the execution conditions.

6. The AI-based intelligent robot voice interaction method according to claim 1, characterized in that, The information gain The calculation formula is: Where Q represents the candidate clarification question. For the space of possible values ​​for the answer to the question, for One of the specific answers, Y represents the prior probability of the answer to the question, obtained from the prior distribution of the state variables stored in the causal graph; Y is the random variable representing the probability of task success. The entropy is the probability of task success in the current state. The success probability of the task is calculated based on the current posterior probability distribution, which is obtained through causal reasoning; To obtain the answer by assumption Then, the conditional entropy of the task success probability under the updated posterior probability distribution.

7. The AI-based intelligent robot voice interaction method according to claim 1, characterized in that, The step of updating the posterior probability distribution of the causal variable nodes using causal inference based on the user's clarification response includes: The user's clarification response is parsed as an assignment to a specific causal variable; the assignment is divided into hard assignment and soft assignment. Hard assignment means that the probability distribution of the causal variable is completely focused on the value of the user's response, while soft assignment means that the probability distribution of the causal variable is adjusted according to the confidence level of the user's response. The confidence level of the user's response is determined by the confidence score of speech recognition or the clarity of the user's answer. Perform a do operation on the causal graph, wherein the do operation deletes all edges of the assigned causal variable and sets the value of the causal variable to the value indicated by the user clarification response; Run the Bayesian network inference algorithm on the causal graph after the do operation is completed, and recalculate the posterior probability distribution of all other causal variable nodes.

8. The AI-based intelligent robot voice interaction method according to claim 5, characterized in that, The posterior probability distribution is obtained as follows: The causal graph is viewed as a dynamic Bayesian network, and the value of the state variable node at each time step in the dynamic Bayesian network depends on the node value at the previous time step and the current observation data. Define the prior probability distribution ;in It is the vector of values ​​of all state variables at the initial time, and the joint prior distribution of all state variables at the initial time is obtained from the prior probability distribution of each state variable stored in the causal graph; For each time step t, the state transition probability is calculated based on the structure of the cause-effect graph. ;in, Let be the vector of values ​​for all state variables at time t. Let be the action variable executed by the robot at time t-1; the state transition probability is given by the conditional probability table of each node in the causal graph with its parent node as the condition; Acquire multimodal observation data at the current moment And calculate the observation probability. The observation probability is obtained by mapping the state variables to an observation model in the sensor space. Based on the state transition probability and the observation probability, the posterior probability distribution of the causal node at the current time is updated using the Bayesian filtering recursive formula. The formula is: ;in, Let be the posterior probability distribution of the previous time step. This indicates that in the given historical observation sequence Under the condition of current observation data Marginal likelihood of spending money.

9. An intelligent robot voice interaction system based on artificial intelligence, characterized in that, include: The module comprises a cause-effect graph construction module, an instruction understanding module, an interaction clarification module, and an execution update module; among which, The causal graph construction module is used to construct a causal graph containing state variables and causal dependencies based on historical interaction data; the causal graph is used to represent the causal dependencies between the robot's perceived environmental state variables, user intention variables, robot action variables, and task result variables; The instruction understanding module is used to acquire natural language instructions input by the user, map the entities in the natural language instructions to causal variable nodes in the causal graph, and use the causal graph to track the posterior probability distribution of the causal variable nodes based on the multimodal data of the current environment. The interactive clarification module is used to determine whether the natural language instruction meets the instruction execution conditions based on the posterior probability distribution and preset task preconditions; if not, it selects the candidate clarification question with the highest information gain to ask the user and receives the user's clarification response; and updates the posterior probability distribution of the causal variable node using causal inference based on the user's clarification response. The execution update module is used to execute the action sequence corresponding to the natural language instruction based on the updated posterior probability distribution, and update the structure of the causal graph online according to the difference between the execution result and the prediction result.

10. An intelligent robot voice interaction device based on artificial intelligence, characterized in that, include: Communication unit and processing unit; The communication unit is used to acquire natural language commands input by the user and multimodal data of the current environment, and to receive the user's clarification response to the clarification question; The processing unit is configured to construct a causal graph containing state variables and causal dependencies based on historical interaction data, map entities in the natural language instructions to causal variable nodes in the causal graph, and use the causal graph to track the posterior probability distribution of the causal variable nodes based on the multimodal data. Based on the posterior probability distribution and the preset task preconditions, it is determined whether the natural language instruction meets the instruction execution conditions. If not, the candidate clarification question with the highest information gain is selected and asked to the user through the communication unit. Based on the received user clarification response, the posterior probability distribution of the causal variable node is updated using causal reasoning. Execute the action sequence corresponding to the natural language instruction based on the updated posterior probability distribution, and update the structure of the causal graph online according to the difference between the execution result and the prediction result.

Images