Robot trial and error learning and skill generation method based on causal reasoning

Abstract

The invention belongs to the field of intelligent robots with bodies, and relates to a robot trial and error learning and skill generation method based on causal reasoning. The method comprises the following steps: performing robot exploration in a safe simulation or limited real environment, and collecting state and action data; learning a causal relationship between the environment state and the robot action from the interaction data by using a causal discovery algorithm, and constructing a causal graph; hypothesis analysis is carried out when a task fails through an anti-factual reasoning method, and results possibly brought by different actions are reasoned; and based on an anti-factual reasoning result, dynamically correcting a current strategy through a strategy correction and skill generation method, and generating a reusable new skill. According to the method, causal reasoning and anti-fact analysis are introduced, so that the autonomous learning efficiency, the skill interpretability and the adaptive capacity in a new task and a new environment of the robot are remarkably improved.

Description

Technical Field

[0001] This invention belongs to the field of embodied intelligent robots and relates to a method for robot trial-and-error learning and skill generation based on causal reasoning. Background Technology

[0002] With the rapid development of artificial intelligence and robotics, reinforcement learning has become an important paradigm for robots to achieve autonomous learning through trial and error and interaction with the environment. Traditional reinforcement learning methods (such as deep Q-networks and policy gradient algorithms) mainly rely on the statistical correlation between data to optimize policies. However, this type of correlation-based learning has inherent limitations: sample efficiency is generally low, often requiring a large amount of expensive and time-consuming interaction data; the generalization ability of the learned policies is limited, and slight changes in environmental or task conditions can lead to a significant drop in performance; at the same time, its decision-making process lacks interpretability and is difficult to provide clear behavioral justifications, thus limiting its application in safety-sensitive scenarios.

[0003] Causal reasoning, as a theoretical framework capable of revealing the intrinsic causal mechanisms between variables, has been gradually introduced into the field of machine learning in recent years. This method enables agents to perform counterfactual reasoning, that is, to infer "the possible consequences of performing different actions," thereby potentially improving learning efficiency, generalization ability, and interpretability. However, current research on integrating causal reasoning techniques into the robot's trial-and-error learning loop is still in its early stages, particularly in key areas such as automatically constructing causal graphs from interaction data, optimizing strategies in real time using counterfactual analysis, and generating transferable skills. Complete, efficient, and scalable technical solutions are still lacking in these crucial aspects. Summary of the Invention

[0004] In view of this, the purpose of this invention is to provide a robot trial-and-error learning and skill generation method based on causal reasoning, which can effectively improve the robot's adaptability and generalization ability in unknown environments.

[0005] To achieve the above objectives, the present invention provides the following technical solution: A method for robot trial-and-error learning and skill generation based on causal reasoning, comprising: By conducting random or guided exploration with robots, the interaction data sequence between environmental state variables and robot action variables is collected; Based on the collected interaction data, the causal relationship between state variables and action variables is learned through the causal discovery algorithm, and a causal graph is constructed. In this causal graph, nodes represent state variables and action variables, and edges represent causal relationships. When the robot fails to execute a task, the constructed cause-effect graph is used as the basis for calculation, and the impact of different action sequences on the task result is calculated through hypothesis analysis. Based on the results of counterfactual reasoning, the robot's current control strategy is dynamically modified, and new action sequences are generated to form a reusable skill set.

[0006] Furthermore, the collected raw interaction data is preprocessed, and the preprocessed data is input into the causal discovery algorithm to learn the causal dependency structure between variables and generate an initial causal graph. Then, the initial causal graph is instantiated into a structural causal model, and a structural equation is established for each variable in the graph to clarify its functional relationship with the parent node and exogenous noise variables. Finally, the learned structural causal model is verified and refined to form a complete causal representation.

[0007] Specifically, using preprocessed interaction data To perform causal structure learning, inferring causal dependencies between variables and constructing an initial causal graph. A score-based causal discovery method is employed, which searches for the optimal directed acyclic graph structure by optimizing the scoring function. The scoring function is: , Representing graph structure Data Log-likelihood, Representing the complexity of a graph, The regularization coefficient is used; through optimization techniques such as greedy search, a coefficient is found that makes the score... Largest graph structure ; In a defined graph structure Based on this, parameter learning is performed to estimate the structural equation for each variable in the causal graph. Its structural equation is expressed as:

[0008] In the formula, In a cause-effect graph Medium variables The set of all parent nodes, This is an exogenous noise term independent of the parent node; Using interactive data The corresponding data is used to learn the function using a nonlinear regression method. Estimate by minimizing the objective function parameters The objective function is:

[0009] In the formula, Representing variables At any moment The observed values, Representing variables The vector formed by the values ​​of the parent nodes at the corresponding times. Represents the regularization term for the parameter. To determine the regularization strength; by learning parameters for each causal dependency in the causal graph, a structural causal model is obtained; The learned structural causal model is validated and refined. First, statistical robustness is verified using K-fold cross-validation, which randomly divides the dataset into K subsets. K-1 subsets are used sequentially for causal discovery and parameter learning, and the prediction accuracy of the structural causal model is evaluated on the remaining subset. Then, domain knowledge consistency is verified by comparing the structural causal model with prior knowledge in the relevant domain. Causal edges that clearly violate domain consensus are corrected or removed based on the results of conditional independence tests.

[0010] Furthermore, when the robot fails to execute a task, the complete sequence of events actually observed by the robot during the task execution is used as inference evidence. The inference evidence includes the sequence of environmental states, the sequence of execution actions, and the final task result. Based on the inference evidence, the Bayesian inference method is used to calculate the posterior probability distribution of the exogenous variables. Identify the key decision nodes in the causal graph that have the greatest causal impact on the task outcome, and perform intervention operations on the corresponding action variables, including sampling from the predefined action space to generate hypothetical action values, severing the causal connection between the target variable and the parent node, and modifying the corresponding structural equations; Based on the model after intervention, forward extrapolation is performed using the updated distribution of exogenous variables to calculate the values ​​of endogenous variables, thereby obtaining the counterfactual results corresponding to the assumed actions. The evolution trajectory of the state sequence is calculated until the task terminates, thus obtaining the counterfactual results. The expected value is calculated through Monte Carlo sampling, generating a quantified counterfactual advantage function as the gradient signal for policy optimization.

[0011] Specifically, inference evidence based on observation The posterior probability distribution of exogenous variables is calculated using a Bayesian update mechanism. :

[0012] In the formula, exogenous variables U The prior distribution, Let be the likelihood function; the posterior probability distribution is approximated using variational inference methods. By optimizing the lower bound of evidence to approximate the true posterior:

[0013] In the formula, For variational posterior distribution, This represents the KL divergence.

[0014] Intervention operations on the causal model: The action variables at a key state node in history... Force set to new value By modifying the structural equation of the structural causal model, the action variables to be intervened are... The corresponding structural equation is replaced with a constant function. At the same time, remove intervention variables Functional dependencies between the intervention point and its original parent node; the selection of the intervention point is based on the assessment of the causal effect strength, and the following criteria are used to select the optimal intervention target:

[0015] In the formula, Key indicators of the success or failure of a task. x This indicates the specific constant value that is set; Based on the post-intervention model According to the posterior probability distribution of exogenous variables Counterfactual results are calculated through forward structural equation modeling, and probabilistic deduction is performed using the Monte Carlo method, based on the posterior probability distribution. Extraction Sample For each sample, according to the post-intervention model The structural equation model sequentially calculates the values ​​of each endogenous variable, ultimately yielding counterfactual results. The expected value of the counterfactual outcome is calculated using the sample mean:

[0016] counterfactual results This is quantified into a cumulative reward function, serving as the initial quantification output for counterfactual reasoning:

[0017] in, This is a discount factor used to weigh the importance of current and future rewards; The immediate reward function is used to evaluate the state. Next action The immediate benefits gained; For the task time range, and They are respectively t The environmental state variables at any given time and the action variables performed by the robot.

[0018] Furthermore, based on the counterfactual reasoning results, the robot's current control strategy is dynamically revised to generate a better action sequence confirmed by counterfactual reasoning; Extract key causal relationships from successfully executed action sequences and corresponding environmental states, and construct a skill package represented in the form of a skill causal subgraph; The generated skill packs are stored in a hierarchical skill library, supporting fast retrieval and cross-task migration based on causal structure similarity.

[0019] Specifically, based on the counterfactual reasoning results, they are initially quantified as a cumulative reward function. After calculating the counterfactual action value function and the counterfactual state value function based on the cumulative reward function, a counterfactual dominance function is constructed by subtracting the counterfactual action value function and the counterfactual state value function. The counterfactual dominance function is then integrated into the policy optimization process using a policy gradient update method. The policy parameters... Updates are performed based on the following gradients:

[0020] In the formula, Describe the objective function Relative to policy parameters The gradient is used to guide the optimization direction of the strategy; Indicates regarding following the current strategy generated trajectory The mathematical expectation operator; This represents the total time step of the trajectory; Representation Strategy In state Select action below The log probability relative to the parameter The gradient; For state Next action The actual advantage function; It is the counterfactual advantage function; These are the weighting coefficients; Actual advantage function The generalized dominance estimation method is used to calculate the value from the actual interaction trajectory, and it is expressed as follows:

[0021] In the formula, This is the timing difference error term. Indicates time Instant rewards received As a discount factor, The state value function; This is a hyperparameter unique to GAE, used to control the trade-off between bias and variance; is the GAE smoothing coefficient, a hyperparameter between 0 and 1; Weighting coefficient Configured for adaptive adjustment, determined by the following adaptive adjustment strategy:

[0022] In the formula, and Weighting coefficients The preset upper and lower limits, This is the decay coefficient, used to control the decay rate of the weights over time. Number of training cycles; After generating the action sequence based on the modified control strategy, a subgraph search operation is performed. Starting from the key state node at the moment of successful action execution, the previously verified cause-effect graph is traversed in reverse. ,in This represents the set of all variables in the graph. This represents the set of causal edges between variables; it identifies all state and action variables that have a direct or indirect causal relationship with task success, forming the set of nodes in the initial skill causal subgraph. Then, causal pruning is performed on each variable in the initial skill causal subgraph. Calculate its impact on the success metrics of execution. Causal contribution:

[0023] By removing variables and related edges whose causal contribution is below a preset threshold ξ, a simplified skill causal subgraph is obtained. , This represents the set of state and action variables that are critically related to the subtask. The set of edges representing causal dependencies between variables.

[0024] Skill causal subgraph Structured encoding forms a directed acyclic graph structure. And store it; When the robot performs a new task, a new task causal graph is constructed by extracting the causal features of the task. The graph similarity metric algorithm retrieves structurally matching causal subgraphs of skills from the skill base. The graph similarity is calculated as follows:

[0025] In the formula, Indicates the graph edit distance. Indicates the semantic similarity of nodes. and These are the weighting coefficients. ; For the retrieved skill causal subgraphs, adaptation verification is performed using counterfactual reasoning, including simulating the execution effect under the new task causal graph through intervention operations. After successful verification, policy distillation is used to integrate the policy knowledge contained in the skill causal subgraphs into the new task policy network, achieving knowledge transfer by minimizing the following distillation loss function:

[0026] In the formula, This represents the output of the new task policy network. This represents the distribution of target policies guided by the skill causal subgraph. This represents the KL divergence, used to measure the difference between two probability distributions.

[0027] The beneficial effects of this invention are as follows: This invention provides a robot trial-and-error learning and skill generation method based on causal reasoning. In a safe simulation environment or a restricted real environment, the robot conducts random or guided exploration to collect interaction data between environmental state variables and robot action variables. Using the collected interaction data, a causal discovery algorithm is used to learn the causal relationship between state variables and action variables, constructing a causal graph, where nodes represent state variables and action variables, and edges represent causal relationships. When the robot fails to execute a task, the constructed causal graph is used as the basis for calculation, and the impact of different action sequences on the task result is calculated through hypothesis analysis. Based on the results of counterfactual reasoning, the robot's current control strategy is dynamically corrected, and new action sequences are generated to form a reusable skill set.

[0028] This invention constructs a causal graph to explicitly model the causal relationship between environmental states and robot actions, enabling the learning process to overcome the limitations of traditional correlation-based methods and achieve the expected performance with fewer trial and error attempts. By using counterfactual reasoning to perform hypothesis analysis on failed tasks, it provides learning signals for better actions that were not executed, improving the efficiency and stability of policy optimization. The generated skills are stored in the form of causal subgraphs, possessing both interpretability and structured characteristics, facilitating skill reuse and combination. When faced with new tasks, it can quickly retrieve and adapt skills based on the similarity of causal structures, effectively improving the robot's adaptability and generalization ability in unknown environments.

[0029] Other advantages, objectives, and features of the invention will be set forth in part in the description which follows, and in part will be apparent to those skilled in the art from the following examination, or may be learned from practice of the invention. The objectives and other advantages of the invention can be realized and obtained through the following description. Attached Figure Description

[0030] To make the objectives, technical solutions, and advantages of the present invention clearer, the preferred embodiments of the present invention will be described in detail below with reference to the accompanying drawings, wherein: Figure 1 This is a schematic diagram of a robot trial-and-error learning and skill generation method according to an embodiment of the present invention; Figure 2 A schematic diagram for constructing a causal model; Figure 3 This is a flowchart of counterfactual reasoning. Figure 4 This is a diagram illustrating skill generation and transfer. Detailed Implementation

[0031] The following specific examples illustrate the implementation of the present invention. Those skilled in the art can easily understand other advantages and effects of the present invention from the content disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and various details in this specification can be modified or changed based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that the illustrations provided in the following embodiments are only schematic representations of the basic concept of the present invention. Unless otherwise specified, the following embodiments and features can be combined with each other.

[0032] The accompanying drawings are for illustrative purposes only and are schematic diagrams, not actual pictures. They should not be construed as limiting the invention. To better illustrate the embodiments of the invention, some parts in the drawings may be omitted, enlarged, or reduced, and do not represent the actual product dimensions. It is understandable to those skilled in the art that some well-known structures and their descriptions may be omitted in the drawings.

[0033] In the accompanying drawings of the embodiments of the present invention, the same or similar reference numerals correspond to the same or similar components. In the description of the present invention, it should be understood that if terms such as "upper," "lower," "left," "right," "front," and "rear" indicate the orientation or positional relationship based on the orientation or positional relationship shown in the drawings, they are only for the convenience of describing the present invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, the terms used to describe positional relationships in the drawings are only for illustrative purposes and should not be construed as limiting the present invention. For those skilled in the art, the specific meaning of the above terms can be understood according to the specific circumstances.

[0034] One embodiment of the present invention provides a robot trial-and-error learning and skill generation method based on causal reasoning. The method includes: collecting interaction data of environmental state variables and robot action variables through random or guided exploration in a safe simulation environment or a restricted real environment; using the collected interaction data, learning the causal relationship between state variables and action variables through a causal discovery algorithm to construct a causal graph, where nodes represent state variables and action variables, and edges represent causal relationships; when the robot fails to execute a task, using the constructed causal graph as the basis for calculation, calculating the impact of different action sequences on the task result through hypothesis analysis; and dynamically correcting the robot's current control strategy based on the results of counterfactual reasoning, and generating new action sequences to form a reusable skill set.

[0035] like Figure 1 As shown, the method is as follows: First, in a safe simulation environment or a restricted real environment, the robot conducts random exploration or guided exploration based on current causal knowledge, systematically collecting the interaction data sequence between environmental state variables and robot action variables, providing a sufficient data foundation for subsequent causal analysis.

[0036] During the data acquisition process, the robot perceives the environmental state in real time through its onboard sensors. The environmental state variables include at least the robot's pose, joint angles, end effector coordinates, and the positions and physical properties of objects in the environment. At the same time, the robot records the motion variables it performs, which include at least joint torque output, movement trajectory commands, or grasping force parameters.

[0037] The exploration strategy dynamically integrates random exploration and causal-guided exploration using mathematical formulas: the random exploration part uses uniform or Gaussian distributions to randomly generate actions to ensure extensive sampling of the environmental space; the guided exploration part, based on a pre-constructed causal graph, recommends candidate actions with high causal influence by querying the causal relationships between state nodes and action nodes. The exploration strategy is represented as follows:

[0038] in, This indicates the final exploration strategy. This indicates a purely random strategy to ensure breadth of exploration. Indicates based on the current causal graph Strategies for providing action suggestions. The exploration rate parameter is dynamically adjusted to balance exploration and utilization. Indicates the state of the environment. This indicates the robot's actions.

[0039] The exploration rate parameter is dynamically decayed or adjusted based on the task execution progress, learning stability indicators, or environmental complexity to achieve a balanced optimization of exploration and utilization.

[0040] The collected interaction data, after time series alignment, noise filtering, and normalization preprocessing, forms a standardized state-action pair dataset, providing high-quality, low-redundancy data input for subsequent causal discovery.

[0041] Second, by utilizing the collected interaction data, the inherent causal relationship between state variables and action variables is automatically learned through the causal discovery algorithm, and a structured causal graph model is constructed. In this model, the nodes in the causal graph represent the environmental state and the robot's actions, and the directed edges between the nodes accurately depict the causal influence relationship between the variables.

[0042] The collected raw interaction data is preprocessed, including data cleaning to remove outliers and standardization to eliminate the influence of units. Simultaneously, the state sequences and action sequences are strictly aligned based on timestamps. The preprocessed data is then input into a causal discovery algorithm, which learns the causal dependency structure between variables through conditional independence testing or a scoring search method, generating an initial causal graph. This initial causal graph is then instantiated into a structural causal model, establishing a structural equation for each variable in the graph and clarifying its functional relationship with parent nodes and exogenous noise variables. Finally, the learned structural causal model is validated and refined, including statistical robustness verification and domain knowledge consistency verification, to ensure its reliability and robustness, forming a complete causal representation.

[0043] like Figure 2 As shown, the processing steps of the cause-effect graph construction step in this embodiment are as follows: 1. Preprocess the collected raw interaction data to prepare high-quality input for causal discovery.

[0044] Data cleaning is performed to remove obvious outliers caused by sensor malfunctions or communication anomalies; state and action variables with different dimensions are standardized to have a mean of 0 and a variance of 1 to eliminate the impact of numerical scale differences on the causal discovery algorithm; finally, the state observation sequence and action execution sequence are strictly aligned according to precise timestamps to ensure data temporal consistency. After the above processing, a standardized dataset is obtained: ,in, For a moment The environment state vector, This represents the robot's motion vector at the corresponding moment. This represents the total number of time steps.

[0045] 2. Utilize the preprocessed data Perform causal structure learning to infer causal dependencies between variables and construct an initial causal graph. .

[0046] A score-based causal discovery method is employed, which searches for the optimal directed acyclic graph (DAG) structure by optimizing a scoring function. The scoring function is defined as follows:

[0047] in, Representing graph structure Data The log-likelihood is used to measure the goodness of fit between the graph and the data; The complexity of a graph is usually measured by the number of edges. This is the regularization coefficient, used to control model complexity and prevent overfitting.

[0048] By employing optimization techniques such as greedy search, we can find ways to improve the rating. Largest graph structure , graph structure This is the initial causal graph obtained through learning, where the nodes correspond to state and action variables, and the directed edges represent the direct causal influence between variables.

[0049] 3. In a defined cause-effect graph structure Based on this, parameter learning is performed to estimate the specific form of the structural equation for each variable in the graph.

[0050] For any variable in the causal graph (These can be state or action variables), assuming their structural equations are: ,

[0051] in, In a cause-effect graph Medium variables The set of all parent nodes, This is an exogenous noise term independent of the parent node. n This represents the total number of variables.

[0052] use The corresponding data is used to learn the function using a nonlinear regression method. Specifically, it is estimated by minimizing the following objective function. parameters :

[0053] in, Representing variables At any moment The observed values, Representing variables The vector formed by the values ​​of the parent nodes at the corresponding times. Represents the regularization term for the parameter. This corresponds to the regularization strength. This step quantifies the influence strength and functional form of each causal dependency in the causal graph, which together constitute a complete structural causal model.

[0054] 4. Verify and refine the learned structural causal model to ensure its reliability and robustness.

[0055] First, statistical robustness verification is performed using K-fold cross-validation. The dataset is randomly divided into K subsets, and causal discovery and parameter learning are performed sequentially on K-1 subsets. The model's prediction accuracy is then evaluated on the remaining subset. Simultaneously, domain knowledge consistency verification is conducted by comparing the structural causal model with prior knowledge in the relevant domain. Causal edges that clearly violate domain consensus are corrected or removed based on the results of conditional independence tests. The final verified structural causal model will serve as the basis for subsequent counterfactual inference and policy optimization.

[0056] Third, when the robot task fails, based on the constructed causal graph, counterfactual reasoning and hypothesis analysis are performed by intervening in key action variables in the causal model to deduce and quantify the potential impact of different action sequences on the task outcome, such as... Figure 3 As shown.

[0057] The complete sequence of events actually observed by the robot during task execution is used as inference evidence, including the environmental state sequence, the sequence of executed actions, and the final task result. Based on this inference evidence, Bayesian inference methods are used to calculate the posterior probability distribution of exogenous variables. Approximate inference is then performed using variational inference or Markov chain Monte Carlo methods to accurately capture the specific conditions and unobserved confounding factors leading to the current failure scenario. Subsequently, key decision nodes with the greatest causal impact on the task result are identified in the causal graph, and intervention operations are performed on the corresponding action variables. These operations include sampling hypothetical action values ​​from a predefined action space, severing the causal connection between the target variable and its parent node, modifying the corresponding structural equations, and constructing a complete hypothetical scenario model.

[0058] Based on the post-intervention causal model, forward extrapolation is performed using the updated exogenous variable distribution to calculate the evolution trajectory of the state sequence until task termination. Counterfactual results are obtained, and their expected values ​​are calculated using Monte Carlo sampling. A quantified counterfactual advantage function is generated as the gradient signal for policy optimization. Simultaneously, causally validated effective action sequences are stored in a hierarchically structured skill library, supporting retrieval and reuse based on task context and performance indicators. Efficient skill management and cross-task transfer are achieved through embedding representation and abstraction mechanisms.

[0059] Furthermore, by integrating long-term sequence reasoning and multi-task learning mechanisms, it can handle sparse reward and noisy data scenarios, enhancing robustness in practical applications. Through the deep integration of causal reasoning and reinforcement learning, it can achieve accurate diagnosis of the root causes of task failure and efficient optimization of strategies.

[0060] In this embodiment, the counterfactual reasoning processing steps are as follows: 1. Use observed event sequences as evidence, including environmental conditions. Robot Actions and the result of mission failure Used to update and infer exogenous variables The posterior distribution. In practice, inference evidence is based on observations. The posterior probability distribution of exogenous variables is calculated using a Bayesian update mechanism. Its formal expression is as follows:

[0061] in, For the prior distribution of exogenous variables, This is the likelihood function, determined by the structural equations in the structural causal model. Exogenous variables. This includes parameters that are difficult to observe directly, such as environmental physical parameters, sensor noise characteristics, and actuator dynamic characteristics. A variational inference method is used to approximate the posterior probability distribution. By optimizing the lower bound of evidence to approximate the true posterior:

[0062] in, For variational posterior distribution, This represents the KL divergence. This step ensures that the estimation of exogenous variables fully reflects the random factors and inherent uncertainties in the actual execution environment.

[0063] 2. Intervene in the causal model: This involves modifying the action variables at a key state node in the historical timeline. Forced to be set to a hypothetical new value , represented as The specific implementation of this operation involves modifying the structural equation of the structural causal model, thereby affecting the action variables being intervened upon. The corresponding structural equation is replaced with a constant function. This also removes the functional dependency between the variable and its original parent node. The formal expression of the intervention operation is: In the original model M middle:

[0064] Post-intervention model middle,

[0065] in, In a cause-effect graph Medium variables The set of all parent nodes, This represents the exogenous noise term, independent of the parent node. The selection of intervention points is based on causal effect strength assessment, using the following criteria to select the optimal intervention target:

[0066] in The partial derivative quantifies the action variables, representing the key indicator of task success or failure. Regarding the results The strength of the causal effect.

[0067] 3. The model after intervention Below, based on the posterior distribution of exogenous variables The counterfactual result is calculated through forward structural equation modeling. In practice, the Monte Carlo method is used for probabilistic deduction, starting from the posterior distribution. Extraction Sample For each sample, according to the post-intervention model The structural equation model sequentially calculates the values ​​of each endogenous variable, ultimately yielding counterfactual results. The expected value of the counterfactual outcome is calculated from the sample mean:

[0068] This expected value provides a crucial counterfactual learning signal for policy correction. The counterfactual results... Quantified as a cumulative reward function As a preliminary quantitative output of counterfactual reasoning:

[0069] in, This is a discount factor used to weigh the importance of current and future rewards. The immediate reward function is used to evaluate the state. Next action The immediate benefits obtained For the task time range, and They are respectively t The environmental state variables at any given time and the action variables performed by the robot.

[0070] 4. To effectively incorporate counterfactual outcomes into policy optimization, the cumulative reward function needs to be further optimized. Transform into a counterfactual advantage function The counterfactual advantage function quantifies the specific actions taken in a counterfactual situation. The calculation process for the superiority of the average action in this state is as follows: First, the cumulative reward derived from counterfactual reasoning. Estimating the counterfactual action value function It indicates the state. Next action And the expected cumulative reward that can be obtained by following the counterfactual strategy in the future:

[0071] in, k For time indexing, starting from the current moment t Accumulate until the end of the task ; and They are respectively k The state and actions at any given moment; This refers to the counterfactual model after intervention in a causal model.

[0072] Secondly, calculate the counterfactual state value function. It indicates the state. The following follows the counterfactual strategy The expected cumulative reward can be obtained by weighting the counterfactual action-value function of all possible actions according to the policy distribution:

[0073] in, Counterfactual strategies are those evaluated under intervention conditions.

[0074] Ultimately, the counterfactual advantage function Determined by the difference between the counterfactual action value function and the counterfactual state value function:

[0075] Counterfactual advantage function As the core gradient signal, it is provided to the policy optimization module to guide the robot control policy to update in a better direction as verified by counterfactual evidence.

[0076] Fourth, based on the results of counterfactual reasoning, dynamically correct the robot's current control strategy, generate better action sequences verified by counterfactual reasoning, and abstract and encapsulate these successful action sequences and their causal contexts into reusable skill packages to achieve knowledge accumulation and transfer.

[0077] Based on the counterfactual advantage function and counterfactual result evaluation output from the counterfactual reasoning process, a reinforcement learning algorithm is used to dynamically update the robot's control policy parameters, causing the policy to converge towards the better direction verified by counterfactual evidence. Specifically, by integrating counterfactual advantage information into policy gradient calculation, learning signals of actions that have not been executed but have been proven to be better through reasoning are provided for policy updates, thereby significantly improving the efficiency and stability of policy optimization. Simultaneously, key causal relationships are extracted from successfully executed action sequences and their corresponding environmental states, constructing a skill package represented in the form of a skill causal subgraph. This skill package not only encapsulates the action sequence but also includes its causal context, giving the skills interpretability and structured characteristics. The generated skill package is stored in a hierarchical skill library, supporting rapid retrieval and cross-task transfer based on causal structure similarity, thereby achieving effective knowledge accumulation and reuse, and enhancing the robot's adaptability in unknown environments. A schematic diagram of skill generation and transfer is shown below. Figure 4 As shown.

[0078] In this embodiment, the processing steps for strategy modification and skill generation are as follows: 1. Based on counterfactual reasoning results, the current control policy of the robot is dynamically corrected using the policy gradient method. The policy gradient update method incorporates the counterfactual advantage function into the policy optimization process. Policy parameters. Updates are performed based on the following gradients:

[0079] in, Describe the objective function Relative to policy parameters The gradient is used to guide the optimization direction of the strategy; Indicates regarding following the current strategy generated trajectory The mathematical expectation operator; This represents the total time step of the trajectory; Representation Strategy In state Select action below The log probability relative to the parameter The gradient; Defined as state Next action The actual advantage function is used to evaluate the superiority of the action compared to the average level; Defined as a counterfactual advantage function, provided by the counterfactual reasoning engine, it is used to evaluate the expected advantage level of the action in a counterfactual situation; It is a weighting coefficient used to balance the relative contribution weights of actual advantages and counterfactual advantages in policy updates.

[0080] The actual advantage function The expression for the generalized advantage estimation (GAE) method, calculated from the actual interaction trajectory, is as follows:

[0081] in, This is the timing difference error term. Indicates time Instant rewards received As a discount factor, The state value function; This is a hyperparameter unique to GAE, used to control the trade-off between bias and variance; is the GAE smoothing coefficient, a hyperparameter between 0 and 1.

[0082] Counterfactual advantage function Provided by the counterfactual reasoning engine, it calculates the result based on the difference between the counterfactual reward and the counterfactual state value:

[0083] in, For counterfactual action value function, It is a counterfactual state value function.

[0084] The weighting coefficient Configured for adaptive adjustment, determined by the following adaptive adjustment strategy:

[0085] in, and Weighting coefficients The preset upper and lower limits, This is the decay coefficient, used to control the decay rate of the weights over time. This represents the number of training epochs. This design allows for the full exploration of high-quality signals provided by counterfactual reasoning in the early stages of training, while relying more on actual interaction data in the later stages when the policy tends to stabilize, thus ensuring the stability of the algorithm's convergence.

[0086] 2. Extract key state and action variables from successfully executed action sequences, and construct and encode the skill causal subgraph.

[0087] Perform a subgraph search operation, starting from the key state node at the moment the action is successfully executed, and traverse the previously verified cause-effect graph in reverse order. ,in This represents the set of all variables in the graph. This represents the set of causal edges between variables; it identifies all state and action variables that have a direct or indirect causal relationship with task success, forming the set of nodes in the initial skill causal subgraph. Causal pruning is performed on each variable in the initial skill causal subgraph. Calculate its impact on the success metrics of execution. The causal contribution is calculated using the following formula:

[0088] By removing variables and their associated edges whose causal contribution is below a preset threshold ξ, a simplified skill causal subgraph is obtained. After completing the structured encoding and storage, the skill causal subgraph is formally defined as a directed acyclic graph structure: ,in, This represents the set of state and action variables that are critically related to the subtask. A set of edges representing causal dependencies between variables. The encoded skill causal subgraph is persistently stored in a hierarchical skill library.

[0089] 3. Implement a structured combination of skills and a cross-task transfer mechanism.

[0090] When the robot performs a new task, a new task causal graph is constructed by extracting the causal features of the task. The algorithm retrieves structurally matching causal subgraphs of skills from a skill base based on graph similarity measurement. Graph similarity is calculated using the following formula:

[0091] in, Indicates the graph edit distance. Indicates the semantic similarity of nodes. and Let be the weight coefficient, and satisfy... .

[0092] For the retrieved skill causal subgraphs, counterfactual reasoning is used for adaptation verification, including simulating their execution effects under the new task causal graph through intervention operations. After successful verification, policy distillation is employed to integrate the policy knowledge contained in the skill causal subgraphs into the new task policy network. Knowledge transfer is achieved by minimizing the following distillation loss function:

[0093] in, This represents the output of the new task policy network. This represents the distribution of target policies guided by the skill causal subgraph. This represents the KL divergence, used to measure the difference between two probability distributions.

[0094] This invention constructs a causal graph to explicitly model the causal relationship between environmental states and robot actions, enabling the learning process to overcome the limitations of traditional correlation-based methods and achieve the expected performance with fewer trial and error attempts. By using counterfactual reasoning to perform hypothesis analysis on failed tasks, it provides learning signals for better actions that were not executed, improving the efficiency and stability of policy optimization. The generated skills are stored in the form of causal subgraphs, possessing both interpretability and structured characteristics, facilitating skill reuse and combination. When faced with new tasks, it can quickly retrieve and adapt skills based on the similarity of causal structures, effectively improving the robot's adaptability and generalization ability in unknown environments.

[0095] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the present invention, and all such modifications or substitutions should be covered within the scope of the claims of the present invention.

Claims

1. A method for robot trial-and-error learning and skill generation based on causal reasoning, characterized in that, By conducting random or guided exploration with robots, the interaction data sequence between environmental state variables and robot action variables is collected; Based on the collected interaction data, the causal relationship between state variables and action variables is learned through the causal discovery algorithm, and a causal graph is constructed. In this causal graph, nodes represent state variables and action variables, and edges represent causal relationships. When the robot fails to execute a task, the constructed cause-effect graph is used as the basis for calculation, and the impact of different action sequences on the task result is calculated through hypothesis analysis. Based on the results of counterfactual reasoning, the robot's current control strategy is dynamically modified, and new action sequences are generated to form a reusable skill set.

2. The method according to claim 1, characterized in that, The collected raw interaction data is preprocessed and then input into the causal discovery algorithm to learn the causal dependency structure between variables and generate an initial causal graph. The initial causal graph is then instantiated into a structural causal model, and a structural equation is established for each variable in the graph to clarify its functional relationship with the parent node and exogenous noise variables. Finally, the learned structural causal model is verified and refined to form a complete causal representation.

3. The method according to claim 2, characterized in that, Using preprocessed interaction data To perform causal structure learning, inferring causal dependencies between variables and constructing an initial causal graph. A score-based causal discovery method is employed, which searches for the optimal directed acyclic graph structure by optimizing the scoring function. The scoring function is: , Representing graph structure Data The log-likelihood, Representing the complexity of a graph, The regularization coefficient is used; through optimization techniques such as greedy search, a coefficient is found that makes the score... Largest graph structure ; In a defined graph structure Based on this, parameter learning is performed to estimate the structural equation for each variable in the causal graph. Its structural equation is expressed as: in, In a cause-effect graph Medium variables The set of all parent nodes, This is an exogenous noise term independent of the parent node; Using interactive data The corresponding data is used to learn the function using a nonlinear regression method. Estimate by minimizing the objective function parameters The objective function is: In the formula, Representing variables At any moment The observed values, Representing variables The vector formed by the values ​​of the parent nodes at the corresponding times. Represents the regularization term for the parameter. To determine the regularization strength; by learning parameters for each causal dependency in the causal graph, a structural causal model is obtained; The learned structural causal model is validated and refined. First, statistical robustness is verified using K-fold cross-validation, which randomly divides the dataset into K subsets. K-1 subsets are used sequentially for causal discovery and parameter learning, and the prediction accuracy of the structural causal model is evaluated on the remaining subset. Then, domain knowledge consistency is verified by comparing the structural causal model with prior knowledge in the relevant domain. Causal edges that clearly violate domain consensus are corrected or removed based on the results of conditional independence tests.

4. The method according to claim 2, characterized in that, When the robot fails to execute a task, the complete sequence of events actually observed by the robot during the task execution is used as inference evidence. The inference evidence includes the sequence of environmental states, the sequence of execution actions, and the final task result. Based on the inference evidence, the Bayesian inference method is used to calculate the posterior probability distribution of the exogenous variables. Identify the key decision nodes in the causal graph that have the greatest causal impact on the task outcome, and perform intervention operations on the corresponding action variables, including sampling from the predefined action space to generate hypothetical action values, severing the causal connection between the target variable and the parent node, and modifying the corresponding structural equations; Based on the model after intervention, forward extrapolation is performed based on the updated distribution of exogenous variables to calculate the values ​​of endogenous variables, thereby obtaining the counterfactual results corresponding to the assumed actions; the evolution trajectory of the state sequence is calculated until the task terminates, thereby obtaining the counterfactual results and calculating their expected values ​​through Monte Carlo sampling, generating a quantified counterfactual advantage function as the gradient signal for policy optimization.

5. The method according to claim 4, characterized in that, Inference based on observation The posterior probability distribution of exogenous variables is calculated using a Bayesian update mechanism. : In the formula, exogenous variables U The prior distribution, Let be the likelihood function; the posterior probability distribution is approximated using variational inference methods. By optimizing the lower bound of evidence to approximate the true posterior: In the formula, For variational posterior distribution, Indicates KL divergence; Intervention operations on the causal model: The action variables at a key state node in history... Forced to be set to a new value By modifying the structural equation of the structural causal model, the action variables to be intervened are... The corresponding structural equation is replaced with a constant function. At the same time, remove intervention variables Functional dependencies between the intervention point and its original parent node; the selection of the intervention point is based on the assessment of the causal effect strength, and the following criteria are used to select the optimal intervention target: In the formula, Key indicators of the success or failure of a task. x This indicates the specific constant value that is set; Based on the post-intervention model According to the posterior probability distribution of exogenous variables Counterfactual results are calculated through forward structural equation modeling, and probabilistic deduction is performed using the Monte Carlo method, based on the posterior probability distribution. Extraction Sample For each sample, according to the post-intervention model The structural equation model sequentially calculates the values ​​of each endogenous variable, ultimately yielding counterfactual results. The expected value of the counterfactual outcome is calculated using the sample mean: Counterfactual results This is quantified into a cumulative reward function, serving as the initial quantification output for counterfactual reasoning: In the formula, This is a discount factor used to weigh the importance of current and future rewards; The immediate reward function is used to evaluate the state. Next action The immediate benefits gained; For the task time range, and They are respectively t The environmental state variables at any given time and the action variables performed by the robot.

6. The method according to claim 4, characterized in that, Based on the counterfactual advantage function obtained from counterfactual reasoning, the robot's current control strategy is dynamically modified to generate a better action sequence verified by counterfactual reasoning. Extract key causal relationships from successfully executed action sequences and corresponding environmental states, and construct a skill package represented in the form of a skill causal subgraph; The generated skill packs are stored in a hierarchical skill library, supporting fast retrieval and cross-task migration based on causal structure similarity.

7. The method according to claim 6, characterized in that, Based on the counterfactual reasoning results, they are initially quantified as a cumulative reward function. After calculating the counterfactual action-value function and counterfactual state-value function using the cumulative reward function, a counterfactual dominance function is constructed by subtracting the counterfactual action-value function and counterfactual state-value function. The counterfactual dominance function is then integrated into the policy optimization process using a policy gradient update method. (Policy parameters...) Updates are performed based on the following gradients: In the formula, Describe the objective function Relative to policy parameters The gradient is used to guide the optimization direction of the strategy; Indicates regarding following the current strategy generated trajectory The mathematical expectation operator; This represents the total time step of the trajectory; Representation Strategy In state Select action below The log probability relative to the parameter The gradient; For state Next action The actual advantage function; It is the counterfactual advantage function; These are the weighting coefficients; Actual advantage function The generalized dominance estimation method is used to calculate the value from the actual interaction trajectory, and it is expressed as follows: In the formula, This is the timing difference error term. Indicates time Instant rewards received As a discount factor, The state value function; This is a hyperparameter unique to GAE, used to control the trade-off between bias and variance; is the GAE smoothing coefficient, a hyperparameter between 0 and 1; Weighting coefficient Configured for adaptive adjustment, determined by the following adaptive adjustment strategy: In the formula, and Weighting coefficients The preset upper and lower limits, This is the decay coefficient, used to control the decay rate of the weights over time. The number of training cycles; After generating the action sequence based on the revised control strategy, a subgraph search operation is performed, starting from the key state node at the moment the action is successfully executed, and traversing the verified cause-effect graph in reverse. ,in This represents the set of all variables in the graph. This represents the set of causal edges between variables; it identifies all state and action variables that have a direct or indirect causal relationship with task success, forming the set of nodes in the initial skill causal subgraph. Then, causal pruning is performed on each variable in the initial skill causal subgraph. Calculate its impact on the success metrics of execution. Causal contribution: By removing variables and related edges whose causal contribution is below a preset threshold ξ, a simplified skill causal subgraph is obtained. , This represents the set of state and action variables that are critically related to the subtask. The set of edges representing causal dependencies between variables.

8. The method according to claim 7, characterized in that, Skill causal subgraph Structured encoding to form a directed acyclic graph structure And store it; When the robot performs a new task, a new task causal graph is constructed by extracting the causal features of the task. The graph similarity metric algorithm retrieves structurally matching causal subgraphs of skills from the skill base. The graph similarity is calculated as follows: In the formula, Indicates the graph edit distance. Indicates the semantic similarity of nodes. and These are the weighting coefficients. ; For the retrieved skill causal subgraphs, adaptation verification is performed using counterfactual reasoning, including simulating the execution effect under the new task causal graph through intervention operations. After successful verification, policy distillation is used to integrate the policy knowledge contained in the skill causal subgraphs into the new task policy network, achieving knowledge transfer by minimizing the following distillation loss function: In the formula, This represents the output of the new task policy network. This represents the distribution of target policies guided by the skill causal subgraph. This represents the KL divergence, used to measure the difference between two probability distributions.

Images