A robot intelligent control method and system based on a causal mechanism

Abstract

The application discloses a robot intelligent control method and system based on a causal mechanism, and relates to the technical field of intelligent robots, which comprises the following steps: inputting an operation task and an environment state of a target robot at a current time into a causal structure model to determine each candidate action for pushing a target object from a current time position to a next time position; inputting the candidate action into a graph recurrent network to obtain a predicted position of the target object under the candidate action at the next time; screening an execution action of the target robot at the next time according to the predicted position of the target object under each candidate action and a target position of the target object; and updating the predicted position corresponding to the execution action of the target robot at the next time to the current time position until the target object reaches the target position. The application can effectively improve the intelligent control capability of the robot by determining the action to be executed according to the predicted position of the target object under the execution action and the target position based on the causal structure model and the graph recurrent network.

Description

Technical Field

[0001] This invention relates to the field of intelligent robots, and in particular to a robot intelligent control method and system based on a causal mechanism. Background Technology

[0002] Robots play a pivotal role in the field of automation, and their ability to handle objects of different shapes is crucial for improving production efficiency and quality.

[0003] Currently, the main control methods used by robots include reinforcement learning and imitation learning. However, reinforcement learning typically requires a significant amount of time for trial and error, and due to sparse rewards, it is prone to learning failures, struggles with long-term tasks, and faces safety issues during the exploration process. Imitation learning, on the other hand, faces challenges such as high data acquisition costs, difficult data acquisition methods, and the susceptibility to obtaining poor teaching data. These problems make it difficult for robots to effectively learn skills, thus hindering the achievement of highly efficient intelligent control. Summary of the Invention

[0004] The purpose of this invention is to provide a robot intelligent control method and system based on a causal mechanism, which enables robots to achieve highly efficient intelligent control.

[0005] To achieve the above objectives, the present invention provides the following solution:

[0006] In a first aspect, the present invention provides a robot intelligent control method based on a causal mechanism, comprising:

[0007] The current operation task and environmental state of the target robot are obtained; the operation task is to push the target object from the initial position to the target position, and the environmental state is the environment in which the target robot is located.

[0008] The operation task and the environmental state are input into the causal structure model to determine the candidate actions for the target robot to push the target object from the current position to the next position. The causal structure model is a model trained based on a graph neural network and trained on historical non-target-oriented data. The historical non-target-oriented data includes the position information of the target object before the action is performed, the position information of the target object after the action is performed, and the action performed by the target robot when the target robot performs the historical operation task.

[0009] For each candidate action, the candidate action is input into a graph recurrent network, and the predicted position of the target object under the candidate action at the next time step is obtained based on a multivariate Gaussian distribution.

[0010] Based on the predicted position of the target object under each candidate action at the next moment and the target position of the target object, the execution action of the target robot at the next moment is selected from each candidate action, and the predicted position corresponding to the execution action of the target robot at the next moment is updated to the current position. Then, the process returns to the step: inputting the operation task and the environmental state into the causal structure model to determine each candidate action of the target robot to push the target object from the current position to the next position, until the target object reaches the target position.

[0011] Optionally, the process for determining the causal structure model is as follows:

[0012] Obtain key point information of the target object; the key point information includes a vertex set and an edge set; the vertex set is the encoding information of each executable action point of the target object, and the edge set is the directed relationship from one executable action point to another.

[0013] Based on the position information of the target object before the action is performed and the position information of the target object after the action is performed in the historical non-target guidance data, the key point sequence of the target object when the target robot performs the historical operation task is determined; the key point sequence includes the key point information of the target object at each moment;

[0014] The keypoint sequence is input into a first graph neural network to obtain the keypoints and keypoint pairs of the target object; the keypoint pairs are determined based on the edge set in the keypoint information; the keypoints are the executable action points of the target object.

[0015] The embedded representations of the key points and key point pairs are input into the second graph neural network to obtain the predicted probability distribution of the relationship type of the key point pairs;

[0016] The predicted probability distribution of the relationship type of the key point pair is input into the third graph neural network to obtain continuous parameters of the degree of correlation between the two key points in each key point pair.

[0017] The causal structure model is obtained based on the embedded representation of the key points and key point pairs, the predicted probability distribution of the relationship type of the key point pairs, and the continuous parameters of the correlation between the two key points in the key point pairs.

[0018] Optionally, the keypoint sequence is input into a first graph neural network to obtain the keypoints and keypoint pairs of the target object, as follows:

[0019]

[0020] in, These are the embedding representations of keypoints and keypoint pairs, respectively. This is the first graph neural network. For edge set, For keypoint sequences, O m,i This represents the information for the i-th key point.

[0021] Optionally, the embedded representations of the key points and key point pairs are input into a second graph neural network to obtain the predicted probability distribution of the key point pair relationship type, as follows:

[0022]

[0023] in, The probability distribution for predicting the relationship type of key points. and For the embedding representation of key point i, For the embedding representation of key point j,

[0024] An embedded representation of the relationship between keypoint i and keypoint j. This is the second neural network.

[0025] Optionally, the predicted probability distribution of the keypoint pair relationship type is input into a third graph neural network to obtain continuous parameters of the correlation between the two keypoints in each keypoint pair, as follows:

[0026]

[0027] in, This is a continuous parameter representing the degree of correlation between the two key points in the keypoint pair. This is the third graph neural network. For the embedding representation of key point i, For the embedding representation of key point j, An embedded representation of the relationship between keypoint i and keypoint j. This represents the predicted probability distribution for the relationship type of key points.

[0028] Optionally, the causal structure model is as follows:

[0029]

[0030] in, Continuous parameters representing the predictive probability distribution and correlation degree of the relationship between keypoint pairs; An embedded representation of key points from time 1 to T, o m,i Represented as the i-th key point information, o m,j This is represented as the j-th key point information.

[0031] Optionally, the environmental state of the target robot specifically includes the two-dimensional coordinates of the target object on its plane and the rotation angle of the target object along the normal axis of the plane.

[0032] Optionally, based on the predicted position of the target object under each candidate action at the next moment and the target position of the target object, the action to be executed by the target robot at the next moment is selected from each candidate action, specifically including:

[0033] According to the formula Ω g :={s∈S||xx g |≤δ x ,|yy g |≤δ y ,|θ-θ g |≤δ θ} Determine the action that the target robot needs to perform from among the candidate actions;

[0034] Among them, Ω g Here, δx represents the predicted position of the target object under each of the candidate actions, δy represents the threshold distance from the target position along the x-axis, δθ represents the threshold distance along the y-axis, and δθ represents the rotation angle threshold. g Let y be the x-coordinate of the target object. g Let θ be the target ordinate of the target object. g Let θ be the target rotation angle of the target object, x be the x-coordinate of the target object at the current moment, y be the y-coordinate of the target object at the current moment, and θ be the rotation angle of the target object at the current moment.

[0035] Secondly, the present invention provides a robot intelligent control system based on a causal mechanism, comprising:

[0036] The acquisition module is used to acquire the target robot's current operation task and environmental state; the operation task is to push the target object from its initial position to its target position, and the environmental state is the environment in which the target robot is located;

[0037] The calculation module is used to input the operation task and the environmental state into the causal structure model to determine the candidate actions of the target robot to push the target object from the current position to the next position. The causal structure model is a model trained based on a graph neural network and trained on historical non-target-oriented data. The historical non-target-oriented data includes the position information of the target object before the action is performed, the position information of the target object after the action is performed, and the action performed by the target robot when the target robot performs the historical operation task.

[0038] The action prediction module is used to input the candidate action into a graph recurrent network for each candidate action and obtain the predicted position of the target object under the candidate action at the next time step based on a multivariate Gaussian distribution.

[0039] The filtering module is used to filter out the target robot's execution action in the next moment from the candidate actions based on the predicted position of the target object under each candidate action in the next moment and the target position of the target object, update the predicted position corresponding to the target robot's execution action in the next moment to the current position, and return to the step: inputting the operation task and the environmental state into the causal structure model to determine each candidate action of the target robot to push the target object from the current position to the next position, until the target object reaches the target position.

[0040] Optionally, a model building module is also included, specifically including:

[0041] An information acquisition unit is used to acquire key point information of the target object; the key point information includes a vertex set and an edge set; the vertex set is the encoding information of each executable action point of the target object, and the edge set is the directed relationship from one executable action point to another.

[0042] A key point sequence unit is used to determine the key point sequence of the target object when the target robot performs a historical operation task, based on the position information of the target object before the action is performed and the position information of the target object after the action is performed in the historical non-target guidance data; the key point sequence includes the key point information of the target object at each moment;

[0043] The first graph neural network unit is used to input the key point sequence into the first graph neural network to obtain the key points and key point pairs of the target object; the key point pairs are determined based on the edge set in the key point information; the key points are the executable action points of the target object.

[0044] The second graph neural network unit is used to input the embedded representation of the key points and key point pairs into the second graph neural network to obtain the predicted probability distribution of the key point pair relationship type;

[0045] The third graph neural network unit is used to input the predicted probability distribution of the key point pair relationship type into the third graph neural network to obtain continuous parameters of the correlation between the two key points in each key point pair;

[0046] The model building unit is used to obtain the causal structure model based on the embedding representation of the key points and key point pairs, the predicted probability distribution of the relationship type of the key point pairs, and the continuous parameters of the correlation between the two key points in the key point pairs.

[0047] According to specific embodiments provided by the present invention, the present invention discloses the following technical effects:

[0048] This invention provides a robot intelligent control system based on a causal mechanism, comprising: acquiring the target robot's current operational task and environmental state; inputting the target robot's current operational task and environmental state into a causal structure model to determine candidate actions that push the target object from its current position to its next position; inputting the candidate actions into a graph cyclic network to obtain the predicted position of the target object under the candidate actions at the next time step; selecting the target robot's execution action for the next time step from the predicted position and the target position of the target object under each candidate action; and updating the predicted position corresponding to the target robot's execution action at the next time step to the current position, until the target object reaches the target position. This invention can effectively improve the robot's intelligent control capability, enabling the robot to better adapt to and cope with various situations in a dynamic environment. Attached Figure Description

[0049] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0050] Figure 1 A flowchart of a robot intelligent control method based on a causal mechanism provided in Embodiment 1 of the present invention;

[0051] Figure 2 is a schematic diagram of the existing learning method provided in Embodiment 1 of the present invention. Figure 2(a) is a schematic diagram of imitation learning, and Figure 2(b) is a schematic diagram of reinforcement learning.

[0052] Figure 3 This is a schematic diagram of causal reinforcement learning provided in Embodiment 1 of the present invention;

[0053] Figure 4 This is a general framework diagram of causal reinforcement learning provided in Embodiment 1 of the present invention;

[0054] Figure 5 This is a schematic diagram of the position of the target object's executable actions provided in Embodiment 1 of the present invention;

[0055] Figure 6 This is a schematic diagram of a graph neural network provided in Embodiment 1 of the present invention;

[0056] Figure 7 This is a schematic diagram of a robot intelligent control system based on a causal mechanism, provided in Embodiment 2 of the present invention. Detailed Implementation

[0057] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0058] Current primary skill learning methods for robots are reinforcement learning and imitation learning. These methods aim to help robots learn certain strategies to perform required tasks. They model state-action pairs in a state space and estimate the probability distribution of performing each action in a given state. This allows the robot to better understand the likelihood of taking different actions in different states, thereby optimizing its decision-making process. Algorithms that learn skill strategies from experience or demonstrations can provide safety and performance guarantees during the learning process, but each method has its limitations when used individually.

[0059] As shown in Figure 2(a), reinforcement learning solves decision-making problems in a human-like manner through trial-and-error interaction with the environment. However, the success of reinforcement learning relies on the data-driven trial-and-error nature of methods that freely explore the search space. These methods typically require a significant amount of time for trial-and-error learning, frequently fail due to sparse rewards, struggle with long-term tasks, and raise safety concerns during the exploration process.

[0060] As shown in Figure 2(b), imitation learning uses the state-action sequences collected from the optimal policy as training data for supervised learning algorithms, attempting to minimize the action differences between the agent's policy and the expert's policy, and learning the robot's policy; this is often referred to as behavior cloning. However, imitation learning faces challenges including high data acquisition costs, difficult acquisition methods, and the ease of obtaining poor teaching data.

[0061] The purpose of this invention is to provide a robot intelligent control method and system based on a causal mechanism, which enables robots to achieve highly efficient intelligent control.

[0062] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0063] Example 1

[0064] like Figure 1 As shown, this embodiment provides a robot intelligent control method based on a causal mechanism, including:

[0065] Step 101: Obtain the target robot's current operation task and environmental state; the operation task is the task of pushing the target object from its initial position to its target position, and the environmental state is the environment in which the target robot is located.

[0066] Step 102: Input the operation task and the environmental state into the causal structure model to determine the candidate actions of the target robot to push the target object from the current position to the next position; the causal structure model is a model trained based on a graph neural network and based on historical non-target-oriented data; the historical non-target-oriented data includes the position information of the target object before the action is performed, the position information of the target object after the action is performed, and the action performed by the target robot when the target robot performs the historical operation task.

[0067] Step 103: For each candidate action, input the candidate action into a graph recurrent network, and obtain the predicted position of the target object under the candidate action at the next time step based on a multivariate Gaussian distribution.

[0068] Step 104: Based on the predicted position of the target object under each of the candidate actions at the next moment and the target position of the target object, select the execution action of the target robot at the next moment from each of the candidate actions, update the predicted position corresponding to the execution action of the target robot at the next moment to the current position, and return to step: Input the operation task and the environmental state into the causal structure model to determine each candidate action of the target robot to push the target object from the current position to the next position, until the target object reaches the target position.

[0069] In this embodiment, the target object is as follows: Figure 5 The building blocks shown are used to learn the entire pushing process by being pushed by the robot's robotic arm to cover the target area with a reward.

[0070] Specifically, step 101 may include:

[0071] The system acquires the target robot's current operational task and environmental state, as well as the target robot's historical non-target-oriented data. The operational task is to push a target object from its initial position to its target position. The specific implementation of this task requires the robot to accurately identify and locate the object, and to apply appropriate force to achieve displacement. The historical non-target-oriented data includes the target object's position information before the action was performed, the target object's position information after the action was performed, and the target robot's actions. This data can reflect the robot's performance and reaction when completing the task, as well as the object's position changes under different states.

[0072] Specifically, the methods for obtaining historical non-goal-oriented data can be as follows:

[0073] To construct an accurate and complete causal structure model, this embodiment defines the locations of ten executable actions (e.g. Figure 5 The blocks are marked with numbers 0-9 and have a random range of force values ​​to manipulate them, simulating most possible scenarios. In discovering causal relationships, the agent aims to interact with the environment by actively executing interventions (completely randomized actions), ignoring its own latent tendencies (intuition), and randomly specifying the location and force of the action. Each time an action is executed, the current (s) value is saved. t ,a t ,s t+1 To model, where state s t ,s t+1 This indicates the position information of the block "T" before and after it is pushed, action a. t This indicates the position and magnitude of the action being performed at the current moment. In this embodiment, a total of 2000 rounds of data were collected, where the initial position s of the random building blocks in each round... t Fixed target position s g Each round consists of 300 (s) t ,a t ,s t+1 Data. Specifically, data (s) t ,a t ,s t+1 This refers to non-goal-oriented data. Non-goal-oriented means that the entire process does not have a pre-set task of "pushing the blocks from the starting position to the target position", but instead randomly selects the position of the blocks to be pushed.

[0074] Specifically, the process of determining the causal structure model during step 102 may include:

[0075] Obtain key point information of the target object; the key point information includes a vertex set and an edge set; the vertex set is the encoded information of the key points, and the edge set is the directed relationship from one key point to another.

[0076] Based on the position information of the target object before the action is performed and the position information of the target object after the action is performed in the historical non-target guidance data, a key point sequence of key points of the target object is determined when the target robot performs the historical operation task; the key point sequence includes key point information of the target object at each moment.

[0077] The key point sequence is input into a first graph neural network to obtain the key points and key point pairs of the target object; the key point pairs are determined based on the edge set in the key point information.

[0078] The embedded representations of the key points and key point pairs are input into a second graph neural network to obtain the predicted probability distribution of the key point pair relationship type.

[0079] The predicted probability distribution of the key point pair relationship type is input into the third graph neural network to obtain continuous parameters of the correlation between the two key points in each key point pair.

[0080] The causal structure model is obtained based on the embedded representation of the key points and key point pairs, the predicted probability distribution of the relationship type of the key point pairs, and the continuous parameters of the correlation between the two key points in the key point pairs.

[0081] Specifically, the keypoint sequence is input into a first graph neural network to obtain the keypoints and keypoint pairs of the target object, as follows:

[0082] The first Graph Neural Network (GNN) takes a time window of size T as input, which contains the sequence of keypoints acquired at each time step, and then passes it through the graph neural network φ. enc Information is transmitted in space. This process generates a single keypoint embedding for each time step and each pair of keypoint embeddings, where the edge set... This represents a fully connected graph, containing edges between every pair of keypoints, and these edges have zero properties. Note that the length of the time window T can be adjusted as needed. The output of the first graph neural network. These represent the embedding representations of keypoints and keypoint pairs, respectively. The specific output is as follows:

[0083]

[0084] In the formula, These are the embedding representations of keypoints and keypoint pairs, respectively. For the first graph neural network, For edge set, For keypoint sequences, O m,i This represents the information for the i-th key point.

[0085] Specifically, the embedded representations of the key points and key point pairs are input into the second graph neural network to obtain the predicted probability distribution of the key point pair relationship type, as follows:

[0086] The second graph neural network uses the output of the first graph neural network as its input. The second graph neural network φ d The aggregated key points and key point pairs are embedded with the execution action a output from the fully connected layer. 1:T The 256-dimensional embeddings are concatenated as input to predict the probability distribution of keypoint pair relation types. Output: This represents the probability distribution for each edge type. When an edge is classified as type I, i.e. In this case, the patent designates it as a "weakly related edge," which will be removed in subsequent calculations and no information will pass through it.

[0087]

[0088] In the formula, The probability distribution for predicting the relationship type of key points. and For the embedding representation of key point i, For the embedding representation of key point j, An embedded representation of the relationship between keypoint i and keypoint j. This is the second neural network.

[0089] Specifically, the predicted probability distribution of the key point pair relationship type is input into a third graph neural network to obtain continuous parameters of the correlation between the two key points in each key point pair, as follows:

[0090] The third graph neural network infers keypoint pair types. Given conditions, predict continuous parameters representing the degree of correlation between each pair of keypoints.

[0091]

[0092] in, This is a continuous parameter representing the degree of correlation between the two key points in the keypoint pair. This is the third graph neural network. For the embedding representation of key point i, For the embedding representation of key point j, An embedded representation of the relationship between keypoint i and keypoint j. This represents the predicted probability distribution for the relationship type of key points.

[0093] The final causal structure model is as follows:

[0094]

[0095] in, Continuous parameters representing the predictive probability distribution and correlation degree of the relationship between keypoint pairs; An embedded representation of key points from time 1 to T, o m,i Let i be the information of the i-th key point, and j be the information of the j-th key point.

[0096] like Figure 6 As shown, the overall causal structure model is F. R This module, later used in conjunction with external actions, predicts how interactions and dynamics affect objects. The graph neural network diagram illustrates this model, which combines objects and relationships...<i,j,> As input, their correlations are calculated using a relational model. Then, the correlations are aggregated and combined with object and external influences, serving as input to an object model to predict how interactions and dynamics affect objects. This graph neural network model primarily consists of two multilayer perceptrons (MLPs): fR and fO. fR has four hidden layers of length 150 and an output length of 50, while fO has only one hidden layer of length 100 and an output length of 2, used to predict velocities in the x and y directions in the dynamic module. The model normalizes the inputs.

[0097] Specifically, step 103 may include:

[0098] To accurately predict the future state corresponding to each candidate action, this embodiment uses a graph recurrent network (GRN). As a dynamic network, it takes a causal structure model, the current environment state, and all encoded candidate actions as input to the graph recurrent network, and outputs the predicted keypoints (predicted positions) corresponding to each candidate action in the next time step. To avoid noise causing inaccurate prediction results, this embodiment uses a multivariate Gaussian distribution to represent the position in future steps and predicts the mean and covariance matrix of the next state of each keypoint.

[0099]

[0100] In the formula, Represents the entire sequence of key points from time 1 to T, Om,i (This represents the information of the i-th key point). It represents the probability distribution and correlation degree of the relationship between key point pairs as continuous parameters.

[0101] When performing step 104, the specific steps may include:

[0102] Based on the predicted position of the target object under each candidate action at the next moment and the target position of the target object, the execution action of the target robot at the next moment is selected from each candidate action, and the predicted position corresponding to the execution action of the target robot at the next moment is updated to the current position. Then, the process returns to the step: inputting the operation task and the environmental state into the causal structure model to determine each candidate action of the target robot to push the target object from the current position to the next position, until the target object reaches the target position.

[0103] After the agent acquires the ability to predict future states, this embodiment allows the agent to perform reinforcement learning training under self-supervision. This embodiment defines the state space of the environment as S = {x, y, θ}, where (x, y) ∈ R. 2 Represents the position of an object on a plane (e.g., floor or tabletop), and θ∈[-π,π] is the object's orientation or rotation angle around an axis perpendicular to the plane.

[0104] Specifically, the environmental state of the target robot includes the two-dimensional coordinates of the target object on its plane and the rotation angle of the target object along the normal axis of the plane.

[0105] This embodiment focuses on pushing an object from its initial state s0 to its target position s0. g The task is judged as successful by checking if the coverage ratio between the building blocks and the target location is greater than a predefined threshold. Where Ω g This represents the set of all completed task states, where δx, δy, and δθ are the threshold numbers representing the distance from the target location.

[0106] Specifically, the action with the smallest L2 distance to the target state among all predicted next states is selected. To avoid noise causing inaccurate predictions, this embodiment uses a multivariate Gaussian distribution to represent the positions in future steps, predicting the mean and covariance matrix of the next state for each keypoint. The formulas are as follows:

[0107] Ω g :={s∈S||xx g |≤δ x ,|yy g |≤δ y ,|θ-θ g |≤δθ}

[0108] Among them, Ω g Here, δx represents the predicted position of the target object under each of the candidate actions, δy represents the threshold distance from the target position along the x-axis, δθ represents the threshold distance along the y-axis, and δθ represents the rotation angle threshold. g Let y be the x-coordinate of the target object. g Let θ be the target ordinate of the target object. g Let θ be the target rotation angle of the target object, x be the x-coordinate of the target object at the current moment, y be the y-coordinate of the target object at the current moment, and θ be the rotation angle of the target object at the current moment.

[0109] Furthermore, in the classic reinforcement learning ε-greedy action selection strategy, the agent randomly selects an action with probability ε, and with probability 1-ε, the agent uses its current knowledge to select the best action. In this way, the agent achieves a balance between exploitation and exploration. This paper proposes an improved greedy strategy that, before the agent randomly selects an action, uses counterfactual predictions of all future states that could potentially execute the action to select the optimal action. A set of actions is defined, where each element represents an action that can be performed. By querying the causal model, this embodiment aims to determine which action will bring the agent closer to the goal. By querying the causal model, this embodiment benefits from narrowing the possible action space, gradually transferring prior knowledge from the causal structure, and rapidly "downloading" optimal experience into the agent.

[0110] In one optional implementation provided in this embodiment, to evaluate the effectiveness of combining causal structures with the classic greedy DQN algorithm, this embodiment compared it with other algorithms in the same environment. It was found that the DQN network trained based on counterfactual methods, even with only 10% of the number of training rounds and 50% of the training time, could complete the goal-oriented task with a higher success rate. This demonstrates that causal structures can help neural networks learn more effectively from incomplete physical information and imperfect data, thereby shortening training time and achieving better results.

[0111] Example 2

[0112] like Figure 7 As shown, the present invention provides a robot intelligent control system based on a causal mechanism, comprising:

[0113] The acquisition module 701 is used to acquire the current operation task and environmental state of the target robot; the operation task is the task of pushing the target object from the initial position to the target position, and the environmental state is the environment in which the target robot is located.

[0114] The calculation module 702 is used to input the operation task and the environmental state into the causal structure model to determine the candidate actions of the target robot to push the target object from the current position to the next position. The causal structure model is a model trained based on a graph neural network and based on historical non-target-oriented data. The historical non-target-oriented data includes the position information of the target object before the action is performed, the position information of the target object after the action is performed, and the action performed by the target robot when the target robot performs the historical operation task.

[0115] The action prediction module 703 is used to input the candidate action into a graph recurrent network for each candidate action and obtain the predicted position of the target object under the candidate action at the next time step based on a multivariate Gaussian distribution.

[0116] The filtering module 704 is used to filter out the target robot's execution action from the candidate actions based on the predicted position of the target object under each candidate action at the next moment and the target position of the target object, update the predicted position corresponding to the target robot's execution action at the next moment to the current position, and return to the step: inputting the operation task and the environmental state into the causal structure model to determine each candidate action of the target robot to push the target object from the current position to the next position, until the target object reaches the target position.

[0117] The model construction module 702 specifically includes:

[0118] An information acquisition unit is used to acquire key point information of the target object; the key point information includes a vertex set and an edge set; the vertex set is the encoded information of the key points, and the edge set is the directed relationship from one key point to another.

[0119] The key point sequence unit is used to determine the key point sequence of the target object's key points when the target robot performs a historical operation task, based on the target object's position information before the action is performed and the target object's position information after the action is performed in the historical non-target guidance data; the key point sequence includes the key point information of the target object at each moment.

[0120] The first graph neural network unit is used to input the key point sequence into the first graph neural network to obtain the key points and key point pairs of the target object; the key point pairs are determined based on the edge set in the key point information.

[0121] The second graph neural network unit is used to input the embedded representation of the key points and key point pairs into the second graph neural network to obtain the predicted probability distribution of the key point pair relationship type.

[0122] The third graph neural network unit is used to input the predicted probability distribution of the key point pair relationship type into the third graph neural network to obtain continuous parameters of the correlation between the two key points in each key point pair.

[0123] The model building unit is used to obtain the causal structure model based on the embedding representation of the key points and key point pairs, the predicted probability distribution of the relationship type of the key point pairs, and the continuous parameters of the correlation between the two key points in the key point pairs.

[0124] In summary, the present invention has the following effects:

[0125] This invention draws on the advantages of both reinforcement learning and imitation learning to propose a causal reinforcement learning algorithm. It focuses on obtaining a structural causal model (SCM) through interaction with the environment via intervention, and counterfactually infers what would happen if the agent performed different actions in reinforcement learning based on the learned causal relationships. Considering that environmental variables in a reinforcement learning environment cannot be directly intervened, this invention employs a sub-goal-based policy to obtain intervention data. Benefiting from a variable-based goal space, sub-goal-based policies can effectively change the distribution of controllable variables. Finally, by proposing this causal reinforcement learning algorithm, this invention aims to promote research on causal structure learning and endow AI agents with the ability to reason causally to complete goal-oriented tasks. This invention develops a reinforcement learning-based method that introduces causal knowledge in the form of a directed acyclic graph. This method can be used to place learned goal-conditional policies within a context to perform tasks in new environments with potential causal structures. This invention progressively generates a causal graph by intervening in the interaction between the agent and the environment and selectively uses inductive graphs to determine actions. Experiments of this invention demonstrate that the method of this invention not only achieves higher initiation rewards and stability, but also effectively extends to completing new tasks in new environments with previously unseen causal structures.

[0126] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the systems disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the descriptions are relatively simple; relevant parts can be referred to the method section.

[0127] This document uses specific examples to illustrate the principles and implementation methods of the present invention. The descriptions of the above embodiments are only for the purpose of helping to understand the method and core ideas of the present invention. Furthermore, those skilled in the art will recognize that, based on the ideas of the present invention, there will be changes in the specific implementation methods and application scope. Therefore, the content of this specification should not be construed as a limitation of the present invention.

Claims

1. A robot intelligent control method based on a causal mechanism, characterized in that, include: Obtain the target robot's current operational tasks and environmental status; The operation task is to push the target object from its initial position to its target position, and the environmental state is the environment in which the target robot is located; The operation task and the environmental state are input into the causal structure model to determine the candidate actions for the target robot to push the target object from the current position to the next position. The causal structure model is a model trained based on a graph neural network and trained on historical non-target-oriented data. The historical non-target-oriented data includes the position information of the target object before the action is performed, the position information of the target object after the action is performed, and the action performed by the target robot when the target robot performs the historical operation task. For each candidate action, the candidate action is input into a graph recurrent network, and the predicted position of the target object under the candidate action at the next time step is obtained based on a multivariate Gaussian distribution. Based on the predicted position of the target object under each of the candidate actions at the next moment and the target position of the target object, the execution action of the target robot at the next moment is selected from each of the candidate actions, and the predicted position corresponding to the execution action of the target robot at the next moment is updated to the current position, and the process returns to the step: inputting the operation task and the environmental state into the causal structure model to determine each candidate action of the target robot to push the target object from the current position to the next position, until the target object reaches the target position; The process of determining the causal structure model is as follows: Obtain key point information of the target object; the key point information includes a vertex set and an edge set; the vertex set is the encoding information of each executable action point of the target object, and the edge set is the directed relationship from one executable action point to another. Based on the position information of the target object before the action is performed and the position information of the target object after the action is performed in the historical non-target guidance data, the key point sequence of the target object when the target robot performs the historical operation task is determined; the key point sequence includes the key point information of the target object at each moment; The key point sequence is input into the first graph neural network to obtain the key points and key point pairs of the target object. The key point pairs are determined based on the edge set in the key point information; the key points are the executable action points of the target object. The embedded representations of the key points and key point pairs are input into the second graph neural network to obtain the predicted probability distribution of the relationship type of the key point pairs; The predicted probability distribution of the relationship type of the key point pair is input into the third graph neural network to obtain continuous parameters of the degree of correlation between the two key points in each key point pair. The causal structure model is obtained based on the embedding representation of the key points and key point pairs, the predicted probability distribution of the relationship type of the key point pairs, and the continuous parameters of the correlation between the two key points in the key point pairs. The keypoint sequence is input into the first graph neural network to obtain the keypoints and keypoint pairs of the target object, as follows: ； in, These are the embedding representations of keypoints and keypoint pairs, respectively. This is the first graph neural network. For edge set, For key point sequence, This refers to the information of the i-th key point; The embedded representations of the key points and key point pairs are input into the second graph neural network to obtain the predicted probability distribution of the key point pair relationship type, as follows: ； in, The probability distribution for predicting the relationship type of key points. and ; For the embedding representation of key point i, For the embedding representation of key point j, Embed a representation of the relationship between keypoint i and keypoint j. This is the second neural network; The predicted probability distribution of the keypoint pair relationship type is input into the third graph neural network to obtain continuous parameters of the correlation between the two keypoints in each keypoint pair, as follows: ； in, This is a continuous parameter representing the degree of correlation between the two key points in the keypoint pair. , This is a third-graph neural network; The causal structure model is as follows: ； in, , Continuous parameters representing the predictive probability distribution and correlation degree of the relationship between keypoint pairs; An embedded representation of key points from time 1 to T. This represents the information of the j-th key point.

2. The robot intelligent control method based on causal mechanism according to claim 1, characterized in that, The environmental state of the target robot specifically includes the two-dimensional coordinates of the target object on its plane and the rotation angle of the target object along the normal axis of the plane.

3. The robot intelligent control method based on causal mechanism according to claim 1, characterized in that, Based on the predicted position of the target object and the target position of the target object under each of the candidate actions at the next moment, the execution action of the target robot at the next moment is selected from each of the candidate actions, specifically including: According to the formula Determine the action that the target robot needs to perform from among the candidate actions; in, Here, δx represents the predicted position of the target object under each of the candidate actions, and δy represents the threshold value of the distance from the target position along the x-axis and the threshold value of the distance from the target position along the y-axis. δθ is the rotation angle threshold; The x-coordinate of the target object. The target's ordinate is the vertical coordinate of the target object. Let θ be the target rotation angle of the target object, x be the x-coordinate of the target object at the current moment, y be the y-coordinate of the target object at the current moment, and θ be the rotation angle of the target object at the current moment.

4. A robot intelligent control system based on a causal mechanism, used to implement the robot intelligent control method based on a causal mechanism as described in claim 1, characterized in that, include: The acquisition module is used to acquire the target robot's current operational tasks and environmental status. The operation task is to push the target object from its initial position to its target position, and the environmental state is the environment in which the target robot is located; The calculation module is used to input the operation task and the environmental state into the causal structure model to determine the candidate actions of the target robot to push the target object from the current position to the next position. The causal structure model is a model trained based on a graph neural network and trained on historical non-target-oriented data. The historical non-target-oriented data includes the position information of the target object before the action is performed, the position information of the target object after the action is performed, and the action performed by the target robot when the target robot performs the historical operation task. The action prediction module is used to input the candidate action into a graph recurrent network for each candidate action and obtain the predicted position of the target object under the candidate action at the next time step based on a multivariate Gaussian distribution. The filtering module is used to filter out the target robot's execution action in the next moment from the candidate actions based on the predicted position of the target object under each candidate action in the next moment and the target position of the target object, update the predicted position corresponding to the target robot's execution action in the next moment to the current position, and return to the step: inputting the operation task and the environmental state into the causal structure model to determine each candidate action of the target robot to push the target object from the current position to the next position, until the target object reaches the target position.

Images