Humanoid robot control method and system based on reinforcement learning

Abstract

The invention provides a humanoid robot control method and system based on reinforcement learning, and the method comprises the steps: carrying out the path planning of a humanoid robot through a robot control model according to the current position, stacking task and residual energy of the humanoid robot, and obtaining an optimal path, executing control processing of the humanoid robot in the target stacking area according to the optimal path; the robot control model is obtained by training through a GNN graph neural network, reinforcement learning and a Monte Carlo tree search method. The method has the beneficial effects that the carrying efficiency of the humanoid robot is improved, the energy consumption is reduced, and the flexibility of path planning is enhanced.

Description

Technical Field

[0001] This invention relates to the field of robotics, and in particular to a humanoid robot control method and system based on reinforcement learning. Background Technology

[0002] In modern logistics and warehousing management, humanoid robots are playing an increasingly important role. In a scenario where multiple palletizing robots are sorting materials to be palletized, their handling behavior mainly includes the grasping action of items on the materials to be palletized, path movement, and placement at the target palletizing location. Existing technologies mainly predict the optimal path and optimal placement action through simulation combined with reinforcement learning. However, existing technologies have the following drawbacks: reinforcement learning has high computational complexity when the state space and action space are large, which may lead to insufficient real-time performance, especially in dynamic environments; long training time, reinforcement learning usually requires a large amount of training data and time to converge, especially in complex tasks, which may lead to extended deployment cycles; sensitivity to environmental changes, reinforcement learning models may be optimized for specific environments during training, and if the environment changes, the experience gained by a single robot through its experience pool needs to be re-accumulated, resulting in poor adaptability; and the lack of consideration for the placement height of different materials to be sorted and the stacking height of the palletizing points leads to additional energy consumption by the humanoid robots. Summary of the Invention

[0003] Aimed at at least in solving one of the technical problems existing in the prior art, the present invention provides a humanoid robot control method and system based on reinforcement learning, which improves the handling efficiency of humanoid robots and reduces handling energy consumption.

[0004] One aspect of the present invention provides a humanoid robot control method based on reinforcement learning, comprising: Based on the current position of the humanoid robot, the palletizing task, and the remaining energy, a robot control model is used to plan the path of the humanoid robot to obtain the optimal path. The control processing of the humanoid robot in the target palletizing area is then executed according to the optimal path. The robot control model is obtained through the following steps: Based on the height constraints of the sorting and stacking points, the task completion time constraints, and the path conflict constraints, the robot's handling constraints are determined, and the objective function is determined based on the handling constraints. Acquire training data, construct node and edge features of GNN graph neural network using the training data, and obtain embedded representations by performing graph convolution on the node and edge features of GNN graph neural network; Reinforcement learning is performed based on the embedded representation and the objective function to obtain the first optimal prediction path; The Monte Carlo tree search method is used to infer the optimal transport path for the first predicted path to obtain the second optimal predicted path. The robot control model is obtained by performing a preset number of training sessions.

[0005] According to the reinforcement learning-based humanoid robot control method, the robot's handling constraints are determined based on the height constraints of the sorting and stacking points, the task completion time constraints, and the path conflict constraints. The objective function is then determined based on these handling constraints, including: The height constraint is the maximum allowable stacking height of the classified stacking point; the time constraint is the maximum allowable handling time of the material pile to be transported; the path conflict constraint is the minimum allowable safe distance between humanoid robots. The objective function is determined based on the aforementioned handling constraints, where the objective function is the minimum energy consumption and minimum handling time for the humanoid robot to handle, classify, and place the material pile to be handled. Represented as: in, , , and This refers to the energy consumption coefficient during the movement process. The humanoid robot moves from its current position to the pile of materials to be transported. This refers to the path length of the humanoid robot from the pile of materials to be transported to the sorting and stacking point. For path energy consumption, Energy consumption weighting coefficient For time weighting coefficients, For humanoid robots, The energy consumption of humanoid robots grasping materials in a pile awaiting transport. Energy consumption for lifting materials.

[0006] According to the reinforcement learning-based humanoid robot control method, training data is acquired, node features and edge features of a GNN (Graph Neural Network) are constructed using the training data, and the node features and edge features are processed by graph convolution of the GNN to obtain an embedded representation, including: The training data samples include a material handling task graph, which consists of nodes and edges. Nodes include node features, and edges include edge features. Node features include the coordinates of the material pile to be handled, the maximum allowable handling time of the material pile to be handled, the type of material pile, the coordinates of the material pile, height constraints, and the current stack height. Edge features include path distance, path energy consumption, and path time. The embedding representation is obtained by performing graph convolution processing through multiple graph convolutional layers of a GNN (Graph Neural Network), where the graph convolution processing includes: in, and For the first Layer node representation and the first Layer node representation, For the ReLU activation function, the adjacency matrix... This represents the weight of the edges between nodes. is a trainable weight matrix.

[0007] According to the reinforcement learning-based humanoid robot control method described above, the training data includes: A handling task map is constructed using at least one of digital simulation, historical task data, and logistics simulation data, and the training data is determined using the handling task map.

[0008] According to the reinforcement learning-based humanoid robot control method, the reinforcement learning training based on the embedded representation and the objective function to obtain a first optimal prediction path includes: Using the embedded representation as the environment state, a greedy strategy is employed to select adjacent nodes or edges; The reward value is calculated using the objective function and objective constraints. Based on the reward value, the Q value or strategy is updated using TD learning. The reinforcement learning training is repeated to obtain the first optimal prediction path. The Q value represents the value of the action selected by the humanoid robot at each node, and the actions include grasping materials, lifting materials, and transporting and moving.

[0009] According to the reinforcement learning-based humanoid robot control method, the first predicted path is inferred using a Monte Carlo tree search method to obtain a second optimal predicted path, including: Obtain the environmental state and action of each node in the first predicted path, and use the Monte Carlo tree search method to calculate the maximum upper bound confidence of the child nodes of each node; The optimal child node is determined based on the maximum upper limit confidence score. Based on the optimal child node, the node is expanded. Based on the expanded optimal child node, candidate prediction paths are generated in a random manner. The statistical information calculated for the candidate prediction path using the objective function and objective constraints includes the total reward value and number of visits for each node in the candidate prediction path. The reward estimate is backpropagated along the candidate prediction path, and the reward estimate of the Monte Carlo tree is updated. Based on the reward estimation, the Q-values ​​of the nodes in the first optimal prediction path are updated to obtain the second optimal prediction path. The Q-value update method for the nodes is as follows: in, For nodes The updated Q value, The total reward value of the node in each simulation. This represents the number of times a point is visited.

[0010] According to the reinforcement learning-based humanoid robot control method, the method further includes: The system acquires the real-time status of the humanoid robot, the stack of materials to be transported, and the sorting stacking points. If any of the following changes occur in the real-time status: the humanoid robot's current energy is insufficient to complete one transport operation, the stacking height of the stack of materials to be transported changes, or the stacking height of the sorting stacking points changes, the optimal transport path of the humanoid robot in the target palletizing area is updated through the robot control model.

[0011] Embodiments of the present invention also include a humanoid robot control system based on reinforcement learning, comprising: The first module is used to plan the path of the humanoid robot based on its current position, palletizing task and remaining energy using a robot control model, obtain the optimal path, and execute the control processing of the humanoid robot in the target palletizing area according to the optimal path. The robot control model is obtained through the following modules: The second module is used to determine the robot's handling constraints based on the height constraints of the sorting and stacking points, the task completion time constraints, and the path conflict constraints, and to determine the objective function based on the handling constraints. The third module is used to acquire training data, construct node features and edge features of the GNN graph neural network through the training data, and obtain the embedded representation after graph convolution processing of the node features and edge features through the GNN graph neural network. The fourth module is used to perform reinforcement learning training based on the embedded representation and the objective function to obtain the first optimal prediction path; The fifth module is used to perform optimal transport path reasoning on the first predicted path using the Monte Carlo tree search method to obtain the second optimal predicted path; The sixth module is used to perform a preset number of training iterations to obtain the robot control model.

[0012] The beneficial effects of this invention are as follows: by comprehensively considering factors such as the robot's current position, task requirements, and remaining energy, the path is dynamically optimized using reinforcement learning and GNN graph neural networks; by analyzing constraints such as the height of the stacking point, task time, and path conflicts, an objective function is constructed to minimize energy consumption and handling time; reinforcement learning and Monte Carlo Tree Search (MCTS) are combined to further optimize path planning; at the same time, the system can update the path in real time to adapt to dynamic environmental changes and ensure efficient task completion; this invention has significant effects in improving handling efficiency, reducing energy consumption, and enhancing the flexibility of path planning. Attached Figure Description

[0013] Figure 1 This is a schematic diagram of the control process of a humanoid robot based on reinforcement learning according to an embodiment of the present invention.

[0014] Figure 2 This is a schematic diagram of a humanoid robot, a pile of materials to be transported, and a sorting and stacking point according to an embodiment of the present invention.

[0015] Figure 3 This is a schematic diagram of the first optimal prediction path processing flow in an embodiment of the present invention.

[0016] Figure 4 This is a schematic diagram of the Monte Carlo tree search process according to an embodiment of the present invention.

[0017] Figure 5 This is a schematic diagram of a humanoid robot control system based on reinforcement learning, according to an embodiment of the present invention. Detailed Implementation

[0018] The embodiments of the present invention are described in detail below, examples of which are shown in the accompanying drawings. Throughout the description, the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions. In the following description, suffixes such as "module," "part," or "unit" used to denote elements are used only for the purpose of illustrative purposes and have no specific meaning in themselves. Therefore, "module," "part," or "unit" can be used interchangeably. Terms such as "first," "second," etc., are used only to distinguish technical features and should not be construed as indicating or implying relative importance, or implicitly indicating the number of indicated technical features, or implicitly indicating the sequential relationship of the indicated technical features. In the following description, the consecutive reference numerals for method steps are for ease of review and understanding. Adjusting the implementation order of steps, in conjunction with the overall technical solution of the present invention and the logical relationship between the various steps, will not affect the technical effect achieved by the technical solution of the present invention. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and should not be construed as limiting the present invention.

[0019] refer to Figure 1 ,in Figure 1This is a flowchart illustrating a humanoid robot control method based on reinforcement learning, which includes, but is not limited to, steps S100~S600: S100 uses a robot control model to plan the path of the humanoid robot based on its current position, palletizing task and remaining energy, and obtains the optimal path. Then, it executes the control processing of the humanoid robot in the target palletizing area according to the optimal path.

[0020] In some embodiments, reference Figure 2 The diagram shows a humanoid robot, a pile of materials to be transported, and sorting and stacking points. It includes multiple humanoid robots, multiple piles of materials to be transported, and multiple sorting and stacking points. The path of the humanoid robot to transport materials from the pile of materials to be transported B2-9 to the sorting and stacking point A1-16 is schematically shown as d1 and d2. d1 is the distance from the humanoid robot's current position to the pile of materials to be transported B2-9, and d2 is the distance from the pile of materials to be transported B2-9 to the sorting and stacking point A1-16.

[0021] Understandably, in actual path planning, it is not a straight-line planning, but rather a path point is planned at intervals.

[0022] The robot control model is obtained through steps S200~S600, including: S200 determines the robot's handling constraints based on the height constraints of the sorting and stacking points, the task completion time constraints, and the path conflict constraints, and then determines the objective function based on the handling constraints.

[0023] In some embodiments, the height constraint is the maximum allowable stacking height of the sorting and stacking point; the time constraint is the maximum allowable handling time of the material pile to be transported; and the path conflict constraint is the minimum allowable safe distance between humanoid robots.

[0024] In some embodiments, the determination of transport constraints and objective functions includes: The objective function is determined based on the handling constraints, where the objective function is the minimum energy consumption and minimum handling time for the humanoid robot to handle, classify, and place the material pile to be handled. Represented as: in, , , and This refers to the energy consumption coefficient during the movement process. The humanoid robot moves from its current position to the pile of materials to be transported. This refers to the path length of the humanoid robot from the pile of materials to be transported to the sorting and stacking point. For path energy consumption, Energy consumption weighting coefficient For time weighting coefficients, For humanoid robots, The energy consumption of humanoid robots grasping materials in a pile awaiting transport. Energy consumption for lifting materials.

[0025] S300: Obtain training data, construct node features and edge features of GNN graph neural network through training data, and obtain embedded representations by performing graph convolution on node features and edge features through GNN graph neural network.

[0026] The training data samples include a material handling task graph, which consists of nodes and edges. Nodes include node features, and edges include edge features. Node features include the coordinates of the material pile to be handled, the maximum allowable handling time of the material pile to be handled, the type of material pile, the coordinates of the material pile, height constraints, and the current stack height. Edge features include path distance, path energy consumption, and path time. Graph convolution processing is performed through multiple graph convolutional layers of a GNN (Graph Neural Network) to obtain the embedding representation (of nodes). The graph convolution processing includes: in, and For the first Layer node representation and the first Layer node representation, For the ReLU activation function, the adjacency matrix... This represents the weight of the edges between nodes. is a trainable weight matrix.

[0027] In some embodiments, each node (such as a material stacking point and a sorting placement point) will receive a high-dimensional feature vector representing the information of that node in the path planning graph. For example, for a sorting stacking point node, its embedding will include a representation of information such as the location of the stacking point, stacking height, and time constraints.

[0028] In some embodiments, the training data samples are obtained using at least one of digital simulation, historical task data, and logistics simulation data, wherein: Digital simulation involves creating a digital twin or warehousing / handling simulation environment, setting multiple material stacking points and multiple classification placement points within the simulation environment, and randomly generating different task scenarios. It uses standard shortest path algorithms or heuristic scheduling algorithms to generate "near-optimal" paths as labels. This can automatically generate a large amount of data and flexibly adjust constraints (height, time, safety distance, etc.).

[0029] Historical tasks involve collecting historical warehousing / handling task data (robot execution path, energy consumption, time, whether violations occurred, etc.) in factories or warehousing systems; this historical handling task data serves as a training / validation set.

[0030] Logistics simulation data consists of publicly available datasets for some warehouse robots or multi-robot path planning datasets. It should be noted that the publicly available datasets for warehouse robots or multi-robot path planning datasets are datasets for specific scenarios and differ from the application scenarios of this application. They are only used for auxiliary purposes (such as comparison, effect verification, and pre-training).

[0031] S400, based on the embedding representation and objective function, performs reinforcement learning training to obtain the first optimal prediction path.

[0032] In some embodiments, reference Figure 3 The schematic diagram of the first optimal prediction path processing flow shown includes, but is not limited to, steps S410~S420: S410 uses an embedded representation as the environment state and employs a greedy strategy to select adjacent nodes or edges. S420 calculates the reward value through the objective function and objective constraints, updates the Q value or strategy using TD learning based on the reward value, and repeatedly executes reinforcement learning training to obtain the first optimal prediction path.

[0033] In some embodiments, the Q value represents the value of the action selected by the humanoid robot at each node, wherein the action includes grasping materials, lifting materials, and transporting.

[0034] It is understandable that the value of the action selected by each node can refer to the energy consumed when grabbing, lifting, and transporting materials and moving them.

[0035] S500, the Monte Carlo tree search method is used to infer the optimal transport path for the first predicted path, and the second optimal predicted path is obtained.

[0036] refer to Figure 4 The Monte Carlo tree search process diagram shown includes, but is not limited to, steps S510~S550: S510: Obtain the environmental state and action of each node in the first prediction path, and use the Monte Carlo tree search method to calculate the maximum upper limit confidence of the child nodes of each node; S520 determines the optimal child node based on the maximum upper limit confidence score, expands the node based on the optimal child node, and generates candidate prediction paths in a random manner based on the expanded optimal child node. S530, Statistical information of candidate prediction paths calculated by objective function and objective constraints, wherein the statistical information includes the total reward value and number of visits for each node in the candidate prediction path; S540, the reward estimate is backpropagated along the candidate prediction path and the reward estimate of the Monte Carlo tree is updated. S550, based on the reward estimation, updates the Q-values ​​of the nodes in the first optimal prediction path to obtain the second optimal prediction path.

[0037] The Q-value of the node is updated in the following way: in, For nodes The updated Q value, The total reward value of the node in each simulation. This represents the number of times a point is visited.

[0038] In some embodiments, Monte Carlo Tree Search (MCTS) includes selection, expansion, simulation, and return, specifically: (1) Selection: MCTS selects a child node to expand from the current node based on the Upper Confidence Bound (UCB) strategy of the current tree. This selection not only depends on the historical information of the path (such as energy consumption, time, etc.), but also combines the node embedding information provided by GNN.

[0039] In some embodiments, for a state and optional actions MCTS selects the child node with the largest UCB1. : in, For is a node The average reward value is calculated using the embedding representation of the GNN graph neural network, which represents the path cost and reward. This represents the number of times the current node has been visited. It is a child node Number of visits For exploration factors.

[0040] In some embodiments, assuming the humanoid robot is currently at node P1, it can choose two directions (edges) P1→P2 or P1→P3; MCTS will calculate the UCB value of these two directions based on historical experience and the cost evaluation provided by GNN, and select the best child node, such as P1→P3.

[0041] (2) Expansion: When the selected node is not fully expanded, MCTS will expand it, that is, generate a new child node under the current node. This new node corresponds to an unexplored action (i.e. an unselected path starting from the current node). Through expansion, the tree structure gradually increases and more paths are explored.

[0042] (3) Simulation: After expanding the node, MCTS will start from the new node and perform a series of random simulations until the target node is reached or the maximum number of steps is exceeded. In the path planning task, the simulation can be performed by randomly selecting the path (e.g., randomly selecting adjacent points near the current node). When the simulation ends, the total cost from the current node to the target node (e.g., time, energy consumption, etc.) is recorded as an estimate of the path.

[0043] (4) Backhaul: After the simulation is completed, MCTS will backhaul the reward information of the current path, passing it from the leaf node all the way to the root node, and updating the statistics of each node in the tree, including: the total reward value of the node (e.g., path cost) and the number of times the node is visited; the backhaul process can help MCTS evaluate the path quality of all nodes and update their reward estimates so as to make better decisions in the next round of selection.

[0044] It should be noted that the reason for using Monte Carlo tree search is to avoid local optima caused by the greedy strategy of reinforcement learning. At the same time, reinforcement learning can use Monte Carlo tree search to avoid slow convergence speed and system resource consumption caused by over-reliance on random simulation. Through this collaborative approach, the optimal path for the humanoid robot can be obtained with maximum efficiency.

[0045] In some embodiments, the system further includes acquiring the real-time status of the humanoid robot, the pile of materials to be transported, and the sorting and stacking points. If any of the following changes occur in the real-time status: the humanoid robot's current energy is insufficient to complete one transport operation, the height of the pile of materials to be transported changes, or the height of the sorting and stacking points changes, the optimal transport path of the humanoid robot in the target palletizing area is updated through the robot control model.

[0046] The humanoid robot obtains its real-time status through built-in chips and sensors, while the height of the material pile to be transported is monitored in real time by sensors installed on the top of the pile to monitor the height of each material (usually a regular matrix).

[0047] S600 performs a preset number of training iterations to obtain the robot control model.

[0048] In some embodiments, training can be stopped when the model parameters are stable during training to obtain the robot control model.

[0049] Figure 5This is a schematic diagram of a humanoid robot control system based on reinforcement learning according to an embodiment of the present invention. The device includes a first module 510, a second module 520, a third module 530, a fourth module 540, and a fifth module 550.

[0050] The system comprises six modules: The first module plans the path of the humanoid robot based on its current position, palletizing task, and remaining energy using a robot control model, obtaining the optimal path and executing control processing for the target palletizing area. The robot control model is obtained through the following modules: The second module determines the robot's handling constraints based on the height constraints of the sorting and stacking points, task completion time constraints, and path conflict constraints, and determines the objective function based on these constraints; The third module acquires training data, constructs node and edge features of a GNN (Graph Neural Network) using the training data, and performs graph convolution processing on the node and edge features to obtain embedded representations; The fourth module performs reinforcement learning training based on the embedded representations and the objective function to obtain a first optimal predicted path; The fifth module uses a Monte Carlo tree search method to infer the optimal handling path from the first predicted path to obtain a second optimal predicted path; The sixth module performs a preset number of training iterations to obtain the robot control model.

[0051] For example, with the cooperation of the first, second, third, fourth, fifth, and sixth modules in the device, the embodiment device can implement any of the aforementioned reinforcement learning-based humanoid robot control methods. Specifically, based on the humanoid robot's current position, palletizing task, and remaining energy, a robot control model is used to plan the humanoid robot's path to obtain an optimal path. Control processing of the humanoid robot in the target palletizing area is then executed according to the optimal path. The robot control model is obtained through the following steps: determining the robot's handling constraints based on the height constraints of the sorting and stacking points, the task completion time constraints, and the path conflict constraints; determining the objective function based on the handling constraints; acquiring training data; constructing node and edge features of a GNN graph neural network using the training data; performing graph convolution processing on the node and edge features through the GNN graph neural network to obtain an embedded representation; performing reinforcement learning training based on the embedded representation and the objective function to obtain a first optimal predicted path; using a Monte Carlo tree search method to infer the optimal handling path from the first predicted path to obtain a second optimal predicted path; and performing training a preset number of times to obtain the robot control model. The beneficial effects of this invention are as follows: by comprehensively considering factors such as the robot's current position, task requirements, and remaining energy, the path is dynamically optimized using reinforcement learning and GNN graph neural networks; by analyzing constraints such as the height of the stacking point, task time, and path conflicts, an objective function is constructed to minimize energy consumption and handling time; reinforcement learning and Monte Carlo Tree Search (MCTS) are combined to further optimize path planning; at the same time, the system can update the path in real time to adapt to dynamic environmental changes and ensure efficient task completion; this invention has significant effects in improving handling efficiency, reducing energy consumption, and enhancing the flexibility of path planning.

[0052] This invention also provides an electronic device, which includes a processor and a memory; The memory stores the program; The processor executes a program to perform the aforementioned reinforcement learning-based humanoid robot control method; the electronic device has the function of carrying and running the reinforcement learning-based humanoid robot control software system provided in the embodiments of the present invention, such as a personal computer, minicomputer, mainframe, workstation, network or distributed computing environment, standalone or integrated computer platform, or communicating with charged particle tools or other imaging devices, etc.

[0053] This invention also provides a computer-readable storage medium storing a program that is executed by a processor to implement the reinforcement learning-based humanoid robot control method described above.

[0054] In some alternative embodiments, the functions / operations mentioned in the block diagrams may not occur in the order shown in the operation diagrams. For example, depending on the functions / operations involved, two consecutively shown blocks may actually be executed substantially simultaneously, or the blocks may sometimes be executed in reverse order. Furthermore, the embodiments presented and described in the flowcharts of this invention are provided by way of example to provide a more comprehensive understanding of the technology. The disclosed methods are not limited to the operations and logic flows presented in the embodiments of this invention. Alternative embodiments are contemplated, in which the order of various operations is changed and sub-operations described as part of a larger operation are executed independently.

[0055] This invention also discloses a computer program product or computer program, which includes computer instructions stored in a computer-readable storage medium. A processor of a computer device can read the computer instructions from the computer-readable storage medium and execute the computer instructions, causing the computer device to perform the aforementioned reinforcement learning-based humanoid robot control method.

[0056] Furthermore, although the invention has been described in the context of functional modules, it should be understood that, unless otherwise stated, one or more of the described functions and / or features may be integrated into a single physical device and / or software module, or one or more functions and / or features may be implemented in a separate physical device or software module. It is also understood that a detailed discussion of the actual implementation of each module is unnecessary for understanding the invention. Rather, considering the properties, functions, and internal relationships of the various functional modules in the apparatus disclosed in the embodiments of the invention, the actual implementation of the module will be understood within the scope of conventional skill of an engineer. Therefore, those skilled in the art can implement the invention as set forth in the claims using ordinary techniques without excessive experimentation. It is also understood that the specific concepts disclosed are merely illustrative and are not intended to limit the scope of the invention, which is determined by the full scope of the appended claims and their equivalents.

[0057] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, essentially, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0058] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (such as a computer-based system, a processor-included system, or other system that can fetch and execute instructions from, an instruction execution system, apparatus, or device). For the purposes of this specification, "computer-readable medium" can be any means that can include, store, communicate, propagate, or transmit programs for use by, or in conjunction with, an instruction execution system, apparatus, or device.

[0059] More specific examples of computer-readable media (a non-exhaustive list) include: electrical connections (electronic devices) having one or more wires, portable computer disk drives (magnetic devices), random access memory (RAM), read-only memory (ROM), erasable and editable read-only memory (EPROM or flash memory), fiber optic devices, and portable optical disc read-only memory (CDROM). Furthermore, computer-readable media can even be paper or other suitable media on which the program can be printed, because the program can be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, interpreting, or otherwise processing as necessary, and then stored in computer memory.

[0060] It should be understood that various parts of the present invention can be implemented in hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented in software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, it can be implemented using any one or a combination of the following techniques known in the art: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.

[0061] In the description of this specification, references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0062] Although embodiments of the invention have been shown and described, those skilled in the art will understand that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the claims and their equivalents.

[0063] The above is a detailed description of the preferred embodiments of the present invention, but the present invention is not limited to the embodiments described. Those skilled in the art can make various equivalent modifications or substitutions without departing from the spirit of the present invention, and these equivalent modifications or substitutions are all included within the scope defined by the claims of this application.

Claims

1. A humanoid robot control method based on reinforcement learning, characterized in that, include: Based on the current position of the humanoid robot, the palletizing task, and the remaining energy, a robot control model is used to plan the path of the humanoid robot to obtain the optimal path. The control processing of the humanoid robot in the target palletizing area is then executed according to the optimal path. The robot control model is obtained through the following steps: Based on the height constraints of the sorting and stacking points, the task completion time constraints, and the path conflict constraints, the robot's handling constraints are determined, and the objective function is determined based on the handling constraints. Acquire training data, construct node and edge features of GNN graph neural network using the training data, and obtain embedded representations by performing graph convolution on the node and edge features of GNN graph neural network; Reinforcement learning is performed based on the embedded representation and the objective function to obtain the first optimal prediction path; The Monte Carlo tree search method is used to infer the optimal transport path for the first predicted path to obtain the second optimal predicted path. The robot control model is obtained by performing a preset number of training sessions.

2. The humanoid robot control method based on reinforcement learning according to claim 1, characterized in that, The robot's handling constraints are determined based on the height constraints of the sorting and stacking points, the task completion time constraints, and the path conflict constraints. The objective function is then determined based on these handling constraints, including: The height constraint is the maximum allowable stacking height of the classified stacking point; the time constraint is the maximum allowable handling time of the material pile to be transported; the path conflict constraint is the minimum allowable safe distance between humanoid robots. The objective function is determined based on the aforementioned handling constraints, where the objective function is the minimum energy consumption and minimum handling time for the humanoid robot to handle, classify, and place the material pile to be handled. Represented as: in, , , and This refers to the energy consumption coefficient during the movement process. The humanoid robot moves from its current position to the pile of materials to be transported. This refers to the path length of the humanoid robot from the pile of materials to be transported to the sorting and stacking point. For path energy consumption, Energy consumption weighting coefficient For time weighting coefficients, For humanoid robots, The energy consumption of humanoid robots grasping materials in a pile awaiting transport. Energy consumption for lifting materials.

3. The humanoid robot control method based on reinforcement learning according to claim 2, characterized in that, The process of acquiring training data, constructing node and edge features of a GNN graph neural network using the training data, and obtaining embedded representations by performing graph convolution on the node and edge features through the GNN graph neural network includes: The training data samples include a material handling task graph, which consists of nodes and edges. Nodes include node features, and edges include edge features. Node features include the coordinates of the material pile to be handled, the maximum allowable handling time of the material pile to be handled, the type of material pile, the coordinates of the material pile, height constraints, and the current stack height. Edge features include path distance, path energy consumption, and path time. The embedding representation is obtained by performing graph convolution processing through multiple graph convolutional layers of a GNN (Graph Neural Network), where the graph convolution processing includes: in, and For the first Layer node representation and the first Layer node representation, For the ReLU activation function, the adjacency matrix... This represents the weight of the edges between nodes. is a trainable weight matrix.

4. The humanoid robot control method based on reinforcement learning according to claim 3, characterized in that, The training data includes: A handling task map is constructed using at least one of digital simulation, historical task data, and logistics simulation data, and the training data is determined using the handling task map.

5. The humanoid robot control method based on reinforcement learning according to claim 3, characterized in that, The step of performing reinforcement learning training based on the embedded representation and the objective function to obtain the first optimal prediction path includes: Using the embedded representation as the environment state, a greedy strategy is employed to select adjacent nodes or edges; The reward value is calculated using the objective function and objective constraints. Based on the reward value, the Q value or strategy is updated using TD learning. The reinforcement learning training is repeated to obtain the first optimal prediction path. The Q value represents the value of the action selected by the humanoid robot at each node, and the actions include grasping materials, lifting materials, and transporting and moving.

6. The humanoid robot control method based on reinforcement learning according to claim 5, characterized in that, The step of using the Monte Carlo tree search method to perform optimal transport path reasoning on the first predicted path to obtain the second optimal predicted path includes: Obtain the environmental state and action of each node in the first predicted path, and use the Monte Carlo tree search method to calculate the maximum upper bound confidence of the child nodes of each node; The optimal child node is determined based on the maximum upper limit confidence score. Based on the optimal child node, the node is expanded. Based on the expanded optimal child node, candidate prediction paths are generated in a random manner. The statistical information calculated for the candidate prediction path using the objective function and objective constraints includes the total reward value and number of visits for each node in the candidate prediction path. The reward estimate is backpropagated along the candidate prediction path, and the reward estimate of the Monte Carlo tree is updated. Based on the reward estimation, the Q-values ​​of the nodes in the first optimal prediction path are updated to obtain the second optimal prediction path. The Q-value update method for the nodes is as follows: in, For nodes The updated Q value, The total reward value of the node in each simulation. This represents the number of times a point is visited.

7. The humanoid robot control method based on reinforcement learning according to claim 1, characterized in that, The method further includes: The system acquires the real-time status of the humanoid robot, the stack of materials to be transported, and the sorting stacking points. If any of the following changes occur in the real-time status: the humanoid robot's current energy is insufficient to complete one transport operation, the stacking height of the stack of materials to be transported changes, or the stacking height of the sorting stacking points changes, the optimal transport path of the humanoid robot in the target palletizing area is updated through the robot control model.

8. A humanoid robot control system based on reinforcement learning, characterized in that, include: The first module is used to plan the path of the humanoid robot based on its current position, palletizing task and remaining energy using a robot control model, obtain the optimal path, and execute the control processing of the humanoid robot in the target palletizing area according to the optimal path. The robot control model is obtained through the following modules: The second module is used to determine the robot's handling constraints based on the height constraints of the sorting and stacking points, the task completion time constraints, and the path conflict constraints, and to determine the objective function based on the handling constraints. The third module is used to acquire training data, construct node features and edge features of the GNN graph neural network through the training data, and obtain the embedded representation after graph convolution processing of the node features and edge features through the GNN graph neural network. The fourth module is used to perform reinforcement learning training based on the embedded representation and the objective function to obtain the first optimal prediction path; The fifth module is used to perform optimal transport path reasoning on the first predicted path using the Monte Carlo tree search method to obtain the second optimal predicted path; The sixth module is used to perform a preset number of training iterations to obtain the robot control model.

Images