Training method for humanoid robot motion control model based on reinforcement learning, humanoid robot motion control method, device, equipment, humanoid robot and storage medium

Abstract

This disclosure provides a method for training a humanoid robot motion control model based on reinforcement learning, a humanoid robot motion control method, apparatus, device, humanoid robot, and storage medium, relating to the field of robotics technology. The method includes: randomly sampling from the joint space of the humanoid robot and calculating the pose data of the arm's end effector; dividing the pose data into multiple bins based on their positions; configuring initial sampling weights for each bin and adjusting the weights of bins in a specified region; sampling according to the target sampling weights to obtain an instruction dataset; and training the reinforcement learning-based humanoid robot motion control model using the instruction dataset. This disclosure improves the pose tracking accuracy of the control strategy at different spatial positions by adjusting the proportion of pose data in a specified region within the training samples, thus making the distribution of training samples in the workspace more balanced.

Description

Technical Field

[0001] This disclosure relates to the field of robotics technology, and to a method for training a humanoid robot motion control model based on reinforcement learning, a method for controlling the motion of a humanoid robot, a device, an equipment, a humanoid robot, and a storage medium. Background Technology

[0002] When performing reinforcement learning training on a humanoid robot with arm manipulation as the primary objective, it is necessary to use the pose data of several points within the reach of the arm end within the humanoid robot's workspace as the instruction dataset for training.

[0003] One method to obtain this instruction dataset is to randomly sample the positions of all or part of the joints of a humanoid robot (at least all arm joints), and then calculate the pose of the points within the reach of the arm end within the humanoid robot's workspace through forward kinematic fitting. The pose of these points is then used as instruction data, and the instruction dataset is constructed from multiple randomly obtained instruction data.

[0004] However, after training the policy network using the instruction dataset constructed in the above random manner, the applicant found that in the edge areas of the workspace of some humanoid robots, the policy network had poor training performance in terms of the control accuracy of the arm end pose.

[0005] The reason is that the way the above-mentioned instruction dataset is generated results in uneven distribution of instruction data in the workspace. The distribution of instruction data is relatively sparse in some workspaces, especially in the edge areas. This will cause the policy network to learn insufficiently in these areas, affecting the training effect and thus causing a decrease in pose tracking accuracy in these areas.

[0006] For example, the paper "An Adaptive Workspace Bias Method for Sampling Planners" Adaptive workspace biasing for sampling-based planners The method described in M ​​Zucker et al. (2009) divides the workspace into multiple cells and constructs a bias distribution based on features such as occupancy, visibility, and path distance. It then uses a policy gradient algorithm to progressively optimize the sampling bias in path planning tasks, thereby improving planning efficiency. While this method can adaptively adjust the sampling bias of the workspace, it primarily focuses on configuration space search in path planning, emphasizing planning efficiency optimization. It does not model the uneven distribution of the end-effector within the workspace, nor does it provide a mechanism for adjusting sample density in reinforcement learning training. Therefore, it still struggles to address the problem of reduced control accuracy caused by insufficient sampling of the end-effector pose in edge regions.

[0007] It should be noted that the information disclosed in the background section above is only used to enhance the understanding of the background of this disclosure, and therefore may include information that does not constitute prior art known to those skilled in the art. Summary of the Invention

[0008] This disclosure provides a reinforcement learning-based training method for humanoid robot motion control models, a humanoid robot motion control method, a device, an equipment, a humanoid robot, and a storage medium to overcome the problem in related technologies where uneven distribution of instruction data in the workspace leads to insufficient training samples at the edge region of the arm end, resulting in decreased pose tracking accuracy.

[0009] Additional aspects and advantages of this disclosure will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of this disclosure.

[0010] According to a first aspect of this disclosure, a method for training a humanoid robot motion control model based on reinforcement learning is provided, comprising:

[0011] Random sampling is performed on the joint space of the humanoid robot, and multiple pose data of the robot arm end are calculated based on the sampling results;

[0012] Based on the position of each pose data, the pose data is divided into multiple buckets, and each bucket corresponds to a set of pose data within a certain position range;

[0013] Configure initial sampling weights for each bucket and adjust the initial sampling weights of buckets whose location range is within the specified area to obtain the target sampling weights for each bucket;

[0014] Sampling is performed based on the target sampling weights of each bucket to obtain the instruction dataset, and the instruction dataset is then input into the humanoid robot motion control model based on reinforcement learning for training.

[0015] In one exemplary embodiment of this disclosure, adjusting the initial sampling weights of buckets whose location range is within a specified area includes:

[0016] Identify the first bucket whose location is within the edge region of the humanoid robot's workspace;

[0017] Add boundary sampling weights to the first bucket to obtain the target sampling weights for the first bucket.

[0018] In one exemplary embodiment of this disclosure, adding boundary sampling weights to the first bucket to obtain the target sampling weights for the first bucket includes:

[0019] The target sampling weight of the first bucket is calculated based on the initial sampling weight and the boundary sampling weight of the first bucket.

[0020] In one exemplary embodiment of this disclosure, the method further includes:

[0021] Sampling is performed based on the target sampling weights of the first bucket, and the humanoid robot motion control model is trained using the sampled instruction dataset.

[0022] The training performance score of the first bucket is calculated based on the evaluation results after training, and the training performance score is mapped to the performance compensation weight.

[0023] The performance compensation weight is added to the target sampling weight of the first bucket to obtain the final sampling weight of the first bucket, and resampling is performed based on the final sampling weight of the first bucket.

[0024] In one exemplary embodiment of this disclosure, the boundary sampling weight is a constant.

[0025] In one exemplary embodiment of this disclosure, the boundary sampling weights decrease as the training progresses.

[0026] In one exemplary embodiment of this disclosure, adjusting the initial sampling weights of buckets whose location range is within a specified area includes:

[0027] Identify the second bucket whose location is within the central area of ​​the humanoid robot's workspace;

[0028] Reduce the initial sampling weight of the second bucket to obtain the target sampling weight of the second bucket.

[0029] In one exemplary embodiment of this disclosure, adjusting the initial sampling weights of buckets whose location range is within a specified area includes:

[0030] Identify the third bucket, whose location range is within the target area of ​​the humanoid robot's workspace;

[0031] The initial sampling weights of the third bucket are adjusted according to the preset weight adjustment factor to obtain the target sampling weights of the third bucket.

[0032] The target area of ​​the humanoid robot's workspace includes at least one of the following: the spatial area corresponding to the robot's singular configuration, the spatial area corresponding to the task-specified operation position, or the spatial area where the training error meets preset conditions.

[0033] In one exemplary embodiment of this disclosure, random sampling is performed from the joint space of the humanoid robot, and multiple pose data of the robot arm end effector are calculated based on the sampling results, including:

[0034] Within the preset joint space sampling range, multiple joint angles of the humanoid robot are sampled to generate joint configurations corresponding to each joint angle.

[0035] Based on the configuration of each joint, multiple pose data of the robot arm end are calculated.

[0036] In one exemplary embodiment of this disclosure, multiple pose data of the robot arm end effector are calculated based on the configuration of each joint, including:

[0037] For each joint configuration, the position and orientation of the corresponding robot arm end effector are obtained through forward kinematics calculations;

[0038] Based on the position and orientation of the robot arm's end effector, multiple pose data of the robot arm's end effector are obtained.

[0039] In one exemplary embodiment of this disclosure, the pose data is divided into multiple buckets based on the position of each pose data point, including:

[0040] Based on the position of each pose data point, the pose data is divided into grids to obtain multiple buckets.

[0041] In one exemplary embodiment of this disclosure, the pose data is divided into meshes according to the position of each pose data point, including:

[0042] The humanoid robot's workspace is divided into multiple voxel regions according to the preset spatial division step size;

[0043] Based on the position of each pose data point, pose data located within the same voxel region are divided into the same bucket.

[0044] In one exemplary embodiment of this disclosure, the method further includes:

[0045] The spatial partitioning step size is adjusted according to the preset strategy;

[0046] The preset strategies include at least one of the following: changes during the training phase, humanoid robot task requirements, or workspace characteristics.

[0047] In one exemplary embodiment of this disclosure, the method further includes:

[0048] The number of pose data in each bin is counted to obtain the sample size of each bin;

[0049] The initial sampling weights for each bin are determined based on the preset coefficients and the number of samples in each bin.

[0050] In one exemplary embodiment of this disclosure, the initial sampling weight corresponding to each bin is determined based on a preset coefficient and the number of samples in each bin, including:

[0051]

[0052] in, The initial sampling weights are for each bucket. K For preset coefficients, This represents the number of samples in each bin. This represents the set of pose data in each bucket, each... Each corresponds to a separate bucket.

[0053] According to a second aspect of this disclosure, a motion control method for a humanoid robot is provided, comprising:

[0054] Acquire pose data of the robot arm's end effector;

[0055] The pose data is input into a pre-trained humanoid robot motion control model, and the action strategy for controlling the humanoid robot's motion is output based on the humanoid robot motion control model.

[0056] The humanoid robot motion control model is obtained by the reinforcement learning-based humanoid robot motion control model training method in the first aspect of this disclosure.

[0057] According to a third aspect of this disclosure, a training device for a humanoid robot motion control model based on reinforcement learning is provided, comprising:

[0058] The pose data calculation module is used to randomly sample from the joint space of the humanoid robot and calculate multiple pose data of the robot arm end based on the sampling results.

[0059] The pose data partitioning module is used to divide each pose data into multiple buckets based on the position of each pose data. Each bucket corresponds to a set of pose data within a certain position range.

[0060] The sampling weight adjustment module is used to configure the initial sampling weight for each bucket and adjust the initial sampling weight of the buckets whose location range is within the specified area to obtain the target sampling weight for each bucket.

[0061] The control model training module is used to sample based on the target sampling weights of each bucket to obtain the instruction dataset, and then input the instruction dataset into the reinforcement learning-based humanoid robot motion control model for training.

[0062] According to a fourth aspect of this disclosure, a motion control device for a humanoid robot is provided, comprising:

[0063] The pose data acquisition module is used to acquire pose data of the robot arm's end effector.

[0064] The motion strategy output module is used to input pose data into the pre-trained humanoid robot motion control model and output motion strategies to control the humanoid robot's motion based on the humanoid robot motion control model.

[0065] The humanoid robot motion control model is obtained according to the reinforcement learning-based humanoid robot motion control model training method in the first aspect of this disclosure.

[0066] According to a fifth aspect of this disclosure, an electronic device is provided, comprising:

[0067] Processor; and

[0068] A memory that stores computer-readable instructions, which, when executed by a processor, implement the method as described in the above embodiments.

[0069] According to a sixth aspect of this disclosure, a humanoid robot is provided, comprising:

[0070] Processor; and

[0071] A memory that stores computer-readable instructions, which, when executed by a processor, implement the method as described in the above embodiments.

[0072] According to a seventh aspect of this disclosure, a computer-readable storage medium is provided that stores computer program code instructions that, when invoked by a processor, cause the processor to perform the method as described in the above embodiments.

[0073] As can be seen from the above technical solution, this disclosure possesses at least one of the following advantages and positive effects:

[0074] This disclosure obtains pose data of the robot arm's end effector by randomly sampling from the joint space of the humanoid robot, providing an initial sample base covering different spatial positions for training. Furthermore, by constructing multiple buckets based on the positions of the pose data and configuring adjustable sampling weights for each bucket, the training process can allocate pose data from different position regions to participate in training according to the sampling weights of each bucket. By adjusting the sampling weights of the buckets corresponding to a specified region, the proportion of pose data in that region within the training samples can be adjusted without increasing the overall training data size, thereby balancing the spatial distribution density of training samples. Therefore, the finally trained humanoid robot motion control model can achieve a more balanced policy learning effect across the entire space, making the accuracy distribution of the control policy more consistent across different position regions and reducing the possibility of decreased pose tracking accuracy in specified regions. Attached Figure Description

[0075] To more clearly illustrate the technical solutions in the embodiments of this disclosure, the accompanying drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this disclosure. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0076] Figure 1 A system architecture diagram is shown that can be applied to the reinforcement learning-based humanoid robot motion control model training method and humanoid robot motion control method in the embodiments of this disclosure.

[0077] Figure 2 A flowchart illustrating a method for training a humanoid robot motion control model based on reinforcement learning, as described in an embodiment of this disclosure, is shown.

[0078] Figure 3 A schematic diagram of a process for calculating pose data is shown in an embodiment of this disclosure.

[0079] Figure 4 A schematic diagram of another process for calculating pose data is shown in an embodiment of this disclosure.

[0080] Figure 5 A schematic diagram of a process for meshing pose data according to an embodiment of this disclosure is shown.

[0081] Figure 6 A schematic diagram of a bin-dividing method according to an embodiment of this disclosure is shown.

[0082] Figure 7 A schematic diagram of a process for determining initial sampling weights is shown in an embodiment of this disclosure.

[0083] Figure 8 A schematic diagram of a process for adjusting the initial sampling weights of buckets whose location range is within a specified area is shown in an embodiment of this disclosure.

[0084] Figure 9 A schematic diagram of a first bin in an embodiment of this disclosure is shown.

[0085] Figure 10 This illustration shows a flowchart of the initial sampling weights for buckets with a location range within a specified area, as described in another embodiment of this disclosure.

[0086] Figure 11 This illustration shows a flowchart of the initial sampling weights for buckets with a location range within a specified area, according to another embodiment of this disclosure.

[0087] Figure 12 A schematic diagram of a process for adjusting the target sampling weight is shown in an embodiment of this disclosure.

[0088] Figure 13 A flowchart illustrating a motion control method for a humanoid robot according to an embodiment of this disclosure is shown.

[0089] Figure 14 A block diagram of a humanoid robot motion control model training device based on reinforcement learning is shown in an embodiment of this disclosure.

[0090] Figure 15 A block diagram of a humanoid robot motion control device according to an embodiment of the present disclosure is shown.

[0091] Figure 16 A schematic diagram of a humanoid robot according to an embodiment of this disclosure is shown.

[0092] Figure 17 A schematic diagram of the structure of a computer system suitable for implementing the embodiments of the present disclosure is shown.

[0093] Figure 18 A schematic diagram of a computer-readable storage medium according to an embodiment of the present disclosure is shown. Detailed Implementation

[0094] In this disclosure, the terms "first" and "second" are used for description only and do not indicate relative importance or imply the number of technical features. Therefore, the features referred to as "first" or "second" may explicitly or implicitly include at least one of those features. "A plurality of" means at least two, unless otherwise expressly defined.

[0095] First, the relevant terms used in the exemplary embodiments of this disclosure will be explained:

[0096] Joint space: refers to the parameter space formed by using the motion variables of each joint of a humanoid robot as coordinate axes, where each dimension corresponds to the angle, displacement, or combination thereof of a joint. A value point in the joint space corresponds to the specific joint configuration state of the robot at a certain moment.

[0097] Pose data refers to the spatial state information of a robot arm's end effector in the workspace, obtained through forward kinematics calculations based on the joint configuration in joint space. Pose data includes at least the end effector's position coordinates and orientation information, and can be represented using Euler angles or quaternions. Pose data is used to generate a command dataset during training, serving as input to the humanoid robot's motion control model.

[0098] Bucketing: refers to the processing method of classifying pose data into multiple discrete sets according to the positional distribution of pose data in the workspace and according to preset spatial division rules. Each bucket corresponds to a positional range in the workspace and contains pose data located within that positional range.

[0099] Initial sampling weight: refers to the basic sampling weight pre-set for each bucket before region differentiation or performance adjustment, used to characterize the basic sampling ratio of pose data in the corresponding bucket during the construction of the instruction dataset.

[0100] Command dataset: refers to the set of training commands extracted and organized from pose data based on bucketing and its corresponding sampling weights, which is used as input to the robot motion control model during reinforcement learning training.

[0101] Workspace: Refers to the range of workspace that the robot arm's end effector can reach under physical structure and joint motion constraints. This workspace is defined by joint limits, link lengths, and kinematic structure. The workspace defines the sampling range for binning and pose data, providing spatial boundaries for training data.

[0102] Edge region: refers to the positional area located near the boundary of the workspace. The pose in this region usually corresponds to the end position where the joint is close to its limit, the arm is close to its maximum extension or folded state.

[0103] Central region: refers to the location area inside the workspace, away from the edge area. The joint configuration corresponding to the pose in this region usually has a large redundancy and good accessibility.

[0104] Boundary sampling weight: refers to the additional or superimposed sampling weights set for the corresponding buckets located in the edge region, used to increase the sampling ratio of edge region pose data in the instruction dataset construction process.

[0105] Target sampling weight: Based on the initial sampling weight, combined with boundary sampling weight and other adjustment factors (such as reducing the weight of the central region and strengthening the weight of specific task regions), the final sampling weight is determined for each bucket and is used for sampling the actual execution instruction dataset.

[0106] Performance compensation weights: These are weights introduced to further adjust the sampling weights of some buckets based on training results or control effect evaluation, in order to improve the control accuracy or stability of the model in the corresponding spatial region.

[0107] Joint configuration refers to the specific combination of values ​​for each joint of a humanoid robot at a given moment. This joint configuration uniquely determines the robot's overall posture and the positional state of its end effector in the workspace. Joint configuration can be represented by joint angles or displacements and is the basis for generating pose data and command datasets.

[0108] Figure 1 A system architecture diagram is shown that can be applied to the reinforcement learning-based humanoid robot motion control model training method and the humanoid robot motion control method in the embodiments of this disclosure. Figure 1As shown, the system architecture 100 may include a terminal device 101, a robot 102, a network 103, and a server 104.

[0109] The terminal device 101 includes, but is not limited to, desktop computers, laptops, smartphones, and tablets. The terminal device 101 is equipped with a graphical user interface for configuring and managing the training process of the humanoid robot motion control model. Specifically, the terminal device 101 can be used to set parameters related to joint space sampling, pose binning methods, sampling weight adjustment rules, and the reinforcement learning training process, and can also be used to display model performance indicators, sampling distribution, and training status information during the training process.

[0110] Robot 102 can be a humanoid robot, equipped with inertial measurement units, joint angle encoders, and other proprioceptive sensors to collect its own joint states, posture information, and motion data. Robot 102 includes at least a main controller and joint actuators. The main controller deploys and runs a trained reinforcement learning-based humanoid robot motion control model and generates motion strategies based on real-time acquired proprioceptive information. The joint actuators receive the motion strategies and drive the corresponding joints to perform movements, thereby achieving motion control of the humanoid robot.

[0111] Server 104 is equipped with a training module for training the humanoid robot's motion control model. The training module randomly samples from the humanoid robot's joint space and calculates the pose data of the robot arm's end effector based on the sampling results. It then bins the pose data based on its position distribution in the workspace. Sampling weights are configured and adjusted for each bin to obtain target sampling weights. Finally, the pose data is sampled based on these target sampling weights to construct a command dataset, which is then input into the reinforcement learning-based humanoid robot motion control model to complete model training. After model training is complete, server 104 can distribute the trained motion control model to robot 102 for real-time action strategy generation during operation.

[0112] Network 103 serves as a medium to provide a communication link between terminal device 101, robot 102, and server 104, enabling the transmission of sensor data, the distribution of model parameters, and the exchange of control information during the training process. Network 103 may include various connection types, such as wired or wireless communication links or fiber optic cables.

[0113] It should be understood that Figure 1 The number and type of terminal devices, robots, networks, and servers shown are merely illustrative. Depending on implementation needs, there can be any number and type of terminal devices, robots, networks, and servers; for example, the servers could be cloud servers.

[0114] Through the coordinated operation of the various components in the system architecture 100, it is possible to acquire pose data based on random sampling in joint space, perform bucket processing based on workspace position distribution, construct instruction datasets based on sampling weight adjustment, and train and deploy humanoid robot motion control models based on reinforcement learning, thereby improving the motion control performance of the model in different workspace positions.

[0115] This disclosure provides an example implementation of a method for training a humanoid robot motion control model based on reinforcement learning. (See references) Figure 2 As shown, the method may include the following steps S201 to S204:

[0116] Step S201: Randomly sample from the joint space of the humanoid robot and calculate multiple pose data of the robot arm end based on the sampling results;

[0117] Step S202: Based on the position of each pose data, divide each pose data into multiple buckets, with each bucket corresponding to a set of pose data within a position range;

[0118] Step S203: Configure initial sampling weights for each bucket and adjust the initial sampling weights of buckets whose location range is within the specified area to obtain the target sampling weights for each bucket.

[0119] Step S204: Sample based on the target sampling weight of each bucket to obtain the instruction dataset, and input the instruction dataset into the humanoid robot motion control model based on reinforcement learning for training.

[0120] This disclosure improves the spatial distribution balance of training samples by adjusting the sampling weights of corresponding buckets within a specified region, without increasing the overall training data size. As a result, the trained humanoid robot motion control model exhibits more consistent control accuracy across all positional regions, reducing the likelihood of decreased pose tracking accuracy in localized areas.

[0121] The following will provide a detailed description of the reinforcement learning-based humanoid robot motion control model training method in this example embodiment.

[0122] In step S201, random sampling is performed from the joint space of the humanoid robot, and multiple pose data of the robot arm end are calculated based on the sampling results.

[0123] In the exemplary embodiments of this disclosure, the joint space of a humanoid robot can be randomly sampled under preset joint motion range constraints to generate multiple sets of joint configurations. The preset joint motion range constraints refer to the allowable motion range set for each joint of the humanoid robot, including minimum and maximum value limits for the joints, as well as additional constraints related to physical structure, link length, and safety requirements. Random sampling can employ uniform random sampling, weighted random sampling, or restricted random sampling methods to ensure that the sampling results meet the requirements of joint physical constraints, kinematic accessibility, and safety restrictions, and cover the entire joint space, avoiding concentration in a few localized areas.

[0124] Subsequently, the joint configuration of each group is input into the forward kinematics model of the humanoid robot. Based on the spatial connection relationship and geometric parameters of each joint, the pose data of the robot arm's end effector in the workspace under the corresponding joint configuration is calculated. For example, the pose data includes at least the position and orientation of the robot arm's end effector in three-dimensional space. Here, position refers to the coordinate position of the robot arm's end effector in three-dimensional space, and orientation is used to characterize the spatial orientation of the robot arm's end effector.

[0125] In this step, a set of pose data for the robot arm's end effector covering multiple workspace regions can be obtained based on random sampling results from different joint spaces. This provides basic data support for subsequent binning, sampling weight configuration, and training sample construction. Moreover, this step does not rely on manual teaching or predefined trajectories, and can effectively improve the diversity and coverage completeness of the robot arm's end effector pose data in the workspace.

[0126] In some example implementations, references Figure 3 As shown, step S201 may further include the following steps S301 and S302:

[0127] Step S301: Within the preset joint space sampling range, sample multiple joint angles of the humanoid robot to generate joint configurations corresponding to each joint angle.

[0128] Specifically, the humanoid robot arm includes multiple rotatable joints, each corresponding to at least one joint angle. Based on the structural characteristics, mechanical limits, and safety constraints of each joint, a corresponding range of joint angle values ​​can be pre-defined for each joint, thus forming the joint space sampling range of the entire arm.

[0129] For example, within the joint space sampling range, multiple joint angles of the humanoid robot are jointly sampled to obtain multiple sets of joint angle combinations, and each set of joint angle combinations constitutes a complete joint configuration.

[0130] This step generates multiple combinations and diverse joint configurations, providing fundamental data support for subsequent calculations of the robot arm's end-effector pose data using forward kinematics.

[0131] Step S302: Based on the configuration of each joint, calculate multiple pose data of the robot arm end effector.

[0132] For example, refer to Figure 4 As shown, step S302 may further include steps S401 and S402:

[0133] Step S401: For each joint configuration, the position and orientation of the corresponding robot arm end effector are obtained through forward kinematics calculation.

[0134] Using the joint configurations generated in step S301 as input, and combining the structural parameters of the humanoid robot arm with the connection relationships between the joints, a corresponding forward kinematics calculation model is constructed. Based on the forward kinematics calculation model, calculations are performed on each joint configuration to obtain the position and orientation of the robot arm's end effector in the workspace under the corresponding joint configuration.

[0135] Step S402: Based on the position and orientation of the robot arm's end effector, obtain multiple pose data of the robot arm's end effector.

[0136] The obtained position and posture of the robot arm end effector are combined to form the end effector pose data under the corresponding joint configuration. The end effector pose data corresponding to multiple joint configurations are summarized to form the pose data set of the robot arm end effector.

[0137] For example, a set of pose data obtained D for:

[0138] D ={( pos , quat )}

[0139] in, pos This indicates the position of the robot arm's end effector in three-dimensional space, and may include... x , y , z Values ​​in three directions; quat The quaternion information representing the pose of the robot arm's end effector can contain four components ( w , x , y , z ), w The angle is the rotation angle.

[0140] By mapping joint configurations in joint space to end-effector pose data in workspace via positive kinematics, the correspondence between joint configurations and end-effector states can be realized, improving the coverage and distribution diversity of pose data in workspace, thereby providing a data foundation for fine-grained control of different spatial regions during subsequent training.

[0141] In step S202, based on the position of each pose data, each pose data is divided into multiple buckets, and each bucket corresponds to a set of pose data within a position range.

[0142] After obtaining multiple pose data at the end of the robot arm, the position in the pose data is used as the basis for binning, and each pose data is divided.

[0143] Specifically, based on the working space range at the end of the robotic arm, the working space can be divided into intervals along at least one spatial dimension, or it can be divided in combination along multiple spatial dimensions to form multiple spatial regions, each spatial region corresponding to a bin.

[0144] Subsequently, each pose data is assigned to a bucket corresponding to its location range, so that the position parameters of the pose data in the same bucket are located in the same or similar spatial range.

[0145] By binning, pose data distributed in different spatial locations can be classified and organized, forming multiple mutually distinct data sets in the spatial location dimension. This helps to avoid the problem of uneven spatial distribution of pose data and improves the coverage and control accuracy of each spatial region in the subsequent training process.

[0146] In some example implementations, the pose data can be divided into multiple buckets based on the position of each pose data.

[0147] Specifically, based on the position data of each pose, the reachable position range of the robot arm's end effector in the workspace is determined. Then, the workspace is discretized according to a preset grid division rule, dividing the workspace into multiple grid regions. Accordingly, each grid region corresponds to a position range and serves as a bucket.

[0148] For example, during the meshing process, mesh intervals can be set along at least one coordinate axis in space, or mesh intervals can be set along multiple coordinate axes to form a two-dimensional or three-dimensional mesh structure. Each mesh region covers a continuous position interval to accommodate pose data located within that position interval. Subsequently, based on the position of each pose data point, the corresponding pose data is assigned to the bucket corresponding to its mesh region.

[0149] By using a gridded approach to bin the pose data, the workspace can be divided in a regular manner, giving pose data in different spatial locations a clear belonging relationship. This facilitates subsequent configuration of sampling weights and adjustment of data distribution based on the binning results, thereby improving the control accuracy and stability of different spatial regions during model training.

[0150] For example, refer to Figure 5 As shown, the process of meshing pose data may include the following steps S501 and S502:

[0151] Step S501: Divide the humanoid robot's workspace into multiple voxel regions according to the preset spatial division step length.

[0152] The humanoid robot's workspace is a three-dimensional spatial region, the extent of which can be determined by the reachable positions of the robot arm's end effector under extreme joint conditions. Based on the workspace's extent, a spatial division step size is set, and the workspace is discretized along each spatial dimension, thereby dividing the continuous workspace into multiple independent voxel regions.

[0153] The spatial partitioning step size is used to define the size of each voxel region in each spatial dimension. The spatial partitioning step size can be set according to the robot size, task accuracy requirements, or training data scale, and this disclosure does not impose any limitations on it. Each voxel region is used to represent a local spatial range in the workspace, and different voxel regions correspond to different position intervals.

[0154] Step S502: Based on the position of each pose data, the pose data located in the same voxel region are divided into the same bucket.

[0155] In this step, the voxel region corresponding to each pose data in the workspace is determined based on the position of each pose data, and pose data whose positions fall within the same voxel region are divided into the same bin, so that the pose data in the same bin have the same or similar position range in the spatial position dimension.

[0156] For example, a set of pose data D According to its position pos When performing spatial binning, a set of bins can be obtained. H Each bin is formed by dividing the workspace at the end of the robotic arm into a three-dimensional grid, with each grid area corresponding to one bin.

[0157] In practical implementation, the workspace along x, y , zThe three spatial directions are divided into intervals, forming multiple voxel regions. Each voxel region can be identified by a position key, such as a position key represented as... Used in bucketed collections H It enables rapid location and management of data within corresponding spatial regions. These respectively represent the voxel region in x, y , z The starting coordinate value in the direction. This position key is used as an index identifier for bucketing, distinguishing buckets corresponding to different spatial regions.

[0158] A bucket corresponds to all pose data that satisfy the following position range: end-effector position x Coordinates are located in the interval , y Coordinates are located in the interval , z Coordinates are located in the interval ,in, dx , dy , dz They represent along x , y , z The spatial division step size in the direction is used to determine the size of the voxel region.

[0159] By voxelization and binning, the continuously distributed pose data at the end of the robot arm can be transformed into a data set based on spatial region organization, so that different spatial locations correspond to different data bins, which helps to improve the distribution balance of pose data in the workspace.

[0160] It should be noted that during the voxel partitioning process of the humanoid robot's workspace, the partitioning step size is used to determine the size of the voxel region. Different partitioning step sizes directly affect the number of bins and the scale of pose data contained in each bin. To ensure that the pose data partitioning method can adapt to different training needs or task scenarios, the partitioning step size can be adjusted according to a preset strategy, such as dynamic or phased adjustments. The preset strategy may include at least one of the following: changes in the training phase, humanoid robot task requirements, or workspace characteristics.

[0161] For example, in the initial stage of model training, a larger spatial partitioning step size can be used to coarsely divide the workspace, thereby reducing the number of buckets and improving overall training efficiency. As the training process progresses, the spatial partitioning step size can be gradually reduced to perform more refined voxel partitioning of the workspace, thereby enhancing the model's ability to perceive and control local spatial positions.

[0162] For example, when a humanoid robot performs high-precision tasks or needs to focus on a specific local workspace area, the spatial division step size can be adjusted according to the corresponding task requirements to give the relevant voxel area a higher spatial resolution.

[0163] For example, for regions with sparse pose distribution or distinct boundary features within the workspace, the spatial partitioning step size can be adjusted specifically based on the workspace characteristics. Workspace characteristics refer to the differences in accessibility, pose distribution density, motion constraints, and control difficulty exhibited by different spatial regions within the reachable space of the robot arm. These characteristics may include at least spatial accessibility characteristics, pose distribution characteristics, and boundary and limit characteristics. For instance, for spatial regions with dense pose distribution and relatively stable control, a larger spatial partitioning step size can be used to reduce the number of buckets and improve training efficiency. For spatial regions with sparse pose distribution, located at the workspace boundary, or with high control difficulty, the spatial partitioning step size can be reduced to give the corresponding voxel region higher spatial resolution, thereby enhancing the model's perception ability and control accuracy for such spatial locations.

[0164] By further adjusting the spatial partitioning step size, the pose data binning process can be made more flexible and adaptable, thereby improving the control accuracy and robustness of the model in different task scenarios and spatial regions while ensuring training efficiency.

[0165] refer to Figure 6 The diagram illustrates a binning approach. Specifically, the workspace 601 of the humanoid robot is divided into multiple bins 602. Each bin 602 corresponds to a set of pose data within a specific position range. By statistically analyzing the pose data within each bin 602, the distribution of pose data in three-dimensional space can be obtained.

[0166] To facilitate understanding, we will use a planar bin distribution 603 as an example. The color intensity of different bin regions indicates the amount of pose data falling into that bin; a darker color indicates more pose data in the corresponding bin, and a lighter color indicates fewer pose data in the corresponding bin. This illustration visually reflects the density of pose data distribution in the workspace and its spatial differences.

[0167] It should be noted that the embodiments disclosed herein are based on the unified pose data statistics and sampling weight adjustment of all bins formed within the robot's workspace, rather than processing only the binning results within a single plane.

[0168] In step S203, initial sampling weights are configured for each bucket, and the initial sampling weights of buckets whose location range is within the specified area are adjusted to obtain the target sampling weights for each bucket.

[0169] After binning the pose data at the robotic arm's end effector, initial sampling weights are assigned to each of the resulting bins. These initial sampling weights characterize the probability of the pose data within that bin being selected in subsequent training sample construction or data sampling. The initial sampling weights can be set using a uniform basic weight setting method, ensuring that each bin has the same sampling priority in its initial state, or they can be set based on factors such as the quantity and spatial distribution of pose data within the bin; this disclosure does not impose any limitations on this approach.

[0170] In some example implementations, references Figure 7 As shown, the process of determining the initial sampling weights corresponding to each bucket may include the following steps S701 and S702:

[0171] Step S701: Count the number of pose data in each bucket to obtain the number of samples in each bucket.

[0172] For the obtained multiple buckets, the pose data contained in each bucket can be statistically analyzed to obtain the total number of pose data in that bucket, which is also the sample size of each bucket. The sample size reflects the data density of the bucket in spatial location. For example, some buckets are located in the central area of ​​the workspace and have a large number of pose data, while other buckets may be located in the edge area or sparse area and have a smaller number of pose data.

[0173] By counting the number of samples in each bin, we can clarify the distribution of each spatial location, providing basic data for setting subsequent sampling weights.

[0174] Step S702: Determine the initial sampling weight corresponding to each bin based on the preset coefficient and the number of samples in each bin.

[0175] In this step, the sample size of each bucket is calculated using preset coefficients to obtain the initial sampling weight for each bucket. The preset coefficients can be used to normalize the weights of different buckets, making the initial sampling weights suitable for subsequent training sampling processes.

[0176] For example, the initial sampling weights can be set according to the reciprocal or proportion of the number of samples in each bucket to balance the sample distribution. This results in buckets with a larger number of samples having relatively smaller initial sampling weights, while buckets with a smaller number of samples have relatively larger initial sampling weights. This achieves balanced sampling of sparse and dense regions in the early stages of training. In this way, the problem of uneven data distribution during training, which could lead to insufficient learning of certain spatial regions by the model, can be prevented.

[0177] For example, for each bucket, we have:

[0178]

[0179] in, The initial sampling weights are for each bucket. K For preset coefficients, This represents the number of samples in each bin. This represents the set of pose data in each bucket, each... Each corresponds to a separate bucket.

[0180] The initial sampling weights ensure that, during the training sampling process, the buckets with fewer samples have a relatively higher sampling probability, so as to balance the data coverage of different buckets.

[0181] After obtaining the initial sampling weights, the initial sampling weights of the buckets located in the specified regions can be further adjusted. In the example implementation of this disclosure, the specified region may include the edge region of the workspace, a region with relatively sparse pose distribution, a critical operation region frequently involved in the task execution process, or a region with high control accuracy requirements, etc. Of course, the region can be determined according to training needs, task objectives, or spatial characteristics. The adjustment method can be to increase or decrease the sampling weight of the bucket, thereby changing the participation ratio of pose data within the bucket in the subsequent sampling process; this disclosure does not limit this.

[0182] Finally, after initial sampling weight configuration and adjustment of specified region bucket weights, the target sampling weights for each bucket are obtained. These target sampling weights can be directly used in the subsequent sampling process for building the training dataset.

[0183] By adjusting the weights of buckets in a specified region, the pose data can have differentiated sampling probabilities in different spatial locations, thereby guiding the training process to form a more reasonable data distribution in the spatial dimension and improving the model training effect and generalization ability.

[0184] In one example implementation, the designated area is the edge region of the humanoid robot's workspace. Accordingly, refer to Figure 8 As shown, the process of adjusting the initial sampling weights of buckets whose location range is within a specified area may include the following steps S801 and S802:

[0185] Step S801: Identify the first bucket whose position range is located in the edge area of ​​the humanoid robot's workspace.

[0186] In this step, the spatial location range corresponding to each bucket can be extracted based on the generated multiple buckets. By analyzing the boundary positions or spatial geometric features of the workspace, the set of buckets located in the edge region of the workspace is determined. Edge regions are usually sparse in the training sample distribution or have high requirements for model control accuracy.

[0187] By identifying edge regions and dividing them into bins, it becomes clear which bins require extra attention in subsequent sampling processes, ensuring that the model learns the edge locations sufficiently.

[0188] refer to Figure 9 The diagram illustrates a first type of binning. In this diagram, each bin in binning region 901 represents a first bin located at the edge of the humanoid robot's workspace. These bins require further processing, such as adding boundary sampling weights, to ensure that the sampling process covers the entire workspace while also considering the sample acquisition needs of the edge regions.

[0189] Step S802: Add boundary sampling weights to the first bucket to obtain the target sampling weights of the first bucket.

[0190] Specifically, the target sampling weight of the first bucket can be calculated based on the initial sampling weight and the boundary sampling weight of the first bucket. In other words, the pre-set boundary sampling weight is added to the initial sampling weight of each first bucket to obtain the target sampling weight of the first bucket.

[0191] For example, the target sampling weight of the first bucket is obtained by adding its initial sampling weight to the additional boundary sampling weight, that is:

[0192]

[0193] in, The target sampling weight for the first bucket. The initial sampling weights for the first bucket reflect the basic sampling probability of that bucket in the overall training data. The boundary sampling weights are used to increase the priority of buckets located at the edge of the workspace during the training sampling process, thereby enabling the model to obtain more samples in the edge region and enhance its learning ability at the edge location.

[0194] In this example, the boundary sampling weights can be fixed values ​​or dynamically adjusted based on the spatial location of the buckets, the training stage, or task requirements. For instance, the boundary sampling weights can be set to a constant in the initial stage to ensure that edge buckets have sufficient sampling priority in the early stages of training. To accommodate the improvement of the model's learning ability during training, the boundary sampling weights can be gradually reduced as the training progresses. For example, as the number of training iterations increases or the training stage advances, the boundary sampling weights can be reduced using linear, exponential, or other preset decay strategies, so that the sampling probability of edge buckets gradually returns to the initial sampling weight level, thereby balancing the distribution of training data in edge regions and other regions.

[0195] By flexibly adjusting the boundary sampling weights in this example, the target sampling weights of the first bucket can enhance the sampling frequency of edge region pose data in the early stages of training, improving the model's learning effect on edge positions. As training progresses, the boundary sampling weights gradually decay, making model training more dependent on the overall data distribution, thereby achieving balance and stability in the training process. Therefore, it can effectively improve the problem of uneven spatial distribution of training data and improve the model's generalization ability and control accuracy across the entire workspace.

[0196] In one example implementation, the designated area is the central area within the humanoid robot's workspace. Accordingly, refer to... Figure 10 As shown, the process of adjusting the initial sampling weights of buckets whose location range is within a specified area may include the following steps S1001 and S1002:

[0197] Step S1001: Identify the second bucket whose location range is in the central area of ​​the humanoid robot's workspace.

[0198] In this step, based on the generated multiple buckets, the spatial location range information corresponding to each bucket can be extracted. By analyzing the boundary positions or spatial geometric features of the workspace, the set of buckets located in the central region of the workspace is determined. The central region is usually the area with a dense distribution of pose data within the workspace, and it is also the location area that the robotic arm frequently passes through during task execution.

[0199] By identifying the central region bins, it can be determined which bins are relatively abundant in the training data distribution, and their sampling weights can be appropriately adjusted in the subsequent training process to balance the overall spatial sample distribution.

[0200] Step S1002: Reduce the initial sampling weight of the second bucket to obtain the target sampling weight of the second bucket.

[0201] Specifically, the initial sampling weights of the second bucket can be adjusted according to a preset reduction strategy to obtain the target sampling weights of the bucket. The preset reduction strategy can be a fixed reduction, a proportional reduction, or a reduction that decreases with the training progress, so that the probability of the second bucket being selected in subsequent training sampling processes is appropriately reduced. This disclosure does not specify the method of reducing the weights of the bucket.

[0202] By differentially adjusting the sampling weights of the central region buckets, the distribution of training data in the workspace becomes more balanced, while maintaining the sampling priority for key edge regions or sparse regions. This helps to balance the learning effects of different spatial locations during training and improve the control accuracy and robustness of the model across the entire workspace.

[0203] In one example implementation, the designated area is a target area within the humanoid robot's workspace. The target area may include at least one of the following: a spatial area corresponding to the robot's singular configuration, a spatial area corresponding to the task-specified operation position, or a spatial area where the training error meets preset conditions.

[0204] The spatial region corresponding to a robot's singular configuration refers to the corresponding location region in the workspace of the singular configuration generated by a specific combination of joint angles of the robot arm. Within this spatial region, the robot arm's kinematic Jacobian matrix tends to degenerate, the end-effector pose is highly sensitive to changes in joint angles, joint redundancy is reduced or lost, typically leading to decreased control stability. Therefore, the model faces higher learning difficulty in this spatial region, and is prone to decreased control accuracy or motion instability.

[0205] The spatial region corresponding to the task-specified operation position refers to the target area where the humanoid robot needs to frequently perform grasping, assembly, interaction, or manipulation actions during the training phase or actual task execution. This type of spatial region is usually predetermined by specific task requirements, and its spatial distribution may be located anywhere in the workspace, directly affecting the task completion effect and operation success rate. It is understandable that the spatial region corresponding to the task-specified operation position is not equivalent to the central area of ​​the workspace; its location may be off-center, or even located at the edge or in a localized area of ​​the workspace.

[0206] The spatial region where the training error meets the preset conditions refers to the location region with a large prediction error or performance that fails to meet the preset standard, identified during model training based on the error assessment between the model output and the expected control result. This type of spatial region reflects the problem of insufficient learning or weak generalization ability of the model at a specific spatial location. Therefore, targeted reinforcement learning of the model can be performed by increasing the sampling weight of the pose data in the corresponding spatial region to improve the model's control accuracy and stability in that region.

[0207] By identifying the third bucket within the target region, we can clearly define the spatial region that needs to be focused on during training, thus providing special treatment when adjusting the sampling weights.

[0208] It should be noted that the target areas given above are merely illustrative examples, used to indicate spatial locations within the workspace that require special attention or differentiated treatment, and do not constitute a limitation on the type of target area. In other embodiments, the target area can be flexibly set or adjusted according to factors such as the structural characteristics of the humanoid robot, task requirements, training effects, or control performance.

[0209] Accordingly, refer to Figure 11As shown, the process of adjusting the initial sampling weights of buckets whose location range is within a specified area may include the following steps S1101 and S1102:

[0210] Step S1101: Identify the third bucket, whose location range is within the target area of ​​the humanoid robot's workspace.

[0211] In this step, based on the generated multiple buckets, the spatial location range information corresponding to each bucket can be extracted. By analyzing the workspace characteristics of the robotic arm and the training task requirements, the regional attributes of each bucket are determined, thereby identifying the bucket whose location range falls within the target area and designating it as the third bucket.

[0212] Step S1102: Adjust the initial sampling weight of the third bucket according to the preset weight adjustment factor to obtain the target sampling weight of the third bucket.

[0213] Specifically, for the identified third bucket, a preset weight adjustment factor can be applied to its initial sampling weights to increase or decrease the probability of that bucket being selected during the training sampling process. The weight adjustment factor can be set according to the importance of the target region, the training stage, or model performance metrics, and is used to characterize the adjustment magnitude of the sampling frequency for that bucket.

[0214] The weight adjustment factor can take different values. For example, for regions with singular configurations or large errors, the weight adjustment factor can be set to a value greater than 1 to increase the sampling frequency, giving that bucket a higher sampling probability during the construction of the instruction dataset, thereby strengthening the model's learning in that region. Conversely, for non-critical locations in the task, the weight adjustment factor can be set to a value less than 1 to avoid over-concentration of training samples.

[0215] In this embodiment, the target sampling weight of the third bucket can prioritize key spatial regions or difficult-to-learn regions during training, thereby improving the model's learning effect and control accuracy at the target location. Simultaneously, by combining the weight adjustments for edge and center buckets in other example embodiments, a comprehensive and differentiated sampling strategy for each location region of the workspace can be achieved, optimizing the spatial distribution of training samples and improving the model's generalization ability and stability.

[0216] In step S204, sampling is performed based on the target sampling weights of each bucket to obtain the instruction dataset, and the instruction dataset is input into the humanoid robot motion control model based on reinforcement learning for training.

[0217] After obtaining the target sampling weights for each bucket, data sampling can be performed according to the bucket division and the target sampling weights for each bucket. The sampling process can employ random sampling, weighted random sampling, or other sampling strategies suitable for reinforcement learning training, ensuring that the probability of selecting pose data in each bucket is proportional to its target sampling weight, thereby ensuring that the training samples have a reasonable spatial distribution.

[0218] In the exemplary implementation of this disclosure, training samples can be distributed in spatial location according to different strategies such as edge region binning, center region binning, and target region binning, to ensure that the model obtains sufficient samples in difficult-to-learn regions, key task regions, and edge or sparse regions, thereby improving the control accuracy and training effect of the model in different spatial regions.

[0219] After sampling, the selected pose data is converted into a corresponding instruction dataset. For example, the position, attitude, and joint angles calculated through inverse kinematics from the sampled pose data are converted into executable motion commands, and the necessary labels or reward information for training are added to form a complete training item. All training items are then aggregated to form the instruction dataset.

[0220] Using this instruction dataset, the humanoid robot motion control model can learn the mapping relationship from input to desired action, achieving precise control of the robot arm's end effector.

[0221] Specifically, the instruction dataset is input into the humanoid robot motion control model for training. During training, the model generates action predictions based on the input state information and calculates the loss function or reward value by comparing them with the actual actions in the training data. Subsequently, reinforcement learning algorithms such as policy gradient, Actor-Critic, or other algorithms suitable for continuous action spaces are used to iteratively optimize the model parameters.

[0222] Through continuous iteration, the model gradually learns the optimal control strategy for different spatial locations and task conditions, thereby improving the accuracy, stability, and generalization ability of robot motion. Furthermore, embodiments of this disclosure can dynamically adjust the sampling strategy and training sample distribution according to the training stage or task requirements, further enhancing the model's learning performance in key regions and improving its adaptability to edge or sparse spatial locations.

[0223] Furthermore, after obtaining the target sampling weights for the first bucket by executing steps S801 and S802, the target sampling weights can be dynamically optimized to improve the model's learning performance in the corresponding spatial region during training. (Reference) Figure 12 As shown, the process of adjusting the target sampling weights may include the following steps S1201 to S1203:

[0224] Step S1201: Sampling is performed based on the target sampling weights of the first bucket, and the humanoid robot motion control model is trained using the sampled instruction dataset.

[0225] Based on the target sampling weights obtained from the first bucket, pose data is selected from the corresponding bucket using weighted randomness or other applicable sampling strategies. The selected pose data is then transformed to form a command dataset, which is subsequently input into a reinforcement learning-based humanoid robot motion control model for training.

[0226] Step S1202: Calculate the training performance score of the first bucket based on the evaluation results after training, and map the training performance score to the performance compensation weight.

[0227] The training evaluation results may include metrics such as the model's motion error, trajectory deviation, or task completion accuracy at the spatial location corresponding to the first bucket. A training performance score is generated by comprehensively calculating these evaluation metrics to reflect the model's learning performance within that bucket region.

[0228] Furthermore, the training performance score is mapped to a performance compensation weight, which is used to adjust the sampling priority of the first bucket in subsequent sampling processes. The mapping rule can be set according to the training performance score. For example, if the training performance score is low, it means that the model's learning effect in that bucket region is poor, so the performance compensation weight can be set to a larger value to increase the probability of that bucket being selected in the next round of sampling. If the training performance score is high, it means that the model's learning effect in that bucket region is good, so the performance compensation weight can be set to a smaller value to reduce the frequency of that bucket being repeatedly sampled.

[0229] For example, assuming the training performance score ranges from 0 to 100 and the performance compensation weight ranges from 0 to 1, a linear mapping can be used, resulting in:

[0230]

[0231] in, For performance compensation weights, S For example, when the training performance score... S When the training performance score is 20, the corresponding performance compensation weight is 0.8; when the training performance score is 80, the corresponding performance compensation weight is 0.2.

[0232] Through this mapping method, spatial regions with poor training performance will receive higher compensation weights, thus being prioritized in resampling and enabling focused reinforcement training of difficult-to-learn regions.

[0233] Step S1203: The performance compensation weight is added to the target sampling weight of the first bucket to obtain the final sampling weight of the first bucket, and resampling is performed based on the final sampling weight of the first bucket.

[0234] In this step, the performance compensation weight can be directly added to the target sampling weight of the first bucket, or weighted addition can be used to obtain the final sampling weight of the first bucket.

[0235] For example, the final sampling weight of the first bucket This can be represented as the initial sampling weight of the first bucket. Boundary sampling weights and performance compensation weight The sum of:

[0236]

[0237] The weight configuration method in this example enables differentiated sampling of training samples, allowing the model to cover edges, centers, and key regions evenly in space. At the same time, it focuses on strengthening the buckets with poor training performance, thereby improving the accuracy, robustness, and generalization ability of the humanoid robot motion control model in the entire workspace.

[0238] Furthermore, based on the final sampling weights of the first bucket, pose data is resampled from the first bucket to form a new instruction dataset, which is used to further train the humanoid robot motion control model.

[0239] This feedback-driven dynamic adjustment mechanism allows the model to pay more attention to poorly performing or critical spatial regions during training, thereby improving training efficiency and enhancing the model's generalization ability and control accuracy across the entire workspace.

[0240] Of course, after obtaining the target sampling weights of the second bucket by executing steps S1001 and S1002, and obtaining the target sampling weights of the third bucket by executing steps S1101 and S1102, the target sampling weights of each bucket can be further dynamically optimized. The dynamic optimization process can be referred to steps S1201 to S1203, including sampling based on the target sampling weights, training the model, evaluating the training performance and generating performance compensation weights, and then combining the performance compensation weights with the original target sampling weights to obtain the final sampling weights. This will not be elaborated in detail here.

[0241] This method ensures that the final sampling weight of each bucket fully reflects its importance in the training process and the model's learning needs, enabling dynamic optimization and balanced utilization of each bucket. Even if a bucket is not divided into edge or center regions, its sampling priority can be adjusted through performance compensation weights, thereby improving the overall training effect and model control accuracy.

[0242] This disclosure also provides a humanoid robot motion control method, which, after model training is completed, applies the trained humanoid robot motion control model to the humanoid robot's operation phase, enabling the humanoid robot to generate corresponding action strategies in real time based on input pose data, and control the robot arm end effector to perform predetermined operations within the workspace.

[0243] refer to Figure 13 As shown, the method may include the following steps S1301 and S1302:

[0244] Step S1301: Obtain the pose data of the robot arm's end effector.

[0245] Specifically, the pose data of the humanoid robot's end effector can be acquired in real time using the body perception sensors mounted on the robot. This pose data can include the robot arm's coordinates in three-dimensional space, attitude angles (such as roll, pitch, and yaw), and velocity or acceleration information can be added as needed to reflect the position and attitude of the end effector at a specific time or under specific task conditions. By acquiring accurate real-time pose data, precise state input can be provided to the model to support subsequent action decisions and control execution.

[0246] Step S1302: Input the pose data into the pre-trained humanoid robot motion control model, and output the action strategy to control the humanoid robot's motion based on the humanoid robot motion control model.

[0247] The motion strategy may include adjustment commands for the angles of each joint, speed and direction control of the end effector, torque distribution, or other control parameters used to drive the robotic arm to perform specific actions. By executing the motion strategy, the humanoid robot can complete the motion task as expected in the actual workspace, ensuring that the end effector reaches the target position and posture, and achieving precise control of the task objective.

[0248] It should be noted that the humanoid robot motion control model in this example embodiment is based on another example embodiment, such as... Figure 2 The training method shown is for a humanoid robot motion control model based on reinforcement learning.

[0249] In an exemplary embodiment of this disclosure, a training device for a humanoid robot motion control model based on reinforcement learning is also provided. (See reference...) Figure 14 As shown, the humanoid robot motion control model training device 1400 based on reinforcement learning includes a pose data calculation module 1401, a pose data partitioning module 1402, a sampling weight adjustment module 1403, and a control model training module 1404, wherein:

[0250] The pose data calculation module 1401 is used to randomly sample from the joint space of the humanoid robot and calculate multiple pose data of the robot arm end based on the sampling results.

[0251] The pose data partitioning module 1402 is used to divide each pose data into multiple buckets based on the position of each pose data, and each bucket corresponds to a set of pose data within a position range.

[0252] The sampling weight adjustment module 1403 is used to configure the initial sampling weight for each bucket and adjust the initial sampling weight of the buckets whose position range is within the specified area to obtain the target sampling weight for each bucket.

[0253] The control model training module 1404 is used to sample based on the target sampling weights of each bucket to obtain the instruction dataset, and input the instruction dataset into the humanoid robot motion control model based on reinforcement learning for training.

[0254] The specific details of each module in the above-mentioned reinforcement learning-based humanoid robot motion control model training device have been described in detail in the corresponding reinforcement learning-based humanoid robot motion control model training method, so they will not be repeated here.

[0255] In an exemplary embodiment of this disclosure, a motion control device for a humanoid robot is also provided. (See reference...) Figure 15 As shown, the humanoid robot motion control device 1500 includes a pose data acquisition module 1501 and a motion strategy output module 1502, wherein:

[0256] The pose data acquisition module 1501 is used to acquire pose data of the robot arm's end effector.

[0257] The motion strategy output module 1502 is used to input pose data into a pre-trained humanoid robot motion control model and output motion strategies to control the humanoid robot's motion based on the humanoid robot motion control model.

[0258] Among them, the humanoid robot motion control model is based on, for example, Figure 2 The training method shown is for a humanoid robot motion control model based on reinforcement learning.

[0259] The specific details of each module in the aforementioned humanoid robot motion control device have been described in detail in the corresponding humanoid robot motion control device method, so they will not be repeated here.

[0260] In an exemplary embodiment of this disclosure, a humanoid robot is also provided. The humanoid robot includes a processor and a memory, the memory storing computer-readable instructions. When the computer-readable instructions are executed by the processor, they implement the described method. (Reference) Figure 16The diagram shown is a schematic of a humanoid robot.

[0261] refer to Figure 17 As shown, an electronic device capable of implementing the above method is also provided. The electronic device 1700 includes a processor 1701 and a memory 1702. The memory 1702 stores computer-readable instructions, which, when executed by the processor 1701, implement the method of this disclosure.

[0262] In an exemplary embodiment of this disclosure, a computer-readable storage medium is also provided, having stored thereon computer program code instructions that, when invoked by a processor, cause the processor to execute the method described in the embodiments.

[0263] refer to Figure 18 As shown, a program product 1800 for implementing the above-described method according to an embodiment of the present disclosure is described. This product may employ a portable compact disc read-only memory (CD-ROM) and include program code, and may run on a terminal device, such as a personal computer. However, the program product of the present disclosure is not limited thereto. In this document, a readable storage medium may be any tangible medium containing or storing a program that may be used by or in conjunction with an instruction execution system, apparatus, or device.

[0264] From the above description of the embodiments, those skilled in the art will readily understand that the exemplary embodiments described herein can be implemented by software or by combining software with necessary hardware. Therefore, the technical solutions according to the embodiments of this disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (such as a CD-ROM, USB flash drive, external hard drive, etc.) or on a network, including several instructions to cause a computing device (such as a personal computer, server, touch terminal, or network device, etc.) to execute the methods according to the embodiments of this disclosure.

[0265] Finally, the above preferred embodiments are only used to illustrate the technical solutions of this application and are not restrictive. Although this application has been described in detail, those skilled in the art should understand that changes in form and detail can be made without departing from the scope defined by the claims of this application. The dimensions in the drawings are not related to the specific physical object, and the physical object dimensions can be arbitrarily changed.

Claims

1. A method for training a humanoid robot motion control model based on reinforcement learning, characterized in that, The method comprises the following steps: randomly sampling from the joint space of the humanoid robot, and calculating a plurality of pose data of the end of the robot arm based on the sampling results; dividing the pose data into a plurality of sub-buckets based on the positions of the pose data, each sub-bucket corresponding to a set of pose data within a position range; configuring initial sampling weights for each sub-bucket, and adjusting the initial sampling weights of the sub-buckets whose position ranges are in a specified area to obtain target sampling weights of each sub-bucket; sampling based on the target sampling weights of each sub-bucket to obtain an instruction data set, and inputting the instruction data set into a humanoid robot motion control model based on reinforcement learning for training.

2. The method of claim 1, wherein the method is based on a reinforcement learning humanoid robot motion control model training method. The method further comprises the following steps: identifying a first sub-bucket whose position range is in the edge area of the humanoid robot workspace from each sub-bucket; appending a boundary sampling weight to the first sub-bucket to obtain the target sampling weight of the first sub-bucket.

3. The method of claim 2, wherein the method is based on a reinforcement learning model for humanoid robot motion control. The method further comprises the following steps: calculating the target sampling weight of the first sub-bucket according to the initial sampling weight and the boundary sampling weight of the first sub-bucket.

4. The method of claim 2, wherein the method is based on a reinforcement learning-based humanoid robot motion control model training method. The method further comprises the following steps: sampling based on the target sampling weight of the first sub-bucket, and training the humanoid robot motion control model using the instruction data set obtained by sampling; calculating a training performance score of the first sub-bucket according to the evaluation results after training, and mapping the training performance score to a performance compensation weight; appending the performance compensation weight to the target sampling weight of the first sub-bucket to obtain the final sampling weight of the first sub-bucket, and resampling based on the final sampling weight of the first sub-bucket.

5. The method of claim 2, wherein the method is based on a reinforcement learning-based humanoid robot motion control model training method. The boundary sampling weight is a constant.

6. The method of claim 2, wherein the method is based on a reinforcement learning-based humanoid robot motion control model training method. The boundary sampling weight decreases as the training progresses.

7. The method of claim 1, wherein the method is based on a reinforcement learning humanoid robot motion control model training method. The method further comprises the following steps: identifying a second sub-bucket whose position range is in the center area of the humanoid robot workspace from each sub-bucket; reducing the initial sampling weight of the second sub-bucket to obtain the target sampling weight of the second sub-bucket.

8. The method of claim 1, wherein the method is based on a reinforcement learning humanoid robot motion control model training method. The method further comprises the following steps: identifying a third sub-bucket whose position range is in the target area of the humanoid robot workspace from each sub-bucket; adjusting the initial sampling weight of the third sub-bucket according to a preset weight adjustment factor to obtain the target sampling weight of the third sub-bucket; wherein the target area of the humanoid robot workspace includes at least one of a space area corresponding to a singular configuration of the robot, a space area corresponding to a specified operation position of the task, or a space area whose training error meets a preset condition. 9.The method of claim 1, wherein, The method further comprises the following steps: sampling a plurality of joint angles of the humanoid robot within a preset joint space sampling range to generate joint configurations corresponding to each joint angle; calculating a plurality of pose data of the end of the robot arm based on each joint configuration.

10. The method of claim 9, wherein the method is based on a reinforcement learning humanoid robot motion control model training method. The method comprises the following steps: According to each joint configuration, the position and attitude of the robot arm end are calculated by forward kinematics; According to the position and attitude of the robot arm end, the multiple pose data of the robot arm end are obtained.

11. The method of claim 1, wherein the method is based on a reinforcement learning humanoid robot motion control model training method. The method comprises the following steps: According to the position of each pose data, each pose data is divided into multiple buckets.

12. The method of claim 11, wherein the method is based on a reinforcement learning humanoid robot motion control model training method. According to the position of each pose data, each pose data is divided into multiple buckets. The method comprises the following steps: According to the position of each pose data, each pose data is divided into multiple buckets.

13. The method of claim 12, wherein the method is based on a reinforcement learning humanoid robot motion control model training method. According to the position of each pose data, each pose data is divided into multiple buckets. The method further comprises: According to the preset strategy, the space division step is adjusted; 14. The method of claim 1, wherein, The preset strategy includes at least one of the change of the training stage, the task requirement of the humanoid robot, or the characteristic of the working space. The method further comprises: The number of pose data in each bucket is counted to obtain the sample number of each bucket; 15. The method of claim 14, wherein the method is based on a reinforcement learning humanoid robot motion control model training method. According to the preset coefficient and the sample number of each bucket, the initial sampling weight corresponding to each bucket is determined. wherein, is an initial sampling weight corresponding to each sub-bucket, K is a preset coefficient, is a number of samples of each sub-bucket, denotes a set of pose data in each sub-bucket, each corresponds to a sub-bucket.

16. A humanoid robot motion control method characterized by, The method further comprises: The method further comprises: Obtain the pose data of the robot arm end; Input the pose data into the pre-trained humanoid robot motion control model, and output the action strategy for controlling the motion of the humanoid robot based on the humanoid robot motion control model; 17. A humanoid robot motion control model training device based on reinforcement learning, characterized by, The humanoid robot motion control model is obtained according to the humanoid robot motion control model training method based on reinforcement learning in any one of claims 1-15. The method comprises the following steps: The pose data calculation module is used for random sampling from the joint space of the humanoid robot, and multiple pose data of the robot arm end are calculated based on the sampling result; The pose data division module is used for dividing each pose data into multiple buckets based on the position of each pose data, and each bucket corresponds to a pose data set within a position range; The sampling weight adjustment module is used for configuring an initial sampling weight for each bucket, and adjusting the initial sampling weight of the bucket whose position range is in a specified area to obtain the target sampling weight of each bucket; 18. A humanoid robot motion control device, characterized by, The control model training module is used for sampling based on the target sampling weight of each bucket, obtaining an instruction data set, and inputting the instruction data set into the humanoid robot motion control model based on reinforcement learning for training. The method comprises the following steps: The pose data acquisition module is used for obtaining the pose data of the robot arm end; The action strategy output module is used for inputting the pose data into the pre-trained humanoid robot motion control model, and outputting the action strategy for controlling the motion of the humanoid robot based on the humanoid robot motion control model. The human-shaped robot motion control model is obtained according to the human-shaped robot motion control model training method based on reinforcement learning in any one of claims 1-15.

19. An electronic device, comprising: Comprising: a processor; and a memory having computer readable instructions stored thereon, the computer readable instructions, when executed by the processor, implement the method of any one of claims 1-16.

20. A humanoid robot, characterized by Comprising: a processor; and a memory having computer readable instructions stored thereon, the computer readable instructions, when executed by the processor, implement the method of any one of claims 1-16.

21. A computer-readable storage medium, characterized in that, The computer readable storage medium has computer program code instructions stored thereon, when the computer program code instructions are called by the processor, the processor executes the method of any one of claims 1-16.

Images