Humanoid robot motion control model training method based on reinforcement learning, humanoid robot motion control method, device and equipment, humanoid robot and storage medium

Abstract

The invention provides a humanoid robot motion control model training method based on reinforcement learning, a humanoid robot motion control method, device and equipment, a humanoid robot and a storage medium, and relates to the technical field of robots. The method comprises the steps that random sampling is conducted from a joint space of the humanoid robot, and pose data of the tail end of an arm are obtained through calculation; dividing the pose data into a plurality of sub-buckets based on the position of the pose data; configuring an initial sampling weight for each sub-bucket, and adjusting the sub-bucket weight of the designated area; sampling according to the target sampling weight to obtain an instruction data set; and training the humanoid robot motion control model based on reinforcement learning by using the instruction data set. According to the invention, by adjusting the proportion of the pose data of the specified area in the training samples, the distribution of the training samples in the working space is more balanced, so that the pose tracking precision of the control strategy at different spatial positions is improved.

Description

Technical Field

[0001] This disclosure relates to the field of robotics technology, and to a method for training a humanoid robot motion control model based on reinforcement learning, a method for controlling the motion of a humanoid robot, a device, an equipment, a humanoid robot, and a storage medium. Background Technology

[0002] When performing reinforcement learning training on a humanoid robot with arm manipulation as the primary objective, it is necessary to use the pose data of several points within the reach of the arm end within the humanoid robot's workspace as the instruction dataset for training.

[0003] One method to obtain this instruction dataset is to randomly sample the positions of all or part of the joints of a humanoid robot (at least all arm joints), and then calculate the pose of the points within the reach of the arm end within the humanoid robot's workspace through forward kinematic fitting. The pose of these points is then used as instruction data, and the instruction dataset is constructed from multiple randomly obtained instruction data.

[0004] However, after training the policy network using the instruction dataset constructed in the above random manner, the applicant found that in the edge areas of the workspace of some humanoid robots, the policy network had poor training performance in terms of the control accuracy of the arm end pose.

[0005] The reason is that the way the above-mentioned instruction dataset is generated results in uneven distribution of instruction data in the workspace. The distribution of instruction data is relatively sparse in some workspaces, especially in the edge areas. This will cause the policy network to learn insufficiently in these areas, affecting the training effect and thus causing a decrease in pose tracking accuracy in these areas.

[0006] For example, the paper "An Adaptive Workspace Bias Method for Sampling Planners" Adaptive workspace Biasing for sampling-based planners The method described in M ​​Zucker et al. (2009) divides the workspace into multiple units and constructs a bias distribution based on features such as occupancy, visibility, and path distance. It then uses a policy gradient algorithm to progressively optimize the sampling bias in path planning tasks, thereby improving planning efficiency. While this method can adaptively adjust the sampling bias of the workspace, it primarily focuses on configuration space search in path planning, emphasizing planning efficiency optimization. It does not model the uneven distribution of the end-effector within the workspace, nor does it provide a mechanism for adjusting sample density in reinforcement learning training. Therefore, it still struggles to address the problem of decreased control accuracy caused by insufficient sampling of the end-effector pose in edge regions.

[0007] It is to be understood that the information disclosed in the Background section is merely for strengthening the understanding of the background of the present disclosure, and thus can include information that does not constitute prior art known to those of ordinary skill in the art. SUMMARY

[0008] The present disclosure provides a humanoid robot motion control model training method based on reinforcement learning, a humanoid robot motion control method, device, equipment, a humanoid robot and a storage medium, to overcome the problem of declining pose tracking accuracy due to uneven distribution of instruction data in the workspace, insufficient training samples at the edge area of the arm end, and the like in the related art.

[0009] Additional aspects and advantages of the present disclosure will be set forth in part in the following description, and in part will become apparent to those of ordinary skill in the art from the description, or can be learned by practice of the present disclosure.

[0010] According to a first aspect of the present disclosure, a humanoid robot motion control model training method based on reinforcement learning is provided, comprising: Randomly sampling from the joint space of the humanoid robot, and calculating a plurality of pose data of the end of the robot arm based on the sampling results; Based on the position of each pose data, dividing each pose data into a plurality of buckets, each bucket corresponding to a set of pose data within a position range; Configuring an initial sampling weight for each bucket, and adjusting the initial sampling weight of the bucket whose position range is in the specified area to obtain a target sampling weight of each bucket; Based on the target sampling weight of each bucket, sampling is performed to obtain an instruction data set, and the instruction data set is input into a humanoid robot motion control model based on reinforcement learning for training.

[0011] In an exemplary embodiment of the present disclosure, adjusting the initial sampling weight of the bucket whose position range is in the specified area comprises: Identifying a first bucket in each bucket whose position range is in the edge area of the workspace of the humanoid robot; Applying a boundary sampling weight to the first bucket to obtain a target sampling weight of the first bucket.

[0012] In an exemplary embodiment of the present disclosure, applying a boundary sampling weight to the first bucket to obtain a target sampling weight of the first bucket comprises: According to the initial sampling weight and the boundary sampling weight of the first bucket, the target sampling weight of the first bucket is calculated.

[0013] In an exemplary embodiment of the present disclosure, the method further comprises: sampling based on the target sampling weight of the first sub-bucket, and training the humanoid robot motion control model using the sampled instruction dataset; calculating a training performance score of the first sub-bucket according to the evaluation result after training, and mapping the training performance score to a performance compensation weight; appending the performance compensation weight to the target sampling weight of the first sub-bucket to obtain a final sampling weight of the first sub-bucket, and resampling based on the final sampling weight of the first sub-bucket.

[0014] In an exemplary embodiment of the present disclosure, the boundary sampling weight is a constant.

[0015] In an exemplary embodiment of the present disclosure, the boundary sampling weight decreases as the training progresses.

[0016] In an exemplary embodiment of the present disclosure, adjusting the initial sampling weight of the sub-bucket with the position range in the specified area comprises: identifying a second sub-bucket in which the position range is in a central area of the humanoid robot workspace; reducing the initial sampling weight of the second sub-bucket to obtain a target sampling weight of the second sub-bucket.

[0017] In an exemplary embodiment of the present disclosure, adjusting the initial sampling weight of the sub-bucket with the position range in the specified area comprises: identifying a third sub-bucket in which the position range is in a target area of the humanoid robot workspace; adjusting the initial sampling weight of the third sub-bucket according to a preset weight adjustment factor to obtain a target sampling weight of the third sub-bucket; wherein the target area of the humanoid robot workspace includes at least one of a spatial area corresponding to a singular configuration of the robot, a spatial area corresponding to a specified operation position of the task, or a spatial area in which the training error meets a preset condition.

[0018] In an exemplary embodiment of the present disclosure, random sampling is performed from the joint space of the humanoid robot, and a plurality of pose data of the end of the robot arm is calculated based on the sampling result, comprising: sampling a plurality of joint angles of the humanoid robot within a preset joint space sampling range to generate a joint configuration corresponding to each joint angle; based on each joint configuration, calculating a plurality of pose data of the end of the robot arm.

[0019] In an exemplary embodiment of the present disclosure, based on each joint configuration, a plurality of pose data of the end of the robot arm is calculated, comprising: for each joint configuration, the position and attitude of the end of the robot arm corresponding to the joint configuration are calculated through forward kinematics. According to the position and pose of the end of the robot arm, a plurality of pose data of the end of the robot arm is obtained.

[0020] In an example embodiment of the present disclosure, based on the position of each pose data, each pose data is divided into a plurality of buckets, including: According to the position of each pose data, each pose data is grid-divided to obtain a plurality of buckets.

[0021] In an example embodiment of the present disclosure, according to the position of each pose data, each pose data is grid-divided, including: According to a preset spatial division step, the working space of the humanoid robot is divided into a plurality of voxel regions; According to the position of each pose data, the pose data located in the same voxel region is divided into the same bucket.

[0022] In an example embodiment of the present disclosure, the method further includes: According to a preset strategy, the spatial division step is adjusted; The preset strategy includes at least one of a change in a training phase, a task requirement of the humanoid robot, or a working space characteristic.

[0023] In an example embodiment of the present disclosure, the method further includes: The number of pose data in each bucket is counted to obtain a sample number of each bucket; According to a preset coefficient and the sample number of each bucket, an initial sampling weight corresponding to each bucket is determined.

[0024] In an example embodiment of the present disclosure, according to the preset coefficient and the sample number of each bucket, the initial sampling weight corresponding to each bucket is determined, including: wherein, is the initial sampling weight corresponding to each bucket, K is the preset coefficient, is the sample number of each bucket, represents a pose data set in each bucket, and each corresponds to a bucket.

[0025] According to a second aspect of the present disclosure, a humanoid robot motion control method is provided, including: Obtaining pose data of the end of the robot arm; Inputting the pose data into a pre-trained humanoid robot motion control model, and outputting an action strategy for controlling the motion of the humanoid robot based on the humanoid robot motion control model; The humanoid robot motion control model is obtained by the method for training a humanoid robot motion control model based on reinforcement learning in the first aspect of the present disclosure.

[0026] According to a third aspect of the present disclosure, a device for training a humanoid robot motion control model based on reinforcement learning is provided, comprising: a pose data calculation module configured to randomly sample from the joint space of the humanoid robot and calculate a plurality of pose data of the end of the robot arm based on the sampling results; a pose data division module configured to divide each pose data into a plurality of buckets based on the position of each pose data, each bucket corresponding to a set of pose data within a position range; a sampling weight adjustment module configured to configure an initial sampling weight for each bucket and adjust the initial sampling weight of the bucket whose position range is in a specified area to obtain a target sampling weight for each bucket; a control model training module configured to sample based on the target sampling weight of each bucket, obtain an instruction data set, and input the instruction data set to the humanoid robot motion control model based on reinforcement learning for training.

[0027] According to a fourth aspect of the present disclosure, a humanoid robot motion control device is provided, comprising: a pose data acquisition module configured to acquire pose data of the end of the robot arm; an action policy output module configured to input the pose data into a pre-trained humanoid robot motion control model and output an action policy for controlling the motion of the humanoid robot based on the humanoid robot motion control model; The humanoid robot motion control model is obtained by the method for training a humanoid robot motion control model based on reinforcement learning in the first aspect of the present disclosure.

[0028] According to a fifth aspect of the present disclosure, an electronic device is provided, comprising: a processor; and a memory having computer readable instructions stored thereon, the computer readable instructions being executed by the processor to implement the method of the above embodiments.

[0029] According to a sixth aspect of the present disclosure, a humanoid robot is provided, comprising: a processor; and a memory having computer readable instructions stored thereon, the computer readable instructions being executed by the processor to implement the method of the above embodiments.

[0030] According to a seventh aspect of the present disclosure, a computer-readable storage medium is provided, and the computer-readable storage medium has stored thereon computer program code instructions which, when invoked by a processor, cause the processor to perform the method of the above-mentioned embodiments.

[0031] From the above technical solutions, the present disclosure has at least one of the following advantages and positive effects: The present disclosure obtains pose data of the end of the robot arm by randomly sampling from the joint space of the humanoid robot, and provides initial sample basis covering different spatial positions for training. Further, a plurality of sub-buckets are constructed based on the positions of the pose data, and adjustable sampling weights are configured for each sub-bucket, so that the training process can allocate pose data of different position regions according to the sampling weight of each sub-bucket to participate in training. By adjusting the sampling weight of the specified region corresponding to the sub-bucket, the proportion of the pose data in the specified region in the training sample can be adjusted without increasing the overall training data size, thereby balancing the distribution density of the training sample in the space. Therefore, the humanoid robot motion control model finally trained can obtain more balanced policy learning effect in the whole space range, so that the accuracy distribution of the control policy in each position region is more consistent, and the situation of accuracy decline of pose tracking in the specified region is reduced. BRIEF DESCRIPTION OF DRAWINGS

[0032] In order to more clearly illustrate the technical solutions in the embodiments of the present disclosure, the drawings needed to be used in the embodiments or prior art description will be briefly introduced below. Obviously, the drawings in the following description are only some embodiments of the present disclosure, and other drawings can also be obtained by those skilled in the art without creative labor.

[0033] Figure 1 A system architecture diagram is shown, which can apply the humanoid robot motion control model training method and humanoid robot motion control method based on reinforcement learning in the embodiments of the present disclosure.

[0034] Figure 2 A flowchart of a humanoid robot motion control model training method based on reinforcement learning in an embodiment of the present disclosure is shown.

[0035] Figure 3 A flowchart of calculating pose data in an embodiment of the present disclosure is shown.

[0036] Figure 4 A flowchart of calculating pose data in another embodiment of the present disclosure is shown.

[0037] Figure 5 A flowchart of grid division of pose data in an embodiment of the present disclosure is shown.

[0038] Figure 6 A bucketing diagram is shown in the embodiments of the present disclosure.

[0039] Figure 7 A flow diagram of determining initial sampling weights is shown in the embodiments of the present disclosure.

[0040] Figure 8 A flow diagram of adjusting initial sampling weights of buckets whose position ranges are in a specified area is shown in the embodiments of the present disclosure.

[0041] Figure 9 A first bucketing diagram is shown in the embodiments of the present disclosure.

[0042] Figure 10 A flow diagram of adjusting initial sampling weights of buckets whose position ranges are in a specified area is shown in the embodiments of the present disclosure.

[0043] Figure 11 A flow diagram of adjusting initial sampling weights of buckets whose position ranges are in a specified area is shown in the embodiments of the present disclosure.

[0044] Figure 12 A flow diagram of adjusting target sampling weights is shown in the embodiments of the present disclosure.

[0045] Figure 13 A flow diagram of a humanoid robot motion control method is shown in the embodiments of the present disclosure.

[0046] Figure 14 A block diagram of a humanoid robot motion control model training apparatus based on reinforcement learning is shown in the embodiments of the present disclosure.

[0047] Figure 15 A block diagram of a humanoid robot motion control apparatus is shown in the embodiments of the present disclosure.

[0048] Figure 16 A humanoid robot diagram is shown in the embodiments of the present disclosure.

[0049] Figure 17 A structural diagram of a computer system of an electronic device suitable for implementing the embodiments of the present disclosure is shown.

[0050] Figure 18 A schematic diagram of a computer readable storage medium in the embodiments of the present disclosure is shown. DETAILED DESCRIPTION

[0051] In the description of the present disclosure, the terms "first", "second" are only used for description, and do not indicate relative importance or imply the number of technical features. Therefore, the "first", "second" features can be explicitly or implicitly included at least one of the features. The meaning of "multiple" is at least two, unless otherwise explicitly limited.

[0052] First, the related terms involved in the example embodiments of the present disclosure are explained and described: Joint space: refers to a parameter space formed by taking the motion variables of each joint of the humanoid robot as the coordinate axes, where each dimension corresponds to the angle, displacement or combination of one joint. A value point in the joint space corresponds to the specific joint configuration state of the robot at a certain time.

[0053] Pose data: refers to the spatial state information of the robot arm end in the workspace calculated by the forward kinematics of the joint configuration in the joint space. The pose data at least includes the end position coordinates and end attitude direction information, which can be represented by Euler angles or quaternions. The pose data is used to generate the instruction data set in the training process, as the input of the humanoid robot motion control model.

[0054] Bucketing: refers to the processing method of classifying the pose data into multiple discrete sets according to the position distribution of the pose data in the workspace according to the preset spatial division rule. Each bucket corresponds to a position range in the workspace and contains the pose data in the position range.

[0055] Initial sampling weight: refers to the basic sampling weight of each bucket before region division or performance adjustment, which is used to represent the basic sampling proportion of the pose data in the corresponding bucket in the instruction data set construction process.

[0056] Instruction data set: refers to the training instruction set extracted and arranged from the pose data based on the bucket and its corresponding sampling weight, which is used as the input of the robot motion control model in the reinforcement learning training process.

[0057] Workspace: refers to the workspace range that the robot arm end can reach under the constraints of physical structure and joint motion, which is jointly limited by joint limits, link lengths and kinematic structure. The workspace interval defines the sampling range of the bucket and the pose data, which can provide spatial boundaries for the training data.

[0058] Edge region: refers to the position region near the boundary of the workspace. The pose in this region usually corresponds to the end position where the joint is close to the limit, the arm is close to the maximum extension or folding state.

[0059] Center region: refers to the position region located inside the workspace and away from the edge region, the joint configuration corresponding to the pose in this region usually has greater redundancy and better reachability.

[0060] Boundary sampling weight: refers to the sampling weight additionally set or superimposed for the bucket corresponding to the edge region, used to improve the sampling proportion of the edge region pose data in the instruction data set construction process.

[0061] Target sampling weight: refers to the sampling weight finally determined for each bucket on the basis of the initial sampling weight, combined with the boundary sampling weight and other adjustment factors (such as center region weight reduction, specific task region strengthening, etc.), used for actual execution of instruction data set sampling.

[0062] Performance compensation weight: refers to the weight item introduced by further adjusting the sampling weight corresponding to part of the buckets according to the training result or control effect evaluation, used to improve the control accuracy or stability of the model in the corresponding space region.

[0063] Joint configuration: refers to the specific value combination of each joint of the humanoid robot at a certain moment, which can uniquely determine the overall posture of the robot and the pose state of its end in the workspace. The joint configuration can be represented by joint angle or displacement, and is the basis for generating pose data and instruction data set.

[0064] Figure 1 A system architecture diagram of the method for training a humanoid robot motion control model based on reinforcement learning and the method for controlling the motion of a humanoid robot in the embodiments of the present disclosure is shown. As shown in Figure 1 , the system architecture 100 can include a terminal device 101, a robot 102, a network 103, and a server 104.

[0065] Among them, the terminal device 101 includes but is not limited to desktop computers, portable computers, smart phones and tablet computers, etc. The terminal device 101 is configured with a graphical user interface for configuring and managing the humanoid robot motion control model training process. Specifically, the terminal device 101 can be used to set parameters related to joint space sampling, pose bucketing method, sampling weight adjustment rule, and reinforcement learning training process, and can be used to display model performance indicators, sampling distribution, and training state information during the training process.

[0066] The robot 102 can be a humanoid robot, which is equipped with inertial measurement devices, joint angle encoders and other proprioceptive sensors for collecting its joint state, posture information and motion state data. The robot 102 at least includes a main controller and joint drivers, wherein the main controller is configured to deploy and run the trained humanoid robot motion control model based on reinforcement learning, and generate action strategies based on the real-time proprioceptive information. The joint drivers are configured to receive the action strategies and drive the corresponding joints to perform motions accordingly, so as to achieve the motion control of the humanoid robot.

[0067] The server 104 is deployed with a training module for training the humanoid robot motion control model. The training module is configured to randomly sample from the joint space of the humanoid robot, and calculate the pose data of the end of the robot arm based on the sampling results; perform bucketing on the pose data based on the position distribution of the pose data in the work space; configure and adjust the sampling weights for each bucket to obtain target sampling weights; and sample the pose data based on the target sampling weights to construct an instruction data set, and input the instruction data set into the humanoid robot motion control model based on reinforcement learning to complete the model training. After the model training is completed, the server 104 can distribute the trained motion control model to the robot 102 for real-time action strategy generation during its running process.

[0068] The network 103 is used to provide a communication link medium between the terminal device 101, the robot 102 and the server 104, so as to realize the transmission of sensor data, the distribution of model parameters and the interaction of training process control information. The network 103 can include various connection types, such as wired, wireless communication links or optical fiber cables, etc.

[0069] It should be understood that Figure 1 The number and type of terminal devices, robots, networks and servers in the system architecture 100 are only illustrative. According to the implementation needs, there can be any number and any type of terminal devices, robots, networks and servers, for example, the server can be a cloud server.

[0070] Through the cooperative operation of each component in the system architecture 100, the pose data acquisition based on joint space random sampling, the bucketing processing based on the position distribution in the work space, the instruction data set construction based on the sampling weight adjustment, and the training and deployment of the humanoid robot motion control model based on reinforcement learning can be realized, so as to improve the motion control effect of the model in different work space positions.

[0071] The example embodiments of the present disclosure provide a method for training a humanoid robot motion control model based on reinforcement learning. Referring to FIG. 2, Figure 2 The method can include the following steps S201 to S204: Step S201, random sampling is performed from the joint space of the humanoid robot, and a plurality of pose data of the end of the robot arm is calculated based on the sampling result; Step S202, based on the positions of the pose data, the pose data is divided into a plurality of buckets, and each bucket corresponds to a set of pose data within a position range; Step S203, initial sampling weights are configured for each bucket, and the initial sampling weights of the buckets whose position ranges are in the specified area are adjusted to obtain target sampling weights of each bucket; Step S204, sampling is performed based on the target sampling weights of each bucket to obtain an instruction data set, and the instruction data set is input to the humanoid robot motion control model based on reinforcement learning for training.

[0072] The present disclosure adjusts the sampling weights of the buckets corresponding to the specified area, changes the proportion of the pose data in the specified area in the training sample without increasing the overall training data size, and improves the distribution balance of the training sample in the space. Therefore, the humanoid robot motion control model trained has more consistent control accuracy in each position area, and reduces the situation of local area pose tracking accuracy decline.

[0073] In the following, the method for training the humanoid robot motion control model based on reinforcement learning in the example embodiment will be described in detail.

[0074] In step S201, random sampling is performed from the joint space of the humanoid robot, and a plurality of pose data of the end of the robot arm is calculated based on the sampling result.

[0075] In the example embodiment of the present disclosure, random sampling can be performed on the joint space of the humanoid robot under the constraint condition of the preset joint motion range to generate a plurality of joint configurations. The preset joint motion range constraint condition refers to the allowed motion range set for each joint of the humanoid robot, including the minimum and maximum limits of the joint, as well as additional constraints related to physical structure, link length and safety requirements, etc. Random sampling can use uniform random sampling, weighted random sampling or limited random sampling, etc. to ensure that the sampling result meets the joint physical constraints, kinematic accessibility and safety limit requirements, and covers the entire joint space to avoid concentration in a few local areas.

[0076] Subsequently, each joint configuration is input to the forward kinematics model of the humanoid robot, and the pose data of the end of the robot arm in the work space under the corresponding joint configuration is calculated according to the spatial connection relationship and geometric parameters of each joint. For example, the pose data at least includes the position and attitude of the end of the robot arm in the three-dimensional space. The position refers to the coordinate position of the end of the robot arm in the three-dimensional space, and the attitude is used to represent the spatial orientation of the end of the robot arm.​​​

[0077] In this step, based on different joint space random sampling results, a set of pose data of the robot arm end can be obtained, which covers multiple workspace regions, thereby providing basic data support for subsequent bucketing, sampling weight configuration and training sample construction. Moreover, this step does not rely on manual demonstration or predefined trajectories, and can effectively improve the distribution diversity and coverage integrity of the pose data of the robot arm end in the workspace.

[0078] In some example embodiments, with reference to Figure 3 As shown in FIG. 3, step S201 can further include steps S301 and S302. Step S301: sampling the joint angles of the humanoid robot in a preset joint space sampling range to generate joint configurations corresponding to the joint angles.

[0079] Specifically, the humanoid robot arm includes multiple rotatable joints, each joint corresponding to at least one joint angle. The joint angle value range corresponding to each joint can be set in advance according to the structural characteristics, mechanical limits and safety constraints of each joint, thereby forming the joint space sampling range of the entire arm.

[0080] For example, in the joint space sampling range, the joint angles of the humanoid robot are jointly sampled to obtain multiple sets of joint angle combinations, each set of joint angle combinations constituting a complete joint configuration.

[0081] Through this step, multiple combined and diversified joint configurations can be generated, thereby providing basic data support for subsequent calculation of the pose data of the robot arm end by forward kinematics.

[0082] Step S302: based on the joint configurations, multiple pose data of the robot arm end are calculated.

[0083] Exemplarily, with reference to Figure 4 As shown in FIG. 4, step S302 can further include steps S401 and S402. Step S401: for each joint configuration, the position and attitude of the corresponding robot arm end are calculated by forward kinematics.

[0084] Each set of joint configurations generated in step S301 is taken as input, and the structural parameters of the humanoid robot arm and the connection relationship between the joints are combined to construct a corresponding forward kinematics calculation model. Based on the forward kinematics calculation model, each joint configuration is operated to obtain the position and attitude of the robot arm end in the workspace under the corresponding joint configuration.

[0085] In step S402, multiple pose data of the end of the robot arm are obtained according to the position and pose of the end of the robot arm.

[0086] The obtained position and pose of the end of the robot arm are combined to form the end pose data corresponding to the joint configuration, and multiple sets of end pose data corresponding to the joint configuration are summarized to form a pose data set of the end of the robot arm.

[0087] For example, a set of obtained pose data set D is: D {( pos , quat )} wherein, pos represents the position of the end of the robot arm in a three-dimensional space, which can include x , y , z values in three directions; quat represents the pose quaternion information of the end of the robot arm, which can include four components w , x , y , z ), w is a rotation angle.

[0088] By mapping the joint configuration in the joint space to the end pose data in the work space through forward kinematics, the correspondence between the joint configuration and the end space state can be realized, the coverage range and distribution diversity of the pose data in the work space can be improved, and a data basis for fine control of different space regions in the subsequent training process can be provided.

[0089] In step S202, based on the position of each pose data, each pose data is divided into multiple sub-buckets, and each sub-bucket corresponds to a set of pose data within a position range.

[0090] After obtaining multiple pose data of the end of the robot arm, the position in the pose data is used as the sub-bucket basis to divide each pose data.

[0091] Specifically, according to the work space range of the end of the robot arm, the work space can be divided along at least one space dimension, or can be combined and divided along multiple space dimensions, thereby forming multiple space regions, each space region corresponding to a sub-bucket.

[0092] Subsequently, each pose data is classified into a sub-bucket corresponding to the position range of the position of the pose data, so that the position parameters of the pose data in the same sub-bucket are located in the same or similar space range.

[0093] Through the bucketing processing, the pose data distributed in different spatial positions can be classified and organized, so that the pose data forms a plurality of mutually distinguished data sets in the spatial position dimension, which helps to avoid the problem of uneven distribution of pose data in space and improve the coverage effect and control accuracy of subsequent training process on each spatial region.

[0094] In some example embodiments, the plurality of buckets can be obtained by grid division of the pose data according to the positions of the pose data.

[0095] Specifically, the reachable position range of the end of the robot arm in the workspace is determined based on the positions of the pose data. Then, the workspace is discretized according to a preset grid division rule, and the workspace is divided into a plurality of grid regions. Correspondingly, each grid region corresponds to a position range and serves as a bucket.

[0096] For example, in the grid division process, a grid interval can be set along at least one coordinate axis direction of the space, or a grid interval can be set along multiple coordinate axis directions to form a two-dimensional or three-dimensional grid structure. Each grid region covers a continuous position interval for accommodating pose data located in the position interval. Subsequently, the corresponding pose data is classified into the bucket corresponding to the grid region where the pose data is located according to the position of the pose data.

[0097] By using the grid division method to bucket the pose data, the workspace can be regularly divided, so that the pose data in different spatial position ranges has a clear attribution relationship, which facilitates subsequent sampling weight configuration and data distribution adjustment based on the bucketing result, thereby improving the control accuracy and stability of different spatial regions in the model training process.

[0098] For example, referring to FIG. 5, Figure 5 The process of grid division of the pose data can include the following steps S501 and S502: Step S501: The workspace of the humanoid robot is divided into a plurality of voxel regions according to a preset spatial division step.

[0099] The workspace of the humanoid robot is a three-dimensional space region, the range of which can be determined by the reachable position of the end of the robot arm under the limit condition of each joint. According to the range of the workspace, the spatial division step is set, and the workspace is discretized along each spatial dimension of the workspace, so that the continuous workspace is divided into a plurality of mutually independent voxel regions.

[0100] The spatial division step length is used to limit the size of each voxel region in each spatial dimension, and the spatial division step length can be set according to the robot size, task accuracy requirement or training data scale, and the present disclosure does not limit this. Each voxel region is used to represent a local spatial range in the work space, and different voxel regions correspond to different position intervals.

[0101] In step S502, the pose data located in the same voxel region is divided into the same bucket according to the position of each pose data.

[0102] In this step, the voxel region corresponding to each pose data in the work space is determined according to the position of each pose data, and the pose data whose position falls into the same voxel region is divided into the same bucket, so that the pose data in the same bucket has the same or similar position range in the spatial position dimension.

[0103] For example, the pose data set D is divided into buckets according to the position pos of each pose data, and the bucket set H is obtained.

[0104] In a specific implementation, the work space along x, y , z three spatial directions is respectively divided into intervals to form a plurality of voxel regions. Each voxel region can be identified by a position key, such as the position key can be represented as , which is used to quickly locate and manage the data of the corresponding spatial region in the bucket set H . Among them, respectively represent the starting coordinate value of the voxel region in x, y , z direction. The position key is used as an index identifier of the bucket, which is used to distinguish the buckets corresponding to different spatial regions.

[0105] A bucket corresponds to all pose data satisfying the following position range: the x coordinate of the end position is located in the interval , y the coordinate is located in the interval , z the coordinate is located in the interval , wherein dx , dy , dz respectively represent the spatial division step length along x , y , z direction, which is used to determine the size of the voxel region.

[0106] By voxelization division and bucketing processing, the continuous pose data of the robot arm end can be converted into a data set organized based on a space region, so that different space position regions correspond to different data buckets, which helps to improve the distribution balance of the pose data in the workspace.

[0107] It should be noted that in the process of voxelizing the workspace of the humanoid robot, the space division step is used to determine the size of the voxel region, and different space division steps will directly affect the number of buckets and the size of the pose data contained in each bucket. In order to adapt the pose data division mode to different training needs or task scenarios, the space division step can be adjusted according to a preset strategy, such as dynamic or phased adjustment of the space division step. The preset strategy can include at least one of a change in the training phase, a task requirement of the humanoid robot, or a workspace characteristic.

[0108] For example, in the initial stage of model training, a larger space division step can be used to perform coarse-grained division of the workspace to reduce the number of buckets and improve overall training efficiency. As the training process progresses, the space division step can be gradually reduced to perform more fine-grained voxelization of the workspace to enhance the model's perception and control ability of local space positions.

[0109] For another example, when the humanoid robot performs a high-precision operation task or needs to focus on a local workspace region, the space division step can be adjusted according to the corresponding task requirement to make the relevant voxel region have a higher spatial resolution.

[0110] For another example, for regions in the workspace where the pose distribution is sparse or the boundary features are obvious, the space division step can also be adjusted in a targeted manner in combination with the workspace characteristics. The workspace characteristics refer to the differences in accessibility, pose distribution density, motion constraints, and control difficulty of different space regions within the reachable space range of the robot arm end, such as at least including space accessibility characteristics, pose distribution characteristics, and boundary and limit characteristics. For example, for a space region with dense pose distribution and relatively stable control, a larger space division step can be used to reduce the number of buckets and improve training efficiency. For a space region with sparse pose distribution, located at the boundary of the workspace, or with high control difficulty, the space division step can be reduced to make the corresponding voxel region have a higher spatial resolution, thereby enhancing the model's perception ability and control accuracy for such space positions.

[0111] By further adjusting the space division step, the pose data bucketing process can have greater flexibility and adaptability, thereby improving the control accuracy and robustness of the model in different task scenarios and space regions while ensuring training efficiency.

[0112] Reference Figure 6 As shown in FIG. 6, a bucketing diagram is given. Specifically, the workspace 601 of the humanoid robot is divided into multiple buckets 602. Each bucket 602 corresponds to a set of pose data within a position range. By counting the pose data in each bucket 602, the distribution of pose data in three-dimensional space can be obtained.

[0113] For ease of understanding, the distribution of buckets in one plane 603 is taken as an example for illustration. The color depth of different bucket regions is used to represent the number of pose data falling into the bucket. The darker the color, the more pose data in the corresponding bucket. The lighter the color, the less pose data in the corresponding bucket. Through this diagram, the distribution density and spatial difference characteristics of pose data in the workspace can be intuitively reflected.

[0114] It should be noted that the pose data statistics and sampling weight adjustment in the embodiments of the present disclosure are based on all the buckets formed in the robot workspace, rather than only processing the bucket results in a single plane.

[0115] In step S203, initial sampling weights are configured for each bucket, and the initial sampling weights of the buckets whose position ranges are in the specified region are adjusted to obtain target sampling weights of each bucket.

[0116] After completing the bucketing division of the pose data of the end of the robot arm, the initial sampling weight corresponding to each bucket is configured for each of the obtained multiple buckets. The initial sampling weight is used to represent the probability of being selected in the subsequent training sample construction or data sampling process. The initial sampling weight can be set in a unified manner, so that each bucket has the same sampling priority in the initial state, or it can be set according to the number of pose data in the bucket, the spatial position distribution, etc., which is not limited by the present disclosure.

[0117] In some example embodiments, with reference to FIG. 7, the process of determining the initial sampling weight corresponding to each bucket can include the following steps S701 and S702: Figure 7 Step S701, count the number of pose data in each bucket to obtain the sample number of each bucket.

[0118] For the obtained multiple buckets, the pose data contained in each bucket can be counted to obtain the total number of pose data in the bucket, i.e., the sample number of each bucket. The sample number is used to reflect the data density of the bucket in the spatial position. For example, some buckets are located in the central region of the workspace, and the number of pose data is large, while other buckets may be located in the edge region or sparse region, and the number of pose data is small. ​

[0119] By counting the sample quantity of each sub-bucket, the distribution of each spatial position area can be determined, thereby providing basic data for subsequent setting of sampling weights.

[0120] In step S702, initial sampling weights corresponding to each sub-bucket are determined according to a preset coefficient and the sample quantity of each sub-bucket.

[0121] In this step, the sample quantity of each sub-bucket is calculated with the preset coefficient, and the initial sampling weight of each sub-bucket is obtained. The preset coefficient can be used to normalize the weights of different sub-buckets, so that the initial sampling weight is applicable to the subsequent training sampling process.

[0122] For example, the initial sampling weight can be set according to the inverse or proportion of the sample quantity of each sub-bucket, so as to balance the sample distribution, that is, the initial sampling weight of the sub-bucket with a larger sample quantity is relatively smaller, and the initial sampling weight of the sub-bucket with a smaller sample quantity is relatively larger, thereby achieving balanced sampling of sparse areas and dense areas in the early stage of training. In this way, the problem of insufficient learning of the model for some spatial areas caused by uneven data distribution in the training process can be prevented.

[0123] For example, for each sub-bucket, we have: wherein, is the initial sampling weight corresponding to each sub-bucket, K is a preset coefficient, is the sample quantity of each sub-bucket, represents a set of pose data in each sub-bucket, and each corresponds to a sub-bucket.

[0124] The initial sampling weight can ensure that the sub-bucket with a smaller sample quantity has a relatively higher sampling probability in the training sampling process, so as to balance the data coverage of different sub-buckets.

[0125] After obtaining the initial sampling weight, the initial sampling weight of the sub-bucket located in the specified area can be further adjusted. In the example embodiment of the present disclosure, the specified area can include the edge area of the working space, the area with relatively sparse pose distribution, the key operation area frequently involved in the task execution process, or the area with higher control accuracy requirement, etc. Of course, the area can be determined according to the training requirement, the task target or the spatial characteristics. The adjustment method can be to increase or decrease the sampling weight of the sub-bucket, so as to change the participation proportion of the pose data in the sub-bucket in the subsequent sampling process, and the present disclosure does not limit this.

[0126] Finally, after the initial sampling weight configuration and the sub-bucket weight adjustment of the specified area, the target sampling weight of each sub-bucket is obtained, and the target sampling weight can be directly used in the subsequent sampling process of constructing the training data set.

[0127] By adjusting the weight of the specified region by bucketing, the pose data has differentiated sampling probability in different spatial position ranges, thereby guiding the training process to form a more reasonable data distribution in the spatial dimension, improving the model training effect and generalization ability.

[0128] In an example embodiment, the specified region is an edge region of the humanoid robot workspace. Accordingly, referring to Figure 8 As shown, the process of adjusting the initial sampling weight of the bucket whose position range is in the specified region can include the following steps S801 and S802: Step S801, identify the first bucket whose position range is in the edge region of the humanoid robot workspace.

[0129] In this step, the spatial position range corresponding to each bucket can be extracted based on the generated multiple buckets. By analyzing the boundary position or spatial geometric features of the workspace, the set of buckets located in the edge region of the workspace is determined. The edge region is usually sparse in the training sample distribution, or has a higher requirement for model control accuracy.

[0130] By identifying the edge region bucket, it can be determined which buckets need to be given additional attention in the subsequent sampling process to ensure the sufficiency of the model's learning of the edge position.

[0131] Referring to Figure 9 As shown, a first bucket schematic diagram is given. Among them, each bucket in the bucket region 901 represents the first bucket whose position range is in the edge region of the humanoid robot workspace, and these buckets need to be processed differently in the subsequent, such as adding boundary sampling weight, so that the sampling process takes into account the sample acquisition requirements of the edge region while covering the overall range of the workspace.

[0132] Step S802, add boundary sampling weight to the first bucket to obtain the target sampling weight of the first bucket.

[0133] Specifically, the target sampling weight of the first bucket can be calculated according to the initial sampling weight and the boundary sampling weight of the first bucket, that is, the pre-set boundary sampling weight is added to the initial sampling weight of each first bucket, thereby obtaining the target sampling weight of the first bucket.

[0134] For example, the target sampling weight of the first bucket is obtained by adding the initial sampling weight and the additional boundary sampling weight, that is: wherein, is the target sampling weight of the first bucket, is an initial sampling weight of the first bucket, reflecting a basic sampling probability of the bucket in the overall training data; is a boundary sampling weight, used to improve the priority of the bucket located in the edge region of the workspace in the training sampling process, so that the model obtains more samples in the edge region and enhances the learning ability of the edge position.

[0135] In this example, the boundary sampling weight can be a fixed value, or can be dynamically adjusted according to the spatial position of the bucket, the training stage or the task requirement. For example, the boundary sampling weight can be set as a constant in the initial stage to ensure that the edge bucket has sufficient sampling priority in the early training stage. In order to adapt to the improvement of the learning ability of the model in the training process, the boundary sampling weight can be gradually reduced with the increase of the training progress. For example, with the increase of the number of training iterations or the advancement of the training stage, the boundary sampling weight can be reduced according to a linear, exponential or other preset decay strategy, so that the sampling probability of the edge bucket gradually returns to the level of the initial sampling weight, thereby balancing the distribution of the training data in the edge region and other regions.

[0136] Through the flexible adjustment of the boundary sampling weight in this example, the target sampling weight of the first bucket can enhance the sampling frequency of the edge region pose data in the early training stage, and improve the learning effect of the model on the edge position. With the advancement of the training, the boundary sampling weight gradually decays, so that the model training relies more on the overall data distribution, thereby realizing the balance and stability of the training process. Therefore, the unbalanced problem of the training data in the spatial distribution can be effectively improved, and the generalization ability and control precision of the model in the whole workspace can be improved.

[0137] In an example embodiment, the specified region is a central region of the humanoid robot workspace. Accordingly, the reference Figure 10 As shown, the process of adjusting the initial sampling weight of the bucket whose position range is in the specified region can include the following steps S1001 and S1002: Step S1001, identifying a second bucket in each bucket whose position range is in the central region of the humanoid robot workspace.

[0138] In this step, the spatial position range information corresponding to each bucket can be extracted based on the generated multiple buckets. By analyzing the boundary position or spatial geometric features of the workspace, the set of buckets located in the central region of the workspace is determined. The central region is usually a region with dense pose data distribution in the workspace, and is also a position region frequently passed through by the robot arm in the task execution process.

[0139] By identifying the central region bucket, it can be determined which buckets are relatively abundant in the training data distribution, so that the sampling weight of each bucket is adjusted in the subsequent training process to balance the overall spatial sample distribution.

[0140] In step S1002, the initial sampling weight of the second bucket is reduced to obtain a target sampling weight of the second bucket.

[0141] Specifically, the initial sampling weight of the second bucket can be adjusted according to a preset reduction strategy to obtain the target sampling weight of the second bucket. The preset reduction strategy can adopt a fixed decrement, a proportional reduction, or a decreasing manner with the training progress, so that the probability of the second bucket being selected in the subsequent training sampling process is moderately reduced, and the weight reduction manner of the second bucket is not limited in the present disclosure.

[0142] By differentiating the adjustment of the sampling weight of the center area bucket, the distribution of the training data in the workspace is more balanced, while the sampling priority of the key edge area or the sparse area is maintained, which helps to balance the learning effect of different spatial positions in the training process and improve the control accuracy and robustness of the model in the whole workspace range.

[0143] In an example embodiment, the specified area is a target area in the humanoid robot workspace, and the target area can include at least one of a spatial area corresponding to a singular configuration of the robot, a spatial area corresponding to a specified operation position of the task, or a spatial area whose training error satisfies a preset condition.

[0144] The spatial area corresponding to the singular configuration of the robot refers to the corresponding position area of the singular configuration generated by a specific joint angle combination of the robot arm in the workspace. In such a spatial area, the kinematic Jacobian matrix of the robot arm tends to degenerate, the end pose is highly sensitive to joint angle changes, the joint redundancy is reduced or lost, and usually leads to a decrease in control stability. Therefore, the learning difficulty of the model in such a spatial area is higher, and the control accuracy may decrease or the motion may be unstable.

[0145] The spatial area corresponding to the specified operation position of the task refers to a target area where the humanoid robot needs to frequently complete grasping, assembly, interaction, or operation actions during the training phase or actual task execution process. Such a spatial area is usually determined by specific task requirements, and its spatial distribution can be located at any position in the workspace, which directly affects the task completion effect and operation success rate. It can be understood that the spatial area corresponding to the specified operation position of the task is not equal to the center area of the workspace, and its position can deviate from the center area, and even be located at the edge or local area of the workspace.

[0146] The space region where the training error meets the preset condition refers to a position region where the prediction error is large or the performance does not meet the preset standard, which is identified according to the error evaluation between the model output result and the expected control result in the model training process. Such a space region reflects the problem of insufficient learning or weak generalization ability of the model at a specific space position, and therefore the model can be subjected to targeted reinforcement learning by increasing the sampling weight of the pose data in the corresponding space region, so as to improve the control accuracy and stability of the model in the region.

[0147] By identifying the third sub-bucket in the target region, the space region that needs to be focused on in the training process can be determined, so that special processing is given when the sampling weight is adjusted.

[0148] It should be noted that the target region given above is only an example for illustrative purposes, and is used to represent the range of space positions that need to be focused on or treated differently in the work space, and does not constitute a limitation on the type of target region. In other embodiments, the target region can be flexibly set or adjusted according to the structural characteristics of the humanoid robot, task requirements, training effect or control performance, and the like.

[0149] Correspondingly, with reference to FIG. 11, the process of adjusting the initial sampling weight of the sub-bucket whose position range is in the specified region can include the following steps S1101 and S1102: Figure 11 Step S1101, identifying a third sub-bucket whose position range is in the target region of the work space of the humanoid robot.

[0150] In this step, the space position range information corresponding to each sub-bucket can be extracted based on the generated multiple sub-buckets. By analyzing the work space characteristics of the robot arm and the training task requirements, the region attribute of each sub-bucket is determined, so that the sub-bucket whose position range falls within the target region is identified and determined as the third sub-bucket.

[0151] Step S1102, adjusting the initial sampling weight of the third sub-bucket according to a preset weight adjustment factor to obtain a target sampling weight of the third sub-bucket.

[0152] Specifically, for the identified third sub-bucket, the preset weight adjustment factor can be applied to the initial sampling weight of the third sub-bucket to increase or decrease the probability of being selected in the training sampling process. The weight adjustment factor can be set according to the importance of the target region, the training stage or the model performance index, and is used to represent the adjustment amplitude of the sampling frequency of the sub-bucket.

[0153] ​The weight adjustment factor can take different numerical values. For example, for singular configuration regions or regions with large errors, the weight adjustment factor can be set to a value greater than 1 to increase the sampling frequency, so that the sub-bucket has a higher sampling probability during the construction of the instruction data set, thereby strengthening the learning of the model in the region. For non-critical positions in the task, the weight adjustment factor can be set to a value less than 1 to avoid excessive concentration of training samples.

[0154] In this embodiment, the target sampling weight of the third sub-bucket can prioritize the key spatial region or the difficult learning region during training, thereby improving the learning effect and control accuracy of the model at the target position. At the same time, in combination with the weight adjustment of the edge sub-bucket and the center sub-bucket in other example embodiments, a comprehensive and differentiated sampling strategy for each position region of the workspace can be implemented, the spatial distribution of training samples is optimized, and the generalization ability and stability of the model are improved.

[0155] In step S204, sampling is performed based on the target sampling weights of the sub-buckets to obtain an instruction data set, and the instruction data set is input to the humanoid robot motion control model based on reinforcement learning for training.

[0156] After obtaining the target sampling weights of the sub-buckets, data sampling can be performed according to the sub-bucket division and the target sampling weights of the sub-buckets. The sampling process can use random sampling, weighted random sampling, or other sampling strategies suitable for reinforcement learning training, so that the probability of selecting pose data in each sub-bucket is proportional to its target sampling weight, thereby ensuring that the training samples have a reasonable distribution in the spatial position.

[0157] In the example embodiments of the present disclosure, the training samples can be distributed according to different strategies such as edge region sub-buckets, center region sub-buckets, and target region sub-buckets, ensuring that the model obtains sufficient samples in difficult learning regions, key task regions, and edge or sparse regions, thereby improving the control accuracy and training effect of the model in different spatial regions.

[0158] After sampling is completed, the selected pose data is converted into a corresponding instruction data set. For example, the position, attitude, and joint angle combination calculated by inverse kinematics in the sampled pose data are converted into executable action instructions, and the required labels or reward information for training are attached to form complete training entries. All training entries are aggregated to form an instruction data set.

[0159] Through the instruction data set, the humanoid robot motion control model can learn the mapping relationship from the input to the desired action, and achieve precise control of the end of the robot arm.

[0160] Specifically, the instruction dataset is input to the humanoid robot motion control model for training. During the training process, the model generates action predictions based on the input state information and calculates a loss function or reward value by comparing with the actual actions in the training data. Subsequently, the model parameters are iteratively optimized using reinforcement learning algorithms such as policy gradient, Actor-Critic (Actor-Critic algorithm) or other algorithms suitable for continuous action space.

[0161] Through continuous iteration, the model gradually learns the optimal control strategy under different spatial position regions and different task conditions, thereby improving the precision, stability and generalization ability of robot motion. In addition, the embodiments of the present disclosure can also dynamically adjust the sampling strategy and training sample distribution according to the training phase or task requirements, further enhancing the learning effect of the model in key regions and improving the adaptability of the model to edge or sparse spatial positions.

[0162] In addition, after obtaining the target sampling weight of the first sub-bucket by performing step S801 and step S802, the target sampling weight can be further dynamically optimized to improve the learning effect of the model in the corresponding spatial region during the training process. It is shown that the process of adjusting the target sampling weight can include the following steps S1201 to S1203: Figure 12 Step S1201, sampling based on the target sampling weight of the first sub-bucket, and training the humanoid robot motion control model using the instruction dataset obtained by sampling.

[0163] According to the obtained target sampling weight of the first sub-bucket, pose data is selected from the corresponding sub-bucket according to weighted random or other applicable sampling strategies. The selected pose data is converted to form an instruction dataset, which is then input to the humanoid robot motion control model based on reinforcement learning for training.

[0164] Step S1202, calculating the training performance score of the first sub-bucket according to the evaluation result after training, and mapping the training performance score to a performance compensation weight.

[0165] The training evaluation result can include action error, trajectory deviation or task completion accuracy of the model in the first sub-bucket corresponding spatial position and other indicators. By comprehensively calculating these evaluation indicators, the training performance score is generated to reflect the learning effect of the model in the sub-bucket region.

[0166] ​Further, the training performance score is mapped to a performance compensation weight, which is used to adjust the sampling priority of the first sub-bucket in the subsequent sampling process. The mapping rule can be set according to the level of the training performance score. For example, if the training performance score is low, it indicates that the model has poor learning effect in the sub-bucket region, and the performance compensation weight can be set to a large value to increase the probability of selecting the sub-bucket in the next round of sampling. If the training performance score is high, it indicates that the model has good learning effect in the sub-bucket region, and the performance compensation weight can be set to a small value to reduce the frequency of repeated sampling of the sub-bucket.

[0167] For example, assuming that the value range of the training performance score is 0-100 and the value range of the performance compensation weight is 0-1, a linear mapping can be used, that is, wherein, is the performance compensation weight, S is the training performance score. For example, when the training performance score is 20, the corresponding performance compensation weight is 0.8; when the training performance score is 80, the corresponding performance compensation weight is 0.2. S

[0168] Through this mapping method, the space region with poor training performance will obtain a higher compensation weight, so as to be preferentially selected in the resampling, thereby realizing the intensive training of the difficult learning region.

[0169] In step S1203, the performance compensation weight is added to the target sampling weight of the first sub-bucket to obtain the final sampling weight of the first sub-bucket, and resampling is performed based on the final sampling weight of the first sub-bucket.

[0170] In this step, the performance compensation weight can be directly added to the target sampling weight of the first sub-bucket, weighted addition, etc., to obtain the final sampling weight of the first sub-bucket.

[0171] For example, the final sampling weight of the first sub-bucket can be represented as the sum of the initial sampling weight of the first sub-bucket , the boundary sampling weight , and the performance compensation weight . Through the weight configuration method in this example, the differential sampling of the training samples can be realized, so that the model evenly covers the edge, center and key region in the spatial position, and at the same time, the sub-bucket with poor training performance is intensively strengthened, thereby improving the precision, robustness and generalization ability of the humanoid robot motion control model in the whole workspace.

[0172] ​Further, according to the final sampling weight of the first sub-bucket, the pose data in the first sub-bucket is resampled to form a new instruction data set for further training of the humanoid robot motion control model.

[0173] Through the feedback-driven dynamic adjustment mechanism, the model can pay more attention to poor performance or key spatial regions during training, improve training efficiency, and enhance the generalization ability and control accuracy of the model in the whole workspace.

[0174] Of course, after obtaining the target sampling weight of the second sub-bucket by performing steps S1001 and S1002, and obtaining the target sampling weight of the third sub-bucket by performing steps S1101 and S1102, the target sampling weight of each sub-bucket can be further dynamically optimized. The dynamic optimization process can refer to steps S1201 to S1203, including sampling based on the target sampling weight, training the model, evaluating the training performance and generating the performance compensation weight, and then combining the performance compensation weight with the original target sampling weight to obtain the final sampling weight. Here, it will not be described in detail.

[0175] In this way, the final sampling weight of each sub-bucket can fully reflect its importance and model learning needs during training, achieving dynamic optimization and balanced use of each sub-bucket. Even if a sub-bucket is not divided into an edge or center region, its sampling priority can be adjusted through the performance compensation weight, thereby improving the overall training effect and model control accuracy.

[0176] The example embodiments of the present disclosure also provide a humanoid robot motion control method for, after the model training is completed, applying the trained humanoid robot motion control model to the humanoid robot running phase, so that the humanoid robot can generate a corresponding action strategy based on the input pose data to control the robot arm end to perform a predetermined operation in the workspace.

[0177] Reference Figure 13 As shown in the figure, the method can include the following steps S1301 and S1302: Step S1301, obtaining pose data of the robot arm end.

[0178] Specifically, the pose data of the humanoid robot arm end can be obtained in real time by the body perception sensor carried by the humanoid robot. The pose data can include coordinate information of the robot arm end in three-dimensional space, attitude angles (such as roll angle, pitch angle, yaw angle), and can additionally include speed or acceleration information according to task requirements to reflect the position and attitude of the arm end at a specific time or task state. By obtaining accurate real-time pose data, the model can be provided with accurate state input to support subsequent action decision and control execution.

[0179] In step S1302, the pose data is input into the pre-trained humanoid robot motion control model, and an action strategy for controlling the motion of the humanoid robot is output based on the humanoid robot motion control model.

[0180] The action strategy can include adjustment instructions for joint angles, speed and direction control of the end, torque distribution, or other control parameters for driving the robot arm to perform a specific action. By executing the action strategy, the humanoid robot can complete the motion task as expected in the actual workspace, ensuring that the end of the arm reaches the target position and attitude, and achieving precise control of the task target.

[0181] It should be noted that the humanoid robot motion control model in this example embodiment is obtained according to the method for training a humanoid robot motion control model based on reinforcement learning as shown in another example embodiment. Figure 2

[0182] In the example embodiments of the present disclosure, a device for training a humanoid robot motion control model based on reinforcement learning is also provided. As shown in Figure 14 The device for training a humanoid robot motion control model based on reinforcement learning 1400 includes a pose data calculation module 1401, a pose data division module 1402, a sampling weight adjustment module 1403, and a control model training module 1404, wherein: The pose data calculation module 1401 is configured to randomly sample from the joint space of the humanoid robot, and calculate a plurality of pose data of the end of the robot arm based on the sampling results; The pose data division module 1402 is configured to divide each pose data into a plurality of buckets based on the position of each pose data, each bucket corresponding to a set of pose data within a position range; The sampling weight adjustment module 1403 is configured to configure an initial sampling weight for each bucket, and adjust the initial sampling weight of the bucket whose position range is in a specified area to obtain a target sampling weight for each bucket; The control model training module 1404 is configured to sample based on the target sampling weight of each bucket, obtain an instruction data set, and input the instruction data set into the humanoid robot motion control model based on reinforcement learning for training.

[0183] The specific details of each module in the above device for training a humanoid robot motion control model based on reinforcement learning have been described in detail in the corresponding method for training a humanoid robot motion control model based on reinforcement learning, and therefore will not be described here.

[0184] In the example embodiments of the present disclosure, a device for controlling the motion of a humanoid robot is also provided. As shown in Figure 15 ​As shown, the humanoid robot motion control apparatus 1500 comprises a pose data acquisition module 1501 and an action policy output module 1502, wherein: The pose data acquisition module 1501 is configured to acquire pose data of the end of the robot arm. The action policy output module 1502 is configured to input the pose data into a pre-trained humanoid robot motion control model, and output an action policy for controlling the motion of the humanoid robot based on the humanoid robot motion control model. The humanoid robot motion control model is obtained according to the method for training a humanoid robot motion control model based on reinforcement learning as shown. Figure 2

[0185] The specific details of the modules in the above humanoid robot motion control apparatus have been described in detail in the corresponding method for controlling the motion of a humanoid robot, and thus will not be described here.

[0186] In the example embodiments of the present disclosure, a humanoid robot is also provided, which comprises a processor and a memory, and the memory stores computer readable instructions which, when executed by the processor, implement the above method. Referring to Figure 16 As shown, a schematic diagram of a humanoid robot is shown.

[0187] Referring to Figure 17 As shown, an electronic device capable of implementing the above method is also provided. The electronic device 1700 comprises a processor 1701 and a memory 1702, and the memory 1702 stores computer readable instructions which, when executed by the processor 1701, implement the method in the embodiments of the present disclosure.

[0188] In the example embodiments of the present disclosure, a computer readable storage medium is also provided, which stores computer program code instructions, and when the computer program code instructions are called by a processor, the processor executes the method as described in the embodiments.

[0189] Referring to Figure 18 As shown, a program product 1800 for implementing the above method according to the embodiments of the present disclosure is described, which can adopt a portable compact disc read-only memory (CD-ROM) and comprises program code, and can run on a terminal device, such as a personal computer. However, the program product of the present disclosure is not limited thereto, and in this document, the readable storage medium can be any tangible medium containing or storing a program, which can be used or combined with an instruction execution system, device or apparatus.

[0190] ​Those skilled in the art can clearly understand, through the description of the above examples, that the example embodiments described herein can be implemented by software or by software in combination with necessary hardware. Therefore, the technical solutions according to the embodiments of the present disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a U disk, a mobile hard disk, etc.) or on a network, and includes a number of instructions to enable a computing device (which can be a personal computer, a server, a touch terminal, or a network device, etc.) to perform the method according to the embodiments of the present disclosure.

[0191] Finally, the above preferred embodiments are only used to illustrate the technical solutions of the present application and are not limiting. Although the present application has been described in detail, those skilled in the art should understand that changes can be made to the form and details thereof without departing from the scope defined by the claims of the present application. The sizes of the drawings are not related to the actual size, and the actual size can be arbitrarily changed.

Claims

1. A method for training a humanoid robot motion control model based on reinforcement learning, characterized in that, include: Random sampling is performed on the joint space of the humanoid robot, and multiple pose data of the robot arm end are calculated based on the sampling results; Based on the position of each pose data, each pose data is divided into multiple buckets, and each bucket corresponds to a set of pose data within a position range; Configure initial sampling weights for each of the buckets, and adjust the initial sampling weights of the buckets whose position range is within the specified area to obtain the target sampling weights for each of the buckets; Sampling is performed based on the target sampling weights of each of the aforementioned buckets to obtain an instruction dataset, and the instruction dataset is then input into a reinforcement learning-based humanoid robot motion control model for training.

2. The method for training a humanoid robot motion control model based on reinforcement learning according to claim 1, characterized in that, The adjustment of the initial sampling weights of buckets within the specified area includes: Identify the first bucket whose location range is located in the edge region of the humanoid robot's workspace; Boundary sampling weights are added to the first bucket to obtain the target sampling weights for the first bucket.

3. The training method for a humanoid robot motion control model based on reinforcement learning according to claim 2, characterized in that, The step of adding boundary sampling weights to the first bucket to obtain the target sampling weights for the first bucket includes: The target sampling weight of the first bucket is calculated based on the initial sampling weight and the boundary sampling weight of the first bucket.

4. The method for training a humanoid robot motion control model based on reinforcement learning according to claim 2, characterized in that, The method further includes: Sampling is performed based on the target sampling weights of the first bucket, and the humanoid robot motion control model is trained using the sampled instruction dataset. The training performance score of the first bucket is calculated based on the evaluation results after training, and the training performance score is mapped to the performance compensation weight. The performance compensation weight is added to the target sampling weight of the first bucket to obtain the final sampling weight of the first bucket, and resampling is performed based on the final sampling weight of the first bucket.

5. The method for training a humanoid robot motion control model based on reinforcement learning according to claim 2, characterized in that, The boundary sampling weights are constants.

6. The training method for a humanoid robot motion control model based on reinforcement learning according to claim 2, characterized in that, The boundary sampling weights decrease as the training progresses.

7. The method for training a humanoid robot motion control model based on reinforcement learning according to claim 1, characterized in that, The adjustment of the initial sampling weights of buckets within the specified area includes: Identify the second bucket whose location range is located in the central area of ​​the humanoid robot's workspace; Reduce the initial sampling weight of the second bucket to obtain the target sampling weight of the second bucket.

8. The method for training a humanoid robot motion control model based on reinforcement learning according to claim 1, characterized in that, The adjustment of the initial sampling weights of buckets within the specified area includes: Identify the third bucket, whose location range is within the target area of ​​the humanoid robot's workspace; The initial sampling weight of the third bucket is adjusted according to the preset weight adjustment factor to obtain the target sampling weight of the third bucket. The target area of ​​the humanoid robot's workspace includes at least one of the following: the spatial area corresponding to the robot's singular configuration, the spatial area corresponding to the task-specified operation position, or the spatial area where the training error meets preset conditions.

9. The method for training a humanoid robot motion control model based on reinforcement learning according to claim 1, characterized in that, The process involves randomly sampling from the joint space of the humanoid robot and calculating multiple pose data of the robot arm's end effector based on the sampling results, including: Within a preset joint space sampling range, multiple joint angles of the humanoid robot are sampled to generate joint configurations corresponding to each joint angle; Based on the configuration of each joint, multiple pose data of the end effector of the robot arm are calculated.

10. The method for training a humanoid robot motion control model based on reinforcement learning according to claim 9, characterized in that, The calculation of multiple pose data of the robot arm end effector based on the configuration of each joint includes: For each of the aforementioned joint configurations, the position and orientation of the corresponding end effector of the robot arm are obtained through forward kinematics calculations; Based on the position and orientation of the robot arm's end effector, the plurality of pose data of the robot arm's end effector are obtained.

11. The training method for a humanoid robot motion control model based on reinforcement learning according to claim 1, characterized in that, Based on the position of each pose data, the pose data is divided into multiple buckets, including: Based on the position of each pose data, the pose data is divided into grids to obtain the multiple buckets.

12. The training method for a humanoid robot motion control model based on reinforcement learning according to claim 11, characterized in that, The step of dividing the pose data into grids based on the position of each pose data includes: The workspace of the humanoid robot is divided into multiple voxel regions according to the preset spatial division step size; Based on the position of each pose data, pose data located within the same voxel region are divided into the same bucket.

13. The training method for a humanoid robot motion control model based on reinforcement learning according to claim 12, characterized in that, The method further includes: The spatial partitioning step size is adjusted according to a preset strategy; The preset strategy includes at least one of the following: changes in the training phase, humanoid robot task requirements, or workspace characteristics.

14. The method for training a humanoid robot motion control model based on reinforcement learning according to claim 1, characterized in that, The method further includes: The number of pose data in each of the aforementioned buckets is counted to obtain the number of samples in each bucket; The initial sampling weight corresponding to each bucket is determined based on the preset coefficient and the number of samples in each bucket.

15. The method for training a humanoid robot motion control model based on reinforcement learning according to claim 14, characterized in that, The step of determining the initial sampling weight corresponding to each of the buckets based on a preset coefficient and the number of samples in each bucket includes: in, The initial sampling weights are for each bucket. K For preset coefficients, This represents the number of samples in each bin. This represents the set of pose data in each bucket, each... Each corresponds to a separate bucket.

16. A method for motion control of a humanoid robot, characterized in that, include: Acquire pose data of the robot arm's end effector; The pose data is input into a pre-trained humanoid robot motion control model, and an action strategy for controlling the humanoid robot's motion is output based on the humanoid robot motion control model. The humanoid robot motion control model is obtained by the reinforcement learning-based humanoid robot motion control model training method according to any one of claims 1 to 15.

17. A training device for a humanoid robot motion control model based on reinforcement learning, characterized in that, include: The pose data calculation module is used to randomly sample from the joint space of the humanoid robot and calculate multiple pose data of the robot arm end based on the sampling results. The pose data partitioning module is used to divide each pose data into multiple buckets based on the position of each pose data, and each bucket corresponds to a set of pose data within a position range. The sampling weight adjustment module is used to configure an initial sampling weight for each of the buckets and adjust the initial sampling weight of the buckets whose position range is within a specified area to obtain the target sampling weight of each of the buckets. The control model training module is used to sample based on the target sampling weights of each of the aforementioned buckets to obtain an instruction dataset, and then input the instruction dataset into the reinforcement learning-based humanoid robot motion control model for training.

18. A motion control device for a humanoid robot, characterized in that, include: The pose data acquisition module is used to acquire pose data of the robot arm's end effector. The motion strategy output module is used to input the pose data into the pre-trained humanoid robot motion control model, and output the motion strategy for controlling the humanoid robot's motion based on the humanoid robot motion control model. The humanoid robot motion control model is obtained by the reinforcement learning-based humanoid robot motion control model training method according to any one of claims 1 to 15.

19. An electronic device, characterized in that, include: processor; as well as A memory storing computer-readable instructions that, when executed by the processor, implement the method as described in any one of claims 1 to 16.

20. A humanoid robot, characterized in that, include: processor; as well as A memory storing computer-readable instructions that, when executed by the processor, implement the method as described in any one of claims 1 to 16.

21. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer program code instructions that, when invoked by a processor, cause the processor to perform the method as described in any one of claims 1 to 16.

Images