Walking control method, retrieval method, motion control strategy network training method, device, electronic equipment, humanoid robot and storage medium for autonomous retrieval of humanoid robots.
By inputting the target's relative position, orientation, environmental space, and body state information into the motion control strategy network, joint control commands are directly generated, solving the problem of insufficient target position control accuracy for humanoid robots in confined box-shaped scenarios, and achieving higher accuracy and stability in reaching the target position during retrieval.
Patent Information
- Application Number
- CN202611641492.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-10-22
- Publication Date
- 2026-11-17
AI Technical Summary
In warehousing scenarios where the container space is limited, humanoid robots struggle to maintain motion stability while improving the precision of control over the target position, especially when approaching the target position, where they are prone to overshooting or repeatedly correcting.
By using the target's relative position information, relative orientation information, environmental spatial information, and body state information as observation information, and inputting them into the motion control strategy network trained by reinforcement learning, joint control commands are directly generated, reducing intermediate command conversions. By combining real-time observation information to update and adjust joint movements, the accuracy of reaching the target position and the stability of control are improved.
It simplifies the control link from target information to joint movements, improves the robot's adaptability to the box entry area and internal space, reduces position deviation and repeated corrections, and improves the target position arrival accuracy and control stability of the homing movement.
Smart Images

Figure CN123113906A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of robotics, specifically to a walking control method, a retrieval method, a motion control strategy network training method, an electronic device, a humanoid robot, and a storage medium for autonomous retrieval of a humanoid robot. Background Technology
[0002] In scenarios involving transportation and storage, humanoid robots need to move to the target container for robot storage and adjust their position and orientation within the corresponding storage area of the container to prepare for subsequent storage operations. Since the entry area and internal space of the container are usually relatively limited, the robot not only needs to maintain stable movement during the storage process, but also needs to adapt to the spatial constraints around the container and accurately reach the target storage location.
[0003] One approach to achieving autonomous robot movement is through a hierarchical control architecture. A path planning module generates the walking path, and based on the robot's current state and the path, it generates a body speed command. The walking controller then tracks this speed command and generates joint movements. In this control method, the relationship between the target position and the robot's current position needs to be converted via the speed command before it is applied to the robot's walking control.
[0004] Humanoid robot walking involves alternating support, gait switching, and body balance adjustments. The accuracy of speed command tracking is not equivalent to the accuracy of reaching the target position. Especially when approaching the target location, the space available for the robot to continue moving and adjusting is limited. The upper-level control module needs to continuously correct the speed command based on the positional deviation, while the lower-level walking controller needs to execute corresponding actions based on the robot's current motion state. In cases of speed tracking deviation, control response lag, or gait switching issues, the robot may overshoot the target position, under-adjust its position, or repeatedly correct itself near the target position.
[0005] Therefore, how to improve the control accuracy of humanoid robots in the process of walking in the confined space of the container, while taking into account the constraints of the surrounding space and the stability of movement, is a technical problem that needs to be solved. Summary of the Invention
[0006] In view of the technical problems in related technologies, such as insufficient control accuracy of target position during walking and difficulty in balancing surrounding spatial constraints and motion stability when humanoid robots are in storage scenarios with limited space in the box entry area, this invention provides a walking control method, a humanoid robot autonomous storage method, a training method, device, electronic device, humanoid robot, and computer-readable storage medium for autonomous storage of humanoid robots. The aim is to input the target relative position information, relative orientation information, environmental spatial information, and the humanoid robot's body state information as observation information into the motion control strategy network, and have the motion control strategy network directly output joint control commands. This allows the positional relationship between the storage target and the robot to directly participate in the generation of joint movements, reducing response lag and error propagation introduced by intermediate command conversion and transmission links, and continuously adjusting subsequent actions based on updated observation information, thereby improving the target position arrival accuracy and control stability during storage walking.
[0007] To achieve the above objectives, according to one aspect of the present invention, a walking control method for autonomous retrieval of a humanoid robot is provided, comprising: acquiring target relative position information, relative orientation information, environmental spatial information, and the body state information of the humanoid robot; wherein, the target relative position information represents the positional relationship between the retrieval target position corresponding to the target box and the humanoid robot, and the relative orientation information represents the orientation relationship between the target box and the humanoid robot; inputting the target relative position information, relative orientation information, environmental spatial information, and body state information as observation information into a motion control strategy network trained through reinforcement learning, and the motion control strategy network directly outputting joint control commands for driving the humanoid robot to walk towards the retrieval target position based on the observation information; controlling the joint movement of the humanoid robot according to the joint control commands to drive the humanoid robot to perform retrieval walking; updating the observation information during the retrieval walking process, inputting the updated observation information into the motion control strategy network, and generating and executing subsequent joint control commands.
[0008] The environmental spatial information includes depth maps or local point cloud information representing the spatial structure of the target box and its entry area; the joint movement of the humanoid robot is controlled according to joint control commands to drive the humanoid robot to perform homing walking, including: during the process of the humanoid robot passing through the entry area, according to the joint control commands generated by the motion control strategy network based on the target relative position information, relative orientation information, environmental spatial information and body state information, for lower limb stepping and body posture adjustment, the humanoid robot is controlled to move towards the homing target position and adjust its body posture to adapt to the passable space of the entry area; the observation information is updated during the homing walking process, including: updating the observation information according to the state of the humanoid robot after performing lower limb stepping and body posture adjustment.
[0009] The joint control commands include lower limb joint control commands generated by the motion control strategy network based on the target distance and direction, relative orientation information, and body state information represented by the target relative position information, used for stride length adjustment and walking direction correction; controlling the joint movement of the humanoid robot according to the joint control commands to drive the humanoid robot to perform homing walking, including: when the humanoid robot approaches the homing target position, reducing the walking stride through the lower limb joint control commands, and controlling the body posture adjustment in combination with environmental spatial information during the walking process of reducing the stride; updating the observation information during the homing walking process, inputting the updated observation information into the motion control strategy network, generating and executing subsequent joint control commands, including: updating the observation information according to the state after the lower limb joint control commands are executed, inputting the updated observation information into the motion control strategy network, generating and executing joint control commands for subsequent position correction.
[0010] The method further includes: when the humanoid robot is far from the target location, generating joint control commands through a motion control strategy network for walking towards the target location with a larger stride; when the humanoid robot approaches the target location, reducing the walking stride through lower limb joint control commands, and adjusting the body posture in combination with environmental spatial information during the reduced stride walking process, including: when the humanoid robot approaches the target location, controlling the humanoid robot to walk with a smaller stride and adjust the body posture through lower limb joint control commands based on depth map or local point cloud information to adapt to the spatial constraints of the entered area.
[0011] In the steps of acquiring the target's relative position information, relative orientation information, environmental spatial information, and the humanoid robot's body state information, acquiring the target's relative position information and relative orientation information includes: acquiring two-dimensional image information of the target box and identifying visual markers set on the target box; determining the pose of the target box relative to the humanoid robot's current base based on the identification results of the visual markers, and determining the relative position vector of the homing target's position in the current base coordinate system and the relative yaw angle of the target box relative to the humanoid robot; and using the relative position vector and relative yaw angle as the target's relative position information and relative orientation information, respectively.
[0012] The visual marker is a QR code set inside the target box, and the target location corresponds to the center position of the QR code. Based on the recognition result of the visual marker, the pose of the target box relative to the current base of the humanoid robot is determined, and the relative position vector of the target location in the current base coordinate system and the relative yaw angle of the target box relative to the humanoid robot are determined. This includes: based on the recognition result of the QR code, determining the pose of the target box relative to the current base of the humanoid robot, determining the relative position vector of the center position of the QR code in the current base coordinate system, and the relative yaw angle of the target box, so as to obtain the target's relative position information and relative orientation information.
[0013] Specifically, during the homing process, the observation information is updated, and the updated observation information is input into the motion control strategy network to generate and execute subsequent joint control commands. This includes: when a QR code is recognized, as the position and orientation of the humanoid robot's base change, the relative position vector and relative yaw angle of the QR code's center position in the humanoid robot's current base coordinate system are updated according to the QR code recognition result; the updated relative position vector and relative yaw angle are input into the motion control strategy network to generate and execute subsequent joint control commands.
[0014] The joint control commands include joint target position commands or joint torque commands. Controlling the joint movement of the humanoid robot according to the joint control commands to drive the humanoid robot to perform homing walking includes: when the joint control command is a joint target position command, controlling the movement of the corresponding joint according to the joint target position command; when the joint control command is a joint torque command, driving the movement of the corresponding joint according to the joint torque command.
[0015] According to another aspect of the present invention, a method for autonomously returning a humanoid robot to a storage container is provided, comprising: acquiring target relative position information, relative orientation information, environmental spatial information, and the humanoid robot's body state information; wherein, the storage target position is a preparatory position within the target container for the humanoid robot to perform a sitting action, the target relative position information represents the positional relationship between the storage target position and the humanoid robot, and the relative orientation information represents the orientation relationship between the target container and the humanoid robot; the target relative position information, relative orientation information, environmental spatial information, and body state information are used as observation information and input into a motion control policy obtained through reinforcement learning training. The motion control strategy network directly outputs joint control commands to drive the humanoid robot to the target location based on observation information. The network then controls the robot's joint movements according to these commands, driving it through the target container's entry area to the seating preparation position. During this movement, the network updates the observation information and inputs it into the motion control strategy network to generate and execute subsequent joint control commands. When the updated observation information determines that the humanoid robot meets the preset positioning conditions corresponding to the seating preparation position, the network controls the robot to perform a seating action, allowing it to be seated and stored inside the target container.
[0016] The method of controlling the joint movement of the humanoid robot according to joint control commands to drive the humanoid robot to walk from the entrance area of the target box to the seating preparation position includes: adjusting the joint control commands output by the motion control strategy network according to the target relative position information during the humanoid robot's movement to the seating preparation position, so that the humanoid robot reduces its walking stride when approaching the seating preparation position; and adjusting the joint control commands according to the environmental space information when the humanoid robot passes through the entrance area of the target box, so as to control the humanoid robot to adjust its body posture and pass through the entrance area.
[0017] The method further includes: before the humanoid robot enters the target box through the entry area, controlling the humanoid robot to perform a turning action at the position corresponding to the entry area outside the target box, so that its back faces the inside of the target box; controlling the joint movement of the humanoid robot according to joint control commands to drive the humanoid robot to walk from the entry area of the target box to the sitting preparation position, including: after the turning action, generating joint control commands for backward walking by the motion control strategy network according to the updated target relative position information, relative orientation information, environmental space information and body state information; driving the humanoid robot to back into the target box according to the joint control commands for backward walking, and generating and executing subsequent joint control commands according to the updated observation information during the backward movement to adjust the position and orientation of the humanoid robot relative to the sitting preparation position.
[0018] The preset positioning conditions include that the magnitude of the positional deviation between the humanoid robot and the ready-to-sit position is less than a preset position threshold. The humanoid robot is determined to meet the preset positioning conditions corresponding to the ready-to-sit position based on the updated observation information, including: the state machine determines the positional deviation based on the updated target relative position information, and when the magnitude of the positional deviation is less than the preset position threshold, the humanoid robot is determined to meet the preset positioning conditions.
[0019] The process of controlling the humanoid robot to perform a sitting action includes: after the state machine determines that the humanoid robot meets the preset positioning conditions, outputting a sitting action sequence through the motion control strategy network, or switching the control of the humanoid robot to a sitting control strategy and outputting a sitting action sequence by the sitting control strategy; controlling the humanoid robot to perform a sitting action according to the sitting action sequence, so that the humanoid robot forms a sitting posture passively supported by the support structure inside the target box.
[0020] According to another aspect of the present invention, a training method for a motion control strategy network for autonomous retrieval of a humanoid robot is provided, comprising: acquiring target relative position information, relative orientation information, environmental spatial information, and the body state information of the humanoid robot in a retrieval training scenario; wherein, the target relative position information represents the positional relationship between the retrieval target position corresponding to the target box and the humanoid robot, and the relative orientation information represents the orientation relationship between the target box and the humanoid robot; inputting the target relative position information, relative orientation information, environmental spatial information, and body state information as observation information into the motion control strategy network to be trained, and the motion control strategy network directly outputs joint control commands based on the observation information; controlling the joint movement of the humanoid robot according to the joint control commands to perform retrieval walking with the retrieval target position as the target in the retrieval training scenario, and acquiring feedback information after executing the joint control commands; updating the parameters of the motion control strategy network using reinforcement learning based on the feedback information, and updating the observation information according to the state of the humanoid robot after executing the joint control commands; iteratively executing the processes of joint control command generation, joint movement control, feedback information acquisition, and parameter updating based on the updated observation information and the updated motion control strategy network to obtain the trained motion control strategy network.
[0021] Specifically, based on feedback information, reinforcement learning is used to update the parameters of the motion control strategy network, including: determining the target distance change information based on the target relative position information before and after the humanoid robot executes joint control commands; determining training feedback to characterize the degree of approach of the humanoid robot to the homing target position based on the target distance change information, and updating the parameters of the motion control strategy network based on the training feedback.
[0022] The process involves acquiring target relative position information, relative orientation information, environmental spatial information, and humanoid robot body state information in the warehousing training scenario. This includes acquiring target relative position information, relative orientation information, environmental spatial information, and body state information when the humanoid robot is at different relative distances from the warehousing target. The environmental spatial information includes depth maps or local point cloud information representing the target box and its entry area. Based on feedback information, reinforcement learning is used to update the parameters of the motion control strategy network. This includes updating the parameters of the motion control strategy network based on feedback information after the humanoid robot executes joint control commands at different target distances. This enables the trained motion control strategy network to generate joint control commands based on the target relative position information and environmental spatial information, thereby reducing the walking stride when approaching the warehousing target position and adjusting the humanoid robot's body posture when passing through the entry area.
[0023] According to another aspect of the present invention, a walking control device for autonomous warehousing of a humanoid robot is also provided, comprising a processor and a memory, wherein the memory stores a computer program, and when the computer program is executed by the processor, the above-described walking control method for autonomous warehousing of a humanoid robot is implemented.
[0024] According to another aspect of the present invention, a humanoid robot autonomous storage control device is also provided, including a processor and a memory, wherein a computer program is stored in the memory, and when the computer program is executed by the processor, the above-described humanoid robot autonomous storage method is implemented.
[0025] According to another aspect of the present invention, a training apparatus for a motion control strategy network is also provided, including a processor and a memory, wherein a computer program is stored in the memory, and when the computer program is executed by the processor, the above-described training method for the motion control strategy network for autonomous retrieval of a humanoid robot is implemented.
[0026] According to another aspect of the present invention, an electronic device is also provided, comprising: a processor; and a memory storing a computer program, wherein when the computer program is executed by the processor, the processor performs the above-described training method for the motion control strategy network for autonomous retrieval of a humanoid robot.
[0027] According to another aspect of the present invention, a humanoid robot is also provided, including a controller and a joint actuator connected to the controller. The controller is configured to execute the above-described walking control method for autonomous retrieval of the humanoid robot, or the above-described autonomous retrieval method for the humanoid robot, and to control the joint actuator to perform corresponding joint movements.
[0028] According to another aspect of the present invention, a computer-readable storage medium is also provided, on which a computer program is stored. When the computer program is executed by a processor, it implements the above-described walking control method for autonomous retrieval of a humanoid robot, the above-described autonomous retrieval method for a humanoid robot, or the above-described training method for a motion control strategy network for autonomous retrieval of a humanoid robot.
[0029] As can be seen from the above technical solution, the present invention possesses at least one of the following advantages and positive effects: This invention uses the target's relative position information, relative orientation information, environmental spatial information, and the humanoid robot's body state information as observation information. These are input into a motion control strategy network trained through reinforcement learning. The motion control strategy network then directly outputs joint control commands based on the observation information, allowing the positional relationship between the target and the robot to directly participate in the generation of joint movements. Compared to a hierarchical control method that first generates body velocity commands based on the target position and then generates joint movements through velocity tracking, this method simplifies the control chain from target information to joint movements, reducing response lag and error propagation introduced by intermediate command conversion and transmission links.
[0030] By combining relative orientation information, environmental spatial information, and body state information, the motion control strategy network can generate joint movements based on the position and orientation relationship between the robot and the target box, the surrounding spatial conditions, and the robot's own state. This provides a joint control basis for position adjustment, orientation adjustment, and stable walking in confined spaces, which helps improve the robot's adaptability to the box's entry area and internal space.
[0031] During the homing process, by updating the observation information and generating and executing subsequent joint control commands accordingly, the subsequent actions are continuously adjusted based on the changed target position relationship, environmental conditions, and robot state. This forms a feedback control process that connects observation and action execution, which helps to reduce the continuous impact of position deviations generated during movement on subsequent walking, improves the fine adjustment capability when approaching the homing target position, reduces repeated corrections near the target position, and improves the accuracy of reaching the target position and the stability of control during homing. Attached Figure Description
[0032] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings involved in the embodiments are briefly described below. The drawings only show some embodiments of the present invention, and those skilled in the art can obtain other drawings based on the description of the present invention.
[0033] Figure 1 A flowchart illustrating a walking control method for autonomous retrieval of a humanoid robot according to an embodiment of the present invention is shown.
[0034] Figure 2A schematic diagram illustrating the process of obtaining the target's relative position information and relative orientation information in an embodiment of the present invention is shown.
[0035] Figure 3 A flowchart illustrating a method for autonomous warehousing of a humanoid robot according to an embodiment of the present invention is shown.
[0036] Figure 4 This illustration shows a schematic diagram of the process by which the humanoid robot turns around outside the target box and retreats into the target box in an embodiment of the present invention.
[0037] Figure 5 The image shows two simulated images of the humanoid robot in a simulated environment: the initial state of its return walk outside the target box and the subsequent images of it walking towards the target box.
[0038] Figure 6 The image shows two simulated images in a simulated environment: one showing the humanoid robot continuing to approach the target box, and the other showing the robot turning around outside the target box.
[0039] Figure 7 The image shows two simulated images of a humanoid robot in a simulated environment: one after it enters the target box from the entry area, and the other after it enters the box.
[0040] Figure 8 The image shows two simulated images of a humanoid robot adjusting its posture and approaching its seat position within a target box in a simulated environment.
[0041] Figure 9 This diagram illustrates a state in which the humanoid robot is housed in a seated position inside the target box, according to an embodiment of the present invention.
[0042] Figure 10 A flowchart illustrating the training method for the motion control strategy network for autonomous retrieval of a humanoid robot is shown in an embodiment of the present invention.
[0043] Figure 11 A schematic diagram of the process for updating motion control strategy network parameters based on feedback information in an embodiment of the present invention is shown.
[0044] Figure 12 A schematic diagram of the structure of an electronic device according to an embodiment of the present invention is shown.
[0045] Figure 13 A schematic diagram of the controller and joint actuator of the humanoid robot in an embodiment of the present invention is shown.
[0046] Figure 14 An image of one structure of a humanoid robot according to an embodiment of the present invention is shown.
[0047] Figure 15 A schematic diagram of a computer-readable storage medium according to an embodiment of the present invention is shown. Detailed Implementation
[0048] In this invention description, the terms "first" and "second" are used for description only and do not indicate relative importance or imply the number of technical features. Therefore, the features referred to as "first" and "second" may explicitly or implicitly include at least one of those features. "Multiple" means at least two, unless otherwise expressly defined.
[0049] The main technical terms involved in the embodiments of this invention are explained in the table below. The following description is for the purpose of facilitating understanding of the embodiments of this invention.
[0050] The specific implementation methods of each embodiment are described below with reference to the accompanying drawings.
[0051] This invention relates to humanoid robots autonomously returning to a predetermined storage area. In some embodiments, the storage area is the internal space of a target container. The robot moves from outside the container to the target storage location, adjusting its position and orientation within the entry area and inside the container to prepare for subsequent placement and storage.
[0052] In some embodiments, the target container can be a shipping container for a humanoid robot to autonomously enter and store. The shipping container includes an outer container and a detachable inner container assembly. The outer container is formed by a bottom plate, back plate, side plates, an openable top cover, and a front plate, creating a storage cavity. The front plate, when opened, forms an opening for the robot to enter and exit. An outwardly rotating footplate can be provided at the opening, forming a transition ramp connecting the interior of the container to the external ground when unfolded. The inner container assembly may include an upper support member, a front support member, a rear support member, and a bottom support member to support or limit the robot's head, torso, arms, legs, and hips in the stored state. The bottom support member can support the robot's body after it sits down. Wheels can also be provided on the outer side of the outer container's back plate for movement after the container changes its placement. When the robot autonomously returns to the container, the top cover and front plate are open. The detachable supports along the entry path are removed as needed for passage. The robot enters the container via the footplate and opening and sits down in the designated seating position. Afterward, the appropriate supports can be assembled as needed for storage. The above-described box structure is used to illustrate the storage scenario of this embodiment and does not limit the box form to which this method is applicable.
[0053] This embodiment can be implemented by three parts: visual perception, relative position calculation, and motion control strategy. A red-green-blue (RGB) camera identifies the QR code inside the enclosure, while a depth camera captures depth maps or local point clouds of the enclosure opening and transition areas. The relative position calculation part calculates the target's relative position vector and relative yaw angle in the current base coordinate system. A motion control strategy network trained through reinforcement learning combines the above information with the body's state information to directly generate joint control commands. Related processing can be performed by the robot controller or collaboratively with external computing devices.
[0054] The following is combined Figures 1 to 15 The specific implementation of the solution provided in the embodiments of the present invention is explained.
[0055] refer to Figure 1 The homing control method can be executed by the robot's controller or a communication-connected computing device. The motion control strategy network directly generates joint control commands based on the target position, relative orientation, surrounding space, and the robot's state.
[0056] like Figure 1 As shown, the walking control method for autonomous retrieval of a humanoid robot provided in this embodiment of the invention includes the following steps S101 to S104: Step S101: Obtain the target's relative position information, relative orientation information, environmental spatial information, and the humanoid robot's body state information.
[0057] Step S102: The target's relative position information, relative orientation information, environmental spatial information, and body state information are used as observation information and input into the motion control strategy network obtained through reinforcement learning training. The network directly outputs joint control commands to drive the humanoid robot to walk towards the target location.
[0058] Step S103: Control the joint movement of the humanoid robot according to the joint control command to drive the humanoid robot to perform the return walking.
[0059] Step S104: During the return-to-base walking process, update the observation information, input the updated observation information into the motion control strategy network, and generate and execute subsequent joint control commands.
[0060] The following describes in detail the walking control method for autonomous retrieval of a humanoid robot in this embodiment.
[0061] The system acquires target relative position information, relative orientation information, environmental spatial information, and humanoid robot body state information. The target relative position information represents the positional relationship between the target location (corresponding to the target container) and the humanoid robot, while the relative orientation information represents the orientation relationship between the target container and the humanoid robot. This target relative position information, relative orientation information, environmental spatial information, and body state information are used as observation information and input into a motion control strategy network trained through reinforcement learning. Based on the observation information, the motion control strategy network directly outputs joint control commands to drive the humanoid robot to walk towards the target location. The system controls the joint movements of the humanoid robot according to these joint control commands to drive it to perform the homing movement. During the homing movement, the observation information is updated and input into the motion control strategy network to generate and execute subsequent joint control commands.
[0062] The retrieval target position refers to the location the humanoid robot needs to reach within the target container. In some embodiments, the target container is an aviation case for housing the humanoid robot, and the retrieval target position can be set inside the target container. Once the humanoid robot reaches this position, it can perform subsequent storage actions. Target relative position information characterizes the positional relationship between the retrieval target position and the humanoid robot, and may include the relative position vector of the retrieval target position in the humanoid robot's current base coordinate system. This relative position vector is determined by the position difference between the retrieval target position and the current base, its magnitude represents the distance between the retrieval target position and the humanoid robot, and its direction represents the orientation of the retrieval target position relative to the humanoid robot. Relative orientation information characterizes the orientation relationship between the target container and the humanoid robot, and may include the relative yaw angle of the target container relative to the humanoid robot. It is understood that the positional relationship between the retrieval target position and the humanoid robot is not fixed but changes continuously as the humanoid robot moves; therefore, the target relative position information needs to be continuously acquired or updated during the retrieval process.
[0063] Environmental spatial information is used to describe the spatial structure of the enclosure and the access area. It can be composed of depth maps or local point clouds collected by depth cameras to characterize the enclosure openings, borders, and passable spaces.
[0064] In some embodiments, an RGB camera acquires images and identifies visual markers, a depth camera acquires spatial data of openings, borders, and transition areas, and a joint encoder and inertial measurement unit provide the body state. The various data types are preprocessed to form observation information; when sampling frequencies differ, the most recent valid data can be selected and aligned with the body state information.
[0065] Body state information may include at least one of joint angles, joint angular velocities, trunk posture, and base position and velocity, used to adapt joint control commands to the current body state and support conditions.
[0066] The motion control strategy network, trained through reinforcement learning, uses the target's relative position, relative orientation, environmental space, and body state information as observation information to directly generate joint target position or joint torque commands. This motion control strategy network directly generates joint target position or joint torque commands based on the target's relative position, relative orientation, environmental space, and body state information.
[0067] In some embodiments, the motion control strategy network processes different types of observation information separately. For example, relative position vectors, relative yaw angles, joint angles, joint angular velocities, and body posture are organized into numerical features; the depth map is effectively cropped and scaled, or local point clouds are sampled or locally spatially encoded to obtain environmental features; then, the environmental features are combined with the numerical features as input to the strategy network. The strategy network outputs the corresponding target joint position or target joint torque according to the preset arrangement order of the controlled joints of the humanoid robot. When joint target position control is used, the target joint position is executed by the position control loop of the corresponding joint; when joint torque control is used, the target joint torque is executed by the torque control loop of the corresponding joint. The above observation processing method and network structure are optional implementation methods. The strategy network directly generates control quantities at the joint level without first generating body linear velocity or angular velocity commands for the walking controller to track.
[0068] The joint actuator drives the lower limbs to step according to the joint control commands, and coordinates with the torso posture adjustment to move the robot toward the target location.
[0069] During the return journey, the system continuously updates the target position, relative orientation, environmental space, and body status information, and generates subsequent joint control commands accordingly to correct walking deviations and achieve fine adjustments near the target.
[0070] During continuous control cycles, the latest relative position, relative yaw angle, body state, and available spatial observations are acquired and input into the policy network to generate joint control commands. After the robot moves, the relative positional relationship is recalculated in the next cycle; if visual sampling is slower than the joint control cycle, the most recent valid visual observation can be used temporarily until a new image arrives and is updated.
[0071] In some embodiments, the updating of environmental spatial information and joint motion control and observation information during entry into the area can be implemented as follows: the environmental spatial information includes depth maps or local point cloud information characterizing the spatial structure of the target box and its entry area; controlling the joint motion of the humanoid robot according to joint control commands to drive the humanoid robot to perform homing walking includes: during the process of the humanoid robot entering the area, according to the joint control commands generated by the motion control strategy network based on the target relative position information, relative orientation information, environmental spatial information and body state information, for lower limb stepping and body posture adjustment, controlling the humanoid robot to move towards the homing target position and adjust the body posture to adapt to the passable space of the entry area; updating the observation information during homing walking includes: updating the observation information according to the state after the humanoid robot performs lower limb stepping and body posture adjustment.
[0072] The entry area includes the box opening and the transition area outside the opening. When the robot enters the box through this area, it needs to control its stepping and body posture simultaneously, based on the available space.
[0073] The depth map reflects the opening height, width, and border position using pixel depth; the local point cloud reflects the surface of the box and the entry area using three-dimensional spatial coordinates. Both can be used as spatial observation inputs, and the specific method of selection is not limited.
[0074] During the process of the humanoid robot entering the area, the motion control strategy network generates joint control commands for lower limb stepping and body posture adjustment based on the target's relative position information, relative orientation information, environmental spatial information, and the robot's own state information. These joint control commands serve two purposes: first, they control the robot's lower limbs to step, enabling the robot to continue moving towards the target location; second, they control the robot's body posture to adapt to the passable space of the entry area. For example, when the humanoid robot approaches the opening of the target box, if the environmental spatial information indicates that the free space above the opening is relatively limited, the joint control commands can control the robot's torso to lean forward or lower its center of gravity while simultaneously controlling the lower limbs to continue stepping, so that the overall outline matches the passable space at the opening. In this way, the humanoid robot can pass through the entry area while walking, reducing the need for pausing and repeated adjustments at the box opening.
[0075] After stepping and adjusting body posture, the observation information is updated based on the changed base position, torso posture and joint status, and then the strategy network generates subsequent joint control commands.
[0076] In some embodiments, the lower limb joint control commands and stride adjustments and subsequent position corrections when approaching the target location are defined as follows: the joint control commands include lower limb joint control commands generated by the motion control strategy network based on the target distance and direction, relative orientation information, and body state information represented by the target relative position information, for stride adjustment and walking direction correction; controlling the joint movement of the humanoid robot according to the joint control commands to drive the humanoid robot to perform homing walking includes: when the humanoid robot approaches the target location, reducing the stride length through the lower limb joint control commands, and controlling the body posture adjustment in combination with environmental spatial information during the stride reduction walking process; updating the observation information during the homing walking process, inputting the updated observation information into the motion control strategy network, generating and executing subsequent joint control commands, including: updating the observation information according to the state after the lower limb joint control commands are executed, inputting the updated observation information into the motion control strategy network, generating and executing joint control commands for subsequent position correction.
[0077] Lower limb joint control commands are used to adjust step length and direction of travel. The target relative position vector provides the target distance and orientation, while relative orientation information is used for orientation correction.
[0078] As the target location approaches, the strategy network generates lower limb joint commands with smaller strides based on the reduced target distance, reducing the likelihood of crossing the target location in a single step.
[0079] Even when stride length decreases, the policy network still adjusts the aircraft's posture by incorporating depth maps or point clouds. For example, when the space above the enclosure is limited, the torso can be lowered while taking small steps to adapt to the space inside the enclosure.
[0080] After the lower limb joint commands are executed, the observation information is updated and input into the strategy network to generate joint control commands for subsequent position correction, so as to reduce residual position bias.
[0081] In some embodiments, when the humanoid robot is far from the target location, the following processing is performed, and the stride length and body posture adjustment methods are further defined when it approaches the target location: When the humanoid robot is far from the target location, joint control commands for walking towards the target location with a larger stride length are generated through a motion control strategy network; when the humanoid robot approaches the target location, the walking stride length is reduced through lower limb joint control commands, and during the walking process with a reduced stride length, the body posture is adjusted in combination with environmental spatial information, including: when the humanoid robot approaches the target location, based on the depth map or local point cloud information, the humanoid robot is controlled to walk with a smaller stride length and adjust its body posture through lower limb joint control commands to adapt to the spatial constraints of the area being entered.
[0082] In some embodiments, a distant stage and an approach stage can be distinguished based on a preset target distance value, which can be set according to the size of the enclosure and the passage conditions of the entry area.
[0083] When the robot is far from the target location and in an open area outside the container, the policy network can output joint control commands with larger strides to speed up the target approach process.
[0084] Upon approaching the opening of the enclosure, the depth map or local point cloud reflects the spatial constraints of the boundary and transition areas. Based on this, the policy network outputs joint commands that coordinate small steps and posture adjustments to control the robot to enter the enclosure.
[0085] In some embodiments, joint control commands may take the following forms and control joint movement accordingly: joint control commands include joint target position commands or joint torque commands; controlling the joint movement of the humanoid robot according to the joint control commands to drive the humanoid robot to perform homing walking includes: when the joint control command is a joint target position command, controlling the movement of the corresponding joint according to the joint target position command; when the joint control command is a joint torque command, driving the movement of the corresponding joint according to the joint torque command.
[0086] A joint target position command refers to a joint control command that uses the target position that the joint needs to reach as the control variable. When the joint control command is a joint target position command, the motion control strategy network outputs the target position that each joint needs to reach, and controls the movement of the corresponding joint according to the joint target position command, so that each joint moves to the corresponding target position. In some embodiments, the joint target position command can be input to the lower-level position control loop of the joint, which drives the joint movement based on the difference between the current angle of the joint and the target position. When using a joint target position command, the lower-level position control loop can be used to track the target joint position given by the network.
[0087] The joint torque command specifies the torque that the controlled joint should output, and drives the corresponding joint through the underlying torque control loop.
[0088] Joint target position commands and joint torque commands are optional output formats, both generated directly by the policy network based on observation information, rather than obtained through conversion of body velocity commands.
[0089] refer to Figure 2 As shown, Figure 2 A schematic diagram of the process for obtaining the target's relative position information and relative orientation information is shown. Figure 2 The steps can be performed by the humanoid robot's controller or by a computing device that is connected to the humanoid robot in communication.
[0090] like Figure 2As shown, the walking control method for autonomous retrieval of a humanoid robot provided in this embodiment of the invention includes the following steps S201 to S203 in the steps of acquiring target relative position information, relative orientation information, environmental space information, and the humanoid robot's body state information: Step S201: Obtain two-dimensional image information of the target box and identify the visual markers set on the target box.
[0091] Step S202: Based on the recognition results of the visual markers, determine the pose of the target box relative to the current base of the humanoid robot, and determine the relative position vector of the target location in the current base coordinate system and the relative yaw angle of the target box relative to the humanoid robot.
[0092] Step S203: The relative position vector and the relative yaw angle are used as the target's relative position information and relative orientation information, respectively.
[0093] The process of obtaining the target's relative position information and relative orientation information in this embodiment is described in detail below.
[0094] In the steps of acquiring target relative position information, relative orientation information, environmental spatial information, and humanoid robot body state information, acquiring target relative position information and relative orientation information includes: acquiring two-dimensional image information of the target box and identifying visual markers set on the target box; determining the pose of the target box relative to the current base of the humanoid robot based on the recognition results of the visual markers, and determining the relative position vector of the target location in the current base coordinate system and the relative yaw angle of the target box relative to the humanoid robot; and using the relative position vector and relative yaw angle as target relative position information and relative orientation information, respectively.
[0095] Two-dimensional image information can be acquired by an RGB camera mounted on the robot's head or torso and transmitted to the processor. Visual markers are set on the target housing, which in this embodiment can be located inside the housing, and are used to calculate the housing's pose.
[0096] After acquiring a 2D image, the robot identifies visual markers within it. It can rotate its head to scan the surrounding area and stop searching upon detecting a visual marker.
[0097] Upon recognition of visual markers, the pose of the target container relative to the humanoid robot's current base is determined based on the recognition results. This pose includes the target container's positional and orientational relationships relative to the humanoid robot. After obtaining the target container's pose, the relative position vector of the homing target's location in the current base coordinate system and the relative yaw angle of the target container relative to the humanoid robot are further determined. The current base coordinate system is a coordinate system established with the humanoid robot's current base position as the origin, and it is updated as the humanoid robot's base position and orientation change. The relative position vector is determined by the position difference between the homing target's location and the current base; its magnitude represents the distance between the two, and its direction represents the homing target's orientation relative to the humanoid robot. The relative yaw angle characterizes the deviation between the humanoid robot's current orientation and the target container's orientation, and it can be used to determine what orientation adjustments the humanoid robot needs to make during its movement towards the homing target's location.
[0098] In some embodiments, the camera intrinsic parameters, the camera's mounting relationship with its mounting component, the actual size of the visual marker, and the mounting relationship of the visual marker with respect to the target housing are first calibrated. When a visual marker is identified, the marker corner points or positioning pattern positions in the image are extracted, and the pose of the visual marker relative to the camera is calculated by combining the camera intrinsic parameters and the marker size. For cameras fixedly mounted on the robot base, the marker pose can be converted to the current base coordinate system based on the pre-calibrated pose transformation between the camera and the base. For cameras mounted on movable components such as rotatable heads, the pose transformation of the camera relative to the current base is updated based on the current head joint state, robot kinematics, and the pre-calibrated camera mounting relationship, and then the coordinate transformation is completed. This yields the pose of the target housing relative to the current base, as well as the relative position vector and relative yaw angle of the homing target position corresponding to the visual marker.
[0099] For example, let Indicates the robot's current base coordinate system, Represents the camera coordinate system, Represents the visual marker coordinate system. To represent a homogeneous pose transformation between coordinate systems, the following formula can be used to transform the visual marker pose to the current base coordinate system: In the formula, Indicates time Pose transformation of the visual marker coordinate system relative to the base coordinate system. Indicates time Pose transformation of the camera coordinate system relative to the base coordinate system Indicates time Pose transformation of the visual marker coordinate system relative to the camera coordinate system. (The above) , and The double-letter subscripts in the text indicate the target coordinate system and the source coordinate system, respectively; for fixed-mount cameras, This can be a fixed value obtained from calibration; for cameras that rotate with the head, Updated based on the current state of the head joints.
[0100] After obtaining the relative position vector and relative yaw angle, they are used as the target's relative position information and relative orientation information, respectively, so that the strategy network can generate joint control commands based on the position deviation and orientation deviation.
[0101] In some embodiments, the visual markers and warehousing target locations are configured as follows, and the target's relative position information and relative orientation information are determined accordingly: The visual marker is a QR code set inside the target box, and the target storage location corresponds to the center of the QR code. Based on the visual marker recognition results, the pose of the target container relative to the current base of the humanoid robot is determined, and the relative position vector of the target location in the current base coordinate system and the relative yaw angle of the target container relative to the humanoid robot are determined, including: Based on the QR code recognition results, the pose of the target box relative to the current base of the humanoid robot is determined, and the relative position vector of the center position of the QR code in the current base coordinate system and the relative yaw angle of the target box are determined, so as to obtain the target's relative position information and relative orientation information.
[0102] A QR code is a visual marker presented in the form of a two-dimensional barcode. In some embodiments, the QR code is placed inside the target container, with the target location corresponding to the center of the QR code. Corner points or positioning patterns of the QR code can be identified based on its image positioning features, and the pose of the QR code relative to the camera can be calculated by combining camera calibration parameters, the actual size of the QR code, and its installation relationship; then, the pose transformation between the camera and the current base is converted to the current base coordinate system; when the camera is mounted on a movable head, this pose transformation is updated according to the current head joint state. The encoded information of the QR code is used to identify the target marker and its associated container information, and by combining the image geometric features of the QR code, camera calibration parameters, and the actual size of the QR code, the position and orientation of the QR code in three-dimensional space are determined.
[0103] The observation information is determined based on the position vector of the QR code center in the current base coordinate system and the relative yaw angle of the box. The QR code center is a reference point for positioning. The actual sitting position of the robot can be determined based on the internal support structure of the box and the predetermined sitting posture, and is mapped to this reference point through a preset coordinate relationship.
[0104] In some embodiments, the updating of the relative position vector and relative yaw angle during the homing process, as well as the generation of subsequent joint control commands, are subject to the following limitations: During the homing process, the observation information is updated, and the updated observation information is input into the motion control strategy network to generate and execute subsequent joint control commands, including: When the QR code is recognized, as the position and orientation of the humanoid robot's base change, the relative position vector and relative yaw angle of the QR code's center position in the humanoid robot's current base coordinate system are updated according to the QR code recognition result. The updated relative position vector and relative yaw angle are input into the motion control strategy network to generate and execute subsequent joint control commands.
[0105] During the homing process, the humanoid robot's base position and orientation continuously change, and the current base coordinate system is updated accordingly. Therefore, the relative position vector and relative yaw angle of the QR code's center position in the current base coordinate system also continuously change. In some embodiments, the humanoid robot continuously acquires 2D image information during walking. When a QR code is recognized, the relative position vector and relative yaw angle of its center position in the current base coordinate system are recalculated based on the QR code recognition result. When the QR code temporarily leaves the camera's field of view, the target's relative position and relative orientation information can be updated based on the most recent valid visual positioning result, combined with the robot's subsequent body motion estimation. When the QR code is observed again, the new recognition result is used for correction. If the reliability of the positional relationship update result is insufficient, the target's relative position and relative orientation information are updated after a valid visual positioning result is acquired again.
[0106] The updated relative position vector and relative yaw angle are input into the strategy network to generate subsequent joint control commands and correct minor position and orientation deviations as the target position approaches.
[0107] refer to Figure 3 As shown, Figure 3 A flowchart illustrating a method for autonomous storage of a humanoid robot is shown. Figure 1 Based on the warehousing and walking control shown, the warehousing target position is set as the sitting preparation position inside the target box, and the humanoid robot is controlled to perform the sitting action after the preset positioning conditions are met.
[0108] like Figure 3 As shown, the humanoid robot autonomous storage method provided in this embodiment of the invention includes the following steps S301 to S305: Step S301: Obtain the target's relative position information, relative orientation information, environmental space information, and the humanoid robot's body state information; wherein, the target position is the sitting preparation position inside the target box for the humanoid robot to perform the sitting action.
[0109] In step S302, the target's relative position information, relative orientation information, environmental spatial information, and body state information are used as observation information and input into the motion control strategy network obtained through reinforcement learning training. The network directly outputs joint control commands to drive the humanoid robot to walk towards the target location.
[0110] Step S303: Control the joint movement of the humanoid robot according to the joint control command to drive the humanoid robot to walk from the entry area of the target box to the sitting position. During the walking process, update the observation information, input the updated observation information into the motion control strategy network, generate and execute subsequent joint control commands.
[0111] Step S304: Based on the updated observation information, determine whether the humanoid robot meets the preset positioning conditions corresponding to the sitting preparation position; if yes, proceed to step S305; if no, return to step S303.
[0112] Step S305: Control the humanoid robot to perform a sitting action, so that the humanoid robot is stored in the target box in a sitting position.
[0113] The judgment result of step S304 determines the subsequent control: if the preset positioning conditions are met, step S305 is executed; if not, the process returns to step S303, and joint control commands are generated and executed based on the updated observation information to reduce the positional deviation relative to the seating preparation position.
[0114] The method for autonomous retrieval of the humanoid robot in this embodiment is described in detail below.
[0115] The system acquires the target's relative position information, relative orientation information, environmental spatial information, and the humanoid robot's body state information. The target location is the preparatory position within the target container for the humanoid robot to perform a sitting action. The target's relative position information represents the positional relationship between the target location and the humanoid robot, while the relative orientation information represents the orientation relationship between the target container and the humanoid robot. The target's relative position information, relative orientation information, environmental spatial information, and body state information are used as observation information and input into a motion control strategy network trained through reinforcement learning. Based on the observation information, the motion control strategy network directly outputs joint control commands to drive the humanoid robot to walk towards the target location. The system controls the humanoid robot's joint movements according to these joint control commands, driving the robot through the target container's entry area to the preparatory position. During this movement, the observation information is updated and input into the motion control strategy network to generate and execute subsequent joint control commands. When the updated observation information determines that the humanoid robot meets the preset positioning conditions corresponding to the preparatory position, the system controls the humanoid robot to perform a sitting action, allowing the humanoid robot to be seated inside the target container.
[0116] The sitting preparation position refers to the location within the target container where the humanoid robot can perform a sitting action. In some embodiments, the humanoid robot is housed in a seated position inside the target container at the sitting preparation position. Setting the storage target position as the sitting preparation position ensures that the humanoid robot's storage movement has a clear objective that connects with the subsequent storage action. It is understood that the humanoid robot's seated storage requires not only that its entire body be located inside the target container, but also that its position and orientation when seated match the spatial distribution within the target container. Therefore, before performing the sitting action, it is advisable to ensure that both its position and orientation match the storage space inside the container. Positioning can be determined based on a preset position threshold, while orientation can be continuously corrected based on relative orientation information during previous walking control, or it can be confirmed separately in optional embodiments. By using the sitting preparation position as the storage target position, and having the motion control strategy network generate joint control commands based on the target relative position information and relative orientation information, the approach to the position and the adjustment of the orientation are completed simultaneously during the storage movement.
[0117] The humanoid robot is continuously controlled by the same motion control strategy network to complete tasks such as approaching the outside of the box, entering the area, and walking inside the box.
[0118] During the walking process, the observation information is updated and input into the motion control strategy network to generate and execute subsequent joint control commands. This walking process includes both the humanoid robot's movement outside the target enclosure and its movement through the entry area and after entering the target enclosure. As the humanoid robot walks, the distance and direction represented by the target's relative position information, the orientation relationship represented by the relative orientation information, the spatial structure represented by the environmental spatial information, and the motion state represented by the body state information all change. The updated observation information reflects these changes accordingly. Through continuous updating and input, the joint control commands output by the motion control strategy network are always generated based on the latest observation information.
[0119] During the walking process, the updated observation information is used to determine whether the robot meets the preset positioning conditions corresponding to the sitting preparation position; this determination and the walking control use the same target relative position data.
[0120] Once the humanoid robot meets the preset positioning conditions, it is controlled to perform a sitting motion, allowing it to be stored in a seated position inside the target container. The sitting motion refers to the action of changing the humanoid robot from a standing to a sitting position. In some embodiments, the humanoid robot gradually lowers its center of gravity during the sitting motion, and leans its torso backward or squats onto the support structure inside the target container. The purpose of storing the humanoid robot in a seated position inside the target container is that, compared to a standing position, the overall height is lower and the outline is more compact, allowing it to match the internal space of the target container, thus enabling the target container to store the humanoid robot after the lid is closed.
[0121] In some embodiments, when the humanoid robot walks from the entry area to the seating position, the joint control commands are adjusted in the following manner: Controlling the joint movements of the humanoid robot according to joint control commands to drive the humanoid robot to walk from the entry area of the target box to the seating preparation position includes: During the process of the humanoid robot walking towards the sitting position, the joint control commands output by the motion control strategy network are adjusted according to the target relative position information, so that the humanoid robot reduces its walking stride when it approaches the sitting position. When the humanoid robot passes through the entry area of the target box, the joint control commands are adjusted according to the environmental space information to control the humanoid robot to adjust its body posture and pass through the entry area.
[0122] As the system approaches the seating position, the strategy network reduces the stride length of the joint movements based on the target distance. This adjustment can occur either before entering the enclosure or inside the enclosure to reduce the likelihood of overshooting the target position.
[0123] When the robot passes through the box and enters the area, the policy network adjusts the joint commands according to the opening size and border position represented by the depth map or point cloud, and controls the robot's posture and stepping movements to complete the passage.
[0124] refer to Figure 4 As shown, Figure 4 The diagram illustrates the process of a humanoid robot turning around before entering a target box and then backing up to enter the target box, showing the alternative implementation methods for turning around and walking backward.
[0125] like Figure 4 As shown, the humanoid robot autonomous storage method provided in this embodiment of the invention further includes the following steps S401 to S403: Step S401: Before the humanoid robot enters the target box through the entry area, control the humanoid robot to perform a turning action at the position corresponding to the entry area outside the target box so that its back faces the inside of the target box.
[0126] In step S402, after the turning action, the motion control strategy network generates joint control commands for backward walking based on the updated target relative position information, relative orientation information, environmental space information and body state information.
[0127] Step S403: Drive the humanoid robot to move backward into the target box according to the joint control command for backward walking, and generate and execute subsequent joint control commands according to the updated observation information during the backward movement to adjust the position and orientation of the humanoid robot relative to the sitting preparation position.
[0128] The process of the humanoid robot entering the target box in this embodiment is described in detail below.
[0129] In some embodiments, before the humanoid robot enters the target box through the entry area, the humanoid robot is controlled to perform a turning motion at a position outside the target box corresponding to the entry area, so that its back faces the inside of the target box; the step of controlling the joint movement of the humanoid robot according to the joint control command to drive the humanoid robot to walk from the entry area of the target box to the sitting preparation position includes: after the turning motion, generating joint control commands for backward walking by the motion control strategy network according to the updated target relative position information, relative orientation information, environmental space information and body state information; driving the humanoid robot to back into the target box according to the joint control commands for backward walking, and generating and executing subsequent joint control commands according to the updated observation information during the backward movement to adjust the position and orientation of the humanoid robot relative to the sitting preparation position.
[0130] The robot can first walk to the corresponding position outside the box opening, and then turn around so that its back faces the inside of the box; one way to achieve this is to rotate about 180 degrees so that it can then back into the box.
[0131] After turning around, the robot retreats into the box with its back facing inward, reducing the need to turn around again in the narrow box and ensuring that its back faces the internal support structure when seated.
[0132] After the turning action, based on the updated target relative position information, relative orientation information, environmental spatial information, and body state information, the motion control strategy network generates joint control commands for backward walking. Since the humanoid robot's orientation changes after turning, if the direction in front of the humanoid robot is still taken as the forward direction, the humanoid robot will move away from the target box. Therefore, the direction of motion driven by the joint control commands for backward walking is opposite to the direction the humanoid robot is facing. Here, the joint control commands for backward walking are also directly output by the motion control strategy network based on the observation information, and the control method is consistent with the aforementioned retrieval walking, the difference being that the driven motion direction is backward. For implementations requiring backward retraction into the box, the observation state after turning, with the target box located behind the robot, and the backward walking process can be added to the retrieval training scenario, enabling the trained motion control strategy network to generate joint control commands for backward walking based on the corresponding observation information.
[0133] Driven by the backward walking command, the robot moves its lower limbs and trunk joints, allowing it to gradually enter the box while keeping its front facing outwards.
[0134] During the backward movement, subsequent joint control commands are generated and executed based on updated observation information to adjust the humanoid robot's position and orientation relative to the seating preparation position. This adjustment includes both position and orientation. Position adjustment allows the humanoid robot to continue approaching the seating preparation position; orientation adjustment ensures that after the backward movement is completed, the humanoid robot's orientation relative to the seating preparation position matches the orientation required for the subsequent seating action. For example, if the humanoid robot experiences an orientation shift during backward movement, subsequent joint control commands are generated based on the available updated relative orientation information to correct this shift. When the robot turns, causing the QR code inside the enclosure to be out of the forward camera's field of view, the most recent valid enclosure pose observation can be used, combined with base motion estimation, to continuously calculate the position and orientation relationship between the enclosure and the base; when the QR code becomes visible again, corrections are made based on the recognition results.
[0135] In some embodiments, during the process of retreating into the target container, the humanoid robot can also adjust its posture based on environmental spatial information to adapt to the internal space of the target container. For example, when the humanoid robot retreats, its body is within the range of the target container's opening. For the spatial constraints in the retreating direction, depth camera observations covering that direction can be used, or local point clouds of the container and opening acquired before turning can be combined with robot pose changes and transformed to the current base coordinate system to form usable environmental spatial information in the retreating direction. During the retreating movement, depth observation information covering the retreating direction is used, or local spatial information acquired before turning and transformed by pose transformation is used. The posture is constrained by environmental spatial information that reflects the free space above and to the sides of the opening, controlling the humanoid robot to retreat through the opening. By combining retreating movement with posture adjustment, the humanoid robot can adapt to the internal space of the container while maintaining movement.
[0136] In some embodiments, the preset positioning conditions and the corresponding positioning determination process are implemented in the following manner: The preset positioning conditions include that the modulus of the positional deviation between the humanoid robot and the ready-to-sit position is less than a preset positional threshold. Based on the updated observation information, it was determined that the humanoid robot met the preset positioning conditions corresponding to its seat preparation position, including: The state machine determines the position deviation based on the updated target relative position information, and determines that the humanoid robot meets the preset positioning conditions when the magnitude of the position deviation is less than the preset position threshold.
[0137] The state machine determines the positional deviation between the robot and its ready-to-be-placed position based on the updated target relative position information, and compares the magnitude of this deviation with a preset position threshold. The threshold can be determined based on the internal dimensions of the housing and the placement space.
[0138] When the magnitude of the positional deviation is less than a preset position threshold, the humanoid robot is determined to have met the preset positioning condition. This determination method limits the criteria for arrival to the distance between the humanoid robot and its ready-to-sit position. This condition determines whether the robot has entered the position tolerance range corresponding to the ready-to-sit position. The positional deviation is used to determine whether the humanoid robot has reached the preset position range, and the seating orientation can be further determined by combining relative orientation information. To ensure the robot sits in the required orientation, the orientation can be adjusted based on relative orientation information during the previous walking control process, or a confirmation condition for the relative yaw angle can be added in an optional embodiment. It is understood that the state machine's determination process and the motion control strategy network's generation of joint control commands occur in parallel. The state machine is used to determine the triggering timing of the seating action, and the motion control strategy network is used to generate joint control commands.
[0139] In some embodiments, in addition to the modulus of the position deviation being less than a preset position threshold, the remaining unobstructed space in front of the humanoid robot for continued movement is determined based on environmental spatial information obtained from the depth camera. When the remaining unobstructed space is less than a preset safety threshold, it can be used as one of the conditions for stopping further approach and triggering the sitting determination. For example, when the humanoid robot approaches a predetermined position inside the box along the homing direction, the remaining free distance corresponding to continuing to move in that direction can be estimated through depth information; when the remaining free distance is less than the safety threshold and the modulus of the position deviation is less than the preset position threshold, the state machine determines that the set homing endpoint condition has been reached. It should be noted that the remaining free distance being less than the safety threshold indicates that the space margin for continuing to move in the current direction is small, and does not directly indicate that the lateral or lower space required during the sitting process is sufficient. When actually triggering the sitting, it can also be confirmed based on the spatial matching relationship between the robot's predetermined sitting posture and the box support structure. The above determination of the remaining unobstructed space retains the implementation method of using less than the safety threshold as the trigger condition, and its measurement direction corresponds to the robot's current movement direction: when moving forward, it can be the front of the robot, and when moving backward, it should be the backward direction. The safety threshold can be set according to the target enclosure structure.
[0140] In some embodiments, after the humanoid robot meets the preset positioning conditions, it performs a sitting action in the following manner: controlling the humanoid robot to perform a sitting action includes: after the state machine determines that the humanoid robot meets the preset positioning conditions, outputting a sitting action sequence through a motion control strategy network, or switching the control of the humanoid robot to a sitting control strategy, and having the sitting control strategy output a sitting action sequence; controlling the humanoid robot to perform a sitting action according to the sitting action sequence, so that the humanoid robot forms a sitting posture passively supported by the support structure inside the target box.
[0141] The sitting sequence is a sequence of joint commands that causes the robot to change from a standing to a sitting posture. It can be generated by a motion control strategy network or an independent sitting control strategy. Support structures such as internal padding support the robot body after it has settled, providing passive support.
[0142] After the state machine determines that the humanoid robot meets the preset positioning conditions, there are two ways to obtain the sitting action sequence. First, the sitting action sequence is output through a motion control strategy network. In this method, after the state machine confirms that the preset positioning conditions are met, it can trigger the motion control strategy network to generate a sequence of joint control commands for executing the sitting action based on the current observation information. Using this method, the humanoid robot's walking and sitting processes are continuously controlled by the same motion control strategy network, resulting in a more coherent control process. Second, the control of the humanoid robot is switched to a sitting control strategy, and the sitting action sequence is output by this strategy. Here, the sitting control strategy refers to a control strategy different from the motion control strategy network used to output the sitting action sequence. Using this method, walking control and sitting control are handled by different control strategies, and the sitting control strategy can be designed separately for the sitting action, thereby improving the stability and consistency of the sitting action. This invention does not limit which method is used.
[0143] The robot lowers its body according to the seating sequence and sits on the support structure inside the box. This structure bears part of the robot's weight to maintain its folded posture.
[0144] The following is combined Figures 5 to 8 This describes a simulation of a humanoid robot autonomously returning to its storage location. Figures 5 to 8 To simulate the robot's autonomous retrieval process in a simulation environment, each image includes two images at different times, corresponding to the first to eighth times in the order of the actions. Each time only represents the sequential relationship of the states and does not limit the actual time interval between adjacent times. Figures 5 to 8 The robot is shown in sequence in different states: approaching from outside the box, turning around, retreating into the box and adjusting its posture. Figure 9 The illustration shows a humanoid robot housed in a seated position inside a target container, according to another embodiment. The specific configuration of the robot and container in the figure is merely illustrative.
[0145] refer to Figure 5 As shown, Figure 5 This includes simulated images from the first and second moments in the simulation environment, corresponding sequentially to the humanoid robot's initial state of returning to the target container and its state of walking towards the target container. Figure 5 In the first image, the target container is an aviation case with its lid fully open. The interior of the case provides space for the humanoid robot to be stored in a seated position. One or more visual markers are located inside the target container; in some embodiments, these are QR codes. The humanoid robot stands in front of the target container, offset from its centerline, with its front facing the direction of the target container. In this position, the humanoid robot acquires two-dimensional image information facing the target container and identifies the presence of visual markers within the image.
[0146] In some embodiments, the humanoid robot's head can rotate to scan the surrounding environment, continuously acquiring and recognizing two-dimensional image information during the rotation. Once a visual marker set on the target container is detected, the target is considered to have been captured. Upon detection of the visual marker, a local coordinate system is immediately established with the humanoid robot's current base as the origin, and the coordinates of the target's position in this local coordinate system are calculated. In one embodiment of the invention, the target's position corresponds to the center position of a QR code set inside the target container. Therefore, the coordinates of the QR code's center position in the humanoid robot's current base coordinate system can be calculated, and the positional deviation used for control is determined based on a preset geometric relationship between the target's reference position and the robot's controlled reference position. The target's relative position information is generated based on these coordinates, and the pose of the target container relative to the humanoid robot's current base is determined based on the visual marker recognition result. The relative yaw angle of the target container relative to the humanoid robot is then determined as relative orientation information.
[0147] The target's relative position and orientation information obtained through the above methods, along with the environmental spatial information acquired by the depth camera and the body state information acquired by the body sensor, are input as observation information into the motion control strategy network. It is understandable that during the target acquisition phase, the humanoid robot has not yet begun to move, and the positional relationship between the target and itself is mainly determined by the distance and orientation between the two. At this time, the distance represented by the target's relative position information is usually relatively large.
[0148] exist Figure 5 In the second image, the humanoid robot is walking towards the target box. One of the robot's legs is raised, while the other leg is supporting it. Its overall position is relative to... Figure 5 The first image shows a position closer to the target container. In this pose, the humanoid robot's base position changes, and the current base coordinate system is updated accordingly. Consequently, the relative position vector and relative yaw angle of the target location in the current base coordinate system are also updated. The updated relative position vector and relative yaw angle are input into the motion control strategy network, which outputs joint control commands for continuing to move towards the target location.
[0149] During this stage, the distance between the humanoid robot and the target location is relatively large, placing the robot in a state of considerable distance. In this state, the motion control strategy network generates joint control commands for walking towards the target location with a larger stride, based on the target distance and direction represented by the target relative position information. Since the humanoid robot is located outside the target enclosure at this time, its surrounding spatial constraints are relatively small. Walking with a larger stride can shorten the distance between the humanoid robot and the target location in a shorter time, thereby improving the overall efficiency of the homing movement.
[0150] refer to Figure 6 As shown, Figure 6 This includes two simulated images from the third and fourth time points in the simulation environment. In the first image, the robot approaches the enclosure further and continues to generate joint commands based on the updated observation information.
[0151] Simultaneously, the humanoid robot begins to enter the area corresponding to the target box's entry zone. The entry zone includes the target box's opening and the transition area located outside the opening. During this phase, the humanoid robot's relative yaw angle is used to correct its direction of travel, gradually aligning its front with the target box's opening.
[0152] exist Figure 6 In the second image, the humanoid robot has reached the position in front of the target box opening, corresponding to the entry area, and has completed a turning motion at that position. At this point, the humanoid robot's back is facing the inside of the target box, and its front is facing the outside of the target box, enabling it to subsequently enter the target box by backing up.
[0153] In some embodiments, when a humanoid robot is stored in a seated position, its legs typically need to extend or fold forward, while its back needs to lean against the support structure inside the target container. If the humanoid robot enters the target container facing forward, it needs to turn around inside the container to achieve a seated position. However, the space inside the target container is relatively limited, and turning around inside can easily interfere with the container's edges or padding. Therefore, performing a turning motion outside the target container before entering, so that the humanoid robot enters with its back facing inward, ensures that the posture after entering directly matches the direction required for seated storage, avoiding a space-consuming turning motion inside the container.
[0154] After the turning motion, based on the updated target relative position information, relative orientation information, environmental spatial information, and body state information, the motion control strategy network generates joint control commands for backward walking. Since the humanoid robot's orientation has changed, the direction of motion driven by the joint control commands for backward walking is opposite to the direction the humanoid robot is facing, i.e., the humanoid robot moves backward into the target box.
[0155] refer to Figure 7 As shown, Figure 7 The simulation includes two images from time points five and six in the simulation environment. In the first image, the humanoid robot is already positioned at the opening of the target box, its body surrounded by the frame of the target box, and the humanoid robot is moving from the outside of the target box to the inside of the box.
[0156] During this process, the humanoid robot is in the entry area of the target container. At this point, the role of environmental spatial information becomes apparent. In some embodiments, environmental spatial information includes depth maps or local point cloud information characterizing the spatial structure of the target container and its entry area. Depth maps and local point cloud information can be acquired by a depth camera. For the backward entry state illustrated, the spatial structure of the entry area can be acquired in real time by a depth camera capable of observing the backward direction, or calculated by combining the opening and local point cloud of the container obtained before turning with the current base pose. As the humanoid robot passes through the entry area, the motion control strategy network generates joint control commands for lower limb stepping and body posture adjustment based on the target relative position information, relative orientation information, environmental spatial information, and body state information. This controls the humanoid robot to move towards the target location and adjust its body posture to adapt to the passable space of the entry area. Through the same set of joint control commands, the humanoid robot synchronously adjusts its body posture while performing lower limb stepping. For example, if the environmental spatial information indicates that the passable space at the opening is restricted in the vertical direction, the joint control command can control the humanoid robot's torso to lean forward or lower its center of gravity while controlling the lower limbs to continue walking, so that the robot's outline falls within the passable space.
[0157] Local point clouds can cover the transition area outside the box opening, allowing the policy network to consider the landing space of this area when generating joint control commands.
[0158] In some embodiments, the depth camera can treat the ramp in front of the flight case entrance as part of the local environment, collecting depth information of the upper surface of the ramp and the vicinity of the case opening. When the motion control strategy network receives this environmental spatial information, it also receives the target relative position vector calculated from the QR code and the relative orientation information of the target case, and combines it with body state information such as joint angles and torso posture to output joint control commands, enabling the robot to adjust its stepping point and body posture according to the spatial relationship between the ramp and the opening. Here, the depth information is used to reflect the spatial constraints of the ramp and the case, while the target relative position information is used to determine the direction in which the robot needs to continue approaching. Both are used together to generate joint control commands, rather than separately generating body velocity commands.
[0159] exist Figure 7 In the second image, the robot has entered the box, and its body has lowered further compared to the fifth moment.
[0160] The robot continued to back up to its seating position, further reducing its stride, lowering its body, and correcting its posture by taking into account the space constraints of the inner lining and frame.
[0161] refer to Figure 8 As shown, Figure 8 This includes two simulated images from time points seven and eight in the simulation environment. In the first image, the robot further enters the box and lowers its body into a crouching posture.
[0162] In this pose, the humanoid robot is approaching the sitting position. Based on the updated target relative position information, the motion control strategy network continues to reduce its stride and adjusts the robot's posture according to the updated environmental spatial information, gradually lowering the robot's center of gravity. The state machine determines the humanoid robot's positional deviation relative to the sitting position based on the updated target relative position information and compares the magnitude of the positional deviation with a preset positional threshold.
[0163] exist Figure 8 In the second image, the robot lowers its body further inside the box, approaching a ready-to-sit position.
[0164] exist Figure 8 The second image shows the approach to the sitting preparation position stage. The state machine can calculate the position deviation based on the updated target relative position information and compare its magnitude with a preset position threshold. When the magnitude of the position deviation is less than the preset position threshold and the preset positioning condition is met, the motion control strategy network outputs a sitting action sequence, or the control is switched to the sitting control strategy and it outputs a sitting action sequence, causing the humanoid robot to perform the sitting action.
[0165] refer to Figure 9 As shown, Figure 9This illustration shows a humanoid robot housed in a seated position inside a target housing, according to another embodiment of the invention. The target housing is padded, and the humanoid robot is positioned within the storage area enclosed by the padded material, forming a compact seated posture. Once seated, the humanoid robot is passively supported by the internal support structure of the housing, thus maintaining its stored posture.
[0166] Figures 5 to 8 The simulation sequentially illustrates eight key moments in the environment, from approaching from outside the box, turning around, retreating into the box, to adjusting the posture inside the box. Figure 9 This demonstrates another storage state of the humanoid robot after it has settled into its seat. During the storage and walking process, joint commands are continuously generated directly based on observation information.
[0167] Throughout the return process, the target's relative position, relative orientation, environmental space, and the body's state all participate in the generation of joint commands, reducing intermediate speed command conversions, and making small strides and body posture adjustments near the target.
[0168] In some embodiments, the motion control strategy network directly generates joint control commands for controlling lower limb stepping and body posture adjustment based on target orientation and local spatial observation.
[0169] In the above process, environmental spatial information can be obtained by a depth camera capturing the spatial structure of the target container itself and the area it enters. For the backward movement phase after turning, it can be acquired in real time using a depth camera with a backward field of view, or updated based on the local spatial information obtained before turning and combined with changes in the base pose. Its coverage area can include the target container itself and the transition area outside the container's entrance. This transition area can be the ground area or ramp outside the target container's opening for the humanoid robot to walk on. If a ramp is provided outside the target container's opening, the local point cloud information captured by the depth camera also includes the surface of the ramp. The motion control strategy network generates joint control commands for going up and down the ramp, ensuring the humanoid robot maintains stable posture as it enters the target container from the ground via the ramp. It is understood that whether a ramp is provided in front of the target container depends on the actual placement conditions of the target container, and this invention does not limit this.
[0170] In some embodiments, the motion control policy network is trained through reinforcement learning. The following describes... Figure 10 and Figure 11 Explain the training process of the motor control strategy network.
[0171] refer to Figure 10 As shown, Figure 10 A flowchart illustrating a training method for a motion control strategy network used in autonomous warehousing of a humanoid robot is shown. This method can be executed by a server, computing device, or training platform.
[0172] like Figure 10 As shown, the training method for the motion control strategy network for autonomous retrieval of a humanoid robot provided in this embodiment of the invention includes the following steps S1001 to S1005: Step S1001: Obtain the target's relative position information, relative orientation information, environmental spatial information, and the humanoid robot's body state information in the warehousing training scenario.
[0173] Step S1002: The target's relative position information, relative orientation information, environmental spatial information, and body state information are used as observation information and input into the motion control strategy network to be trained, and joint control commands are directly output.
[0174] Step S1003: Control the joint movement of the humanoid robot according to the joint control command to perform warehousing walking with the warehousing target position as the target in the warehousing training scenario, and obtain feedback information after executing the joint control command.
[0175] Step S1004: Based on the feedback information, the parameters of the motion control strategy network are updated using reinforcement learning, and the observation information is updated according to the state of the humanoid robot after executing joint control commands.
[0176] Step S1005: Based on the updated observation information and the updated motion control strategy network, iteratively execute the processes of joint control command generation, joint motion control, feedback information acquisition, and parameter updating to obtain the trained motion control strategy network.
[0177] In some embodiments, the training process of the motion control policy network can be carried out in the following manner.
[0178] The process involves acquiring the target's relative position, relative orientation, environmental space, and the humanoid robot's body state information within a warehousing training scenario. The target's relative position represents the positional relationship between the target location (corresponding to the target container) and the humanoid robot, while the relative orientation represents the orientation relationship between the target container and the humanoid robot. This relative position, relative orientation, environmental space, and body state information are used as observation information and input into the motion control strategy network to be trained. The motion control strategy network directly outputs joint control commands based on the observation information. The humanoid robot's joint movements are controlled according to these commands to perform warehousing walking with the target location as the goal in the warehousing training scenario, and feedback information is acquired after executing the joint control commands. Based on the feedback information, reinforcement learning is used to update the parameters of the motion control strategy network, and the observation information is updated based on the humanoid robot's state after executing the joint control commands. Based on the updated observation information and the updated motion control strategy network, the processes of joint control command generation, joint movement control, feedback information acquisition, and parameter updating are iteratively executed to obtain the trained motion control strategy network.
[0179] A retrieval training scenario refers to a scenario used to train a motion control strategy network. In some embodiments, a target box is provided in the retrieval training scenario, and the humanoid robot performs retrieval walking with the target retrieval location as the objective. The retrieval training scenario can be the same as the scenario for the actual application described above, that is, the retrieval training scenario also provides a box in the form of an aviation case for storing the humanoid robot, and the humanoid robot needs to walk from the outside of the box to the target retrieval location inside the box. In one embodiment of the present invention, the retrieval training scenario can be constructed in a simulation environment, and the feedback information of the humanoid robot after executing joint control commands can be obtained through the simulation environment; the retrieval training scenario can also be a physical scenario, in which a physical humanoid robot executes joint control commands and obtains feedback information. The present invention does not limit the specific form of the retrieval training scenario.
[0180] The target's relative position, relative orientation, environmental space, and ontology state information acquired during the training phase have the same meaning as those acquired during the actual execution phase and are used as input to the policy network to be trained.
[0181] The robot executes training actions according to the joint commands output by the network and acquires information on changes in robot position, posture, and other states to determine the corresponding training feedback.
[0182] In some embodiments, the feedback information may include not only the robot's approach to the target location, but also information reflecting changes in body posture, joint movement amplitude, and interference within the box space. For example, the difference in the magnitude of the target's relative position vector between two training moments can represent the reduction in target distance; body posture deviation or joint movement amplitude can represent the smoothness of walking; and, if contact states can be obtained in the simulation environment, constraint feedback can be formed based on the contact between the robot body and unexpected parts such as the box frame. These feedback information are used together with set weights for training, guiding the motion control strategy network to shorten the target distance while considering body motion stability and box space constraints. This combination method is an optional extension to the feedback calculation; the specific weights and feedback functions should be determined by the actual training scenario.
[0183] Based on training feedback, the motion control strategy network parameters are updated using a reinforcement learning algorithm. Simultaneously, the observed information is updated based on the robot's state after executing joint commands, and this process is repeated. The reinforcement learning algorithm can be selected based on the training conditions.
[0184] In some embodiments, the homing training scenario sets multiple initial relative positions and orientations, allowing the humanoid robot to experience approaching the outside of the box, traversing the area, and fine-tuning its position inside the box. After each execution of a joint control command, the training environment returns new target distances, robot motion states, and environmental contact information. Based on this information, training feedback is calculated, and the policy network parameters are updated using the reinforcement learning algorithm employed. During training, behaviors that output large-amplitude joint movements when approaching the target homing position can receive lower overall feedback, while behaviors that shorten positional deviations, maintain robot stability, and avoid unintended contact with the box structure can receive higher overall feedback. This allows the policy network to gradually learn corresponding stride and posture adjustment methods at different target distances. For implementations including turning outside the box and backing into the box, training observations can also cover the state where the robot's back faces the target box and it approaches the sitting position by moving backward. The simulation feedback considers changes in target distance in the backward direction, robot stability, and unintended contact with the box structure to support the policy output during the backward walking phase.
[0185] Based on the updated observation information and the updated motion control strategy network, the processes of joint control command generation, joint motion control, feedback information acquisition, and parameter updating are iteratively executed to obtain the trained motion control strategy network. The four iterative processes are as follows: the updated motion control strategy network generates joint control commands based on the updated observation information; the joint motion of the humanoid robot is controlled according to the joint control commands; feedback information after executing the joint control commands is acquired; and the parameters of the motion control strategy network are updated based on the feedback information. This process continues until the motion control strategy network can generate joint control commands that enable the humanoid robot to move towards the target location based on the observation information. In some embodiments, the termination condition of the iteration can be determined based on the training effect; the present invention does not limit its specific form.
[0186] The trained network can be deployed on robots to directly output joint control commands based on observation information during the actual warehousing process.
[0187] In some embodiments, the process of updating the motion control strategy network parameters based on feedback information further includes: Based on feedback information, reinforcement learning is used to update the parameters of the motion control policy network, including: Based on the relative position information of the target before and after the humanoid robot executes joint control commands, determine the target distance change information; Based on the target distance change information, training feedback is determined to characterize the degree to which the humanoid robot approaches the target location, and the parameters of the motion control strategy network are updated according to the training feedback.
[0188] The target distance change information is determined by the target's relative position information before and after executing the joint control command, and can be characterized by the difference in magnitude of the target's relative position vector at the two moments.
[0189] A shorter target distance corresponds to approaching the target location, while an increased distance corresponds to moving away from the target location. This change can be used as training feedback to update the policy network parameters.
[0190] Distance changes can be calculated at adjacent control times or accumulated over multiple control times to form training feedback at different time scales.
[0191] In some embodiments, the target distances covered by the training process and the corresponding parameter update methods are limited as follows: Acquire the target's relative position information, relative orientation information, environmental spatial information, and the humanoid robot's body state information in the warehousing training scenario, including: Acquire the target's relative position information, relative orientation information, environmental spatial information, and body state information when the humanoid robot and the target location are at different relative distances. Among them, the environmental spatial information includes depth maps or local point cloud information representing the target box and its entry area. Based on feedback information, reinforcement learning is used to update the parameters of the motion control policy network, including: Based on the feedback information after the humanoid robot executes joint control commands at different target distances, the parameters of the motion control strategy network are updated. The trained motion control strategy network generates joint control commands based on the target relative position information and environmental spatial information, so as to reduce the walking stride when approaching the target position and adjust the body posture of the humanoid robot when passing through the entry area.
[0192] During training, different target distances are set to cover two phases: approaching from a distance and adjusting near the target, so that the policy network can learn the corresponding actions.
[0193] The spatial structure covered by the depth map or point cloud varies depending on the target distance: at greater distances, it mainly reflects the box outline and external space, while at closer distances, it can reflect the opening, border, and internal space constraints.
[0194] An execution feedback update strategy network is adopted at different target distances, which generates joint control commands based on the target's relative position and environmental spatial information, and reduces stride when approaching the target and adjusts body posture when passing through the entry area.
[0195] In some embodiments, to cover the spatial differences in the box entry area during training, the initial position, orientation, and spatial state of the target box entry area of the humanoid robot in the warehousing training scenario can be changed. Each set of training observations includes target relative position information, relative orientation information, environmental spatial information, and body state information; the depth map or local point cloud information can reflect the spatial distribution of the box border, entry area, and optional ramps. During training, the network continuously outputs joint-level control commands based on the corresponding observations and updates parameters based on the target distance changes and optional motion stability feedback after execution. By covering both distant and near passage positions, the policy network can simultaneously learn behaviors such as approaching the target, reducing the near-distance step size, and adapting to the spatial constraints of the box entrance.
[0196] refer to Figure 11 , Figure 11 The process of determining training feedback and updating motion control policy network parameters based on target distance change information is shown, including steps S1101 to S1102: Step S1101: Determine the target distance change information based on the target relative position information before and after the humanoid robot executes the joint control command.
[0197] Step S1102: Based on the target distance change information, determine the training feedback used to characterize the degree of approach of the humanoid robot to the retrieval target position, and update the parameters of the motion control strategy network according to the training feedback.
[0198] The following describes how to update the motion control strategy network parameters based on target distance change information.
[0199] Based on the target relative position information before and after the humanoid robot executes joint control commands, the target distance change information is determined; based on the target distance change information, training feedback is determined to characterize the degree to which the humanoid robot approaches the homing target position, and the parameters of the motion control strategy network are updated according to the training feedback.
[0200] The relative position information of the target before and after the humanoid robot executes the joint control command corresponds to the positional relationship of the humanoid robot relative to the target location before the joint control command is executed and the positional relationship after the execution. In some embodiments, the relative position information of the target includes the relative position vector of the target location in the current base coordinate system of the humanoid robot. The magnitude of the relative position vector represents the distance between the two. Therefore, the change in target distance can be determined based on the relative position vector before and after the execution of the joint control command.
[0201] In some embodiments, the difference in magnitude between the relative position vectors of the target before and after execution is used as the target distance change information. To avoid missing lateral position changes by using only the projection in one direction, this embodiment calculates the difference using the target distance under the same definition. The target distance can be defined as the three-dimensional distance between the robot's controlled reference position and the corresponding homing target position, or as the planar position relationship when using in-plane positioning determination; the same distance definition is used before and after training.
[0202] set up Indicates the current control moment. This represents the relative position vector of the target location at that moment relative to the robot's controlled reference position. To represent the corresponding target distance, the following formula can be used for calculation: In the formula, The Euclidean norm is represented; the relative position vector can be determined according to the aforementioned three-dimensional spatial relationships or a consistently selected positional relationship within the travel plane. The target distance at the next control moment is denoted as... The target distance change information can be represented as: In the formula, Indicates from time At that time The amount of time the target distance is shortened. A value greater than zero indicates that the robot is approaching its target location for retrieval; A value less than zero indicates that the robot is far from the target location for retrieval; A value of zero indicates that the target distance is the same at both moments. Training feedback can be determined based on this shortening amount and can be further combined with information on organism stability and spatial contact.
[0203] Training feedback is determined based on target distance change information. In some embodiments, when the target distance change information indicates that the distance between the humanoid robot and the target location is decreasing, the determined training feedback is used to characterize the humanoid robot approaching the target location; when the target distance change information indicates that the distance between them is increasing, the determined training feedback is used to characterize the humanoid robot moving away from the target location. The degree of approach represented by the training feedback is positively correlated with the degree of decrease in the distance between the humanoid robot and the target location. In one embodiment of the present invention, the specific calculation form of the training feedback can be determined according to the reinforcement learning algorithm used, and the present invention does not limit the specific calculation form.
[0204] The parameters of the motion control policy network are updated based on training feedback, enabling the network to progressively increase the likelihood of generating joint control commands that facilitate approaching the target position under corresponding observation conditions. It should be noted that reinforcement learning optimizes the policy based on feedback; the motion control policy network iteratively updates its parameters to progressively optimize joint control commands under different observation conditions. The actual parameter update results also depend on the reinforcement learning algorithm used, the sampled data, and other training feedback.
[0205] Training feedback can also incorporate state information such as posture stability, rather than being limited to changes in target distance.
[0206] The trained policy network takes four types of observation information as input, outputs joint control commands for homing walking, and can adjust walking movements at different target distances.
[0207] In some embodiments, the present invention also provides a walking control device for autonomous retrieval of a humanoid robot, comprising a processor and a memory. The memory stores a computer program, which, when executed by the processor, implements the aforementioned walking control method for autonomous retrieval of a humanoid robot and any of its optional embodiments.
[0208] The processor can be a central processing unit or other processing circuit, and the memory can be read-only memory, random access memory, or non-volatile memory; the processor executes programs to acquire observation information, generate joint instructions, and update the state.
[0209] The walking control device can be installed inside the humanoid robot, connecting to vision sensors, body sensors, and joint actuators; alternatively, a computing device that communicates with the humanoid robot can handle some of the data processing functions.
[0210] In some embodiments, the present invention also provides a humanoid robot autonomous storage control device, including a processor and a memory. The memory stores a computer program, which, when executed by the processor, implements the aforementioned humanoid robot autonomous storage method and any of its optional embodiments.
[0211] In addition to controlling the robot's walking motion, the control device also determines the positioning conditions based on updated observation information. Once the conditions are met, it triggers the robot to sit down, allowing it to be stored in the box in a seated position.
[0212] The state machine calculates the position deviation based on the target's relative position information. When the position deviation magnitude is lower than a preset threshold, a sitting action sequence is triggered.
[0213] In some embodiments, the present invention also provides a training apparatus for a motion control strategy network, including a processor and a memory. The memory stores a computer program, which, when executed by the processor, implements the aforementioned training method for a motion control strategy network for autonomous retrieval of a humanoid robot and any of its optional embodiments.
[0214] The training device runs a program through a processor, generates joint commands based on observation information, obtains training feedback, and updates network parameters using reinforcement learning.
[0215] The training device can be set up on a server or training platform, and deployed to the robot control device after network training is completed; the three types of devices correspond to walking control, complete retrieval and network training, respectively.
[0216] refer to Figure 12 As shown, the present invention also provides an electronic device 1200, including a processor 1201 and a memory 1202. The memory 1202 stores a computer program. When the computer program is executed by the processor 1201, the processor 1201 executes the aforementioned training method for the motion control strategy network for autonomous retrieval of humanoid robots and any of its optional implementation methods.
[0217] Processor 1201 runs the program, according to Figure 10 , Figure 11 The training process involves acquiring observations, generating joint commands, obtaining feedback, and updating network parameters; this hardware architecture can also serve as an example of implementing other control devices.
[0218] The electronic device 1200 may also have a communication interface for exchanging observation information, feedback information, and training results with the humanoid robot in the simulation environment or training scenario. The electronic device may be a server or a computer.
[0219] refer to Figure 13 As shown, the present invention also provides a humanoid robot 1300, including a controller 1301 and a joint actuator 1302 connected to the controller 1301. The controller 1301 is configured to execute the aforementioned walking control method for autonomous retrieval of the humanoid robot or the autonomous retrieval method for the humanoid robot and corresponding optional implementations, and to control the joint actuator 1302 to perform corresponding joint movements.
[0220] The controller 1301 is used to acquire observation information, run the strategy network and generate joint control commands, and the joint actuator 1302 drives the corresponding joint according to the target position command or torque command.
[0221] The controller 1301 inputs four types of observation information into the trained policy network, and the network directly outputs joint commands to drive the return-to-base movement, and adjusts subsequent actions according to the updated observation information.
[0222] The controller 1301 can also control the joint actuator 1302 to perform the sitting action after the conditions for the sitting preparation position are met.
[0223] The robot 1300 can be equipped with an RGB camera, a depth camera, and sensors for measuring the state of joints and body, and provides the observation data to the controller 1301.
[0224] refer to Figure 14 Humanoid robots may include a head, torso, two arms and two legs, and are equipped with cameras, body sensors and joint actuators, with the specific installation location not limited to those shown in the figure.
[0225] refer to Figure 15 As shown, the present invention also provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, it implements the aforementioned walking control method for autonomous retrieval of a humanoid robot, the autonomous retrieval method for a humanoid robot, or the training method for a motion control strategy network for autonomous retrieval of a humanoid robot, and their respective optional implementations.
[0226] Computer-readable storage media can be tangible media such as read-only memory, flash memory, hard disk or optical disk; Figure 15 The program 1500 stored within it is shown.
[0227] Program 1500 can acquire observations, directly generate joint commands through the strategy network, and continue to control the return-to-base movement based on the updated observation information.
[0228] Program 1500 can also realize positioning judgment and seat placement action control to complete the entire autonomous retrieval.
[0229] Program 1500 can also realize feedback acquisition, reinforcement learning parameter update and network training in the warehousing training scenario.
[0230] Each processing step can be implemented by the processor executing program instructions, or by the collaboration of software and hardware circuits.
[0231] The above embodiments are used to illustrate the technical solution of the present invention, and those skilled in the art can make adjustments without departing from the technical solution of the present invention.
Claims
1. A walking control method for autonomous retrieval of a humanoid robot, characterized in that, include: Acquire the target's relative position information, relative orientation information, environmental spatial information, and the body state information of the humanoid robot; The target relative position information represents the positional relationship between the target box and the humanoid robot, and the relative orientation information represents the orientation relationship between the target box and the humanoid robot. The target relative position information, the relative orientation information, the environmental space information, and the body state information are used as observation information and input into the motion control strategy network obtained through reinforcement learning. The motion control strategy network directly outputs joint control commands to drive the humanoid robot to walk towards the target location based on the observation information. The joint movement of the humanoid robot is controlled according to the joint control command to drive the humanoid robot to perform the return walking. During the homing process, the observation information is updated, and the updated observation information is input into the motion control strategy network to generate and execute subsequent joint control commands.
2. The walking control method for autonomous retrieval of a humanoid robot according to claim 1, characterized in that, The environmental spatial information includes depth maps or local point cloud information characterizing the spatial structure of the target container and its entry area; The step of controlling the joint movement of the humanoid robot according to the joint control commands to drive the humanoid robot to perform retrieval walking includes: During the process of the humanoid robot passing through the entry area, according to the joint control commands generated by the motion control strategy network based on the target relative position information, the relative orientation information, the environmental space information and the body state information, for lower limb stepping and body posture adjustment, the humanoid robot is controlled to move towards the return target position and adjust its body posture to adapt to the passable space of the entry area. The updating of the observation information during the retrieval process includes: The observation information is updated based on the state of the humanoid robot after it performs lower limb steps and adjusts its body posture.
3. The walking control method for autonomous retrieval of a humanoid robot according to claim 2, characterized in that, The joint control commands include lower limb joint control commands generated by the motion control strategy network based on the target distance and direction represented by the target relative position information, the relative orientation information, and the body state information, for adjusting walking stride and correcting walking direction; The step of controlling the joint movement of the humanoid robot according to the joint control commands to drive the humanoid robot to perform retrieval walking includes: When the humanoid robot approaches the target location, it reduces its walking stride through the lower limb joint control commands, and adjusts its body posture in combination with the environmental spatial information during the walking process with reduced stride. The process of updating the observation information during the homing process, inputting the updated observation information into the motion control strategy network, and generating and executing subsequent joint control commands includes: The observation information is updated based on the state after the lower limb joint control command is executed, and the updated observation information is input into the motion control strategy network to generate and execute joint control commands for subsequent position correction.
4. The walking control method for autonomous retrieval of a humanoid robot according to claim 3, characterized in that, The method further includes: When the humanoid robot is far from the target location, the motion control strategy network generates joint control commands for walking towards the target location with a larger stride. As the humanoid robot approaches the target location, it reduces its stride length via lower limb joint control commands. During this reduced stride length movement, it adjusts its body posture based on environmental spatial information, including: As the humanoid robot approaches the target location, based on the depth map or local point cloud information, the lower limb joint control commands control the humanoid robot to walk with a smaller stride and adjust its body posture to adapt to the spatial constraints of the area it is entering.
5. The walking control method for autonomous retrieval of a humanoid robot according to claim 1, characterized in that, In the step of acquiring the target relative position information, relative orientation information, environmental spatial information, and the body state information of the humanoid robot, acquiring the target relative position information and the relative orientation information includes: Acquire two-dimensional image information of the target box and identify visual markers set on the target box; Based on the recognition results of the visual markers, the pose of the target box relative to the current base of the humanoid robot is determined, and the relative position vector of the retrieval target position in the current base coordinate system and the relative yaw angle of the target box relative to the humanoid robot are determined. The relative position vector and the relative yaw angle are respectively used as the target relative position information and the relative orientation information.
6. The walking control method for autonomous retrieval of a humanoid robot according to claim 5, characterized in that, The visual marker is a QR code set inside the target box, and the storage target location corresponds to the center of the QR code. The step of determining the pose of the target container relative to the current base of the humanoid robot based on the recognition result of the visual marker, and determining the relative position vector of the retrieval target position in the current base coordinate system and the relative yaw angle of the target container relative to the humanoid robot, includes: Based on the recognition result of the QR code, the pose of the target box relative to the current base of the humanoid robot is determined, and the relative position vector of the center position of the QR code in the coordinate system of the current base and the relative yaw angle of the target box are determined, so as to obtain the target relative position information and the relative orientation information.
7. The walking control method for autonomous retrieval of a humanoid robot according to claim 6, characterized in that, The process of updating the observation information during the homing process, inputting the updated observation information into the motion control strategy network, and generating and executing subsequent joint control commands includes: Upon recognition of the QR code, as the base position and orientation of the humanoid robot change, the relative position vector of the center position of the QR code in the current base coordinate system of the humanoid robot and the relative yaw angle are updated according to the recognition result of the QR code. The updated relative position vector and the relative yaw angle are input into the motion control strategy network to generate and execute subsequent joint control commands.
8. The walking control method for autonomous retrieval of a humanoid robot according to claim 1, characterized in that, The joint control commands include joint target position commands or joint torque commands; The step of controlling the joint movement of the humanoid robot according to the joint control commands to drive the humanoid robot to perform retrieval walking includes: When the joint control command is a joint target position command, the movement of the corresponding joint is controlled according to the joint target position command; When the joint control command is a joint torque command, the corresponding joint is driven to move according to the joint torque command.
9. A method for autonomous storage of a humanoid robot, characterized in that, include: Acquire the target's relative position information, relative orientation information, environmental spatial information, and the body state information of the humanoid robot; The target location is the preparatory position for the humanoid robot to perform a sitting action inside the target box. The target relative position information represents the positional relationship between the target location and the humanoid robot. The relative orientation information represents the orientation relationship between the target box and the humanoid robot. The target relative position information, the relative orientation information, the environmental space information, and the body state information are used as observation information and input into the motion control strategy network obtained through reinforcement learning. The motion control strategy network directly outputs joint control commands to drive the humanoid robot to walk towards the target location based on the observation information. The joint movement of the humanoid robot is controlled according to the joint control command to drive the humanoid robot to walk from the entry area of the target box to the sitting preparation position. During the walking process, the observation information is updated and the updated observation information is input into the motion control strategy network to generate and execute subsequent joint control commands. When the updated observation information determines that the humanoid robot meets the preset positioning conditions corresponding to the sitting preparation position, the humanoid robot is controlled to perform a sitting action, so that the humanoid robot is stored in the target box in a sitting posture.
10. The method for autonomous warehousing of a humanoid robot according to claim 9, characterized in that, The step of controlling the joint movement of the humanoid robot according to the joint control command to drive the humanoid robot to walk from the entry area of the target box to the seating preparation position includes: During the process of the humanoid robot walking towards the sitting position, the joint control commands output by the motion control strategy network are adjusted according to the target relative position information, so that the humanoid robot reduces its walking stride when approaching the sitting position. When the humanoid robot passes through the entry area of the target box, the joint control commands are adjusted according to the environmental space information to control the humanoid robot to adjust its body posture and pass through the entry area.
11. The method for autonomous warehousing of a humanoid robot according to claim 10, characterized in that, The method further includes: Before the humanoid robot enters the target box through the entry area, the humanoid robot is controlled to perform a turning motion at the position outside the target box corresponding to the entry area, so that its back faces the inside of the target box; The step of controlling the joint movement of the humanoid robot according to the joint control command to drive the humanoid robot to walk from the entry area of the target box to the seating preparation position includes: After the turning action, the motion control strategy network generates joint control commands for backward walking based on the updated target relative position information, relative orientation information, environmental space information and body state information. The humanoid robot is driven to move backward into the target box according to the joint control commands for backward walking. During the backward movement, subsequent joint control commands are generated and executed based on the updated observation information to adjust the position and orientation of the humanoid robot relative to the seating preparation position.
12. The method for autonomous warehousing of a humanoid robot according to claim 9, characterized in that, The preset positioning condition includes that the magnitude of the positional deviation between the humanoid robot and the ready-to-sit position is less than a preset positional threshold. The step of determining that the humanoid robot meets the preset positioning conditions corresponding to the sitting preparation position based on the updated observation information includes: The state machine determines the position deviation based on the updated target relative position information, and determines that the humanoid robot meets the preset positioning condition when the magnitude of the position deviation is less than the preset position threshold.
13. The method for autonomous warehousing of a humanoid robot according to claim 12, characterized in that, The control of the humanoid robot to perform the sitting action includes: After the state machine determines that the humanoid robot meets the preset positioning conditions, it outputs a sitting action sequence through the motion control strategy network, or switches the control of the humanoid robot to a sitting control strategy, and the sitting control strategy outputs a sitting action sequence. The humanoid robot is controlled to perform a sitting action according to the sitting action sequence, so that the humanoid robot assumes a sitting posture passively supported by the support structure inside the target box.
14. A training method for a motion control strategy network for autonomous retrieval of a humanoid robot, characterized in that, include: Acquire the target's relative position information, relative orientation information, environmental spatial information, and the humanoid robot's body state information in the warehousing training scenario; The target relative position information represents the positional relationship between the target box and the humanoid robot, and the relative orientation information represents the orientation relationship between the target box and the humanoid robot. The target relative position information, the relative orientation information, the environmental spatial information, and the body state information are used as observation information and input into the motion control strategy network to be trained. The motion control strategy network directly outputs joint control commands based on the observation information. The joint movement of the humanoid robot is controlled according to the joint control command to perform warehousing walking with the warehousing target position as the target in the warehousing training scenario, and to obtain feedback information after the execution of the joint control command. Based on the feedback information, reinforcement learning is used to update the parameters of the motion control strategy network, and the observation information is updated according to the state of the humanoid robot after executing the joint control command; Based on the updated observation information and the updated motion control strategy network, the process of generating joint control commands, controlling joint motion, acquiring feedback information, and updating parameters is executed iteratively to obtain the trained motion control strategy network.
15. The training method for the motion control strategy network for autonomous retrieval of a humanoid robot according to claim 14, characterized in that, The step of updating the parameters of the motion control policy network using reinforcement learning based on the feedback information includes: Based on the target's relative position information before and after the humanoid robot executes joint control commands, the target distance change information is determined; Based on the target distance change information, training feedback is determined to characterize the degree to which the humanoid robot approaches the target location, and the parameters of the motion control strategy network are updated according to the training feedback.
16. The training method for the motion control strategy network for autonomous retrieval of a humanoid robot according to claim 15, characterized in that, The acquisition of target relative position information, relative orientation information, environmental spatial information, and humanoid robot body state information in the warehousing training scenario includes: The robot acquires target relative position information, relative orientation information, environmental space information, and body state information when the humanoid robot is at different relative distances from the target location. The environmental space information includes a depth map or local point cloud information characterizing the target box and its entry area. The step of updating the parameters of the motion control policy network using reinforcement learning based on the feedback information includes: Based on the feedback information after the humanoid robot executes joint control commands at different target distances, the parameters of the motion control strategy network are updated, so that the trained motion control strategy network generates joint control commands according to the target relative position information and environmental spatial information, so as to reduce the walking stride when approaching the return target position and adjust the body posture of the humanoid robot when passing through the entry area.
17. A walking control device for autonomous retrieval of a humanoid robot, characterized in that, It includes a processor and a memory, wherein the memory stores a computer program, and when the computer program is executed by the processor, it implements the walking control method for autonomous retrieval of a humanoid robot as described in any one of claims 1 to 8.
18. A humanoid robot autonomous warehousing control device, characterized in that, It includes a processor and a memory, wherein the memory stores a computer program, and when the computer program is executed by the processor, it implements the humanoid robot autonomous storage method according to any one of claims 9 to 13.
19. A training device for a motion control strategy network, characterized in that, It includes a processor and a memory, wherein the memory stores a computer program, which, when executed by the processor, implements the training method for a motion control strategy network for autonomous retrieval of a humanoid robot as described in any one of claims 14 to 16.
20. An electronic device, characterized in that, include: processor; as well as A memory storing a computer program, which, when executed by the processor, causes the processor to perform the training method for a motion control strategy network for autonomous retrieval of a humanoid robot, as described in any one of claims 14 to 16.
21. A humanoid robot, characterized in that, The device includes a controller and a joint actuator connected to the controller. The controller is configured to execute the walking control method for autonomous retrieval of a humanoid robot as described in any one of claims 1 to 8, or the autonomous retrieval method for a humanoid robot as described in any one of claims 9 to 13, and to control the joint actuator to perform corresponding joint movements.
22. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the walking control method for autonomous retrieval of a humanoid robot according to any one of claims 1 to 8, the autonomous retrieval method for a humanoid robot according to any one of claims 9 to 13, or the training method for the motion control strategy network for autonomous retrieval of a humanoid robot according to any one of claims 14 to 16.
