Autonomous operation method for humanoid robot based on ros

Abstract

This invention discloses a method for autonomous operation of humanoid robots based on ROS. It employs a decision-making and control framework that fuses hierarchical finite state machines with YOLOv8 vision to achieve continuous autonomous execution of multiple tasks in complex unstructured environments. The method includes: system initialization and standardized processing of multimodal perception data; target 3D localization fused with YOLOv8 and coordinate transformation; task scheduling and state transitions using hierarchical finite state machines; segmented navigation and motion planning of joint trajectories; and ROS command issuance and full-process closed-loop control. This invention addresses the rigidity of traditional script-based control by fusing hierarchical state machines with visual servoing. Combined with a full-process anomaly handling mechanism and multi-target scene processing rules, it can dynamically adapt to changes in complex unstructured environments, effectively avoiding problems such as state freezes and erroneous grasping.

Description

Technical Field

[0001] This invention belongs to the field of robot control technology, specifically relating to a method for autonomous operation of a humanoid robot based on ROS. Background Technology

[0002] Controlling bipedal humanoid robots is an extremely challenging task in the field of robotics research. The core difficulty lies in how to achieve a closed loop from environmental perception, decision-making and planning to whole-body motion control in complex and ever-changing unstructured environments.

[0003] Existing technologies mostly employ single-threaded linear scripts to serially execute navigation, recognition, and grasping, lacking task-level state switching and failure recovery mechanisms. This results in the inability to trigger conditions between task stages based on real-time observations. Furthermore, existing solutions often directly use visual recognition results for grasping without jointly determining depth validity and target category consistency, leading to falsely detected targets entering the grasping process. On the motion execution side, chassis movement and arm movements are usually controlled separately, lacking unified busy / idle state constraints and posture positioning criteria, resulting in unstable whole-body coordination timing.

[0004] The aforementioned technical defects can cause issues such as state freezing, incorrect capture, and action conflicts in continuous tasks across different scenarios, making it difficult to stably complete continuous operations such as climbing stairs, traversing complex terrain, and transporting targets. Summary of the Invention

[0005] The purpose of this invention is to overcome the shortcomings of existing technologies, such as single-threaded serial execution without state switching / failure recovery, direct use of visual recognition for grasping without depth / category joint determination, and unstable collaborative timing caused by separate control of chassis and arm without unified constraints. This invention provides a humanoid robot autonomous operation method based on ROS, which achieves continuous execution of multiple tasks in complex environments through hierarchical finite state machine and vision fusion.

[0006] This invention is achieved through the following technical solution: A method for autonomous operation of a humanoid robot based on ROS includes the following steps: Step 1: System initialization and multimodal sensing data standardization processing The system initializes and loads the visual model and establishes a ROS communication connection, then acquires RGB-D images, IMU pose data, and joint encoder data in real time. After preprocessing, it generates standardized observation states. At the same time, it completes full-process initialization verification and establishes a basic mechanism for anomaly handling.

[0007] Initialization operations: Load the YOLOv8 pre-trained model, establish communication connections between ROS nodes / publishers / subscribers, start the modular controller (vision module, motion planning module, arm control module, shared resource management module, and master state machine node), and load configuration information such as camera intrinsics, joint limits, and control parameters through the ROS parameter server.

[0008] Multimodal data acquisition: Environmental images, robot posture, and joint position / velocity / torque data are acquired through an RGB-D camera, IMU, and joint encoder, respectively. The acquisition frequencies are 30Hz for the RGB-D camera, 200Hz for the IMU, and 50Hz for the joint encoder.

[0009] Data preprocessing includes: 1. IMU data: High-frequency noise is removed by sliding window filtering (window size 5), quaternion attitude angles are calculated, and outliers with linear acceleration > 2g and angular velocity > 5rad / s are removed; 2. Joint encoder data: Median filtering is used to remove glitch, and timestamps are used to synchronize joint position / velocity / torque data; 3. RGB-D image frames: The RGB and depth images are aligned using a timestamp synchronization method (error ≤ 0.01s), and unaligned frames are discarded directly; 4. Unified format: Map all preprocessed data to the world coordinate system, unify the data format to float32, mark missing data as NaN and trigger the completion mechanism.

[0010] Standardized observation state generation: The preprocessed data is encapsulated into a unified dictionary structure `obs`. Simultaneously, at the sensor's data publishing end, all sensing data is uniformly encapsulated into a message in `sensorsData` format, with a timestamp in `header.stamp` format set for this message. The message's coordinate system is also calibrated to the world coordinate system. The specific structure of the `obs` dictionary is defined as follows: The camera-related perception data in OBS consists of two parts: three-channel image frame data and depth map data. The IMU data in OBS covers linear acceleration, angular velocity, and quaternion pose-related data collected by the device. The joint state data in OBS adapted for the Kuavo robot includes the position, velocity, and torque data of the 12 joints of the robot's legs, 14 joints of the arms, and 2 joints of the head.

[0011] The initialization check includes: sequentially performing the existence check of the model file, the YOLO model loading and zero-image inference verification, the ROS node / publisher / subscriber creation check, and the hardware interface connectivity check; if any link in the system initialization check has an exception, the system will automatically record the error log, keep the visual detection result empty, and not enter the subsequent grasping process; at the same time, perform abnormal process cleaning and resource release operations at the process level to prevent the system from entering an irreparable failure state due to process exceptions.

[0012] Step 2: 3D Target Localization by Integrating YOLOv8 Model and Coordinate Transformation Use the YOLOv8 model and coordinate transformation algorithm to process multi-modal perception data, complete 2D target detection and 3D localization, and at the same time perform joint determination of depth validity and category consistency to filter out invalid targets. The specific workflow is as follows: 2.1 2D Target Detection: Read the RGB and depth data output in Step 1, convert RGB to BGR and then input it into the YOLOv8 model for inference. This model is trained based on the COCO dataset and a custom job target dataset (including stairs, boxes, shelves, etc.), retain the detection boxes with a confidence level ≥ 0.7, and output the detection box coordinates, confidence level, and category numbers. 2.2 Depth Validity Filtering: Extract the central pixel coordinates (u, v) for each retained box, and obtain the corresponding depth value depth in the depth map. Perform double filtering: the depth valid value constraint 0 < depth ≤ 10m, and the near-distance filtering depth > 0.7m. The valid targets after filtering enter the next step. 2.3 Target Attribute Fusion: Integrate the category results and depth results into the target attribute {class_name, confidence, u, v, depth}, and directly filter out the targets that are not in the preset job categories. 2.4 3D World Coordinate Calculation: Use the pinhole imaging model and external parameter mapping to achieve the conversion from pixel coordinates to world coordinates. The camera internal parameters (fx, fy, cx, cy) are obtained through Zhang Zhengyou calibration method and stored in the configuration file camera_calib.yaml. The specific steps are as follows: ① Calculate the three-dimensional points in the camera coordinate system according to the camera internal parameters: x cam =(u - cx)*depth / fx, y cam =(v - cy)*depth / fy, z cam =depth; ② Solving the world pose rotation matrix: IMU data is fused with wheel odometry data (Kalman filtering) to obtain the robot's world pose quaternion (w,x,y,z). The rotation matrix R is then solved using the quaternion-to-rotation matrix formula.

[0013] ③ Coordinate transformation: Combining the camera-to-body translation [0,0,1] and the above rotation matrix R, the three-dimensional points of the camera system are transformed from the body coordinate system to the world coordinate system to obtain the target's three-dimensional position {x,y,z}; 2.5 Multi-target processing: If multiple valid targets exist, a weighted scoring method is used to select the optimal target. The weighting factors are: closest to the robot (0.5), target category consistent with the task preset (0.3), and no occlusion (0.2). The target with the highest total score is the optimal target. If the optimal target is occluded, the robot pose adjustment mechanism is triggered (small range movement ≤0.5m + head rotation ±30°), and the target is re-detected. If the target is still occluded after 3 adjustments, the second-best target is selected. 2.6 Step 2 Exception Handling: If there is no valid target (empty after depth filtering / category mismatch), trigger the target re-search mechanism, expand the visual detection range, update candidate target points, and if there is still no valid target after 3 seconds of continuous perception, send a "detection failure" signal to the state machine. The state machine maintains the current navigation state or returns to the previous task node.

[0014] Step 3: Task scheduling and state transition of hierarchical finite state machine A hierarchical finite state machine is scheduled based on the task ID. By evaluating the matching degree between real-time observation data and preset logical conditions, the current robot behavior mode and state transition are determined. The hierarchical finite state machine is implemented in "task layer - action layer" and has clear coding rules, switching priorities and scheduling logic, specifically as follows: 3.1 State Machine Core Definition Encoding rules: Two-level encoding is adopted, with the task layer being T1 / T2 / T3 (corresponding to navigation, grab and place, and transport tasks respectively), and the action layer being Tn_Sm (n is the task layer number, and m is the action layer state number, such as T1_S0 = going to the stairs, T2_S1 = grab ready). Switching priority: Emergency states (hardware failure, pose instability) have the highest priority and can interrupt any task; Task layer priority (configurable): T1 (navigation) > T3 (transportation) > T2 (grab and place); Action layer priority: Position determination > trajectory execution > perception update; Scheduling logic: Based on event-triggered scheduling, the task ID and state transition signal are published through the ROS topic task command channel. The master state machine node subscribes to the topic and executes the scheduling. At the same time, the current state is published through the state feedback channel to achieve two-way feedback.

[0015] 3.2 Task-level state definition Task 1 (T1, Unstructured Environment Navigation): The states are as follows: T1_S0 Heading to the stairs, T1_S1 Climbing the stairs, T1_S2 Heading to the beach initialization point, T1_S3 Heading to the beach starting point, T1_S4 Heading to the beach ending point, T1_S5 Heading to the task ending point; Task 2 (T2, visual servoing operation): The state cycles through T2_S0 grab preparation, T2_S1 grab, T2_S2 place preparation, and T2_S3 place. When the preset conditions are met, it switches to Task 3. Task 3 (T3, Full-body Coordinated Transport): The status progresses in the following order: T3_S0 Head to box, T3_S1 Grab box, T3_S2 Head to shelf, T3_S3 Place box, T3_S4 Task completion.

[0016] 3.3 Conditional Matching Criteria: A non-continuous scoring conditional matching criterion is adopted. If the criterion is met, the state transition is executed and the corresponding action is triggered. If the criterion is not met, it is handled according to the type. If the recoverable criterion is not met, the sensing is repeated and the robot pose is fine-tuned before re-judgment. If the non-recoverable criterion is not met, the current action layer state is terminated, the task layer is returned to the waiting scheduling state, and an error log is recorded. The criteria include: whether the pose error is lower than the threshold, whether the target category matches, whether the depth is valid, whether the subprocess busy / idle flag allows switching, and whether the stage flag is met.

[0017] 3.4 State transition processing rules If the criterion is met, the state transition is executed and the corresponding action is triggered; if the criterion is not met, the current state is maintained, the sensing and control release is repeated, or the next candidate target point is updated and the search continues to prevent accidental switching.

[0018] Step 4: Motion planning with segmented navigation and joint trajectory Based on the current pose and target point, the navigation path and joint trajectory are calculated, generating chassis speed commands and a smooth interpolation sequence for the robotic arm. Multiple constraint checks ensure the trajectory's rationality, and a unified busy / idle flag enables coordinated control of the chassis and arm. Specifically: 4.1 Navigation Path Planning: A segmented planning approach oriented towards key points is adopted, executing segment by segment according to the target point sequence given by the state machine. The core steps are: ① Read the robot's current pose (current_pos) and target point (target_pos), and calculate the translation error and orientation error; ② If the orientation error is greater than 3° (orientation error threshold), only issue angular velocity commands (≤0.5 rad / s) to execute the steering until the orientation error is ≤3°; ③ After the turn is completed, generate a linear velocity command based on the relative position of the target point (≤0.2m / s in unstructured environments, ≤0.1m / s when climbing stairs) and move towards the target point; ④ Real-time detection of pose error; navigation is considered in place when translation is ≤0.1m and rotation is ≤5° (pose error threshold).

[0019] 4.2 Joint Trajectory Planning: A predefined key pose sequence is used, approximated by cubic spline interpolation, to ensure the continuity of joint position, velocity, and acceleration. The core steps are: ① Read the current angle of the 14th joint of the arm and use it as the starting point of the trajectory; ② Perform cubic spline interpolation on predefined key attitude points to generate intermediate points; ③ Sampling interpolation points at a control frequency of 50Hz to generate a complete joint trajectory sequence; ④ Perform joint limit checks (12 joints in the legs ±1.57 rad, 14 joints in the arms ±2.0 rad, 2 joints in the head ±1.0 rad). If the limits are exceeded, the trajectory will be automatically adjusted. ⑤ Compare the joint error between the current joint and the target posture in real time. If the error is ≤0.02rad (joint position tolerance), proceed to the next posture.

[0020] 4.3 Trajectory rationality verification: Perform three types of constraint verification. If any constraint is not met, the process will not proceed to the next state: orientation error threshold, target point distance threshold, and joint positioning tolerance. 4.4 Whole-body coordinated control: By using unified busy / idle indicators and task status to jointly constrain the movements of the chassis and arm, the chassis and arm are locked when performing key segments such as climbing stairs or grabbing to avoid conflicts; the chassis and arm are unlocked and subsequent actions are advanced only after the positioning criteria are met.

[0021] 4.5 Motion Control Algorithm: A position-based PID algorithm is used to achieve chassis speed control and joint angle control. The PID parameters are individually tuned for different motions and stored in the ROS parameter server. For example, for the leg joint, KP=5.0, KI=0.1, KD=0.5, and for the chassis linear velocity, KP=3.0, KI=0.05, KD=0.3. 4.6 Anomaly Handling: If the trajectory exceeds the joint limit or there is an obstacle in the path, path replanning is triggered. The target point is adjusted by combining the environmental information detected by vision, and the navigation path and joint trajectory are regenerated. If the replanning fails 3 times, the task is paused and reported to the main state machine.

[0022] Step 5: ROS command issuance and full-process closed-loop control The planned motion control commands are sent to the underlying hardware via ROS topics to drive the actuators to complete physical operations. Multiple validity checks are performed before the commands are sent, and sensor feedback is used to achieve a closed-loop control process of "perception-determination-action-feedback," specifically: Command generation: Chassis control commands are encapsulated as Twist speed commands, and joint execution commands are uniformly assembled into leg torque, arm position, and head position control quantities through joint command callbacks.

[0023] Validity determination before issuance: The current action object is not empty, the posture or distance criteria meet the corresponding state requirements, the depth and category are valid, and the busy / idle flag allows execution.

[0024] Command issuance: Chassis control commands are issued through the chassis motion command channel, and joint control commands are issued through the hardware interface. The issuance frequency is consistent with the control frequency (50Hz). The underlying hardware execution: The underlying hardware is the leg drive, arm joint drive and head joint drive in the humanoid robot execution link (the simulation and the actual machine interface are consistent). After receiving the instructions, the execution mechanism completes the physical operation, and the feedback sensors (IMU, RGB-D camera, joint encoder) collect execution status data in real time. Full-process closed-loop control: perception update OBS → state machine determination → action generation → ROS issuance → actuator feedback → re-perception and determination; if the action has not yet been generated, the system redistributes sensor messages at the control frequency and maintains the current state until the issuance conditions are met; Anomaly Handling: If there is no response when ROS commands are issued / the sensor feedback data is interrupted, the process restart mechanism is triggered to re-establish the ROS communication connection and restart the corresponding execution module; if a hardware failure is detected (such as joint driver failure), the current task is terminated, the faulty module is marked and the error log is recorded, and the fault information is reported to the master control terminal.

[0025] Compared with the prior art, the beneficial effects of the present invention are: This invention solves the rigidity problem of traditional script-based control by integrating a hierarchical state machine with visual servoing. Combined with a full-process exception handling mechanism and multi-target scene processing rules, it can dynamically adapt to changes in complex unstructured environments and effectively avoid problems such as state deadlock and erroneous grasping. This invention adopts multimodal perception and whole-body motion planning, and achieves parallel processing of chassis movement and arm grasping through unified busy / idle flags and priority scheduling. It has high operational stability and high task completion rate in scenarios such as climbing stairs, traversing complex terrain, and grasping and transporting objects.

[0026] This invention adopts a modular controller design, clearly defining the ROS node division and interaction logic of each module. It features automatic dependency checking, error logging, and process cleanup, reducing development and deployment costs and facilitating later maintenance and functional expansion. Through data preprocessing, deep validity filtering, trajectory constraint verification, and PID algorithm control, it achieves pose error and joint error both below preset thresholds, resulting in high operation control accuracy and meeting the requirements for high-precision autonomous operation. Attached Figure Description

[0027] Figure 1 This is a flowchart of a humanoid robot autonomous operation method based on ROS according to the present invention; Figure 2 This is a simulation diagram of the stair climbing method of the present invention; Figure 3 This is a simulation diagram of the present invention crossing a slope; Figure 4 This is a simulation diagram of the present invention over speed bumps; Figure 5 This is a simulation diagram of the present invention grasping a specified object; Figure 6 This is a simulation diagram of the box-moving process of this invention. Detailed Implementation

[0028] The present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments: The baseline hardware platform of this invention is the Kuavo Pro bipedal humanoid robot, equipped with the Ubuntu 20.04+ROSNoetic system. The simulation environment is built based on Gazebo 11+ROS MoveIt1.1. The ROS interface of the simulation and the actual machine are completely consistent, realizing seamless connection between simulation debugging and actual deployment.

[0029] Hardware platform and development environment configuration: Baseline hardware platform: Kuavo Pro bipedal humanoid robot, with the following hardware configuration: RGB-D camera (1280×720 resolution, depth detection range 0.3-10m), 6-axis IMU (sampling frequency 200Hz), 12 servo joints in the legs, 14 servo joints in the arms, and 2 servo joints in the head, supporting ROS topic / service communication; Hardware compatibility requirements: For other bipedal humanoid robots, the following conditions must be met for compatibility: ① Equipped with a ROS system that supports topic / service communication; ② Equipped with sensors such as an RGB-D camera, IMU, and joint encoders; ③ Possesses motion control capabilities for at least 12 leg joints and 14 arm joints; ④ Provides a low-level ROS interface; The compatibility method is to modify the hardware configuration file robot_hw.yaml, replacing parameters such as joint limits, camera intrinsics, and communication interfaces; ROS Development Environment: Uses the standard Catkin workspace, path / catkin_ws. Functional packages are divided as follows: robot_sensors (sensor data acquisition), robot_vision (vision processing), robot_fsm (state machine control), robot_motion (motion planning), and robot_hw (hardware interface). All functional packages are compiled using catkin_make. At startup, roslaunch, robot_bringup, and robot.launch are executed, automatically starting all nodes and loading configuration files. All configuration files are stored in the / catkin_ws / src / robot_config / config directory. Simulation environment setup: Simulation scene files are stored in / catkin_ws / src / robot_gazebo / worlds, supporting one-click loading of simulation scenes such as climbing stairs, ramps, speed bumps, grabbing, and moving. The simulation parameters are completely consistent with the actual machine parameters.

[0030] Figure 1 This paper presents a method for autonomous operation of a humanoid robot based on ROS, encompassing a complete closed-loop control process from sensor data acquisition, vision processing, state machine decision-making, motion planning to command execution, and the data flow between modules. The specific implementation process of this invention is illustrated below using three typical tasks as examples: Task 1: Walking and navigation capabilities in unstructured environments, including sub-tasks such as climbing stairs, crossing slopes, and navigating speed bumps.

[0031] During system initialization, the YOLOv8 visual model is automatically loaded and a ROS connection is established. All modular controllers are started, and navigation-related parameters (chassis speed 0.1 m / s, pose error threshold 0.1 m / 5°) are loaded. Sensors collect multimodal data in real time: the RGB-D camera acquires environmental images, the IMU provides pose information, and the joint encoder returns leg status. This data is processed by the `process_obs` method to generate standardized observation states, including world coordinates `current_pos` and quaternion orientation. After the sensor data is input, the state machine schedules behavior according to the task ID. The initial state of Task 1 is `taskone_state=0`, and the robot navigates to the starting point of the stairs using the `walk_to_point_task1` method. When a change in the stair height is detected, the state transitions to `taskone_state=1`, triggering the `start_stair_climb` method to start the stair-climbing subprocess. Figure 2 In the simulation environment shown, the stair-climbing subroutine controls the robot to smoothly climb the stairs. The motion planning module generates chassis speed commands based on a PID algorithm and issues them via the ROS topic / cmd_pose. Simultaneously, the robotic arm assists in balancing along a predefined trajectory. In scenarios involving ramps and speed bumps, the state machine dynamically adjusts the path to adapt to terrain changes and achieves closed-loop control through sensor feedback. Figure 3 , Figure 4 In the simulation environment shown, a state machine controls a robot to smoothly traverse a ramp and a speed bump.

[0032] Task 2: Visual servoing operations, including target detection, grasping, and placement.

[0033] Figure 5 In the simulation environment shown, the robot smoothly moves to the placement location after grasping the target object. The robot processes the RGB-D image using the `detect_objects` method, the YOLOv8 model identifies the target object, and background interference is filtered in conjunction with depth information. The state machine schedules behavior according to the task objective. If the stage is "grab", the robot moves to the grasping point using the `walk_to_point_task2` method. When the detection result matches the grasping target, the state machine transitions to the grasping state, triggering the `grab_object` method to grasp the object. After successful grasping, the robot carries the object to the placement point and calls the `place_object` method.

[0034] Task 3: Full-body coordination, including the handling and placement of boxes and the inspection of target shelves.

[0035] Figure 6In the simulation environment shown, a robot picks up a box. The robot identifies the target on the shelf using a vision module and obtains its world coordinates. The state machine initializes the task state and navigates to the box using the `walk_to_point` method, calculating the optimal orientation based on the `ideal_dir_str_task3` method. When approaching the target, the state machine switches to the handling state. The robotic arm executes a smooth trajectory, interpolating joint commands using the `arm_traj_change` method. After the robot places the box on the shelf, the state machine marks the task as complete and logs the information.

[0036] The undescribed parts involved in this invention are the same as or implemented using existing technology.

Claims

1. A method for autonomous operation of a humanoid robot based on ROS, characterized in that, A decision control framework integrating hierarchical finite state machines and YOLOv8 vision is adopted, including the following steps: Step 1, System Initialization and Multimodal Sensing Data Standardization Processing: Load the YOLOv8 vision model and establish a communication connection with the ROS system. Real-time acquisition of RGB-D images, IMU pose and joint encoder data. After preprocessing, a standardized observation state is generated. The standardized observation state is a unified dictionary structure. Sensor acquisition data is uniformly encapsulated into sensorData messages with timestamps and world coordinate system calibration. At the same time, the entire initialization process is verified. If the verification fails, error log recording and failure process cleanup are performed. Step 2, Target 3D localization by fusing YOLOv8 with coordinate transformation: 2D target detection is performed on RGB data using the YOLOv8 model. After extracting the valid detection boxes, the depth map is combined to perform dual filtering of depth validity and proximity. The target category and depth information are fused to form target attributes. Then, the pixel coordinates are converted into the target's three-dimensional physical position in the world coordinate system through the pinhole imaging model and extrinsic mapping algorithm. Step 3, Task scheduling and state transition of the hierarchical finite state machine: The hierarchical finite state machine consisting of the task layer and the action layer is scheduled according to the task ID. The current behavior mode of the robot is determined by comparing the real-time observation data with the preset condition matching criteria and the results are used to achieve orderly state transition. The condition matching criteria include pose error, target category, depth validity, subprocess busy / idle flag and stage flag determination. Step 4, Segmented navigation and joint trajectory motion planning: Based on the robot's current pose and target point, a segmented planning method is used to generate the chassis navigation path. The smooth joint trajectory of the robotic arm is generated through predefined key posture sequences and interpolation algorithms. At the same time, a unified busy / idle flag is set to constrain the coordination timing of the chassis and the arm, and three types of constraint checks on trajectory rationality are performed. Step 5, ROS command issuance and full-process closed-loop control: The chassis speed command and joint execution command generated by motion planning are issued to the underlying hardware through the ROS system-specific topic. Before the command is issued, multiple validity checks are performed. After the underlying actuator completes the physical operation, the execution status is collected in real time by the feedback sensor, forming a full-process closed-loop control of "perception update - state machine judgment - motion generation - ROS issuance - execution feedback". Each step is configured with corresponding exception handling and recovery mechanisms.

2. The ROS-based autonomous operation method for humanoid robots according to claim 1, characterized in that, The preprocessing described in Step 1 includes performing sliding window filtering and outlier removal on IMU data, median filtering and timestamp synchronization on joint encoder data, and timestamp alignment on RGB-D image frames. All preprocessed data is uniformly converted to the float32 format and mapped to the world coordinate system. After marking the missing data, the completion mechanism is triggered; the dictionary structure of the standardized observation state includes the three-channel image frame and depth map data of the camera, the linear acceleration, angular velocity, and quaternion attitude data of the IMU, and the position, velocity, and torque data of the 12 joints of the legs, 14 joints of the arms, and 2 joints of the head of the bipedal humanoid robot.

3. The ROS-based autonomous operation method for humanoid robots according to claim 1, characterized in that, The YOLOv8 model described in Step 2 is trained based on the COCO dataset and the job-customized target dataset, and the detection boxes with a confidence level ≥ 0.7 are retained; the conditions for depth validity and near-distance double filtering are 0 < depth ≤ 10m and depth > 0.7m; the external parameter mapping algorithm includes obtaining the robot world pose quaternion by fusing IMU and wheeled odometer data through Kalman filtering, then solving the rotation matrix through the quaternion-to-rotation matrix formula, and completing the unified transformation of the coordinate system in combination with the translation parameters from the camera to the body.

4. The ROS-based autonomous operation method for humanoid robots according to claim 1, characterized in that, The hierarchical finite state machine described in Step 3 adopts a two-level encoding rule. The task layer is encoded as T1, T2, T3, corresponding to unstructured environment navigation, visual servo grasping and placing, and whole-body coordinated handling tasks respectively. The action layer is encoded as Tn_Sm, where n is the task layer number and m is the action layer state serial number; the state transition sets priorities, with the emergency state having the highest priority. The task layer priorities are T1 > T3 > T2, and the action layer priorities are in-place determination > trajectory execution > perception update; the state machine adopts event-triggered scheduling, publishes the task ID and state transition instructions through the dedicated channel for ROS system task scheduling, and transmits the current execution state back through the dedicated channel for state feedback to achieve two-way communication.

5. The ROS-based autonomous operation method for humanoid robots according to claim 1, characterized in that, The processing rule for the comparison result of the condition matching criterion described in Step 3 is: adopt a non-continuous scoring condition matching criterion. If the criterion is satisfied, the state transition is executed and the corresponding action is triggered. If the criterion is not satisfied, it is processed according to the type. If the recoverable criterion is not satisfied, the perception is repeated and the robot pose is fine-tuned and then re-judged; If the non-recoverable criterion is not satisfied, the current action layer state is terminated, the task layer is returned to the pending scheduling state, and the error log is recorded.

6. The ROS-based autonomous operation method for humanoid robots according to claim 1, characterized in that, The specific process of the segmented navigation path planning described in Step 4 is: first calculate the deviation between the current orientation and the target orientation. When the deviation is greater than 3°, only the angular velocity command is issued to execute the turn. After the deviation ≤ 3°, the linear velocity command is issued to move towards the target point. When the pose error satisfies translation ≤ 0.1m and rotation ≤ 5°, it is determined that the navigation is in place.

7. The ROS-based autonomous operation method for humanoid robots according to claim 1, characterized in that, The ROS system topics mentioned in step 5 include chassis control commands, which are encapsulated as Twist speed commands. Joint execution commands are assembled into leg torque, arm position, and head position control quantities. The multiple validity determinations include the current action object being non-empty, posture or distance criteria meeting the corresponding state requirements, depth and category determinations being valid, and busy / idle flags allowing execution. The underlying hardware includes leg drives, arm joint drives, and head joint drives. The simulation and actual machine interfaces are consistent, and the feedback sensors include an IMU and an RGB-D camera.

8. The ROS-based autonomous operation method for humanoid robots according to claim 1, characterized in that, The anomaly handling and recovery mechanism includes: Step 2: When there is no valid target, trigger the target re-search mechanism; if there is no valid target after 3 seconds of continuous sensing, send a detection failure signal to the state machine; Step 4: When the trajectory is invalid or there are obstacles, trigger path replanning; if the replanning fails 3 times, suspend the task and report it; Step 5: When there is no response to the command or the feedback data is interrupted, trigger the process restart mechanism; if a hardware fault is detected, terminate the task and mark the faulty module.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, which, when executed by a processor, implements the ROS-based autonomous operation method for humanoid robots as described in any one of claims 1-8.

10. A humanoid robot autonomous operation control system based on ROS, characterized in that, The system includes a ROS system, a sensor module, a hierarchical finite state machine decision module, a motion planning module, a low-level execution module, and a feedback module. The sensor module is used to acquire RGB-D images, IMU attitude data, and joint encoder data. The hierarchical finite state machine decision module is used to implement the task scheduling and state transition as described in claim 1. The motion planning module is used to implement the navigation path and joint trajectory planning as described in claim 1. The low-level execution module is used to receive instructions from the ROS system and complete physical operations. The feedback module is used to collect the execution status in real time and realize closed-loop control. The modules work together to execute the ROS-based humanoid robot autonomous operation method as described in any one of claims 1-8.

Images