A humanoid robot control system and method based on reinforcement learning

Abstract

The present invention relates to the field of robot control and proposes a humanoid robot control system and method based on reinforcement learning. The humanoid robot control system includes a first control subsystem and a second control subsystem; the first control subsystem includes a strategy reasoning module, a state transition module and a robot control module; the second control subsystem includes a data acquisition module and a drive control module; the data acquisition module is used to complete the timestamp alignment and outlier filtering of sensor data, and transmit them to the state transition module; the state transition module is used to fuse multi-source sensor data and construct a timing state feature containing real-time measurement values ​​and historical timing information; the strategy reasoning module is used to generate a multi-joint angle target value of the robot according to the timing state feature provided by the state transition module; the robot control module is used to parse the multi-joint angle target value output by the strategy reasoning module, select a control mode, and generate a control command including a parameter adjustment instruction.

Description

Technical Field

[0001] The present invention relates to the field of robot control, and in particular to a humanoid robot control system and method based on reinforcement learning. Background Art

[0002] Humanoid robots have broad application prospects due to their human-like structure and ability to work and interact naturally in human environments. However, the control of humanoid robots faces challenges such as high dimensionality, nonlinearity, and uncertainty. Traditional model-based control methods are difficult to adapt to complex and changing environments.

[0003] In recent years, reinforcement learning, as a data-driven control method, has shown great potential in the field of robotic control. Reinforcement learning learns optimal strategies by interacting with the environment, does not require precise system models, and can adapt to complex nonlinear dynamic systems. However, the application of reinforcement learning in actual humanoid robots faces technical difficulties such as real-time requirements, limited hardware resources, sensor data fusion and processing, and safety assurance.

[0004] In existing technologies, most reinforcement learning robot control systems are mainly verified in simulation environments or are based on a single platform architecture, which cannot meet the real-time, stable and safe control requirements of real humanoid robots. Summary of the Invention

[0005] In response to the above-mentioned defects, the purpose of the present invention is to propose a humanoid robot control system and method based on reinforcement learning, aiming to solve the real-time response, environmental perception and execution safety problems encountered by reinforcement learning in physical robot deployment through the construction of a hierarchical reinforcement learning architecture and temporal state characteristics.

[0006] To achieve this object, the present invention adopts the following technical solutions:

[0007] A humanoid robot control system based on reinforcement learning, the humanoid robot control system comprising a first control subsystem and a second control subsystem;

[0008] The first control subsystem includes a strategy reasoning module, a state transition module and a robot control module;

[0009] The second control subsystem includes a data acquisition module and a drive control module;

[0010] The data acquisition module is used to synchronously collect joint motion parameters, posture perception data and contact force information based on sensors, complete the timestamp alignment and outlier filtering of sensor data, and transmit them to the state conversion module;

[0011] The state conversion module is used to fuse multi-source sensor data and construct time series state features that include real-time measurements and historical time series information;

[0012] The strategy reasoning module is used to generate the robot's multi-joint angle target values ​​through a reinforcement learning strategy network based on the temporal state features provided by the state transition module. The reinforcement learning strategy network uses a deep learning framework to achieve parallel computing acceleration.

[0013] The robot control module is used to analyze the multi-joint angle target values ​​output by the strategy reasoning module, select the control mode, and generate control commands including parameter adjustment instructions;

[0014] The drive control module is used to receive control commands from the robot control module, convert the robot's multi-joint target angles into motor drive signals to drive the robot's joints, and monitor the actuator status in real time to trigger the safety protection mechanism.

[0015] Preferably, the state transition module includes:

[0016] Receive data from multiple source sensors and synchronize timestamps of joint encoder data and inertial measurement data;

[0017] Constructing a single-frame observation vector including periodic phase encoding, motion parameters, and posture information, wherein: the periodic phase encoding is achieved by converting the gait periodic phase value into a sine-cosine dual-channel encoding, the motion parameters are normalized based on the target velocity instruction; the posture information is obtained by fusing joint kinematics solution with inertial measurement data;

[0018] The current single-frame observation vector is stacked with a preset number of historical frame data in time series. When the number of historical frames is less than the preset number, the missing frames are filled with zero vectors to obtain the time series state features.

[0019] Furthermore, the state transition module includes:

[0020] receiving original position parameters and original velocity parameters of multiple joints in real time, performing a first normalization process on the original position parameters after subtracting pre-stored joint position reference values, and generating a normalized joint position feature vector;

[0021] The original velocity parameter is multiplied by the motion state scaling coefficient for a second normalization process to generate a normalized joint velocity feature vector, the normalized joint position feature vector is filled into the first dimension interval in the observation vector, the normalized joint velocity feature vector is filled into the second dimension interval in the observation vector, and the normalized joint position feature vector and the normalized joint velocity feature vector are stored in a history cache queue in a time series.

[0022] Furthermore, the state transition module includes:

[0023] Receiving multi-source attitude parameters, the multi-source attitude parameters including joint attitude Euler angle data and angular velocity and linear acceleration data of an inertial measurement unit, performing quaternion conversion calculation on the Euler angle data, and generating a quaternion attitude representation including spatial rotation information;

[0024] Multiplying the angular velocity data by a first scaling factor to perform first motion feature normalization processing to generate a normalized angular velocity feature vector, multiplying the Euler angle data by a second scaling factor to perform second posture feature normalization processing to generate a normalized posture feature vector, filling the normalized angular velocity feature vector into a third dimensional interval in the observation vector, and filling the normalized posture feature vector into a fourth dimensional interval in the observation vector;

[0025] A posture monitoring message including coordinate transformation parameters is generated according to the quaternion posture representation and is visualized, and the normalized angular velocity feature vector and the normalized posture feature vector are stored in a history cache queue in a time series.

[0026] Preferably, the policy reasoning module includes:

[0027] Constructing a reinforcement learning strategy network including three hidden layers, and inputting the temporal state features into the reinforcement learning strategy network;

[0028] The reinforcement learning policy network infers the optimal action with GPU acceleration and outputs a multi-dimensional joint angle target vector whose dimension matches the number of robot joints.

[0029] An inference trigger cycle is set, and joint motion range limiting processing is performed on the output vector according to the inference trigger cycle to generate multi-joint angle target values ​​of the robot.

[0030] Furthermore, the robot control module includes:

[0031] Analyzing the multi-joint angle target values ​​and converting the multi-joint angle target values ​​into a numerical array;

[0032] Detect the numerical distribution characteristics of each dimension and select the current control mode based on the preset dimension grouping threshold, where:

[0033] If it is detected that the dimension value of the left leg in the numerical array is greater than the dimension value of the right leg and the difference between the two is greater than a first preset value, the left leg control mode is activated;

[0034] If it is detected that the dimension value of the right leg in the numerical array is greater than the dimension value of the left leg and the difference between the two is greater than a first preset value, the right leg control mode is activated;

[0035] If all dimension values ​​are within the preset balance range and the difference between each dimension is less than the set second preset value, the balance maintenance mode is activated;

[0036] Dynamically adjust joint control parameters based on the identified control mode, including: adding periodic compensation to the target joint angle in walking mode, increasing the proportional gain coefficient of the joint position loop in balance mode, and shortening the control period and increasing the differential gain coefficient in recovery mode;

[0037] The adjusted multi-joint angle target values ​​and control parameters are encapsulated into a structured data packet, added with a protocol identifier matching the control mode, and then transmitted to the drive control module.

[0038] Furthermore, when the inference cycle is triggered, it includes:

[0039] Call the policy inference function at a fixed period that matches the preset control frequency, record the initial timestamp before the policy inference process begins, and calculate the phased time value after the inference is completed;

[0040] Comparing the periodic time consumption value with the theoretical time consumption threshold corresponding to the fixed period, and triggering a resource dynamic allocation compensation strategy when it is detected that the periodic time consumption value continuously exceeds the theoretical time consumption threshold for a preset number of times;

[0041] The time consumption data of each inference process is added to the performance statistical log in real time, and the performance statistical log is analyzed regularly to generate a performance analysis report including the average time consumption and the extreme time consumption. When it is detected that the stage time consumption exceeds the preset deviation threshold, the abnormal alarm mechanism is triggered.

[0042] Preferably, the second control submodule further includes a communication module;

[0043] The communication module is used to encapsulate sensor data into data packets with integrity verification, including:

[0044] receiving a binary data stream carrying multiple source sensors through a preset communication protocol, and verifying that a packet header identifier of the binary data stream matches a preset packet header identifier;

[0045] Parse the data packet type field and identify it as a sensing data packet or a heartbeat maintenance packet. If it is identified as a sensing data packet, extract the floating-point parameter set in the data payload based on the data length field.

[0046] Performing byte order conversion and data type decoding operations on the floating-point parameter set to generate a sensing parameter set, and mapping different physical quantity parameters in the sensing parameter set to a ROS2 message data structure to generate the data packet.

[0047] Preferably, the communication module further includes a heartbeat detection mechanism:

[0048] Receive a heartbeat data packet carrying a status identifier, verify the matching of a preset identifier and a protocol identifier in the heartbeat data packet, and extract a timestamp and a status code;

[0049] Update the current status record according to the timestamp, reset the counting period of the timeout timer, and execute the degradation control strategy when the number of consecutive heartbeat data packet losses exceeds the preset timeout threshold, trigger the automatic reconnection process and broadcast the connection abnormality reminder to the first control subsystem;

[0050] A heartbeat confirmation signal including the operating status is periodically sent to the second control subsystem. When it is detected that the communication link is restored, the degraded control strategy is released and switched to the standard control strategy.

[0051] A humanoid robot control method based on reinforcement learning, wherein the humanoid robot control method is applied to the humanoid robot control system as described above, and the humanoid robot control method comprises:

[0052] Based on the synchronous collection of joint motion parameters, posture perception data and contact force information by sensors, the timestamp alignment and outlier filtering of sensor data are completed;

[0053] Fuse multi-source sensor data and construct time series state features that include real-time measurements and historical time series information;

[0054] According to the temporal state characteristics, a reinforcement learning strategy network is used to generate target values ​​for the multi-joint angles of the robot, wherein the reinforcement learning strategy network adopts a deep learning framework to achieve parallel computing acceleration;

[0055] Analyzing the target values ​​of the multi-joint angles, selecting a control mode, and generating a control command including a parameter adjustment instruction;

[0056] The control command is received, the multi-joint target angle of the robot is converted into a motor drive signal to drive the joints of the robot, and the actuator status is monitored in real time to trigger a safety protection mechanism.

[0057] One of the above technical solutions has the following advantages or beneficial effects:

[0058] The present invention uses sensors to synchronously collect joint motion parameters, posture perception data and contact force information, and completes timestamp alignment and outlier filtering, which can ensure the accuracy and consistency of input data, solve the control error problem caused by inconsistent multi-source data, and provide high-quality input for subsequent modules; the state conversion module fuses multi-source sensor data and constructs a time series state feature containing real-time measurement values ​​and historical time series information, which helps to enhance the input information of the reinforcement learning strategy network, enabling it to perceive the environmental state more comprehensively, thereby improving the robustness and adaptability of strategy reasoning; the strategy reasoning module uses the parallel computing acceleration of the deep learning framework to generate the multi-joint angle target value of the robot, and uses the GPU to calculate the target value of the robot's multi-joint angle. Acceleration and real-time guarantee mechanism, this module significantly improves the reasoning speed of reinforcement learning strategies, meeting the high requirements of humanoid robots for real-time control; the robot control module parses the target values ​​of multi-joint angles, dynamically selects control modes (such as left leg control, right leg control or balance maintenance mode), and generates control commands containing parameter adjustment instructions. This dynamic mode selection and parameter adjustment mechanism can enhance the adaptability of the system, enabling it to quickly respond to different motion states; the drive control module converts control commands into motor drive signals, monitors the actuator status in real time, and triggers safety protection mechanisms under abnormal conditions, which can ensure the stability and safety of the system and avoid system crashes due to hardware failures or abnormalities. BRIEF DESCRIPTION OF THE DRAWINGS

[0059] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are merely embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying any creative work.

[0060] Figure 1 Schematic diagram of the structure of a humanoid robot control system based on reinforcement learning provided by an embodiment of the present invention;

[0061] Figure 2 is a flow chart of a humanoid robot control method based on reinforcement learning provided by an embodiment of the present invention;

[0062] Figure 3 3 is a schematic diagram of the structure of the reinforcement learning strategy network of the reinforcement learning-based humanoid robot control system provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0063] The following describes embodiments of the present invention in detail. Examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended only to explain the present invention and are not to be construed as limiting the present invention.

[0064] In this application, the terms "comprises," "comprising," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, method, article, or apparatus that includes a list of elements includes not only those elements but also other elements not explicitly listed, or elements inherent to such process, method, article, or apparatus. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not preclude the presence of additional identical elements in the process, method, article, or apparatus that includes the element.

[0065] A humanoid robot control system based on reinforcement learning, such as Figure 1 As shown, in a preferred embodiment of the present invention, the humanoid robot control system includes a first control subsystem and a second control subsystem;

[0066] The first control subsystem includes a strategy reasoning module, a state transition module and a robot control module;

[0067] The second control subsystem includes a data acquisition module and a drive control module;

[0068] The data acquisition module is used to synchronously collect joint motion parameters, posture perception data and contact force information based on sensors, complete the timestamp alignment and outlier filtering of sensor data, and transmit them to the state conversion module;

[0069] The state conversion module is used to fuse multi-source sensor data and construct time series state features that include real-time measurements and historical time series information;

[0070] The strategy reasoning module is used to generate the robot's multi-joint angle target values ​​through a reinforcement learning strategy network based on the temporal state features provided by the state transition module. The reinforcement learning strategy network uses a deep learning framework to achieve parallel computing acceleration.

[0071] The robot control module is used to analyze the multi-joint angle target values ​​output by the strategy reasoning module, select the control mode, and generate control commands including parameter adjustment instructions;

[0072] The drive control module is used to receive control commands from the robot control module, convert the robot's multi-joint target angles into motor drive signals to drive the robot's joints, and monitor the actuator status in real time to trigger the safety protection mechanism.

[0073] In this embodiment, the first control subsystem can be an upper-level control system based on the Jetson platform, responsible for high-level policy reasoning and state transitions. The second control subsystem can be a lower-level control system based on an STM32 microcontroller, responsible for data acquisition and drive control. The two achieve closed-loop control through efficient data exchange. In the data acquisition module, joint motion parameters can refer to the robot's joint angles, joint velocities, etc., and posture perception data can refer to the angular velocity and linear acceleration of the inertial measurement unit, as well as contact force information such as ground contact force. The function of the data acquisition module is to provide real-time and accurate robot status data to subsequent modules. Timestamp alignment refers to synchronizing the timestamps of sensor data to ensure that multi-source data is on the same time basis, avoiding control errors caused by data inconsistencies. Outlier filtering refers to preprocessing sensor data to eliminate outliers (such as noise or erroneous data) and improve data quality.

[0074] The reinforcement learning policy network can be a reinforcement learning model implemented based on a deep learning framework (such as TensorFlow or PyTorch), which accelerates the inference process through parallel computing and outputs multi-joint angle target values.

[0075] Specifically, in the data acquisition module, an inertial measurement unit (IMU) can be used to collect posture data, joint encoders can collect joint position and velocity data, and force sensors can collect contact force information. The data acquisition frequency can be set to 100 Hz or higher to ensure real-time performance. Timestamp alignment can be achieved through hardware synchronization or software interpolation. Outlier filtering can use median filtering, sliding average, or statistical-based methods.

[0076] The state transition module integrates real-time measurements with historical time series information through time series stacking. For example, the current frame is stacked with a preset number of historical frames in chronological order. Insufficient historical frames can be padded with zero vectors to ensure consistent input feature dimensions. Time series state features can include periodic phase encoding (such as sine-cosine encoding), motion parameters (such as target velocity commands), and posture information (such as quaternion representation). The reinforcement learning policy network in the policy inference module can use the Deep Deterministic Policy Gradient (DDPG) or Proximal Policy Optimization (PPO) algorithm. GPU acceleration can significantly improve inference speed, for example, meeting control frequency requirements above 50Hz. The robot control module can dynamically select left-leg control, right-leg control, or balance maintenance mode based on the distribution characteristics of multi-joint angle target values. Parameter adjustment instructions can include proportional gains and differential gains of the joint position loop to adapt to different motion states.

[0077] In the drive control module, the motor drive signal can be realized through pulse width modulation (PWM) or digital control. The safety protection mechanism may include overcurrent protection, overtemperature protection and emergency stop mechanism to ensure that the robot will not be damaged under abnormal circumstances.

[0078] Therefore, the present invention realizes efficient and real-time control through a layered architecture. The first control subsystem is responsible for high-level strategy reasoning and state transition, and the second control subsystem is responsible for data acquisition and drive control. The two realize closed-loop control through efficient data exchange. The data acquisition module synchronously collects joint motion parameters, posture perception data and contact force information through sensors, and performs timestamp alignment and outlier filtering on the data to ensure the accuracy and consistency of the data. The state transition module fuses multi-source sensor data to construct a time series state feature containing real-time measurement values ​​and historical time series information, providing comprehensive input for the reinforcement learning strategy network. The strategy reasoning module uses the parallel computing acceleration of the deep learning framework to generate the multi-joint angle target value of the robot. The robot control module parses the target value, selects the control mode and generates the control command. The drive control module converts the control command into a motor drive signal, monitors the actuator status in real time and triggers the safety protection mechanism to ensure the stability and safety of the system.

[0079] Preferably, the state conversion module includes:

[0080] Receive data from multiple source sensors and synchronize timestamps of joint encoder data and inertial measurement data;

[0081] Specifically, by receiving heterogeneous sensor data from joint encoders and inertial measurement units (IMUs), time deviations caused by sensor sampling frequency or communication delays are eliminated, ensuring strict alignment of data from different physical quantities in the temporal dimension. Its core purpose is to address the spatiotemporal misalignment of multi-source data caused by asynchronous acquisition times. For example, the instantaneous correspondence between joint angles and body acceleration is disrupted, thereby avoiding control strategy errors caused by data asynchrony. The role of timestamp synchronization alignment is to provide a temporally consistent data foundation for subsequent state feature construction, enabling accurate correlation of cross-modal information such as cycle phase and motion parameters. Among them, joint encoder data refers to the joint angle / angular velocity information measured by a rotary encoder or potentiometer, which is used to reflect the real-time position and motion status of each joint of the robot and is the basis for kinematic solution; inertial measurement data includes the three-axis acceleration and angular velocity output by the IMU, which is used to estimate the posture (such as pitch angle and roll angle) and motion trend (such as instantaneous acceleration direction) of the robot body; timestamp synchronization alignment can map the data of different sensors to a unified time base through hardware clock synchronization or software interpolation algorithms (such as linear interpolation) to ensure the time consistency of the data.

[0082] Constructing a single-frame observation vector including periodic phase encoding, motion parameters, and posture information, wherein: the periodic phase encoding is achieved by converting the gait periodic phase value into a sine-cosine dual-channel encoding, the motion parameters are normalized based on the target velocity instruction; the posture information is obtained by fusing joint kinematics solution with inertial measurement data;

[0083] Specifically, the synchronized multi-source data is converted into a standardized feature vector that can be processed by the reinforcement learning policy network, and the gait cycle (such as the swing / support phase during walking) is encoded into a continuous sine-cosine value through periodic phase encoding, such as mapping the gait cycle phase value (such as the cycle ratio of 0-1) to , , solving the problem of numerical jumps at the cycle boundary and enabling the policy network to smoothly handle periodic motion; the motion parameters are normalized based on the target speed command to eliminate dimensional differences and enhance the policy's generalization ability for speed commands. For example, the target speed command (such as a forward speed of 0.5m / s) is linearly scaled to [-1,1] within a preset range (such as 0-1m / s) to eliminate dimensional effects and adapt to the input range of the policy network's activation function; the posture information is integrated with the joint kinematics solution (forward kinematics to calculate the body posture) and the IMU data (directly measure the posture), and the complementarity of sensors is used to improve the robustness of the posture estimation. Its purpose is to compress high-dimensional heterogeneous sensor data into low-dimensional, interpretable feature vectors while retaining key motion state information. For example, the posture solved by the joint kinematics solution (which depends on the encoder accuracy) and the posture measured by the IMU (which is susceptible to instantaneous acceleration interference) are integrated through the extended Kalman filter (EKF) or complementary filter to improve the posture estimation accuracy in dynamic environments.

[0084] The current single-frame observation vector is stacked with a preset number of historical frame data in time series. When the number of historical frames is less than the preset number, the missing frames are filled with zero vectors to obtain the time series state features.

[0085] Specifically, the observation vector of the current single frame is stacked with historical data from several past frames in chronological order to form state features that include temporal context. When the number of historical frames is insufficient, missing frames are padded with zero vectors to ensure a fixed input dimension. This allows the reinforcement learning policy network to capture the temporal dependencies of the motion state (such as phase coherence and posture change trends within the gait cycle), thereby making control decisions that conform to the laws of dynamic evolution. Zero-vector padding prevents insufficient frames due to system startup or communication interruptions, maintaining the stability of the network input structure. Temporal stacking involves concatenating observation vectors from multiple consecutive time steps along the channel dimension. For example, the current frame plus the previous three frames form a 4-frame × N-dimensional matrix, which serves as the input to the policy network. Zero-vector padding involves filling missing historical frames with all-zero vectors during system initialization or data loss to prevent network inference errors caused by changes in input dimension. The preset number of historical frames can be set based on the time scale of the motion task, for example, a walking task requires 3 frames (approximately 150ms history window) and a jumping task requires 10 frames (500ms).

[0086] Specifically, timestamp synchronization and alignment avoids time deviation between joint encoders and IMU data, ensuring the physical correlation between phase encoding and posture information; single-frame observation vector construction extracts low-dimensional and high-information-density features through periodic encoding, motion parameter normalization and posture fusion; time series stacking gives the policy network the ability to capture gait cycle evolution and motion trends.

[0087] Furthermore, the state transition module includes:

[0088] receiving original position parameters and original velocity parameters of multiple joints in real time, performing a first normalization process on the original position parameters after subtracting pre-stored joint position reference values, and generating a normalized joint position feature vector;

[0089] The original velocity parameter is multiplied by the motion state scaling coefficient for a second normalization process to generate a normalized joint velocity feature vector, the normalized joint position feature vector is filled into the first dimension interval in the observation vector, the normalized joint velocity feature vector is filled into the second dimension interval in the observation vector, and the normalized joint position feature vector and the normalized joint velocity feature vector are stored in a history cache queue in a time series.

[0090] By receiving the original position (such as angle value) and speed (such as angular velocity value) parameters fed back by the joint encoder in real time, the absolute numerical deviation caused by mechanical structure differences or sensor zero drift is eliminated, and the data is mapped to a unified dimensional range. Its purpose is to solve the problem of inconsistent feature scales caused by differences in physical properties (such as range of motion and maximum speed) of different joints. For example, ±180° of the shoulder joint and ±90° of the knee joint, if directly input into the network, will lead to an imbalance in the model weight distribution. In this embodiment, the absolute position is converted into an offset relative to the reference position through the first normalization process (subtracting the reference value) to highlight the joint movement trend; the second normalization process (multiplying by the scaling factor) scales the speed value to the [-1,1] range to avoid model convergence difficulties caused by mixing speed units (such as rad / s and m / s).

[0091] The raw position parameter refers to the angle / displacement value directly measured by the joint encoder (e.g., a knee currently bent 30°), reflecting the instantaneous spatial position of the joint and serving as the fundamental input for kinematic solutions. The joint position reference value refers to a pre-stored reference position (e.g., the neutral position of the joint when the robot is upright), used to convert the absolute position into a relative offset (e.g., current angle - neutral angle = +5°), eliminating the effects of mechanical installation errors on the absolute position. The motion state scaling factor is a scaling factor (e.g., 1 / 2 = 0.5) set based on the joint's maximum motion capacity (e.g., a maximum angular velocity of 2 rad / s for the hip joint). This factor scales the raw velocity value to the range [-1, 1], making the velocity characteristics of different joints comparable. The dimensional range of the observation vector represents a pre-partitioned feature vector storage area. For example, the first 12 dimensions of the vector are position features (one dimension per joint), and the last 12 dimensions are velocity features, ensuring a standardized data structure.

[0092] The normalized joint position / velocity feature vector is filled into the specified dimensional interval of the observation vector according to preset rules, and stored in the historical cache queue in time series to construct a state representation with spatiotemporal continuity. Its purpose is to solve the problem of lack of temporal correlation of single-frame data. For example, when the current speed is high but the historical speed continues to decline, the policy network needs to combine multiple frames of data to determine whether it is in the deceleration stage. The first dimensional interval (such as 0-11 dimensions) stores position features, and the second dimensional interval (such as 12-23 dimensions) stores speed features. Feature confusion is avoided through dimensional isolation; the historical cache queue retains the most recent N frames of data, provides a buffer for zero-value filling, ensures that the input dimension is fixed, and meets the structural requirements of the neural network input.

[0093] Furthermore, the state transition module includes:

[0094] Receiving multi-source attitude parameters, the multi-source attitude parameters including joint attitude Euler angle data and angular velocity and linear acceleration data of an inertial measurement unit, performing quaternion conversion calculation on the Euler angle data, and generating a quaternion attitude representation including spatial rotation information;

[0095] Multiplying the angular velocity data by a first scaling factor to perform first motion feature normalization processing to generate a normalized angular velocity feature vector, multiplying the Euler angle data by a second scaling factor to perform second posture feature normalization processing to generate a normalized posture feature vector, filling the normalized angular velocity feature vector into a third dimensional interval in the observation vector, and filling the normalized posture feature vector into a fourth dimensional interval in the observation vector;

[0096] A posture monitoring message including coordinate transformation parameters is generated according to the quaternion posture representation and is visualized, and the normalized angular velocity feature vector and the normalized posture feature vector are stored in a history cache queue in a time series.

[0097] By receiving Euler angle data from joint encoders and angular velocity and linear acceleration data from the IMU, the Euler angles are converted to quaternions to resolve the gimbal deadlock problem associated with Euler angles in three-dimensional rotations and unify the mathematical representation of data from different sensors. Quaternions describe spatial rotations using four-dimensional vectors (w, x, y, z). This avoids singularities in Euler angles caused by rotation order (e.g., ZYX). They also support efficient interpolation and compound rotation calculations, providing an unambiguous mathematical foundation for subsequent posture fusion and kinematic solutions. For example, when a robot rapidly rolls, quaternions can continuously and smoothly describe posture changes, while Euler angles may cause posture solution failure due to singularities. Euler angle data for joint posture refers to the pitch, roll, and yaw angles calculated through forward kinematics or directly measured by joint encoders. They are used to describe the discrete rotation angles of the robot's local coordinate system relative to the world coordinate system, but are subject to the gimbal deadlock problem. The angular velocity and linear acceleration data of the inertial measurement unit are the three-axis angular velocity (unit: rad / s) and linear acceleration (unit: m / s²) measured by the IMU sensor, reflecting the instantaneous rotation trend and linear motion state of the robot body. However, accelerometer data is susceptible to vibration interference. Quaternion posture representation is a continuous rotation representation generated by the Euler angle to quaternion conversion formula (such as the Tait-Bryan angle formula). It supports efficient rotation superposition (quaternion multiplication) and coordinate system transformation to avoid singularities in posture description.

[0098] During normalization and observation vector padding, the IMU angular velocity is scaled to the range [-1, 1] using a first scaling factor (e.g., 0.01) to prevent weight imbalance in the reinforcement learning policy network caused by dimensional differences between different physical quantities (e.g., rad / s and m / s²). A second scaling factor (e.g., π / 180) is used to convert Euler angles to radians or normalize them to eliminate the impact of joint range differences (e.g., ±90° for the knee and ±180° for the hip) on feature weights. The normalized angular velocity feature vector (e.g., 3D) and posture feature vector (e.g., 3D) are padded to the third dimension (indexes 41-43) and fourth dimension (indexes 44-46) of the observation vector, respectively. This dimensional isolation prevents feature confusion while meeting the structural requirements of the neural network input.

[0099] The quaternion posture is then converted into coordinate transformation parameters (such as a rotation matrix or translation vector), generating a TF (Transform) message for 3D posture rendering in ROS visualization tools (such as RVIZ). At the same time, the normalized angular velocity / posture feature vectors are stored in a historical cache queue in a time series. The purpose is to provide real-time visual monitoring capabilities (such as detecting whether the robot is tilted beyond the limit). By caching the latest N frames of data, temporal state features are constructed, enabling the policy network to capture motion trends (such as the continuous increase in angular velocity before a fall). The zero-padding mechanism fills the historical frames with all-zero vectors when data is lost, ensuring that the input dimension is constant.

[0100] In another embodiment, the policy reasoning module includes:

[0101] Construct a reinforcement learning policy network consisting of three hidden layers. The reinforcement learning policy network is as follows Figure 3 As shown, the temporal state features are input into the reinforcement learning strategy network;

[0102] The reinforcement learning policy network infers the optimal action with GPU acceleration and outputs a multi-dimensional joint angle target vector whose dimension matches the number of robot joints.

[0103] An inference trigger cycle is set, and joint motion range limiting processing is performed on the output vector according to the inference trigger cycle to generate multi-joint angle target values ​​of the robot.

[0104] The policy reasoning module maps 705-dimensional temporal state features (47-dimensional single-frame features × 15-frame historical stacks) to 12-dimensional joint angle target values ​​by designing an MLP network architecture consisting of three fully connected hidden layers (512-256-128 neurons). The three hidden layers extract high-order motion pattern features (such as gait phase and dynamic balance trend) through layer-by-layer nonlinear transformations (such as the ReLU activation function), solving the problem that shallow networks are unable to capture complex spatiotemporal correlations. Its purpose is to utilize the feature abstraction capabilities of deep networks to convert high-dimensional sensor data (joint angles, IMU postures, historical frame stacks) into control strategies that adapt to dynamic environments, such as automatically adjusting stride length and leg lift height in unstructured terrain. The 705-dimensional features of the input layer contain 15 frames of historical data, enabling the network to perceive the continuity of the motion state (such as the increasing trend of angular velocity before imbalance) and avoid short-sighted decision-making caused by single-frame input.

[0105] The policy network is then inferred at high speed using the parallel computing capabilities of GPUs (such as Jetson's CUDA cores), reducing the single inference time to less than 5ms, meeting the real-time requirements of a 50Hz control frequency. By deploying the network model to GPU memory and optimizing the computational graph (such as layer fusion and FP16 quantization) using the TensorRT framework, high throughput batch inference is achieved. This aims to address control lag caused by the high inference latency of traditional CPUs (e.g., over 20ms). For example, when the robot is running quickly, GPU acceleration ensures that joint target angles are generated every 20ms, matching the 100Hz sensor data update frequency and preventing loss of synchronization between control commands and perception data.

[0106] Finally, a fixed 20ms inference trigger cycle (50Hz) is set to limit the motion range of the 12-dimensional joint angles output by the network (for example, the knee joint is limited to ±90°) to ensure that the output instructions meet the mechanical structure constraints. The goal is to strike a balance between policy flexibility and hardware security: the 50Hz frequency matches the minimum real-time requirements of humanoid robot dynamic balance control (at least 10Hz) while leaving sufficient time for GPU inference (≤5ms inference + 15ms communication and execution within a 20ms cycle). Limiting processing uses hard truncation (such as np.clip) or dynamic scaling to prevent motor overload or mechanical damage caused by network output exceeding the limit. For example, when the policy network outputs a target knee angle of 120°, it is forcibly truncated to 90°, triggering security logging.

[0107] Preferably, the robot control module includes:

[0108] Analyzing the multi-joint angle target values ​​and converting the multi-joint angle target values ​​into a numerical array;

[0109] Specifically, the multi-dimensional tensor (such as PyTorch Tensor) output by the policy network is converted into a numerical array (such as a NumPy array), eliminating the data structure differences that the framework depends on and ensuring data compatibility with the underlying control system. Through explicit data type conversion (such as float32) and device switching (such as GPU→CPU), the problem of data flow interruption in heterogeneous computing environments is solved. For example, the CUDA tensor is copied to the host memory to avoid the driver controller from being unable to parse GPU data. Its purpose is to build a standardized data interface so that the upper-level policy output can seamlessly connect to the underlying execution unit, while improving transmission efficiency through memory continuity optimization.

[0110] Detect the numerical distribution characteristics of each dimension, select the current control mode based on the preset dimension grouping threshold, and dynamically select the control mode by statistically analyzing the distribution characteristics of the joint angle target value (such as the difference in the mean value of the left and right leg dimensions and the range of numerical fluctuations) and combining it with the preset threshold (such as the first preset value of 10° and the second preset value of 5°). The purpose is to enable the robot to adaptively switch the behavior strategy according to the task requirements: when the left leg joint angle is significantly greater than the right leg (difference > 10°), activate the left leg control mode to perform the single-leg support action; when the numerical distribution is balanced (the difference in each dimension is < 5°), switch to the balance mode to maintain posture stability. The threshold setting is based on the robot's dynamic characteristics (such as the tolerance of the center of mass offset), and the optimal value is calibrated through offline simulation and actual machine testing. Among them:

[0111] If it is detected that the dimension value of the left leg in the numerical array is greater than the dimension value of the right leg and the difference between the two is greater than a first preset value, the left leg control mode is activated;

[0112] If it is detected that the dimension value of the right leg in the numerical array is greater than the dimension value of the left leg and the difference between the two is greater than a first preset value, the right leg control mode is activated;

[0113] If all dimension values ​​are within the preset balance range and the difference between each dimension is less than the set second preset value, the balance maintenance mode is activated;

[0114] Dynamically adjust joint control parameters based on the identified control mode, including: adding periodic compensation to the target joint angle in walking mode, increasing the proportional gain coefficient of the joint position loop in balance mode, and shortening the control period and increasing the differential gain coefficient in recovery mode;

[0115] The adjusted multi-joint angle target values ​​and control parameters are encapsulated into a structured data packet, added with a protocol identifier matching the control mode, and then transmitted to the drive control module.

[0116] In one embodiment, the PID parameters and compensation strategy of the joint controller are adjusted according to the selected control mode: in walking mode, a sinusoidal wave compensation amount (such as amplitude ±2° and frequency 2Hz) is superimposed on the hip / knee joint angle to simulate the cycle characteristics of human gait to enhance stability; in balancing mode, the position loop proportional gain (Kp) is increased by 30% to resist external disturbances through stronger corrective force; in recovery mode, the control period is shortened from 20ms to 10ms, and the differential gain (Kd) is increased to quickly suppress posture deviation. The purpose is to enable the same controller to cope with multi-task requirements through real-time parameter adaptation, such as switching to recovery mode when the risk of falling increases, and quickly correcting the posture with higher frequency control. The adjusted joint angle target value and control parameters are then encapsulated into a structured data packet (such as a binary protocol), and a mode identifier (such as 0x01 for walking mode) is added for the driver module to parse. The data packet structure includes: frame header (0xAA55), mode identifier (1 byte), joint angle array (12×float32), and control parameters (such as Kp / Kd value). It is transmitted to the STM32 controller via the UDP protocol, utilizing its low latency feature (<1ms) to ensure real-time performance. At the same time, a retransmission mechanism is used to cope with packet loss. The purpose is to establish a lightweight communication pipeline to achieve lossless connection between upper-level decision-making and underlying execution while ensuring the timeliness of instructions.

[0117] Preferably, when triggering the inference cycle, it includes:

[0118] The policy inference function is called at a fixed period that matches the preset control frequency. The initial timestamp is recorded before the policy inference process begins, and the phased time value is calculated after the inference is completed.

[0119] Comparing the periodic time consumption value with the theoretical time consumption threshold corresponding to the fixed period, and triggering a resource dynamic allocation compensation strategy when it is detected that the periodic time consumption value continuously exceeds the theoretical time consumption threshold for a preset number of times;

[0120] The time consumption data of each inference process is added to the performance statistical log in real time, and the performance statistical log is analyzed regularly to generate a performance analysis report including the average time consumption and the extreme time consumption. When it is detected that the stage time consumption exceeds the preset deviation threshold, the abnormal alarm mechanism is triggered.

[0121] Specifically, the policy inference task is triggered by a preset fixed control period (such as 50Hz corresponding to 20ms), and a timer (such as create_timer of ROS2) is used to ensure the stability of periodic scheduling to avoid control frequency drift due to system load fluctuations. The initial timestamp (such as time.time()) is recorded at the start of each inference, and the stage time consumption (such as inference time and total processing time) is calculated after the inference is completed. By comparing the theoretical time consumption threshold, such as 80% of 20ms, that is, 16ms, the real-time violation risk is detected in real time. The purpose is to establish a time baseline and provide a quantitative basis for system performance evaluation and dynamic resource allocation. For example, when the single inference time exceeds 16ms, a warning of potential timeout risk is issued to prevent motion instability caused by loss of step in the control loop.

[0122] When it is detected that the stage time consumption continues to exceed the theoretical threshold (such as exceeding 16ms three times in a row), the dynamic resource allocation strategy is triggered to compensate for the real-time loss by adjusting computing resources (such as GPU memory allocation priority), reducing data accuracy or enabling redundant computing nodes (such as multi-core parallelism). Its purpose is to quickly restore control frequency stability when the system is overloaded. For example, when the network inference time suddenly increases, the temporary engine switches to another gear mode to reduce the amount of calculation, ensuring that the inference task is completed within a 20ms cycle and avoiding motion control interruption.

[0123] The timing data for each inference is written to the performance statistics log in real time. Periodic analysis (e.g., every minute) generates an analysis report containing the average timing (e.g., 15.2ms) and extreme timings (maximum 22ms, minimum 12ms). When the timing data consistently deviates from a preset deviation threshold (e.g., ±3ms), an anomaly alert (e.g., email / SMS notification) is triggered, and the stack trace is recorded for offline diagnosis. This establishes a long-term performance monitoring mechanism that can identify systemic bottlenecks (e.g., memory leaks) through historical trend analysis, while also providing real-time warnings for sudden anomalies (e.g., GPU frequency reduction due to excessive temperature).

[0124] Preferably, the second control submodule further includes a communication module;

[0125] The communication module is used to encapsulate sensor data into data packets with integrity verification, including:

[0126] receiving a binary data stream carrying multiple source sensors through a preset communication protocol, and verifying that a packet header identifier of the binary data stream matches a preset packet header identifier;

[0127] Parse the data packet type field and identify it as a sensing data packet or a heartbeat maintenance packet. If it is identified as a sensing data packet, extract the floating-point parameter set in the data payload based on the data length field.

[0128] Performing byte order conversion and data type decoding operations on the floating-point parameter set to generate a sensing parameter set, and mapping different physical quantity parameters in the sensing parameter set to a ROS2 message data structure to generate the data packet.

[0129] Binary data streams from multiple sensors are received via a preset communication protocol (such as a custom UDP protocol). The system first verifies the legitimacy of the packet header identifier (such as 0x55 0xAA) and filters out illegal or damaged data packets (such as garbled data caused by noise interference) to ensure the reliability of the data source. After parsing the packet type field (such as the 1-byte 0x01 for a sensor data packet and 0x02 for a heartbeat maintenance packet), differentiated processing is performed based on the type: the sensor data packet requires the extraction of a floating-point parameter set (such as joint angles and IMU acceleration), while the heartbeat packet only needs to update the connection status. This aims to establish a data access mechanism at the communication layer to prevent invalid data from entering subsequent processing flows. At the same time, efficient resource allocation is achieved through type diversion (such as separating the sensor data processing thread from the heartbeat maintenance thread).

[0130] After determining that it is a sensor data packet based on the packet type field, the data payload part is accurately intercepted based on the data length field (such as 2 bytes of 0x0018 indicating a payload length of 24 bytes), and the original byte stream is converted into a floating-point parameter set. Data compatibility issues such as byte order differences on heterogeneous hardware platforms are resolved through byte order conversion (such as big endian to little endian). Then, according to the preset data type (such as float32, int16), the sensor parameter set (such as 12 float32 values ​​for joint angles and 3 float32 values ​​for IMU three-axis acceleration) is decoded to generate a sensor parameter set. Its purpose is to convert the original binary data into structured data that can be used by upper-level applications, while ensuring the numerical accuracy and unit consistency of the physical quantity parameters.

[0131] The decoded sensor parameter set is mapped to the ROS2 message data structure according to the physical quantity type. Through the ROS2 interface definition language, standardized message types are generated to achieve seamless integration with the ROS2 ecosystem. Its purpose is to build a unified middleware interface so that heterogeneous sensor data can be directly consumed by upper-level modules such as navigation and control. At the same time, ROS2's QoS policies (such as reliability and persistence) are used to optimize data transmission efficiency and reliability.

[0132] Furthermore, the communication module also includes a heartbeat detection mechanism:

[0133] Receive a heartbeat data packet carrying a status identifier, verify the matching of a preset identifier and a protocol identifier in the heartbeat data packet, and extract a timestamp and a status code;

[0134] Update the current status record according to the timestamp, reset the counting period of the timeout timer, and execute the degradation control strategy when the number of consecutive heartbeat data packet losses exceeds the preset timeout threshold, trigger the automatic reconnection process and broadcast the connection abnormality reminder to the first control subsystem;

[0135] A heartbeat confirmation signal including the operating status is periodically sent to the second control subsystem. When it is detected that the communication link is restored, the degraded control strategy is released and switched to the standard control strategy.

[0136] Specifically, the legitimacy of the heartbeat data packet is verified through the preset protocol identifier (such as the packet header 0x55 0xAA), noise interference or illegal data packets are filtered out to ensure the credibility of the communication link, the timestamp is extracted to update the time base of the current connection status, and the timeout timer (such as a 500ms countdown) is reset to maintain the active connection status; the status code (such as 0x00 normal, 0x01 warning) is used to diagnose the operating health of the remote device (such as the STM32 controller). Its purpose is to monitor the quality of the communication link in real time, and determine whether there is packet loss or delay anomalies through the continuity of the timestamp, providing a basic guarantee for the stability of the subsequent control strategy.

[0137] Then, by counting the number of consecutive heartbeat packet losses (such as timeout threshold = 3 times), the connection is judged to be interrupted, and the degradation control strategy is triggered (such as the joint motor enters the damping mode) to avoid the risk of loss of control. At the same time, the automatic reconnection process is started (such as trying the handshake protocol every 200ms). Its purpose is to maintain the system's minimum safe operation capability in the event of communication anomalies. For example, when the wireless signal is interrupted, it switches to the pre-stored trajectory tracking mode and broadcasts abnormal reminders to the upper-level system (such as the navigation module) to ensure global state synchronization.

[0138] After detecting that the heartbeat packet has been restored (such as two consecutive successful receptions), the degraded control strategy is released and switched back to the standard control strategy (such as full-degree-of-freedom motion planning). At the same time, the timeout counter (such as clearing the number of loss statistics) and the number of reconnection attempts are reset. The purpose is to quickly return to a high-precision control state after communication is restored. For example, the IMU data fusion algorithm is re-enabled to improve attitude stability, and the state machines at both ends are synchronized through the heartbeat confirmation signal (such as ACK response) to ensure the timing consistency of the control instructions.

[0139] A humanoid robot control method based on reinforcement learning, such as Figure 2 As shown, the humanoid robot control method is applied to the humanoid robot control system as described above, and the humanoid robot control method includes the following steps:

[0140] S1: Based on the synchronous acquisition of joint motion parameters, posture perception data and contact force information by sensors, the timestamp alignment and outlier filtering of sensor data are completed;

[0141] S2: Fusing multi-source sensor data and constructing time series state features that include real-time measurements and historical time series information;

[0142] S3: Generate multi-joint angle target values ​​of the robot through a reinforcement learning strategy network based on the temporal state characteristics, wherein the reinforcement learning strategy network adopts a deep learning framework to achieve parallel computing acceleration;

[0143] S4: Analyze the multi-joint angle target value, select a control mode, and generate a control command including a parameter adjustment instruction;

[0144] S5: Receive the control command, convert the multi-joint target angle of the robot into a motor drive signal to drive the joints of the robot, and monitor the actuator status in real time to trigger a safety protection mechanism.

[0145] Throughout this specification, reference to terms such as "one embodiment," "some embodiments," "illustrative embodiments," "examples," "specific examples," or "some examples" means that a specific feature, structure, material, or characteristic described in conjunction with that embodiment or example is included in at least one embodiment or example of the present invention. In this specification, illustrative uses of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in any one or more embodiments or examples.

[0146] While embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions, and variations may be made to the embodiments without departing from the principles and spirit of the invention, and that the scope of the invention is defined by the claims and their equivalents.

Claims

1. A humanoid robot control system based on reinforcement learning, characterized in that: The humanoid robot control system includes a first control subsystem and a second control subsystem; The first control subsystem includes a strategy reasoning module, a state transition module and a robot control module; The second control subsystem includes a data acquisition module and a drive control module; The data acquisition module is used to synchronously collect joint motion parameters, posture perception data and contact force information based on sensors, complete the timestamp alignment and outlier filtering of sensor data, and transmit them to the state conversion module; The state conversion module is used to fuse multi-source sensor data and construct time series state features that include real-time measurements and historical time series information; The strategy reasoning module is used to generate the robot's multi-joint angle target values ​​through a reinforcement learning strategy network based on the temporal state features provided by the state transition module. The reinforcement learning strategy network uses a deep learning framework to achieve parallel computing acceleration. The robot control module is used to analyze the multi-joint angle target values ​​output by the strategy reasoning module, select the control mode, and generate control commands including parameter adjustment instructions; The drive control module is used to receive control commands from the robot control module, convert the robot's multi-joint target angles into motor drive signals to drive the robot's joints, and monitor the actuator status in real time to trigger the safety protection mechanism; The state transition module includes: Receive data from multiple source sensors and synchronize timestamps of joint encoder data and inertial measurement data; Constructing a single-frame observation vector including periodic phase encoding, motion parameters, and posture information, wherein: the periodic phase encoding is achieved by converting the gait periodic phase value into a sine-cosine dual-channel encoding, the motion parameters are normalized based on the target velocity instruction; the posture information is obtained by fusing joint kinematics solution with inertial measurement data; The current single-frame observation vector is stacked with a preset number of historical frame data in time series. When the number of historical frames is less than the preset number, the missing frames are filled with zero vectors to obtain the time series state features; The strategy reasoning module includes: Constructing a reinforcement learning strategy network including three hidden layers, and inputting the temporal state features into the reinforcement learning strategy network; The reinforcement learning policy network infers the optimal action with GPU acceleration and outputs a multi-dimensional joint angle target vector whose dimension matches the number of robot joints. An inference trigger cycle is set, and joint motion range limiting processing is performed on the output vector according to the inference trigger cycle to generate multi-joint angle target values ​​of the robot.

2. The humanoid robot control system according to claim 1, characterized in that: The state transition module includes: receiving original position parameters and original velocity parameters of multiple joints in real time, performing a first normalization process on the original position parameters after subtracting pre-stored joint position reference values, and generating a normalized joint position feature vector; The original velocity parameter is multiplied by the motion state scaling coefficient for a second normalization process to generate a normalized joint velocity feature vector, the normalized joint position feature vector is filled into the first dimension interval in the observation vector, the normalized joint velocity feature vector is filled into the second dimension interval in the observation vector, and the normalized joint position feature vector and the normalized joint velocity feature vector are stored in a history cache queue in a time series.

3. The humanoid robot control system according to claim 1, characterized in that: The state transition module includes: Receiving multi-source attitude parameters, the multi-source attitude parameters including joint attitude Euler angle data and angular velocity and linear acceleration data of an inertial measurement unit, performing quaternion conversion calculation on the Euler angle data, and generating a quaternion attitude representation including spatial rotation information; Multiplying the angular velocity data by a first scaling factor to perform first motion feature normalization processing to generate a normalized angular velocity feature vector, multiplying the Euler angle data by a second scaling factor to perform second posture feature normalization processing to generate a normalized posture feature vector, filling the normalized angular velocity feature vector into a third dimensional interval in the observation vector, and filling the normalized posture feature vector into a fourth dimensional interval in the observation vector; A posture monitoring message including coordinate transformation parameters is generated according to the quaternion posture representation and is visualized, and the normalized angular velocity feature vector and the normalized posture feature vector are stored in a history cache queue in a time series.

4. The humanoid robot control system according to claim 1, characterized in that: The robot control module includes: Analyzing the multi-joint angle target values ​​and converting the multi-joint angle target values ​​into a numerical array; Detect the numerical distribution characteristics of each dimension and select the current control mode based on the preset dimension grouping threshold, where: If it is detected that the dimension value of the left leg in the numerical array is greater than the dimension value of the right leg and the difference between the two is greater than a first preset value, the left leg control mode is activated; If it is detected that the dimension value of the right leg in the numerical array is greater than the dimension value of the left leg and the difference between the two is greater than a first preset value, the right leg control mode is activated; If all dimension values ​​are within the preset balance range and the difference between each dimension is less than the set second preset value, the balance maintenance mode is activated; Dynamically adjust joint control parameters based on the identified control mode, including: adding periodic compensation to the target joint angle in walking mode, increasing the proportional gain coefficient of the joint position loop in balance mode, and shortening the control period and increasing the differential gain coefficient in recovery mode; The adjusted multi-joint angle target values ​​and control parameters are encapsulated into a structured data packet, added with a protocol identifier matching the control mode, and then transmitted to the drive control module.

5. The humanoid robot control system according to claim 1, characterized in that: When an inference cycle is triggered, it includes: Call the policy inference function at a fixed period that matches the preset control frequency, record the initial timestamp before the policy inference process begins, and calculate the phased time value after the inference is completed; Comparing the periodic time consumption value with the theoretical time consumption threshold corresponding to the fixed period, and triggering a resource dynamic allocation compensation strategy when it is detected that the periodic time consumption value continuously exceeds the theoretical time consumption threshold for a preset number of times; The time consumption data of each inference process is added to the performance statistical log in real time, and the performance statistical log is analyzed regularly to generate a performance analysis report including the average time consumption and the extreme time consumption. When it is detected that the stage time consumption exceeds the preset deviation threshold, the abnormal alarm mechanism is triggered.

6. The humanoid robot control system according to claim 1, characterized in that: The second control subsystem further includes a communication module; The communication module is used to encapsulate sensor data into data packets with integrity verification, including: receiving a binary data stream carrying multiple source sensors through a preset communication protocol, and verifying that a packet header identifier of the binary data stream matches a preset packet header identifier; Parse the data packet type field and identify it as a sensing data packet or a heartbeat maintenance packet. If it is identified as a sensing data packet, extract the floating-point parameter set in the data payload based on the data length field. Performing byte order conversion and data type decoding operations on the floating-point parameter set to generate a sensing parameter set, and mapping different physical quantity parameters in the sensing parameter set to a ROS2 message data structure to generate the data packet.

7. The humanoid robot control system according to claim 6, characterized in that: The communication module also includes a heartbeat detection mechanism: Receive a heartbeat data packet carrying a status identifier, verify the matching of a preset identifier and a protocol identifier in the heartbeat data packet, and extract a timestamp and a status code; Update the current status record according to the timestamp, reset the counting period of the timeout timer, and execute the degradation control strategy when the number of consecutive heartbeat data packet losses exceeds the preset timeout threshold, trigger the automatic reconnection process and broadcast the connection abnormality reminder to the first control subsystem; A heartbeat confirmation signal including the operating status is periodically sent to the second control subsystem. When it is detected that the communication link is restored, the degraded control strategy is released and switched to the standard control strategy.

8. A humanoid robot control method based on reinforcement learning, characterized in that: The humanoid robot control method is applied to the humanoid robot control system according to any one of claims 1 to 7, and the humanoid robot control method includes: Based on the synchronous collection of joint motion parameters, posture perception data and contact force information by sensors, the timestamp alignment and outlier filtering of sensor data are completed; Fuse multi-source sensor data and construct time series state features that include real-time measurements and historical time series information; According to the temporal state characteristics, a reinforcement learning strategy network is used to generate target values ​​for the multi-joint angles of the robot, wherein the reinforcement learning strategy network adopts a deep learning framework to achieve parallel computing acceleration; Analyzing the target values ​​of the multi-joint angles, selecting a control mode, and generating a control command including a parameter adjustment instruction; The control command is received, the multi-joint target angle of the robot is converted into a motor drive signal to drive the joints of the robot, and the actuator status is monitored in real time to trigger a safety protection mechanism.

Images