Model-based prediction for humanoid robot control methods and systems

Abstract

This invention provides a model-based prediction-based humanoid machine control method and system. First, it acquires joint sensing data (hip joint motion angle sequence, knee joint force signal) and environmental interaction data (depth image stream data, foot contact pressure data) from the humanoid machine. These data are then processed through feature association to generate a dynamic state feature set, including joint coordination feature vectors and environmental constraint feature matrices. Next, a pre-trained model is invoked to perform spatiotemporal joint prediction on the feature set, outputting a future motion control sequence containing gait cycle features. Based on this, a joint synchronization control model is constructed, generating a drive control instruction set with time coordination constraints. Finally, the drive control instruction set is transmitted to a distributed execution unit, which collects joint state feedback data for the next round of processing, thereby improving the motion control performance of the humanoid machine in complex environments.

Description

Technical Field

[0001] This invention relates to the field of robot control technology, and more specifically, to a humanoid robot control method and system based on model prediction. Background Technology

[0002] In the field of humanoid robot control, achieving precise, stable, and adaptable motion control in complex environments has always been a key challenge. Traditional humanoid robot control methods often rely on preset motion patterns or simple feedback adjustment mechanisms. For example, some methods rely solely on joint sensor data, such as joint angles or force signals, for independent joint control. However, these approaches neglect the collaborative relationships between the humanoid robot's joints and its dynamic interaction with the surrounding environment. This results in poor motion coordination and a tendency to lose balance when facing complex terrain or external disturbances.

[0003] Other methods, while considering environmental information, often only make local adjustments based on environmental data, lacking comprehensive analysis and prediction of the robot's overall dynamic state. For example, adjusting gait solely based on plantar contact pressure data, without combining joint movement and environmental spatial information reflected in depth images, makes it difficult for the robot to plan motion strategies to adapt to the environment in advance, severely limiting its motion flexibility and adaptability in complex scenarios. Therefore, existing technologies struggle to meet the needs of humanoid robots for efficient and stable motion control in diverse and dynamically changing environments. Summary of the Invention

[0004] In view of the aforementioned problems, and in conjunction with the first aspect of the present invention, embodiments of the present invention provide a humanoid machine control method based on model prediction, the method comprising:

[0005] Acquire joint sensing data and environmental interaction data of a humanoid robot. The joint sensing data includes hip joint motion angle sequence and knee joint force signal. The environmental interaction data includes depth image stream data and plantar contact pressure data.

[0006] The joint sensing data and the environmental interaction data are subjected to feature association processing to generate a dynamic state feature set of the humanoid machine. The dynamic state feature set includes joint coordination feature vectors and environmental constraint feature matrices.

[0007] A pre-trained motion sequence prediction model is invoked to perform spatiotemporal joint prediction processing on the dynamic state feature set, and the output is a future motion control sequence containing gait cycle features. The future motion control sequence includes the angle change trajectory of each joint and the force adjustment parameters of the actuator.

[0008] Based on the future motion control sequence, a joint synchronization control model is constructed, and a drive control instruction set containing time coordination constraints is generated. The drive control instruction set includes the position drive signal and force feedback adjustment parameters for each joint.

[0009] The drive control instruction set is transmitted to the distributed execution unit of the humanoid machine, the drive control instruction set is executed and the joint state feedback data during the movement is collected, and the joint state feedback data is used as the input data for the next round of feature association processing.

[0010] In another aspect, embodiments of the present invention also provide a humanoid machine control system based on model prediction, including a processor and a machine-readable storage medium connected to the processor. The machine-readable storage medium is used to store programs, instructions, or code, and the processor is used to execute the programs, instructions, or code in the machine-readable storage medium to implement the above-described method.

[0011] Based on the above, by comprehensively acquiring joint sensing data and environmental interaction data of the humanoid robot, and performing feature association processing to generate a dynamic state feature set containing joint coordination feature vectors and environmental constraint feature matrices, a comprehensive and accurate characterization of the robot's own motion state and its interaction with the environment is achieved. A pre-trained motion sequence prediction model is then invoked to perform spatiotemporal joint prediction processing on the dynamic state feature set, outputting a future motion control sequence containing gait periodic features. This allows for advance planning of the robot's motion over a future period, enabling the robot to better adapt to environmental changes and make adjustments in advance, thus enhancing the predictability and stability of its motion. Based on the future motion control sequence, a joint synchronization control model is constructed, generating a drive control instruction set containing time coordination constraints. This ensures the synchronization and coordination of each joint during motion, avoiding motion errors and energy waste caused by asynchronous joint movements. Finally, the drive control instruction set is transmitted to a distributed execution unit, and joint state feedback data is collected as input for the next round. This allows for real-time adjustment of the control strategy, further improving the accuracy and adaptability of control, enabling the robot to achieve efficient, stable, and flexible motion in various complex environments. Attached Figure Description

[0012] Figure 1 This is a schematic diagram of the execution flow of the humanoid machine control method based on model prediction provided in an embodiment of the present invention.

[0013] Figure 2 This is a schematic diagram of exemplary hardware and software components of a model-predictive humanoid machine control system provided in an embodiment of the present invention. Detailed Implementation

[0014] The present invention will now be described in detail with reference to the accompanying drawings. Figure 1 This is a flowchart illustrating a model-predictive humanoid machine control method according to an embodiment of the present invention. The following is a detailed description of the model-predictive humanoid machine control method.

[0015] This embodiment uses the scenario of a humanoid robot walking indoors as an example to illustrate in detail the specific implementation process of the humanoid robot control method based on model prediction, so that those skilled in the art can clearly understand and successfully implement the method.

[0016] Step S110: Acquire joint sensing data and environmental interaction data of the humanoid robot. The joint sensing data includes hip joint motion angle sequence and knee joint force signal. The environmental interaction data includes depth image stream data and plantar contact pressure data.

[0017] During humanoid robot walking, angle sensors installed at the hip joint operate continuously at a fixed sampling frequency, collecting the hip joint's active angle at equal time intervals. These angle values ​​are arranged in chronological order of collection, forming a sequence of hip joint active angles. For example, during walking, the hip joint undergoes a process where it starts at an initial angle, gradually increases in angle with each step, reaches its maximum angle, and then gradually decreases back to near the initial angle. This sequence of hip joint active angles fully records this change.

[0018] Force sensors at the knee joint also collect data at a fixed frequency, sensing the tension and pressure experienced by the knee joint during movement, and generating a force signal for the knee joint. When the humanoid robot's foot touches the ground to support its body, the force on the knee joint increases, and when the foot leaves the ground and swings, the force decreases. This force signal for the knee joint can reflect these changes in real time.

[0019] A depth camera mounted on the front of the humanoid robot's torso continuously captures images of the environment ahead. Each frame contains distance information between the camera and objects in the scene, and these consecutive images together form a depth image stream. This data allows for the identification of environmental information such as the presence of obstacles and the smoothness of the ground.

[0020] The pressure sensor array on the sole of the foot continuously monitors the pressure distribution when the foot contacts the ground during walking, generating foot contact pressure data. When the heel strikes the ground, the pressure in the heel area increases first; when the ball of the foot lands to support the body, the pressure in the forefoot area becomes the main component; when the foot leaves the ground, the pressure in each area gradually decreases to zero.

[0021] Step S120: Perform feature association processing on the joint sensing data and the environmental interaction data to generate a dynamic state feature set of the humanoid machine. The dynamic state feature set includes joint coordination feature vectors and environmental constraint feature matrices.

[0022] After acquiring joint sensing data and environmental interaction data, it is necessary to process these raw data, extract features that reflect the machine state and environmental constraints, and associate them to form a dynamic state feature set.

[0023] Step S121: Extract time-domain fluctuation features from the hip joint motion angle sequence in the joint sensing data. Capture the periodic change features of the angle sequence through a sliding time window, and extract the main peak periodic features and fluctuation amplitude features of the angle fluctuation as dynamic features of the hip joint.

[0024] First, set the length of the sliding time window, which is determined based on the gait cycle of a humanoid robot during normal walking, ensuring that the window can cover the angle changes within a complete gait cycle. Set the sliding step size to a value smaller than the window length to guarantee the continuity of feature extraction.

[0025] The sliding time window is moved sequentially across the hip joint range of motion angle sequence. For each angle sequence within the window, a time-domain analysis is performed. By calculating the autocorrelation function of the angle sequence, the time interval corresponding to the maximum value of the autocorrelation function is found. This time interval is the main peak periodic characteristic of the angle fluctuation, which reflects the main periodic pattern of hip joint movement.

[0026] Simultaneously, the difference between the maximum and minimum values ​​of the angle sequence within the window is calculated to obtain the fluctuation amplitude feature, which reflects the range of change in the hip joint's range of motion angles within the corresponding time period of the window. The main peak periodic features and fluctuation amplitude features extracted from each window are arranged in sequence to form the dynamic features of the hip joint.

[0027] Step S122: Perform frequency band component analysis on the force signal of the knee joint, decompose the force signal into component signals of different frequency ranges, and extract the energy ratio characteristics and waveform distortion characteristics of each component signal as knee joint load characteristics.

[0028] The force signal of the knee joint is processed by wavelet decomposition algorithm, which decomposes it into multiple preset frequency ranges to obtain the component signals of each frequency band. The frequency range is determined according to the frequency of force changes that may occur in the knee joint during walking. For example, the low frequency range corresponds to slow load changes, and the high frequency range corresponds to rapid impact loads.

[0029] For each component signal, its energy value is calculated, and then the energy value is divided by the total energy value of all component signals to obtain the energy proportion characteristic of that component signal. This energy proportion characteristic reflects the importance of the force change in that frequency range in the overall force signal.

[0030] The deviation of the component signal waveform from the standard sine wave is calculated, specifically by calculating the kurtosis value of the waveform. The larger the kurtosis value, the more severe the waveform distortion, which serves as a waveform distortion characteristic. This waveform distortion characteristic reflects the irregularity of the force applied to the knee joint. The energy proportion characteristics of all component signals and the waveform distortion characteristic are combined to form the knee joint load characteristics.

[0031] Step S123: The dynamic features of the hip joint and the load features of the knee joint are concatenated by feature dimensions to generate a joint coordination feature vector. The dimension of the joint coordination feature vector is equal to the sum of the dimensions of the dynamic features of the hip joint and the load features of the knee joint. The coupling representation of the motion state of the lower limb joints is realized through feature concatenation.

[0032] The dynamic characteristics of the hip joint are a multidimensional vector, assuming its dimension is a, with each dimension corresponding to the main peak period characteristics or fluctuation amplitude characteristics of different windows; the load characteristics of the knee joint are also a multidimensional vector, with dimension b, with each dimension corresponding to the energy proportion characteristics or waveform distortion characteristics of different component signals.

[0033] The dynamic features of the hip joint and the load features of the knee joint are concatenated dimensionally. Specifically, each dimension of the knee joint load feature is sequentially added to the dimension of the hip joint dynamic feature, forming a joint-coordinated feature vector with dimension a+b. This concatenation method preserves the information of both features, enabling the joint-coordinated feature vector to simultaneously reflect the motion state of the hip joint and the load condition of the knee joint, thus achieving a coupled representation of the lower limb joint motion state.

[0034] Step S124: Perform spatial structure analysis on the depth image stream data in the environmental interaction data, generate scene depth information through stereo matching, extract the spatial location distribution features of obstacles and the ground slope change features, and construct an environmental spatial feature matrix.

[0035] Each frame of the depth image stream is preprocessed, including noise removal and image enhancement, to improve image quality. A stereo matching algorithm is used to calculate the 3D coordinates of each point in the scene by comparing the positional differences of corresponding points in images taken by two cameras at the same time, thereby generating scene depth information.

[0036] Based on scene depth information, a distance threshold is set, and objects within a distance of less than this threshold are identified as obstacles. For each identified obstacle, its 3D coordinates are recorded. These coordinates are then divided into a defined spatial grid, and the number and average distance of obstacles within each grid are counted to form the spatial distribution characteristics of the obstacles.

[0037] Simultaneously, points in the ground region are selected from the scene depth information, and the ground slope is calculated by fitting a plane equation. The ground is divided into multiple small regions, and a slope value is calculated for each region, forming ground slope variation characteristics. The spatial distribution characteristics of obstacles and ground slope variation characteristics are arranged in rows and columns to construct an environmental spatial feature matrix.

[0038] Step S125: Extract regional pressure features from the plantar contact pressure data, divide the plantar area into the forefoot area, midfoot area and heel area, and extract the pressure peak features and pressure center offset features of each area as plantar interaction features.

[0039] Based on the anatomical structure of the foot, the foot sensor array is divided into the forefoot, midfoot, and heel regions, with each region containing multiple pressure sensors. For each region, the maximum value collected by the pressure sensors within a gait cycle is identified as the peak pressure characteristic of that region, reflecting the maximum pressure experienced by that region during walking.

[0040] Calculate the coordinates of the point of application of the resultant pressure in each region, i.e., the pressure center. Then, calculate the offset of this pressure center relative to the geometric center of the region, including the offset in the x and y directions, as the pressure center offset feature. This pressure center offset feature reflects the distribution and offset of pressure within the region. Combine the pressure peak features and pressure center offset features of the three regions in sequence to form the plantar interactive feature.

[0041] Step S126: The environmental spatial feature matrix and the foot interaction features are fused using tensor dimensions to generate an environmental constraint feature matrix. The row dimension of the environmental constraint feature matrix corresponds to the spatial location distribution, and the column dimension corresponds to the pressure interaction features.

[0042] The environmental spatial feature matrix is ​​a two-dimensional matrix, assuming its dimension is m×n, where m corresponds to the number of rows of the spatial position and n corresponds to the number of columns of the spatial position. The plantar interaction feature is a one-dimensional vector with dimension p.

[0043] The plantar interaction features are expanded into a matrix with the same number of rows, m, as the environmental space feature matrix. This expansion is achieved by repeating each element of the plantar interaction feature m times, forming an m×p matrix. Then, the environmental space feature matrix and the expanded plantar interaction feature matrix are concatenated column-wise to generate an m×(n+p) environmental constraint feature matrix.

[0044] In this way, the row dimension of the environmental constraint feature matrix still corresponds to the original spatial location distribution, while the column dimension includes the original spatial features and the newly added pressure interaction features, thus realizing the fusion of environmental spatial information and foot interaction information.

[0045] Step S127: Align and associate the joint coordination feature vector and the environmental constraint feature matrix according to the time acquisition order, establish the correlation between joint motion features and environmental constraint features through a nonlinear mapping function, and generate a dynamic state feature set containing the kinematic and environmental interaction relationship. Each feature element of the dynamic state feature set contains joint motion parameters and corresponding environmental constraint parameters.

[0046] First, obtain the acquisition timestamps of the joint coordination feature vector and the environmental constraint feature matrix. Then, align the two in chronological order according to the timestamps to ensure that the joint coordination features and environmental constraint features at the same time point correspond to each other.

[0047] A multilayer perceptron is used as the nonlinear mapping function. This multilayer perceptron includes an input layer, a hidden layer, and an output layer. The aligned joint cooperative feature vector and the expanded vector of the environmental constraint feature matrix are used as inputs to the input layer. After nonlinear transformation in the hidden layer, the associated features are obtained from the output layer.

[0048] The aforementioned associated features are arranged in chronological order to form a dynamic state feature set. Each feature element contains corresponding joint motion parameters (such as hip joint angle and knee joint force-related features) and environmental constraint parameters (such as obstacle position, ground slope, and plantar pressure-related features), fully reflecting the interaction between machine motion and the environment.

[0049] Step S130: Call the pre-trained motion sequence prediction model to perform spatiotemporal joint prediction processing on the dynamic state feature set, and output a future motion control sequence containing gait cycle features. The future motion control sequence includes the angle change trajectory of each joint and the force adjustment parameters of the actuator.

[0050] The pre-trained motion sequence prediction model is trained using a large amount of humanoid robot walking data and can predict future motion based on the input dynamic state feature set. After the dynamic state feature set is input into the model, it performs spatiotemporal joint prediction processing and finally outputs a future motion control sequence containing gait cycle features.

[0051] Step S131: Input the dynamic state feature set into the spatiotemporal feature encoding module of the motion sequence prediction model, mine key spatial location features in the environmental constraint feature matrix through a spatial attention mechanism, and capture the temporal dependencies in the joint coordination feature vector through a temporal attention mechanism to generate a spatiotemporal correlation feature tensor; the spatial attention mechanism is implemented by calculating the influence weight of each spatial location in the environmental constraint feature matrix on joint motion, and the temporal attention mechanism is implemented by calculating the correlation weight of features at different time points in the joint coordination feature vector.

[0052] The dynamic state feature set contains information from both time and space perspectives. The role of the spatiotemporal feature encoding module is to encode this information, highlight key information, and establish spatiotemporal correlations.

[0053] Step S1311: Add time position features to the joint coordination feature vector in the dynamic state feature set. By generating time identifier features of different periods and superimposing them with the joint coordination feature vector, time-enhanced joint features are obtained.

[0054] Generate various time signature features with different periods, such as sine and cosine wave features with periods of 1 / 4, 1 / 2, and 1 times the gait period. The dimensions of the aforementioned time signature features are the same as the dimensions of the joint coordination feature vector.

[0055] Each time signature feature is element-wise added to the joint co-location feature vector to obtain multiple intermediate features. These intermediate features are then concatenated dimensionally to form the time-enhanced joint features. In this way, the joint co-location feature vector incorporates temporal location information, which helps the model capture temporal dependencies.

[0056] Step S1312: Add spatial location features to the environmental constraint feature matrix in the dynamic state feature set. By generating a spatial identifier feature matrix that reflects the spatial coordinate position and superimposing it with the environmental constraint feature matrix, spatial enhanced environmental features are obtained. The spatial identifier feature matrix is ​​realized by an eigenvalue matrix that reflects the spatial position relationship.

[0057] Based on the spatial range corresponding to the environmental constraint feature matrix, a spatial identification feature matrix is ​​generated. Each element in this spatial identification feature matrix is ​​a feature value obtained by normalizing the coordinates of its location. For example, the x-coordinate and y-coordinate are normalized to the range of 0-1, which serves as the spatial identification feature of that location.

[0058] The spatial identifier feature matrix has the same dimension as the environmental constraint feature matrix. By adding the two at the element level, we obtain the spatially enhanced environmental features, which makes the environmental constraint feature matrix contain clear spatial location information.

[0059] Step S1313: Use the temporal augmentation joint features as query features, and the spatial augmentation environment features as key features and value features. Generate a spatial attention weight matrix by calculating the correlation between the query features and the key features.

[0060] Temporally augmented joint features are used as query features Q, and spatially augmented environment features are flattened and used as key features K and value features V. The dot product between query features Q and key features K is calculated, and then the result is normalized using the softmax function to obtain the spatial attention weight matrix W.

[0061] Each element in the weight matrix represents the correlation between the query feature and the corresponding key feature, that is, the influence weight of the spatial location in the environment on joint movement. The larger the weight, the more important the information of the spatial location is to joint movement.

[0062] Step S1314: Weight and aggregate the spatial enhancement environment features according to the spatial attention weight matrix to generate the environmental key location feature vector.

[0063] Perform matrix multiplication on the spatial attention weight matrix W and the value feature V to obtain the weighted feature matrix. Then flatten the matrix into a vector, which is the environmental key location feature vector.

[0064] The above weighted aggregation highlights the features of key locations in the environment that have a significant impact on joint movement, while suppressing the features of secondary locations, thus making the environmental features more focused.

[0065] Step S1315: The environmental key location feature vector and the time-enhanced joint feature are used to generate interactive features. The joint features and environmental features are fused through the correlation operation between feature elements to obtain spatiotemporal interactive features.

[0066] Element-level multiplication is performed on the environmental key location feature vectors and time-enhanced joint features to obtain the interaction feature matrix. Then, the matrix is ​​summed in the column direction to obtain a vector, which serves as the spatiotemporal interaction feature.

[0067] The above-mentioned operation method enables the interaction between joint features and environmental features. The resulting spatiotemporal interactive features contain both the temporal information of the joints and the key spatial information of the environment, reflecting the interaction between the two.

[0068] Step S1316: Increase the feature expression dimension of the spatiotemporal interaction features through feature dimension expansion processing to generate a spatiotemporal correlation feature tensor containing spatiotemporal correlation, so that the spatiotemporal correlation feature tensor contains both time dependency information and spatial location information.

[0069] A convolutional layer is used to process the spatiotemporal interaction features. The size and number of convolutional kernels are determined according to the required dimensions to be expanded. Through convolution operations, the dimensions of the spatiotemporal interaction features are expanded to generate a three-dimensional spatiotemporal correlation feature tensor, where the first dimension corresponds to time, and the second and third dimensions correspond to spatial location.

[0070] This spatiotemporal correlation feature tensor can clearly reflect both time-dependent information and spatial location information, as well as the correlation between the two.

[0071] Step S132: Input the spatiotemporal correlation feature tensor into the graph structure modeling layer of the motion sequence prediction model to construct a humanoid machine joint connection topology graph. Calculate the motion coupling relationship between joints through graph node feature propagation to generate graph structure motion features. The nodes of the joint connection topology graph represent each joint, and the edges represent the motion transmission relationship between joints.

[0072] The graph structure modeling layer aims to capture the motion relationships between joints, transforming spatiotemporal correlation feature tensors into features that reflect the coupling relationships between joints.

[0073] First, a joint connection topology graph is constructed based on the skeletal structure of the humanoid robot. The nodes in the graph correspond to the hip joint, knee joint, ankle joint, shoulder joint, elbow joint, wrist joint, etc. The edges between nodes are determined according to the connection relationship of the joints, such as the hip joint being connected to the knee joint, the knee joint being connected to the ankle joint, etc. The weight of the edges is initialized to 1, indicating that there is a motion transmission relationship.

[0074] Features corresponding to each joint are extracted from the spatiotemporal correlation feature tensor and used as the initial features for each node in the topological graph. Then, node features are propagated through graph convolution operations, and the features of each node are updated based on the features of its neighboring nodes, taking edge weights into account during the update process.

[0075] Specifically, for each node, the features of its neighboring nodes are weighted and summed (the weights are the edge weights), and then fused with the node's own features to obtain updated node features. Through multiple graph convolution operations, the node features are continuously propagated and updated. The final node features can reflect the motion coupling relationship between joints. These features are arranged in node order to form graph structure motion features.

[0076] Step S133: Input the graph structure motion features into the temporal prediction layer of the motion sequence prediction model, and generate joint motion state prediction values ​​for multiple future time steps through recursive prediction. The joint motion state prediction values ​​include the angle prediction values ​​of each joint and the force prediction parameters of the actuator.

[0077] The role of the time series prediction layer is to predict the joint motion state at multiple time steps in the future based on the motion characteristics of the graph structure. The recursive prediction method can make the prediction results more consistent with the continuity of the time series.

[0078] Step S1331: Input the graph structure motion features into the feature input unit of the time-series prediction layer, and determine the key motion features that should be included in the prediction at the current moment through the feature selection mechanism.

[0079] The feature input unit includes a feature importance evaluation module. This module assesses the importance of each feature by calculating the correlation between each dimension of the graph structure motion features and historical motion states. The higher the correlation, the more critical the feature is to predicting future motion states.

[0080] Set an importance threshold, and filter out features that are more important than the threshold as key motion features that should be included in the prediction at the current moment. These features may include the rate of change of hip joint angle, the trend of force change of knee joint, etc.

[0081] Step S1332: Input the key motion features and the predicted state features of the previous moment into the temporal memory unit. The long-term motion trend features are retained through the state update mechanism, while the short-term motion change features are updated. The temporal memory unit achieves collaborative memory of long-term and short-term features through feature fusion method.

[0082] The temporal memory unit comprises long-term memory subunits and short-term memory subunits. Key motion features and predicted state features from the previous time step are input into the two subunits respectively.

[0083] The long-term memory subunit extracts low-frequency components from the features through a low-pass filter operation. These low-frequency components reflect long-term motion trends, such as the stable trend of cadence during walking. The short-term memory subunit, on the other hand, extracts high-frequency components from the features through a high-pass filter operation, reflecting short-term motion changes, such as the rapid changes in stride angle during steps.

[0084] Then, an attention mechanism is used to fuse the output features of the long-term memory sub-unit and the short-term memory sub-unit. An attention weight is calculated for each sub-unit's output feature, with the weight determined by the feature's importance to the current prediction; features with higher importance have larger weights. The output features of the two sub-units are then weighted and summed according to their attention weights to obtain the fused memory feature, achieving collaborative memorization of long-term and short-term features.

[0085] Step S1333: The output features of the temporal memory unit are processed by the feature conversion unit to generate the joint motion state prediction value at the current moment. The joint motion state prediction value includes angle prediction features and force prediction parameters. The feature conversion unit realizes the conversion from features to prediction values ​​through a nonlinear mapping method.

[0086] The feature transformation unit contains multiple fully connected layers and activation functions. The number of neurons in each fully connected layer is determined by the dimensions of the input features and the output predicted values. The fused memory features output from the temporal memory unit are input into the first fully connected layer. After a linear transformation, they undergo non-linear processing using the ReLU activation function to obtain intermediate features.

[0087] The intermediate features are then input into the next fully connected layer, where linear transformations and activation functions are applied again. This process is repeated until the last fully connected layer. The output dimension of the last fully connected layer is consistent with the dimension of the joint motion state prediction values. One part of the output corresponds to the angle prediction features of each joint, and the other part corresponds to the force prediction parameters of the actuator.

[0088] Through the above multi-layer nonlinear mapping, the fused memory features are converted into specific joint motion state predictions, ensuring that the predictions can reflect the long-term and short-term motion information contained in the memory features.

[0089] Step S1334: Input the predicted joint motion state value at the current moment as the predicted state feature of the next moment into the temporal memory unit, repeat the feature input, temporal memory update and feature transformation process, and recursively generate the predicted joint motion state value for subsequent time steps.

[0090] After obtaining the predicted value of the joint motion state at the current moment, it is used as the predicted state feature for the next moment, and input into the temporal memory unit along with the key motion features for the next moment. Following the process of steps S1331 to S1333, feature selection, temporal memory update, and feature transformation are performed again to generate the predicted value of the joint motion state for the next moment.

[0091] This process is repeated recursively to generate predicted joint motion states for each subsequent time step, enabling the prediction sequence to continuously reflect joint motion conditions for multiple future time steps.

[0092] Step S1335: Determine the number of predicted time steps based on gait cycle characteristics. Stop recursive prediction when the generated joint motion state prediction values ​​cover the entire gait cycle, and obtain a sequence of joint motion state prediction values ​​for multiple future time steps. The number of time steps in the joint motion state prediction value sequence is consistent with the number of time divisions included in the gait cycle.

[0093] Based on the gait cycle characteristics of the humanoid robot, the number of time segments contained in a complete gait cycle is determined in advance; this number of time segments is the total number of time steps to be predicted. During the recursive prediction process, the number of time steps of the generated joint motion state prediction values ​​is counted in real time.

[0094] Recursive prediction stops when the number of statistical time steps reaches the preset total, meaning the generated predicted value sequence covers the entire gait cycle. At this point, the obtained joint motion state predicted value sequence contains the same number of time steps as the number of gait cycle time divisions, and can fully reflect the joint motion state in the next gait cycle.

[0095] Step S134: Perform kinematic feasibility verification on the predicted joint motion state value to ensure that the predicted angle is within the joint range of motion and the predicted force parameter is within the rated load range of the actuator, and generate a preliminary motion sequence.

[0096] The predicted values ​​of joint motion states may exceed the actual motion capabilities of humanoid robots. Therefore, kinematic feasibility verification is required to ensure the reasonableness of the prediction results.

[0097] For the angle prediction features of each joint, the pre-stored range of motion limit data for each joint is retrieved. The range of motion limit data for each joint includes the minimum and maximum range of motion angles. The predicted angle value of each joint at each time step is compared with the corresponding range of motion limit. If the predicted angle value is less than the minimum range of motion angle, it is adjusted to the minimum range of motion angle; if it is greater than the maximum range of motion angle, it is adjusted to the maximum range of motion angle; if it is within the range, it remains unchanged.

[0098] For the force prediction parameters of the actuators, the rated load range data of each actuator is retrieved. This rated load range data includes the minimum and maximum output force of each actuator. The force prediction parameter of each actuator at each time step is compared with the corresponding rated load range. If the force prediction parameter is less than the minimum output force, it is adjusted to the minimum output force; if it is greater than the maximum output force, it is adjusted to the maximum output force; if it is within the range, it remains unchanged.

[0099] After the above adjustments, the predicted joint motion states of all time steps are arranged in order to generate a preliminary motion sequence.

[0100] Step S135: Divide the preliminary motion sequence into gait cycles. Determine gait cycle features by analyzing the periodic changes in plantar contact pressure. The gait cycle features include sustained support phase features and sustained swing phase features. Embed the gait cycle features into the preliminary motion sequence to generate a future motion control sequence containing gait cycle features. The time step of the future motion control sequence is consistent with the time division of the gait cycle features.

[0101] Predictive features related to plantar contact pressure were extracted from the initial movement sequence, and the periodic changes of these features were analyzed. Based on the pattern of plantar contact pressure increasing, decreasing, returning to zero, and then increasing again, the start and end points of each gait cycle were determined.

[0102] Within each gait cycle, the phase is divided into a stance phase and a swing phase based on whether the plantar contact pressure is greater than zero. The stance phase is the stage where the foot is in contact with the ground and the pressure is greater than zero, while the swing phase is the stage where the foot leaves the ground and the pressure is zero. The number of time steps contained in the stance phase is calculated as the duration characteristic of the stance phase; the number of time steps contained in the swing phase is calculated as the duration characteristic of the swing phase. Together, they constitute the gait cycle characteristics.

[0103] The support phase continuity feature and swing phase continuity feature from the gait cycle feature are embedded into the corresponding positions of the preliminary motion sequence in chronological order, so that each time step in the sequence can be associated with its corresponding gait phase. After embedding, a future motion control sequence containing gait cycle features is generated. The time step of this future motion control sequence is consistent with the time division of the gait cycle feature, which can clearly reflect the correspondence between motion and gait phase.

[0104] Step S140: Construct a joint synchronization control model based on the future motion control sequence, and generate a drive control instruction set containing time coordination constraints. The drive control instruction set includes the position drive signal and force feedback adjustment parameters for each joint.

[0105] The future motion control sequence provides motion targets and gait information for each joint, and based on this, a joint synchronization control model is constructed to generate a set of drive control instructions that can achieve coordinated movement of each joint.

[0106] Step S141: Analyze the trajectory of each joint angle change in the future motion control sequence, extract the target angle features and angle change rate features of each joint at each time step, and construct joint kinematic constraint relationships, which reflect the correlation of angle changes of adjacent joints.

[0107] The trajectory of joint angle changes in the future motion control sequence is analyzed. For each joint, its angle value at each time step is extracted as the target angle feature. The angle change rate feature at each time step is obtained by calculating the difference between the target angle features of two adjacent time steps and dividing by the time interval.

[0108] Based on the limb structure of the humanoid machine, the connection relationships between adjacent joints are determined, such as the hip joint being adjacent to the knee joint, and the knee joint being adjacent to the ankle joint. For each pair of adjacent joints, the relationship between their target angle characteristics and angle change rate characteristics at each time step is analyzed. By calculating the correlation coefficient between the two, a joint kinematic constraint relationship is constructed, which quantifies the degree of correlation of angle changes between adjacent joints.

[0109] Step S142: Based on the joint kinematic constraints and the limb linkage structure characteristics of the humanoid machine, calculate the position driving parameters of each joint through inverse kinematics analysis. The position driving parameters include joint rotation angle parameters and rotation direction parameters.

[0110] The structural feature data of the humanoid robot's limbs and links is retrieved, including information such as the length and mass distribution of each link. Combining the kinematic constraints of the joints, inverse kinematics analysis is employed, using the target angle features of each joint as known quantities, to calculate the joint rotation angle parameters required to achieve the target angle.

[0111] Based on the sign of the rate of change of angle, the rotation direction parameter of the joint is determined. If the rate is positive, the rotation direction is positive; if the rate is negative, the rotation direction is negative. The joint rotation angle parameter and rotation direction parameter are combined to form the position drive parameters of each joint.

[0112] Step S143: Analyze the actuator force adjustment parameters in the future motion control sequence, combine them with the current load state characteristics of the joint, and calculate the actuator force feedback adjustment parameters through the force feedback adjustment mechanism. The force feedback adjustment parameters are used to compensate for the influence of external load changes during the motion process.

[0113] The force adjustment parameters of the actuator are extracted from the future motion control sequence, and the current load state characteristics of each joint are collected. These load state characteristics include the actual load force and load change rate currently experienced by the joint.

[0114] The force feedback adjustment mechanism calculates the force compensation amount based on the difference between the actuator's force adjustment parameters and the current load characteristics of the joint. If the actual load force is greater than the force adjustment parameters, the force compensation amount is negative, used to reduce the actuator's output force; if the actual load force is less than the force adjustment parameters, the force compensation amount is positive, used to increase the actuator's output force. Adding the force adjustment parameters to the force compensation amount yields the actuator's force feedback adjustment parameters, which can compensate for the influence of external load changes on motion.

[0115] Step S144: Input the position driving parameters and the force feedback adjustment parameters into the time coordination layer of the joint synchronization control model to establish the motion time synchronization relationship between multiple joints and generate time coordination constraints.

[0116] The role of the time coordination layer is to ensure that the movements of multiple joints remain synchronized in time, thus avoiding any uncoordinated movements.

[0117] Step S1441: Divide the position driving parameters into lower limb joint group parameters and upper limb joint group parameters according to the joint limb group. The lower limb joint group parameters include the position driving parameters of the hip joint, knee joint and ankle joint, and the upper limb joint group parameters include the position driving parameters of the shoulder joint, elbow joint and wrist joint.

[0118] Based on the humanoid robot's limb structure, all joints are divided into lower limb joint groups and upper limb joint groups. The lower limb joint group includes the hip, knee, and ankle joints. The position driving parameters of these joints are extracted to form the lower limb joint group parameters. The upper limb joint group includes the shoulder, elbow, and wrist joints. The position driving parameters of these joints are extracted to form the upper limb joint group parameters.

[0119] Step S1442: Perform time series alignment processing on the position driving parameters within each joint group, calculate the time difference characteristics of the target angle changes of adjacent joints, find the optimal time correspondence through the time series matching method, and determine the motion synchronization deviation characteristics between joints.

[0120] For the parameters of the lower limb joint group and the upper limb joint group, time series alignment processing was performed separately. Using the time series of the position-driving parameters of the first joint within the joint group as a benchmark, the time series of the position-driving parameters of other joints were compared with the benchmark series.

[0121] The time difference between adjacent joints when they reach the same target angle change is calculated to obtain the time difference feature. Dynamic time warping and other time series matching methods are used to match the time series of adjacent joints to find the optimal time correspondence that minimizes the time difference. Based on this correspondence, the motion synchronization deviation feature between joints is determined, which reflects the degree of deviation of joint motion in time.

[0122] Step S1443: Construct a joint synchronization error function based on the action synchronization deviation characteristics, compare the synchronization deviation characteristics with a preset synchronization allowable deviation range, and generate an error compensation signal when the synchronization deviation characteristics exceed the allowable range.

[0123] Based on the characteristics of motion synchronization deviation, a joint synchronization error function is constructed. This function takes the synchronization deviation characteristics as input and outputs an error value, which increases as the synchronization deviation increases. A preset allowable range of synchronization deviation is established, which is determined based on the joint synchronization accuracy requirements during normal humanoid robot movement.

[0124] Substituting the motion synchronization deviation characteristics into the joint synchronization error function yields the error value. If the error value is within the allowable synchronization deviation range, no error compensation signal is generated; if the error value exceeds the range, an error compensation signal is generated, and the magnitude of the compensation signal is proportional to the excess error value.

[0125] Step S1444: Input the error compensation signal into the time coordination controller, calculate the time compensation amount of the position drive parameters, and achieve motion synchronization within the joint group by adjusting the start time of the joint motion. The time compensation amount is determined according to the magnitude of the synchronization deviation characteristics.

[0126] After receiving the error compensation signal, the time coordination controller calculates the time compensation amount of the position drive parameters according to the preset compensation algorithm. If the synchronization deviation characteristic is positive, it indicates that the joint movement is lagging, and the time compensation amount is positive, which is used to advance the start time of the joint movement; if the synchronization deviation characteristic is negative, it indicates that the joint movement is ahead, and the time compensation amount is negative, which is used to delay the start time of the joint movement.

[0127] The time compensation is applied to the corresponding joint position drive parameters to adjust the start time of the movement, so that the movements of each joint in the joint group are synchronized in time.

[0128] Step S1445: Synchronize the force feedback adjustment parameters, calculate the rate of change characteristics of the force adjustment parameters within the joint group, and generate force synchronization constraint characteristics.

[0129] For each joint group, the rate of change of the force feedback adjustment parameter within the group at each time step is calculated, which is the ratio of the difference in the force feedback adjustment parameter between adjacent time steps to the time interval, thus obtaining the rate of change characteristic. Based on the rate of change characteristic, the synchronous change range of the force adjustment parameter is set, generating a force synchronization constraint characteristic. This force synchronization constraint characteristic requires that the rate of change of the force adjustment parameter within the joint group remain consistent within the same time step or within an allowable range of difference.

[0130] Step S1446: Combine the position driving parameters and force synchronization constraint features to form a multi-joint collaborative control rule containing time synchronization relationship, which serves as a time coordination constraint condition. The time coordination constraint condition is used to limit the action sequence and synchronization requirements of each joint at different time steps.

[0131] The position drive parameters, adjusted by time compensation, are combined with force synchronization constraints according to joint groups and time steps to form multi-joint cooperative control rules. These rules specify the sequence of actions of each joint at different time steps, such as which joint moves first and which moves later, as well as the synchronization requirements during actions, such as starting simultaneously, ending simultaneously, or maintaining a specific time difference. These rules together constitute time coordination constraints, which are used to regulate the time relationship of actions of each joint.

[0132] Step S145: Based on the time coordination constraints, perform time calibration on the position drive parameters and force feedback adjustment parameters to keep the control parameters of each joint synchronized on the time axis, and generate calibrated joint control parameters.

[0133] Based on the action sequence and synchronization requirements specified in the time coordination constraints, the position drive parameters and force feedback adjustment parameters are time-calibrated. For joints that need to move simultaneously, the timestamps of their control parameters are adjusted to ensure that their start and end times on the time axis are consistent; for joints that move in a sequential manner, the timestamps of the control parameters are adjusted according to the specified time intervals to ensure the correct action sequence.

[0134] After time calibration, the position drive parameters and force feedback adjustment parameters of each joint can remain synchronized on the time axis or be executed in a prescribed order, generating calibrated joint control parameters.

[0135] Step S146: Convert the calibrated joint control parameters into a drive signal format recognizable by the actuator, and generate a drive control instruction set containing the position drive signal and force feedback adjustment parameters of each joint. The drive control instruction set is arranged in the order of joint number and time step. The drive control instruction of each joint includes a time identifier, a position drive signal and a corresponding force feedback adjustment parameter.

[0136] The calibrated joint control parameters are converted into a signal format that the actuator can recognize in order to generate the final drive control instruction set.

[0137] Step S1461: Input the calibrated joint position drive parameters into the actuator drive model, and convert the angle parameters into the position drive signal of the actuator through a signal conversion method. The features of the position drive signal correspond to the joint target angle and motion speed.

[0138] The actuator drive model stores the conversion relationship between joint angle parameters and position drive signals, which is determined based on the actuator's working principle. When the joint rotation angle and rotation direction parameters are input into the actuator drive model, the model uses signal conversion methods such as D / A conversion to convert the angle parameters into corresponding analog voltage or pulse signals, which serve as the position drive signals. The amplitude or frequency of these position drive signals corresponds to the target joint angle and movement speed; a larger amplitude or frequency corresponds to a larger target angle or a faster movement speed.

[0139] Step S1462: Perform anti-interference encoding processing on the position drive signal to generate an encoded signal with anti-transmission interference capability.

[0140] Error control coding methods are employed to perform anti-interference coding processing on the position drive signal, such as adding a check bit. The position drive signal is divided into data blocks according to a set rule, a check value is calculated for each data block, and the check value is appended to the end of the data block to form the coded signal. During transmission, if the signal is interfered with and an error occurs, the receiving end can detect the error through the check value and correct it, thereby improving the reliability of signal transmission.

[0141] Step S1463: Convert the force feedback adjustment parameter into a force control signal for the actuator. Convert the adjustment parameter into an analog control signal recognizable by the actuator through a parameter signal conversion method. The characteristics of the analog control signal affect the magnitude of the stress feedback adjustment parameter.

[0142] Force feedback adjustment parameters are digital quantities and need to be converted into analog control signals, such as current or voltage signals, that the actuator can recognize through parameter signal conversion methods. During the conversion, a corresponding analog signal value is generated according to a preset proportional relationship based on the magnitude of the force feedback adjustment parameter. The magnitude of the current or voltage in the analog control signal corresponds to the magnitude of the force feedback adjustment parameter; the larger the current or voltage, the greater the impact on the stress feedback adjustment parameter.

[0143] Step S1464: After adding data verification features to the drive signals of each joint, the position drive signals, force control signals and verification features are combined according to a preset instruction format to generate joint drive instructions. The format of the joint drive instructions includes an instruction header, joint identifier, position signal segment, force control signal segment, verification segment and instruction tail.

[0144] Add a data verification feature to the position drive signal and force control signal of each joint. This data verification feature can be a hash value calculated by a hash algorithm. According to the preset instruction format, the instruction header (used to identify the start of the instruction), joint identifier (used to distinguish different joints), position signal segment (containing the encoded position drive signal), force control signal segment (containing the force control signal), verification segment (containing the data verification feature), and instruction tail (used to identify the end of the instruction) are combined in sequence to form the joint drive instruction.

[0145] Step S1465: Arrange the drive instructions of all joints in order of time step to generate a drive control instruction set. The drive control instruction for each time step contains the synchronization control signals of all joints of the humanoid machine.

[0146] The drive commands for all joints corresponding to each time step are collected and arranged in chronological order to form a drive control command set. Within this command set, each time step's command contains synchronization control signals for all joints of the humanoid robot at that time step, ensuring that each joint can execute the corresponding action according to the command within the same time step.

[0147] Step S150: Transmit the drive control instruction set to the distributed execution unit of the humanoid machine, execute the drive control instruction set and collect joint state feedback data during the movement process, and use the joint state feedback data as input data for the next round of feature association processing.

[0148] After the drive control instruction set is generated, it needs to be transmitted to the distributed execution unit for execution and to collect feedback data during the motion process in order to achieve closed-loop control.

[0149] For example, in step S151: the drive control instruction set is grouped according to the control range of the distributed execution unit. Each distributed execution unit is responsible for executing the control instructions of a specific joint, generating a joint control instruction package. The joint control instruction package includes a joint identifier, a time identifier, a position drive signal, and a force feedback adjustment parameter.

[0150] In humanoid robots, each of the distributed execution units is responsible for controlling a specific joint. Therefore, the drive control instruction set needs to be grouped according to the control range of the execution unit. For example, the execution unit responsible for the lower limb joints corresponds to the drive instructions for the lower limb joints, and the execution unit responsible for the upper limb joints corresponds to the drive instructions for the upper limb joints.

[0151] The joint identifier, time identifier, position drive signal, and force feedback adjustment parameters from each set of instructions are packaged to generate a joint control instruction package. Each instruction package corresponds to a distributed execution unit, ensuring that each execution unit can accurately obtain the joint control information it is responsible for. For example, in the joint control instruction package corresponding to the distributed execution unit responsible for the left hip joint, the joint identifier is "left hip joint," the time identifier is the specific time point of the instruction execution, the position drive signal corresponds to the angle and movement speed that the left hip joint needs to achieve, and the force feedback adjustment parameters are used to adjust according to the actual force during movement.

[0152] Step S152: The joint control command packet is transmitted to the corresponding distributed execution unit through a real-time communication link, so that the distributed execution unit can receive the joint control command packet, parse the command, extract the position drive signal and force feedback adjustment parameter, convert the position drive signal into the drive signal of the joint actuator, and control the force feedback adjustment parameter to realize force closed-loop regulation control.

[0153] A time-sensitive network-based real-time communication link is used to transmit joint control command packets. This real-time communication link ensures low latency and high reliability of command transmission, meeting the real-time requirements of humanoid robot motion control. During transmission, a sequence number and checksum are added to each joint control command packet for the receiving end to verify the integrity and order of the command packets.

[0154] After receiving the joint control command packet, the distributed execution unit first verifies the packet by checking the checksum to ensure the data is complete and error-free. If an error is found, it requests a retransmission. If the verification passes, the command packet is parsed to extract the position drive signal and force feedback adjustment parameters.

[0155] The position drive signal is converted into an electrical signal that the joint actuator can recognize. The amplitude and frequency of this electrical signal correspond to the angle and speed at which the joint needs to rotate. Simultaneously, the threshold and adjustment coefficient for force closed-loop control are set according to force feedback adjustment parameters. During joint movement, the force signal acting on the joint is collected in real time and compared with the preset force parameters. When the deviation exceeds the threshold, the output force of the actuator is adjusted according to the adjustment coefficient, achieving force closed-loop regulation control and ensuring the stability and safety of joint movement.

[0156] Step S153: During motion execution, the distributed execution unit uses the sensing device at the joint to collect the current joint angle characteristics, force characteristics and temperature characteristics in real time to generate raw state sensing data. The acquisition time of the raw state sensing data is consistent with the time step of the drive control command.

[0157] The distributed actuator collects data in real time through angle sensors, force sensors, and temperature sensors installed at the joints. The angle sensors collect the current actual angle of the joint at the same frequency as the time step of the drive control command, forming an angle feature. This angle feature is a multi-dimensional vector containing angle values ​​at different times, used to reflect the motion trajectory of the joint.

[0158] Force sensors collect the forces acting on the joint during movement, forming force characteristics, which are also multi-dimensional vectors containing force values ​​at different times, reflecting the load changes borne by the joint. Temperature sensors collect the temperature of the joint actuator, generating temperature characteristics to monitor the actuator's operating status and prevent equipment damage due to excessive temperature.

[0159] The collected angle, force, and temperature features are combined in chronological order to form raw state sensing data. The raw state sensing data at each time point corresponds to the time step of the corresponding drive control command, which facilitates subsequent state analysis and feedback.

[0160] Step S154: Clean the original state sensing data, package the cleaned state sensing data according to joint identifier and time identifier to generate a joint state feedback data packet, and transmit it to the central control module through the data backhaul link as input data for the next round of feature association processing. The joint state feedback data packet contains execution state feedback information corresponding to the drive control instruction set.

[0161] The raw state sensing data is cleaned up by removing outliers that are clearly outside the reasonable range, such as angle values ​​exceeding the maximum range of motion of the joint or negative force values ​​(which would not occur under normal movement). These outliers are replaced with the average value of the previous and next time points.

[0162] Then, the data is smoothed by using a moving average filter to eliminate high-frequency noise and make the data curve smoother. For example, for each data point in the angular feature, the average value of that point and its several adjacent points is used to reduce the impact of sensor noise on the data.

[0163] The purified state sensor data is categorized and organized according to joint and time identifiers. Angle, force, and temperature characteristics of the same joint and at the same time step are packaged together to generate joint state feedback data packages. Each data package includes the data acquisition time, joint identifier, and data verification information to ensure data traceability and accuracy.

[0164] The joint status feedback data packets are transmitted to the central control module via a data feedback link. The feedback link uses the same real-time communication mechanism as the command transmission to ensure the timeliness of the feedback data. The central control module receives and stores these feedback data packets as input data for feature association processing in the next step S110, forming a closed-loop control. This allows the humanoid robot to continuously adjust its control strategy according to the actual motion state, improving the adaptability and accuracy of its motion.

[0165] The method also includes a training step for the motion sequence prediction model, as detailed below:

[0166] Step S210: Collect historical data of the humanoid robot under different environments and motion states, including historical joint sensing data, historical environmental interaction data and corresponding historical motion control sequences, and preprocess and label the historical data.

[0167] A large amount of historical data was collected through motion experiments of humanoid robots in various scenarios. Different environments included flat ground, slopes, and ground with obstacles; different motion states included walking, turning, and climbing stairs. Historical joint sensor data included angle sequences and force signals of each joint; historical environmental interaction data included depth image stream data and plantar contact pressure data; and historical motion control sequences were the actual motion control commands corresponding to these data.

[0168] The collected historical data undergoes preprocessing, including data cleaning, missing value imputation, and data standardization. Data cleaning removes obvious errors and noise; missing value imputation uses interpolation to estimate missing values ​​based on data from different time points; data standardization transforms data of different magnitudes to the same numerical range, for example, transforming angle data to between 0 and 1, which facilitates feature processing during model training.

[0169] The preprocessed historical data is labeled with information such as movement type (e.g., walking, turning), gait cycle phase (e.g., support phase, swing phase), and environment type (e.g., flat ground, slope). This labeling information is used to guide the model training process.

[0170] Step S220: Construct the network structure of the motion sequence prediction model, which includes a spatiotemporal feature encoding module, a graph structure modeling layer, and a temporal prediction layer, and set the initial values ​​of the parameters for each layer.

[0171] The spatiotemporal feature encoding module employs a structure combining convolutional neural networks and an attention mechanism. Convolutional layers extract local correlations of features, while the attention mechanism highlights key features. The graph structure modeling layer uses a graph convolutional network to construct a graph structure based on the joint connections of a humanoid machine, where nodes represent joints and edges represent connections between joints. The temporal prediction layer uses a long short-term memory network to capture the dependencies between time series sequences.

[0172] The initial values ​​of the parameters for each layer are set. The weight parameters of the convolutional layer are initialized with a random normal distribution, and the bias parameters are initialized to 0. The weight parameters of the attention mechanism are initialized with a uniform distribution. The adjacency matrix of the graph convolutional network is initialized to 0 or 1 according to the joint connection relationship, and the weight parameters are randomly initialized. The weights and bias parameters of the long short-term memory network are both randomly initialized.

[0173] Step S230: Divide the preprocessed and labeled historical data into training set, validation set and test set, and input the training set into the motion sequence prediction model for training according to the preset batch size.

[0174] The historical data was divided into training, validation, and test sets in a 7:2:1 ratio. The training set was used to learn the model parameters, the validation set was used to tune the model's hyperparameters, and the test set was used to evaluate the model's final performance.

[0175] The training set data is divided into predefined batch sizes, for example, each batch contains 32 samples. The historical joint sensing data and historical environmental interaction data of each batch are input into the motion sequence prediction model. After processing by the spatiotemporal feature encoding module, graph structure modeling layer, and temporal prediction layer, the predicted motion control sequence is output.

[0176] Step S240: Calculate the loss value between the predicted motion control sequence and the labeled historical motion control sequence, adjust the parameters of each layer of the model through the backpropagation algorithm, and repeat the training process until the loss value converges to the preset threshold.

[0177] The mean squared error loss function is used to calculate the loss value between the predicted motion control sequence and the labeled historical motion control sequence. The mean squared error can reflect the overall deviation between the predicted value and the true value.

[0178] The backpropagation algorithm calculates the gradient of each layer's parameters with respect to the loss value, starting from the loss value. Then, the parameters are adjusted using gradient descent to reduce the loss value. For example, for the weight parameters of a convolutional layer, the parameter values ​​are updated according to the magnitude and direction of their gradients and a preset learning rate.

[0179] Repeat the training iterations, using different batches of data each time, until the loss value on the validation set converges to a preset threshold, i.e., the loss value no longer decreases significantly. At this point, the model training is complete, and a pre-trained motion sequence prediction model is obtained.

[0180] Step S250: Use the test set to evaluate the performance of the trained motion sequence prediction model. Evaluation metrics include prediction accuracy, root mean square error, etc. If the evaluation results meet the preset requirements, the model can be used for actual prediction; otherwise, adjust the model structure or parameters and retrain.

[0181] The test set data is input into the trained motion sequence prediction model to obtain the prediction results. The prediction accuracy is calculated, which is the proportion of the predicted motion control sequence that matches the real sequence; the root mean square error is calculated, which reflects the average deviation between the predicted value and the real value.

[0182] If the evaluation metrics meet the preset requirements, such as a prediction accuracy greater than 90% and a root mean square error less than the preset value, then the model performance meets the requirements and can be used for actual humanoid robot motion prediction. If the evaluation results do not meet the requirements, analyze the reasons, adjust the model's network structure (such as increasing the number of convolutional layers or adjusting the parameters of the attention mechanism) or training parameters (such as learning rate and batch size), and retrain and evaluate until the model performance meets the requirements.

[0183] The method also includes a step of constructing a joint synchronization control model, as follows:

[0184] Step S310: Collect the kinematic and dynamic parameters of each joint of the humanoid robot, including the joint range of motion, the rated power of the actuator, and the transmission ratio between joints, as the basic data for constructing the joint synchronization control model.

[0185] By consulting the design manuals and technical parameter tables of the humanoid robot, we collected the kinematic parameters of each joint, such as the maximum and minimum range of motion of the hip, knee, and ankle joints, as well as the range of rotational speeds of the joints.

[0186] Collect dynamic parameters, including the rated power, rated torque, and maximum load of each joint actuator, as well as the transmission ratio between joints, i.e., the ratio of motion transmission between active and passive joints. These parameters reflect the joint's motion capability and mechanical characteristics and are an important basis for constructing a joint synchronization control model.

[0187] Step S320: Based on the collected parameters, establish a joint kinematic model and a dynamic model. The kinematic model describes the relationship between the joint angle and the position of the end effector, and the dynamic model describes the relationship between the joint force and the motion acceleration.

[0188] Based on robot kinematics theory, a joint kinematic model is established. For lower limb joints, a coordinate system is established with the hip joint as the origin. The positional relationship between the knee and ankle joints relative to the hip joint is described by a homogeneous transformation matrix. The mathematical relationship between joint angles and foot position is established, that is, given the angle values ​​of each joint, the spatial position of the foot can be calculated through the kinematic model.

[0189] A dynamic model is established based on the Lagrange equations, considering factors such as the joint's mass, moment of inertia, gravitational potential energy, and kinetic energy, to derive the relationship between the force on the joint and its acceleration. The dynamic model reflects the magnitude of the force or torque that needs to be applied given the joint's acceleration.

[0190] Step S330: Based on the joint kinematic model and dynamic model, design a control strategy for the joint synchronization control model, including a position control strategy and a force control strategy. The position control strategy is used to achieve precise position tracking of the joint, and the force control strategy is used to achieve force adjustment of the joint.

[0191] The position control strategy employs a proportional-integral-derivative (PI-DE) control algorithm, calculating the control input based on the deviation between the target angle and the actual angle of the joint. The proportional term directly outputs the control input based on the magnitude of the deviation, the integral term eliminates steady-state deviation, and the derivative term suppresses overshoot, improving system stability. By adjusting the proportional gain, integral time, and derivative time, the joint can quickly and accurately track the target angle.

[0192] The force control strategy employs an impedance control algorithm, treating the joint as an impedance system with certain stiffness, damping, and inertia. Based on the deviation between the desired and actual force, the position or velocity of the joint is adjusted to ensure the force on the joint reaches the desired value. For example, when the actual force is greater than the desired force, the force is reduced by decreasing the joint's movement speed or adjusting its position; conversely, when the actual force is less than the desired force, the movement speed is increased or the position is adjusted to increase the force.

[0193] Step S340: Integrate the position control strategy and force control strategy into the joint synchronous control model. Set the input of the model to the joint target angle and desired force, and the output to the position drive signal and force feedback adjustment parameters. Verify the synchronous control effect of the model through simulation experiments.

[0194] Position control and force control strategies are programmed as algorithm modules and integrated into the software architecture of the joint synchronization control model. The model's input consists of the joint target angle and desired force extracted from the future motion control sequence. The position control module calculates the position drive signal, and the force control module calculates the force feedback adjustment parameters.

[0195] A simulation environment was built to simulate the movement of a humanoid robot in different walking scenarios. The output of the joint synchronization control model was used as the control commands for the joints in the simulation environment. The motion synchronization, position tracking accuracy, and force adjustment effect of each joint were observed. By adjusting the control parameters in the model, such as the proportional coefficient and impedance parameters, the model performance was optimized to ensure that each joint can move according to the expected synchronization relationship, thus meeting the motion control requirements of the humanoid robot.

[0196] Throughout the data acquisition and processing process, data such as the joint angles and forces acting on the humanoid robot are involved. This data may contain sensitive information such as the robot's motion state. To protect data privacy and prevent leakage, the following technical measures are adopted:

[0197] The collected raw data is encrypted using a symmetric encryption algorithm before storage and transmission. Only authorized modules and devices can decrypt the data. During data transmission, secure communication protocols, such as transport layer security protocols, are used to establish encrypted communication channels, preventing data from being stolen or tampered with during transmission.

[0198] The data is anonymized to remove information that could identify specific machines or scenarios, such as machine numbers or locations, making it impossible to associate the data with any particular entity. Simultaneously, data access permissions are set, allowing only authorized personnel and programs to access sensitive data, and data access is logged to facilitate traceability of data usage.

[0199] Figure 2Schematic diagrams are shown of exemplary hardware and software components of a model-predictive humanoid machine control system 100 that can implement the ideas of this application, according to some embodiments of this application. For example, a processor 120 may be used in the model-predictive humanoid machine control system 100 and to perform the functions described in this application.

[0200] The model-predictive humanoid machine control system 100 can be a general-purpose server or a special-purpose server, both of which can be used to implement the model-predictive humanoid machine control method of this application. Although only one server is shown in this application, for convenience, the functions described in this application can be implemented in a distributed manner on multiple similar platforms to balance the load.

[0201] For example, a model-based predictive humanoid machine control system 100 may include a network port 110 connected to a network, one or more processors 120 for executing program instructions, a communication bus 130, and various forms of storage media 140, such as a disk, ROM, or RAM, or any combination thereof. Exemplarily, the model-based predictive humanoid machine control system 100 may also include program instructions stored in ROM, RAM, or other types of non-transitory storage media, or any combination thereof. The methods of this application can be implemented according to these program instructions. The model-based predictive humanoid machine control system 100 also includes an I / O interface 150 between the computer and other input / output devices.

[0202] For ease of explanation, only one processor is described in the model-based predictive humanoid machine control system 100. However, it should be noted that the model-based predictive humanoid machine control system 100 of this application may also include multiple processors, and therefore the steps executed by one processor described in this application may also be executed jointly or individually by multiple processors. For example, if the processor of the model-based predictive humanoid machine control system 100 executes steps A and B, it should be understood that steps A and B may also be executed jointly by two different processors or individually by one processor. For example, the first processor executes step A, the second processor executes step B, or the first processor and the second processor jointly execute steps A and B.

[0203] Furthermore, this embodiment of the invention also provides a readable storage medium, wherein computer-executable instructions are preset in the readable storage medium, and when the processor executes the computer-executable instructions, the above-mentioned humanoid machine control method based on model prediction is implemented.

[0204] It should be noted that, in order to simplify the description of the present invention and thus help to understand one or more embodiments of the invention, multiple features may sometimes be grouped into one embodiment, drawing or description thereof in the foregoing description of the embodiments of the present invention.

Claims

1. A humanoid machine control method based on model prediction, characterized in that, The method includes: Acquire joint sensing data and environmental interaction data of a humanoid robot. The joint sensing data includes hip joint motion angle sequence and knee joint force signal. The environmental interaction data includes depth image stream data and plantar contact pressure data. The joint sensing data and the environmental interaction data are subjected to feature association processing to generate a dynamic state feature set of the humanoid machine. The dynamic state feature set includes joint coordination feature vectors and environmental constraint feature matrices. A pre-trained motion sequence prediction model is invoked to perform spatiotemporal joint prediction processing on the dynamic state feature set, and the output is a future motion control sequence containing gait cycle features. The future motion control sequence includes the angle change trajectory of each joint and the force adjustment parameters of the actuator. Based on the future motion control sequence, a joint synchronization control model is constructed, and a drive control instruction set containing time coordination constraints is generated. The drive control instruction set includes the position drive signal and force feedback adjustment parameters for each joint. The drive control instruction set is transmitted to the distributed execution unit of the humanoid machine, the drive control instruction set is executed and the joint state feedback data during the movement is collected, and the joint state feedback data is used as the input data for the next round of feature association processing; The step of performing feature association processing on the joint sensing data and the environmental interaction data to generate a dynamic state feature set for the humanoid machine includes: The time-domain fluctuation features of the hip joint motion angle sequence in the joint sensing data are extracted. The periodic change features of the angle sequence are captured by a sliding time window, and the main peak periodic features and fluctuation amplitude features of the angle fluctuation are extracted as dynamic features of the hip joint. Frequency band component analysis is performed on the force signal of the knee joint, decomposing the force signal into component signals of different frequency ranges, and extracting the energy proportion characteristics and waveform distortion characteristics of each component signal as knee joint load characteristics. The hip joint dynamic features and the knee joint load features are concatenated by feature dimensions to generate a joint coordination feature vector. The dimension of the joint coordination feature vector is equal to the sum of the dimensions of the hip joint dynamic features and the knee joint load features. The coupled representation of the lower limb joint motion state is achieved through feature concatenation. Spatial structure analysis is performed on the depth image stream data in the environmental interaction data. Scene depth information is generated through stereo matching. Spatial location distribution features of obstacles and ground slope change features are extracted to construct an environmental spatial feature matrix. Regional pressure features are extracted from the plantar contact pressure data. The plantar area is divided into forefoot, midfoot and heel regions. The pressure peak features and pressure center offset features of each region are extracted as plantar interaction features. The environmental spatial feature matrix and the foot interaction features are fused using tensor dimensions to generate an environmental constraint feature matrix. The row dimension of the environmental constraint feature matrix corresponds to the spatial location distribution, and the column dimension corresponds to the pressure interaction features. The joint coordination feature vector and the environmental constraint feature matrix are aligned and associated according to the time acquisition order. The correlation between joint motion features and environmental constraint features is established through a nonlinear mapping function to generate a dynamic state feature set containing the kinematic and environmental interaction relationship. Each feature element of the dynamic state feature set contains joint motion parameters and corresponding environmental constraint parameters. The pre-trained motion sequence prediction model is invoked to perform spatiotemporal joint prediction processing on the dynamic state feature set, outputting a future motion control sequence containing gait cycle features, including: The dynamic state feature set is input into the spatiotemporal feature encoding module of the motion sequence prediction model. The key spatial location features in the environmental constraint feature matrix are mined through the spatial attention mechanism, and the temporal dependencies in the joint coordination feature vector are captured through the temporal attention mechanism to generate a spatiotemporal correlation feature tensor. The spatial attention mechanism is implemented by calculating the influence weight of each spatial location in the environmental constraint feature matrix on the joint motion, and the temporal attention mechanism is implemented by calculating the correlation weight of features at different time points in the joint coordination feature vector. The spatiotemporal correlation feature tensor is input into the graph structure modeling layer of the motion sequence prediction model to construct a humanoid robot joint connection topology graph. The motion coupling relationship between joints is calculated through graph node feature propagation to generate graph structure motion features. The nodes of the joint connection topology graph represent each joint, and the edges represent the motion transmission relationship between joints. The graph structure motion features are input into the temporal prediction layer of the motion sequence prediction model, and joint motion state prediction values ​​for multiple future time steps are generated through recursive prediction. The joint motion state prediction values ​​include the angle prediction values ​​of each joint and the force prediction parameters of the actuator. The kinematic feasibility of the predicted joint motion state is verified to ensure that the predicted angle is within the joint range of motion and the predicted force parameter is within the rated load range of the actuator, thereby generating a preliminary motion sequence. The preliminary motion sequence is divided into gait cycles. Gait cycle characteristics are determined by analyzing the periodic changes in plantar contact pressure. The gait cycle characteristics include the continuous characteristics of the support phase and the continuous characteristics of the swing phase. The gait cycle characteristics are embedded into the preliminary motion sequence to generate a future motion control sequence containing the gait cycle characteristics. The time step of the future motion control sequence is consistent with the time division of the gait cycle characteristics. The joint synchronization control model is constructed based on the future motion control sequence, generating a drive control instruction set containing time coordination constraints, including: The trajectory of joint angle change in the future motion control sequence is analyzed, the target angle features and angle change rate features of each joint at each time step are extracted, and the joint kinematic constraint relationship is constructed. The joint kinematic constraint relationship reflects the correlation of angle changes of adjacent joints. Based on the joint kinematic constraints and the limb linkage structure characteristics of the humanoid machine, the position driving parameters of each joint are calculated through inverse kinematics analysis. The position driving parameters include joint rotation angle parameters and rotation direction parameters. The actuator force adjustment parameters in the future motion control sequence are analyzed, and combined with the current load state characteristics of the joint, the force feedback adjustment parameters of the actuator are calculated through the force feedback adjustment mechanism. The force feedback adjustment parameters are used to compensate for the influence of external load changes during the motion process. The position driving parameters and the force feedback adjustment parameters are input into the time coordination layer of the joint synchronization control model to establish the motion time synchronization relationship between multiple joints and generate time coordination constraints. Based on the aforementioned time coordination constraints, the position drive parameters and force feedback adjustment parameters are time-calibrated to keep the control parameters of each joint synchronized on the time axis, thereby generating calibrated joint control parameters. The calibrated joint control parameters are converted into a drive signal format recognizable by the actuator, generating a drive control instruction set containing the position drive signal and force feedback adjustment parameters for each joint. The drive control instruction set is arranged in the order of joint number and time step, and the drive control instruction for each joint includes a time identifier, a position drive signal, and a corresponding force feedback adjustment parameter.

2. The humanoid machine control method based on model prediction according to claim 1, characterized in that, The step of concatenating the dynamic features of the hip joint and the load features of the knee joint along their feature dimensions to generate a joint collaborative feature vector includes: Extract the main peak period feature and fluctuation amplitude feature from the dynamic features of the hip joint to form a hip joint feature sub-vector. The feature dimension of the hip joint feature sub-vector corresponds to the number of main peak period features and fluctuation amplitude features. Extract the energy proportion features and waveform distortion features of each component signal in the knee joint load features to form a knee joint feature subvector. The feature dimension of the knee joint feature subvector corresponds to the number of energy proportion features and waveform distortion features. The hip joint feature vector and the knee joint feature vector are subjected to feature standardization processing. The standardized hip joint feature vector and the knee joint feature vector are then concatenated in the order of feature dimensions to form a preliminary joint co-feature vector. Redundant features are removed from the preliminary joint coordination feature vector to generate the final joint coordination feature vector.

3. The humanoid machine control method based on model prediction according to claim 1, characterized in that, The step of inputting the dynamic state feature set into the spatiotemporal feature encoding module of the motion sequence prediction model, mining key spatial location features in the environmental constraint feature matrix through a spatial attention mechanism, and capturing temporal dependencies in the joint coordination feature vector through a temporal attention mechanism to generate a spatiotemporal correlation feature tensor includes: The joint coordination feature vector in the dynamic state feature set is augmented with time position features. By generating time identifier features of different periods and superimposing them with the joint coordination feature vector, time-enhanced joint features are obtained. Spatial location features are added to the environmental constraint feature matrix in the dynamic state feature set. By generating a spatial identifier feature matrix that reflects spatial coordinates and superimposing it with the environmental constraint feature matrix, spatial enhanced environmental features are obtained. The spatial identifier feature matrix is ​​realized by an eigenvalue matrix that reflects spatial positional relationships. The temporal augmentation joint features are used as query features, and the spatial augmentation environment features are used as key and value features. A spatial attention weight matrix is ​​generated by calculating the correlation between the query features and the key features. Based on the spatial attention weight matrix, the spatial augmented environment features are weighted and aggregated to generate a feature vector of key environmental locations. The environmental key location feature vectors and time-enhanced joint features are used to generate interactive features. The joint features and environmental features are fused through the correlation operation between feature elements to obtain spatiotemporal interactive features. The spatiotemporal interaction features are expanded by feature dimension extension to increase the feature expression dimension, generating a spatiotemporal correlation feature tensor that contains spatiotemporal relationships, so that the spatiotemporal correlation feature tensor contains both time dependency information and spatial location information.

4. The humanoid machine control method based on model prediction according to claim 1, characterized in that, The step of inputting the graph structure motion features into the temporal prediction layer of the motion sequence prediction model, and generating predicted joint motion states for multiple future time steps through recursive prediction, includes: The graph structure motion features are input into the feature input unit of the time-series prediction layer, and the key motion features that should be included in the prediction at the current moment are determined through the feature selection mechanism. The key motion features and the predicted state features of the previous moment are input into the temporal memory unit. The long-term motion trend features are retained through the state update mechanism, while the short-term motion change features are updated. The temporal memory unit achieves the collaborative memory of long-term and short-term features through the feature fusion method. The feature conversion unit processes the output features of the temporal memory unit to generate a predicted value of the joint motion state at the current moment. The predicted value of the joint motion state includes angle prediction features and force prediction parameters. The feature conversion unit realizes the conversion from features to predicted values ​​through a nonlinear mapping method. The predicted joint motion state at the current moment is used as the predicted state feature for the next moment and input into the temporal memory unit. The process of feature input, temporal memory update and feature transformation is repeated to recursively generate the predicted joint motion state for subsequent time steps. The number of predicted time steps is determined based on gait cycle characteristics. Recursive prediction stops when the generated joint motion state prediction values ​​cover the entire gait cycle, resulting in a sequence of joint motion state prediction values ​​for multiple future time steps. The number of time steps in the joint motion state prediction value sequence is consistent with the number of time divisions included in the gait cycle.

5. The humanoid machine control method based on model prediction according to claim 1, characterized in that, The step involves inputting the position driving parameters and the force feedback adjustment parameters into the time coordination layer of the joint synchronization control model to establish the motion time synchronization relationship between multiple joints and generate time coordination constraints, including: The position driving parameters are divided into lower limb joint parameters and upper limb joint parameters according to the joints and limbs. The lower limb joint parameters include the position driving parameters of the hip joint, knee joint and ankle joint, and the upper limb joint parameters include the position driving parameters of the shoulder joint, elbow joint and wrist joint. The position driving parameters within each joint group are aligned over time. The time difference characteristics of the target angle changes of adjacent joints are calculated. The optimal time correspondence is found through time series matching method, and the motion synchronization deviation characteristics between joints are determined. Based on the motion synchronization deviation characteristics, a joint synchronization error function is constructed. The synchronization deviation characteristics are compared with a preset synchronization allowable deviation range. When the synchronization deviation characteristics exceed the allowable range, an error compensation signal is generated. The error compensation signal is input into the time coordination controller to calculate the time compensation amount of the position drive parameters. The action synchronization within the joint group is achieved by adjusting the start time of the joint action. The time compensation amount is determined according to the magnitude of the synchronization deviation characteristics. The force feedback adjustment parameters are processed synchronously to calculate the rate of change characteristics of the force adjustment parameters within the joint group and generate force synchronization constraint characteristics. The position driving parameters and force synchronization constraint features are combined to form a multi-joint collaborative control rule that includes time synchronization relationship, which serves as a time coordination constraint condition. The time coordination constraint condition is used to limit the action sequence and synchronization requirements of each joint at different time steps.

6. The humanoid machine control method based on model prediction according to claim 1, characterized in that, The step of converting the calibrated joint control parameters into a drive signal format recognizable by the actuator, and generating a drive control instruction set containing drive signals for each joint position and force feedback adjustment parameters, includes: The calibrated joint position drive parameters are input into the actuator drive model, and the angle parameters are converted into the position drive signal of the actuator through a signal conversion method. The features of the position drive signal correspond to the joint target angle and the motion speed. The position drive signal is subjected to anti-interference coding processing to generate a coded signal with anti-transmission interference capability; The force feedback adjustment parameter is converted into a force control signal for the actuator. The adjustment parameter is then converted into an analog control signal that the actuator can recognize through a parameter signal conversion method. The characteristics of the analog control signal affect the magnitude of the stress feedback adjustment parameter. After adding data verification features to the drive signals of each joint, the position drive signals, force control signals and verification features are combined according to a preset instruction format to generate joint drive instructions. The format of the joint drive instructions includes an instruction header, joint identifier, position signal segment, force control signal segment, verification segment and instruction tail. The drive commands for all joints are arranged in order of time step to generate a drive control command set. The drive control command for each time step contains the synchronization control signals for all joints of the humanoid machine.

7. A humanoid machine control system based on model prediction, characterized in that, The device includes a processor and a memory, the memory being connected to the processor. The memory is used to store programs, instructions, or code, and the processor is used to execute the programs, instructions, or code in the memory to implement the model prediction-based humanoid machine control method according to any one of claims 1-6.

Images