Humanoid machine control method and system based on model prediction

Abstract

The invention provides a humanoid machine control method and system based on model prediction, and the method comprises the steps: firstly obtaining joint sensing data (hip joint movement angle sequences and knee joint stress signals) and environment interaction data (depth image flow data and sole contact pressure data) of a humanoid machine, and generating a dynamic state feature set through feature correlation processing; the feature matrix comprises a joint cooperation feature vector and an environment constraint feature matrix; calling a pre-training model to carry out spatio-temporal joint prediction on the feature set, and outputting a future motion control sequence containing gait cycle features; then, a joint synchronous control model is constructed on the basis, and a driving control instruction set containing time coordination constraint is generated; and finally, the driving control instruction set is transmitted to the distributed execution unit, and joint state feedback data is collected for next round of processing, so that the motion control performance of the humanoid machine in a complex environment is improved.

Description

Technical Field

[0001] The present invention relates to the field of robot control technology, and in particular to a humanoid machine control method and system based on model prediction. Background Art

[0002] In the field of humanoid robot control, achieving precise, stable motion control that adapts to complex environments has always been a key challenge. Traditional humanoid robot control methods often rely on preset motion patterns or simple feedback adjustment mechanisms. For example, some methods perform independent joint control based solely on joint sensor data, such as joint angles or force signals. However, these methods ignore the collaborative relationship between the humanoid robot's joints and its dynamic interaction with the surrounding environment. As a result, the robot's motion coordination is poor and it is prone to loss of balance when faced with complex terrain or external interference.

[0003] While other methods do consider environmental information, they often simply make local adjustments based on this data, lacking comprehensive analysis and prediction of the robot's overall dynamic state. For example, gait adjustment based solely on plantar contact pressure data, without incorporating joint motion and the spatial information of the environment reflected by depth images, makes it difficult for the robot to plan adaptive motion strategies in advance, significantly limiting its flexibility and adaptability in complex scenarios. Consequently, existing technologies struggle to meet the demands for efficient and stable motion control for humanoid robots in diverse and dynamically changing environments. Summary of the Invention

[0004] In view of the above-mentioned problems, in combination with the first aspect of the present invention, an embodiment of the present invention provides a humanoid robot control method based on model prediction, the method comprising:

[0005] Acquire joint sensor data and environmental interaction data of the humanoid machine, wherein the joint sensor data includes a hip joint motion angle sequence and a knee joint force signal, and the environmental interaction data includes depth image stream data and plantar contact pressure data;

[0006] Performing feature association processing on the joint sensing data and the environmental interaction data to generate a dynamic state feature set of the humanoid machine, wherein the dynamic state feature set includes a joint coordination feature vector and an environmental constraint feature matrix;

[0007] Calling a pre-trained motion sequence prediction model to perform spatiotemporal joint prediction processing on the dynamic state feature set, and outputting a future motion control sequence containing gait cycle characteristics, the future motion control sequence including angle change trajectories of each joint and force adjustment parameters of the actuator;

[0008] Building a joint synchronization control model based on the future motion control sequence, generating a drive control instruction set including a time coordination constraint, wherein the drive control instruction set includes a position drive signal and a force feedback adjustment parameter for each joint;

[0009] The drive control instruction set is transmitted to the distributed execution unit of the humanoid machine, the drive control instruction set is executed and the joint state feedback data during the movement is collected, and the joint state feedback data is used as input data for the next round of feature association processing.

[0010] On the other hand, an embodiment of the present invention also provides a humanoid machine control system based on model prediction, including a processor and a machine-readable storage medium, wherein the machine-readable storage medium is connected to the processor, the machine-readable storage medium is used to store programs, instructions or codes, and the processor is used to execute the programs, instructions or codes in the machine-readable storage medium to implement the above method.

[0011] Based on the above aspects, the humanoid robot's joint sensor data and environmental interaction data are comprehensively acquired and feature-correlated to generate a dynamic state feature set consisting of joint coordination feature vectors and environmental constraint feature matrices. This comprehensively and accurately characterizes the robot's motion state and its interaction with the environment. A pre-trained motion sequence prediction model is then used to perform spatiotemporal joint prediction on the dynamic state feature set, outputting future motion control sequences containing gait cycle characteristics. This allows for pre-planning of the robot's motion over a period of time, enabling it to better adapt to environmental changes and make adjustments in advance, thus enhancing the predictability and stability of its motion. A joint synchronization control model is constructed based on the future motion control sequence, generating a drive control instruction set with temporal coordination constraints. This ensures the synchronization and coordination of all joints during motion, avoiding motion errors and energy waste caused by asynchronous joint motion. Finally, the drive control instruction set is transmitted to a distributed execution unit, where joint state feedback data is collected as input for the next round. This allows for real-time adjustment of the control strategy, further improving control accuracy and adaptability, and enabling the robot to achieve efficient, stable, and flexible motion in various complex environments. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] Figure 1 4 is a schematic diagram of the execution flow of the humanoid machine control method based on model prediction provided by an embodiment of the present invention.

[0013] Figure 2 Schematic diagram of exemplary hardware and software components of a humanoid machine control system based on model prediction provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0014] The present invention will be described in detail below with reference to the accompanying drawings. Figure 1 FIG1 is a flow chart of a humanoid machine control method based on model prediction provided by an embodiment of the present invention. The humanoid machine control method based on model prediction is introduced in detail below.

[0015] This embodiment uses a humanoid robot walking indoors as a scenario to elaborate on the specific implementation process of the humanoid robot control method based on model prediction, so that those skilled in the art can clearly understand and smoothly implement the method.

[0016] Step S110: Acquire joint sensing data and environmental interaction data of the humanoid machine, wherein the joint sensing data includes hip joint motion angle sequence and knee joint force signal, and the environmental interaction data includes depth image stream data and plantar contact pressure data.

[0017] During walking, the angle sensor installed at the humanoid robot's hip joint operates continuously at a fixed sampling frequency, collecting hip joint angle data at regular intervals. These angle values ​​are arranged in chronological order to form a hip joint angle sequence. For example, during walking, the hip joint begins at an initial angle, gradually increases with each step, reaches its maximum angle, and then gradually decreases back to a near-initial angle. This hip joint angle sequence fully records this change.

[0018] The force sensor at the knee joint also collects data at a fixed frequency, sensing the tension and pressure applied to the knee joint during movement, generating a knee force signal. When the humanoid robot's foot touches the ground to support the body, the force on the knee joint increases. When the foot leaves the ground and swings, the force decreases. This knee force signal reflects these changes in real time.

[0019] A depth camera mounted on the front of the humanoid robot continuously captures the environment ahead. Each frame contains information about the distance between each object in the scene and the camera. These continuous images together form a depth image stream. This data can be used to identify environmental information such as whether there are obstacles ahead and whether the ground is flat.

[0020] The pressure sensor array on the sole of the foot continuously detects the pressure distribution when the sole of the foot contacts the ground during walking, generating sole contact pressure data. When the heel strikes the ground, the pressure in the heel area increases first; when the sole of the foot supports the body, the pressure in the forefoot area becomes the main component; when the foot leaves the ground, the pressure in each area gradually decreases to zero.

[0021] Step S120: performing feature association processing on the joint sensing data and the environmental interaction data to generate a dynamic state feature set of the humanoid machine, wherein the dynamic state feature set includes a joint coordination feature vector and an environmental constraint feature matrix.

[0022] After obtaining the joint sensor data and environmental interaction data, these raw data need to be processed to extract features that can reflect the machine state and environmental constraints, and associate them to form a dynamic state feature set.

[0023] Step S121: extracting the time domain fluctuation features of the hip joint motion angle sequence in the joint sensing data, capturing the periodic change features of the angle sequence through a sliding time window, and extracting the main peak period features and fluctuation amplitude features of the angle fluctuation as the dynamic features of the hip joint.

[0024] First, the length of the sliding time window is set. This length is determined based on the gait cycle of a normal humanoid robot, ensuring that the window can cover the angle changes within a complete gait cycle. The sliding step size is set to a value smaller than the window length to ensure continuity in feature extraction.

[0025] The sliding time window is moved sequentially across the hip joint motion angle sequence, and time-domain analysis is performed on the angle sequence within each window. The autocorrelation function of the angle sequence is calculated, and the time interval corresponding to the maximum value of the autocorrelation function is found. This time interval is the main peak period characteristic of the angle fluctuation, which reflects the main periodic pattern of hip joint motion.

[0026] At the same time, the difference between the maximum and minimum values ​​of the angle sequence within the window is calculated to obtain the fluctuation amplitude feature, which reflects the range of variation in the hip joint motion angle within the time period corresponding to the window. The main peak period feature and fluctuation amplitude feature extracted from each window are arranged in sequence to form the dynamic feature of the hip joint.

[0027] Step S122: performing frequency band component analysis on the knee joint force signal, decomposing the force signal into component signals in different frequency ranges, and extracting energy proportion characteristics and waveform distortion characteristics of each component signal as knee joint load characteristics.

[0028] A wavelet decomposition algorithm is used to process the knee joint force signal, decomposing it into multiple preset frequency ranges to obtain component signals in each frequency band. The above frequency ranges are determined based on the frequency of force changes that may occur in the knee joint during walking. For example, the low frequency range corresponds to slow load changes, and the high frequency range corresponds to rapid impact loads.

[0029] For each component signal, its energy value is calculated, and then the energy value is divided by the total energy value of all component signals to obtain the energy proportion feature of the component signal. The energy proportion feature reflects the importance of the force change within this frequency range in the overall force signal.

[0030] The deviation of the component signal waveform from a standard sine wave is calculated by calculating the waveform's kurtosis. A larger kurtosis value indicates more severe waveform distortion. This is used as a waveform distortion feature, which reflects the degree of irregularity in the knee joint's load. The energy contribution features and waveform distortion features of all component signals are combined to form a knee joint load signature.

[0031] Step S123: Perform feature dimension splicing on the hip joint dynamic feature and the knee joint load feature to generate a joint coordination feature vector. The dimension of the joint coordination feature vector is equal to the sum of the hip joint dynamic feature dimension and the knee joint load feature dimension. The coupled representation of the lower limb joint motion state is achieved through feature splicing.

[0032] The dynamic feature of the hip joint is a multidimensional vector. Assuming its dimension is a, each dimension corresponds to the main peak period feature or fluctuation amplitude feature of a different window; the load feature of the knee joint is also a multidimensional vector with dimension b. Each dimension corresponds to the energy proportion feature or waveform distortion feature of a different component signal.

[0033] The hip joint dynamic features and knee joint load features are concatenated by adding each dimension of the knee joint load feature to the dimensions of the hip joint dynamic features, forming a joint coordination feature vector with dimensions a + b. This concatenation preserves the information of both features, allowing the joint coordination feature vector to simultaneously reflect the motion state of the hip joint and the load on the knee joint, achieving a coupled representation of the motion state of the lower limb joints.

[0034] Step S124: performing spatial structure analysis on the depth image stream data in the environmental interaction data, generating scene depth information through stereo matching, extracting the spatial position distribution characteristics of obstacles and the ground slope change characteristics, and constructing an environmental spatial feature matrix.

[0035] Each frame in the depth image stream is preprocessed, including noise removal and image enhancement, to improve image quality. A stereo matching algorithm is used to calculate the 3D coordinates of each point in the scene by comparing the positional differences of corresponding points in the images captured by the left and right cameras at the same time, generating scene depth information.

[0036] Based on the scene depth information, a distance threshold is set and objects closer than this threshold are identified as obstacles. For each identified obstacle, its 3D coordinates are recorded. These coordinates are then divided into a set spatial grid. The number and average distance of obstacles within each grid are counted to form a spatial distribution feature of the obstacles.

[0037] At the same time, points in the ground area are selected from the scene depth information. The ground slope is calculated by fitting a plane equation. The ground is divided into multiple small areas, and a slope value is calculated for each area to form a ground slope change feature. The spatial position distribution features of obstacles and the ground slope change features are arranged in rows and columns to construct an environmental spatial feature matrix.

[0038] Step S125: performing regional pressure feature extraction on the sole contact pressure data, dividing the sole into the forefoot area, the midfoot area and the heel area, and extracting the pressure peak feature and the pressure center offset feature of each area as the sole interaction feature.

[0039] Based on the anatomical structure of the sole, the plantar sensor array is divided into the forefoot, midfoot, and heel regions, each containing multiple pressure sensors. For each region, the maximum value collected by the pressure sensors during a gait cycle is determined as the peak pressure feature for that region, reflecting the maximum pressure experienced by that region during walking.

[0040] The coordinates of the point where the resultant pressure acts within each region, known as the pressure center, are calculated. The offset of this pressure center relative to the region's geometric center, including both x- and y-direction offsets, is then calculated as the pressure center offset feature, which reflects the offset in the pressure distribution within the region. The pressure peak features and pressure center offset features of the three regions are sequentially combined to form a plantar interaction feature.

[0041] Step S126: Perform tensor dimension fusion on the environmental space feature matrix and the plantar interaction feature to generate an environmental constraint feature matrix, where the row dimension of the environmental constraint feature matrix corresponds to the spatial position distribution, and the column dimension corresponds to the pressure interaction feature.

[0042] The environmental spatial feature matrix is ​​a two-dimensional matrix with a dimension of m×n, where m corresponds to the number of rows and n corresponds to the number of columns. The plantar interaction feature is a one-dimensional vector with a dimension of p.

[0043] The plantar interaction features are expanded into a matrix with the same number of rows, m, as the environment spatial feature matrix. This expansion is done by repeating each element of the plantar interaction features m times, forming an m × p matrix. The environment spatial feature matrix and the expanded plantar interaction feature matrix are then concatenated column by column to generate an m × (n + p) environmental constraint feature matrix.

[0044] In this way, the row dimension of the environmental constraint feature matrix still corresponds to the original spatial position distribution, and the column dimension contains the original spatial features and the newly added pressure interaction features, realizing the fusion of environmental spatial information and plantar interaction information.

[0045] Step S127: Align and associate the joint collaboration feature vector and the environmental constraint feature matrix in the order of time acquisition, establish the correlation between the joint motion feature and the environmental constraint feature through a nonlinear mapping function, and generate a dynamic state feature set containing the interaction relationship between kinematics and the environment. Each feature element of the dynamic state feature set contains joint motion parameters and corresponding environmental constraint parameters.

[0046] First, the acquisition timestamps of the joint collaboration feature vector and the environmental constraint feature matrix are obtained, and the two are aligned in chronological order according to the timestamps to ensure that the joint collaboration features and environmental constraint features at the same time point correspond to each other.

[0047] A multilayer perceptron (MLP) is used as a nonlinear mapping function. The MLP consists of an input layer, a hidden layer, and an output layer. The aligned joint collaboration feature vector and the expanded vector of the environmental constraint feature matrix are used as inputs to the input layer. After nonlinear transformation in the hidden layer, the associated features are obtained from the output layer.

[0048] The above-mentioned associated features are arranged in chronological order to form a dynamic state feature set, in which each feature element contains the corresponding joint motion parameters (such as hip joint angle, knee joint force-related characteristics) and environmental constraint parameters (such as obstacle position, ground slope, and plantar pressure-related characteristics), which fully reflects the interactive relationship between machine motion and the environment.

[0049] Step S130: calling the pre-trained motion sequence prediction model to perform spatiotemporal joint prediction processing on the dynamic state feature set, and outputting a future motion control sequence containing gait cycle characteristics, wherein the future motion control sequence includes the angle change trajectory of each joint and the force adjustment parameters of the actuator.

[0050] The pre-trained motion sequence prediction model is trained on a large amount of humanoid robot walking data. It can predict future motion based on the input dynamic state feature set. After the dynamic state feature set is input into the model, the model performs a joint spatiotemporal prediction process, ultimately outputting a future motion control sequence containing gait cycle characteristics.

[0051] Step S131: Input the dynamic state feature set into the spatiotemporal feature encoding module of the motion sequence prediction model, mine the key spatial position features in the environmental constraint feature matrix through the spatial attention mechanism, capture the temporal dependency in the joint coordination feature vector through the temporal attention mechanism, and generate a spatiotemporal correlation feature tensor; the spatial attention mechanism is realized by calculating the influence weight of each spatial position on the joint motion in the environmental constraint feature matrix, and the temporal attention mechanism is realized by calculating the correlation weight of the features at different time points in the joint coordination feature vector.

[0052] The dynamic state feature set contains information in both time and space. The role of the spatiotemporal feature encoding module is to encode this information, highlight key information, and establish spatiotemporal associations.

[0053] Step S1311: adding time position features to the joint coordination feature vector in the dynamic state feature set, and generating time identification features of different periods and superimposing them with the joint coordination feature vector to obtain time-enhanced joint features.

[0054] Generate time identification features of various periods, such as sine wave and cosine wave features with periods of 1 / 4, 1 / 2, and 1 times the gait period. The dimensions of the above time identification features are the same as the dimensions of the joint coordination feature vector.

[0055] Each time signature feature is element-wise added to the joint collaboration feature vector to generate multiple intermediate features. These intermediate features are then concatenated dimensionally to form the time-enhanced joint feature. This incorporates temporal position information into the joint collaboration feature vector, helping the model capture temporal dependencies.

[0056] Step S1312: Add spatial position features to the environmental constraint feature matrix in the dynamic state feature set, and obtain spatial enhanced environmental features by generating a spatial identification feature matrix reflecting the spatial coordinate position and superimposing it with the environmental constraint feature matrix. The spatial identification feature matrix is ​​realized by an eigenvalue matrix reflecting the spatial position relationship.

[0057] Based on the spatial range corresponding to the environmental constraint feature matrix, a spatial identification feature matrix is ​​generated. Each element in the spatial identification feature matrix is ​​the feature value obtained by normalizing the coordinate value of the location at which it is located. For example, the x-coordinate and y-coordinate are normalized to the range of 0-1, which serves as the spatial identification feature of the location.

[0058] The dimension of the spatial identification feature matrix is ​​the same as that of the environmental constraint feature matrix. The two are added at the element level to obtain the spatial enhanced environmental features, so that the environmental constraint feature matrix contains clear spatial location information.

[0059] Step S1313: Using the temporally enhanced joint features as query features, and the spatially enhanced environment features as key features and value features, a spatial attention weight matrix is ​​generated by calculating the correlation between the query features and the key features.

[0060] The temporal enhanced joint features are used as query features Q, and the spatial enhanced environment features are flattened as key features K and value features V. The dot product between the query feature Q and the key feature K is calculated, and then the result is normalized by the softmax function to obtain the spatial attention weight matrix W.

[0061] Each element in the weight matrix represents the correlation between the query feature and the corresponding key feature, that is, the weight of the influence of the spatial position in the environment on the joint movement. The larger the weight, the more important the information of the spatial position is to the joint movement.

[0062] Step S1314: Perform weighted aggregation on the spatially enhanced environmental features according to the spatial attention weight matrix to generate an environmental key position feature vector.

[0063] Perform matrix multiplication on the spatial attention weight matrix W and the value feature V to obtain the weighted feature matrix, and then flatten the matrix into a vector, namely the environment key position feature vector.

[0064] Through the above weighted aggregation, the features of key positions in the environment that have a greater impact on joint movement are highlighted, and the features of secondary positions are suppressed, making the environmental features more focused.

[0065] Step S1315: Generate interactive features by performing interactive operations on the key environmental position feature vectors and the time-enhanced joint features, and achieve fusion of the joint features and the environmental features through correlation operations between feature elements to obtain spatiotemporal interactive features.

[0066] Perform element-wise multiplication on the environment key position feature vector and the time-enhanced joint feature to obtain the interaction feature matrix, and then sum the matrix in the column direction to obtain a vector as the spatiotemporal interaction feature.

[0067] The above operation method enables the joint features and environmental features to interact with each other. The generated spatiotemporal interaction features contain both the temporal information of the joints and the key spatial information of the environment, reflecting the interactive relationship between the two.

[0068] Step S1316: The spatiotemporal interaction features are subjected to feature dimension expansion processing to increase feature expression dimensions, generating a spatiotemporal correlation feature tensor containing a spatiotemporal correlation relationship, so that the spatiotemporal correlation feature tensor contains both time dependency information and spatial position information.

[0069] A convolutional layer is used to process the spatiotemporal interaction features. The size and number of convolution kernels are determined by the desired expanded dimensions. Through the convolution operation, the dimensions of the spatiotemporal interaction features are expanded to generate a three-dimensional spatiotemporal correlation feature tensor, where the first dimension corresponds to time and the second and third dimensions correspond to spatial position.

[0070] The spatiotemporal correlation feature tensor can clearly reflect the temporal dependency information and spatial location information, as well as the correlation between the two.

[0071] Step S132: Input the spatiotemporal correlation feature tensor into the graph structure modeling layer of the motion sequence prediction model, construct a humanoid machine joint connection topology graph, calculate the motion coupling relationship between the joints through graph node feature propagation, and generate graph structure motion features. The nodes of the joint connection topology graph represent each joint, and the edges represent the motion transfer relationship between the joints.

[0072] The graph structure modeling layer aims to capture the motion association between joints and convert the spatiotemporal association feature tensor into features that reflect the coupling relationship between joints.

[0073] First, a joint connection topology diagram is constructed based on the humanoid robot's skeletal structure. The nodes in the diagram correspond to the hip, knee, ankle, shoulder, elbow, wrist, and other joints. The edges between nodes are determined by the connection between the joints, for example, the hip connects to the knee, the knee connects to the ankle, and so on. The edge weights are initialized to 1, indicating the existence of a motion transfer relationship.

[0074] The features corresponding to each joint in the spatiotemporal correlation feature tensor are extracted and used as the initial features of each node in the topological graph. Node features are then propagated through graph convolution operations. The features of each node are updated based on the features of its neighboring nodes, taking edge weights into account during the update process.

[0075] Specifically, for each node, the features of its adjacent nodes are weighted and summed (the weights are the edge weights), which are then fused with the node's own features to obtain the updated node features. Through multiple graph convolution operations, node features are continuously propagated and updated. The resulting node features can reflect the kinematic coupling relationship between joints. These features are arranged in node order to form graph-structured motion features.

[0076] Step S133: Input the graph structure motion features into the timing prediction layer of the motion sequence prediction model, and generate joint motion state prediction values ​​for multiple future time steps through recursive prediction. The joint motion state prediction values ​​include the angle prediction values ​​of each joint and the force prediction parameters of the actuator.

[0077] The role of the time series prediction layer is to predict the joint motion state of multiple time steps in the future based on the motion characteristics of the graph structure. The use of recursive prediction can make the prediction results more consistent with the continuity of the time series.

[0078] Step S1331: input the graph structure motion features into the feature input unit of the time series prediction layer, and determine the key motion features that should be included in the prediction at the current moment through the feature selection mechanism.

[0079] The feature input unit includes a feature importance assessment module. This module evaluates the importance of each feature by calculating the correlation between each dimension of the graph structure motion feature and the historical motion state. The higher the correlation, the more critical the feature is for predicting future motion states.

[0080] An importance threshold is set and features with importance higher than the threshold are screened out as key motion features that should be included in the prediction at the current moment. The above features may include the angle change rate of the hip joint, the force change trend of the knee joint, etc.

[0081] Step S1332: Input the key motion features and the predicted state features of the previous moment into the temporal memory unit, retain the long-term motion trend features through the state update mechanism, and update the short-term motion change features at the same time. The temporal memory unit realizes the collaborative memory of long-term and short-term features through the feature fusion method.

[0082] The temporal memory unit consists of a long-term memory subunit and a short-term memory subunit. The key motion features and the predicted state features of the previous moment are input into the two subunits respectively.

[0083] The long-term memory subunit uses a low-pass filter to extract low-frequency components from the features. These low-frequency components reflect long-term motion trends, such as the stability of cadence during walking. The short-term memory subunit uses a high-pass filter to extract high-frequency components from the features, reflecting short-term motion changes, such as rapid changes in angle during walking.

[0084] Then, an attention mechanism is used to fuse the output features of the long-term and short-term memory subunits. The attention weight of each subunit's output feature is calculated. The weight is determined by the importance of the feature to the current prediction, with more important features having a larger weight. The output features of the two subunits are weighted and summed according to the attention weight to obtain the fused memory feature, achieving collaborative memory of long-term and short-term features.

[0085] Step S1333: The output features of the temporal memory unit are processed by the feature conversion unit to generate a joint motion state prediction value at the current moment. The joint motion state prediction value includes angle prediction features and force prediction parameters. The feature conversion unit realizes the conversion from features to prediction values ​​through a nonlinear mapping method.

[0086] The feature conversion unit consists of multiple fully connected layers and activation functions. The number of neurons in a fully connected layer is determined by the dimensions of the input features and the output prediction value. The fused memory features output by the temporal memory unit are input into the first fully connected layer. After linear transformation, they are nonlinearly processed using the ReLU activation function to obtain intermediate features.

[0087] The intermediate features are fed into the next fully connected layer, where they are again linearly transformed and activated, and this process is repeated until the last fully connected layer. The output dimension of the last fully connected layer matches the dimension of the joint motion state prediction value, with one portion of the output corresponding to the angle prediction features of each joint, and the other portion corresponding to the actuator force prediction parameters.

[0088] Through the above-mentioned multi-layer nonlinear mapping, the fused memory features are converted into specific joint motion state prediction values, ensuring that the prediction values ​​can reflect the long-term and short-term motion information contained in the memory features.

[0089] Step S1334: The predicted value of the joint motion state at the current moment is input into the temporal memory unit as the predicted state feature of the next moment, and the feature input, temporal memory update and feature conversion processes are repeated to recursively generate the predicted value of the joint motion state of the subsequent time step.

[0090] After obtaining the predicted joint motion state value at the current moment, it is used as the predicted state feature for the next moment and input into the temporal memory unit along with the key motion features for the next moment. Following the process from steps S1331 to S1333, feature selection, temporal memory update, and feature conversion are repeated to generate the predicted joint motion state value for the next moment.

[0091] This process is repeated and recursive to generate the predicted value of the joint motion state for each subsequent time step, so that the prediction sequence can continuously reflect the joint motion status for multiple time steps in the future.

[0092] Step S1335: Determine the number of prediction time steps based on the gait cycle characteristics, stop the recursive prediction when the generated joint motion state prediction value covers the complete gait cycle, and obtain a joint motion state prediction value sequence for multiple time steps in the future. The number of time steps of the joint motion state prediction value sequence is consistent with the number of time divisions contained in the gait cycle.

[0093] Based on the gait cycle characteristics of the humanoid robot, the number of time divisions included in a complete gait cycle is determined in advance. This number of time divisions is the total number of time steps required for prediction. During the recursive prediction process, the number of time steps of the generated joint motion state prediction values ​​is counted in real time.

[0094] When the number of counted time steps reaches the preset total, that is, the generated prediction value sequence covers the entire gait cycle, the recursive prediction stops. At this point, the resulting joint motion state prediction value sequence contains the same number of time steps as the time divisions of the gait cycle, and can fully reflect the joint motion state within the next gait cycle.

[0095] Step S134: Perform kinematic feasibility verification on the predicted value of the joint motion state so that the predicted angle is within the joint motion range limit and the predicted force parameter is within the actuator rated load range, thereby generating a preliminary motion sequence.

[0096] The predicted values ​​of joint motion states may exceed the actual motion capabilities of the humanoid machine, so kinematic feasibility verification is required to ensure the rationality of the prediction results.

[0097] For each joint's angle prediction feature, the pre-stored range of motion limit data for each joint is retrieved. This data includes the minimum and maximum range of motion for each joint. The predicted angle value for each joint at each time step is compared with the corresponding range of motion limit. If the predicted angle value is less than the minimum range of motion, it is adjusted to the minimum range of motion; if it is greater than the maximum range of motion, it is adjusted to the maximum range of motion; if it is within the range of motion, it remains unchanged.

[0098] For the actuator force prediction parameters, the rated load range data for each actuator is retrieved. This rated load range data includes the minimum and maximum output forces of each actuator. The force prediction parameters for each actuator at each time step are compared with the corresponding rated load range. If the force prediction parameter is less than the minimum output force, the minimum output force is adjusted; if it is greater than the maximum output force, the maximum output force is adjusted; if it is within the range, the force remains unchanged.

[0099] After the above adjustments, the predicted values ​​of the joint motion states of all time steps are arranged in sequence to generate a preliminary motion sequence.

[0100] Step S135: Divide the preliminary motion sequence into gait cycles, and determine the gait cycle characteristics by analyzing the periodic change characteristics of the plantar contact pressure. The gait cycle characteristics include the support phase duration characteristics and the swing phase duration characteristics. The gait cycle characteristics are embedded in the preliminary motion sequence to generate a future motion control sequence containing the gait cycle characteristics. The time step of the future motion control sequence is consistent with the time division of the gait cycle characteristics.

[0101] Predictive features related to plantar contact pressure are extracted from the preliminary motion sequence and their cyclical variations are analyzed. The starting and ending points of each gait cycle are determined based on the pattern of plantar contact pressure increasing, decreasing, then zero, and then increasing again.

[0102] Each gait cycle is divided into a stance phase and a swing phase based on whether the plantar contact pressure is greater than zero. The stance phase is when the plantar contact pressure is greater than zero, while the swing phase is when the plantar contact pressure is zero. The number of time steps in the stance phase is calculated as the stance phase duration feature, while the number of time steps in the swing phase is calculated as the swing phase duration feature. Together, these two features constitute the gait cycle characteristics.

[0103] The stance and swing phase duration features from the gait cycle characteristics are embedded into corresponding positions in the preliminary motion sequence in chronological order, so that each time step in the sequence can be associated with its gait phase. After embedding, a future motion control sequence containing the gait cycle characteristics is generated. The time steps of this future motion control sequence are consistent with the time divisions of the gait cycle characteristics, clearly reflecting the correspondence between motion and gait phases.

[0104] Step S140: constructing a joint synchronization control model based on the future motion control sequence, generating a drive control instruction set including time coordination constraints, wherein the drive control instruction set includes a position drive signal and force feedback adjustment parameters for each joint.

[0105] The future motion control sequence provides the motion goals and gait information of each joint, based on which a joint synchronization control model is constructed to generate a drive control instruction set that can achieve coordinated motion of each joint.

[0106] Step S141: Analyze the trajectory of each joint angle change in the future motion control sequence, extract the target angle characteristics and angle change rate characteristics of each joint at each time step, and construct a joint kinematic constraint relationship, which reflects the correlation between adjacent joint angle changes.

[0107] The trajectory of each joint angle change in the future motion control sequence is analyzed. For each joint, its angle value at each time step is extracted as the target angle feature. The angle change rate feature for each time step is obtained by calculating the difference between the target angle features of two adjacent time steps and dividing it by the time interval.

[0108] Based on the humanoid robot's limb structure, the connection relationship between adjacent joints is determined, such as the hip and knee joints, the knee and ankle joints, and so on. For each pair of adjacent joints, the relationship between their target angle characteristics and angle change rate characteristics at each time step is analyzed. By calculating the correlation coefficient between the two, a joint kinematic constraint relationship is constructed. This kinematic constraint relationship quantifies the degree of correlation between the angle changes of adjacent joints.

[0109] Step S142: Based on the joint kinematic constraint relationship and the limb link structure characteristics of the humanoid robot, the position drive parameters of each joint are calculated through inverse kinematic analysis, and the position drive parameters include joint rotation angle parameters and rotation direction parameters.

[0110] The humanoid robot's limb linkage structural characteristic data, including information such as the length and mass distribution of each limb link, is retrieved. Combined with the joint kinematic constraints, inverse kinematics analysis is used, using the target angle characteristics of each joint as known quantities to calculate the joint rotation angle parameters required to achieve that target angle.

[0111] The rotation direction parameters of the joints are determined based on the sign of the angle change rate. If the rate is positive, the rotation direction is forward; if the rate is negative, the rotation direction is reverse. The joint rotation angle parameters and rotation direction parameters are combined to form the position drive parameters of each joint.

[0112] Step S143: Analyze the actuator force adjustment parameters in the future motion control sequence, combine the current load state characteristics of the joint, and calculate the actuator force feedback adjustment parameters through the force feedback adjustment mechanism. The force feedback adjustment parameters are used to compensate for the influence of external load changes during the motion process.

[0113] The force adjustment parameters of the actuator are parsed from the future motion control sequence, and the current load state characteristics of each joint are collected at the same time. The load state characteristics include the actual load force and load change rate currently exerted on the joint.

[0114] The force feedback control mechanism calculates a force compensation based on the difference between the actuator's force adjustment parameter and the joint's current load characteristics. If the actual load force is greater than the force adjustment parameter, the force compensation is negative, reducing the actuator's output force. If the actual load force is less than the force adjustment parameter, the force compensation is positive, increasing the actuator's output force. The force adjustment parameter is added to the force compensation to obtain the actuator's force feedback control parameter, which compensates for the effects of external load changes on motion.

[0115] Step S144: inputting the position drive parameters and the force feedback adjustment parameters into the time coordination layer of the joint synchronization control model, establishing a time synchronization relationship of actions among multiple joints, and generating time coordination constraint conditions.

[0116] The role of the time coordination layer is to ensure that the movements of multiple joints are synchronized in time to avoid uncoordinated movements.

[0117] Step S1441: Divide the position drive parameters into lower limb joint group parameters and upper limb joint group parameters according to the joint limb groups. The lower limb joint group parameters include the position drive parameters of the hip joint, knee joint and ankle joint, and the upper limb joint group parameters include the position drive parameters of the shoulder joint, elbow joint and wrist joint.

[0118] Based on the humanoid robot's limb structure, all joints are divided into lower limb joint groups and upper limb joint groups. The lower limb joint group includes the hip, knee, and ankle joints, and the position drive parameters of these joints are extracted to form the lower limb joint group parameters. The upper limb joint group includes the shoulder, elbow, and wrist joints, and the position drive parameters of these joints are extracted to form the upper limb joint group parameters.

[0119] Step S1442: Perform time series alignment processing on the position drive parameters in each joint group, calculate the time difference characteristics of the target angle changes of adjacent joints, find the optimal time correspondence through the time series matching method, and determine the movement synchronization deviation characteristics between joints.

[0120] For the parameters of the lower limb joint group and the upper limb joint group, time series alignment is performed separately. The time series of the position drive parameters of the first joint in the joint group is used as the benchmark, and the time series of the position drive parameters of the other joints are compared with the benchmark sequence.

[0121] The time difference between adjacent joints when they reach the same target angle change is calculated to obtain a time difference feature. Using time series matching methods such as dynamic time warping, the time series of adjacent joints are matched to find the optimal time correspondence that minimizes the time difference. Based on this correspondence, the motion synchronization deviation feature between the joints is determined. This motion synchronization deviation feature reflects the degree of temporal deviation of the joint motion.

[0122] Step S1443: constructing a joint synchronization error function based on the motion synchronization deviation characteristics, comparing the synchronization deviation characteristics with a preset synchronization allowable deviation range, and generating an error compensation signal when the synchronization deviation characteristics exceed the allowable range.

[0123] Based on the motion synchronization deviation characteristics, a joint synchronization error function is constructed. This joint synchronization error function takes the synchronization deviation characteristics as input and outputs an error value, which increases with the synchronization deviation. A preset synchronization tolerance range is determined based on the joint synchronization accuracy requirements during normal humanoid robot motion.

[0124] The motion synchronization deviation characteristics are substituted into the joint synchronization error function to obtain the error value. If the error value is within the synchronization allowable deviation range, no error compensation signal is generated; if the error value exceeds the range, an error compensation signal is generated, and the size of the compensation signal is proportional to the exceeded error value.

[0125] Step S1444: Input the error compensation signal into the time coordination controller, calculate the time compensation amount of the position drive parameter, and achieve movement synchronization within the joint group by adjusting the starting time of the joint movement. The time compensation amount is determined according to the size of the synchronization deviation feature.

[0126] After receiving the error compensation signal, the time coordination controller calculates the time compensation for the position drive parameters based on a preset compensation algorithm. If the synchronization deviation characteristic is positive, it indicates that the joint movement is lagging, and the time compensation is positive, which is used to advance the start time of the joint movement. If the synchronization deviation characteristic is negative, it indicates that the joint movement is ahead, and the time compensation is negative, which is used to delay the start time of the joint movement.

[0127] The time compensation is applied to the corresponding joint position drive parameters to adjust the start time of its action so that the actions of each joint in the joint group remain synchronized in time.

[0128] Step S1445: Synchronize the force feedback adjustment parameters, calculate the change rate characteristics of the force adjustment parameters in the joint group, and generate force synchronization constraint characteristics.

[0129] For each joint group, the rate of change of the force feedback adjustment parameters within the group at each time step is calculated. This is the ratio of the difference between the force feedback adjustment parameters in adjacent time steps to the time interval, resulting in a rate of change feature. Based on this rate of change feature, the synchronous change range of the force adjustment parameters is set to generate a force synchronization constraint feature. This force synchronization constraint feature requires that the rate of change of the force adjustment parameters within the joint group remain consistent within the same time step or within the allowable difference range.

[0130] Step S1446: Combine the position drive parameters and force synchronization constraint features to form a multi-joint collaborative control rule containing a time synchronization relationship as a time coordination constraint condition. The time coordination constraint condition is used to limit the action sequence and synchronization requirements of each joint at different time steps.

[0131] The time-compensated position drive parameters and force synchronization constraint features are combined according to joint groups and time steps to form a multi-joint coordinated control rule. This multi-joint coordinated control rule specifies the order in which joints move at different time steps, such as which joint moves first and which moves later, as well as synchronization requirements during movement, such as simultaneous start and end, or maintaining a specific time difference. These rules together constitute the temporal coordination constraint, which regulates the temporal relationship between the movements of each joint.

[0132] Step S145: performing time calibration processing on the position drive parameters and the force feedback adjustment parameters based on the time coordination constraint condition, so that the control parameters of each joint are synchronized on the time axis, and generating calibrated joint control parameters.

[0133] Time-calibrate the position drive parameters and force feedback control parameters according to the motion sequence and synchronization requirements specified in the time coordination constraints. For joints that need to move simultaneously, adjust the timestamps of their control parameters to align their start and end times on the timeline. For joints that move sequentially, adjust the timestamps of their control parameters at the specified intervals to ensure the correct motion sequence.

[0134] After time calibration, the position drive parameters and force feedback adjustment parameters of each joint can be kept synchronized on the time axis or executed in a prescribed order to generate calibrated joint control parameters.

[0135] Step S146: Convert the calibrated joint control parameters into a drive signal format recognizable by the actuator, and generate a drive control instruction set containing each joint position drive signal and force feedback adjustment parameters. The drive control instruction set is arranged in the order of joint number and time step. The drive control instruction of each joint contains a time identifier, a position drive signal and a corresponding force feedback adjustment parameter.

[0136] The calibrated joint control parameters are converted into a signal format that the actuator can recognize to generate the final drive control instruction set.

[0137] Step S1461: Input the calibrated joint position drive parameters into the actuator drive model, and convert the angle parameters into the actuator position drive signal through a signal conversion method, wherein the characteristics of the position drive signal correspond to the joint target angle and movement speed.

[0138] The actuator drive model stores the conversion relationship between joint angle parameters and position drive signals, which is determined based on the actuator's operating principle. Joint rotation angle and direction parameters are input into the actuator drive model. The model uses signal conversion methods such as D / A conversion to convert the angle parameters into corresponding analog voltage or pulse signals, which serve as the position drive signals. Characteristics such as the amplitude and frequency of this position drive signal correspond to the target joint angle and movement speed. A larger amplitude or frequency corresponds to a larger target angle or faster movement speed.

[0139] Step S1462: performing anti-interference coding processing on the position drive signal to generate a coded signal with anti-transmission interference capability.

[0140] Error control coding methods are used to perform interference-resistant coding on the position drive signal, such as adding a check bit. The position drive signal is segmented into data blocks according to a set rule. A check value is calculated for each data block and appended to the data block to form the coded signal. During transmission, if the signal is interfered with or an error occurs, the receiver can detect the error using the check value and correct it, improving signal transmission reliability.

[0141] Step S1463: converting the force feedback adjustment parameter into a force control signal of the actuator, and converting the adjustment parameter into an analog control signal recognizable by the actuator through a parameter signal conversion method, wherein the characteristics of the analog control signal correspond to the size of the force feedback adjustment parameter.

[0142] The force feedback control parameter is a digital quantity that needs to be converted into an analog control signal, such as a current or voltage signal, that the actuator can recognize through a parameter signal conversion method. During conversion, the corresponding analog signal value is generated according to a preset proportional relationship based on the magnitude of the force feedback control parameter. The current or voltage of the analog control signal corresponds to the magnitude of the force feedback control parameter. The larger the current or voltage, the larger the corresponding force feedback control parameter.

[0143] Step S1464: After adding data verification features to the drive signal of each joint, the position drive signal, force control signal and verification features are combined according to the preset instruction format to generate a joint drive instruction. The format of the joint drive instruction includes an instruction header, a joint identifier, a position signal segment, a force control signal segment, a verification segment and an instruction tail.

[0144] A data verification feature is added to the position drive signal and force control signal of each joint. This data verification feature can be a hash value calculated using a hash algorithm. According to the preset instruction format, the instruction header (used to mark the beginning of the instruction), the joint identifier (used to distinguish different joints), the position signal segment (containing the encoded position drive signal), the force control signal segment (containing the force control signal), the verification segment (containing the data verification feature), and the instruction tail (used to mark the end of the instruction) are combined in sequence to form a joint drive instruction.

[0145] Step S1465: Arrange the driving instructions of all joints in order of time steps to generate a driving control instruction set. The driving control instruction of each time step contains synchronous control signals of all joints of the humanoid robot.

[0146] The drive instructions for all joints corresponding to each time step are collected and arranged in order of time steps to form a drive control instruction set. In this instruction set, the instructions for each time step contain the synchronous control signals for all joints of the humanoid robot at that time step, ensuring that each joint can execute the corresponding action according to the instructions within the same time step.

[0147] Step S150: transmitting the drive control instruction set to the distributed execution unit of the humanoid machine, executing the drive control instruction set and collecting joint state feedback data during the movement, and using the joint state feedback data as input data for the next round of feature association processing.

[0148] After the drive control instruction set is generated, it needs to be transmitted to the distributed execution unit for execution and feedback data during the motion process is collected to achieve closed-loop control.

[0149] For example, step S151: group the drive control instruction set according to the control range of the distributed execution unit, each distributed execution unit is responsible for executing the control instructions of a specific joint, and generates a joint control instruction package, which includes a joint identifier, a time identifier, a position drive signal and a force feedback adjustment parameter.

[0150] The distributed execution units of a humanoid robot are each responsible for controlling a specific joint. Therefore, the drive control instruction sets need to be grouped according to the execution unit's control range. For example, the execution units responsible for the lower limb joints correspond to the drive instructions for the lower limb joints, while the execution units responsible for the upper limb joints correspond to the drive instructions for the upper limb joints.

[0151] The joint identifier, time identifier, position drive signal, and force feedback adjustment parameters in each set of instructions are packaged to generate a joint control instruction package. Each instruction package corresponds to a distributed execution unit, ensuring that each execution unit can accurately obtain the control information of the joint it is responsible for. For example, in the joint control instruction package corresponding to the distributed execution unit responsible for the left hip joint, the joint identifier is "left hip joint", the time identifier is the specific time point when the instruction is executed, the position drive signal corresponds to the angle and movement speed that the left hip joint needs to achieve, and the force feedback adjustment parameters are used to adjust according to the actual force during movement.

[0152] Step S152: The joint control instruction package is transmitted to the corresponding distributed execution unit through a real-time communication link, so that the distributed execution unit performs instruction parsing after receiving the joint control instruction package, extracts the position drive signal and force feedback adjustment parameters, converts the position drive signal into a drive signal of the joint actuator, and controls the force feedback adjustment parameters to realize force closed-loop adjustment control.

[0153] Joint control command packets are transmitted using a real-time communication link based on a time-sensitive network. This real-time communication link ensures low latency and high reliability, meeting the real-time requirements of humanoid robot motion control. During transmission, a transmission sequence number and checksum are added to each joint control command packet to verify the packet's integrity and sequence at the receiving end.

[0154] After receiving the joint control instruction packet, the distributed execution unit first verifies the packet, using a checksum to verify the data integrity and request retransmission if an error is found. After verification, the instruction packet is parsed to extract the position drive signal and force feedback adjustment parameters.

[0155] The position drive signal is converted into an electrical signal that the joint actuator can recognize. The amplitude and frequency of this electrical signal correspond to the angle and speed at which the joint needs to rotate. Simultaneously, the threshold and adjustment coefficient of the force closed-loop control are set based on the force feedback adjustment parameters. During joint movement, the joint force signal is collected in real time and compared with the preset force parameters. If the deviation exceeds the threshold, the actuator's output force is adjusted based on the adjustment coefficient, achieving force closed-loop adjustment control and ensuring the stability and safety of the joint movement.

[0156] Step S153: During the execution of the movement, the distributed execution unit is used to collect the current joint angle characteristics, force characteristics and temperature characteristics in real time through the sensor device at the joint to generate original state sensor data. The collection time point of the original state sensor data is consistent with the time step of the drive control instruction.

[0157] Distributed execution units collect data in real time from angle sensors, force sensors, and temperature sensors installed at the joints. The angle sensors collect the actual joint angle at a frequency consistent with the time step of the drive control instructions, generating an angle signature. This angle signature is a multidimensional vector containing angle values ​​at different times, reflecting the joint's motion trajectory.

[0158] Force sensors collect the forces acting on joints during movement, generating a force signature. This signature is also a multidimensional vector containing force values ​​at different times, reflecting the load changes experienced by the joint. Temperature sensors collect the temperature of the joint actuators, generating a temperature signature that monitors the actuator's operating status and prevents damage due to excessive temperatures.

[0159] These collected angle features, force features, and temperature features are combined in chronological order to form raw state sensing data. The raw state sensing data at each time point corresponds to the corresponding drive control instruction time step, facilitating subsequent state analysis and feedback.

[0160] Step S154: Purify the original state sensing data, package the purified state sensing data according to the joint identifier and time identifier, generate a joint state feedback data packet, and transmit it to the central control module through the data return link as input data for the next round of feature association processing. The joint state feedback data packet contains execution state feedback information corresponding to the drive control instruction set.

[0161] The original state sensor data is purified. First, outliers that are obviously beyond the reasonable range are removed, such as angle values ​​exceeding the maximum range of joint motion, negative force values ​​(which are impossible under normal movement conditions), etc. For these outliers, the average of the previous and next moments is used to replace them.

[0162] The data is then smoothed using a sliding average filter to remove high-frequency noise and create a smoother curve. For example, each data point in the angle feature is replaced with the average of its preceding and succeeding points, reducing the impact of sensor noise on the data.

[0163] The cleaned state sensor data is categorized and organized by joint ID and time stamp. The angle, force, and temperature characteristics of the same joint and time step are packaged together to generate a joint state feedback data packet. The data acquisition time, joint ID, and data verification information are added to each packet to ensure data traceability and accuracy.

[0164] Joint state feedback packets are transmitted to the central control module via a data backhaul link. This backhaul link utilizes the same real-time communication mechanism used for command transmission to ensure timely feedback. The central control module receives and stores these feedback packets, which serve as input for the next round of feature association processing in step S110. This forms a closed-loop control mechanism, enabling the humanoid robot to continuously adjust its control strategy based on its actual motion state, improving its adaptability and accuracy.

[0165] The method further includes a step of training a motion sequence prediction model, specifically as follows:

[0166] Step S210: Collect historical data of the humanoid robot in different environments and motion states, including historical joint sensor data, historical environment interaction data and corresponding historical motion control sequences, and pre-process and annotate the historical data.

[0167] Through motion testing of humanoid robots in various scenarios, a large amount of historical data is collected. Different environments include flat surfaces, slopes, and surfaces with obstacles; different motion states include walking, turning, and ascending and descending stairs. Historical joint sensor data includes angle sequences and force signals of each joint; historical environmental interaction data includes depth image streams and plantar contact pressure data; and historical motion control sequences are the actual motion control instructions corresponding to this data.

[0168] The collected historical data is preprocessed, including data cleaning, missing value filling, and data standardization. Data cleaning removes obvious erroneous data and noise; missing value filling uses interpolation to estimate missing values ​​based on data from previous and subsequent moments; and data standardization converts data of different magnitudes to the same numerical range, such as converting angle data to a range of 0-1, to facilitate feature processing during model training.

[0169] The preprocessed historical data is labeled with the movement type (such as walking, turning), gait cycle phase (such as stance phase, swing phase), environment type (such as flat ground, slope), etc. This labeled information is used to guide the model training process.

[0170] Step S220: constructing a network structure of a motion sequence prediction model, the network structure including a spatiotemporal feature encoding module, a graph structure modeling layer and a time series prediction layer, and setting initial values ​​of parameters of each layer.

[0171] The spatiotemporal feature encoding module utilizes a structure that combines a convolutional neural network and an attention mechanism. The convolutional layer extracts local feature correlations, while the attention mechanism highlights key features. The graph structure modeling layer uses a graph convolutional network to construct a graph structure based on the joint connections of the humanoid robot. Nodes in the graph represent joints, and edges represent connections between joints. The time series prediction layer uses a long short-term memory network to capture time series dependencies.

[0172] The initial values ​​of the parameters of each layer are set. The weight parameters of the convolutional layer are initialized with random normal distribution, and the bias parameters are initialized to 0; the weight parameters of the attention mechanism are initialized with uniform distribution; the adjacency matrix of the graph convolutional network is initialized to 0 or 1 according to the joint connection relationship, and the weight parameters are randomly initialized; the weight and bias parameters of the long short-term memory network are both randomly initialized.

[0173] Step S230: Divide the pre-processed and annotated historical data into a training set, a validation set, and a test set, and input the training set into the motion sequence prediction model for training according to a preset batch size.

[0174] The historical data is divided into training set, validation set, and test set in a ratio of 7:2:1. The training set is used to learn model parameters, the validation set is used to adjust the model's hyperparameters, and the test set is used to evaluate the model's final performance.

[0175] The training set data is divided into preset batch sizes, for example, each batch contains 32 samples. Each batch of historical joint sensor data and historical environmental interaction data is input into the motion sequence prediction model. After processing by the spatiotemporal feature encoding module, the graph structure modeling layer, and the time series prediction layer, the predicted motion control sequence is output.

[0176] Step S240: Calculate the loss value between the predicted motion control sequence and the labeled historical motion control sequence, adjust the parameters of each layer of the model through the back propagation algorithm, and repeat the training process until the loss value converges to a preset threshold.

[0177] The mean square error loss function is used to calculate the loss value between the predicted motion control sequence and the labeled historical motion control sequence. The mean square error can reflect the overall deviation between the predicted value and the true value.

[0178] Using the backpropagation algorithm, starting from the loss value, the gradient of each layer's parameters with respect to the loss value is calculated. Then, the parameters are adjusted using the gradient descent method to reduce the loss value. For example, for the weight parameters of the convolutional layer, the parameter values ​​are updated according to the size and direction of the gradient and the preset learning rate.

[0179] The training iterations are repeated, using different batches of data in each iteration, until the loss value on the validation set converges to the preset threshold, that is, the loss value no longer decreases significantly. At this time, the model training is completed and the pre-trained motion sequence prediction model is obtained.

[0180] Step S250: Use the test set to perform performance evaluation on the trained motion sequence prediction model. The evaluation indicators include prediction accuracy, root mean square error, etc. If the evaluation result meets the preset requirements, the model can be used for actual prediction; otherwise, adjust the model structure or parameters and retrain.

[0181] The test set data is input into the trained motion sequence prediction model to obtain the prediction results. The prediction accuracy of the prediction results is calculated, that is, the proportion of the predicted motion control sequence that matches the true sequence; the root mean square error is calculated to reflect the average deviation between the predicted value and the true value.

[0182] If the evaluation metrics meet the preset requirements, such as a prediction accuracy greater than 90% and a root mean square error less than a preset value, the model's performance meets the requirements and can be used for actual humanoid robot motion prediction. If the evaluation results do not meet the requirements, analyze the reasons and adjust the model's network structure (such as increasing the number of convolutional layers or adjusting the parameters of the attention mechanism) or training parameters (such as the learning rate and batch size), and retrain and reevaluate until the model performance meets the requirements.

[0183] The method further includes the step of constructing a joint synchronization control model, which is specifically as follows:

[0184] Step S310: Collect kinematic and dynamic parameters of each joint of the humanoid robot, including joint range of motion, actuator rated power, transmission ratio between joints, etc., as basic data for constructing a joint synchronization control model.

[0185] By referring to the design manual and technical parameter table of the humanoid machine, the kinematic parameters of each joint are collected, such as the maximum and minimum angles of motion of the hip joint, knee joint, ankle joint, etc., and the rotation speed range of the joint.

[0186] Collect dynamic parameters, including the rated power, rated torque, and maximum load of each joint actuator, as well as the transmission ratio between joints, that is, the motion transmission ratio between active joints and passive joints. The above parameters reflect the motion ability and mechanical characteristics of the joints and are an important basis for constructing a joint synchronization control model.

[0187] Step S320: Based on the collected parameters, a joint kinematic model and a dynamic model are established. The kinematic model describes the relationship between the joint angle and the end effector position, and the dynamic model describes the relationship between the joint force and the motion acceleration.

[0188] Based on robot kinematics theory, a joint kinematic model was established. For the lower limb joints, a coordinate system was established with the hip joint as the origin. A homogeneous transformation matrix was used to describe the positional relationship between the knee and ankle joints relative to the hip joint. A mathematical relationship was established between joint angles and foot position. That is, given the angle values ​​of each joint, the kinematic model could be used to calculate the spatial position of the foot.

[0189] A dynamic model based on the Lagrange equations takes into account factors such as joint mass, moment of inertia, gravitational potential energy, and kinetic energy, and derives the relationship between joint force and acceleration. The dynamic model reflects the amount of force or torque required for a given joint acceleration.

[0190] Step S330: Design a control strategy for the joint synchronization control model based on the joint kinematic model and the dynamic model, including a position control strategy and a force control strategy. The position control strategy is used to achieve precise position tracking of the joint, and the force control strategy is used to achieve force regulation of the joint.

[0191] The position control strategy utilizes a proportional-integral-derivative control algorithm, calculating the control variable based on the deviation between the joint's target angle and actual angle. The proportional component directly outputs the control variable based on the magnitude of the deviation. The integral component eliminates steady-state deviations, and the derivative component suppresses overshoot, improving system stability. By adjusting the proportional coefficient, integral time, and derivative time, the joint can quickly and accurately track the target angle.

[0192] The force control strategy uses an impedance control algorithm, treating the joint as an impedance system with a certain stiffness, damping, and inertia. Based on the deviation between the desired force and the actual force, the joint position or speed is adjusted to achieve the desired force. For example, when the actual force is greater than the desired force, the force is reduced by reducing the joint's movement speed or adjusting its position. When the actual force is less than the desired force, the movement speed is increased or the position is adjusted to increase the force.

[0193] Step S340: Integrate the position control strategy and force control strategy into the joint synchronization control model, set the model input as the joint target angle and expected force, and the output as the position drive signal and force feedback adjustment parameters, and verify the synchronization control effect of the model through simulation experiments.

[0194] The position and force control strategies are written as algorithm modules and integrated into the software architecture of the joint synchronization control model. The model inputs are the target joint angles and desired forces extracted from the future motion control sequence. The position control module calculates the position drive signal, while the force control module calculates the force feedback adjustment parameters.

[0195] A simulation environment was built to simulate the motion of a humanoid robot in different walking scenarios. The output of the joint synchronization control model was used as the control instructions for the joints in the simulation environment to observe the motion synchronization, position tracking accuracy, and force regulation of each joint. By adjusting the control parameters in the model, such as the scale factor and impedance parameters, the model performance was optimized to ensure that each joint moves in the expected synchronous relationship and meet the motion control requirements of the humanoid robot.

[0196] The entire data collection and processing process involves data such as the joint angles and forces of the humanoid robot. This data may contain sensitive information such as the machine's motion state. To protect data privacy and prevent leakage, the following technical measures are used:

[0197] The collected raw data is encrypted using a symmetric encryption algorithm before storage and transmission. Only authorized modules and devices can decrypt the data. During data transmission, secure communication protocols, such as Transport Layer Security (TLS), are used to establish encrypted communication channels to prevent data from being stolen or tampered with during transmission.

[0198] Anonymize the data, removing information that could identify specific machines or scenarios, such as machine numbers and specific locations, so that the data cannot be associated with specific entities. Additionally, set data access permissions so that only authorized personnel and programs can access sensitive data. Data access is logged to facilitate data usage tracing.

[0199] Figure 2A schematic diagram illustrates exemplary hardware and software components of a model-based prediction humanoid robot control system 100 that can implement the concepts of the present application, as provided in some embodiments of the present application. For example, a processor 120 can be used in the model-based prediction humanoid robot control system 100 to perform the functions of the present application.

[0200] The model-prediction-based humanoid robot control system 100 can be a general-purpose server or a special-purpose server, both of which can be used to implement the model-prediction-based humanoid robot control method of this application. Although only one server is shown in this application, for convenience, the functions described in this application can be implemented in a distributed manner on multiple similar platforms to balance the processing load.

[0201] For example, the model-prediction-based humanoid machine control system 100 may include a network port 110 connected to a network, one or more processors 120 for executing program instructions, a communication bus 130, and various forms of storage media 140, such as a disk, ROM, or RAM, or any combination thereof. For example, the model-prediction-based humanoid machine control system 100 may also include program instructions stored in ROM, RAM, or other types of non-transitory storage media, or any combination thereof. The method of the present application may be implemented based on these program instructions. The model-prediction-based humanoid machine control system 100 also includes an I / O interface 150 between the computer and other input and output devices.

[0202] For ease of explanation, only one processor is described in the model-prediction-based humanoid machine control system 100. However, it should be noted that the model-prediction-based humanoid machine control system 100 in this application may also include multiple processors, and therefore the steps performed by one processor described in this application may also be performed jointly or individually by multiple processors. For example, if the processor of the model-prediction-based humanoid machine control system 100 performs steps A and B, it should be understood that steps A and B may also be performed jointly by two different processors or individually in one processor. For example, the first processor performs step A and the second processor performs step B, or the first processor and the second processor perform steps A and B together.

[0203] In addition, an embodiment of the present invention further provides a readable storage medium, in which computer-executable instructions are preset. When a processor executes the computer-executable instructions, the above-mentioned humanoid machine control method based on model prediction is implemented.

[0204] It should be noted that in order to simplify the description of the present invention and thus help understand one or more embodiments of the invention, in the foregoing description of the embodiments of the present invention, multiple features are sometimes combined into one embodiment, figure or description thereof.

Claims

1. A humanoid robot control method based on model prediction, characterized in that: The method comprises: Acquire joint sensor data and environmental interaction data of the humanoid machine, wherein the joint sensor data includes a hip joint motion angle sequence and a knee joint force signal, and the environmental interaction data includes depth image stream data and plantar contact pressure data; Performing feature association processing on the joint sensing data and the environmental interaction data to generate a dynamic state feature set of the humanoid machine, wherein the dynamic state feature set includes a joint coordination feature vector and an environmental constraint feature matrix; Calling a pre-trained motion sequence prediction model to perform spatiotemporal joint prediction processing on the dynamic state feature set, and outputting a future motion control sequence containing gait cycle characteristics, the future motion control sequence including angle change trajectories of each joint and force adjustment parameters of the actuator; Building a joint synchronization control model based on the future motion control sequence, generating a drive control instruction set including a time coordination constraint, wherein the drive control instruction set includes a position drive signal and a force feedback adjustment parameter for each joint; The drive control instruction set is transmitted to the distributed execution unit of the humanoid machine, the drive control instruction set is executed and the joint state feedback data during the movement is collected, and the joint state feedback data is used as input data for the next round of feature association processing.

2. The humanoid robot control method based on model prediction according to claim 1, characterized in that: The performing feature association processing on the joint sensor data and the environment interaction data to generate a dynamic state feature set of the humanoid machine includes: Extracting time-domain fluctuation features of the hip joint motion angle sequence in the joint sensor data, capturing the periodic variation features of the angle sequence through a sliding time window, and extracting the main peak period features and fluctuation amplitude features of the angle fluctuation as dynamic features of the hip joint; Performing frequency band component analysis on the knee joint force signal, decomposing the force signal into component signals in different frequency ranges, and extracting energy proportion characteristics and waveform distortion characteristics of each component signal as knee joint load characteristics; The hip joint dynamic feature and the knee joint load feature are spliced ​​in feature dimensions to generate a joint coordination feature vector, where the dimension of the joint coordination feature vector is equal to the sum of the hip joint dynamic feature dimension and the knee joint load feature dimension, and the coupled representation of the lower limb joint motion state is achieved through feature splicing; Performing spatial structural analysis on the depth image stream data in the environmental interaction data, generating scene depth information through stereo matching, extracting the spatial position distribution characteristics of obstacles and the ground slope change characteristics, and constructing an environmental spatial feature matrix; performing regional pressure feature extraction on the plantar contact pressure data, dividing the plantar into a forefoot region, a midfoot region, and a heel region, and extracting pressure peak features and pressure center offset features of each region as plantar interaction features; Performing tensor dimension fusion on the environmental spatial feature matrix and the plantar interaction feature to generate an environmental constraint feature matrix, wherein the row dimension of the environmental constraint feature matrix corresponds to the spatial position distribution, and the column dimension corresponds to the pressure interaction feature; The joint coordination feature vector and the environmental constraint feature matrix are aligned and associated in the order of time acquisition, and the correlation between the joint motion features and the environmental constraint features is established through a nonlinear mapping function to generate a dynamic state feature set containing the interaction relationship between kinematics and the environment. Each feature element of the dynamic state feature set contains joint motion parameters and corresponding environmental constraint parameters.

3. The humanoid robot control method based on model prediction according to claim 2, characterized in that: The step of performing feature dimension splicing on the hip joint dynamic feature and the knee joint load feature to generate a joint coordination feature vector includes: Extracting the main peak period feature and the fluctuation amplitude feature from the hip joint dynamic feature to form a hip joint feature sub-vector, wherein the feature dimension of the hip joint feature sub-vector corresponds to the number of the main peak period feature and the fluctuation amplitude feature; Extracting energy proportion features and waveform distortion features of each component signal in the knee joint load feature to form a knee joint feature subvector, wherein the feature dimension of the knee joint feature subvector corresponds to the number of energy proportion features and waveform distortion features; performing feature normalization processing on the hip joint feature subvector and the knee joint feature subvector, and concatenating the normalized hip joint feature subvector and the knee joint feature subvector in the order of feature dimensions to form a preliminary joint collaboration feature vector; Redundant features are removed from the preliminary joint collaboration feature vector to generate a final joint collaboration feature vector.

4. The humanoid robot control method based on model prediction according to claim 1, characterized in that: The calling of the pre-trained motion sequence prediction model to perform spatiotemporal joint prediction processing on the dynamic state feature set, and outputting a future motion control sequence containing gait cycle features, includes: The dynamic state feature set is input into the spatiotemporal feature encoding module of the motion sequence prediction model, and the key spatial position features in the environmental constraint feature matrix are mined through the spatial attention mechanism, and the temporal dependency in the joint coordination feature vector is captured through the temporal attention mechanism to generate a spatiotemporal correlation feature tensor; the spatial attention mechanism is implemented by calculating the influence weight of each spatial position on the joint motion in the environmental constraint feature matrix, and the temporal attention mechanism is implemented by calculating the correlation weight of features at different time points in the joint coordination feature vector; Inputting the spatiotemporal correlation feature tensor into the graph structure modeling layer of the motion sequence prediction model, constructing a joint connection topology graph of the humanoid robot, calculating the motion coupling relationship between the joints through graph node feature propagation, and generating graph structure motion features, wherein the nodes of the joint connection topology graph represent each joint, and the edges represent the motion transmission relationship between the joints; Inputting the graph structure motion features into the time series prediction layer of the motion sequence prediction model, and generating joint motion state prediction values ​​for multiple future time steps through a recursive prediction method, wherein the joint motion state prediction values ​​include angle prediction values ​​of each joint and force prediction parameters of the actuator; Performing kinematic feasibility verification on the predicted values ​​of the joint motion state so that the predicted angle is within the joint motion range limit and the predicted force parameter is within the actuator rated load range, thereby generating a preliminary motion sequence; The preliminary motion sequence is divided into gait cycles, and the gait cycle characteristics are determined by analyzing the periodic change characteristics of the plantar contact pressure. The gait cycle characteristics include the support phase duration characteristics and the swing phase duration characteristics. The gait cycle characteristics are embedded in the preliminary motion sequence to generate a future motion control sequence containing the gait cycle characteristics. The time step of the future motion control sequence is consistent with the time division of the gait cycle characteristics.

5. The humanoid robot control method based on model prediction according to claim 4, characterized in that: The dynamic state feature set is input into the spatiotemporal feature encoding module of the motion sequence prediction model, the key spatial position features in the environmental constraint feature matrix are mined through the spatial attention mechanism, the temporal dependency in the joint coordination feature vector is captured through the temporal attention mechanism, and the spatiotemporal correlation feature tensor is generated, including: Adding time position features to the joint collaborative feature vector in the dynamic state feature set, and generating time identification features of different periods and superimposing them with the joint collaborative feature vector to obtain time enhanced joint features; Adding spatial position features to the environmental constraint feature matrix in the dynamic state feature set, generating a spatial identification feature matrix reflecting the spatial coordinate position and superimposing it with the environmental constraint feature matrix to obtain spatially enhanced environmental features, wherein the spatial identification feature matrix is ​​realized by an eigenvalue matrix reflecting the spatial position relationship; The temporal enhanced joint feature is used as a query feature, the spatial enhanced environment feature is used as a key feature and a value feature, and a spatial attention weight matrix is ​​generated by calculating the correlation between the query feature and the key feature; Performing weighted aggregation on the spatially enhanced environment features according to the spatial attention weight matrix to generate an environment key position feature vector; Generate interactive features by combining the key position feature vector of the environment with the time-enhanced joint feature, and fuse the joint feature with the environment feature through correlation operations between feature elements to obtain spatiotemporal interactive features; The spatiotemporal interaction features are processed by feature dimension expansion to increase feature expression dimensions, and a spatiotemporal correlation feature tensor containing spatiotemporal correlation relationships is generated, so that the spatiotemporal correlation feature tensor contains both time dependency information and spatial position information.

6. The humanoid robot control method based on model prediction according to claim 4, characterized in that: The step of inputting the graph structure motion features into the time series prediction layer of the motion sequence prediction model and generating joint motion state prediction values ​​for multiple future time steps by recursive prediction includes: Input the graph structure motion features into the feature input unit of the time series prediction layer, and determine the key motion features that should be included in the prediction at the current moment through a feature selection mechanism; The key motion features and the predicted state features of the previous moment are input into the temporal memory unit, and the long-term motion trend features are retained through the state update mechanism, while the short-term motion change features are updated. The temporal memory unit realizes the collaborative memory of long-term and short-term features through the feature fusion method; The output features of the temporal memory unit are processed by the feature conversion unit to generate a joint motion state prediction value at the current moment, wherein the joint motion state prediction value includes angle prediction features and force prediction parameters. The feature conversion unit realizes the conversion from features to prediction values ​​through a nonlinear mapping method; The predicted value of the joint motion state at the current moment is input into the temporal memory unit as the predicted state feature of the next moment, and the feature input, temporal memory update and feature conversion process are repeated to recursively generate the predicted value of the joint motion state of the subsequent time step; The number of prediction time steps is determined according to the gait cycle characteristics. When the generated joint motion state prediction value covers the complete gait cycle, the recursive prediction is stopped to obtain a sequence of joint motion state prediction values ​​for multiple time steps in the future. The number of time steps of the joint motion state prediction value sequence is consistent with the number of time divisions contained in the gait cycle.

7. The humanoid robot control method based on model prediction according to claim 1, characterized in that: The step of constructing a joint synchronization control model based on the future motion control sequence and generating a drive control instruction set including time coordination constraints includes: Analyzing the trajectory of each joint angle change in the future motion control sequence, extracting the target angle characteristics and angle change rate characteristics of each joint at each time step, and constructing a joint kinematic constraint relationship, wherein the joint kinematic constraint relationship reflects the correlation between adjacent joint angle changes; Based on the joint kinematic constraint relationship and the limb link structure characteristics of the humanoid robot, the position drive parameters of each joint are calculated through inverse kinematic analysis, and the position drive parameters include joint rotation angle parameters and rotation direction parameters; Analyzing the actuator force adjustment parameters in the future motion control sequence, and calculating the actuator force feedback adjustment parameters through a force feedback adjustment mechanism based on the current load state characteristics of the joint, wherein the force feedback adjustment parameters are used to compensate for the influence of external load changes during the motion process; Inputting the position drive parameters and the force feedback adjustment parameters into the time coordination layer of the joint synchronization control model, establishing the action time synchronization relationship between multiple joints, and generating time coordination constraint conditions; Performing time calibration processing on the position drive parameters and the force feedback adjustment parameters based on the time coordination constraint condition, so that the control parameters of each joint are synchronized on the time axis, and generating calibrated joint control parameters; The calibrated joint control parameters are converted into a drive signal format recognizable by the actuator, and a drive control instruction set containing each joint position drive signal and force feedback adjustment parameters is generated. The drive control instruction set is arranged in the order of joint number and time step, and the drive control instruction of each joint contains a time identifier, a position drive signal and the corresponding force feedback adjustment parameter.

8. The humanoid robot control method based on model prediction according to claim 7, characterized in that: The step of inputting the position drive parameters and the force feedback adjustment parameters into the time coordination layer of the joint synchronization control model, establishing a time synchronization relationship between multiple joints, and generating time coordination constraints includes: The position driving parameters are divided into lower limb joint group parameters and upper limb joint group parameters according to the joint limb group, the lower limb joint group parameters include the position driving parameters of the hip joint, knee joint and ankle joint, and the upper limb joint group parameters include the position driving parameters of the shoulder joint, elbow joint and wrist joint; Perform time series alignment on the position drive parameters within each joint group, calculate the time difference characteristics of the target angle changes of adjacent joints, find the optimal time correspondence through time series matching method, and determine the movement synchronization deviation characteristics between joints; Constructing a joint synchronization error function based on the motion synchronization deviation feature, comparing the synchronization deviation feature with a preset synchronization allowable deviation range, and generating an error compensation signal when the synchronization deviation feature exceeds the allowable range; The error compensation signal is input into a time coordination controller to calculate a time compensation amount of a position drive parameter, and the movement synchronization within the joint group is achieved by adjusting the start time of the joint movement, wherein the time compensation amount is determined according to the size of the synchronization deviation feature; Synchronously processing the force feedback adjustment parameters, calculating the change rate characteristics of the force adjustment parameters within the joint group, and generating force synchronization constraint characteristics; The position drive parameters and force synchronization constraint features are combined to form a multi-joint collaborative control rule containing a time synchronization relationship as a time coordination constraint condition. The time coordination constraint condition is used to limit the action sequence and synchronization requirements of each joint at different time steps.

9. The humanoid robot control method based on model prediction according to claim 7, characterized in that: The step of converting the calibrated joint control parameters into a drive signal format recognizable by the actuator to generate a drive control instruction set including a drive signal for each joint position and a force feedback adjustment parameter includes: The calibrated joint position drive parameters are input into the actuator drive model, and the angle parameters are converted into the actuator position drive signal through a signal conversion method, wherein the characteristics of the position drive signal correspond to the joint target angle and motion speed; Performing anti-interference coding processing on the position drive signal to generate a coding signal with anti-transmission interference capability; Converting the force feedback adjustment parameter into a force control signal of the actuator, and converting the adjustment parameter into an analog control signal recognizable by the actuator through a parameter signal conversion method, wherein the characteristics of the analog control signal correspond to the magnitude of the force feedback adjustment parameter; After adding data verification features to the drive signal of each joint, the position drive signal, force control signal and verification features are combined according to a preset instruction format to generate a joint drive instruction. The format of the joint drive instruction includes an instruction header, a joint identifier, a position signal segment, a force control signal segment, a verification segment and an instruction tail; The driving instructions of all joints are arranged in time step order to generate a driving control instruction set. The driving control instruction of each time step contains the synchronous control signals of all joints of the humanoid robot.

10. A humanoid machine control system based on model prediction, characterized in that: It includes a processor and a memory, the memory is connected to the processor, the memory is used to store programs, instructions or codes, and the processor is used to execute the programs, instructions or codes in the memory to implement the humanoid machine control method based on model prediction as described in any one of claims 1 to 9.

Images