Humanoid robot action recognition method based on self-supervised learning

Abstract

The application discloses a humanoid robot action recognition method based on self-supervised learning and belongs to the technical field of robot action recognition. The method comprises the following steps: acquiring a multi-modal time series data set collected by a joint sensor during the execution of an action of a humanoid robot, wherein the data set comprises a joint angle sequence, inertial measurement unit data and a plantar pressure distribution sequence; performing action intrinsic structure decomposition processing on the multi-modal time series data set to generate action skeleton graph structure features comprising a joint topological connection matrix and a time series motion trajectory tensor; calling a pre-trained self-supervised encoder backbone network to perform comparative learning pre-training processing on the action skeleton graph structure features, and extracting action representation embedding features; and inputting the action representation embedding features into a downstream action classifier to generate an action recognition result of the humanoid robot.

Description

Technical Field

[0001] This invention relates to the field of robot motion recognition technology, specifically a method for humanoid robot motion recognition based on self-supervised learning. Background Technology

[0002] Humanoid robot motion recognition requires accurately extracting the robot's motion intentions and execution states from high-dimensional sensor data streams. Existing solutions generally employ a supervised learning paradigm, relying on a large number of manually labeled motion sequence samples to train deep neural networks. These methods classify data by extracting single-modal features such as joint angles and accelerations, or by simply concatenating multimodal data before feeding it into a temporal model. These existing solutions have significant drawbacks. On the one hand, single-modal data has limited information representation capabilities; joint angles cannot reflect the global posture drift of the torso, and inertial data lacks force feedback from foot-to-ground interaction, resulting in incomplete criteria for judging complex movements. On the other hand, multimodal fusion methods are too coarse, directly stacking heterogeneous data at the input layer, ignoring the inherent correlation structure of different physical sensor signals in spatial and temporal dimensions. This leads to high feature redundancy, with effective information being buried by noise, and the model's generalization ability deteriorates sharply when labeled samples are scarce. More importantly, existing pre-training strategies mostly target single skeleton flows or temporal flows when designing proxy tasks, failing to establish a comparative constraint between spatial structure and temporal dynamics. This results in the learned representation vectors not being sharp enough in depicting the essential attributes of the actions. The technical problems to be solved by the technical solution of this application are: first, how to eliminate the interference of global body motion of the humanoid robot on local joint coordinates and reconstruct an accurate motion skeleton diagram description with pose invariance from multimodal sensor readings; second, how to establish cross-modal mutual information constraints between joint spatial connection patterns and motion temporal evolution laws under label-free conditions, so that the motion embedding obtained by self-supervised pre-training can capture the spatiotemporal cooperative characteristics of the execution process with fine granularity. Summary of the Invention

[0003] The purpose of this invention is to provide a humanoid robot action recognition method based on self-supervised learning. Without relying on manual category labeling, it utilizes the internal structural relationships of multimodal sensor signals to learn representational features with strong discriminative ability for action categories, thus solving the technical bottlenecks of supervised schemes being highly sensitive to the amount of labeled data and insufficient utilization of multimodal information.

[0004] To achieve the above objectives, the present invention provides the following technical solution: The present invention provides a humanoid robot action recognition method based on self-supervised learning. This method includes: acquiring a multimodal temporal data set collected by joint sensors during the humanoid robot's action execution, the multimodal temporal data set including joint angle sequences, inertial measurement unit data, and plantar pressure distribution sequences; performing action intrinsic structure decomposition on the multimodal temporal data set to generate action skeleton graph structural features, the action skeleton graph structural features including joint topology connection matrices and temporal motion trajectory tensors; calling a pre-trained self-supervised encoder backbone network to perform comparative learning pre-training on the action skeleton graph structural features to extract action representation embedding features; and inputting the action representation embedding features into a downstream action classifier to generate the humanoid robot's action recognition result.

[0005] The above method extracts structured action skeleton features from multimodal sensor data, and then obtains a highly discriminative action representation through self-supervised contrastive learning. This method can make full use of unlabeled data to improve feature quality, enabling high-precision action recognition even with a small number of labeled samples, and effectively enhancing robustness to action execution speed, perspective, and individual differences.

[0006] In a preferred embodiment of the present invention, when performing action intrinsic structure decomposition on a multimodal temporal data set, the joint angle sequence is first sliced ​​using a time sliding window to obtain multiple continuous action frame segments. Spatial position encoding is then performed on the joint angle data in each action frame segment to generate instantaneous node coordinate vectors for each joint in three-dimensional space. Attitude angle calculation is then performed on the inertial measurement unit data to generate a rotation matrix of the humanoid robot's torso relative to the global coordinate system. Based on this rotation matrix, global coordinate system alignment correction is performed on each instantaneous node coordinate vector to generate a corrected node coordinate sequence. A joint adjacency matrix is ​​constructed based on the corrected node coordinate sequence, and the coordinate trajectories of all nodes are stacked according to the time dimension to generate an action skeleton graph structural feature containing a joint topological connection matrix and a temporal motion trajectory tensor. This process unifies the originally scattered multimodal data into a spatially aligned skeleton representation, eliminating coordinate deviations caused by changes in torso posture. This allows subsequent graph structure modeling to more accurately capture the real motion dependencies between joints, thereby improving the spatiotemporal consistency of action feature expression.

[0007] Preferably, when calling the pre-trained self-supervised encoder backbone network to process the action skeleton graph structural features, the action skeleton graph structural features are input into the graph convolutional coding branch and the temporal convolutional coding branch of the network, respectively. The graph convolutional coding branch performs spatial message passing processing on the joint topology connection matrix to generate a joint spatial feature map; the temporal convolutional coding branch performs causal convolution processing on the temporal motion trajectory tensor to generate a temporal dynamic feature map; the joint spatial feature map and the temporal dynamic feature map are fused through cross-modal cross-attention processing to generate an action fusion feature tensor; and the action fusion feature tensor is then subjected to global average pooling to generate action representation embedding features. By mining spatial structural relationships and temporal evolution patterns through separate encoding channels, and by achieving adaptive aggregation of bilateral information through a cross-attention mechanism, the final action representation can perceive both the instantaneous configuration of the skeleton and capture the dynamic evolution pattern of the action, significantly enhancing the expressive power of the features.

[0008] In the aforementioned cross-modal attention fusion stage, a preferred implementation is as follows: A joint spatial feature map is constructed as the query matrix, and a temporal dynamic feature map is constructed as the key and value matrices. The dot product similarity between the query matrix and the key matrix is ​​calculated to generate a spatial-temporal attention weight matrix. This weight matrix is ​​then subjected to row-direction Softmax normalization to generate a normalized attention distribution matrix. The normalized attention distribution matrix and the value matrix are multiplied to obtain an attention-weighted temporal feature projection. The temporal feature projection and the joint spatial feature map are then subjected to residual connections and layer normalization to generate an action fusion feature tensor. This approach, while preserving the intrinsic structure of the spatial features, injects the most relevant temporal dynamic information in a residual manner, enhancing the response to key motion moments and effectively mitigating gradient degradation, making the fused features more stable in multi-scale action recognition.

[0009] As another important technical solution of this invention, before the contrastive learning pre-training process, the structural features of the action skeleton graph are first subjected to random augmentation processing to generate a first augmented view and a second augmented view. The random augmentation processing includes random pruning of the node degree in the joint topology connection matrix and random masking of the temporal dimension of the temporal motion trajectory tensor. The first augmented view is input into the self-supervised encoder backbone network to extract a first embedding vector, and the second augmented view is input into the same network to extract a second embedding vector. The cosine similarity between the first embedding vector and the second embedding vector is calculated to obtain the positive sample pair similarity score. Based on the contrastive loss function value of the positive sample pair similarity score and the negative sample similarity score in the negative sample pool, the parameters of the self-supervised encoder backbone network are adjusted. This self-supervised pre-training strategy enables the network to learn to map different augmented views of the same action to similar embedding positions without any manual annotation, while pushing out the representation of different actions, thereby providing efficient initialization and generalization capabilities for downstream tasks.

[0010] When generating the enhanced view, preferably, nodes in the joint topology connection matrix are randomly discarded, and all connecting edges of the discarded nodes are set to zero to generate a first spatial enhancement matrix; a random interval masking operation is performed on the time dimension of the temporal motion trajectory tensor, and the trajectory values ​​within the masked interval are replaced with zero to generate a first temporal enhancement tensor; the first spatial enhancement matrix and the first temporal enhancement tensor are combined to generate a first enhanced view; and the above operations are repeated with a random discard rate and random masking rate different from the first enhanced view to generate a second enhanced view. This dual-view enhancement method simulates partial occlusion and missing motion segments in actual perception, forcing the encoder to learn high-level semantic features that are insensitive to local missing parts, thus improving the robustness of action representation in complex environments.

[0011] Furthermore, after contrastive learning pre-training, global max pooling is applied to the action representation embedding features in the time dimension to generate action-level embedding vectors. These vectors are then input into the clustering pseudo-label generation module, where discrete clustering pseudo-labels are assigned using the K-means clustering algorithm. A clustering contrastive loss function is constructed to calculate the similarity loss value between each action-level embedding vector and its corresponding cluster center. This loss value is then used to fine-tune and update the self-supervised encoder backbone network, ensuring that action representation embedding features with the same clustering pseudo-labels cluster together in the feature space. During this process, the clustering process automatically mines the inherent category structure of the data, further refining the compactness and separability of the feature distribution in an unsupervised manner. This achieves similar feature clustering results to supervised learning even without labels, significantly reducing the reliance on manual labels.

[0012] As a preferred embodiment of the present invention, the self-supervised encoder backbone network may further include a temporal mask prediction branch. Specifically, the structural features of the action skeleton diagram are input into this temporal mask prediction branch to mask a specified time step in the temporal motion trajectory tensor, generating a mask tensor; the mask tensor is subjected to multi-scale convolutional encoding to generate a context-aware feature map of the masked region; then, a fully connected decoder is used to reconstruct the trajectory from the context-aware feature map to generate a predicted trajectory tensor of the masked region; the mean square error between the predicted trajectory tensor and the true trajectory at the corresponding time step in the original temporal motion trajectory tensor is calculated to obtain the trajectory reconstruction loss function value; the encoding parameters of the self-supervised encoder backbone network are updated based on this reconstruction loss function value. By introducing the self-supervised proxy task of mask trajectory reconstruction, the network is forced to learn to infer the motion trend of the occluded part using contextual information, thereby mastering the implicit dynamic laws at a higher level, making the extracted action representation more recognizable to changes in action speed and rhythm.

[0013] When embedding action representations into the downstream action classifier, the preferred approach is to first compress the feature dimension of the input fully connected projection layer to generate a low-dimensional classification feature vector. This low-dimensional feature vector is then normalized using BatchNormalization to generate a normalized classification feature vector. The output layer of the downstream action classifier then applies Softmax probability mapping to the normalized classification feature vector, generating probability distribution vectors for each preset action category. Finally, the action category label corresponding to the highest probability value in the probability distribution vector is selected as the action recognition result for the humanoid robot. This classification process combines feature compression, normalization, and probability calibration, effectively preventing overfitting and ensuring the interpretability and stability of the output probabilities. This facilitates rapid deployment and real-time inference on real-world humanoid robot platforms.

[0014] Training the self-supervised encoder backbone network can also be performed using a teacher-student knowledge distillation approach. The action skeleton graph structural features are input into the teacher and student network branches respectively, extracting teacher and student action representation embedding features. The parameters of the teacher network branch are updated from the student network branch using an exponential moving average. The mean squared error between the student and teacher features is calculated to generate a knowledge distillation loss function. The encoding parameters of the student network branch are updated based on this loss function value, while the teacher network branch parameters are updated simultaneously using a moving average. This self-distillation mechanism requires no additional labeled data and can suppress feature fluctuations in the student network through the stable supervision signal provided by the teacher network, further improving the smoothness and consistency of action representations and enhancing the model's robustness while maintaining real-time requirements.

[0015] The technical effects and advantages provided by the present invention in the above technical solution are as follows: By calculating the attitude angles of the inertial measurement unit (IMU) data to generate a rotation matrix of the torso relative to the global coordinate system, and using this rotation matrix to perform global coordinate system alignment correction on the coordinate vectors of each instantaneous node, the local joint positions affected by the robot's orientation and torso sway can be uniformly transformed to a global space with an absolute reference. This correction mechanism ensures that the generated motion skeleton graph structural features possess both the relative kinematic constraints of local joints and a consistent expression of global pose, eliminating coordinate offsets introduced by differences in robot initial orientation or accidental torso tilting during execution in different acquisition scenarios. The intrinsic motion structure gains pose generalization invariance, providing a cleaner graph input for subsequent encoding networks. The motion skeleton graph structural features are fed into graph convolutional coding and temporal convolutional coding branches respectively. By constructing a joint space feature map as the query matrix and a temporal dynamic feature map as the key and value matrices, cross-modal cross-attention fusion processing is performed, forcing the static spatial relationship of joint topological connections and the dynamic evolution mode of motion trajectory to be matched one by one and reweighted for relevance on the feature channels. This fusion method enables the activated joints in the spatial branches to be accurately located to highly correlated motion stages in the temporal domain. At the same time, the temporal features flowing through the attention gating are projected back to the spatial joint representation and supplemented with information through residual connections. The resulting action fusion feature tensor retains the semantic interpretability of the spatial structure and internalizes the contextual awareness of temporal changes, significantly increasing the inter-class representation distance and compressing the intra-class representation dispersion without introducing additional annotations. Attached Figure Description

[0016] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in this invention. For those skilled in the art, other drawings can be obtained based on these drawings.

[0017] Figure 1 This is a flowchart of a humanoid robot motion recognition method based on self-supervised learning; Figure 2 This is a flowchart of a humanoid robot motion recognition method based on self-supervised learning; Figure 3 It is a flowchart of self-supervised contrastive learning pre-training for action skeleton diagram structural features; Figure 4 This is a schematic diagram of the timing mask prediction branch structure; Figure 5 This is a schematic diagram of the action recognition process based on self-supervised contrastive learning. Detailed Implementation

[0018] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0019] See Figure 1 This invention provides a humanoid robot motion recognition method based on self-supervised learning. The method includes: acquiring a multimodal temporal data set collected by joint sensors during the humanoid robot's motion execution, the multimodal temporal data set including joint angle sequences, inertial measurement unit data, and plantar pressure distribution sequences; performing motion intrinsic structure decomposition on the multimodal temporal data set to generate motion skeleton graph structural features, the motion skeleton graph structural features including joint topology connection matrices and temporal motion trajectory tensors; calling a pre-trained self-supervised encoder backbone network to perform comparative learning pre-training on the motion skeleton graph structural features to extract motion representation embedding features; and inputting the motion representation embedding features into a downstream motion classifier to generate the humanoid robot's motion recognition result.

[0020] Example 1:

[0021] In specific implementation, please refer to Figure 2 The self-supervised learning-based humanoid robot motion recognition method is implemented through the following steps: A multimodal temporal data set, including joint angle sequences, inertial measurement unit data, and plantar pressure distribution sequences, is acquired from joint sensors during the humanoid robot's motion execution. The intrinsic structure decomposition of the multimodal temporal data set is then performed to generate a motion skeleton diagram.

[0022] During the intrinsic structure decomposition of motion, the joint angle sequence is sliced ​​using a time-sliding window to obtain multiple consecutive motion frame segments. The length of the time-sliding window is set to 60 frames, and the sliding step size is set to 30 frames. The 60-frame window length can cover the typical duration of a single complete motion cycle of a humanoid robot under the condition of a joint angle sequence sampling frequency of 100 Hz. Spatial position encoding is performed on the joint angle data in each motion frame segment to generate instantaneous node coordinate vectors of each joint in three-dimensional space. The spatial position encoding process uses the forward kinematics model of the humanoid robot, with the torso coordinate system as the reference coordinate system. Based on the joint angles and the link lengths between adjacent joints, the three-dimensional spatial coordinates of each joint relative to the torso coordinate system are calculated step by step, starting from the root joint of the torso. The link length is a fixed design parameter of the robot's mechanical structure and is pre-stored in the processing system. Attitude angle calculation is performed on the inertial measurement unit data to generate the rotation matrix of the humanoid robot's torso relative to the global coordinate system. The inertial measurement unit (IMU) data includes triaxial angular velocity and triaxial acceleration measurements. Attitude angle calculation employs a complementary filtering algorithm, fusing the dynamic attitude obtained from angular velocity integration with the static gravity-direction attitude calculated from the accelerometers to output a torso attitude quaternion. This quaternion is then converted into a 3×3 rotation matrix. Based on the rotation matrix, a global coordinate system alignment correction is performed on each instantaneous node coordinate vector, generating a corrected node coordinate sequence. This global coordinate system alignment correction transforms each instantaneous node coordinate vector from the torso coordinate system to the global coordinate system. Specifically, the instantaneous node coordinate vector is left-multiplied by the rotation matrix, unifying the node coordinates to the same global reference frame. A joint adjacency matrix is ​​then constructed based on the corrected node coordinate sequence. When constructing the joint adjacency matrix, a predefined template for humanoid robot physical joint connections is used. In this template, if two joints are mechanically connected, the corresponding element is marked as 1; otherwise, it is marked as 0. The Euclidean distance between each pair of physically connected joints is then calculated based on the corrected node coordinate sequence. If the Euclidean distance exceeds a preset breakage threshold, the corresponding connection in the joint adjacency matrix is ​​set to 0, indicating a broken connection; otherwise, it remains at 1. The preset breakage threshold is set to 1.5 times the Euclidean distance between two joints in the robot's standard standing posture. The corrected node coordinates of all nodes are stacked along the time dimension to generate a temporal motion trajectory tensor. The temporal motion trajectory tensor has a dimension of T×N×3, where T represents the number of time frames, N represents the number of joints, and 3 represents the three-dimensional coordinate axes. The motion skeleton graph structural features include the aforementioned joint adjacency matrix and temporal motion trajectory tensor.

[0023] A pre-trained self-supervised encoder backbone network is invoked to perform contrastive learning pre-training on the action skeleton graph structural features to extract action representation embedding features. The self-supervised encoder backbone network includes a graph convolutional encoding branch, a temporal convolutional encoding branch, a cross-modal attention fusion module, and a global average pooling layer. The input to the graph convolutional encoding branch is the joint adjacency matrix and the temporal motion trajectory tensor from the action skeleton graph structural features. The graph convolutional encoding branch consists of three stacked graph convolutional layers. Each graph convolutional layer symmetrically normalizes the joint adjacency matrix to obtain a normalized adjacency matrix. The normalized adjacency matrix is ​​multiplied by the joint space feature matrix input to the current layer, then multiplied by the learnable weight matrix, and finally activated by the ReLU function to achieve spatial message passing. The initial input feature matrix is ​​taken from the joint coordinates of each joint in the current time frame of the temporal motion trajectory tensor. After three layers of graph convolution, a joint space feature map is generated, with dimensions T×N×. ,in The spatial feature dimension is set to 128. The input to the temporal convolutional coding branch is the temporal motion trajectory tensor in the action skeleton graph structural features. The temporal convolutional coding branch consists of four stacked causal convolutional layers. Each causal convolutional layer uses a one-dimensional convolutional kernel to convolve the three-dimensional trajectory sequence of each joint along the time dimension. The kernel size is 3, the stride is 1, and zero padding of length 2 on the left is used to ensure causality, that is, the output at the current time step depends only on the input at the past time step. After convolution, the ReLU activation function is applied. After four layers of causal convolution processing, a temporal dynamic feature map is generated, with a dimension of T×N× ,in The time feature dimension is set to 128.

[0024] Cross-modal cross-attention fusion processing is performed on joint space feature maps and temporal dynamic feature maps to generate an action fusion feature tensor. The specific process of cross-modal cross-attention fusion processing is as follows: The joint space feature map is constructed as the source of the query matrix, and a query matrix Q is obtained through linear transformation. The temporal dynamic feature map is constructed as the source of the key matrix and value matrix, and a key matrix K and value matrix V are obtained through linear transformation. The weight matrix used in the linear transformation... , , They are respectively × , × , × The matrix, where The value is 128. The value is 128. Calculate the dot product similarity between the query matrix Q and the key matrix K, and divide by the scaling factor. The resulting matrix is ​​then subjected to Softmax normalization along its rows to generate a normalized attention distribution matrix. This normalized attention distribution matrix is ​​multiplied by the value matrix V to obtain the attention-weighted temporal feature projection. The attention-weighted temporal feature projection is then residual-joined with the joint space feature map, i.e., element-wise added together. The sum is then layer-normalized to obtain the action fusion feature tensor, which has dimensions T×N×. ,in The value is 128. The layer normalization operation normalizes the mean and variance of the same feature dimension for each sample.

[0025] The core computation of the above cross-modal attention fusion process can be expressed as:

[0026] in, This represents the query matrix obtained by the first linear transformation of the joint spatial feature map, where each row corresponds to the spatial feature representation of a joint in a certain time frame. This represents the transpose of the key matrix obtained by the second linear transformation of the temporal dynamic feature map. Each row of the key matrix corresponds to the temporal dynamic feature representation of a joint in a certain time frame. The column dimension of the key matrix is ​​128, which is equivalent to the feature dimension of the time-series dynamic feature map. This represents the value matrix obtained by the third linear transformation of the temporal dynamic feature map, where each row corresponds to the temporal dynamic feature representation of a joint in a certain time frame; A joint space feature map representing the original input; This indicates that an exponential normalization operation is performed on each row of the matrix; Presentation layer normalization operation; This represents the output action fusion feature tensor.

[0027] Global average pooling is applied to the action fusion feature tensor to generate action representation embedding features. Global average pooling averages the action fusion feature tensor over all frames in the time dimension T, resulting in an N×... The matrix is ​​then processed by max pooling the features along the joint dimension N, resulting in a matrix with dimension N. The action representation is embedded in the feature vector. This action representation embedded feature vector is the action representation embedded feature extracted by the self-supervised encoder backbone network.

[0028] The action representation embedded in the feature vector is input into the downstream action classifier to generate the action recognition result for the humanoid robot. The downstream action classifier consists of a fully connected projection layer, a batch normalization layer, and an output layer connected sequentially. The fully connected projection layer compresses the feature dimension of the action representation embedded in the feature vector, outputting a low-dimensional classification feature vector. The dimension of the low-dimensional classification feature vector is set to the number of action categories, for example, 12. The batch normalization layer standardizes the low-dimensional classification feature vector, making its mean 0 and variance 1, resulting in a normalized classification feature vector. The output layer performs Softmax probability mapping on the normalized classification feature vector, generating a probability distribution vector for each preset action category. The action category label corresponding to the highest probability value in the probability distribution vector is selected as the action recognition result for the humanoid robot.

[0029] Example 2:

[0030] In specific implementation, please refer to Figure 3 Before calling the pre-trained self-supervised encoder backbone network to perform contrastive learning pre-training processing on the action skeleton graph structural features to extract action representation embedding features, random augmentation processing and contrastive loss calculation are performed on the action skeleton graph structural features to adjust the parameters of the self-supervised encoder backbone network.

[0031] The motion skeleton graph structural features include a joint topology connection matrix and a temporal motion trajectory tensor. Random augmentation is performed on the motion skeleton graph structural features to generate a first augmented view and a second augmented view. The random augmentation process includes random pruning of node degrees in the joint topology connection matrix and random masking of the temporal dimension of the temporal motion trajectory tensor.

[0032] In some embodiments, the process of generating the first enhanced view is as follows: A random discard operation is performed on the nodes in the joint topology connection matrix, setting all connection edges of the discarded nodes to zero, generating a first spatial enhancement matrix. The random discard operation follows a preset first node discard rate, which is set to 0.2, meaning that in each random discard operation, each node has a 20% probability of being selected as a discarded node. All element values ​​in the corresponding row and column of the selected discarded node in the joint topology connection matrix are forcibly set to zero, indicating that the connection relationship between the discarded node and all other nodes is removed. A random interval masking operation is performed on the time dimension of the temporal motion trajectory tensor, replacing the trajectory values ​​within the masked interval with zero, generating a first temporal enhancement tensor. The random interval masking operation follows a preset first time masking rate, which is set to 0.15, meaning that 15% of the time frames in the total number of time frames of the temporal motion trajectory tensor will be covered by the masked interval. The length of the mask interval is obtained by uniformly and randomly sampling from the interval length set {3, 5, 7}, and the starting position of the mask interval is uniformly and randomly selected across the entire time dimension. After determining the mask interval, the trajectory values ​​of all joints within the mask interval are replaced with zero values ​​in all coordinate dimensions. The first spatial enhancement matrix and the first temporal enhancement tensor are combined to generate the first enhanced view. The combination operation incorporates the first spatial enhancement matrix as a component of the joint topology connection matrix and the first temporal enhancement tensor as a component of the temporal motion trajectory tensor, together forming the first enhanced view.

[0033] The process of generating the second enhanced view uses the same operation type as generating the first enhanced view, but with different random parameters. Following a different random drop rate and random masking rate than the first enhanced view, the above operations are repeated on the original motion skeleton diagram structural features to generate the second enhanced view. Specifically, a second random drop operation is performed on the nodes in the joint topology connection matrix. This second random drop operation follows a preset second node drop rate, set to 0.3, meaning that each node has a 30% probability of being selected as a dropped node in the second random drop operation. All the connection edges in the rows and columns corresponding to the selected dropped nodes in the joint topology connection matrix are set to zero, generating the second spatial enhancement matrix. A second random interval masking operation is performed on the temporal motion trajectory tensor, following a preset second time masking rate, set to 0.25, meaning that 25% of the time frames in the total time frame of the temporal motion trajectory tensor are covered by the masked interval. The mask interval length for the second random interval masking operation is also obtained by uniformly random sampling from the interval length set {3, 5, 7}. The starting position of the mask interval is uniformly randomized across the entire time dimension, and all trajectory values ​​within the mask interval are replaced with zero values ​​to generate a second temporal augmentation tensor. The second spatial augmentation matrix and the second temporal augmentation tensor are combined to generate a second augmented view.

[0034] The first augmented view is input into the self-supervised encoder backbone network to extract the first embedding vector. The processing of the first augmented view by the self-supervised encoder backbone network includes graph convolutional coding of the first spatial augmentation matrix, temporal convolutional coding of the first temporal augmentation tensor, cross-modal cross-attention fusion, and global average pooling, ultimately outputting the first embedding vector representing the action features of the first augmented view. The second augmented view is input into the same self-supervised encoder backbone network to extract the second embedding vector. The processing of the second augmented view by the self-supervised encoder backbone network shares the same parameters as the processing of the first augmented view, undergoing the same forward propagation process, ultimately outputting the second embedding vector representing the action features of the second augmented view.

[0035] Calculate the cosine similarity between the first and second embedding vectors to generate a positive sample pair similarity score. The cosine similarity is calculated by normalizing the first and second embedding vectors according to their L2 norm, so that the L2 norm of the first and second embedding vectors are both 1. Then, calculate the dot product of the normalized vectors. The dot product result is the positive sample pair similarity score, which ranges from -1 to 1.

[0036] A negative sample pool is constructed within each training batch. For a batch containing B original action skeleton graph structural features, each original action skeleton graph structural feature generates a pair of augmented views, corresponding to B first embedding vectors and B second embedding vectors. For the first embedding vector corresponding to the u-th original action skeleton graph structural feature, the remaining B-1 second embedding vectors are used as negative samples. The cosine similarity between the first embedding vector and each negative sample is calculated, generating B-1 negative sample similarity scores. The parameters of the self-supervised encoder backbone network are adjusted based on the contrastive loss function value between the positive sample pair similarity score and the negative sample similarity score in the negative sample pool. The contrastive loss function adopts the information noise contrastive estimation loss function, the specific calculation form of which is as follows:

[0037] in, This represents the number of original action skeleton graph structural features contained in a training batch, with a value of 256. This represents the index of the original action skeleton graph structural feature corresponding to the first embedding vector used to calculate the loss; This represents the index of the original action skeleton graph structural feature corresponding to the second embedding vector in the negative sample pool; Indicates the first The similarity score between positive sample pairs between the first and second embedding vectors generated corresponding to the structural features of the original action skeleton diagram; Indicates the first The first embedding vector generated corresponding to the structural features of the original action skeleton diagram is related to the... Negative sample similarity scores between the second embedding vectors generated corresponding to the structural features of the original action skeleton diagram; The temperature hyperparameter controls the concentration of similarity distribution. It is set to 0.07, a value that provides suitable gradient discriminative power under mini-batch contrastive learning. The contrastive loss function is calculated using the backpropagation algorithm to determine the gradients of each layer in the self-supervised encoder backbone network. The parameters of the self-supervised encoder backbone network are then updated using a stochastic gradient descent optimizer with a learning rate of 0.001 and a weight decay coefficient of 0.0001.

[0038] Example 3:

[0039] In practice, the self-supervised encoder backbone network performs comparative learning pre-training on the action skeleton graph structural features and extracts action representation embedding features. Then, it performs fine-tuning updates on the action representation embedding features guided by clustering pseudo-labels. The action representation embedding features output by the self-supervised encoder backbone network are action fusion feature tensors, with dimensions T×N×. Where T represents the number of time frames and N represents the number of joints. This indicates the dimension of action fusion features. The value is set to 128. Global max pooling is performed on the action representation embedding features along the temporal dimension to generate action-level embedding vectors. Specifically, the global max pooling process along the temporal dimension T involves taking the maximum value across all time frames for each feature channel of each joint along the time dimension T, and then concatenating all feature channels of all joints to form a vector of length N× The action-level embedding vector serves as the representation of each action sample at the clustering level.

[0040] Action-level embedding vectors are input into the cluster pseudo-label generation module, where discrete cluster pseudo-labels are assigned using the K-means clustering algorithm. The cluster pseudo-label generation module consists of a cluster center initialization unit, a distance calculation unit, a label allocation unit, and a cluster center update unit. The cluster center initialization unit uses the K-means++ initialization method, selecting initial cluster centers from all action-level embedding vectors. When selecting the first cluster center, the K-means++ initialization method selects a uniformly random cluster center from all action-level embedding vectors. Subsequent cluster centers are selected with a probability proportional to the square of the distance from the action-level embedding vector to its nearest existing cluster center. The number of cluster centers in the K-means clustering algorithm is set to 500. Choosing 500 as the number of cluster centers ensures that the clustering granularity can cover the diverse action subtypes of the humanoid robot. The distance calculation unit calculates the cosine distance from each action-level embedding vector to all cluster centers. The cosine distance is defined as 1 minus the normalized dot product of the action-level embedding vector and the cluster center. The label assignment unit assigns each action-level embedding vector to the cluster center with the smallest cosine distance, obtaining a discrete cluster pseudo-label for each action-level embedding vector. The cluster center update unit calculates the mean vector of all action-level embedding vectors belonging to each cluster, using this as the updated cluster center. The cluster center initialization unit, distance calculation unit, label assignment unit, and cluster center update unit are executed iteratively 50 times. This 50-iteration run is designed to converge when the change in the clustering objective function value is less than one ten-thousandth.

[0041] After obtaining discrete cluster pseudo-labels using the K-means clustering algorithm, a cluster contrastive loss function is constructed. This function calculates the similarity loss value between each action-level embedding vector and its corresponding cluster center. The cluster contrastive loss function adopts the Softmax cross-entropy form based on a temperature scaling factor, specifically:

[0042] in, This represents the total number of action-level embedding vectors in a batch, with a batch size of 256. This indicates the ordinal number of the action-level embedding vectors in this batch; Indicates the first Each action-level embedding vector; Indicates the first Discrete cluster pseudo-labels are assigned to each action-level embedding vector; Represents discrete cluster pseudo-labels The corresponding cluster center vector; Indicates the number of cluster centers. The value is 500; This represents the ordinal number used to iterate over all cluster centers; Indicates the first Cluster center vectors; Represents action-level embedding vectors With cluster center The cosine similarity between them is calculated by first normalizing the action-level embedding vector and the cluster center vector using the L2 norm, and then calculating the dot product of the normalized vectors. The temperature hyperparameter for cluster contrast is used to adjust the smoothness of the similarity distribution. The temperature hyperparameter for cluster contrast is set to 0.1. The temperature hyperparameter value of 0.1 makes the clustering loss provide an effective gradient signal on clusters with different pseudo-label discriminative abilities.

[0043] The self-supervised encoder backbone network is fine-tuned using the cluster contrast loss function. During fine-tuning, the cluster center vectors in the cluster pseudo-label generation module remain unchanged. These cluster center vectors are obtained by re-running the K-means clustering algorithm using the current action-level embedding vectors from the full training dataset before each training cycle. In each fine-tuning iteration, the gradients of the parameters of each layer in the self-supervised encoder backbone network are calculated using the backpropagation algorithm based on the cluster contrast loss function value. The parameters of the self-supervised encoder backbone network are then updated using a stochastic gradient descent optimizer. The fine-tuning learning rate of the stochastic gradient descent optimizer is set to 0.0001, which is one-tenth of the learning rate in the pre-training stage. The fine-tuning weight decay coefficient is also set to 0.00001. After multiple iterations, the action representation embedding features output by the self-supervised encoder backbone network are adjusted, increasing the cosine similarity in the feature space for action-level embedding vectors assigned the same discrete cluster pseudo-labels across different humanoid robot action samples. This results in the clustering of action representation embedding features with the same cluster pseudo-labels.

[0044] Example 4:

[0045] In specific implementation, please refer to Figure 4In another implementation method that uses a pre-trained self-supervised encoder backbone network to perform comparative learning pre-training processing on the structural features of the action skeleton graph to extract action representation embedding features, the self-supervised encoder backbone network includes a temporal mask prediction branch. The temporal mask prediction branch is composed of a mask setting module, a multi-scale convolutional encoder, and a fully connected decoder connected sequentially.

[0046] The motion skeleton graph structural features include a temporal motion trajectory tensor with dimensions T×N×3, where T represents the number of time frames (T=60), N represents the number of joints, and 3 represents the three-dimensional coordinate axes. The motion skeleton graph structural features are input into the temporal mask prediction branch of the self-supervised encoder backbone network. The mask setting module sets masks for specified time steps in the temporal motion trajectory tensor to generate a mask tensor. The specified time steps are determined as follows: On the temporal dimension of the temporal motion trajectory tensor, a preset masking ratio of 15% is used to determine the total number of time steps to be masked. Then, time step indices, equal to the total number of masked time steps, are randomly selected according to a uniform distribution. All three-dimensional coordinate values ​​of all joints corresponding to the selected time step indices are replaced with zero values, while all three-dimensional coordinate values ​​of all joints corresponding to the unselected time step indices remain unchanged, forming the mask tensor. The dimensions of the mask tensor are consistent with the temporal motion trajectory tensor, being T×N×3. Meanwhile, the mask setting module records the index set of the masked time step and the original true trajectory value corresponding to the masked time step for subsequent loss calculation.

[0047] The multi-scale convolutional encoder receives a mask tensor as input and performs multi-scale convolutional encoding on it to generate a context-aware feature map of the mask region. The multi-scale convolutional encoder contains three parallel convolutional encoding branches, each with a different receptive field scale. The first convolutional encoding branch consists of three stacked one-dimensional convolutional layers with kernel sizes of 3, 5, and 7, a stride of 1, and 64 kernels per layer. Each convolutional layer is followed by a ReLU activation function, and the output feature map of the first convolutional encoding branch has a dimension of T×N×64. The second convolutional encoding branch consists of three stacked dilated one-dimensional convolutional layers with kernel sizes of 3, 5, and 7, dilation rates of 2, 4, and 8, a stride of 1, and 64 kernels per layer. Each convolutional layer is followed by a ReLU activation function, and the output feature map of the second convolutional encoding branch also has a dimension of T×N×64. The third convolutional coding branch consists of a global average pooling layer and two fully connected layers. First, it performs global average pooling on the mask tensor in the time dimension T, obtaining an N×3 feature matrix. This feature matrix is ​​then flattened into an N×3 vector, mapped to a 64-dimensional intermediate feature map through the first fully connected layer, processed by the ReLU activation function, and then mapped to a 64×N-dimensional output through the second fully connected layer. This output is then rearranged into an N×64 matrix and copied T times in the time dimension, forming a T×N×64 output feature map. The output feature maps from the first, second, and third convolutional coding branches are concatenated along the feature channel dimension, resulting in a concatenated feature map of dimension T×N×192. This concatenated feature map is then compressed back to 64 channels by a 1×1 convolutional layer, processed by the ReLU activation function, and generates a context-aware feature map of the mask region, with a dimension of T×N×64.

[0048] The fully connected decoder receives a context-aware feature map as input, performs trajectory reconstruction on the feature map, and generates a predicted trajectory tensor for the masked region. The fully connected decoder consists of three fully connected layers stacked sequentially. The first fully connected layer maps the context-aware feature map along the feature channel dimension, with 64 input channels and 128 output channels, and outputs an intermediate feature map after ReLU activation. The second fully connected layer maps the intermediate feature map from 128 to 64 feature channels, and outputs a compressed feature map after ReLU activation. The third fully connected layer maps the compressed feature map from 64 to 3 feature channels, generating a predicted trajectory tensor with dimensions T×N×3. Each value in the predicted trajectory tensor corresponds to the predicted trajectory value at the corresponding time step, joint, and coordinate axis in the mask tensor.

[0049] The mean squared error between the predicted trajectory tensor and the true trajectory at the corresponding time step in the original temporal motion trajectory tensor is calculated to generate the trajectory reconstruction loss function value. The trajectory reconstruction loss function value is calculated only for the time steps in the masked time step index set. The predicted values ​​of all time steps in the masked time step index set in the predicted trajectory tensor are collected as a predicted value set, and the true values ​​of the corresponding time steps in the original temporal motion trajectory tensor are collected as a true value set. The element-wise squared difference between the predicted value set and the true value set is calculated, and then the mean of all squared differences is obtained to obtain the trajectory reconstruction loss function value. The formula for calculating the trajectory reconstruction loss function value is:

[0050] in, A set of indices representing the masked time steps; This represents the number of elements in the masked time step index set. Its value is T multiplied by the preset masking ratio of 15% and then rounded down, with a value of 9. This represents any time step index in the set of masked time step indices; This indicates the joint number of a humanoid robot. The value range is from 1 to N; Represents the ordinal numbers of the three-dimensional coordinate axes. The value range is from 1 to 3; This indicates that the predicted trajectory tensor is at time step ,joint coordinate axes The predicted trajectory value on the surface; This represents the original temporal trajectory tensor at time step. ,joint coordinate axes The true value of the trajectory on the surface.

[0051] The encoding parameters of the self-supervised encoder backbone network are updated based on the trajectory reconstruction loss function value. The gradients of the parameters of each layer in the multi-scale convolutional encoder and fully connected decoder of the self-supervised encoder backbone network are calculated using the trajectory reconstruction loss function value via backpropagation. The parameters of the multi-scale convolutional encoder and fully connected decoder are then updated using an adaptive moment estimation optimizer. The learning rate of the adaptive moment estimation optimizer is set to 0.0005, the first-order momentum coefficient to 0.9, the second-order momentum coefficient to 0.999, and the weight decay coefficient to 0.0001. The parameters of the graph convolutional coding branch and the temporal convolutional coding branch in the self-supervised encoder backbone network also undergo gradient calculation during the backpropagation of the trajectory reconstruction loss function value and are updated with the same adaptive moment estimation optimizer parameter settings.

[0052] Example 5:

[0053] In specific implementation, please refer to Figure 5 The complete process of inputting action representation embedding features into a downstream action classifier to generate action recognition results for a humanoid robot is as follows. The action representation embedding features are extracted by a self-supervised encoder backbone network, and the dimension of the action representation embedding feature vector is... , The value is 128. The downstream action classifier consists of a fully connected projection layer, a BatchNormalization layer, and an output layer connected sequentially.

[0054] The action representation embedded in the features is input into the fully connected projection layer of the downstream action classifier for feature dimensionality compression, generating a low-dimensional classification feature vector. The fully connected projection layer contains a weight matrix and a bias vector; the weight matrix has dimensions of... ,in Indicates the number of preset action categories. The value is 12, corresponding to 12 predefined basic action categories for humanoid robots, including walking, running, jumping, squatting, kicking, punching, turning, waving, bending over, tilting to the side, stepping, and standing. The dimension of the bias vector is... Feature dimension compression involves embedding the action representation into the feature vector, performing matrix multiplication with the weight matrix of the fully connected projection layer, and then adding the result of the matrix multiplication element-wise to the bias vector to obtain a dimension of... The low-dimensional classification feature vector.

[0055] The low-dimensional categorical feature vector is normalized using BatchNormalization to generate normalized categorical feature vectors. The BatchNormalization normalization layer contains a learnable scaling parameter vector and a learnable translation parameter vector. The scaling parameter vector has a dimension of [missing value]. The dimension of the translation parameter vector is BatchNormalization first calculates the mean and variance of all low-dimensional classification feature vectors in the current training batch for each preset action category dimension. The calculated mean and variance are then used to standardize the low-dimensional classification feature vectors, ensuring that each dimension of the standardized vector has a mean of 0 and a variance of 1. Next, the standardized vector is multiplied element-wise by the scaling parameter vector, and the result is added element-wise by the translation parameter vector to obtain the normalized classification feature vector. During the inference phase, the BatchNormalization layer uses the global mean and global variance accumulated during training via a moving average to replace the batch-level mean and variance, with the moving average decay coefficient set to 0.9.

[0056] The output layer of the downstream action classifier performs Softmax probability mapping on the normalized classification feature vector to generate probability distribution vectors for each preset action category. The output layer receives the normalized classification feature vector as input, and the normalized classification feature vector contains... Each element in the normalized categorical feature vector is a real value. The Softmax probability mapping processes each element... Calculate the exponential function ,get Each index value, all Summing up the exponent values ​​yields a total exponent value. Dividing each exponent value by the total exponent value gives the probability value for each preset action category. The probability values ​​form a probability distribution vector, and the sum of all probability values ​​in the probability distribution vector is 1.

[0057] The action category label corresponding to the highest probability value in the probability distribution vector is selected as the action recognition result for the humanoid robot. Specifically, the operation involves iterating through all values ​​in the probability distribution vector. Given a probability value, find the index of the location with the highest probability value, and map the location index to the corresponding label in the predefined action category label list. The predefined action category label list corresponds to walking, running, jumping, squatting, kicking, punching, turning, waving, bending over, leaning to the side, stepping, and standing in order from index 0 to 11.

[0058] In some embodiments, when a pre-trained self-supervised encoder backbone network is invoked to perform contrastive learning pre-training on action skeleton graph structural features to extract action representation embedding features, the self-supervised encoder backbone network adopts a two-branch architecture with a teacher network branch and a student network branch. Both the teacher and student network branches have the same network structure, including a graph convolutional coding branch, a temporal convolutional coding branch, a cross-modal cross-attention fusion module, and a global average pooling layer, which is completely consistent with the composition structure of the self-supervised encoder backbone network described above. The parameters of the teacher network branch are updated from the student network branch using an exponential moving average method, and the parameters of the student network branch are updated using a backpropagation algorithm and an optimizer.

[0059] The structural features of the action skeleton graph are input into the teacher network branch of the self-supervised encoder backbone network to extract the teacher action representation embedding features. The forward propagation process of the teacher network branch is consistent with the standard forward propagation process of the self-supervised encoder backbone network: the graph convolutional coding branch of the teacher network branch performs spatial message passing on the joint topology connection matrix; the temporal convolutional coding branch of the teacher network branch performs causal convolution on the temporal motion trajectory tensor; the cross-modal cross-attention fusion module of the teacher network branch fuses the joint spatial feature map and the temporal dynamic feature map; and the global average pooling layer of the teacher network branch outputs the teacher action representation embedding feature vector, which has a dimension of 128.

[0060] The action skeleton graph structural features are input into the student network branch of the self-supervised encoder backbone network to extract student action representation embedding features. The forward propagation process of the student network branch is exactly the same as that of the teacher network branch. The student network branch outputs a student action representation embedding feature vector with a dimension of 128.

[0061] The mean squared error between the embedded features of student action representations and those of teacher action representations is calculated to generate the knowledge distillation loss function value. The knowledge distillation loss function value is calculated as follows: Subtract the student action representation embedding feature vector from the teacher action representation embedding feature vector element by element, calculate the square of the difference, sum the squared differences across all 128 dimensions, and then divide by the number of dimensions (128) to obtain the mean squared error value as the knowledge distillation loss function value. The formula for calculating the knowledge distillation loss function value is:

[0062] in, The action represents the number of dimensions of the embedded feature vector. The value is 128; The action represents the ordinal number of the dimension embedded in the feature vector. The value range is from 1 to 128; The student action represents the embedded feature vector at the th... Values ​​in each dimension; The teacher's action represents the embedded feature vector at the th... The values ​​in each dimension.

[0063] The encoding parameters of the student network branch are updated based on the knowledge distillation loss function value. The gradient of the parameters of each layer in the student network branch is calculated by backpropagation using the knowledge distillation loss function value. The encoding parameters of the student network branch are then updated using a stochastic gradient descent optimizer. The learning rate and weight decay coefficient of the student network branch are set to 0.001.

[0064] The coding parameters of the teacher network branch are updated synchronously using an exponential moving average method. The exponential moving average update operation is performed after each update of the student network branch parameters. Specifically, for each parameter of the teacher network branch… Parameters corresponding to student network branches According to the formula Update, among which The momentum coefficient represents the momentum of an exponential moving average. Set to 0.999. Momentum coefficient. The rationale for setting it to 0.999 is to allow the parameters of the teacher network branch to slowly and smoothly converge with those of the student network branch. Within a pre-training cycle containing 50,000 training batches, the parameter update rate of the teacher network branch is slow enough to maintain a stable target representation while continuously absorbing the learning progress of the student network branch.

[0065] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any changes or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application.

Claims

1. A humanoid robot action recognition method based on self-supervised learning, characterized in that, The method includes: A set of multimodal time-series data collected by joint sensors during the execution of actions by a humanoid robot is obtained. The set of multimodal time-series data includes joint angle sequences, inertial measurement unit data, and plantar pressure distribution sequences. The multimodal temporal data set is subjected to action intrinsic structure decomposition to generate action skeleton graph structural features, which include joint topology connection matrix and temporal motion trajectory tensor. The pre-trained self-supervised encoder backbone network is invoked to perform comparative learning pre-training on the structural features of the action skeleton graph, and action representation embedding features are extracted. The motion representation embedded features are input into a downstream motion classifier to generate the motion recognition result of the humanoid robot.

2. The humanoid robot action recognition method based on self-supervised learning according to claim 1, characterized in that, The step of performing action intrinsic structure decomposition on the multimodal time-series data set to generate action skeleton diagram structural features includes: The joint angle sequence is sliced ​​using a time sliding window to obtain multiple continuous motion frame segments; Spatial position encoding is performed on the joint angle data in each action frame segment to generate instantaneous node coordinate vectors of each joint in three-dimensional space; The inertial measurement unit data is processed to calculate attitude angles, generating a rotation matrix of the humanoid robot torso relative to the global coordinate system. Based on the rotation matrix, a global coordinate system alignment correction is performed on each instantaneous node coordinate vector to generate a corrected node coordinate sequence; Based on the corrected node coordinate sequence, a joint adjacency matrix is ​​constructed, and the coordinate trajectories of all nodes are stacked according to the time dimension to generate an action skeleton graph structural feature containing a joint topology connection matrix and a temporal motion trajectory tensor.

3. The humanoid robot action recognition method based on self-supervised learning according to claim 2, characterized in that, The process of calling a pre-trained self-supervised encoder backbone network to perform comparative learning pre-training on the action skeleton graph structural features, and extracting action representation embedding features, includes: The motion skeleton graph structural features are input into the graph convolutional coding branch of the self-supervised encoder backbone network, and spatial message passing processing is performed on the joint topology connection matrix to generate a joint spatial feature map. The structural features of the motion skeleton graph are input into the temporal convolutional coding branch of the self-supervised encoder backbone network to perform causal convolution processing on the temporal motion trajectory tensor and generate a temporal dynamic feature map. The joint space feature map and the temporal dynamic feature map are subjected to cross-modal cross-attention fusion processing to generate an action fusion feature tensor. The action fusion feature tensor is subjected to global average pooling to generate action representation embedding features.

4. The humanoid robot action recognition method based on self-supervised learning according to claim 3, characterized in that, The step of performing cross-modal cross-attention fusion processing on the joint space feature map and the temporal dynamic feature map to generate an action fusion feature tensor includes: Construct the joint space feature map as a query matrix, and construct the temporal dynamic feature map as a key matrix and a value matrix; Calculate the dot product similarity between the query matrix and the key matrix to generate a spatial-temporal attention weight matrix; The spatial-temporal attention weight matrix is ​​subjected to row-direction Softmax normalization to generate a normalized attention distribution matrix; Perform matrix multiplication on the normalized attention distribution matrix and the value matrix to obtain the attention-weighted temporal feature projection; The temporal feature projection and the joint space feature map are subjected to residual connection and layer normalization to generate an action fusion feature tensor.

5. The humanoid robot action recognition method based on self-supervised learning according to claim 1, characterized in that, Before invoking the pre-trained self-supervised encoder backbone network to perform comparative learning pre-training processing on the action skeleton graph structural features and extracting action representation embedding features, the process also includes: The structural features of the motion skeleton graph are subjected to random enhancement processing to generate a first enhanced view and a second enhanced view. The random enhancement processing includes random pruning of the node degree in the joint topology connection matrix and random masking of the time dimension of the temporal motion trajectory tensor. The first enhanced view is input into the self-supervised encoder backbone network to extract the first embedding vector; The second enhanced view is input into the self-supervised encoder backbone network to extract the second embedding vector; Calculate the cosine similarity between the first embedding vector and the second embedding vector, and generate positive sample pair similarity scores; The parameters of the self-supervised encoder backbone network are adjusted based on the comparison loss function value between the similarity scores of the positive sample pairs and the similarity scores of the negative samples in the negative sample pool.

6. The humanoid robot action recognition method based on self-supervised learning according to claim 5, characterized in that, The random enhancement processing of the structural features of the motion skeleton diagram to generate a first enhanced view and a second enhanced view includes: Randomly discard nodes in the joint topology connection matrix, set all connection edges of the discarded nodes to zero, and generate a first spatial enhancement matrix; A random interval masking operation is performed on the time dimension of the temporal motion trajectory tensor, and the trajectory values ​​within the masked interval are replaced with zeros to generate a first time-enhanced tensor. The first spatial enhancement matrix is ​​combined with the first temporal enhancement tensor to generate a first enhanced view; The above operations are repeated on the original action skeleton diagram structural features according to a random drop rate and random masking rate that are different from those of the first enhanced view to generate a second enhanced view.

7. The humanoid robot action recognition method based on self-supervised learning according to claim 1, characterized in that, After the pre-trained self-supervised encoder backbone network is invoked to perform comparative learning pre-training on the action skeleton graph structural features and extract action representation embedding features, the process further includes: The action representation embedding features are subjected to global max pooling in the time dimension to generate action-level embedding vectors; The action-level embedding vector is input into the clustering pseudo-label generation module, and discrete clustering pseudo-labels are assigned using the K-means clustering algorithm; Construct a clustering contrastive loss function and calculate the similarity loss value between each action-level embedding vector and its cluster center; The self-supervised encoder backbone network is fine-tuned and updated using the cluster contrast loss function value, so that the action representation embedding features of the same cluster pseudo-labels are clustered together in the feature space.

8. The humanoid robot action recognition method based on self-supervised learning according to claim 1, characterized in that, The process of calling a pre-trained self-supervised encoder backbone network to perform comparative learning pre-training on the action skeleton graph structural features, and extracting action representation embedding features, includes: The motion skeleton graph structural features are input into the temporal mask prediction branch of the self-supervised encoder backbone network to set the mask for a specified time step in the temporal motion trajectory tensor and generate a mask tensor. The mask tensor is subjected to multi-scale convolutional encoding to generate a context-aware feature map of the mask region; The context-aware feature map is processed by a fully connected decoder to reconstruct the trajectory, generating a predicted trajectory tensor for the masked region. Calculate the mean square error between the predicted trajectory tensor and the actual trajectory at the corresponding time step in the original temporal motion trajectory tensor, and generate the trajectory reconstruction loss function value; The encoding parameters of the self-supervised encoder backbone network are updated based on the trajectory reconstruction loss function value.

9. The humanoid robot action recognition method based on self-supervised learning according to claim 1, characterized in that, The step of inputting the action representation embedded features into a downstream action classifier to generate the action recognition result of the humanoid robot includes: The action representation embedded features are input into the fully connected projection layer of the downstream action classifier to perform feature dimension compression processing and generate a low-dimensional classification feature vector. The low-dimensional classification feature vector is normalized using BatchNormalization to generate a normalized classification feature vector. The normalized classification feature vector is processed by Softmax probability mapping through the output layer of the downstream action classifier to generate the probability distribution vector of each preset action category. The action category label corresponding to the highest probability value in the probability distribution vector is selected as the action recognition result of the humanoid robot.

10. The humanoid robot action recognition method based on self-supervised learning according to claim 1, characterized in that, The process of calling a pre-trained self-supervised encoder backbone network to perform comparative learning pre-training on the action skeleton graph structural features, and extracting action representation embedding features, includes: The action skeleton graph structural features are input into the teacher network branch of the self-supervised encoder backbone network to extract the teacher action representation embedding features. The parameters of the teacher network branch are updated from the student network branch using an exponential moving average method. The action skeleton graph structural features are input into the student network branch of the self-supervised encoder backbone network to extract the student action representation embedding features. Calculate the mean squared error between the student action representation embedding features and the teacher action representation embedding features, and generate the knowledge distillation loss function value; The coding parameters of the student network branch are updated based on the knowledge distillation loss function value, and the coding parameters of the teacher network branch are updated synchronously using the exponential moving average method.

Images