An open-set skeleton action recognition method and device based on outlier prototype learning

Abstract

This application provides a method and apparatus for open-set skeleton action recognition based on outlier prototype learning. The method includes: constructing a neural network model; preprocessing human skeleton data by obtaining initial features through a multi-branch feature extraction network; processing the initial features using a classifier and a hypersphere feature mapper to obtain logical prediction values ​​for action categories and branch features; training the neural network model based on a training set; after the first iteration, optimizing the feature space of in-distribution samples using multi-class loss; synthesizing virtual outliers in the optimized feature space of in-distribution samples; optimizing the energy boundary by combining in-distribution samples and virtual outliers; after the second iteration, selecting the optimal model weights based on the combined performance of open-set recognition and closed-set classification on the validation set; and determining the action type and triggering the corresponding robot operation based on the energy score output by the trained neural network model and the action classification result.

Description

Technical Field

[0001] This application relates to the field of action recognition technology, and in particular to an open set skeleton action recognition method and apparatus based on outlier prototype learning. Background Technology

[0002] In human-robot collaborative assembly scenarios, the complexity and diversity of product structures mean that relying solely on manual assembly presents efficiency bottlenecks, while fully automated assembly faces limitations such as insufficient flexibility and high costs. Therefore, combining human flexibility and adaptability with the repeatability of robot operations to construct an efficient human-robot collaborative assembly model has become an effective way to achieve flexible production and improve manufacturing efficiency. In this process, the robot's perception and understanding of the operator's actions is the core of achieving high-level collaboration.

[0003] Currently, human motion recognition methods in human-computer collaborative systems are mainly divided into two categories: methods based on video / image input and methods based on skeleton input. The former relies on video frames or image sequences, recognizing human motion through convolutional neural networks and spatiotemporal feature modeling, but it has limitations in terms of lighting changes, occlusion, and privacy protection. Skeleton input-based methods utilize spatiotemporal sequence data of human joints for modeling; however, existing research on skeleton input-based human motion recognition mostly focuses on closed-set scenarios, meaning the motion category remains consistent during training and testing. In actual human-computer collaborative assembly processes, operators may exhibit undefined actions or abnormal behaviors, leading to an open-set recognition scenario. Because research on open-set human motion recognition based on skeleton input is relatively limited, existing methods still fall short in terms of open-set detection accuracy, making it difficult to meet the security and reliability requirements of motion recognition in complex human-computer collaborative applications.

[0004] Therefore, there is an urgent need for a method to improve the accuracy of open set skeleton action recognition. Summary of the Invention

[0005] In view of this, this application provides an open set skeleton action recognition and apparatus based on outlier prototype learning, in order to improve the accuracy of open set skeleton action recognition.

[0006] Specifically, this application is implemented through the following technical solution:

[0007] The first aspect of this application provides an open-set skeleton action recognition method based on outlier prototype learning, the method comprising:

[0008] Construct a neural network model for skeleton action recognition;

[0009] Human skeleton data is preprocessed, and initial features are obtained through a multi-branch feature extraction network;

[0010] The initial features are processed by a classifier and a hypersphere feature mapper respectively to obtain the logical prediction value of the action category and the branch features in the unified hypersphere feature space.

[0011] The human skeleton data is divided into a training set and a validation set. The skeleton action recognition neural network model is trained based on the training set. After the first iteration, the feature space of the in-distribution samples is optimized by multi-class loss. Virtual outliers are synthesized in the optimized feature space of the in-distribution samples. The energy boundary is optimized by combining the in-distribution samples and the virtual outliers. After the second iteration, the optimal model weights are selected based on the comprehensive performance of open set recognition and closed set classification of the validation set.

[0012] The real-time collected artificial skeleton data is input into the trained skeleton motion recognition neural network model. The energy score and motion classification results output by the neural network model are used to determine the motion type and trigger the corresponding robot operation.

[0013] The second aspect of this application provides an open set skeleton action recognition device based on outlier prototype learning, the device comprising a construction module, a processing module, an optimization module and a recognition module;

[0014] The construction module is used to construct a skeleton action recognition neural network model; the skeleton action recognition neural network model includes a data preprocessing module, a multi-branch feature extraction network, a classifier, and a hypersphere feature mapper.

[0015] The processing module is used to preprocess human skeleton data and obtain initial features through the multi-branch feature extraction network;

[0016] The processing module is also used to process the initial features through the classifier and the hypersphere feature mapper respectively to obtain the action category logical prediction value and the branch features in the unified hypersphere feature space;

[0017] The optimization module is used to divide the human skeleton data into a training set and a validation set, train the skeleton action recognition neural network model based on the training set, optimize the feature space of the in-distribution samples through multi-class loss after the first iteration, synthesize virtual outliers in the optimized in-distribution sample feature space, optimize the energy boundary by combining the in-distribution samples and virtual outliers, and select the optimal model weights based on the comprehensive performance of open set recognition and closed set classification of the validation set after the second iteration.

[0018] The recognition module is used to input real-time collected artificial skeleton data into a trained skeleton motion recognition neural network model. By using the energy score and motion classification results output by the neural network model, the motion type is determined and the corresponding robot operation is triggered.

[0019] The open-set skeleton action recognition method and apparatus based on outlier prototype learning provided in this application constructs a neural network model including a multi-branch feature extraction network, a classifier, and a hypersphere feature mapper. Combining virtual outlier synthesis and prototype learning mechanisms, it first extracts initial features using multi-branch features and maps them to a hypersphere space to form class prototypes. Multi-class loss is used to optimize the feature space of in-distribution samples, making the class prototypes more compact and distinguishable. Then, based on the optimized space, virtual outliers located outside the boundary are synthesized to simulate unknown actions. The energy boundary is optimized by combining in-distribution samples and virtual outliers. Finally, recognition and operation triggering are achieved based on energy scores and classification results. From the perspective of outliers, this setup provides the model with simulated samples of unknown actions, making the energy boundary optimization more closely resemble known and unknown actions in real-world scenarios. The distribution differences of unknown actions avoid the boundary ambiguity problem caused by the lack of unknown samples, allowing the model to more accurately distinguish between known and unknown actions. From the perspective of prototype learning, the class prototypes in the hypersphere space are continuously optimized through multi-class loss, ensuring the clustering of similar action features and the separation of dissimilar ones. This provides a stable reference benchmark for energy score calculation and outlier identification, making the classification results more reliable. Overall, this design enables skeleton recognition to have both high-precision known action classification capability (closed-set performance) and effective recognition of unknown actions (open-set performance), reducing the risk of misjudgment. Ultimately, when the robot receives skeleton action instructions, it can accurately respond to the operation corresponding to known actions and remain reasonably silent for unknown actions, improving the applicability and reliability of the entire recognition system in complex real-world scenarios. Attached Figure Description

[0020] Figure 1 A flowchart of the open set skeleton action recognition method based on outlier prototype learning provided in Embodiment 1 of this application;

[0021] Figure 2 This is a schematic diagram of the open set skeleton action recognition device based on outlier prototype learning provided in Embodiment 2 of this application. Detailed Implementation

[0022] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application.

[0023] The terminology used in this application is for the purpose of describing particular embodiments only and is not intended to be limiting of the application. The singular forms “a,” “the,” and “the” used herein are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used herein refers to and includes any and all possible combinations of one or more of the associated listed items.

[0024] It should be understood that although the terms first, second, third, etc., may be used in this application to describe various information, such information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another. For example, without departing from the scope of this application, first information may also be referred to as second information, and similarly, second information may also be referred to as first information. Depending on the context, the word "if" as used herein may be interpreted as "when," "when," or "in response to determination."

[0025] The following specific embodiments are given to illustrate the technical solution of this application in detail.

[0026] Figure 1 This is a flowchart of the open set skeleton action recognition method based on outlier prototype learning provided in Embodiment 1 of this application. Please refer to... Figure 1 The method provided in this embodiment may include:

[0027] S101. Construct a neural network model for skeleton action recognition.

[0028] The skeletal motion recognition neural network model includes a data preprocessing module, a multi-branch feature extraction network, a classifier, and a hypersphere feature mapper. The data preprocessing module performs normalization on the received human skeletal data, decomposing the processed data into joint position data, joint velocity data, and bone data, which are then fed to the corresponding branches of the multi-branch feature extraction network. The multi-branch feature extraction network comprises three parallel feature extraction branches, each with a graph neural network as its backbone. These three branches receive joint position data, joint velocity data, and bone data respectively, extracting the spatial topological relationships and motion features of each data type through the graph neural network, and outputting the initial features corresponding to each branch. The classifier comprises three sub-classifiers corresponding one-to-one with the branches of the multi-branch feature extraction network. Each sub-classifier receives the initial features output by its corresponding branch, performs category mapping operations on the initial features, and outputs a logical prediction value indicating whether the sample under the corresponding branch belongs to a different motion category. The input of the hypersphere feature mapper receives the initial features output by the three branches, transforms the three initial features into the same hypersphere feature space through a unified feature mapping operation, and outputs the hypersphere features corresponding to each branch.

[0029] Specifically, the skeleton motion recognition neural network model is a deep learning model used to recognize human skeleton movements. Through the collaborative work of multiple modules, it extracts features from input human skeleton data and performs tasks such as motion classification. The skeleton motion recognition network model includes a data preprocessing module, a multi-branch feature extraction network, a classifier, and a hypersphere feature mapper. The data preprocessing module receives human skeleton data, first normalizes it to eliminate issues such as data scale differences, and then decomposes the processed data into joint position data, joint velocity data, and bone data, feeding them respectively to the corresponding branches of the multi-branch feature extraction network, providing standardized and clearly categorized data input for subsequent feature extraction. The multi-branch feature extraction network contains three parallel feature extraction branches with graph neural networks as the backbone. These three branches receive joint position data, joint velocity data, and bone data respectively, and use graph neural networks to extract the spatial topological relationships and motion features contained in each type of data, ultimately outputting the initial features corresponding to each branch, providing basic features for subsequent classification and feature mapping. The classifier contains three sub-classifiers, each corresponding to a branch of the multi-branch feature extraction network. Each sub-classifier receives the initial features from the corresponding branch output, performs a category mapping operation on the initial features, and outputs the logical prediction value of the sample under that branch belonging to different action categories, used to determine the known category to which the action belongs. The input of the hypersphere feature mapper receives the initial features from the three branches output. Through a unified feature mapping operation, it transforms the three initial features into the same hypersphere feature space, outputting the hypersphere features corresponding to each branch. This helps in subsequent more effective model training and action recognition inference based on the feature space.

[0030] S102. Preprocess the human skeleton data and obtain initial features through a multi-branch feature extraction network.

[0031] Specifically, the initial features are the results obtained by the multi-branch feature extraction network after extracting features from the human skeleton data (including joint position data, joint velocity data, and bone data) processed by the data preprocessing module.

[0032] In specific implementation, the preprocessing of the human skeleton data includes: acquiring a sequence of human skeleton data to be processed; identifying and extracting the three-dimensional spatial coordinates of the root joint from the 0th frame of the human skeleton data sequence as reference coordinates; for the target frame data in the human skeleton data sequence, subtracting the three-dimensional spatial coordinates of the root joint of the 0th frame from the three-dimensional spatial coordinates of each joint point in the target frame to obtain the relative three-dimensional spatial coordinates of each joint point in the target frame relative to the root joint of the 0th frame; and repeating the coordinate transformation process for each frame of data to obtain normalized human skeleton data.

[0033] Specifically, the process begins by acquiring a complete sequence of human skeleton data, consisting of multiple consecutive frames. Each frame records the three-dimensional spatial coordinates of each joint. The sequence is then located at frame 0 (the initial frame), and the root joints (such as the center point of the pelvis or hip joint) are identified from this frame. Their three-dimensional spatial coordinates are extracted and used as reference coordinates. A specific frame in the sequence is selected as the target frame. All joints within this target frame are traversed, and the three-dimensional coordinates of each joint are calculated: the X-coordinate of the target frame is subtracted from the X-reference coordinate of the root joint in frame 0 to obtain the relative X-coordinate; the Y-coordinate is subtracted from the Y-reference coordinate of the root joint in frame 0 to obtain the relative Y-coordinate; and the Z-coordinate is subtracted from the Z-reference coordinate of the root joint in frame 0 to obtain the relative Z-coordinate. This yields the relative three-dimensional spatial coordinates of each joint in the target frame relative to the root joint in frame 0. This process is repeated for each frame in the human skeleton data sequence, transforming the coordinates of each joint in all frames to obtain the normalized human skeleton data for the entire sequence.

[0034] For example, in one embodiment, the normalization process can be represented as:

[0035] ;

[0036] Among them, the The normalized human skeleton data; The human skeleton data to be processed; These are the three-dimensional spatial coordinates of the root joint in frame 0.

[0037] Furthermore, the normalized human skeleton data is decomposed into joint position data, joint velocity data, and bone data. These three types of data are then input into three corresponding branches of a multi-branch feature extraction network (each branch uses a graph neural network as its backbone). Each branch extracts spatial topological relationships and motion features from the input data, outputting the initial features corresponding to its respective branch. For details on the specific working principle of the multi-branch feature extraction network, please refer to the descriptions in related technologies; they will not be elaborated upon here.

[0038] For example, in one embodiment, the process of obtaining joint positions, joint velocities, and bone data streams from human skeletal data can be represented as:

[0039] ;

[0040] ;

[0041] ;

[0042] Among them, the , , These are the joint position, joint velocity, and skeletal data stream, respectively; For the first Frame of human skeleton data; the The number of joints; The total number of frames; This is a topology diagram of the skeleton connections.

[0043] S103. The initial features are processed by the classifier and the hypersphere feature mapper respectively to obtain the action category logical prediction value and the branch features in the unified hypersphere feature space.

[0044] Specifically, the action category logical prediction value is the probability or likelihood value output by the classifier after processing the initial features, used to determine whether a sample belongs to different action categories; the unified hypersphere feature space is a feature space constructed by the hypersphere feature mapper through specific operations, which can uniformly transform the initial features of different branches into it; the branch features are the feature results corresponding to each branch after the initial features are transformed into the unified hypersphere feature space by the hypersphere feature mapper.

[0045] In practice, the multi-branch feature extraction network outputs initial features from three branches. The initial features of each branch are then fed into the corresponding sub-classifier within the classifier. Each sub-classifier performs a category mapping operation on the received initial features of its corresponding branch, obtaining a logical prediction value for each preset action category under that branch. Each action category corresponds to one logical prediction value. The initial features from the three branches output by the multi-branch feature extraction network are simultaneously input into a hypersphere feature mapper. The hypersphere feature mapper performs a unified feature mapping operation on these three types of initial features, transforming the initial features originally belonging to different branches into the same hypersphere feature space. After the transformation, it outputs the features corresponding to each branch within this unified hypersphere feature space, i.e., the branch features.

[0046] The method provided in this embodiment processes initial features in parallel using a classifier and a hypersphere feature mapper, allowing the same initial feature to undergo two different types of processing simultaneously: the classifier performs category mapping operations on the initial features of each branch through sub-classifiers to output logical prediction values ​​of action categories, directly serving the classification and recognition of known actions, ensuring that the model can accurately determine which preset action category the input skeleton data belongs to; the hypersphere feature mapper transforms the initial features of each branch to the same hypersphere feature space through a unified mapping operation and outputs branch features, providing a structured feature foundation for subsequent calculation of energy scores, generation of virtual outliers, and optimization of energy boundaries, enabling the model to effectively distinguish between known and unknown actions. This approach retains the classifier's accurate classification ability for known actions (ensuring closed-set recognition performance) while providing necessary feature structure support for open-set recognition through the hypersphere feature space, allowing the model to make judgments based on the boundary characteristics of the feature space when facing unknown actions. Ultimately, the model achieves dual capabilities of closed-set classification and open-set recognition, accurately recognizing trained known skeleton actions and effectively distinguishing untrained unknown skeleton actions, thus improving the applicability of skeleton recognition in complex real-world scenarios.

[0047] S104. Divide the human skeleton data into a training set and a validation set. Train the skeleton action recognition neural network model based on the training set. After the first iteration, optimize the feature space of the in-distribution samples through multi-class loss. Synthesize virtual outliers in the optimized in-distribution sample feature space. Optimize the energy boundary by combining the in-distribution samples and virtual outliers. After the second iteration, select the optimal model weights based on the comprehensive performance of open set recognition and closed set classification of the validation set.

[0048] Specifically, multi-class loss is a combination of loss functions used to optimize the model's classification accuracy for multiple action categories, rather than a single loss. In skeleton action recognition scenarios, its core is to simultaneously constrain classification accuracy and feature space clustering. The core purpose of optimizing the feature space of in-distribution samples is to lay a reliable feature foundation for open-set recognition, avoiding the inability to distinguish between known and unknown actions due to feature chaos: because an unoptimized feature space may have problems such as intra-class sample dispersion or inter-class sample overlap, for example, different samples of the same running action may be too far apart in space, or the features of running and brisk walking samples may overlap; this chaos will make it impossible to determine whether a sample belongs to a known category by the distance from the feature to the class center. If the features of the known class are themselves scattered, unknown class samples may mistakenly fall into the feature region of the known class, and conversely, known class samples may be misclassified as unknown.

[0049] Furthermore, virtual outliers are artificially synthesized virtual samples simulating "unknown action features" within the optimized distributed sample feature space, rather than real human skeleton data. The core attribute of these virtual samples is that they do not belong to any known action category. In the feature space, they are far from all known class feature clusters, used to simulate unknown action features that may be encountered in real-world scenarios and that the model has not trained on. The energy boundary is the critical line in the feature space used to distinguish between known and unknown action samples (essentially the energy score threshold of the sample feature to the corresponding class center). The core purpose of optimizing it is to solve the problem of known / unknown misclassification in open set recognition: an unoptimized energy boundary may be too wide or too narrow. If it is too wide, the energy score of unknown action samples (or virtual outliers) will be lower than the threshold, and they will be misclassified as known actions; if it is too narrow, the energy score of some known action samples (especially samples whose features are close to the cluster boundary) will be higher than the threshold, and they will be misclassified as unknown actions.

[0050] Optionally, the multi-class loss includes cross-entropy loss, intra-modal prototype contrast loss, and inter-modal prototype contrast loss. The calculation process of the cross-entropy loss includes: counting the total number of samples in the training set and the total number of action categories; determining the true action category to which the sample belongs for each sample in the training set; obtaining the predicted probability of the sample belonging to all action categories output by the classifier; taking the logarithm of the predicted probability corresponding to the true action category for each sample and taking the negative value; calculating the average of the negative values ​​of all samples in the training set to obtain the cross-entropy loss value.

[0051] Specifically, cross-entropy loss is a loss function used to measure the difference between the prediction results of a classification model and the true labels of samples. In skeleton action recognition, its core is to quantify the model's classification error by calculating the deviation between the model's predicted probability of the action category to which the sample belongs and the true category. The magnitude of its value reflects the accuracy of the model's classification; the smaller the value, the closer the model's prediction of the action category to which the sample belongs is to the true situation, and the better the classification effect.

[0052] In practice, the following steps are taken: First, the samples within the distribution in the training set are statistically analyzed to determine the total number of samples and the total number of action categories they involve. Then, each sample within the distribution in the training set is processed one by one to determine its actual action category (i.e., the label information corresponding to the sample). For each sample within the distribution, the predicted probability of the sample belonging to all action categories is extracted from the classifier's output (each action category corresponds to a probability value). For each sample within the distribution, the predicted probability corresponding to the sample's actual action category is determined from the predicted probabilities of all action categories. This predicted probability is then logarithmically calculated, and the result is negative. After all samples within the distribution have undergone the above logarithmic negative operation, all the negative values ​​are summed and then divided by the total number of samples within the distribution. The result is the cross-entropy loss value.

[0053] For example, in one embodiment, the cross-entropy loss value can be expressed as:

[0054] ;

[0055] Among them, the The cross-entropy loss value; The total number of samples within the training set distribution; The total number of action categories; The actual action category to which the sample belongs; For in-distribution samples Predicted as category The predicted probability.

[0056] Optionally, the calculation process of the intramodal prototype contrast loss includes: determining the class prototype of each action category; wherein, historical class prototypes are fused with the hypersphere features of similar in-distribution samples in the current training set according to a preset ratio, and the fusion result is normalized; for each in-distribution sample in the training set, the hypersphere features of the in-distribution sample are extracted to determine the class prototype of the action category to which it belongs and the class prototypes of all other action categories; the first similarity between the hypersphere features of the in-distribution sample and the class prototype of its category, and the second similarity between the hypersphere features of the in-distribution sample and the class prototype of each other category are calculated; the first similarity is converted into an exponential form as the numerator, and the sum of the numerator and the exponential forms of all the second similarities is used as the denominator, and the class assignment probability of the in-distribution sample is obtained by dividing the numerator by the denominator; the logarithm of the class assignment probability of each in-distribution sample is taken and the negative value is taken, and the average value of the negative values ​​of all in-distribution samples is calculated to obtain the intramodal prototype contrast loss value.

[0057] Specifically, intramodal prototype contrastive loss is a loss function used to optimize the relationship between sample features and corresponding class prototypes within the same modality (such as the feature space of the same branch). Its core principle is to enhance the similarity between similar sample features and class prototypes, and to increase the difference between similar and dissimilar features, thereby improving the compactness and distinguishability of class clusters in the feature space. A smaller intramodal prototype contrastive loss value indicates higher similarity between sample features and similar prototypes, more significant differences with dissimilar prototypes, and better class distinguishability in the feature space.

[0058] In practice, for each action category, the historical class prototype of that category and the hypersphere features of all in-distribution samples belonging to that category in the current training set are obtained. These two features are then fused according to a pre-set ratio, and the fused result is normalized to obtain the class prototype of that action category. The hypersphere features of each in-distribution sample are extracted one by one to determine the class prototype of the action category to which the sample belongs, while simultaneously determining the class prototypes of all other action categories. For each in-distribution sample, the first similarity between its hypersphere features and the class prototype of its corresponding action category, and the second similarity between it and the class prototypes of each other action category, are calculated. The first similarity is converted to an exponential form and used as the numerator. This numerator is added to the exponential forms of all the second similarities to obtain the denominator. The numerator is divided by the denominator to obtain the class assignment probability of the in-distribution sample. The class assignment probability of each in-distribution sample is logarithmically calculated, and the result is then negative. The negative values ​​obtained from the above operations for all in-distribution samples are summed and then divided by the total number of in-distribution samples; the average value obtained is the in-modal prototype contrast loss value.

[0059] For example, in one embodiment, the process of generating a class prototype can be represented as:

[0060] ;

[0061] Among them, the For the first The class prototype of the action; the For the first The historical prototype of the action class; the For updating coefficients; the This represents the mapping features of the training set samples in the hypersphere space.

[0062] Intramodal prototype contrast loss can be expressed as:

[0063] ;

[0064] Among them, the The intramodal prototype contrast loss; The total number of samples within the training set distribution; As the normalization factor, , For lumped parameters, , For category , class prototype, For the sample The hyperspherical characteristics.

[0065] Optionally, the calculation process of the inter-modal prototype comparison loss includes: obtaining the class prototype set of each branch in the multi-branch feature extraction network; the class prototype set of each branch contains the class prototypes of all action categories; for every two different branches, calculating the similarity between the class prototype of each action category in the first branch and the class prototypes of all action categories in the second branch; performing an exponential transformation on the similarity between the class prototype of the target action category in the first branch and the class prototypes of the same action category in the second branch, using this as the numerator; using the sum of the numerator and the exponential transformation result of the similarity between the class prototype of the target action category in the first branch and the class prototypes of all action categories in the second branch as the denominator, and obtaining the cross-branch matching probability of the target action category by dividing the numerator by the denominator; taking the logarithm and negative value of the cross-branch matching probability of each target action category, calculating the average of the negative values ​​of all target action categories, and obtaining the prototype comparison loss between the two branches; calculating the prototype comparison loss of all different branch combinations, and summing all the prototype comparison losses to obtain the total inter-modal prototype comparison loss value.

[0066] Specifically, the intermodal prototype comparison loss is a loss function used to optimize the consistency of class prototypes in different modal branches (such as different feature extraction paths) in a multi-branch feature extraction network. The core is to calculate the matching probability of class prototypes of the same and different action classes between different branches, and force the same class prototypes across modal branches to be more similar and the different class prototypes to be more different, thereby aligning the multimodal feature space.

[0067] In practical implementation, firstly, from each branch of the multi-branch feature extraction network, extract the class prototype set containing all action categories, ensuring that each branch has prototype data for all categories. For any two different branches (e.g., branch A and branch B), calculate the similarity between the class prototype of each action category in one branch (e.g., branch A) and the class prototypes of all action categories in the other branch (e.g., branch B). Taking the "target action category" of branch A as an example, first convert the similarity between its class prototype and the prototypes of the "same action category" in branch B into an exponential form as the numerator; then sum this numerator with the "exponential form of the similarity between the target category prototype of branch A and the prototypes of all categories in branch B" as the denominator; divide the numerator by the denominator to obtain the cross-branch matching probability of the target category. Take the logarithm and negative of the matching probability of each target category, and the average of the negative values ​​of all target categories is the prototype comparison loss between the two branches. Iterate through all different branch combinations, calculate the prototype comparison loss of each group of branches, and finally add up the losses of all branch combinations to obtain the total intermodal prototype comparison loss value.

[0068] For example, in one embodiment, the prototype contrast loss between the first branch and the second branch can be expressed as:

[0069] ;

[0070] Among them, the The prototype contrast loss between the first and second branches; , These are the class prototype sets for the first branch and the second branch, respectively. Total number of action categories; For the first branch The class prototype of the action class; and Similarly, Indicates the temperature coefficient. , and The class prototype collections that represent joint position, joint velocity, and bone branches, respectively.

[0071] The total loss value of intermodal prototype comparison can be expressed as:

[0072] ;

[0073] Among them, the This represents the total loss value for intermodal prototype comparison; The prototype contrast loss between the first and second branches; The prototype contrast loss between the second and third branches; The prototype contrast loss is used between the first and third branches.

[0074] Optionally, the step of synthesizing virtual outliers in the optimized distributed sample feature space includes: in the hypersphere feature space optimized by multi-class loss, for each action category, calculating the distance between the hypersphere feature of each distributed sample in the action category and the hypersphere features of all other samples in the same category, selecting the top preset number of distance values ​​with the largest distance values, and determining the samples corresponding to the distance values ​​as candidate boundary points; calculating the average distance between the candidate boundary points and all samples in the same category, and selecting the sample with the largest average distance as the category boundary point; obtaining the hypersphere features of distributed samples of other action categories, and calculating the difference between the category boundary point and the hypersphere features of samples of other categories; using the category boundary point as a reference, amplifying the difference according to a preset ratio, adding the amplified difference to the features of the category boundary point to generate a new feature point; and using the generated new feature point as a virtual outlier, the virtual outlier being located outside the boundary of the distributed sample feature space.

[0075] In practical implementation, within the hyperspherical feature space optimized by multi-class losses, each action category is processed sequentially: For each in-distribution sample under that category, the distance between its hyperspherical feature and the hyperspherical features of all other samples in the same category is calculated. From all distance values, the top preset number of the largest distance values ​​are selected, and the samples corresponding to these distance values ​​are marked as candidate boundary points. For each candidate boundary point, the average distance between it and the hyperspherical features of all samples in the same category is calculated. Among all candidate boundary points, the sample with the largest average distance is selected as the category boundary point for that action category. Hyperspherical features of in-distribution samples from all other action categories are collected, and the difference (e.g., feature vector difference) between the determined category boundary point and the hyperspherical features of these other category samples is calculated. Using the hyperspherical features of the category boundary point as a benchmark, the obtained difference is amplified according to a preset ratio (e.g., 1.2 times). The amplified difference is added to the hyperspherical features of the category boundary point to generate a new feature point, which is then identified as a virtual outlier.

[0076] For example, in one embodiment, the process of filtering category boundary points can be represented as:

[0077] ;

[0078] in, For branches Normalized hypersphere spatial characteristics The categories extracted for this branch in the training set The set of features of a hypersphere space, For the first distance Recent normalized hypersphere space samples, This indicates the calculation of the Euclidean distance between sample points.

[0079] The process of generating virtual outliers can be represented as:

[0080] ;

[0081] in, for The coefficient of the distribution, For category boundary points, These are sample points for other categories in the hypersphere space.

[0082] The method provided in this embodiment first selects candidate boundary points for each action category in the optimized hypersphere feature space, then selects the boundary point with the largest average distance from all samples of the same category. Next, it calculates the difference between the boundary point and the features of other category samples and amplifies it according to a preset ratio. Finally, it adds the amplified difference to the boundary point features to generate a virtual outlier located outside the boundary of the feature space within the distribution. This method can generate virtual samples that closely resemble the distribution of real unknown action features and are located outside the known category boundaries, providing data support for the model to simulate unknown actions. This allows the model to learn the boundary range of the known category feature space during training. By comparing with the virtual outlier and optimizing the energy boundary, the model can more accurately distinguish between known and unknown actions in actual skeleton recognition. This improves the stability of closed-set classification and enhances the reliability of open-set recognition, ultimately improving the adaptability and accuracy of overall skeleton recognition in complex scenes.

[0083] Optionally, the optimization of the energy boundary by combining in-distribution samples and virtual outliers includes: for in-distribution samples and synthesized virtual outliers in the training set, based on the logistic prediction values ​​of all action categories output by the classifier, summing the logistic prediction values ​​after exponential transformation, and then taking the logarithm and negative of the summation result to obtain the energy score of each in-distribution sample and each virtual outlier; statistically analyzing the energy scores of all in-distribution samples in the training set according to a preset period, sorting the energy scores, and selecting the score value corresponding to a specific quantile to determine the current in-distribution energy boundary; setting a preset out-of-distribution energy boundary; the energy score of each in-distribution sample is further analyzed. The energy boundary outside the distribution is higher than the energy boundary inside the distribution. The first part of the energy score of the samples inside the distribution exceeds the energy boundary inside the distribution, and the second part of the energy score of the virtual outliers is lower than the energy boundary outside the distribution. Positive values ​​are taken for the values ​​of the first part and the second part that exceed or fall short of the energy boundary. The average of the positive values ​​of all samples inside the distribution and the virtual outliers is calculated as the energy score loss function value. The parameters of the skeleton action recognition neural network model are updated through backpropagation of the energy score loss function. The updated energy scores of the samples inside the distribution are clustered below the energy boundary inside the distribution, and the energy scores of the virtual outliers are clustered above the energy boundary outside the distribution.

[0084] In practice, for each in-distribution sample and each synthesized virtual outlier in the training set, the logistic prediction values ​​of all action categories output by the classifier are extracted. Each logistic prediction value is exponentially transformed, and the transformed results are summed. The sum is then logarithmically calculated and negatively taken to obtain the energy score corresponding to each in-distribution sample and each virtual outlier. Energy scores of all in-distribution samples in the training set are collected according to a preset time period (e.g., after a certain number of iterations). These energy scores are sorted in ascending or descending order. From the sorted results, a specific quantile (e.g., the 95th percentile) is selected, and this value is determined as the current in-distribution energy boundary. A preset out-of-distribution energy boundary value is directly set, ensuring that the value of this out-of-distribution energy boundary is greater than the value of the in-distribution energy boundary. For each in-distribution sample, calculate the difference between its energy score and the in-distribution energy boundary. If the difference is positive (i.e., the energy score exceeds the in-distribution energy boundary), this difference is the first part. For each virtual outlier, calculate the difference between the out-of-distribution energy boundary and its energy score. If the difference is positive (i.e., the energy score is below the out-of-distribution energy boundary), this difference is the second part. Sum all positive values ​​(0 for negative values) from both parts 1 and 2, and calculate the sum of these positive values. Divide the sum by the total number of in-distribution samples and virtual outliers; the average value is the energy score loss function value. Based on the energy score loss function value, adjust the parameters of the skeleton action recognition neural network model using the backpropagation algorithm. After parameter updates, the energy scores of in-distribution samples are concentrated below the in-distribution energy boundary, and the energy scores of virtual outliers are concentrated above the out-of-distribution energy boundary. The specific implementation process of adjusting model parameters using the backpropagation algorithm can be found in the descriptions in related technologies, and will not be elaborated here.

[0085] For example, in one embodiment, the calculation process of the energy fraction can be expressed as follows:

[0086] ;

[0087] in, For input The corresponding classifier for the first The logical predicted value of the class; Energy fraction; This represents the total number of action categories.

[0088] The energy fraction loss function value can be expressed as:

[0089] ;

[0090] in, This represents the energy fraction loss function value. For the training dataset; The energy score of the training set samples; The energy fraction of the virtual outlier; The energy boundary of the sample within the distribution. This represents the external energy boundary of the distribution.

[0091] The method provided in this embodiment first calculates the energy scores of in-distribution samples and virtual outliers based on the logistic prediction values ​​output by the classifier. Then, it statistically analyzes the energy scores of in-distribution samples at a preset period and selects a specific quantile as the in-distribution energy boundary. Simultaneously, it sets an out-of-distribution energy boundary higher than the in-distribution boundary. Next, it calculates the portion of the in-distribution sample energy score that exceeds the in-distribution boundary and the portion of the virtual outlier energy score that is lower than the out-of-distribution boundary. After taking positive values ​​for these portions, it averages them to obtain the energy score loss function value. Finally, it updates the model parameters through backpropagation to cause the in-distribution sample energy scores to cluster below the in-distribution boundary and the virtual outliers to cluster above the out-of-distribution boundary. Clustering, by simultaneously constraining the energy score distribution of known action samples and virtual samples simulating unknown actions in the feature space, can accurately optimize the energy boundary. This allows the model to more clearly distinguish between known actions (energy score below the inner boundary of the distribution) and unknown actions (energy score above the outer boundary of the distribution) based on energy scores during skeleton recognition. This reduces the chance of known actions being misclassified as unknown, and also lowers the probability of unknown actions being misclassified as known. Furthermore, continuous optimization makes the energy boundary more closely match the actual data distribution, ultimately improving the accuracy and robustness of skeleton recognition in open-set scenarios, enabling the model to better adapt to complex recognition tasks that include unknown actions.

[0092] S105. Input the real-time collected artificial skeleton data into the trained skeleton action recognition neural network model. Based on the energy score and action classification results output by the neural network model, determine the action type and trigger the corresponding robot operation.

[0093] In specific implementation, the step of determining the action type and triggering the corresponding robot operation by using the energy score output by the neural network model and the action classification result includes: obtaining the energy score output by the neural network model and the logical prediction value of the multi-branch feature extraction network; comparing the energy score with a predefined judgment threshold; if the energy score is greater than the judgment threshold, determining that the corresponding action belongs to an unknown action and not triggering the robot to perform any operation; if the energy score is less than the judgment threshold, determining the action type based on the logical prediction value and triggering the robot to perform the operation corresponding to the action type.

[0094] Specifically, from the trained skeleton motion recognition neural network model, two key outputs are extracted: one is the energy score calculated by the model, and the other is the logical prediction value of each action category output by the multi-branch feature extraction network. A predefined threshold for determining whether an action is known is retrieved from the system. The obtained energy score is compared with this threshold. If the energy score is greater than the threshold, the current action is determined to be an unknown action, and no operation command is sent to the robot; the robot maintains its current state. If the energy score is less than the threshold, the specific type of the action (e.g., "waving" or "nodding") is determined based on the logical prediction value output by the multi-branch feature extraction network. Based on the determined action type, a pre-defined instruction mapping relationship is invoked to generate and send the corresponding operation command to the robot, triggering the robot to execute an operation matching the action type.

[0095] The method provided in this embodiment, in its first aspect, involves calculating the distance between samples within each action category and other samples of the same category in the hypersphere feature space optimized by multiple loss classes to screen candidate boundary points. Then, it determines the category boundary points based on the average distance between the candidate boundary points and all samples of the same category. Next, it calculates the feature difference between the category boundary points and samples of other categories and amplifies it according to a preset ratio. Finally, it generates virtual outliers located outside the boundary of the feature space within the distribution. This method can simulate the feature distribution of real unknown actions, providing the model with a reference of unknown action samples that are lacking. It avoids the problem that the model cannot learn the boundaries between known and unknown actions due to the scarcity of real unknown samples, enabling the model to more accurately perceive the range of the known category feature space in subsequent training, reducing misjudgments of unknown actions, and improving the reliability of open set recognition.

[0096] Secondly, a stable class prototype learning mechanism is constructed by using intramodal prototype contrast loss (which fuses and normalizes the hypersphere features of historical class prototypes and current class samples proportionally to obtain class prototypes, and calculates the similarity between samples and class / dissimilar prototypes to optimize feature clustering) and intermodal prototype contrast loss (which calculates the cross-branch matching probability of class / dissimilar prototypes between multiple branches to align the multimodal feature space). This mechanism enables class action features to present a state of "compact intra-class and separated inter-class" in the hypersphere space, while ensuring that the feature representation of the same action is consistent across multiple branches. This provides an accurate feature benchmark for energy score calculation and action classification, which not only improves the accuracy of closed set classification, but also lays a clear feature foundation for distinguishing known and unknown actions in open set recognition.

[0097] Thirdly, by first calculating the energy scores of in-distribution samples and virtual outliers based on the classifier's logical prediction values, then selecting specific quantiles of the in-distribution sample energy scores as the in-distribution energy boundary according to a preset period, and setting an out-of-distribution energy boundary above this boundary, the energy score loss function is constructed by calculating the out-of-distribution portion of in-distribution samples and the portion of virtual outliers below the out-of-distribution boundary through backpropagation. Finally, the model parameters are updated through backpropagation to make the energy scores of in-distribution samples cluster below the in-distribution boundary and the virtual outliers cluster above the out-of-distribution boundary. This method can dynamically optimize the energy boundary, making the boundary more consistent with the actual data distribution, avoiding the problem of known actions being misjudged as unknown or unknown actions being misjudged as known due to fixed boundaries. This allows the model to clearly distinguish action types based on energy scores during actual recognition, further improving the accuracy and robustness of open-set skeleton action recognition.

[0098] Corresponding to the aforementioned embodiment of an open set skeleton action recognition method based on outlier prototype learning, this application also provides an embodiment of an open set skeleton action recognition device based on outlier prototype learning.

[0099] Figure 2 This is a schematic diagram of the open set skeleton action recognition device based on outlier prototype learning provided in Embodiment 2 of this application. Please refer to... Figure 2 The apparatus provided in this embodiment includes a construction module 210, a processing module 220, an optimization module 230, and an identification module 240;

[0100] The construction module 210 is used to construct a skeleton action recognition neural network model; the skeleton action recognition neural network model includes a data preprocessing module, a multi-branch feature extraction network, a classifier, and a hypersphere feature mapper.

[0101] The processing module 220 is used to preprocess the human skeleton data and obtain initial features through the multi-branch feature extraction network;

[0102] The processing module 220 is further configured to process the initial features through the classifier and the hypersphere feature mapper respectively to obtain the action category logical prediction value and the branch features in the unified hypersphere feature space;

[0103] The optimization module 230 is used to divide the human skeleton data into a training set and a validation set, train the skeleton action recognition neural network model based on the training set, optimize the feature space of the in-distribution samples through multi-class loss after the first iteration, synthesize virtual outliers in the optimized in-distribution sample feature space, optimize the energy boundary by combining the in-distribution samples and virtual outliers, and select the optimal model weights based on the comprehensive performance of open set recognition and closed set classification of the validation set after the second iteration.

[0104] The recognition module 240 is used to input the real-time collected artificial skeleton data into the trained skeleton action recognition neural network model, and determine the action type and trigger the corresponding robot operation by using the energy score and action classification result output by the neural network model.

[0105] The apparatus of this embodiment can be used to perform... Figure 1 The steps of the method embodiment shown are similar in principle and process, and will not be repeated here.

[0106] The specific implementation process of the functions and roles of each unit in the above device can be found in the implementation process of the corresponding steps in the above method, and will not be repeated here.

[0107] For the device embodiments, since they basically correspond to the method embodiments, the relevant parts can be referred to in the description of the method embodiments. The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this application according to actual needs. Those skilled in the art can understand and implement this without creative effort.

[0108] The above description is merely a preferred embodiment of this application and is not intended to limit this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of protection of this application.

Claims

1. A method for open-set skeleton action recognition based on outlier prototype learning, characterized in that, The method includes: Construct a neural network model for skeleton action recognition; Human skeleton data is preprocessed, and initial features are obtained through a multi-branch feature extraction network; The initial features are processed by a classifier and a hypersphere feature mapper respectively to obtain the action category logical prediction value and the branch features in the unified hypersphere feature space; the action category logical prediction value is the probability or likelihood value output by the classifier after processing the initial features, used to determine whether a sample belongs to different action categories. The human skeleton data is divided into a training set and a validation set. The skeleton action recognition neural network model is trained based on the training set. After the first iteration, the feature space of the in-distribution samples is optimized by multi-class loss. Virtual outliers are synthesized in the optimized feature space of the in-distribution samples. The energy boundary is optimized by combining the in-distribution samples and the virtual outliers. After the second iteration, the optimal model weights are selected based on the comprehensive performance of open set recognition and closed set classification of the validation set. The real-time collected artificial skeleton data is input into the trained skeleton motion recognition neural network model. The energy score and motion classification results output by the neural network model are used to determine the motion type and trigger the corresponding robot operation.

2. The method according to claim 1, characterized in that, The process of synthesizing virtual outliers in the optimized feature space of the distributed samples includes: In the hypersphere feature space after multi-class loss optimization, for each action category, the distance between the hypersphere feature of each sample in the distribution of the action category and the hypersphere feature of all other samples of the same category is calculated. The top preset number of distance values ​​with the largest distance values ​​are selected, and the samples corresponding to the distance values ​​are determined as candidate boundary points. Calculate the average distance between the candidate boundary point and all samples of the same class, and select the sample with the largest average distance as the class boundary point; Obtain the hypersphere features of in-distribution samples of other action categories, and calculate the difference in hypersphere features between the category boundary points and other category samples; Based on the category boundary point, the difference is amplified according to a preset ratio, and the amplified difference is added to the feature of the category boundary point to generate a new feature point; The newly generated feature points are used as virtual outliers, which are located outside the boundary of the feature space of the in-distribution samples.

3. The method according to claim 1, characterized in that, The optimization of the energy boundary by combining in-distribution samples and virtual outliers includes: For the in-distribution samples and the synthesized virtual outliers in the training set, the energy scores of each in-distribution sample and each virtual outlier are obtained by summing the logistic prediction values ​​of all action categories output by the classifier after exponential transformation of each logistic prediction value, and then taking the logarithm and negative of the summation result. According to the preset period, the energy scores of all samples within the distribution in the training set are statistically analyzed. After sorting the energy scores, the score value corresponding to a specific quantile is selected to determine the current energy boundary within the distribution. A preset external energy boundary is set; the external energy boundary is higher than the internal energy boundary. Calculate the first part of the energy fraction of the in-distribution sample that exceeds the in-distribution energy boundary, and the second part of the energy fraction of the virtual outlier that is below the out-of-distribution energy boundary; For the values ​​in the first and second parts that are exceeded or insufficient, positive values ​​are taken respectively. The average of the positive values ​​of all in-distribution samples and virtual outliers is calculated as the energy fraction loss function value. The parameters of the skeleton action recognition neural network model are updated through backpropagation of the energy fraction loss function. The updated energy fractions of in-distribution samples are clustered below the in-distribution energy boundary, and the energy fractions of virtual outliers are clustered above the out-of-distribution energy boundary.

4. The method according to claim 1, characterized in that, The multiple types of losses include cross-entropy loss, intra-modal prototype contrast loss, and inter-modal prototype contrast loss. The calculation process of the cross-entropy loss includes: The total number of samples and the total number of action categories within the statistical training set distribution are calculated. For each in-distribution sample in the training set, determine the true action category to which the in-distribution sample belongs; Obtain the predicted probability that a sample in the distribution output by the classifier belongs to all action categories; For each sample within the distribution, take the logarithm of the predicted probability corresponding to the actual action category and then take the negative value; The average of the negative values ​​for all samples within the distribution is used to obtain the cross-entropy loss value.

5. The method according to claim 4, characterized in that, The calculation process of the intramodal prototype contrast loss includes: Determine the class prototype for each action category; among them, fuse the hypersphere features of historical class prototypes with samples of the same distribution in the current training set according to a preset ratio, and normalize the fusion result; For each in-distribution sample in the training set, extract the hypersphere features of the in-distribution sample to determine the class prototype of the action category to which it belongs and the class prototypes of all other action categories. Calculate the first similarity between the hypersphere features of the samples within the distribution and the class prototype of their respective category, and the second similarity between the samples and the class prototypes of each other category; The first similarity is converted into an exponential form as the numerator, and the sum of the numerator and the exponential forms of all the second similarities is used as the denominator. The class assignment probability of the samples in the distribution is obtained by dividing the numerator by the denominator. Take the logarithm and negative value of the class assignment probability of each sample within the distribution, and calculate the average of the negative values ​​of all samples within the distribution to obtain the intramodal prototype contrast loss value.

6. The method according to claim 4, characterized in that, The calculation process of the intermodal prototype contrast loss includes: Obtain the class prototype set of each branch in the multi-branch feature extraction network; the class prototype set of each branch contains the class prototypes of all action categories; For each pair of distinct branches, calculate the similarity between the class prototype of each action category in the first branch and the class prototype of all action categories in the second branch; The similarity between the class prototype of the target action category in the first branch and the class prototype of the same action category in the second branch is exponentially transformed and used as the numerator. The sum of the exponential transformation results of the similarity between the numerator and the class prototype of the target action category in the first branch and the class prototype of all action categories in the second branch is used as the denominator. The cross-branch matching probability of the target action category is obtained by dividing the numerator by the denominator. Take the logarithm and negative value of the cross-branch matching probability for each target action category, calculate the average of the negative values ​​for all target action categories, and obtain the prototype contrast loss between the two branches; Calculate the prototype contrast loss for all different branch combinations, and sum all the prototype contrast losses to obtain the total prototype contrast loss value between modes.

7. The method according to claim 1, characterized in that, The skeleton motion recognition neural network model includes a data preprocessing module, a multi-branch feature extraction network, a classifier, and a hypersphere feature mapper. The data preprocessing module is used to perform normalization processing on the received human skeleton data, and decompose the processed data into joint position data, joint velocity data and bone data, which are then sent to the corresponding branches of the multi-branch feature extraction network. The multi-branch feature extraction network contains three parallel feature extraction branches, each of which uses a graph neural network as its backbone network. The three branches are used to receive joint position data, joint velocity data, and skeleton data respectively. The graph neural network extracts the spatial topological relationship and motion features of various types of data and outputs the initial features corresponding to each branch. The classifier includes three sub-classifiers that correspond one-to-one with the branches of the multi-branch feature extraction network. Each sub-classifier is used to receive the initial features output by the corresponding branch, perform class mapping operations on the initial features, and output the logical prediction value of the sample under the corresponding branch belonging to different action categories. The input of the hypersphere feature mapper is used to receive the initial features output from the three branches. Through a unified feature mapping operation, the three types of initial features are transformed into the same hypersphere feature space, and the hypersphere features corresponding to each branch are output.

8. The method according to claim 1, characterized in that, The preprocessing of the human skeleton data includes: Obtain the human skeleton data sequence to be processed, and identify and extract the three-dimensional spatial coordinates of the root joints from the 0th frame of the human skeleton data sequence as the reference coordinates; For the target frame data in the human skeleton data sequence, the three-dimensional spatial coordinates of each joint point in the target frame are subtracted from the three-dimensional spatial coordinates of the root joint of the 0th frame to obtain the relative three-dimensional spatial coordinates of each joint point in the target frame relative to the root joint of the 0th frame. The coordinate transformation process is repeated for each frame of data to obtain normalized human skeleton data.

9. The method according to claim 1, characterized in that, The process of determining the action type and triggering the corresponding robot operation based on the energy score and action classification result output by the neural network model includes: Obtain the energy score output by the neural network model and the logical prediction value of the multi-branch feature extraction network; The energy score is compared with a predefined judgment threshold. If the energy score is greater than the judgment threshold, the corresponding action is determined to be an unknown action, and the robot is not triggered to perform any operation. If the energy score is less than the judgment threshold, the action type is determined based on the logical prediction value, and the robot is triggered to perform the operation corresponding to the action type.

10. An open-set skeleton action recognition device based on outlier prototype learning, characterized in that, The device includes a construction module, a processing module, an optimization module, and an identification module; The construction module is used to construct a skeleton action recognition neural network model; the skeleton action recognition neural network model includes a data preprocessing module, a multi-branch feature extraction network, a classifier, and a hypersphere feature mapper. The processing module is used to preprocess human skeleton data and obtain initial features through the multi-branch feature extraction network; The processing module is further configured to process the initial features using the classifier and the hypersphere feature mapper respectively, to obtain the action category logical prediction value and the branch features in the unified hypersphere feature space; the action category logical prediction value is the probability or likelihood value output by the classifier after processing the initial features, used to determine whether a sample belongs to different action categories. The optimization module is used to divide the human skeleton data into a training set and a validation set, train the skeleton action recognition neural network model based on the training set, optimize the feature space of the in-distribution samples through multi-class loss after the first iteration, synthesize virtual outliers in the optimized in-distribution sample feature space, optimize the energy boundary by combining the in-distribution samples and virtual outliers, and select the optimal model weights based on the comprehensive performance of open set recognition and closed set classification of the validation set after the second iteration. The recognition module is used to input real-time collected artificial skeleton data into a trained skeleton motion recognition neural network model. By using the energy score and motion classification results output by the neural network model, the motion type is determined and the corresponding robot operation is triggered.

Images