Offline evaluation method and device of robot prediction model and electronic equipment

Abstract

This disclosure provides an offline evaluation method, apparatus, and electronic device for a robot prediction model, comprising: dividing the robot's operation dataset into a training set and a test set; performing clustering operations based on each real motion vector in the training set to determine multiple motion prototype categories; inputting sensor data from the test set into the robot prediction model to perform prediction, obtaining predicted motion vectors for each time series; converting each real motion vector in the test set and each predicted motion vector output by the model into real category labels and predicted category labels according to each motion prototype category; performing error analysis based on the real motion vectors, real category labels, predicted motion vectors, and predicted category labels output by the model for the corresponding time series in the test set, and generating model evaluation results. This disclosure achieves an offline evaluation scheme with semantic interpretability and no human intervention by clustering continuous motions output by the prediction model into discrete motion prototype categories.

Description

Technical Field

[0001] This disclosure relates to the field of robotic artificial intelligence technology, specifically to an offline evaluation method, apparatus, electronic device, and storage medium for a robot prediction model. Background Technology

[0002] Vision-Language-Action (VLA) models directly output continuous action vectors (such as 7-dimensional joint angles plus gripper opening and closing angles) from multimodal inputs (images, verbal commands, proprioception). Essentially, it is a continuous sequence prediction / regression problem. Current mainstream evaluation methods suffer from the following fundamental flaws: The first category is task success rate evaluation schemes. This scheme deploys the model to a physical robot or simulation environment to execute a complete task, using the ratio of successful tasks to the total number of trials as the performance evaluation metric. Physical robot evaluation offers the highest realism, but it is costly to deploy, has a long testing cycle, and poses safety risks related to equipment operation. Simulation environment evaluation has the advantages of speed and no safety risks, but it generally suffers from a simulation-to-real gap. The core shortcomings of this type of scheme are: heavy reliance on costly and inefficient physical / simulation deployment processes, difficulty in quickly iterating and optimizing the model using static datasets, and the ability to output only a single quantitative indicator, unable to pinpoint the sub-steps and error types corresponding to task failures.

[0003] The second category is continuous error metric evaluation schemes (common metrics: MSE, MAE, L2 loss). This scheme is based on static datasets and calculates the numerical deviation between the model's predicted actions and the actual labeled actions. Its inherent shortcomings are: the error value is not directly related to the actual success of the task; low action error does not mean the task can be completed successfully; the metric lacks semantic interpretability and cannot determine whether the model has mastered the standard operating paradigm; it only outputs the global overall error value, making it difficult to locate the specific joint or action execution stage corresponding to the failure.

[0004] The third category is evaluation schemes based on manually predefined discrete action classifications (such as DriveAction). This scheme involves manually defining several discrete action categories and mapping the continuous actions output by the model to these categories. Model performance is measured by classification accuracy. This scheme has three main drawbacks: the action categories are entirely predefined manually, failing to cover all diverse actions within the robot's continuous action space; action categories need to be redefined after changing tasks or robot models, resulting in poor generalization ability; and it only outputs the overall classification accuracy, lacking fine-grained diagnostic information such as confusion matrices and sub-class accuracy.

[0005] In summary, existing evaluation methods each have significant shortcomings: some solutions rely on high-cost physical deployments, some offline quantitative indicators lack semantic interpretation capabilities, and the remaining solutions depend on manual prior settings and have coarse diagnostic granularity. Therefore, there is an urgent need for a novel evaluation method that eliminates the need for manual pre-definition of action categories, automatically mines action prototypes, and outputs fine-grained diagnostic analysis reports to form a closed-loop solution for model iterative optimization. Summary of the Invention

[0006] In view of this, this disclosure provides an offline evaluation scheme for robot prediction models, which solves the technical problems of existing evaluation methods that cannot transform continuous action prediction problems into interpretable classification problems, lack semantic diagnostic capabilities, rely on manual presets, and cannot directly guide model improvement.

[0007] According to a first aspect of this disclosure, an offline evaluation method for a robot prediction model is provided, comprising: dividing a dataset of robot task execution operations into a training set and a test set, wherein the training set includes real motion vectors at corresponding time intervals, and the test set includes real motion vectors at corresponding time intervals and sensor data; performing a clustering operation based on each real motion vector in the training set to determine multiple motion prototype categories; inputting the sensor data at each time interval in the test set into the robot prediction model to perform prediction, thereby obtaining predicted motion vectors at each time interval; converting each real motion vector in the test set and each predicted motion vector output by the robot prediction model into real category labels and predicted category labels, respectively, according to the multiple motion prototype categories; performing error analysis based on the real motion vectors and their real category labels at each time interval in the test set and the predicted motion vectors and their predicted category labels at the corresponding time interval output by the robot prediction model, thereby obtaining an evaluation result of the robot prediction model, wherein the evaluation result is used to measure the performance of the robot prediction model.

[0008] According to a second aspect of this disclosure, an offline evaluation apparatus for a robot prediction model is provided, comprising: a processing module configured to divide a dataset of robot task execution operations into a training set and a test set, wherein the training set includes real motion vectors at corresponding time intervals, and the test set includes real motion vectors at corresponding time intervals and sensor data; a clustering module configured to perform clustering operations based on each real motion vector in the training set to determine multiple motion prototype categories; a prediction module configured to input sensor data at each time interval in the test set into the robot prediction model to perform prediction and obtain predicted motion vectors at each time interval; a conversion module configured to convert each real motion vector in the test set and each predicted motion vector output by the robot prediction model into real category labels and predicted category labels, respectively, according to the multiple motion prototype categories; and an evaluation module configured to perform error analysis based on the real motion vectors and their real category labels at each time interval in the test set and the predicted motion vectors and their predicted category labels at the corresponding time interval output by the robot prediction model, wherein the evaluation result is used to measure the performance of the robot prediction model.

[0009] According to a third aspect of this disclosure, a computer-readable storage medium is provided that stores computer instructions, which, when executed by a processor, cause the processor to perform an offline evaluation method for a robot prediction model as described in the first aspect.

[0010] According to a fourth aspect of this disclosure, an electronic device is provided, comprising: a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other via the communication bus; the memory is used to store at least one executable instruction, the executable instruction causing the processor to perform an operation corresponding to the offline evaluation method of the robot prediction model described in the first aspect.

[0011] The offline evaluation scheme for robot prediction models provided in this disclosure converts the continuous regression output of the robot prediction model into a classification task, enabling semantically interpretable offline evaluation. This scheme does not require a simulation environment or physical robot deployment; the entire evaluation process can be completed using only a static dataset, which can significantly reduce the cost of model evaluation. Attached Figure Description

[0012] To more clearly illustrate the technical solutions in the embodiments of this disclosure or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in the embodiments of this disclosure. For those skilled in the art, other drawings can be obtained based on these drawings.

[0013] Figure 1 This is a flowchart illustrating an offline evaluation method for a robot prediction model, which is an exemplary embodiment of this disclosure.

[0014] Figure 2 This is a flowchart illustrating an offline evaluation method for a robot prediction model, which is another exemplary embodiment of this disclosure.

[0015] Figure 3 This is a flowchart illustrating an offline evaluation method for a robot prediction model, which is another exemplary embodiment of this disclosure.

[0016] Figure 4 This is a structural block diagram of an offline evaluation apparatus for a robot prediction model, which is an exemplary embodiment of this disclosure.

[0017] Figure 5 A simplified structural diagram of an electronic device that is an exemplary embodiment of this disclosure. Detailed Implementation

[0018] To enable those skilled in the art to better understand the technical solutions in the embodiments of this disclosure, the technical solutions in the embodiments of this disclosure will be clearly and thoroughly described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this disclosure, and not all embodiments. Based on the embodiments of this disclosure, all other embodiments obtained by those skilled in the art should fall within the protection scope of this disclosure.

[0019] Reference is made to the accompanying drawings, which form part of the detailed description and illustrate exemplary embodiments. Furthermore, it should be understood that other embodiments may be utilized, and structural and / or logical changes may be made without departing from the scope of the claimed subject matter. It should also be noted that orientations and references (e.g., up, down, top, bottom, etc.) may be used merely to facilitate the description of features in the drawings. Therefore, the following detailed description is not to be construed in a limiting sense, and the scope of the claimed subject matter is defined only by the appended claims and their equivalents.

[0020] Numerous details are set forth in the following description. However, it will be apparent to those skilled in the art that the embodiments described herein can be practiced without these specific details. In some instances, well-known methods and apparatus are shown in block diagram form rather than in detail to avoid obscuring the embodiments described herein. Throughout this specification, references to “embodiment,” “one embodiment,” or “some embodiments” mean that a particular feature, structure, function, or characteristic described in connection with that embodiment is included in at least one embodiment herein. Therefore, the phrases “in an embodiment,” “in one embodiment,” or “some embodiments” appearing throughout this specification do not necessarily refer to the same embodiment. Furthermore, in one or more embodiments, particular features, structures, functions, or characteristics can be combined in any suitable manner. For example, a first embodiment can be combined with a second embodiment in any way that does not mutually exclude particular features, structures, functions, or characteristics associated with two embodiments.

[0021] As used in the description and appended claims, the singular forms “a” and “the” are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will also be understood that the term “and / or” as used herein refers to and includes any and all possible combinations of one or more of the associated listed items.

[0022] As described throughout this document and in the claims, a list of items connected by the terms “at least one of” or “one or more of” may mean any combination of the listed items. For example, the phrase “at least one of A, B, or C” may mean A; B; C; A and B; A and C; B and C; or A, B, and C.

[0023] With the development of artificial intelligence technology, deep learning-based intelligent decision-making models for robots are increasingly being applied to scenarios such as autonomous robot operation, human-robot collaboration, and complex task execution. Among these, the Vision-Language-Action (VLA) model can integrate visual perception information, natural language commands, and the robot's own state information, directly outputting the continuous motion control quantities required for the robot to perform tasks, such as the joint angles of the robotic arm, the pose of the end effector, and the opening and closing states of the gripper. Compared to traditional rule-based or modular control-based robot systems, the VLA model has stronger environmental understanding and task generalization capabilities, becoming an important direction for the current development of intelligent robots.

[0024] However, since VLA model outputs are typically high-dimensional continuous action sequences, their decision-making process is highly complex and dynamic. Therefore, accurately and effectively evaluating the model's decision-making ability has become a key issue limiting the further development of robotic intelligent systems. Currently, performance evaluation methods for robot prediction models, especially VLA models, mainly include task success rate-based evaluation methods and offline evaluation methods based on prediction errors.

[0025] Among these methods, task success rate-based evaluation typically deploys the model under test on a real robot platform or in a simulation environment. It executes preset tasks and tracks task completion, using the percentage of successful attempts out of the total number of tests as the model's performance metric. While this method can intuitively reflect the model's performance in real-world tasks, it relies on real robot deployment or high-precision simulation environments, requiring significant experimental costs and long testing cycles. Furthermore, real-world testing may present risks such as equipment damage and safety hazards. Additionally, this type of method usually only outputs the overall task success rate, making it difficult to further analyze the causes of errors in specific action phases or local decision-making processes, thus failing to provide effective guidance for model optimization.

[0026] Another type of approach is the offline evaluation method based on predicted action error. This typically utilizes static robot operation datasets, comparing the action vectors predicted by the model with the actual actions performed, and using metrics such as Mean Square Error (MSE), Mean Absolute Error (MAE), or Euclidean distance to evaluate model performance. This type of method does not require deployment of real robots and has advantages such as low cost and high testing efficiency. However, because robot actions are inherently continuous and semantically related, simply relying on numerical error to measure prediction results is insufficient to reflect whether the model has learned the correct operation mode. For example, different action strategies may have small numerical deviations but correspond to completely different task semantics; at the same time, global error metrics cannot pinpoint which type of action, which task stage, or specific execution part of the model exhibits decision-making bias, thus lacking effective interpretability and diagnostic capabilities.

[0027] Furthermore, some existing methods attempt to convert continuous robot actions into discrete action categories and evaluate them using metrics such as classification accuracy. However, these methods typically rely on manually predefined action categories, making it difficult to cover the complex and diverse continuous action space of robots. For different robot structures and task scenarios, the action category system needs to be redesigned, resulting in insufficient generalization ability of the evaluation methods. At the same time, existing discretization evaluation methods usually only provide overall classification results, lacking further analysis of the confusion relationships between different action categories and the sources of errors, making it difficult to directly guide model training and optimization.

[0028] In summary, existing robot prediction model evaluation methods have the following shortcomings: on the one hand, evaluation methods based on real deployments are costly and lack process-level analysis capabilities; on the other hand, evaluation methods based on offline data mainly focus on numerical errors of actions and lack semantic-level decision-making capability verification; while methods based on manually defined action categories suffer from insufficient category coverage, poor generalization, and limited diagnostic capabilities.

[0029] Based on this, the embodiments of this disclosure provide an offline evaluation scheme for robot prediction models, which can realize offline evaluation of models based on real robot data, without the need for manual pre-setting, and can take into account both continuous motion accuracy and motion semantic understanding ability, while providing fine-grained error diagnosis and model optimization guidance.

[0030] The specific implementations of the embodiments of this disclosure will be described in detail below with reference to the accompanying drawings: Figure 1 This is a flowchart illustrating an offline evaluation method for a robot prediction model, which is an exemplary embodiment of this disclosure.

[0031] In some embodiments, the robot prediction model is a vision-language-action (VLA) model. The robot is a robotic arm with multiple joints and grippers.

[0032] like Figure 1 As shown, this embodiment mainly includes: Step 102: Divide the robot's task execution dataset into a training set and a test set.

[0033] In this embodiment, the training set includes real action vectors in the corresponding time series, and the test set includes real action vectors and sensor data in the corresponding time series.

[0034] For example, a real robot operation dataset in LeRobot format can be obtained, containing multi-view images, continuous action sequences, and task labels. The dataset can be hierarchically divided into training and testing sets according to tasks. The training set is used for action clustering and extracting action prototypes, while the testing set is used for model inference and evaluation. This achieves data boundary isolation between the two datasets: the training set retains only the real action vectors corresponding to each time series, while the testing set retains both sensor observation data and real action vectors for the corresponding time series, ensuring that clustered prototypes and evaluation samples are independent of each other.

[0035] In some embodiments, the real motion vector or predicted motion vector includes at least one of the joint angle data, end effector pose data, and actuator state data collected during the robot's task execution; the test dataset also includes sensor data at various time intervals, and the sensor data may include at least one of the visual sensor data, force sensor data, and position sensor data.

[0036] In this embodiment, the robot is a robotic arm with multiple joints and grippers, and the actual motion vector is a multi-dimensional vector including the joint angles corresponding to each joint and the opening and closing degree of the grippers.

[0037] For example, a robot with a seven-DOF collaborative robotic arm (such as the Franka EmikaPanda or UR5e) equipped with a two-finger gripper has a real-time motion vector that is a 7-dimensional continuous value. The first six dimensions correspond to the rotation angles (in radians) of each joint from the base to the end effector, and the seventh dimension corresponds to the opening and closing degree of the gripper (the value range is normalized to 0~1, where 0 represents complete closure and 1 represents complete opening). During data acquisition, the motion vector at each time step is synchronously recorded with the corresponding multi-view RGB image, language command, and proprioceptive state, which together constitute a complete sample in the training or testing set.

[0038] Step 104: Perform clustering operations based on the real action vectors in the training set to determine multiple action prototype categories.

[0039] In some embodiments, the clustering algorithm used in the clustering operation may include at least one of K-means clustering, Gaussian mixture model clustering, density clustering, or segmented clustering based on motion trajectories.

[0040] In some embodiments, a clustering evaluation index can be calculated based on the distance values ​​between each real action vector to determine the number of clusters; clustering is then performed based on the number of clusters to generate multiple action prototype categories corresponding to the number of clusters, wherein each action prototype category corresponds to a cluster center. The clustering evaluation index can be determined using the elbow rule, the profile coefficient method, or error trend analysis.

[0041] For example, a clustering number determination scheme based on the elbow rule could include: setting a candidate range for the number of clusters K (e.g., 2 to 50); for each candidate K value, executing the K-Means clustering algorithm to calculate the sum of squared errors (SSE, i.e., the sum of squared distances from each sample to its cluster center) at that K value; plotting the SSE as a function of K; calculating the second difference of the curve at each point; and taking the K corresponding to the maximum second difference as the optimal number of clusters. Assuming that after calculation, the SSE curve for this dataset shows a clear inflection point at K=23 (maximum second difference), then the number of clusters K can be determined to be 23.

[0042] The physical meaning of the elbow rule is that when the value of K is small, increasing the number of clusters can significantly reduce SSE (i.e., each cluster is more compact). However, when the value of K exceeds the number of actual action patterns, the decrease in SSE from further increasing the value of K decreases sharply, the curve flattens out, and the inflection point is the optimal value of K. Through this adaptive determination method, the number of clusters suitable for the current dataset can be obtained without repeated manual trials.

[0043] Furthermore, K-Means clustering can be performed based on the determined number of clusters K to generate K action prototype categories, each corresponding to a cluster center. By mapping the action vector of each cluster center back to the joint angle trajectory and visualizing it, the semantic meaning of each action prototype can be identified: for example, cluster center 5 corresponds to the "vertical grasp" action, cluster center 8 corresponds to the "horizontal push" action, cluster center 12 corresponds to the "rotation placement" action, and cluster center 15 corresponds to the "gripper closure" action, etc.

[0044] For example, a scheme for determining the number of clusters based on the silhouette coefficient method may include: setting a candidate range for the number of clusters K (e.g., 2 to 50); for each candidate K value, executing the K-Means clustering algorithm to calculate the silhouette coefficient for that K value. The silhouette coefficient is calculated as follows: for each sample, calculate its average distance 'a' to other samples in the same category (intra-cluster dissimilarity) and its average distance 'b' to all nearest samples in other categories (inter-cluster dissimilarity); the silhouette coefficient of this sample is (ba) / max(a,b). The average of the silhouette coefficients of all samples can be taken as the overall silhouette coefficient for that K value. The K that maximizes the overall silhouette coefficient is selected as the optimal number of clusters.

[0045] The silhouette coefficient ranges from -1 to 1, with a larger value indicating better clustering (closer clusters within clusters and more separated clusters between clusters). Assuming that the silhouette coefficient reaches its maximum value (0.72) when K=23, the number of clusters can be determined to be K=23.

[0046] It should be noted that the means of determining the number of clusters are not limited to the elbow rule and the profile coefficient method mentioned in the above embodiments. Those skilled in the art can also use other feasible methods such as error change trend analysis to optimize the number of clusters. This disclosure does not limit the specific method for determining the number of clusters.

[0047] This embodiment utilizes clustering evaluation metrics such as the elbow rule or contour coefficient method. The system can automatically calculate and determine the optimal number of clusters K based on the distance between actual action vectors, eliminating the need for manual preset of action categories. This solves the shortcomings of existing technologies where action categories need to be manually preset, making it impossible to exhaustively cover the diverse continuous action space of robots; and requiring redefinition for different tasks and different robots. Users only need to provide a dataset, and the system can adaptively output a clustering scheme adapted to that dataset.

[0048] Furthermore, the optimal K value determined through clustering evaluation metrics balances the goals of "intra-cluster compactness" and "inter-cluster separation," ensuring that each action prototype category has clear semantic boundaries. A K value that is too small will merge different action patterns (e.g., merging "grasp" and "pre-grasp" into the same category), causing the confusion matrix to fail to distinguish key action types and resulting in distorted diagnostic recommendations. Conversely, a K value that is too large will produce redundant categories, with some categories having too few samples, leading to unreliable statistical metrics. The adaptive mechanism of this invention achieves an optimal balance between these two factors, ensuring the accuracy of subsequent confusion matrix analysis and diagnostic recommendations. It can also adaptively output the appropriate number of clusters based on the characteristics of the data itself, eliminating the need to redesign the action classification system for different robot platforms and task types, and exhibiting excellent cross-platform transferability and generalization.

[0049] Step 106: Input the sensor data of each time series in the test set into the robot prediction model to perform prediction and obtain the predicted action vector for each time series.

[0050] For example, taking the VLA model as an example, the model input is multimodal temporal sensor data (such as multi-view RGB image data of a current frame and at least one historical frame), and the output is a 7-dimensional continuous motion vector (such as 7 joint angle data and a gripper opening and closing degree data). The robot prediction model can reason frame by frame according to the original task time sequence and output a continuous temporal predicted motion sequence, completely reproducing the robot's continuous operation process.

[0051] In this embodiment, the predicted motion vectors output by the robot prediction model have the same dimension and physical meaning as the actual motion vectors. They can be directly used for joint-by-joint error calculation or input into the clustering model as a whole for motion prototype division, thereby ensuring that the two evaluation paths, continuous numerical regression and discrete semantic classification, share a unified data representation foundation.

[0052] Therefore, this embodiment can complete model inference solely based on sensor data from a static test set, eliminating the need to build a simulation environment, deploy a physical robotic arm for hands-on tasks, and avoid steps such as real-machine debugging, scene setup, and repeated test runs. This significantly reduces the hardware, time, and manpower costs required for evaluation, supporting rapid iterative model evaluation. Simultaneously, this solution uses only sensor observation data as model input, with real motion vectors not participating in the model inference process throughout. This completely avoids the problem of inflated evaluation results caused by truth information leakage, ensuring the objectivity and reliability of subsequent error analysis results.

[0053] It should be noted that there is no strict order constraint between steps 104 and 106. They can be executed in parallel or simultaneously, or steps 106 can be executed first and then steps 104.

[0054] Step 108: Based on multiple action prototype categories, convert each real action vector in the test set and each predicted action vector output by the robot prediction model into real category labels and predicted category labels, respectively.

[0055] In this embodiment, the actual motion vector or the predicted motion vector is a multi-dimensional vector that includes the joint angles corresponding to each joint and the opening and closing degree of the gripper.

[0056] In some embodiments, any real action vector or predicted action vector may be compared with the cluster centers of each action prototype category; the action prototype category corresponding to the cluster center closest to the real action vector or predicted action vector may be determined as any real category label or predicted category label.

[0057] In some embodiments, Euclidean distance, Mahalanobis distance, cosine similarity, and other calculation methods can be used to achieve category label assignment.

[0058] For example, taking the Euclidean distance and 23 action prototype categories as an example, each category corresponds to a 7-dimensional cluster center, denoted as C={C1,C2,...,C23}. Each cluster center is a centroid vector in 7-dimensional space. These cluster centers represent 23 different action prototypes, such as C5 corresponding to "vertical grab," C8 corresponding to "horizontal push," C12 corresponding to "rotational placement," etc.

[0059] For a true action vector g(t) = [g{t,1}, g{t,2}, ..., g{t,7}] at a given time step t, it can be compared one by one with 23 cluster centers, and the Euclidean distance is calculated to obtain 23 Euclidean distance values. The cluster center with the smallest distance (e.g., C5) is identified as the true class label corresponding to the true action vector at that time step. Similarly, for a predicted action vector at a given time step t... (t) = [ {t,1}, {t,2}, ..., Similarly, the Euclidean distance between {t,7} and the 23 cluster centers is calculated, and the action prototype category corresponding to the nearest cluster center is determined as the predicted category label for this time series.

[0060] Step 110: Perform error analysis based on the real motion vectors and their real category labels in each time series of the test set and the predicted motion vectors and their predicted category labels in the corresponding time series output by the robot prediction model to obtain the evaluation results of the robot prediction model.

[0061] In this embodiment, the evaluation results are used to measure the performance of the robot prediction model.

[0062] In some embodiments, a continuous error index can be calculated based on the real action vectors at each time series in the test set and the predicted action vectors at the corresponding time series output by the robot prediction model; a discrete classification index can be calculated based on the real category labels corresponding to the real action vectors at each time series in the test set and the predicted category labels corresponding to the predicted action vectors at the corresponding time series output by the robot prediction model; and the evaluation result of the robot prediction model can be obtained based on the discrete classification index and the continuous error index.

[0063] In some embodiments, discrete classification metrics include at least one of classification accuracy, macro average F1, weighted average F1, and action category confusion matrix; continuous error metrics include at least one of mean square error, mean absolute error, mean and standard deviation of Euclidean distance, cosine similarity, and joint-by-joint error.

[0064] Specifically, discrete classification metrics and continuous error metrics can be integrated to output hierarchical, multi-dimensional comprehensive model evaluation results. On the one hand, classification accuracy and weighted average F1 score can be combined to evaluate the model's semantic recognition ability of the overall action pattern, while mean squared error (MSE) and cosine similarity can be used to measure the model's numerical fitting accuracy of continuous action vectors. The overall performance level of the model can be determined by combining the two dimensions of metrics. On the other hand, the confusion matrix can be used to accurately locate action prototypes with low recognition accuracy and action pairs that are easily confused, clarifying the model's defects in action semantic recognition. Then, joint-by-joint error and L2 distance distribution can be used to pinpoint joints and temporal segments with concentrated action numerical deviations.

[0065] In addition, the two types of indicators can be linked to conduct cross-validation: if the classification accuracy of a certain type of action prototype is low, and the samples of this type simultaneously show a large L2 distance and single joint prediction error, then the action type can be identified as a weakness in the model performance; based on the above cross-analysis conclusions, targeted model optimization diagnostic suggestions can be generated and incorporated into the complete comprehensive evaluation results.

[0066] In summary, the offline evaluation method for robot prediction models provided in this embodiment can complete the entire evaluation process relying solely on static datasets, without the need to deploy physical robots or build simulation environments. This significantly reduces the hardware investment, time expenditure, and manpower costs required for evaluation, providing efficient support for rapid model iteration and optimization. By automatically segmenting action prototypes through clustering algorithms, the continuous regression prediction task of the VLA model is transformed into a discrete classification task. This allows the evaluation results to obtain a clear semantic interpretation of the actions using classification metrics, effectively compensating for the inherent deficiency that a single continuous error metric cannot explain the model's behavioral intent. Simultaneously, this method calculates both continuous error metrics (including MSE, MAE, L2 distance, cosine similarity, joint-by-joint error, etc.) and discrete classification metrics (including classification accuracy, macro-average F1, weighted average F1, confusion matrix, etc.), taking into account both the model's performance in terms of numerical fitting accuracy and action pattern recognition ability. This effectively avoids the one-sidedness of evaluation caused by a single metric and achieves a comprehensive and multi-dimensional process verification of the model's decision-making capabilities.

[0067] Figure 2 The offline evaluation process of a robot prediction model according to another exemplary embodiment of this disclosure is shown, which mainly includes: Step 202: Generate a diagnostic report based on the evaluation results of the robot prediction model.

[0068] In this embodiment, it can be based on Figure 1 The evaluation results output in step 110 of the embodiment are used to generate a targeted diagnostic report.

[0069] In some embodiments, a target action prototype with a classification accuracy below a preset threshold and its main confusion category can be determined from multiple action prototype categories based on an action category confusion matrix. Diagnostic recommendations are then generated based on the target action prototype and its main confusion category.

[0070] Step 204: Optimize the robot prediction model based on the diagnostic recommendations.

[0071] In some embodiments, the optimization of the robot prediction model mainly falls into two dimensions: data optimization and model structure optimization. Data-level optimization can supplement grasping action samples and balance the data distribution of different action prototypes to alleviate confusion between different action categories. Model structure optimization can modify the visual attention module to enhance the feature extraction capability of the grasping interaction area and reduce the numerical prediction bias of joint angles and gripper opening / closing degrees. In addition, fine-tuning can be used to load the original model weights, and the robot prediction model can be retrained based on the expanded and balanced dataset.

[0072] In some embodiments, after completing this step, the process can return to step 106 to re-evaluate the optimized robot prediction model on the test set and recalculate the discrete classification index, thereby constructing a complete iterative closed loop of "evaluation → diagnosis → improvement".

[0073] Specifically, the robot prediction model, after data optimization, structural adjustment, and / or fine-tuning training, can be stored as a new model checkpoint. The static test set temporal sensor data used in the previous evaluation is fully reused. The optimized model is input frame by frame for forward inference, outputting all temporal predicted action vectors. This inference process is completed entirely offline, without the need for a physical robot or simulation environment, and the temporal alignment rules remain consistent with the first round of evaluation. A clustering model pre-trained on the training set is invoked to map the predicted action vectors and real action vectors output in this round to new predicted and real category labels, respectively. The label mapping rules and cluster center parameters remain unchanged throughout, ensuring that the metrics from the two rounds can be compared horizontally. Based on the updated real and predicted label datasets, various discrete classification metrics are recalculated, including global classification accuracy, macro-average F1, and weighted average F1, and a new action prototype confusion matrix is ​​generated, fully preserving the subdivision recognition accuracy and category confusion distribution of various action prototypes. Finally, the discrete classification metrics obtained from this retest are compared horizontally with the original metrics before optimization to determine the actual effect of this model optimization.

[0074] The following is an example of model diagnostic suggestions generated from a structured comprehensive evaluation report in JSON format: { "global": { "accuracy": 0.834, "mse": 0.0035, "f1_macro": 0.82, "f1_weighted": 0.84 }, "per_prototype": { "prototype_5": { "accuracy": 0.62, "samples": 1240, "semantic_hint": "grab with both hands", "confused_with": ["prototype_8", "prototype_3"] }, "prototype_12": { "accuracy": 0.91, "samples": 980, "semantic_hint": "single-arm push", "confused_with": [] } }, "confusion_matrix": [[0, 0, ...], [...], ...], "diagnosis": "prototype_5 (two-handed grasping) has an accuracy of only 62%, mainly due to confusion with prototype_8 (pre-grasping) and prototype_3 (close to). It is recommended to increase the training samples for grasping actions and optimize the grasping point detection module."

[0075] The aforementioned comprehensive evaluation report fully lists all continuous error indicators: mean squared error (MSE) is 0.0035, mean absolute error (MAE) is 0.041, mean L2 distance of single-frame action vector is 0.124, standard deviation is 0.068, and cosine similarity of action vector is 0.958. Simultaneously, discrete classification indicators are calculated, with the model achieving an overall classification accuracy of 83.4%, a macro-average F1 score of 0.82, a weighted average F1 score of 0.84, and outputting a 23×23 action prototype confusion matrix (this embodiment classifies 23 action prototypes).

[0076] Based on the comprehensive evaluation report, the researchers clearly found that the recognition accuracy of the "two-handed grasping" action prototype was only 62%, and this prototype sample was easily confused with the "pre-grabbing" and "approaching" action prototypes. Based on the above diagnostic conclusions, the researchers supplemented the training samples corresponding to grasping actions and optimized the model's attention mechanism to focus its features on the region corresponding to the grasping operation.

[0077] In summary, this embodiment, through a re-evaluation mechanism, links the three stages of "evaluation and diagnosis," "model optimization," and "retesting and verification" into a complete closed-loop iterative process. This supports repeated model tuning and continuous correction of its defects in action recognition. The entire process reuses the same static test set, clustering mapping rules, and indicator calculation logic. There are no differences in environmental, data, or algorithm variables between the two rounds of evaluation. Therefore, changes in indicators can truly reflect the performance improvement brought about by model optimization, effectively avoiding biases caused by inconsistent evaluation conditions that could interfere with the judgment of the optimization direction. The retesting process is entirely based on the static dataset, eliminating the need to repeatedly deploy physical robots or build simulation scenarios for task testing. This significantly reduces the manpower and hardware costs associated with real-machine trials and scenario reproduction, greatly accelerating the model iteration pace. Using standardized discrete classification indicators as the basis for judging the optimization effect not only intuitively quantifies the improvement in the recognition accuracy of each action prototype but also clearly identifies the action categories that still have recognition defects, providing clear directional guidance for the next round of optimization.

[0078] Figure 3 An offline evaluation process for a robot prediction model provided by another exemplary embodiment of this disclosure is illustrated. This process can be implemented at an opportune time after the optimization and adjustment of the robot prediction model has substantially met preset requirements (e.g., when the diagnostic report generated in step 202 determines that the current model does not require further optimization).

[0079] As shown in the figure, this embodiment mainly includes: Step 302: Obtain multiple robot prediction models at different training stages as multiple model checkpoints.

[0080] In practical implementation, multiple checkpoints at different training stages can be obtained through various strategies: for example, in supervised learning training, model weights can be saved at fixed intervals (e.g., every 5 epochs) to cover the complete performance evolution trajectory from underfitting to moderate fitting to overfitting; or comparative experiments can be conducted by setting different combinations of hyperparameters such as learning rate, batch size, and number of model parameters to obtain diverse checkpoints with wide performance distribution; checkpoints can also be saved at intervals based on the number of interaction steps in the reinforcement learning fine-tuning stage to capture the gradual process of the strategy from random exploration to stable convergence; alternatively, the same configuration model can be repeatedly trained using different random seeds, or checkpoints can be collected from different fine-tuning epochs in cross-task transfer learning. The multiple checkpoints obtained cover model states with varying performance and convergence levels, providing sufficient and diverse sample support for subsequent calculation of the correlation coefficient between discrete classification accuracy and task success rate, ensuring the representativeness and credibility of the statistical analysis.

[0081] Step 304: Calculate the discrete classification accuracy of each model checkpoint on the test set.

[0082] Specifically, for each acquired model checkpoint, it can be loaded into the evaluation framework, and a complete offline inference and evaluation process can be executed on the same static test set. For each checkpoint, the model is input with sensor data (including multi-view images, language commands, and proprioceptive states) from various time series within the test set. The predicted action vector is then obtained, and a pre-trained clustering model is used to map the predicted action vector to discrete action prototype category labels. These labels are then compared frame-by-frame with the true category labels corresponding to the actual action vectors in the test set, thereby calculating the overall classification accuracy of that checkpoint across all test samples. By performing the above operations on all checkpoints, a set of discrete classification accuracy sequences corresponding to the model training stage, hyperparameter configuration, or fine-tuning level is obtained. This sequence reflects the model's action pattern recognition capability under different states, laying the data foundation for subsequent correlation analysis with task success rate.

[0083] Step 306: Calculate the correlation coefficient between the discrete classification accuracy and the task success rate of each model checkpoint.

[0084] In some embodiments, the correlation coefficient is the Spearman rank correlation coefficient.

[0085] Specifically, after obtaining the discrete classification accuracy and corresponding task success rate for each model checkpoint, these two can be paired into one-to-one data pairs, and correlation analysis can be performed using Spearman's rank correlation coefficient. This coefficient is a nonparametric statistic that does not require the data to follow a normal distribution; it calculates the correlation based solely on the rank of the variable, and can robustly assess the strength of the monotonic association between accuracy and success rate.

[0086] Step 308: If the correlation coefficient exceeds the preset threshold, the discrete classification accuracy is used as an offline proxy indicator of the task success rate, and the optimal model is selected from each robot prediction model.

[0087] Specifically, the discrete classification accuracy of each checkpoint can be sorted from low to high and assigned a rank. Simultaneously, the corresponding task success rate can be sorted and ranked in the same way. Then, the Pearson correlation coefficient between the two sets of ranks is calculated, yielding the ρ value. The value of ρ ranges from [-1, 1]. If ρ approaches +1 (e.g., ρ > 0.7) and the p-value is less than the significance level (e.g., p < 0.05), it indicates a strong positive correlation between the two. That is, the higher the discrete classification accuracy of the model checkpoint, the higher its actual task success rate. This verifies the effectiveness of discrete classification accuracy as an offline proxy indicator of task success rate, providing statistical support for subsequent model selection based solely on offline evaluation.

[0088] Conversely, if the ρ value is low or the p value is not significant, it indicates that the correlation between discrete classification accuracy and task success rate is weak, and offline accuracy cannot reliably represent real deployment performance. In this case, real machine testing should still be used as the final evaluation criterion for model performance, and offline indicators should not be used to replace actual deployment testing.

[0089] In summary, this embodiment's technical solution obtains multiple checkpoints at different performance levels during model training or optimization, calculates their offline discrete classification accuracy and real (or simulated) task success rate, and performs correlation analysis on the two, providing a statistical quantitative verification method for the effectiveness of offline evaluation metrics. When the correlation coefficient exceeds a preset threshold, it indicates a strong positive correlation between discrete classification accuracy and task success rate, thus confirming that the offline metric can reliably reflect the model's task execution capability in a real deployment environment. In this case, it can be used as an offline proxy metric for task success rate, directly used to select the optimal model from multiple candidate models without requiring expensive and time-consuming real-device deployment testing for each model. This significantly reduces the evaluation cost in the model selection and iterative optimization process, accelerating the conversion efficiency from model development to deployment. Conversely, if the correlation does not meet the standard, real-device testing is still required as the final evaluation criterion, ensuring that the reliability of model selection is not compromised by blindly relying on offline metrics.

[0090] Figure 4 This is a simplified structural diagram of an offline evaluation device for a robot prediction model according to an exemplary embodiment of the present disclosure. As shown in the figure, the offline evaluation device 400 for a robot prediction model in this embodiment mainly includes: a processing module 402, a clustering module 404, a prediction module 406, a conversion module 408, and an evaluation module 410.

[0091] The processing module 402 is configured to divide the operation dataset of the robot performing the task into a training set and a test set. The training set includes real motion vectors in the corresponding time series, and the test set includes real motion vectors and sensor data in the corresponding time series. Clustering module 404 is configured to perform clustering operations based on each real action vector in the training set to determine multiple action prototype categories; The prediction module 406 is configured to input sensor data from each time series in the test set into the robot prediction model to perform prediction and obtain the predicted action vector for each time series. The conversion module 408 is configured to convert each real motion vector in the test set and each predicted motion vector output by the robot prediction model into real category labels and predicted category labels, respectively, according to the multiple motion prototype categories. The evaluation module 410 is configured to perform error analysis based on the real motion vectors and their real category labels in each time series of the test set and the predicted motion vectors and their predicted category labels in the corresponding time series output by the robot prediction model. The evaluation results are used to measure the performance of the robot prediction model.

[0092] In some embodiments, the clustering algorithm used in the clustering operation includes at least one of K-means clustering, Gaussian mixture model clustering, density clustering, or segmented clustering based on motion trajectories.

[0093] In some embodiments, the clustering module 404 is configured to: calculate a clustering evaluation index based on the distance values ​​between each real action vector, and determine the number of clusters; perform clustering partitioning based on the number of clusters, and generate multiple action prototype categories corresponding to the number of clusters, with each action prototype category corresponding to a cluster center; wherein the clustering evaluation index is determined by the elbow rule or the profile coefficient method.

[0094] In some embodiments, the conversion module 408 is configured to: compare any real action vector or predicted action vector with the cluster centers of each action prototype category; and determine the action prototype category corresponding to the cluster center closest to the real action vector or the predicted action vector as the real category label or the predicted category label.

[0095] In some embodiments, the evaluation module 410 is configured to: calculate a continuous error index based on the real action vectors in each time series of the test set and the predicted action vectors in the corresponding time series output by the robot prediction model; calculate a discrete classification index based on the real class labels corresponding to the real action vectors in each time series of the test set and the predicted class labels corresponding to the predicted action vectors in the corresponding time series output by the robot prediction model; and obtain the evaluation result of the robot prediction model based on the discrete classification index and the continuous error index.

[0096] In some embodiments, the discrete classification metric includes at least one of classification accuracy, macro average F1, weighted average F1, and action category confusion matrix; the continuous error metric includes at least one of mean square error, mean absolute error, mean and standard deviation of Euclidean distance, cosine similarity, and joint-by-joint error.

[0097] In some embodiments, the evaluation module 410 is further configured to: determine, based on the action category confusion matrix, a target action prototype with a classification accuracy lower than a preset threshold and the main confusion category of the target action prototype from the plurality of action prototype categories; and generate diagnostic suggestions based on the target action prototype and its main confusion category.

[0098] In some embodiments, the evaluation module 410 is further configured to: optimize the robot prediction model based on the diagnostic recommendations; re-evaluate the optimized robot prediction model on the test set; and recalculate the discrete classification index.

[0099] In some embodiments, the evaluation module 410 is further configured to: acquire multiple robot prediction models at different training stages as multiple model checkpoints; calculate the discrete classification accuracy of each model checkpoint on the test set; obtain the task success rate of each model checkpoint through real robot deployment experiments or simulation environment experiments; calculate the correlation coefficient between the discrete classification accuracy and the task success rate of each model checkpoint; if the correlation coefficient exceeds a preset threshold, use the discrete classification accuracy as an offline proxy indicator of the task success rate, and select the optimal model from the multiple robot prediction models.

[0100] In some embodiments, the correlation coefficient is a Spearman rank correlation coefficient.

[0101] In some embodiments, the real motion vector or the predicted motion vector includes at least one of the joint angle data, end effector pose data, and actuator state data collected during the robot's task execution; the test dataset also includes sensor data at various time intervals, and the sensor data includes at least one of the visual sensor data, force sensor data, and position sensor data; the robot prediction model is a vision-language-motion model.

[0102] In some embodiments, the robot is a robotic arm with multiple joints and grippers; the actual motion vector or the predicted motion vector is a multi-dimensional vector including the joint angles corresponding to each joint and the opening and closing degree of the grippers.

[0103] Reference Figure 5 The diagram shows a schematic representation of an electronic device according to an embodiment of the present disclosure. The specific embodiments of the present disclosure do not limit the specific implementation of the electronic device.

[0104] like Figure 5 As shown, the electronic device in this embodiment may include: a processor 502, a communications interface 504, a memory 506, and a communications bus 508.

[0105] The processor 502, communication interface 504, and memory 506 communicate with each other via communication bus 508.

[0106] Communication interface 504 is used to communicate with other electronic devices or servers.

[0107] The processor 502 is used to execute program 510, specifically to execute the relevant steps in the above-described offline evaluation method embodiment for the robot prediction model.

[0108] Specifically, program 510 may include program code that includes computer operation instructions.

[0109] Processor 502 may be a CPU, an Application Specific Integrated Circuit (ASIC), or one or more integrated circuits configured to implement embodiments of this disclosure. The smart device may include one or more processors of the same type, such as one or more CPUs; or it may include processors of different types, such as one or more CPUs and one or more ASICs.

[0110] Memory 506 is used to store program 510. Memory 506 may include high-speed RAM memory, and may also include non-volatile memory, such as at least one disk storage device.

[0111] Program 510 may include multiple computer instructions. Specifically, program 510 may use multiple computer instructions to cause processor 502 to perform the operation corresponding to the offline evaluation method of the robot prediction model described in any of the foregoing multiple method embodiments.

[0112] The specific implementation of each step in program 510 can be found in the corresponding descriptions of the steps and modules in the above method embodiments, and has corresponding beneficial effects, which will not be repeated here. Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working process of the devices and modules described above can be referred to the corresponding process descriptions in the foregoing method embodiments, and will not be repeated here.

[0113] This disclosure also provides a computer storage medium storing a computer program thereon, which, when executed by a processor, implements the method described in any of the foregoing method embodiments. The computer storage medium includes, but is not limited to, compact disc read-only memory (CD-ROM), random access memory (RAM), floppy disk, hard disk, or magneto-optical disk.

[0114] This disclosure also provides a computer program product, including computer instructions that instruct a computing device to perform operations corresponding to the offline evaluation method for the robot prediction model described in any of the above embodiments.

[0115] It should be noted that, depending on the implementation needs, the various components / steps described in the embodiments of this disclosure can be broken down into more components / steps, or two or more components / steps or parts of the operation of components / steps can be combined into new components / steps to achieve the purpose of the embodiments of this disclosure.

[0116] The methods described above according to embodiments of this disclosure can be implemented in hardware, firmware, or as software or computer code that can be stored in a recording medium (such as a CD-ROM, RAM, floppy disk, hard disk, or magneto-optical disk), or as computer code originally stored on a remote recording medium or a non-transitory machine-readable medium and subsequently stored on a local recording medium, downloaded over a network. Thus, the methods described herein can be stored on a recording medium using a general-purpose computer, a dedicated processor, or programmable or dedicated hardware (such as an Application Specific Integrated Circuit (ASIC) or a Field Programmable Gate Array (FPGA)). It is understood that the computer, processor, microprocessor controller, or programmable hardware includes storage components (e.g., Random Access Memory (RAM), Read-Only Memory (ROM), Flash Memory, etc.) capable of storing or receiving software or computer code, implementing the methods described herein when the software or computer code is accessed and executed by the computer, processor, or hardware. Furthermore, when a general-purpose computer accesses code used to implement the methods shown herein, the execution of the code transforms the general-purpose computer into a dedicated computer for performing the methods shown herein.

[0117] Those skilled in the art will recognize that the modules and method steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the embodiments disclosed herein.

[0118] The above embodiments are only used to illustrate the embodiments of this disclosure, and are not intended to limit the embodiments of this disclosure. Those skilled in the art can make various changes and modifications without departing from the spirit and scope of the embodiments of this disclosure. Therefore, all equivalent technical solutions also fall within the scope of the embodiments of this disclosure, and the patent protection scope of the embodiments of this disclosure should be defined by the claims.

Claims

1. An offline evaluation method for a robot prediction model, comprising: The operation dataset of the robot performing the task is divided into a training set and a test set. The training set includes the real motion vectors under the corresponding time series, and the test set includes the real motion vectors and sensor data under the corresponding time series. Clustering operations are performed based on the real action vectors in the training set to determine multiple action prototype categories; The sensor data from each time series in the test set are input into the robot prediction model to perform prediction and obtain the predicted action vector for each time series. Based on the multiple action prototype categories, each real action vector in the test set and each predicted action vector output by the robot prediction model are converted into real category labels and predicted category labels, respectively. Error analysis is performed on the actual motion vectors and their actual class labels at each time step in the test set, and the predicted motion vectors and their predicted class labels at the corresponding time steps output by the robot prediction model. The evaluation results of the robot prediction model are then obtained, and these evaluation results are used to measure the performance of the robot prediction model.

2. The method according to claim 1, wherein, The clustering algorithm used in the clustering operation includes at least one of the following: K-means clustering algorithm, Gaussian mixture model clustering algorithm, density clustering algorithm, or segmented clustering algorithm based on motion trajectory.

3. The method according to claim 1, wherein, The determination of multiple action prototype categories includes: The clustering evaluation index is calculated based on the distance values ​​between each real action vector to determine the number of clusters; Clustering is performed based on the number of clusters to generate multiple action prototype categories corresponding to the number of clusters, with each action prototype category corresponding to a cluster center; The clustering evaluation index is determined by the elbow rule or the silhouette coefficient method.

4. The method according to claim 3, wherein, The step of converting each real action vector in the test set and each predicted action vector output by the robot prediction model into real category labels and predicted category labels, respectively, based on the multiple action prototype categories, includes: Compare any real action vector or predicted action vector with the cluster centers of each action prototype category; The action prototype category corresponding to the cluster center that is closest to the real action vector or the predicted action vector is determined as either the real category label or the predicted category label.

5. The method according to claim 1, wherein, The evaluation result of the robot prediction model is obtained by performing error analysis based on the real action vectors and their real class labels in each time series of the test set and the predicted action vectors and their predicted class labels in the corresponding time series output by the robot prediction model, including: Based on the real motion vectors at each time step in the test set and the predicted motion vectors at the corresponding time step output by the robot prediction model, the continuous error index is calculated. Based on the real category labels corresponding to the real action vectors in each time series of the test set and the predicted category labels corresponding to the predicted action vectors in the corresponding time series output by the robot prediction model, a discrete classification index is calculated. The evaluation results of the robot prediction model are obtained based on the discrete classification index and the continuous error index.

6. The method according to claim 5, wherein, The discrete classification metric includes at least one of classification accuracy, macro average F1, weighted average F1, and action category confusion matrix; The continuous error index includes at least one of the following: mean square error, mean absolute error, mean and standard deviation of Euclidean distance, cosine similarity, and joint-by-joint error.

7. The method according to claim 6, wherein, Also includes: Based on the action category confusion matrix, target action prototypes with classification accuracy below a preset threshold and the main confusion categories of the target action prototypes are determined from the plurality of action prototype categories; Based on the target action prototype and its main confusion categories, diagnostic suggestions are generated.

8. The method according to claim 7, wherein, Also includes: Based on the diagnostic recommendations, optimize the robot prediction model; The optimized robot prediction model is re-evaluated on the test set, and the discrete classification metric is recalculated.

9. The method according to claim 1, wherein, Also includes: Multiple robot prediction models at different training stages are obtained as multiple model checkpoints; Calculate the discrete classification accuracy of each model checkpoint on the test set; The success rate of each model checkpoint is obtained through real robot deployment tests or simulation environment tests. Calculate the correlation coefficient between the discrete classification accuracy and the task success rate at each model checkpoint; If the correlation coefficient exceeds a preset threshold, the discrete classification accuracy is used as an offline proxy indicator of the task success rate, and the optimal model is selected from the multiple robot prediction models.

10. The method according to claim 9, wherein, The correlation coefficient is the Spearman rank correlation coefficient.

11. The method according to claim 1, wherein, The actual motion vector or the predicted motion vector includes at least one of the joint angle data, end effector pose data, and actuator state data collected during the robot's task execution. The test dataset also includes sensor data at various time intervals, and the sensor data includes at least one of visual sensor data, force sensor data, and position sensor data. The robot prediction model is a vision-language-action model.

12. The method according to claim 11, wherein, The robot includes a robotic arm with multiple joints and grippers; The actual motion vector or the predicted motion vector is a multi-dimensional vector that includes the joint angles corresponding to each joint and the opening and closing degree of the gripper.

13. An offline evaluation device for a robot prediction model, comprising: The processing module is configured to divide the operation dataset of the robot performing the task into a training set and a test set. The training set includes real motion vectors in the corresponding time series, and the test set includes real motion vectors and sensor data in the corresponding time series. The clustering module is configured to perform clustering operations based on each real action vector in the training set to determine multiple action prototype categories; The prediction module is configured to input sensor data from each time series in the test set into the robot prediction model to perform prediction and obtain the predicted action vector for each time series. The conversion module is configured to convert each real action vector in the test set and each predicted action vector output by the robot prediction model into real category labels and predicted category labels, respectively, according to the multiple action prototype categories. The evaluation module is configured to perform error analysis based on the real motion vectors and their real class labels in each time series of the test set and the predicted motion vectors and their predicted class labels in the corresponding time series output by the robot prediction model. The evaluation results are used to measure the performance of the robot prediction model.

14. A computer-readable storage medium having stored thereon computer instructions, wherein, When the computer instructions are executed by the processor, the processor performs the method as described in any one of claims 1 to 12.

15. An electronic device comprising a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other via the communication bus; the memory is used to store at least one executable instruction, the executable instruction causing the processor to perform the method as described in any one of claims 1 to 12.

Images