Hand-object interaction data quality verification system

Abstract

The embodiment of the invention relates to the technical field of robots, in particular to a hand-object interaction data quality verification system. The system comprises a data preprocessing module which is configured to process collected dynamic interaction data into target interaction data when an operator wears a data glove to grab an object; the physical scene construction module is configured to construct an interaction scene of the hand model and the object model by using the target interaction data based on the simulation engine; the simulation replay module is configured to control the hand model and the object model to perform simulation replay on the target interaction data to obtain physical verification information; and the quality verification module is configured to verify the physical verification information based on multiple verification rules and determine abnormal interaction data. According to the embodiment of the invention, the interaction process is reproduced in the simulation environment, so that the physical anomaly which is difficult to perceive by naked eyes or cannot be judged by a traditional algorithm can be detected, the physical rationality of interaction data is ensured, and the reliability of related data research and application results is improved.

Description

Technical Field

[0001] This application relates to the field of robotics, and in particular to a hand-object interaction data quality verification system. Background Technology

[0002] Hand-object interaction data is acquired through data gloves, depth cameras, and other acquisition devices. It is structured data that reflects the hand's movement state and the relationship between the hand and the object (including relative position, contact timing, etc.) during the interaction between the hand and the object (such as grasping, touching, and releasing). It is the core basic data for robot grasping control scenarios.

[0003] In existing technologies, the process of verifying the quality of hand-object interaction data typically involves deploying inertial measurement units (IMUs) or optical markers on data gloves and scene objects. By capturing the spatial displacement changes of these sensors, the kinematic trajectories of the hand and the object are reconstructed. This primarily focuses on checking the data integrity and smoothness at the kinematic level. For example, existing solutions filter raw readings based on preset sensor range thresholds, eliminating obvious outliers caused by signal loss or hardware saturation. Digital filters are applied to smooth the joint angle sequences to suppress high-frequency noise.

[0004] However, the core focus of this verification mechanism is on the electrical signal characteristics and motion continuity of the data itself. Its verification logic is based on the assumption that "the data is valid if the sensor is working properly". It generally accepts the denoised motion trajectory as a valid approximation of the real motion, without considering the physical rationality of the data itself, such as the biomechanical constraints of the hand, the limits of joint movement, and the physical rationality that the hand cannot penetrate objects. Summary of the Invention

[0005] One objective of this application is to provide a hand-object interaction data quality verification system to solve the technical problem that the prior art does not consider the physical rationality of hand-object interaction data when verifying data quality.

[0006] In a first aspect, embodiments of this application provide a hand-object interaction data quality verification system, comprising: The data preprocessing module is configured to process the dynamic interactive data collected during the process of the operator wearing data gloves grasping objects into target interactive data. The data gloves are equipped with multiple sensors for collecting joint data. The physical scene construction module is configured to: construct an interactive scene of the hand model and the object model based on the target interaction data using the simulation engine; The simulation replay module is configured to control the hand model and the object model to simulate and replay the target interaction data to obtain physical verification information; The quality verification module is configured to verify the physical verification information based on multiple verification rules and identify abnormal interaction data.

[0007] Optionally, the dynamic interactive data includes glove sensor data, object pose data, and object tags. The glove sensor data includes raw data from multiple joint sensors and tactile sensor data. The data preprocessing module is further configured to: convert the raw joint sensor data into standard-format joint simulation data; correlate the three data based on the timestamps in the joint simulation data, the tactile sensor data, and the object pose data to obtain time-series data at a unified time; and calculate a transformation matrix based on the time-series data under a known pose, the transformation matrix being used to transform the coordinate system of the time-series data to the simulation world coordinate system.

[0008] Optionally, the target interaction data includes the time-series data, the transformation matrix, and the object labels. The physical scene construction module is further configured to: add joint colliders and mass attributes to each joint of the hand model based on a predefined hand URDF file, and construct the hand model; select the corresponding 3D mesh file based on the object labels, generate mesh colliders, and set physical material parameters, and construct the object model; set physical constraints on the hand model, including joint angle constraints and maximum velocity / acceleration constraints; and set contact parameters for the hand model and the object model, the contact parameters being used to implement collision detection.

[0009] Optionally, the simulation replay module is further configured to: control the hand model and the object model to perform simulation replay of the time series data, perform collision detection on each frame of the time series data, wherein the simulation replay reproduces the time series data in a simulation world; and obtain the physical verification information based on the collision detection.

[0010] Optionally, the physical verification information includes a sequence of contact points and the normal vector and / or penetration depth and / or contact force and / or torque of each contact point. Obtaining the physical verification information based on the collision detection includes: for each frame of temporal data, obtaining all sets of colliders based on the collision detection, the set of colliders including a joint collider of the hand model and a mesh collider of the object model; calculating the spatial distance between the nearest point sets of each set of colliders, the nearest point set including the point where the joint collider and the mesh collider are closest to each other in the set of colliders; determining that the joint collider and the mesh collider in the set of colliders are in contact if the spatial distance is less than a preset distance threshold; calculating the average of the spatial coordinates of the two points in the nearest point set as the spatial coordinates of the contact point; and calculating the normal vector and / or penetration depth and / or contact force and / or torque at each contact point.

[0011] Optionally, the physical verification information also includes the joint angle and velocity / acceleration of each joint of the hand model. The multiple verification rules include penetration depth verification and / or contact force verification and / or dynamic verification and / or velocity / acceleration verification and / or joint angle verification. The quality verification module is further configured to: for the physical verification information of each frame of temporal data, if the penetration depth at each contact point does not meet the penetration depth verification, the verification result is abnormal; and / or, if the contact force at each contact point does not meet the contact force verification, the verification result is determined to be abnormal; and / or, if the torque at each contact point does not meet the dynamic verification, the verification result is determined to be abnormal; and / or, if the velocity / acceleration of the joint where the contact point is located does not meet the velocity / acceleration verification, the verification result is abnormal; and / or, if the joint angle of the joint where the contact point is located does not meet the joint angle verification, the verification result is determined to be abnormal; in response to the existence of an abnormal verification result, the temporal data of this frame is determined to be the abnormal interaction data.

[0012] Optionally, the quality verification module is further configured to: in response to the presence of abnormal interaction data in the target interaction data, mark the frame containing the abnormal interaction data to obtain an abnormal problem frame, and generate quality verification information.

[0013] Optionally, it also includes a visualization output module, which is communicatively connected to the simulation replay module and the quality verification module, respectively. The visualization output module is configured to: render each frame of target interaction data during simulation replay to visualize the simulation replay; mark the abnormal content in the abnormal problem frame when rendering to the abnormal problem frame; and construct a time-series curve of key data based on the physical verification information and the quality verification information.

[0014] Optionally, it also includes a contact analysis module, which is communicatively connected to the simulation replay module. The contact analysis module is configured to: obtain contact analysis information based on the collision detection and identification of contact events and mechanical statistical analysis of the simulation replay module. The contact analysis information is used to quantify the contact parameters in the contact events.

[0015] Optionally, a report generation module is also included, which is communicatively connected to the contact analysis module, the quality verification module, and the simulation replay module, respectively. The report generation module is configured to generate a contact event sequence and a mechanical analysis report based on the physical verification information, the contact analysis information, and the quality verification information. The mechanical analysis report is used to evaluate the quality of the dynamic interaction data collected in the current time period.

[0016] The embodiments of this application can achieve the following technical effects: By constructing an automated analysis process deeply integrated with physical simulation, this application enables the verification of the physical rationality of hand-object interaction data. Going beyond traditional data range or format checks, by reproducing the real-time interaction process in a simulation environment, it can detect deep physical anomalies that are difficult to detect with the naked eye or that traditional algorithms cannot determine, ensuring the reproducibility and rationality of the data in the physical world. This application transforms subjective data quality assessment relying on human experience into an automated analysis process based on clear physical rules and thresholds, improving verification efficiency and objectivity, ensuring the repeatability and comparability of verification results, and is suitable for batch quality screening of large-scale datasets. This application can effectively filter out physically unreliable or contradictory noise data, providing a high-quality, high-confidence data foundation for subsequent data applications such as model training, simulation analysis, and algorithm development, thereby improving the reliability of related research and application results. Attached Figure Description

[0017] To more clearly illustrate the technical solutions of the embodiments of this application, the drawings used in the description of the embodiments of this application will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0018] Figure 1 A schematic diagram of the structure of a hand-object interaction data quality verification system provided in this application embodiment; Figure 2 A physical scene diagram illustrating model interaction provided in an embodiment of this application; Figure 3 This application provides a schematic diagram of a physical scenario in which a model comes into contact. Figure 4This is a schematic diagram of the structure of a hand-object interaction data quality verification system provided in another embodiment of this application. Detailed Implementation

[0019] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application. All other embodiments obtained by those skilled in the art based on the embodiments in this application without inventive effort are within the scope of protection of this application.

[0020] It should be noted that, unless there is a conflict, the various features in the embodiments of this application can be combined with each other, all of which are within the protection scope of this application. Furthermore, although functional modules are divided in the device schematic diagram and a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in a different order than the module division in the device or the order in the flowchart. Moreover, the terms "first," "second," and "third" used in this application do not limit the data or execution order, but only distinguish identical or similar items with essentially the same function and effect.

[0021] Hand-object interaction data is multimodal temporal data acquired in time and space during the physical interaction between a human hand and a target object through a multi-sensor fusion data acquisition system. The data acquisition system integrates data gloves worn by the data acquisition personnel (such as those with built-in bending sensors, tactile sensors, and inertial measurement units to capture joint angles, contact forces, and postures) and external object tracking systems (such as optical motion capture cameras or radar to acquire the object's six degrees of freedom pose). It records and fuses the complete dynamic relationship between human hand operations and object response movements in real time, fully representing the complete interactive dynamic process from contact, grasping, manipulation to release.

[0022] The data acquisition system includes a hand data acquisition system and an object tracking system. The hand data acquisition system includes a high-precision data glove, in which each finger joint and key part of the palm integrates a bending sensor, a tactile sensor, and an inertial measurement unit (IMU). The hand data acquisition system is responsible for outputting in real time the bending angles, contact forces, and raw data (three-axis acceleration and three-axis angular velocity) of the 21 joints of the hand, and can calculate the overall spatial posture of the hand and the motion trajectory of each joint through sensor fusion algorithms.

[0023] The object tracking system deploys multiple infrared optical motion capture cameras around the data acquisition compartment, constructing a measurement field covering the entire interactive space. A set of reflective markers with known three-dimensional geometric relationships are firmly attached to the surface of the target object. The object tracking system can capture images of the reflective markers through the cameras and calculate the target object's six-degree-of-freedom pose in three-dimensional space in real time, namely its three-dimensional position and three-dimensional rotational attitude. The data acquisition system also has a master synchronization controller that sends unified hardware trigger pulse signals to the data glove's built-in control module and the object tracking system, forcing all data streams to start based on the same physical clock and be timestamped. The data from both is transmitted in real time to the central processing server of the data acquisition system via a wired or low-latency wireless network.

[0024] To enable quality verification of hand-object interaction data obtained from a data acquisition system based on physical plausibility, this application provides a hand-object interaction data quality verification system. Please refer to... Figure 1 , Figure 1 This is a schematic diagram of the structure of a hand-object interaction data quality verification system provided in an embodiment of this application. The hand-object interaction data quality verification system 100 provided in this embodiment of the application includes a data preprocessing module 101, a physical scene construction module 102, a simulation replay module 103, and a quality verification module 104.

[0025] The data preprocessing module 101 is configured to process the dynamic interactive data collected during the process of the operator wearing data gloves grasping objects into target interactive data. The data gloves are equipped with multiple sensors that collect joint data.

[0026] Specifically, the dynamic interactive data includes glove sensor data, object pose data, and object labels. The glove sensor data includes raw data from multiple joint sensors and tactile sensor data. The data preprocessing module 101 converts the raw joint sensor data into standard-format joint simulation data. Specifically, the data preprocessing module 101 receives the raw joint sensor data collected by the data glove, typically in the format {sensor_id, raw_value, timestamp}. Based on a pre-loaded sensor configuration mapping table, the data preprocessing module 101 identifies and classifies the sensor_id into specific hand joints (joint_id) and distinguishes the sensor type (e.g., bending sensor, inertial measurement unit).

[0027] The data preprocessing module 101 applies the corresponding calibration model to convert physical quantities according to the sensor type. For bending sensors, a linear or nonlinear calibration curve is used to convert the raw voltage or count value (raw_value) into a precise joint angle. The conversion formula is: θ = f(raw_value), where the function f is determined by the calibration process. For example, if a bending sensor is calibrated and its output value range [0, 1] linearly corresponds to the physiological range of joint motion [θ_min, θ_max], then the conversion formula is: θ = raw_value * (θ_max - θ_min) + θ_min. For the raw data (acceleration, angular velocity, magnetic field strength) of the inertial measurement unit, the data preprocessing module 101 calls the sensor fusion algorithm to solve it into the quaternion posture of the corresponding hand segment, representing its rotation in three-dimensional space.

[0028] The converted joint angles and raw IMU data are filtered in real time. A low-pass digital filter is typically used to suppress high-frequency components introduced by hand tremors or sensor electronic noise, ensuring the smoothness and physiological plausibility of the motion curve. The data preprocessing module 101 encapsulates the converted data into a unified standard format required by the simulation replay module 103. The data output format for each joint at each time point is: {joint_id, angle / position, quaternion, timestamp}.

[0029] Where: joint_id is the joint identifier. angle / position is the joint's rotation angle or the 3D coordinates of a specific reference point. quaternion is the spatial rotational pose (represented by a quaternion) of the joint or the corresponding hand region. timestamp is a high-precision timestamp.

[0030] The data preprocessing module 101 correlates the joint simulation data and the object pose data based on the timestamps in the two data sets to obtain time-series data at a unified time. Specifically, the data preprocessing module 101 synchronously reads the hardware synchronization timestamps carried in the joint simulation data, tactile sensor data, and object pose data. Based on these timestamps, the data preprocessing module 101 loads the three types of data streams onto a unified time axis. To handle possible minor differences in data acquisition frequency, the data preprocessing module 101 uses linear interpolation or spline interpolation algorithms to resample the three types of data onto a common and fixed time series. This ensures that each processed time point contains a strictly corresponding hand joint state and the object's six-degree-of-freedom pose, generating time-series data at a unified time.

[0031] The data preprocessing module 101 calculates a transformation matrix based on the time-series data with known poses. This transformation matrix is ​​used to transform the coordinate system of the time-series data to the simulation world coordinate system. Specifically, to transform the time-series data to the simulation world coordinate system, the data preprocessing module 101 uses a segment of time-series data with known poses during the acquisition process, typically a static data segment during system initialization where the hand and object are in a predetermined calibration posture. From this static data segment, the poses of key hand reference points (such as the palm) and the target object are extracted. Then, by calculating the transformation from the hand key reference point coordinate system to the object coordinate system, and combining this with the known transformation relationship from the object coordinate system to the simulation world coordinate system, a transformation matrix is ​​obtained from the sensor coordinate system of the data glove to the simulation world coordinate system. The data preprocessing module 101 applies this transformation matrix to the hand pose data in all time-series data, completing the spatial alignment and fusion of all data in the simulation world coordinate system.

[0032] The data preprocessing module 101 combines time-series data, transformation matrix, and object labels to obtain target interactive data.

[0033] The physical scene construction module 102 is configured to: construct an interactive scene of hand models and object models based on the simulation engine and using target interaction data. Specifically, the physical scene construction module 102, based on the simulation engine, uses preprocessed target interaction data (including time series data, transformation matrices, and object labels) to construct hand models and object models with physical attributes in the simulation environment, and configures their interaction rules.

[0034] Please see Figure 2 , Figure 2This is a schematic diagram of a physical scene for model interaction provided in an embodiment of this application. The physical scene construction module 102 adds joint colliders and mass attributes to each joint of the hand model 21 based on a predefined hand URDF (Unified Robot Description Format) file, thus constructing the hand model 21. URDF is a standardized XML format file in the field of robotics. Its core function is to structurally describe the hardware entity attributes and motion constraints of a robot, providing unified model data support for simulation, visualization, and motion control. Specifically, the physical scene construction module 102 constructs the hand model 21 in the USD (Universal Scene Description) scene of the IsaacSim simulation engine based on the predefined hand URDF file. Using IsaacSim's URDF import tool, the hand URDF file is instantiated into a hinge body containing 21 degrees of freedom. Subsequently, joint colliders are added to each finger joint link, typically using simplified geometry such as capsules or cylinders for approximation. By editing the physical properties of the corresponding hand model 21 in the USD scene, mass attributes are configured for each finger joint link, including mass and inertia tensors, the values ​​of which are determined based on biomechanical measurement data or CAD model calculations. For example, the mass of each finger joint is set to approximately 0.02 kg.

[0035] The physics scene construction module 102 selects the corresponding 3D mesh file based on the object label, generates a mesh collider, and sets physical material parameters to construct the object model 22. Specifically, the physics scene construction module 102 retrieves the corresponding 3D mesh file (such as USD or OBJ format) from the resource library based on the object label (e.g., cup_001) in the target interaction data. A rigid body is created in the USD scene, and the 3D mesh file is attached as its visual and collision geometry. To improve the efficiency of physics simulation, IsaacSim's collision mesh processing tool is called to automatically generate simplified mesh colliders from complex meshes. By configuring the physical material of the rigid body, its friction coefficient (e.g., about 0.4-0.6 for plastic surfaces, about 0.8-1.2 for rubber), elastic coefficient, and density are set. The mass is automatically calculated based on the material density (e.g., setting the density of plastic to about 700-1400 kg / m³) and the object volume, or the mass can be directly specified.

[0036] The physics scene construction module 102 sets physical constraints on the hand model 21, including joint angle constraints and maximum velocity / acceleration constraints. Specifically, the physics scene construction module 102 sets physical constraints on the hinges of the hand model 21. By configuring the driving attributes and limit parameters of each rotational joint, joint angle constraints are achieved. For example, the range of motion of the interphalangeal joints is limited to [-10°, 90°], and the range of motion of the metacarpophalangeal joints is limited to [-30°, 90°]. At the same time, the maximum velocity (e.g., 10 rad / s) and the maximum acceleration (e.g., 50 rad / s²) are set to simulate the physiological limits of the human hand.

[0037] The physics scene construction module 102 sets contact parameters for the hand model 21 and the object model 22, which are used to implement collision detection. Specifically, in the physics scene settings of IsaacSim, continuous collision detection is enabled. The contact stiffness (e.g., 10000 N / m) and contact damping (e.g., 100 N·s / m) of the hand model 21 and the object model 22 are set to define the characteristics of the objects resisting penetration and absorbing energy upon contact, thus finely controlling the collision response behavior and ensuring the stability and realism of the interaction.

[0038] It should be noted that the simulation replay module 103 binds the preprocessed time-series data (such as hand joint angles and object poses) to two models in the simulation scene. During simulation, the simulation replay module 103 drives the hand joint controller frame by frame according to the timestamp, so that it reproduces the recorded motion trajectory and synchronously sets the corresponding pose of the object, reproducing and further studying the dynamic interaction process between hand and object in the simulation engine.

[0039] The simulation replay module 103 is configured to control the hand model and the object model to simulate and replay the target interaction data to obtain physical verification information.

[0040] Specifically, the simulation replay module 103 controls the hand model and object model to perform simulation replay of the time series data, and performs collision detection on the time series data of each frame. The simulation replay reproduces the time series data in the simulation world and obtains physical verification information based on the collision detection.

[0041] Please see Figure 3 , Figure 3This is a schematic diagram of a physical scene where models come into contact, provided in an embodiment of this application. Specifically, the simulation replay module 103 loads the constructed physical scene and binds the temporal data (hand joint angle sequence, object pose sequence) in the target interaction data with the corresponding models in the scene. The articulated body of the hand model 21 is driven by the hand joint controller, and the rigid body of the object model 22 is synchronized by directly setting its global pose. During simulation, the simulation replay module 103 updates the scene state frame by frame according to the timestamp sequence. In each simulation frame (corresponding to a frame of temporal data): based on the current frame data, the target angles of each joint of the hand and the target pose of the object are set, the IsaacSim simulation engine is called to perform simulation calculations, the motion and contact between the hand model 21 and the object model 22 are calculated, and all colliders in the current frame state are traversed and queried through the collision detection API provided by IsaacSim to obtain the set of all possible contact colliders. In some embodiments, when a contact force is detected in the tactile sensor data of the current frame, the system will call the collision detection API provided by IsaacSim to traverse and query all colliders in the current frame state to obtain a set of all possible contact colliders, thereby reducing unnecessary computational overhead and improving computational efficiency and real-time performance.

[0042] Each collider set includes a joint collider from the hand model 21 and a mesh collider from the object model 22. For each collider set retrieved, the engine function is called to calculate the nearest point set (p_hand, p_object) and the Euclidean distance d between the two points. The nearest point set includes the point in the collider set where the joint collider and the mesh collider are closest to each other. If d is less than a preset distance threshold ε, it is determined that the collider set has made contact in the current frame. In this embodiment, the distance threshold ε is set to 1 mm. The distance threshold ε is usually set to 1 mm or less to handle cases where there is visual contact but no geometric penetration. The arithmetic mean of the coordinates of the nearest points of the contacting collider set is taken as the spatial coordinates of the contact point 31 in the simulation world coordinate system: p_contact = (p_hand + p_object) / 2. The simulation replay module 103 calculates the normal vector and / or penetration depth and / or contact force and / or torque at each contact point 31.

[0043] It should be noted that, for each frame, the simulation replay module 103 performs the above calculations on all detected contact points. The simulation replay module 103 encapsulates the spatial coordinates p_contact, normal vector n, penetration depth d_penetration, contact force F_contact, and torque τ_contact of each contact point into physical verification information. The physical verification information is transmitted in real time to the quality verification module 104 at a sampling frequency (e.g., 60Hz) that is completely consistent with the timing data for quality analysis and physical consistency verification.

[0044] The quality verification module 104 is configured to verify physical verification information based on multiple verification rules and identify abnormal interaction data.

[0045] Specifically, the physical verification information also includes the joint angles and velocities / accelerations of each joint of the hand model, and the multiple verification rules include penetration depth verification and / or contact force verification and / or dynamics verification and / or velocity / acceleration verification and / or joint angle verification.

[0046] The quality verification module 104 performs quality verification on the physical verification information of each frame of timing data. If the penetration depth at each contact point does not meet the penetration depth verification, the verification result is abnormal; and / or, if the contact force at each contact point does not meet the contact force verification, the verification result is determined to be abnormal; and / or, if the torque at each contact point does not meet the dynamic verification, the verification result is determined to be abnormal; and / or, if the velocity / acceleration of the joint where the contact point is located does not meet the velocity / acceleration verification, the verification result is abnormal; and / or, if the joint angle of the joint where the contact point is located does not meet the joint angle verification, the verification result is determined to be abnormal; in response to the existence of an abnormal verification result, the timing data of this frame is determined to be abnormal interactive data.

[0047] Specifically, for the physical verification information of the timing data of each frame, the quality verification module 104 performs the following five verifications in parallel. If any verification fails, the timing data of this frame is determined to be abnormal interaction data.

[0048] In the penetration depth verification, if the penetration depth dt at any contact point exceeds the maximum allowable threshold Dmax, the verification result is determined to be abnormal and marked as "physical penetration anomaly". The maximum allowable threshold Dmax is set to 5mm. It is understandable that excessive penetration depth violates the fundamental physical principle that rigid bodies are impenetrable.

[0049] In contact force verification, if contact is detected but the corresponding contact force F_t is lower than the minimum contact force threshold F_min, or if no contact is detected but contact force exists, the verification result is determined to be abnormal and marked "contact force abnormal". It should be noted that the time-series data also includes tactile sensor data, which includes contact forces collected by multiple tactile sensors configured on the data glove. In contact force verification, for any frame of time-series data, if any tactile sensor in the tactile sensor data detects contact force and simultaneously detects contact at the corresponding part of the tactile sensor, but the contact force is less than the minimum contact force threshold F_min, the verification result is determined to be abnormal and marked "contact force abnormal"; or, if any tactile sensor in the tactile sensor data detects contact force but does not simultaneously detect contact at the corresponding part of the tactile sensor, the verification result is determined to be abnormal and marked "contact force abnormal". It is understandable that in physical simulation, contact is a necessary condition for the generation of contact force, and measurable contact force necessarily results in contact.

[0050] In dynamics verification, the angular acceleration and required net torque are estimated based on the changes in angular velocity and moment of inertia of an object across consecutive frames. If the vector sum of the applied torque τ_t contradicts the estimated required net torque in direction or magnitude (e.g., a large torque exists but the object's angular velocity remains unchanged, or no torque exists but the object rotates), the verification result is anomaly and marked as "dynamics inconsistency." It is understandable that the dynamics verification rules directly verify the rotational manifestation of Newton's second law (∑τ=Iα). Inconsistency indicates that the time-series data violates Newton's second law.

[0051] In velocity / acceleration verification, if the instantaneous angular velocity v_t of any joint exceeds the maximum physiological angular velocity v_max, or the instantaneous angular acceleration a_t exceeds the maximum physiological angular acceleration a_max, the verification result is determined to be abnormal and marked as "kinematic abnormality". In this embodiment, the maximum physiological angular velocity is set to 10 rad / s, and the maximum physiological angular acceleration is set to 50 rad / s². It is understood that the movement capabilities of each joint in the human hand are constrained by the biomechanical characteristics of muscles, tendons, etc., and there are clear upper limits for velocity and acceleration. Data exceeding these limits lacks biological authenticity.

[0052] In joint angle verification, if the angle θ_t of any joint exceeds its preset physiological range [θ_min, θ_max] (e.g., the angle range of the metacarpophalangeal joint is approximately -30° to 90°), the verification result is determined to be abnormal and marked as "joint angle out of bounds abnormality". It is understood that the range of motion of human hand joints is fixed due to the limitations of skeletal structure and ligaments. Angle data exceeding the corresponding angle range is physically impossible. In this embodiment, for each frame of temporal data, if any verification result is abnormal, the temporal data of that frame is determined to be abnormal interactive data.

[0053] This application embodiment achieves physical plausibility verification of hand-object interaction data by constructing an automated analysis process deeply integrated with physical simulation. Going beyond traditional data range or format checks, by reproducing the real-time interaction process in a simulation environment, it can detect deep physical anomalies that are difficult to detect with the naked eye or that traditional algorithms cannot determine, ensuring the reproducibility and plausibility of the data in the physical world. This application embodiment transforms subjective data quality assessment relying on human experience into an automated analysis process based on clear physical rules and thresholds, improving verification efficiency and objectivity, ensuring the repeatability and comparability of verification results, and is suitable for batch quality screening of large-scale datasets. This application embodiment can effectively filter out physically unreliable or contradictory noise data, providing a high-quality, high-confidence data foundation for subsequent data applications such as model training, simulation analysis, and algorithm development, thereby improving the reliability of related research and application results.

[0054] In some embodiments, the dynamic interactive data includes glove sensor data, object pose data, and object tags. The glove sensor data includes raw data from multiple joint sensors and tactile sensor data. The data preprocessing module 101 is further configured to: The raw data from the joint sensors is converted into joint simulation data in a standard format. Specifically, the data preprocessing module 101 receives the raw data from the joint sensors collected by the data glove. Its typical format is {sensor_id, raw_value, timestamp}. The data preprocessing module 101 identifies and classifies the sensor_id into a specific hand joint (joint_id) according to the preloaded sensor configuration mapping table, and distinguishes the sensor type (such as bending sensor, inertial measurement unit).

[0055] The data preprocessing module 101 applies the corresponding calibration model to perform physical quantity conversion according to the sensor type. For bending sensors, a linear or nonlinear (e.g., polynomial) calibration curve is used to convert the raw voltage or count value (raw_value) into a precise joint angle. The conversion formula is: θ = f(raw_value), where the function f is determined by the calibration process. For example, if a bending sensor is calibrated so that its output value range [0,1] linearly corresponds to the physiological range of joint movement [θ_min, θ_max], then the conversion formula is: θ = raw_value * (θ_max - θ_min) + θ_min. For the raw data (acceleration, angular velocity, magnetic field strength) of the inertial measurement unit, the data preprocessing module 101 calls the sensor fusion algorithm to solve it into the quaternion posture of the corresponding hand segment, representing its rotation in three-dimensional space.

[0056] The converted joint angles and raw IMU data are filtered in real time. A low-pass digital filter is typically used to suppress high-frequency components introduced by hand tremors or sensor electronic noise, ensuring the smoothness and physiological plausibility of the motion curve. The data preprocessing module 101 encapsulates the converted data into a unified standard format required by the simulation replay module 103. The data output format for each joint at each time point is: {joint_id, angle / position, quaternion, timestamp}.

[0057] Where: joint_id is the joint identifier. angle / position is the joint's rotation angle or the 3D coordinates of a specific reference point. quaternion is the spatial rotational pose (represented by a quaternion) of the joint or the corresponding hand region. timestamp is a high-precision timestamp.

[0058] The data preprocessing module 101 correlates the joint simulation data, tactile sensor data, and object pose data based on the timestamps in these three datasets to obtain time-series data at a unified time. Specifically, the data preprocessing module 101 synchronously reads the hardware synchronization timestamps carried in the joint simulation data and object pose data. Based on these timestamps, the data preprocessing module 101 loads the three types of data streams onto a unified timeline. To handle potential minor differences in data acquisition frequency, the data preprocessing module 101 uses linear interpolation or spline interpolation algorithms to resample the three types of data onto a common and fixed time series. This ensures that each processed time point contains a strictly corresponding hand joint state and the object's six-degree-of-freedom pose, generating time-series data at a unified time.

[0059] The data preprocessing module 101 calculates a transformation matrix based on the time-series data in a known pose. This transformation matrix is ​​used to transform the coordinate system of the time-series data to the simulation world coordinate system. Specifically, to transform the time-series data to the simulation world coordinate system, the data preprocessing module 101 uses a segment of time-series data in a known pose during the acquisition process, typically a static data segment during system initialization where the hand and object are in a predetermined calibration posture. The poses of the key reference points of the hand and the target object are extracted from this static data segment. Then, by calculating the transformation from the hand reference point coordinate system to the object coordinate system, and combining this with the known transformation relationship from the object coordinate system to the world coordinate system, a transformation matrix is ​​obtained from the sensor coordinate system of the data glove to the simulation world coordinate system. The data preprocessing module 101 applies this transformation matrix to the hand pose data in all time-series data, completing the spatial alignment and fusion of all data in the simulation world coordinate system.

[0060] In some embodiments, the target interaction data includes time-series data, transformation matrix, and object labels. The physical scene construction module 102 is further configured to: add joint colliders and mass attributes to each joint of the hand model based on a predefined hand URDF file to construct the hand model; select the corresponding 3D mesh file based on the object labels, generate mesh colliders, and set physical material parameters to construct the object model; set physical constraints on the hand model, including joint angle constraints and maximum velocity / acceleration constraints; and set contact parameters for the hand model and the object model, which are used to implement collision detection.

[0061] The physics scene construction module 102 adds joint colliders and mass attributes to each joint of the hand model based on a predefined hand URDF file, thus constructing the hand model. Specifically, the physics scene construction module 102 constructs the hand model in the USD (Universal Scene Description) scene of the IsaacSim simulation engine based on the predefined hand URDF file. Using IsaacSim's URDF import tool, the hand URDF file is instantiated as a hinge with 21 degrees of freedom. Subsequently, joint colliders are added to each finger joint, typically approximated using simplified geometry such as capsules or cylinders. By editing the physical properties of the corresponding hand model in the USD scene, mass attributes are configured for each finger joint, including mass and inertia tensors, the values ​​of which are determined based on biomechanical measurement data or CAD model calculations. For example, the mass of each finger joint is set to approximately 0.02 kg.

[0062] The physics scene construction module 102 selects the corresponding 3D mesh file based on the object label, generates a mesh collider, and sets physical material parameters to construct the object model. Specifically, the physics scene construction module 102 retrieves the corresponding 3D mesh file (such as USD or OBJ format) from the resource library based on the object label (e.g., cup_001) in the target interaction data. A rigid body is created in the USD scene, and this 3D mesh file is attached as its visual and collision geometry. To improve the efficiency of physics simulation, IsaacSim's collision mesh processing tool is called to automatically generate simplified mesh colliders from complex meshes. By configuring the physical material of this rigid body, its friction coefficient (e.g., approximately 0.4-0.6 for plastic surfaces, approximately 0.8-1.2 for rubber), elastic coefficient, and density are set. The mass is automatically calculated based on the material density (e.g., setting the density of plastic to approximately 700-1400 kg / m³) and the object volume, or the mass can be directly specified.

[0063] The physics scene construction module 102 sets physical constraints on the hand model, including joint angle constraints and maximum velocity / acceleration constraints. Specifically, the physics scene construction module 102 sets physical constraints on the hinges of the hand model. By configuring the driving attributes and limit parameters of each rotational joint, joint angle constraints are achieved. For example, the range of motion of the interphalangeal joints is limited to [-10°, 90°], and the range of motion of the metacarpophalangeal joints is limited to [-30°, 90°]. At the same time, maximum velocity (e.g., 10 rad / s) and maximum acceleration (e.g., 50 rad / s²) are set to simulate the physiological limits of the human hand.

[0064] The physics scene construction module 102 sets contact parameters for the hand model and object model, which are used to implement collision detection. Specifically, in the physics scene settings of IsaacSim, continuous collision detection is enabled. The contact stiffness (e.g., 10000 N / m) and contact damping (e.g., 100 N·s / m) of the hand model and object model are set to define the characteristics of the objects resisting penetration and absorbing energy upon contact, finely controlling the collision response behavior and ensuring the stability and realism of the interaction.

[0065] In some embodiments, the simulation replay module 103 is further configured to: control the hand model and the object model to perform simulation replay of the time series data, perform collision detection on each frame of the time series data, and reproduce the time series data in the simulation world; and obtain physical verification information based on the collision detection.

[0066] Specifically, the simulation replay module 103 loads the constructed physical scene and binds the temporal data (hand joint angle sequence, object pose sequence) in the target interaction data to the corresponding models in the scene. The articulated body of the hand model is driven by the hand joint controller, and the rigid body of the object model is synchronized by directly setting its global pose. During simulation runtime, the simulation replay module 103 updates the scene state frame by frame according to the timestamp sequence, performs collision detection, and obtains the corresponding physical verification information.

[0067] In some embodiments, the physical verification information includes a sequence of contact points and the normal vector and / or penetration depth and / or contact force and / or torque of each contact point. Obtaining physical verification information based on collision detection includes: for each frame of temporal data, obtaining all collider sets based on collision detection, the collider set including joint colliders of a hand model and mesh colliders of an object model; calculating the spatial distance between the nearest point sets of each collider set, the nearest point set including the points of the joint colliders and mesh colliders in the collider set that are closest to each other; determining that the joint colliders and mesh colliders in the collider set are in contact in response to the spatial distance being less than a preset distance threshold; calculating the average of the spatial coordinates of the two points in the nearest point set as the spatial coordinates of the contact point; and calculating the normal vector and / or penetration depth and / or contact force and / or torque at each contact point.

[0068] Specifically, in each simulation frame (corresponding to a frame of time-series data): the simulation replay module 103 sets the target angles of each joint of the hand and the target pose of the object according to the current frame data, calls the IsaacSim simulation engine to perform simulation calculations, solves the motion and contact between the hand model and the object model, and uses the collision detection API provided by IsaacSim to traverse and query all collision bodies in the current frame state to obtain a set of all collision bodies that may come into contact.

[0069] Each collider set includes a joint collider from the hand model and a mesh collider from the object model. For each collider set retrieved, the engine function is called to calculate the nearest point set (p_hand, p_object) and the Euclidean distance d between the two points. The nearest point set includes the points in the collider set where the joint collider and the mesh collider are closest to each other. If d is less than a preset distance threshold ε, it is determined that the collider sets are in contact in the current frame. In this embodiment, the distance threshold ε is set to 1 mm. The distance threshold ε is usually set to 1 mm or less to handle cases where there is visual contact but no geometric penetration. The arithmetic mean of the coordinates of the nearest points of the contacting collider sets is taken as the spatial coordinates of the contact point in the simulation world coordinate system: p_contact = (p_hand + p_object) / 2.

[0070] The simulation replay module 103 calculates the normal vector and / or penetration depth and / or contact force and / or torque at each contact point. Specifically, when calculating the normal vector at the contact point, the simulation replay module 103 first determines the direction of the contact surface, queries the local geometry information of the object model at the nearest point p_object, and calculates the contact normal vector n_object of its surface. The direction of the contact normal vector is defined as pointing from the object surface to the joint collider, and the unit normal vector is n = normalize(n_object).

[0071] The simulation replay module 103 calculates the penetration depth at the contact point. Penetration depth characterizes the degree of mutual intrusion between colliders. Its calculation formula is: d_penetration = max(0, -( (p_hand - p_object) · n_object)). Here, (p_hand - p_object) is the vector from the nearest point of the mesh collider in the object model to the nearest point of the joint collider in the hand model. This vector is then multiplied by the contact normal vector n_object and negative to obtain the intrusion distance along the normal direction. The max function ensures that the depth is non-negative; the penetration depth is zero when the two colliders have just separated.

[0072] The simulation replay module 103 calculates the contact force at the contact point. The simulation replay module 103 uses the Penalty algorithm to approximate the calculation of the contact force. The calculation formula is: F_contact = k_contact * d_penetration * n. Where k_contact is a preset contact stiffness (e.g., 10000 N / m), and n is the unit normal vector of the contact point. This contact force is the normal force perpendicular to the contact surface. In some embodiments, if contact is detected, and the contact force calculated by the simulation replay module 103 at the contact point is within the error tolerance range of the contact force detected by the tactile sensor corresponding to the contact point, the system will use the contact force detected by the tactile sensor corresponding to the contact point as physical verification information.

[0073] The simulation replay module 103 calculates the torque at the contact point. The formula is: τ_contact = r_contact × F_contact. Here, r_contact is the vector pointing from the object's center of mass to the contact point p_contact, F_contact is the contact force, and × represents the vector cross product operation.

[0074] In some embodiments, the physical verification information also includes the joint angle and velocity / acceleration of each joint of the hand model. Multiple verification rules include penetration depth verification and / or contact force verification and / or dynamic verification and / or velocity / acceleration verification and / or joint angle verification. The quality verification module is further configured to: for each frame of temporal data, if the penetration depth at each contact point does not meet the penetration depth verification, the verification result is abnormal; and / or, if the contact force at each contact point does not meet the contact force verification, the verification result is determined to be abnormal; and / or, if the torque at each contact point does not meet the dynamic verification, the verification result is determined to be abnormal; and / or, if the velocity / acceleration of the joint where the contact point is located does not meet the velocity / acceleration verification, the verification result is abnormal; and / or, if the joint angle of the joint where the contact point is located does not meet the joint angle verification, the verification result is determined to be abnormal; in response to the existence of an abnormal verification result, the temporal data of this frame is determined to be abnormal interactive data.

[0075] Specifically, for the physical verification information of the timing data of each frame, the quality verification module 104 performs the following five verifications in parallel. If any verification fails, the timing data of this frame is determined to be abnormal interaction data.

[0076] In the penetration depth verification, if the penetration depth dt at any contact point exceeds the maximum allowable threshold Dmax, the verification result is determined to be abnormal and marked as "physical penetration anomaly". The maximum allowable threshold Dmax is set to 5mm. It is understandable that excessive penetration depth violates the fundamental physical principle that rigid bodies are impenetrable.

[0077] In contact force verification, if contact is detected but the corresponding contact force F_t is lower than the minimum contact force threshold F_min, or if no contact is detected but contact force exists, the verification result is determined to be abnormal and marked "contact force abnormal". It should be noted that the time-series data also includes tactile sensor data, which includes contact forces collected by multiple tactile sensors configured on the data glove. In contact force verification, for any frame of time-series data, if any tactile sensor in the tactile sensor data detects contact force and simultaneously detects contact at the corresponding part of the tactile sensor, but the contact force is less than the minimum contact force threshold F_min, the verification result is determined to be abnormal and marked "contact force abnormal"; or, if any tactile sensor in the tactile sensor data detects contact force but does not simultaneously detect contact at the corresponding part of the tactile sensor, the verification result is determined to be abnormal and marked "contact force abnormal". It is understandable that in physical simulation, contact is a necessary condition for the generation of contact force, and measurable contact force necessarily results in contact.

[0078] In dynamics verification, the angular acceleration and required net torque are estimated based on the changes in angular velocity and moment of inertia of an object across consecutive frames. If the vector sum of the applied torque τ_t contradicts the estimated required net torque in direction or magnitude (e.g., a large torque exists but the object's angular velocity remains unchanged, or no torque exists but the object rotates), the verification result is anomaly and marked as "dynamic inconsistency." It is understandable that the dynamics verification rules directly verify the manifestation of Newton's second law in rotation (∑τ=Iα). Inconsistency indicates that the time-series data violates Newton's second law.

[0079] In velocity / acceleration verification, if the instantaneous angular velocity v_t of any joint exceeds the maximum physiological angular velocity v_max, or the instantaneous angular acceleration a_t exceeds the maximum physiological angular acceleration a_max, the verification result is determined to be abnormal and marked as "kinematic abnormality". In this embodiment, the maximum physiological angular velocity is set to 10 rad / s, and the maximum physiological angular acceleration is set to 50 rad / s². It is understood that the movement capabilities of each joint in the human hand are constrained by the biomechanical characteristics of muscles, tendons, etc., and there are clear upper limits for velocity and acceleration. Data exceeding these limits lacks biological authenticity.

[0080] In joint angle verification, if the angle θ_t of any joint exceeds its preset physiological range [θ_min, θ_max] (e.g., the angle range of the metacarpophalangeal joint is approximately -30° to 90°), the verification result is determined to be abnormal and marked as "joint angle out of bounds abnormality". It is understood that the range of motion of human hand joints is fixed due to the limitations of skeletal structure and ligaments. Angle data exceeding the corresponding angle range is physically impossible. In this embodiment, for each frame of temporal data, if any verification result is abnormal, the temporal data of that frame is determined to be abnormal interactive data.

[0081] In some embodiments, the quality verification module 104 is further configured to: in response to the presence of abnormal interaction data in the target interaction data, mark the frame containing the abnormal interaction data to obtain an abnormal problem frame, and generate quality verification information.

[0082] Specifically, the target interactive data includes time-series data. The quality verification module 104 traverses the physical verification information of each frame of time-series data and applies multiple verification rules to the physical verification information for parallel checks. When any verification rule (such as penetration depth verification, contact force verification, velocity / acceleration verification, dynamics verification, joint angle verification) is triggered, the time-series data of that frame is marked as an abnormal problem frame. The quality verification module 104 records a list for each abnormal problem frame consisting of triggering rule identifiers (such as "PENETRATION", "CONTACT_FORCE", "VELOCITY", "ACCELERATION", "DYNAMICS", etc.). Simultaneously, based on the number and type of triggered verification rules and the severity of the violation (such as the percentage of penetration depth exceeding a threshold), a normalized anomaly severity score, ranging from 0 to 1, is calculated using a preset weighted scoring model. This anomaly severity score quantifies the degree of anomaly in the time-series data of the abnormal problem frame.

[0083] For each marked abnormal frame, the quality verification module 104 generates corresponding quality verification information. For example, the core fields of the quality verification information include: Frame ID and Timestamp (frame_id, timestamp): precisely locating the location of the abnormality. Validity Status (is_valid): a boolean value, marked as false. Anomaly Type List (anomaly_types): listing all triggered verification rules. Anomaly Severity Score (anomaly_score): a quantified assessment of the degree of abnormality. Anomaly Related Regions (anomaly_regions): a list detailing the specific joint ID (joint_id) directly associated with the abnormality and its local severity assessment. Key Physical Parameter Snapshots (physical_parameters): recording the instantaneous values ​​of key physical quantities that triggered the abnormality, such as maximum penetration depth, abnormal contact force magnitude, maximum joint velocity, etc., which can be used for problem diagnosis.

[0084] In some embodiments, please refer to Figure 4 , Figure 4 This is a schematic diagram of the structure of a hand-object interaction data quality verification system 100 provided in another embodiment of this application. The hand-object interaction data quality verification system 100 also includes a visualization output module 105, which is communicatively connected to the simulation replay module 103 and the quality verification module 104, respectively. The visualization output module 105 is configured to: render each frame of target interaction data during simulation replay to visualize the simulation replay; mark the abnormal content in the abnormal problem frame when rendering to the abnormal problem frame; and construct the time series curve of key data based on physical verification information and quality verification information.

[0085] Specifically, the visualization output module 105 serves as the system's human-computer interaction and diagnostic interface, providing a comprehensive visualization through 3D scene rendering and 2D time-series charts. The visualization output module 105 renders each frame of target interaction data during simulation replay, achieving visualization of the simulation replay. Specifically, the visualization output module 105 constructs a 3D visualization environment for the interactive scene based on the rendering pipeline of the IsaacSim simulation engine. The interactive scene provides a main free-viewpoint, allowing users to freely rotate, translate, and zoom for global observation; simultaneously, it presets key fixed viewpoints (such as front view, side view, and hand close-up view) to focus on specific analysis dimensions. In particular, it draws the motion trajectory lines of key hand points (such as joints and palm) and the object's center of mass in real time, forming an intuitive spatial path.

[0086] The visualization output module 105 marks the abnormal content within the anomaly frame when rendering. Specifically, when rendering to an anomaly frame marked by the quality verification module 104, the visualization output module 105 activates the corresponding visual highlighting prompt based on the anomaly type. For example, for penetration anomalies: in the surface area of ​​an object where non-physical penetration occurs, a red semi-transparent geometry (such as a highlighted sphere or area overlay) is overlaid to visually display the penetration location and range. Another example is kinematic anomalies: for joints with excessive velocity / acceleration, their motion trajectory lines use color mapping (such as transitioning from green to yellow to red) to indicate the degree of anomaly, and the joint flashes to attract attention. All visual highlighting prompts are strictly bound to the anomaly frame list (anomaly_list) to ensure that visualization and logical judgment are synchronized. The anomaly frame list (anomaly_list) includes all anomaly frames.

[0087] The visualization output module 105 constructs time-series curves of key data based on physical verification information and quality verification information. Specifically, the visualization output module 105 obtains a list of global poses of the hand joints, object poses, and contact point spatial coordinates for each frame from the simulation replay module 103 for trajectory drawing. The visualization output module 105 obtains structured quality verification information from the quality verification module 104, extracting the anomaly frame ID, type, associated joints, and corresponding physical verification information to trigger visual highlighting and annotating time-series charts. The time-series charts include time-series curves of key data; for example, the time-series charts include joint motion curves, contact force curves, kinematic curves, and quality score curves. The joint motion curves represent the changes in the angles of 21 joints over time. The contact force curves represent the normal contact force amplitude at each contact point. The kinematic curves represent the angular velocity and angular acceleration of the selected joints. The quality score curve represents the anomaly severity score. In this process, on the X-axis (time axis) of each curve, anomaly markers are made at the time positions corresponding to all abnormal problem frames, such as drawing vertical red marker lines or highlighting points. Clicking on the anomaly marker will jump to the corresponding frame in the 3D scene, realizing the linkage between the time series data and the time axis of the 3D scene.

[0088] In some embodiments, please continue reading Figure 4 The hand-object interaction data quality verification system 100 also includes a contact analysis module 106, which is connected to the simulation replay module 103. The contact analysis module 106 is configured to: identify contact events and perform mechanical statistical analysis based on the collision detection of the simulation replay module 103 to obtain contact analysis information, which is used to quantify the contact parameters in the contact event.

[0089] Specifically, the contact analysis module 106 receives collision detection data from the simulation replay module 103 to identify contact events. The collision detection data includes the spatial relationships of each collider set and the calculated contact forces. The contact analysis module 106 determines the start and end of a contact event using the following logic: when the amplitude F of the contact force of a certain collider pair first exceeds a preset force threshold F... threshold When the current time t_start is recorded as the start of contact, and F>F... threshold Within a continuous time period, the contact is determined to be continuous, and the contact force F(t) for each frame within that time period is recorded. When F decreases and remains below F... threshold At that time, the previous valid time point t_end is recorded as the end of the contact. The duration of a single contact is calculated as Δt = t_end. t_start.

[0090] The contact analysis module 106 performs mechanical statistical analysis on each contact pair (hand_joint_i, object). Specifically, the contact analysis module 106 records the contact force time sequence F_i(t) of the contact pair throughout the entire interaction process, constructing a contact force curve. Based on F_i(t), the contact analysis module 106 calculates its peak force F_max and average force F_avg. According to the shape of the contact force curve, the contact analysis module 106 automatically divides and marks the time intervals of three stages: "contact establishment," "stable contact," and "contact release," thus defining the contact stages.

[0091] The contact analysis module 106 performs a comprehensive statistical analysis of multiple contact events, calculating the duration of all contact events and categorizing them by associated fingers (such as thumb, index finger, etc.). Based on the proportion of the total contact time of each finger (or joint) to the total interaction time, it calculates the contact frequency percentage. The contact frequency percentage can be used to generate heatmaps characterizing the contact activity of different areas of the hand.

[0092] The contact analysis module 106 generates contact analysis information, which, for example, includes the following fields: total_contact_events: the total number of identified contact events; contact_durations: an array of durations for each contact event; force_curves: storing the time-series force curve data (t, F) corresponding to each finger or joint as a key; peak_forces: recording the peak force generated by each finger or joint during the interaction; and contact_area_distribution: the percentage of contact frequency, recording the proportion of contact duration for each finger, used to describe the spatial distribution characteristics of the contact.

[0093] In some embodiments, please continue reading Figure 4 The hand-object interaction data quality verification system 100 also includes a report generation module 107, which is communicatively connected to the contact analysis module 106, the quality verification module 104, and the simulation replay module 103. The report generation module 107 is configured to generate a contact event sequence and a mechanical analysis report based on physical verification information, contact analysis information, and quality verification information. The mechanical analysis report is used to evaluate the quality of the dynamic interaction data collected in the current time period.

[0094] Specifically, the report generation module 107 synchronously receives quality verification information from each upstream quality verification module 104, contact analysis information from the contact analysis module 106, and physical verification information from the simulation replay module 103. Based on this, the report generation module 107 runs an event state machine algorithm based on multi-condition discrimination to automatically identify and label key contact event nodes in the interaction process. The event state machine algorithm can monitor key variables such as contact force, object pose change rate, and hand configuration in real time. Through preset thresholds and logical combinations, it determines and labels typical contact events such as GRASP_INITIATE (contact establishment), GRASP_STABILIZE (multi-finger stable grip), LIFT (object is lifted), MANIPULATE (operation in progress), and RELEASE (release). The report generation module 107 organizes the identified contact events into a structured contact event sequence, EventSequence, in chronological order. Each contact event sequence includes a unique event ID, event type, timestamp, list of involved joints, peak force (or object acceleration), and descriptive text (such as "four fingers form a stable grip"). Contact event sequences can provide a high-level semantic decomposition of the interaction process.

[0095] The report generation module 107 integrates all input information to generate a mechanical analysis report. For example, the mechanical analysis report includes an execution summary, mechanical analysis, anomaly summary, and assessment and recommendations. The execution summary includes a verall_quality_score (range 0-1) calculated based on the total duration of hand-object interaction data, the total number of contact events, the number of marked problem frames and their percentages, and the rule compliance rate. The mechanical analysis includes force analysis and kinematic analysis. Force analysis reports peak forces and their timing and location, average force during the grasping phase, and the percentage of force distribution for each finger. Kinematic analysis reports the maximum angular velocity and angular acceleration of the hand joints and assesses whether they conform to physiological limitations, while also calculating a trajectory smoothness score. The anomaly summary presents all anomalous problem frames, including the anomalous problem frame ID, anomaly type (e.g., "minor clipping"), and key anomaly parameters (e.g., penetration depth 2mm), for quickly locating anomalous problems. The evaluation and recommendations are based on the overall quality score (verall_quality_score) and mechanical analysis to generate data usability conclusions and specific improvement suggestions, such as "Frames 123-125 are recommended to be reacquired" and "The overall data quality is good and can be used for training".

[0096] The report generation module 107 can convert contact event sequences and mechanical analysis reports into three output formats. Specifically, the three formats include structured JSON, formatted PDF documents, and interactive HTML pages. Structured JSON fully preserves all analysis data and event sequences for subsequent procedural analysis and database storage. Formatted PDF documents include summary charts (such as contact force-time curves and anomaly score time series plots), data tables, and textual conclusions, suitable for archiving and written reports. Interactive HTML pages integrate a 3D scene replay view, interactive time series charts, and highlighted anomaly frames, allowing users to dynamically explore the analysis results.

[0097] It should be noted that in the above embodiments, there is no necessarily a certain order between the steps. Those skilled in the art can understand from the description of the embodiments of this application that the above steps may have different execution orders in different embodiments, that is, they may be executed in parallel or in turn, etc.

[0098] The above-disclosed embodiments are merely preferred embodiments of this application and should not be construed as limiting the scope of this application. Therefore, any equivalent variations made in accordance with the claims of this application shall still fall within the scope of this application.

Claims

1. A hand-object interaction data quality verification system, characterized in that, include: The data preprocessing module is configured to process the dynamic interactive data collected during the process of the operator wearing data gloves grasping objects into target interactive data. The data gloves are equipped with multiple sensors for collecting joint data. The physical scene construction module is configured to: construct an interactive scene of the hand model and the object model based on the target interaction data using the simulation engine; The simulation replay module is configured to control the hand model and the object model to simulate and replay the target interaction data to obtain physical verification information; The quality verification module is configured to verify the physical verification information based on multiple verification rules and identify abnormal interaction data.

2. The system according to claim 1, characterized in that, The dynamic interactive data includes glove sensor data, object pose data, and object tags. The glove sensor data includes raw data from multiple joint sensors and tactile sensor data. The data preprocessing module is further configured to: The raw data from the joint sensors is converted into joint simulation data in a standard format; The three data points—joint simulation data, tactile sensor data, and object pose data—are correlated to obtain time-series data at a unified time. A transformation matrix is ​​calculated based on the time series data under a known pose. The transformation matrix is ​​used to transform the coordinate system of the time series data to the simulation world coordinate system.

3. The system according to claim 2, characterized in that, The target interaction data includes the time-series data, the transformation matrix, and the object labels. The physical scene construction module is further configured to: The hand model is constructed by adding joint colliders and mass attributes to each joint of the hand model based on a predefined hand URDF file; Based on the object label, select the corresponding 3D mesh file, generate the mesh collider and set the physical material parameters to construct the object model; Physical constraints are set for the hand model, including joint angle constraints and maximum velocity / acceleration constraints. Contact parameters are set for the hand model and the object model, and the contact parameters are used to achieve collision detection.

4. The system according to claim 3, characterized in that, The simulation replay module is also configured as follows: The hand model and the object model are controlled to simulate and replay the time-series data, and collision detection is performed on each frame of the time-series data. The simulation replay is to reproduce the time-series data in a simulated world. The physical verification information is obtained based on the collision detection.

5. The system according to claim 4, characterized in that, The physical verification information includes a sequence of contact points and the normal vector and / or penetration depth and / or contact force and / or torque for each contact point. Obtaining the physical verification information based on the collision detection includes: For each frame of the time-series data, a set of all colliders is obtained based on the collision detection. The set of colliders includes joint colliders of the hand model and mesh colliders of the object model. Calculate the spatial distance between the nearest point sets of each collider set, wherein the nearest point set includes the point that is closest to both the joint collider and the mesh collider in the collider set; In response to the spatial distance being less than a preset distance threshold, it is determined that the joint collider and the mesh collider in the collider set have come into contact; The average of the spatial coordinates of two points in the set of closest points is calculated as the spatial coordinates of the contact point; Calculate the normal vector and / or penetration depth and / or contact force and / or torque at each contact point.

6. The system according to claim 5, characterized in that, The physical verification information also includes the joint angle and velocity / acceleration of each joint of the hand model. The multiple verification rules include penetration depth verification and / or contact force verification and / or dynamic verification and / or velocity / acceleration verification and / or joint angle verification. The quality verification module is also configured to: For each frame of time-series data, if the penetration depth at each contact point does not meet the penetration depth verification, the verification result is abnormal. And / or, If the contact force at each contact point does not meet the contact force verification, the verification result is determined to be abnormal. And / or, If the torque at each contact point does not meet the dynamic verification requirements, the verification result is determined to be abnormal; and / or, If the velocity / acceleration of the joint where the contact point is located does not meet the velocity / acceleration verification, the verification result is abnormal; and / or, If the joint angle of the joint where the contact point is located does not meet the joint angle verification, the verification result is determined to be abnormal. If a verification result is found to be abnormal, the timing data of this frame is determined to be the abnormal interaction data.

7. The system according to claim 1, characterized in that, The quality verification module is also configured to: In response to the presence of abnormal interaction data in the target interaction data, the frame containing the abnormal interaction data is marked to obtain an abnormal problem frame, and quality verification information is generated.

8. The system according to claim 7, characterized in that, It also includes a visualization output module, which is communicatively connected to both the simulation replay module and the quality verification module. The visualization output module is configured as follows: Render the target interaction data of each frame during simulation replay to visualize the simulation replay; When rendering to the abnormal problem frame, the abnormal content within the abnormal problem frame is marked; Time-series curves of key data are constructed based on the physical verification information and the quality verification information.

9. The system according to claim 7, characterized in that, It also includes a contact analysis module, which is communicatively connected to the simulation replay module. The contact analysis module is configured as follows: Based on the collision detection and identification of contact events and mechanical statistical analysis of the simulation replay module, contact analysis information is obtained, which is used to quantify the contact parameters in the contact event.

10. The system according to claim 9, characterized in that, It also includes a report generation module, which is communicatively connected to the contact analysis module, the quality verification module, and the simulation replay module, respectively. The report generation module is configured as follows: Based on the physical verification information, the contact analysis information, and the quality verification information, a contact event sequence and a mechanical analysis report are generated. The mechanical analysis report is used to evaluate the quality of the dynamic interaction data collected in the current time period.

Images