Body intelligence multi-finger operation data alignment method, system and device and electronic equipment

Abstract

The application discloses a body intelligent multi-finger operation data alignment method, system and device and electronic equipment, and relates to the technical field of body intelligence. The method comprises the following steps: acquiring a finger pulp tactile array, a wrist torque and multi-finger action data in a body intelligent task, and establishing an inter-modal transformation relationship; generating a candidate contact event based on tactile contact features and action stage features, verifying contact authenticity by using an equivalent wrist torque and an actual torque residual error and auxiliary conditions, merging adjacent effective events into a contact event cluster, and outputting alignment data by taking a contact start time as an alignment anchor point. The multi-modal alignment data output by the application can more accurately reflect the timing relationship between tactile contact, mechanical response and finger action, thereby improving the quality of the real machine operation data set and providing a more reliable data basis for subsequent contact state recognition, dexterous operation strategy learning, robot imitation learning and body intelligent model training.

Description

Technical Field

[0001] This invention relates to the field of embodied intelligence technology, specifically to a method, system, device, and electronic device for aligning multi-finger operation data in embodied intelligence. Background Technology

[0002] In recent years, embodied intelligence technology has developed rapidly. Robots, through visual perception, motion control, and environmental interaction capabilities, have gradually expanded from simple grasping and handling tasks to more complex real-world physical tasks such as assembly, manipulation, organization, and tool use. To enhance the generalization ability of embodied intelligence models, the industry has begun building large-scale real-world robot datasets, such as AgiBot World and Open X-Embodiment. These datasets typically cover multiple robot bodies, multiple task scenarios, and a large number of real-world operational trajectories, providing a crucial data foundation for general robot policy learning, visual-language-motion model training, and cross-body transfer. Publicly available information shows that Open X-Embodiment contains over one million real-world robot trajectories, covering more than twenty types of robot bodies; AgiBot World is also being built for large-scale real-world operational data and dual-arm operational scenarios, and continues to expand its embodied intelligence data ecosystem.

[0003] However, the modal composition of existing large-scale embodied intelligence datasets is still mainly concentrated on data levels such as visual information, joint states, end effector poses, and motion trajectories. Their core value is more reflected in "seeing the scene and reproducing the action." For the high-precision tactile feedback, wrist force or torque feedback, and the collaborative contact relationship of multiple fingers in the contact establishment process that are necessary for multi-fingered dexterous hands to perform fine operations, existing open datasets are still relatively scarce.

[0004] In multi-finger dexterity scenarios, the actual operational effect often depends not only on whether the fingers reach the target position, but also on when the fingers make effective contact with the target object, how the contact force is established, and whether multiple fingers cooperate to form a stable constraint. For example, in tasks such as pinching, inserting, twisting, pressing, and flicking, visual images and motion trajectories can describe what actions the robotic arm "did," but they are difficult to accurately reflect whether effective contact has been made between the fingertips and the target object, whether the contact has transformed from local friction to stable support, and whether the contact has formed an observable mechanical response at the wrist torque level. Therefore, if the dataset lacks high-precision tactile and force feedback, or if there is a lack of reliable temporal alignment between tactile, force, and motion trajectories, the model is prone to learning surface motion trajectories but struggles to learn the contact establishment rules and mechanical interaction rules upon which fine manipulation depends.

[0005] For multi-finger dexterity, the truly valuable alignment anchor point should be the moment when multi-source data forms a physically consistent relationship, rather than the moment when a single sensor's data first changes. If this moment cannot be accurately determined, even if high-precision tactile and force feedback is collected, it is difficult to convert it into high-quality samples that can be used for model training. Consequently, embodied intelligent models still rely on vision and trajectory imitation in contact-rich tasks, lacking the ability to learn from real contact states and mechanical feedback. Summary of the Invention

[0006] To address the problem that existing technologies struggle to generate high-precision sample datasets for embodied intelligent multi-finger dexterity operation using vision and motion trajectories, this invention provides a method, system, device, and electronic device for embodied intelligent multi-finger operation data alignment.

[0007] To achieve the above objectives, the technical solution of the present invention is as follows:

[0008] In a first aspect, this application discloses a method for aligning embodied intelligent multi-finger operation data, comprising the following steps:

[0009] Acquire multimodal data when embodied intelligence performs target operation tasks and establish transformation relationships between data; multimodal data includes fingertip tactile array data, wrist torque data, and multi-finger movement data;

[0010] Tactile contact features are extracted based on fingertip tactile array data, and action phase features are extracted based on multi-finger action data. It is then determined whether the tactile contact features and action phase features meet the contact conditions. If so, candidate contact events are generated.

[0011] Within the time window corresponding to the candidate contact event, the tactile contact features are converted into equivalent wrist torque features based on the transformation relationship. Error calculation is then performed between this conversion and the residual of the actual wrist torque features extracted from the wrist torque data. The result is used to determine if the calculation does not exceed the threshold; if so, auxiliary verification is performed. Auxiliary verification includes at least one of the following: whether the rate of change of the actual wrist torque feature residual meets the corresponding conditions; whether the decrease in fingertip speed or the amount of finger closure and its rate of change in the action phase characteristics meet the corresponding conditions; and whether the candidate contact times of a preset number of fingers are located within the same collaborative time window. The candidate contact time of a finger is the moment when the contact conditions of the candidate contact event are first met. The collaborative time window is set based on the candidate contact time.

[0012] Candidate contact events that pass auxiliary verification and meet the adjacency condition are merged into contact event clusters. The contact start time of the contact event cluster is used as the alignment anchor point to perform time alignment on the multimodal data and output it.

[0013] Secondly, this application discloses an embodied intelligent multi-finger operation data alignment system, including a data acquisition module, a feature extraction and judgment module, a feature verification module, and a data alignment output module.

[0014] The data acquisition module is used to acquire multimodal data when the embodied intelligence performs target operation tasks and establish transformation relationships between data; the multimodal data includes fingertip tactile array data, wrist torque data, and multi-finger movement data;

[0015] The feature extraction and judgment module is used to extract tactile contact features based on fingertip tactile array data and extract action phase features based on multi-finger action data, and to determine whether the tactile contact features and action phase features meet the contact conditions. If so, candidate contact events are generated.

[0016] The feature verification module is used to convert tactile contact features into equivalent wrist torque features based on transformation relationships within the time window corresponding to candidate contact events. It then calculates the error between the converted feature and the residual of the actual wrist torque features extracted from the wrist torque data, determining whether the calculation result does not exceed a threshold. If so, auxiliary verification is performed. Auxiliary verification includes at least one of the following: whether the rate of change of the actual wrist torque feature residual meets the corresponding conditions; whether the decrease in fingertip speed or the amount of finger closure and its rate of change in the action phase feature meet the corresponding conditions; and whether the candidate contact times of a preset number of fingers are located within the same collaborative time window. The candidate contact time of a finger is the moment when the contact conditions of the candidate contact event are first met. The collaborative time window is set based on the candidate contact time.

[0017] The data alignment output module is used to merge candidate contact events that have passed auxiliary verification and meet the adjacency condition into contact event clusters, use the contact start time of the contact event cluster as the alignment anchor point, perform time alignment on the multimodal data and output it.

[0018] Thirdly, this application discloses an embodied intelligent multi-finger operation data alignment device, including an embodied intelligent actuator, a fingertip tactile array sensor, a wrist torque sensor, a multi-finger motion sensor, a memory, and a processor.

[0019] An embodied intelligent actuator is used to perform target operational tasks;

[0020] A fingertip tactile array sensor is installed on the fingertip area of ​​at least one finger of an integrated intelligent actuator to collect fingertip tactile array data when the corresponding finger performs a target operation task.

[0021] A wrist torque sensor is installed in the wrist connection area of ​​the embodied intelligent actuator to collect wrist torque data when the embodied intelligent actuator performs target operation tasks;

[0022] Multi-finger motion sensor is used to collect multi-finger motion data of the embodied intelligent actuator when performing target operation tasks;

[0023] The memory is used to store computer programs, as well as fingertip tactile array data, wrist torque data, and multi-finger movement data.

[0024] The processor, connected to the fingertip haptic array sensor, wrist torque sensor, multi-finger motion sensor, and memory, implements the steps of the embodied intelligent multi-finger operation data alignment method described above when executing a computer program.

[0025] Fourthly, this application discloses an electronic device, including a memory and a processor, wherein the memory stores a computer program, characterized in that the processor executes the computer program to implement the steps of the aforementioned embodied intelligent multi-finger operation data alignment method.

[0026] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0027] Based on the consistency verification between local contact and overall mechanical response, this application further eliminates accidental touches, local slippage, and unstable contact by using auxiliary verification methods such as torque residual change rate, fingertip speed decrease, finger closure amount, or multi-finger cooperative time window. By merging adjacent candidate contact events into contact event clusters and using the contact start time of the contact event cluster as the alignment anchor point, more stable and reliable multimodal time alignment results can be obtained, thereby improving the quality of embodied intelligence real machine datasets and providing an accurate data foundation for contact state modeling, imitation learning, and dexterous operation strategy training. Attached Figure Description

[0028] The disclosure of this invention is illustrated with reference to the accompanying drawings. It should be understood that the drawings are for illustrative purposes only and are not intended to limit the scope of protection of this invention. In the drawings, the same reference numerals are used to refer to the same parts. Wherein:

[0029] Figure 1 This is a flowchart illustrating an embodied intelligent multi-finger operation data alignment method described in this invention;

[0030] Figure 2 For based on Figure 1 A schematic diagram of multimodal time alignment;

[0031] Figure 3 A structural block diagram of an embodied intelligent multi-finger operation data alignment system;

[0032] Figure 4 This is a schematic diagram of a device for embodying intelligent multi-finger operation data alignment. Detailed Implementation

[0033] It is readily understood that, based on the technical solution of this invention, those skilled in the art can propose various interchangeable structural methods and implementations without altering the essential spirit of the invention. Therefore, the following detailed embodiments and accompanying drawings are merely illustrative examples of the technical solution of this invention and should not be considered as the entirety of the invention or as limitations or restrictions on the technical solution of this invention.

[0034] The method of this invention is applied to a robotic system having an embodied intelligent body (such as a humanoid robot body), a five-finger dexterous hand, a fingertip or fingertip tactile array, a wrist torque sensor, and a multi-joint encoder. In an optional embodiment, the robotic system further includes a vision sensor to assist in determining the spatial relationship between the fingers and a target object. The fingertip or fingertip tactile array data is used to characterize the local contact state of the fingers, the wrist torque data is used to characterize the global external forces acting on the robot hand, and the multi-joint angle data or fingertip pose data is used to characterize the finger movement state. Target operation tasks include, for example, picking up a glass, pinching a soft packaging bag, twisting a bottle cap, plugging or unplugging a connector, pressing a buckle, or removing a thin object from a drawer.

[0035] Example 1

[0036] like Figure 1 The diagram illustrates a method for aligning embodied intelligent multi-finger operation data, comprising the following steps:

[0037] S101. Acquire multimodal data when embodied intelligence performs target operation tasks and establish transformation relationships between data; multimodal data includes fingertip tactile array data, wrist torque data and multi-finger movement data.

[0038] The transformation relationship is based on the pre-established transformation relationship between the embodied intelligent finger tactile sensor coordinate system and the wrist base coordinate system.

[0039] Specifically, the embodied intelligent entity is controlled to perform target tasks through its five-fingered dexterous hands, and during the execution, it collects fingertip tactile array data, wrist torque data, and multi-finger motion data. To establish the spatial correspondence between the data of each modality, the first... Root finger tactile sensor coordinate system and wrist coordinate system The tactile sensor coordinate system Used to describe the Contact position on the root finger tactile array, wrist base coordinate system Used to describe the output of the wrist torque sensor. According to the... By determining the joint angles of the fingers and the extrinsic parameters of the tactile sensor installation, the transformation relationship from the tactile sensor coordinate system to the wrist base coordinate system can be established, thereby enabling the first finger to be used as the base finger. The position of the tactile contact point of the finger, when transformed to the wrist coordinate system, is represented as follows: ;

[0040] in, Indicates the finger number. Indicates time, Indicates the first The position of the tactile contact point of the finger in the wrist coordinate system. Indicates by the first Homogeneous transformation from finger coordinate system to wrist base coordinate system and derived from joint angles Decide, Indicates the first The angle of the joint of the root finger, Represents the coordinate system of the tactile sensor to the first... Installation extrinsic parameter transformation of the root finger coordinate system. Indicates the first The pressure center is located within the finger tactile array. This coordinate transformation provides a unified spatial reference for subsequent conversion of local tactile contact forces into equivalent wrist torques. This step pertains to multimodal data acquisition and coordinate system calibration, and can be implemented using conventional methods for robot forward kinematics and sensor calibration, which will not be elaborated upon here.

[0041] S102. Extract tactile contact features based on fingertip tactile array data, and extract action phase features based on multi-finger action data. Determine whether the tactile contact features and action phase features meet the contact conditions. If so, generate candidate contact events.

[0042] Among them, the tactile contact features extracted based on fingertip tactile array data include normal contact force, effective contact area, and pressure center, specifically including:

[0043] The pressure value and area of ​​each tactile unit of the fingertip tactile array are obtained, and the normal contact force of the finger is calculated accordingly.

[0044] The effective contact area is calculated by combining the area of ​​the tactile unit whose pressure value exceeds the threshold, and the pressure center is determined based on the center position and pressure value of each tactile unit.

[0045] Specifically, no. The normal contact force of a finger is obtained by summing the pressure values ​​and areas of each tactile unit on the finger's tactile array:

[0046] ;

[0047] in, Indicates the first The normal contact force of the root finger, Indicates the first The first tactile array Each tactile unit at time Pressure value, This indicates the area of ​​the tactile unit. and They represent the first The number of rows and columns of each tactile array.

[0048] No. The effective contact area of ​​a finger is obtained by accumulating the tactile unit surfaces where the pressure value exceeds the pressure threshold:

[0049] ;

[0050] in, Indicates the first The effective contact area of ​​the fingers This indicates an indicator function, meaning it returns 1 if the condition within the parentheses is true, and 0 otherwise. This indicates the pressure threshold at which the tactile sensor recognizes effective contact. Pressure threshold To filter out weak noise, the value can be 2% to 10% of the full scale of the tactile array unit, for example, 5% of the full scale, and can be adjusted according to the noise level of the tactile sensor.

[0051] No. The pressure center of the root finger is obtained by weighted averaging of the product of the pressure value and area of ​​the center positions of each tactile unit:

[0052] ;

[0053] in, Indicates the first The position of the center of each tactile unit in the coordinate system of the tactile sensor This represents an extremely small positive number to prevent the denominator from being zero, such as 10. -6 .

[0054] In extracting action phase features based on multi-finger motion data, the action phase features include the decrease in fingertip speed, the degree of finger closure, and the action phase state, specifically including:

[0055] Acquire multi-finger movement data, including the joint angles of the fingers and their fingertip positions in the wrist coordinate system;

[0056] The fingertip velocity is calculated based on the fingertip position at adjacent moments, and the decrease in fingertip velocity is calculated based on the magnitude of the fingertip velocity at the current moment and the previous moment.

[0057] The finger joint angles are weighted and summed to obtain the finger closure amount; the action stage state is generated based on the decrease in fingertip speed and the comparison between the finger closure amount and the corresponding threshold. The action stage state includes the closure stage, deceleration stage, and force application stage.

[0058] The contact conditions are that the normal contact force of the current finger exceeds the normal contact force threshold, the effective contact area exceeds the contact area threshold, and the finger is in an action phase state.

[0059] Once a candidate contact event is identified, a short time window is formed by extending the candidate contact time of the candidate contact event by a preset duration. The average pressure center is calculated by averaging the pressure center within the short time window. The average pressure center is then used as the tactile contact position and transformed to the wrist coordinate system through a transformation relationship.

[0060] Regarding the characteristics of the action phase, the fingertip velocity can be obtained by taking the time derivative of the fingertip position with respect to the fingertip position in the multi-finger action data at adjacent time points.

[0061] ;

[0062] in, Indicates the first The velocity of the fingertip in the wrist coordinate system. Indicates the first The position of the fingertip in the wrist coordinate system. Further, the difference between the magnitude of the fingertip velocity at the current moment and the previous moment yields the decrease in fingertip velocity; for the ... The finger closure is obtained by weighted summation of the joint angles of each finger. The weights are normalized weights, and their sum is 1. These weights can be allocated according to the contribution of each joint to finger opening and closing, for example, assigning a larger weight to the proximal phalanges. Subsequently, the action phase state is generated based on the comparison between the fingertip velocity decrease and the finger closure amount with corresponding thresholds, and then... Indicates the first If the finger is in a contact candidate action phase such as closing, deceleration, or applying force, take 1 if it is in this phase and 0 otherwise.

[0063] Specifically, the states of the action phase can be defined as follows: when the rate of change of finger closure is greater than the closure threshold, the i-th finger is determined to be in the closure phase; when the decrease in fingertip speed is greater than the speed decrease threshold, the i-th finger is determined to be in the deceleration phase; when the finger closure is greater than the minimum closure threshold and the tactile contact feature persists, the i-th finger is determined to be in the force application phase. It should be noted that the force application phase is only one of the action phase states; the final determination of the contact condition is still based on the joint judgment of tactile sensation and action.

[0064] In an optional embodiment, if visual data is available, including image data, depth data, or target object pose data obtained through visual recognition, and the visual distance between the finger and the target object is less than a calibrated threshold, the finger can be determined to be in an approach state, which can be used as a feature of the action phase. Furthermore, the determination of visual distance can also serve as one of the auxiliary verification conditions.

[0065] After obtaining the above-mentioned tactile contact characteristics and action phase characteristics, when the first The fingers simultaneously satisfy the condition that the normal contact force exceeds the normal contact force threshold. The effective contact area exceeds the contact area threshold. And it has an action phase state (i.e. When the finger is in contact with a certain force, a candidate contact event is generated. This includes the normal contact force threshold. For example, a value of 0.05~0.5N can be used for the contact area threshold. For example, the area can be equivalent to 1-5 tactile units or 3%-15% of the total area of ​​the tactile array, which can be set according to the contact characteristics between the dexterous hand and the target object. This candidate contact event is not directly identified as a real contact event, but enters the subsequent verification steps. In addition, after the candidate contact event is determined, a short time window is formed by extending the candidate contact time forward and backward by a preset time (e.g., 10-50ms). The average pressure center within the short time window is obtained by averaging the pressure centers, which is used as the tactile contact position and transformed to the wrist base coordinate system through a transformation relationship. This averaging process can suppress the jitter of the pressure center in a single frame, making the tactile contact position more stable.

[0066] For example, in the task of picking up a glass, when the pressure of the tactile array on the pads of the thumb and index finger increases with contact, the effective contact area expands accordingly, and at this time the two fingers are in the closing and force-applying action phase, the contact condition is met, thus generating candidate contact events for the thumb and index finger respectively; conversely, when a tactile unit experiences a momentary pressure jump only due to array noise, but the finger is still in the free movement phase and does not have the action phase state of closing, deceleration or force application, no candidate contact event will be generated.

[0067] Therefore, this step, through the dual joint judgment of tactile contact features and action phase features, can filter out false triggers caused by tactile noise in the free movement phase of the finger, compared with the single judgment based on the tactile pressure threshold alone. This improves the accuracy of the initial screening of candidate contact events and provides fewer but higher-quality candidate objects for subsequent tactile-force consistency verification.

[0068] S103. Within the time window corresponding to the candidate contact event, the tactile contact features are converted into equivalent wrist torque features based on the transformation relationship, and the error is calculated with the residual of the actual wrist torque features extracted from the wrist torque data. It is then determined whether the calculation result does not exceed the threshold. If so, auxiliary verification is performed. The auxiliary verification includes at least one of the following: whether the rate of change of the residual of the actual wrist torque features meets the corresponding conditions; whether the decrease in fingertip speed or the amount of finger closure and its rate of change of the action phase features meet the corresponding conditions; and whether the candidate contact times of the preset number of fingers are located within the same collaborative time window. The candidate contact time of the finger is the moment when the contact conditions of the candidate contact event are first met. The collaborative time window is set based on the candidate contact time.

[0069] The transformation of tactile contact features into equivalent wrist torque features based on the transformation relationship is as follows:

[0070] The lever arm is determined based on the positional relationship between the tactile contact position of the finger in the wrist coordinate system and the origin of the wrist coordinate system.

[0071] The normal contact force of the fingers is transformed into the wrist base coordinate system through a transformation relationship, and then coupled with the lever arm to obtain the equivalent wrist torque;

[0072] The equivalent wrist torque of multiple fingers within a candidate contact event is synthesized to obtain the equivalent wrist torque feature corresponding to the candidate contact event.

[0073] Before performing auxiliary verification, the priority of auxiliary verification conditions is determined based on the type of the target operation task, and auxiliary verification is performed accordingly.

[0074] This step primarily involves cross-validating candidate contact events by examining the physical consistency between local tactile information and global wrist force information, thereby eliminating false contacts. Specifically, for each candidate contact event, the first step is to cross-validate the candidate contact event by examining the physical consistency between local tactile information and global wrist force information. The contact force estimated by the finger in the tactile sensor coordinate system is transformed to the wrist base coordinate system using a rotation matrix:

[0075] ;

[0076] in, Indicates the first The contact force of the finger in the wrist coordinate system. This represents the rotation matrix from the tactile sensor coordinate system to the wrist base coordinate system. Indicates the first The contact force estimated by a finger in the coordinate system of the tactile sensor. Among them, Transformation relationship from S101 tactile sensor coordinate system to wrist base coordinate system The rotational component, that is, only the rotational part of the homogeneous transformation chain is used for the coordinate transformation of the contact force; the th Contact force estimated by a finger in the coordinate system of the tactile sensor Normal contact force obtained from S102 Along the normal of the tactile sensor coordinate system Construction, i.e. (In an embodiment where tangential friction is ignored), the scalar normal contact force of S102 is connected to the contact force vector of this step. This represents the unit normal vector in the coordinate system of the tactile sensor of the i-th finger, that is, the normal unit vector perpendicular to the contact surface of the tactile array of the fingertip.

[0077] It should be noted that the above method of constructing the contact force vector solely from the normal contact force is suitable for implementations where the tangential friction is relatively small, or where only coarse-grained consistency verification is required at the moment of contact. In tasks such as screwing, inserting / removing, sliding, and stable clamping, where tangential friction significantly contributes to wrist torque, any of the following implementation methods can be used to ensure that the equivalent wrist torque characteristics simultaneously reflect the contribution of the tangential force:

[0078] Firstly, in embodiments where the tactile array possesses three-dimensional force or tangential force sensing capabilities, each tactile unit of the i-th finger directly outputs a three-dimensional contact force vector. These vectors are then summed in the tactile sensor coordinate system to obtain a contact force vector that simultaneously includes both normal and tangential components. Then, after being transformed to the wrist base coordinate system by the rotation matrix and cross-multiplied with the lever arm, the equivalent wrist torque characteristics simultaneously include the force and torque components generated by the tangential friction force.

[0079] Secondly, in implementations where the tactile array can only sense normal pressure, the tangential force is estimated based on the friction cone constraint, meaning the magnitude of the tangential force does not exceed the product of the friction coefficient and the normal contact force. , The friction coefficient between the i-th finger and the contact surface of the target object is represented by the direction of the tangential motion of the fingertip relative to the target object, or the resultant force balance relationship of multiple fingers gripping. The estimated tangential force and normal force are combined to construct the contact force vector to approximately compensate for the contribution of tangential friction force to the wrist torque.

[0080] Third, for tasks dominated by tangential force and without tangential force estimation, the consistency error threshold can be appropriately relaxed, or only the torque components less affected by tangential force can be compared for consistency. Consistency verification can be used as a pre-screening step to eliminate gross inconsistencies (such as tactile equivalent torque approaching zero and wrist residual being large during free space emergency stop). The determination of contact effectiveness in such tasks can be left more to auxiliary verification conditions such as wrist force impact peak, fingertip deceleration, and multi-finger coordination.

[0081] Subsequently, the tactile contact position in the wrist coordinate system The positional relationship between the finger and the origin of the coordinate system is taken as the lever arm. The cross product of the contact force and the lever arm yields the equivalent torque generated by the finger contact force on the wrist.

[0082] ;

[0083] in, Indicates the first The contact force of the finger produces an equivalent torque about the origin of the wrist coordinate system. The contact force and the equivalent torque together constitute the first... The equivalent torque of the root finger:

[0084] ;

[0085] in, Indicates the first The equivalent torque of each finger. Summing the torques of multiple fingers involved in the candidate contact event yields the multi-finger tactile equivalent torque, which serves as a characteristic of the equivalent wrist torque.

[0086] ;

[0087] in, Indicates the tactile equivalence moment of multiple fingers. This represents the set of fingers that participated in the candidate contact event.

[0088] On the other hand, the actual wrist torque characteristic residual (i.e., the wrist external torque residual) is extracted from the wrist torque data. To eliminate non-contact torques caused by the robot's own motion, gravity, or inertia, baseline compensation is performed on the measured wrist torque. That is, the wrist baseline torque is estimated within the non-contact time window, and the difference between the measured wrist torque and the wrist baseline torque is used as the wrist external torque residual.

[0089] ;

[0090] in, This indicates the measured torque at the wrist. Indicates the wrist baseline torque. This represents the residual of the external wrist torque caused by external contact. In another implementation, the non-contact dynamic torque determined by joint position, velocity, and acceleration can be calculated based on the robot's dynamics model, and the difference between the measured wrist torque and this non-contact dynamic torque can be used as the residual of the external wrist torque, thereby obtaining a more accurate residual estimate when the robot is moving at high speed. The non-contact time window can be taken as the free motion phase before the start of the target operation task, or a contact-robust recursive estimation can be used.

[0091] Based on this, the tactile equivalent torque of the multi-finger touch is compared with the residual external torque of the wrist to obtain the tactile-force consistency error.

[0092] In practical applications, if not just torque, but force and torque are used, considering the different dimensions of the force component (unit N) and torque component (unit N·m) in force / torque, instead of directly calculating the Euclidean norm of the entire vector, the force component and torque component are compared separately to obtain the force consistency error. Torque Consistency Error :

[0093] ;

[0094] ;

[0095] in, , These are the force component and torque component of the equivalent torque of multi-finger tactile sensation, respectively. , These are the force component and torque component of the external torque residual at the wrist, respectively. , Collectively referred to as tactile-mechanical consistency error When the force consistency error does not exceed the force consistency error threshold. And the torque consistency error does not exceed the torque consistency error threshold. At that time, it was believed that tactile contact and wrist force sensation corroborated each other, and the candidate contact event was verified by wrist force sensation; when Exceed or Exceed When, that is, the consistency error is considered. If the threshold is exceeded, the candidate contact event is judged as a spurious contact event or a low-confidence contact event and is discarded. For example, a value of 0.1~1N can be used. For example, a value of 0.01~0.1 N·m can be used, or the relative error of force and torque can be set to 10%~30%, which can be determined according to the sensor accuracy and the task force.

[0096] For candidate contact events that pass the tactile-force consistency verification, at least one auxiliary verification condition is further combined to determine the valid contact event. The auxiliary verification conditions include: the residual external torque of the wrist exhibits an impact peak within the candidate time window, i.e., its rate of change meets the corresponding condition; the fingertip velocity decreases within the candidate time window, i.e., the amount of fingertip velocity decrease meets the corresponding condition; the finger closure increases within the candidate time window; the target object undergoes a small displacement when visual data is available, or the distance between the finger and the target object is less than a spatial threshold; and a preset number of fingers' candidate contact moments fall within the same collaborative time window. The collaborative time window is set based on the candidate contact moment, i.e., starting from the earliest candidate contact moment among the participating fingers that meets the contact condition, and extending backward for a duration equal to the collaborative time window. When the candidate contact moments of a preset number of fingers all fall within this time window (i.e., the maximum difference between these candidate contact moments is less than the collaborative time window), it is considered that these fingers form a multi-finger collaborative contact relationship, and the corresponding multi-finger collaborative event is established. Here, the candidate contact time of a finger is the moment when the contact condition of the candidate contact event corresponding to that finger is first met. The collaborative time window is, for example, 50~300ms and is set according to different target operation tasks; the preset number is at least two. The number of auxiliary verification conditions satisfied by the candidate contact events is counted.

[0097] ;

[0098] in, This indicates the number of auxiliary verification conditions satisfied by the candidate contact event. This refers to the peak of force impact event. Indicates a deceleration or closing event of an action. Indicates visual micro-movements or spatial proximity events. This indicates a multi-stage collaborative event. This occurs when the number of satisfied auxiliary verification conditions is not less than the threshold for the number of verification conditions. When this occurs, the corresponding candidate contact event is determined as a valid contact event. Among these, the threshold for the number of verification conditions... The value can be set according to the task type, and the range is 1 to 4. For example, for fine-tuning with a single finger, a value of 1 can be used. Multi-finger collaborative operation is preferable This mechanism avoids requiring all conditions—tactile, force, motor, visual, and multi-finger coordination—to be met simultaneously, thus reducing the coupling between modalities. It is important to emphasize that only three conditions are available when visual data is unavailable. The maximum value is 3, so K can be 1 to 3 accordingly.

[0099] Furthermore, before conducting auxiliary verification, the priority of auxiliary verification conditions is determined based on the type of the target operation task, and executed accordingly. For example, for tasks involving changes in force, such as twisting bottle caps, the wrist force-sensing impact peak condition is prioritized; for tasks involving changes in posture, such as plugging and unplugging connectors, the fingertip speed decrease or visual proximity condition is prioritized. This ensures that the most discriminative evidence is verified first when the threshold number of conditions is met, improving verification efficiency. In other words, auxiliary verification conditions are verified one by one in priority order. When the cumulative number of met conditions reaches the verification condition number threshold K, subsequent verification stops, and the candidate contact event is determined as a valid contact event, thus reducing computational overhead.

[0100] This step effectively eliminates the following false contact scenarios. First, when the robot stops abruptly in free space, the wrist torque changes abruptly due to inertia, but the tactile array does not form effective contact. At this time, the multi-finger tactile equivalent torque is close to zero, while the residual external wrist torque is large, resulting in a consistency error. First, if the threshold is exceeded, the candidate contact event is eliminated. Second, when the tactile array generates local noise, the tactile equivalent torque is not zero, but the wrist force sensation does not show a corresponding external action; the two are inconsistent, and the candidate contact event is eliminated. Third, when the finger makes self-contact or local accidental contact, the force it experiences cannot explain the true change in wrist torque; the consistency error exceeds the threshold, and the candidate contact event is also eliminated. Conversely, for example, when pinching a soft packaging bag, if the tactile equivalent torque of the thumb and index finger matches the residual of the wrist external torque in both magnitude and direction, and is accompanied by multiple auxiliary evidences such as a wrist force sensation impact peak, fingertip deceleration, and increased finger closure within the candidate time window, then the candidate contact event is confirmed as a valid contact event.

[0101] Therefore, this step cross-validates the consistency between the multi-finger tactile equivalent torque and the wrist force sensory residual, utilizing the physical consistency between local tactile information and global force sensory information to eliminate various types of pseudo-contacts such as sudden stops in free space, local tactile noise, and accidental finger self-contact; while simultaneously satisfying at least The weakly coupled determination of auxiliary verification conditions replaces the strong constraint that all modal conditions must be satisfied simultaneously. While ensuring the high credibility of effective contact events, it avoids missing real contact due to missing individual modalities or noise, thus balancing the accuracy and recall of contact event identification and significantly improving the robustness and portability of the solution under different tasks.

[0102] As an optional implementation, when the computing power meets preset conditions (e.g., the processor's real-time computing margin is higher than a set value, or in the case of offline dataset construction), the above-mentioned priority-based early termination method can be abandoned. Instead, all auxiliary verification conditions are evaluated for the candidate contact events, and the total number of auxiliary verification conditions satisfied is counted. This allows for confidence classification of valid contact events. Specifically, when... When the number of verification conditions is not less than the threshold K, the candidate contact event is determined as a valid contact event, and simultaneously based on... The value of determines its confidence level: The larger the value, the higher the degree of mutual corroboration between multiple sources of evidence, such as tactile, force, motor, and multi-finger coordination, and the higher the corresponding confidence level. For example, when all four auxiliary verification conditions are available, a condition meeting 1 can be recorded as low confidence, 2 as medium confidence, 3 as high confidence, and 4 as very high confidence. In cases where available conditions decrease due to lack of visual data, the confidence level can be reduced accordingly, or the confidence level can be determined based on the proportion of met conditions to the total number of available conditions. In another implementation, the tactile-force consistency error can also be considered. The confidence level is adjusted so that, when the number of conditions is the same, the smaller the consistency error, the higher the confidence level.

[0103] Furthermore, the aforementioned confidence levels are recorded along with valid contact events, and when merging contact event clusters and outputting aligned data segments in S104, they are output as confidence labels for those contact event clusters. These confidence labels can be used for sample weighting or filtering in downstream imitation learning and operational strategy training; for example, assigning greater training weights to high-confidence samples and reducing or eliminating low-confidence samples, thereby further improving the overall quality of the training data while preserving recall. In this embodiment, for Candidate contact events smaller than the threshold K can be directly eliminated, or they can be retained and labeled as low-confidence samples for subsequent analysis.

[0104] Therefore, this implementation replaces the "early termination by priority" when computing power is limited with "full-condition evaluation + confidence grading" when computing power is limited: the former sacrifices some computing overhead in exchange for fine annotation with confidence, which is suitable for offline high-quality dataset construction; the latter terminates when the threshold of the number of conditions is met, which is suitable for online real-time or computing power-limited scenarios, so that this method can be flexibly adapted to different computing power conditions and application scenarios.

[0105] Furthermore, considering the differences in the discriminative power and reliability of various auxiliary verification conditions for contact authenticity (e.g., wrist force-sensory impact peaks and multi-finger coordination events generally have higher discriminative power than fingertip speed declines), confidence grading is not limited to a simple count of the number of conditions met. Instead, each auxiliary verification condition is assigned a weight reflecting its discriminative power and reliability, and a confidence score is calculated by weighted summation. The confidence level is determined based on the interval in which the confidence score falls. The weights can be set a priori (with larger weights assigned to high-discriminative conditions such as force-sensory impact peaks and multi-finger coordination), or determined statistically based on the true positive and false positive rates of each condition on labeled verification data (e.g., weighted by the likelihood ratio or log-likelihood ratio of each condition to obtain a probabilistic confidence level), or adaptively adjusted according to the target task type (consistent with the aforementioned mechanism of prioritizing auxiliary verification conditions by task type). In another implementation, the degree of satisfaction of each condition can be replaced by a binary indicator with a continuous soft indicator obtained by normalizing the margin exceeding the corresponding threshold, combined with tactile-force consistency error. Together, they constitute a continuous confidence score, which more precisely reflects the true confidence level of the contact event.

[0106] S104. Merge the candidate contact events that pass the auxiliary verification and meet the adjacency condition into a contact event cluster, use the contact start time of the contact event cluster as the alignment anchor point, perform time alignment on the multimodal data and output it.

[0107] Specifically, candidate contact events verified through auxiliary testing are judged as valid contact events, and valid contact events that meet the adjacency condition are merged into a contact event cluster. When two valid contact events correspond to the same finger, the time interval between their candidate contact moments is required to be less than a preset event merging time window (e.g., 20~200ms), and the distance between their tactile contact positions is required to be less than a preset same-finger spatial distance threshold (e.g., 2~20mm), so as to merge repeated events caused by the same contact point due to shaking or momentary loss of contact. When two valid contact events correspond to different fingers but belong to the same multi-finger collaborative operation relationship, since the contact positions of different fingers are physically far apart, the distance between their tactile contact positions is no longer required to be less than the above-mentioned same-finger spatial distance threshold. Instead, the candidate contact moments of the two are required to be within the same collaborative time window, and optionally, the distance between their tactile contact positions is required to not exceed a collaborative spatial distance threshold (e.g., 10~150mm) adapted to the size of the dexterous hand or the target object, so as to merge multi-finger collaborative contacts around the same target object into the same contact event cluster.

[0108] It should be noted that the collaborative time window in S103 is used to verify the validity of a single candidate contact event (as a multi-finger collaborative event is counted in the number of verification conditions), while the finger relationship judgment in the adjacent conditions of this step is used to cluster and merge the determined valid contact events. The two are respectively applied to the two levels of "event validity" and "event attribution".

[0109] A contact event cluster includes at least the contact start time, contact peak time, contact end time, the set of fingers involved in the contact, contact location, contact torque characteristics, and multi-finger coordination relationship.

[0110] After forming a cluster of contact events, the contact start time within the cluster is used as the alignment anchor point (in another implementation, the contact peak time can also be used as the anchor point) to perform time offset correction on the tactile data, wrist torque data, and multi-finger movement data. For the first... For modal data, the corrected timestamp is the original timestamp of the modality minus the time offset of the modality relative to the anchor point of the contact event cluster, i.e.:

[0111] ;

[0112] in, Indicates modal category, Indicates the first Modal The original timestamps of each sampling point This indicates the time offset of the mode relative to the anchor point of the contact event cluster. This represents the corrected timestamp. In one implementation, the time offset is determined based on the cross-correlation between the tactile contact evidence signal and other modal contact evidence signals:

[0113] ;

[0114] in, This indicates tactile contact evidence signals. Indicates the first Modal contact evidence signals, and Let represent the derivatives of the two variables with respect to time. This represents the time offset to be searched. This means taking the time offset that maximizes the integral value, i.e., the offset that makes the changes in the two contact evidence signals most synchronized, as the time offset of this mode. .

[0115] In implementations with visual data, the visual data is not used as the sole reference for millisecond-level contact time, but rather as supplementary spatial state data for alignment in the vicinity of contact event clusters.

[0116] Furthermore, a data slicing window is determined based on the start and end times of the contact event cluster. Specifically, within a time window consisting of a preset pre-contact retention time preceding the contact start time and a preset post-contact retention time following the contact end time, aligned tactile data, wrist torque data, multi-finger movement data, and optional visual data are extracted and output. The pre-contact retention time and post-contact retention time can both be set to, for example, 100~500ms, to fully preserve the approach process before contact and the release process after contact.

[0117] like Figure 2 As shown, in the multimodal time alignment process, Figure 2 The comparison before and after alignment is shown. Before alignment, due to the different sampling frequencies, transmission delays, and signal response characteristics of the tactile sensor, wrist torque sensor, and multi-finger motion sensor, the characteristic changes in tactile evidence, force evidence, and motion evidence may occur at t1, t2, and t3, respectively, resulting in the same physical contact event appearing as temporally misaligned in different modal data.

[0118] To eliminate the aforementioned time misalignment, the contact start time of the contact event cluster is used as a unified alignment anchor point. Subsequently, the time offset of tactile data, wrist torque data, and multi-finger movement data relative to this alignment anchor point is calculated, and the timestamps of each modality are corrected. After correction, key change points in tactile evidence, force evidence, and movement evidence are all aligned to the same time reference, thus enabling different modalities to correspond to the same real physical contact process.

[0119] In one implementation, contact state labels are generated based on tactile, force, and motion changes within the contact event cluster. The contact states include at least contact initiation, stable gripping, sliding, overpressure, and release. Specifically, when the tactile normal contact force first exceeds a threshold (which could be a normal contact force threshold) and the effective contact area begins to increase, it is labeled as contact initiation. When the tactile normal contact force remains stable for a preset time, the effective contact area remains stable, and the change in the center of pressure is less than a stability threshold, it is labeled as stable gripping. When the change in the center of pressure is greater than a sliding threshold, or when the target object undergoes a displacement exceeding a threshold relative to the fingertip, it is labeled as sliding. When the normal contact force exceeds a safety force threshold, or the wrist torque residual exceeds a safety threshold, it is labeled as overpressure. When the tactile normal contact force decreases below a release threshold and the effective contact area decreases below a release area threshold, it is labeled as release. The final output is a multimodal real machine data segment after contact event cluster alignment and status labeling, including aligned haptic array data, aligned wrist torque data, aligned multi-finger joint angle or fingertip pose data, contact event cluster information, and contact status label sequence. In optional implementations, aligned visual data is also included.

[0120] The stability threshold, slip threshold, safety force threshold, safety threshold, release threshold, and release area threshold are status determination thresholds used for contact status labeling. The status determination thresholds are preset or adaptively determined based on the target operation task type, tactile sensor range, wrist torque sensor range, or historical statistical characteristics within the contact event cluster.

[0121] For example, in the task of twisting a bottle cap, the thumb, index finger, and middle finger touch the bottle cap one after another within tens of milliseconds. Each effective contact event is merged into a contact event cluster because they are adjacent in time, close in contact position, and belong to the same multi-finger collaborative operation relationship. Using the contact start time of this contact event cluster as the anchor point, the three types of data of touch, force and motion, which were originally misaligned due to the sampling phase difference, are aligned. A period of time is retained before and after the contact for slicing. Finally, a multimodal data segment with state labels such as contact start, stable clamping, sliding when necessary, and release is output.

[0122] Therefore, this step uses the actual physical contact time corresponding to the contact event cluster as the alignment anchor point. Compared with alignment methods that rely on a unified clock, manual annotation, or visual frames, it can eliminate the sampling phase difference and transmission delay between different sensors, and achieve millisecond-level spatiotemporal alignment of tactile, force, and motion data. At the same time, through event cluster slicing and contact state annotation, the output multimodal real device data fragments are strictly aligned in time and have clear contact state labels in semantics, thus providing high-quality real device data for imitation learning and operation strategy training of multi-finger dexterity operation.

[0123] It should be noted that the above-mentioned process, which uses the start of tactile contact as the anchor point and determines the time offset of each modality through cross-correlation of contact evidence signals, first eliminates the relative timing misalignment between the three types of sensors (tactile, force, and motion) caused by differences in sampling frequency, sampling phase, and transmission delay, thus aligning the three modal evidence to the same time reference. Furthermore, to reduce the systematic deviation between the selected anchor point and the actual physical contact time caused by the hysteresis of the tactile sensor itself and the threshold crossing delay, the following implementation method can be adopted: First, the inherent response delay of each sensor (including tactile, wrist torque, and motion sensors) is pre-calibrated. For example, using known calibrated tapping or step contact events in conjunction with high-speed reference measurements, the inherent delay of each modality relative to the actual contact occurrence is measured, and then... First, compensation is applied simultaneously to make the anchor point approximate the actual physical contact moment rather than the tactile detection moment. Second, instead of directly using the moment when the normal contact force crosses the threshold, the change point of the rising edge of the normal contact force or effective contact area is detected, or its rising segment is extrapolated back to the baseline to estimate the moment when the contact force begins to rise as the contact start, thereby reducing the threshold crossing delay component. In applications where only relative alignment of each modality is required, the inherent delay that is not calibrated and compensated manifests as a consistent common offset between samples, without affecting the relative consistency of multimodal data on the same time axis. In applications that require approximation of the actual physical contact moment, compensation is made through the aforementioned inherent delay calibration and contact start estimation.

[0124] In summary, the embodied intelligent multi-finger operation data alignment method of this invention utilizes three mechanisms in synergy: S102, which generates candidate contact events through joint judgment of tactile contact features and action phase features; S103, which eliminates false contacts and determines valid contact events through tactile-force consistency verification and weak coupling auxiliary verification; and S104, which performs multimodal alignment and state labeling with the physical contact time of contact event clusters as anchor points. The initial screening of candidate contact events provides fewer but higher-quality objects for consistency verification; consistency verification and auxiliary verification ensure the credibility of the merged contact event clusters; and credible contact event clusters provide accurate physical anchor points for multimodal alignment. Therefore, this method effectively eliminates various types of false contacts, such as sudden stops in free space, tactile local noise, and accidental finger self-contact, and achieves millisecond-level multimodal spatiotemporal alignment. Ultimately, it provides high-quality, real-device data with contact state labels for strategy learning in embodied intelligent multi-finger dexterous operation.

[0125] The above-mentioned technical effects can be further illustrated by taking the connector insertion and removal task as an example. In this task, there are millisecond-level sampling phase differences between the tactile, force, and motion signals of the finger in each stage of approach, alignment, insertion, and rebound. If this method is not used and the data is only spliced ​​according to the timestamps of each sensor, the peak of wrist force at the moment of insertion will be misaligned with the tactile contact start and fingertip deceleration on the time axis. Moreover, the small force fluctuations in free space during the alignment stage are easily mislabeled as contact, resulting in inaccurate data segments for policy learning in terms of contact time and contact state. With this method, spurious contact in the alignment stage is first eliminated by tactile-force consistency verification. Then, the tactile, force, and motion trimodal data are aligned and sliced ​​using the actual insertion contact time as the anchor point. Finally, high-quality data segments with consistent contact time and labels for contact start, insertion force application, and rebound release state are obtained, thereby significantly improving the sample quality of downstream imitation learning and the success rate of policy training.

[0126] Example 2

[0127] like Figure 3As shown, this embodiment introduces an embodied intelligent multi-finger operation data alignment system, including:

[0128] The data acquisition module 201 is used to acquire multimodal data when the embodied intelligence performs the target operation task and establish the transformation relationship between the data; the multimodal data includes fingertip tactile array data, wrist torque data and multi-finger movement data;

[0129] The feature extraction and judgment module 202 is used to extract tactile contact features based on fingertip tactile array data and extract action stage features based on multi-finger action data, and to judge whether the tactile contact features and action stage features meet the contact conditions. If so, a candidate contact event is generated.

[0130] The feature verification module 203 is used to convert tactile contact features into equivalent wrist torque features based on transformation relationships within the time window corresponding to the candidate contact event, and to calculate the error between the calculated result and the residual of the actual wrist torque features extracted from the wrist torque data. It then determines whether the calculation result does not exceed a threshold; if so, it performs auxiliary verification. Auxiliary verification includes at least one of the following: whether the rate of change of the actual wrist torque feature residual meets the corresponding conditions; whether the decrease in fingertip speed or the amount of finger closure and its rate of change in the action phase features meet the corresponding conditions; and whether the candidate contact times of a preset number of fingers are located within the same collaborative time window. The candidate contact time of a finger is the moment when the contact conditions of the candidate contact event are first met. The collaborative time window is set based on the candidate contact time.

[0131] The data alignment output module 204 is used to merge candidate contact events that have passed auxiliary verification and meet the adjacency conditions into a contact event cluster, use the contact start time of the contact event cluster as the alignment anchor point, perform time alignment on the multimodal data and output it.

[0132] It should be understood that the data acquisition module 201, feature extraction and judgment module 202, feature verification module 203, and data alignment output module 204 in the embodied intelligent multi-finger operation data alignment system according to the embodiments of the present invention are respectively used to implement the corresponding steps of the embodied intelligent multi-finger operation data alignment method in Embodiment 1. The specific implementation of each module can be found in the relevant description of Embodiment 1, and will not be repeated here for the sake of brevity.

[0133] Example 3

[0134] like Figure 4 As shown, this embodiment introduces a personalized intelligent multi-finger operation data alignment device, including:

[0135] An embodied intelligent actuator is used to perform target operational tasks;

[0136] A fingertip tactile array sensor is installed on the fingertip area of ​​at least one finger of an integrated intelligent actuator to collect fingertip tactile array data when the corresponding finger performs a target operation task.

[0137] A wrist torque sensor is installed in the wrist connection area of ​​the embodied intelligent actuator to collect wrist torque data when the embodied intelligent actuator performs target operation tasks;

[0138] Multi-finger motion sensor is used to collect multi-finger motion data of the embodied intelligent actuator when performing target operation tasks;

[0139] The memory is used to store computer programs, as well as fingertip tactile array data, wrist torque data, and multi-finger movement data.

[0140] The processor is connected to a fingertip tactile array sensor, a wrist torque sensor, a multi-finger motion sensor, and a memory. When the processor executes a computer program, it implements the steps of the embodied intelligent multi-finger operation data alignment method described in Example 1.

[0141] according to Figure 4 As shown, in the data flow direction, the fingertip tactile array sensor outputs tactile array data to the processor, the wrist torque sensor outputs wrist torque data to the processor, and the multi-finger motion sensor outputs multi-finger motion data to the processor. After processing the multi-source data in a unified manner, the processor can store the processing results in the memory, or output control commands, time synchronization information, calibration parameters, or aligned data results to the embodied intelligent actuator.

[0142] It should be understood that in the embodiments of the present invention, the processor may be a central processing unit, or other general-purpose processors, digital signal processors, application-specific integrated circuits, off-the-shelf programmable gate arrays, or other programmable logic devices; the memory may include read-only memory and random access memory, and provides instructions and data to the processor. The steps of the above method can be implemented by integrated logic circuits in the processor hardware or by instructions in software form; to avoid repetition, they will not be described in detail here.

[0143] Example 4

[0144] This embodiment introduces an electronic device, including a memory and a processor. The memory stores a computer program, and when the processor executes the computer program, it implements the steps of the embodied intelligent multi-finger operation data alignment method as described in Embodiment 1.

[0145] The embodied intelligent multi-finger operation data alignment method of Example 1 can be applied in software form, such as by designing it as a standalone program and installing it on an electronic device, which can be a computer, smartphone, etc. Alternatively, it can be designed as an embedded program and installed on a computer terminal, such as a microcontroller.

[0146] The technical scope of this invention is not limited to the content described above. Those skilled in the art can make various modifications and variations to the above embodiments without departing from the technical concept of this invention, and all such modifications and variations should fall within the protection scope of this invention.

Claims

1. A method for aligning data for embodied intelligent multi-finger operation, characterized in that, Includes the following steps: Acquire multimodal data when embodied intelligence performs target operation tasks and establish transformation relationships between data; multimodal data includes fingertip tactile array data, wrist torque data, and multi-finger movement data; Tactile contact features are extracted based on fingertip tactile array data, and action phase features are extracted based on multi-finger action data. It is then determined whether the tactile contact features and action phase features meet the contact conditions. If so, candidate contact events are generated. Within the time window corresponding to the candidate contact event, the tactile contact features are converted into equivalent wrist torque features based on the transformation relationship. Error calculation is then performed between this conversion and the residual of the actual wrist torque features extracted from the wrist torque data. The result is used to determine if the calculation does not exceed the threshold; if so, auxiliary verification is performed. Auxiliary verification includes at least one of the following: whether the rate of change of the actual wrist torque feature residual meets the corresponding conditions; whether the decrease in fingertip speed or the amount of finger closure and its rate of change in the action phase characteristics meet the corresponding conditions; and whether the candidate contact times of a preset number of fingers are located within the same collaborative time window. The candidate contact time of a finger is the moment when the contact conditions of the candidate contact event are first met. The collaborative time window is set based on the candidate contact time. Candidate contact events that pass auxiliary verification and meet the adjacency condition are merged into contact event clusters. The contact start time of the contact event cluster is used as the alignment anchor point to perform time alignment on the multimodal data and output it.

2. The embodied intelligent multi-finger operation data alignment method according to claim 1, characterized in that, The transformation relationship is based on the pre-established transformation relationship between the embodied intelligent finger tactile sensor coordinate system and the wrist base coordinate system.

3. The embodied intelligent multi-finger operation data alignment method according to claim 2, characterized in that, In extracting tactile contact features based on fingertip tactile array data, the tactile contact features include normal contact force, effective contact area, and pressure center, specifically including: The pressure value and area of ​​each tactile unit of the fingertip tactile array are obtained, and the normal contact force of the finger is calculated accordingly. The effective contact area is calculated by combining the area of ​​the tactile unit whose pressure value exceeds the threshold, and the pressure center is determined based on the center position and pressure value of each tactile unit.

4. The embodied intelligent multi-finger operation data alignment method according to claim 3, characterized in that, In extracting action phase features based on multi-finger motion data, the action phase features include the decrease in fingertip speed, the degree of finger closure, and the action phase state, specifically including: Acquire multi-finger movement data, including the joint angles of the fingers and their fingertip positions in the wrist coordinate system; The fingertip velocity is calculated based on the fingertip position at adjacent moments, and the decrease in fingertip velocity is calculated based on the magnitude of the fingertip velocity at the current moment and the previous moment. The finger joint angles are weighted and summed to obtain the finger closure amount; the action stage state is generated based on the decrease in fingertip speed and the comparison between the finger closure amount and the corresponding threshold. The action stage state includes the closure stage, deceleration stage, and force application stage.

5. The embodied intelligent multi-finger operation data alignment method according to claim 4, characterized in that, The contact conditions are that the normal contact force of the current finger exceeds the normal contact force threshold, the effective contact area exceeds the contact area threshold, and the finger is in an action phase state. Once a candidate contact event is identified, a short time window is formed by extending the candidate contact time of the candidate contact event by a preset duration. The average pressure center is calculated by averaging the pressure center within the short time window. The average pressure center is then used as the tactile contact position and transformed to the wrist coordinate system through a transformation relationship.

6. The embodied intelligent multi-finger operation data alignment method according to claim 5, characterized in that, The transformation of tactile contact features into equivalent wrist torque features based on the transformation relationship is as follows: The lever arm is determined based on the positional relationship between the tactile contact position of the finger in the wrist coordinate system and the origin of the wrist coordinate system. The normal contact force of the fingers is transformed into the wrist base coordinate system through a transformation relationship, and then coupled with the lever arm to obtain the equivalent wrist torque; The equivalent wrist torque of multiple fingers within a candidate contact event is synthesized to obtain the equivalent wrist torque feature corresponding to the candidate contact event. Before performing auxiliary verification, the priority of auxiliary verification conditions is determined based on the type of the target operation task, and auxiliary verification is performed accordingly.

7. The embodied intelligent multi-finger operation data alignment method according to claim 5, characterized in that, Candidate contact events that pass auxiliary verification and meet the adjacency conditions are merged into a contact event cluster. The adjacency conditions include: the time interval between the candidate contact times of two candidate contact events is less than a preset event merging time window; when two candidate contact events correspond to the same finger, the distance between the tactile contact positions is less than a preset spatial distance threshold; or when two candidate contact events correspond to different fingers, they belong to the same multi-finger collaborative operation relationship.

8. A personalized intelligent multi-finger operation data alignment system, characterized in that, Its application is the embodied intelligent multi-finger operation data alignment method as described in any one of claims 1-7, and the system includes: The data acquisition module is used to acquire multimodal data when the embodied intelligence performs target operation tasks and establish transformation relationships between data; the multimodal data includes fingertip tactile array data, wrist torque data, and multi-finger movement data; The feature extraction and judgment module is used to extract tactile contact features based on fingertip tactile array data and extract action phase features based on multi-finger action data, and to determine whether the tactile contact features and action phase features meet the contact conditions. If so, candidate contact events are generated. The feature verification module is used to convert tactile contact features into equivalent wrist torque features based on transformation relationships within the time window corresponding to candidate contact events. It then calculates the error between the converted feature and the residual of the actual wrist torque features extracted from the wrist torque data, determining whether the calculation result does not exceed a threshold. If so, auxiliary verification is performed. Auxiliary verification includes at least one of the following: whether the rate of change of the actual wrist torque feature residual meets the corresponding conditions; whether the decrease in fingertip speed or the amount of finger closure and its rate of change in the action phase feature meet the corresponding conditions; and whether the candidate contact times of a preset number of fingers are located within the same collaborative time window. The candidate contact time of a finger is the moment when the contact conditions of the candidate contact event are first met. The collaborative time window is set based on the candidate contact time. The data alignment output module is used to merge candidate contact events that have passed auxiliary verification and meet the adjacency condition into contact event clusters, use the contact start time of the contact event cluster as the alignment anchor point, perform time alignment on the multimodal data and output it.

9. A data alignment device for embodying intelligent multi-finger operation, characterized in that, include: An embodied intelligent actuator is used to perform target operational tasks; A fingertip tactile array sensor is disposed in the fingertip area of ​​at least one finger of a self-contained intelligent actuator, for collecting fingertip tactile array data when the corresponding finger performs the target operation task; A wrist torque sensor is installed in the wrist connection area of ​​the embodied intelligent actuator to collect wrist torque data when the embodied intelligent actuator performs the target operation task; A multi-finger motion sensor is used to collect multi-finger motion data of the embodied intelligent actuator when performing the target operation task; The memory is used to store computer programs, as well as fingertip tactile array data, wrist torque data, and multi-finger movement data. The processor is connected to the fingertip tactile array sensor, the wrist torque sensor, the multi-finger motion sensor, and the memory. When the processor executes a computer program, it implements the steps of the embodied intelligent multi-finger operation data alignment method as described in any one of claims 1-7.

10. An electronic device comprising a memory and a processor, the memory storing a computer program, characterized in that, When the processor executes a computer program, it implements the steps of the embodied intelligent multi-finger operation data alignment method as described in any one of claims 1-7.

Images