Body data acquisition method, system and computer readable storage medium

Abstract

The application discloses a body data acquisition method and system and a computer readable storage medium, relates to the technical field of data acquisition, and is applied to a body data acquisition system. The body data acquisition system comprises a wearable device and a head-mounted device. The image acquisition direction of the head-mounted device is consistent with the line of sight direction of the wearer. The method comprises the following steps: acquiring bioelectric impedance signals collected by the wearable device and image data collected by the head-mounted device; determining a finger joint angle based on the bioelectric impedance signals and / or the image data, and determining a hand operation strength based on the finger joint angle and the bioelectric impedance signals; and obtaining a body data set based on the image data, the finger joint angle and the hand operation strength. The application realizes the collection of high-quality body data without changing the natural bare-hand operation state of an operator.

Description

Technical Field

[0001] This application relates to the field of data acquisition technology, and in particular to an embodied data acquisition method, system, and computer-readable storage medium. Background Technology

[0002] Embodied intelligence is a crucial development direction in the field of robotics, its core being enabling intelligent agents to acquire operational capabilities through interaction with the physical environment. Embodied data serves as the fundamental resource for training embodied models, and the method of its acquisition directly impacts the model's understanding of real-world operational behaviors. Therefore, how to efficiently acquire high-quality embodied data has become a key issue in this field.

[0003] Existing methods for acquiring embodied data mainly fall into two categories: one is based on robot teleoperation, where an operator remotely controls the robot and records its status data; the other relies on wearable hand devices, requiring operators to wear data gloves, exoskeletons, or hand grippers, etc., and capture hand and interaction information through embedded sensors.

[0004] However, all of the above methods have significant shortcomings: teleoperation is limited by specific robot platforms and has poor data transferability; wearable hand devices alter natural tactile sensation and operating habits, making calibration complex and lacking versatility. Therefore, how to achieve high-quality embodied data collection without altering the operator's natural bare-hand operating state has become an urgent technical problem to be solved. Summary of the Invention

[0005] The main objective of this application is to provide a method, system, and computer-readable storage medium for acquiring embodied data, aiming to solve the technical problem of how to achieve high-quality embodied data acquisition without changing the operator's natural bare-handed operating state.

[0006] To achieve the above objectives, this application provides an embodied data acquisition method, applied to an embodied data acquisition system. The embodied data acquisition system includes a wearable device and a head-mounted device. The image acquisition direction of the head-mounted device is consistent with the wearer's line of sight. The embodied data acquisition method includes the following steps: Acquire bioelectrical impedance signals from wearable devices and image data from head-mounted devices; Finger joint angles are determined based on bioelectrical impedance signals and / or image data, and hand manipulation force is determined based on finger joint angles and bioelectrical impedance signals. An embodied dataset is obtained based on image data, finger joint angles, and hand manipulation force.

[0007] In one embodiment, the step of determining the force of hand manipulation based on finger joint angles and bioelectrical impedance signals includes: Gesture operation type is recognized based on finger joint angle; The force regression model is based on matching the type of gesture operation, where different types of gesture operations correspond to different force regression models; The bioelectrical impedance signal is input into a matched force regression model, and the output is the hand operation force.

[0008] In one embodiment, prior to the step of recognizing the gesture operation type based on the finger joint angle, the method further includes: Monitor the changes in finger joint angles within a preset time period; If the change is less than or equal to a preset threshold, then the step of recognizing the gesture operation type based on the finger joint angle is executed; If the change exceeds a preset threshold, the process returns to the steps of acquiring bioelectrical impedance signals collected by the wearable device and image data collected by the head-mounted device.

[0009] In one embodiment, after the step of recognizing the gesture operation type based on the finger joint angle, the method further includes: If the gesture operation type belongs to the preset operation type, then the step of matching the corresponding force regression model based on the gesture operation type is executed. The preset operation type includes one or more of pressing, pinching, grasping and twisting. If the gesture operation type is not a preset operation type, return to the step of acquiring the bioelectrical impedance signal collected by the wearable device and the image data collected by the head-mounted device.

[0010] In one embodiment, the step of determining the finger joint angle based on bioelectrical impedance signals and / or image data includes: Feature extraction is performed on the bioelectrical impedance signal to obtain impedance signal features. These features are then input into a pre-trained first angle regression model, which outputs the finger joint angle; or... Visual features are extracted from image data and then input into a pre-trained second angle regression model to output the finger joint angles; or... Visual features and bioelectrical impedance signal features are extracted from image data and bioelectrical impedance signal respectively. The visual features and bioelectrical impedance signal features are fused to obtain fused features. The fused features are input into a pre-trained third-angle regression model to output the finger joint angle.

[0011] In one embodiment, the step of extracting features from a bioelectrical impedance signal to obtain impedance signal features includes: The bioelectrical impedance signal was sequentially subjected to bandpass filtering and notch filtering to obtain the preprocessed bioelectrical impedance signal. The bioelectrical impedance signal features were obtained by feature extraction from the preprocessed bioelectrical impedance signal.

[0012] In one embodiment, the step of extracting visual features from image data includes: The image data is sequentially subjected to denoising, hand region segmentation, and normalization to obtain preprocessed image data; Visual features are obtained by extracting features from the preprocessed image data.

[0013] In one embodiment, the step of obtaining an embodied dataset based on image data, finger joint angles, and hand manipulation force includes: The image data, finger joint angles, and hand manipulation force are time-aligned, and the time-aligned image data, finger joint angles, and hand manipulation force are encapsulated in a format to obtain an embodied dataset; or, Hand posture and gesture operation type are determined based on finger joint angles. The image data, finger joint angles, hand operation force, hand posture, and gesture operation type are time-aligned. The time-aligned image data, finger joint angles, hand operation force, hand posture, and gesture operation type are encapsulated in a format to obtain an embodied dataset.

[0014] Furthermore, to achieve the above objectives, this application also provides an embodied data acquisition system, which includes: Wearable devices used to collect bioelectrical impedance signals; A head-mounted device used to acquire image data, and the image acquisition direction of the head-mounted device is consistent with the wearer's line of sight; The processing unit is used to execute the steps of the embodied data acquisition method described above.

[0015] In addition, to achieve the above objectives, this application also provides a readable storage medium, which is a computer-readable storage medium storing a computer program that is executed by a processor to implement the steps of the above-described embodied data acquisition method.

[0016] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the embodied data acquisition method described above.

[0017] One or more technical solutions proposed in this application have at least the following technical effects: This application utilizes bioelectrical impedance signals collected by a wearable device (suitable for wearing on the operator's body, such as the arm or wrist) and image data collected by a head-mounted device aligned with the line of sight to determine finger joint angles and hand manipulation force, ultimately obtaining an embodied dataset. This eliminates the need for the operator to wear a hand-worn device or rely on a specific robotic platform for remote operation. The wearable device only collects bioelectrical impedance signals without contacting the hand, allowing the hand to remain in a natural, bare state, completely undisturbed by existing tactile perception and operational habits. In determining finger joint angles, bioelectrical impedance signals reflect the muscle activity controlling finger movement, while image data provides visual information of the hand aligned with the operator's line of sight. Both can be used selectively or complement each other as needed to accurately reconstruct joint angles. Furthermore, the inherent correlation between the determined joint angles and bioelectrical impedance signals is used to calculate the manipulation force, fully recording the fine posture and mechanical characteristics during natural operation. The entire data acquisition process does not apply any additional devices or constraints to the hands, thus avoiding motion distortion and calibration errors caused by external contact. The resulting embodied dataset is fundamentally guaranteed in terms of motion authenticity and information richness, thereby achieving the acquisition of high-quality embodied data without changing the operator's natural bare-hand operation state. Attached Figure Description

[0018] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0019] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0020] Figure 1 This is a flowchart illustrating the first embodiment of the data acquisition method of this application; Figure 2 This is a schematic diagram of the system architecture involved in one embodiment of the data acquisition system of this application; Figure 3 This is a schematic diagram of a process for determining finger joint angles based on image signals, according to an embodiment of the data acquisition system of this application. Figure 4 This is a schematic diagram of a process for determining finger joint angles based on bioelectrical impedance signals, according to an embodiment of the data acquisition system of this application. Figure 5 This is a schematic diagram illustrating the process of determining finger joint angles by fusing image signals and bioelectrical impedance signals, according to an embodiment of the data acquisition system of this application. Figure 6 This is a schematic diagram illustrating the classification process of gesture operation types in an embodiment of the data acquisition system of this application; Figure 7 This is a schematic diagram of the hand operation force regression prediction process according to an embodiment of the data acquisition system of this application.

[0021] The purpose, features, and advantages of this application will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0022] To make the above-mentioned objects, features, and advantages of the present invention more apparent and understandable, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0023] Universal Manipulation Interface (UMI) is a core technology for robots to achieve generalized operation learning and large-scale deployment. Its core idea and goal is to collect multimodal operation information from human operation processes through lightweight and standardized human-computer interaction methods, independent of the robot itself. This provides data support for robot strategy learning and action replication without relying on robot hardware, thereby enabling robots to have general operation capabilities for various everyday objects.

[0024] Existing general-purpose operation interface solutions typically rely primarily on visual information to obtain the operation trajectory. However, to acquire more nuanced feedback information such as force, touch, and joint angles, it is necessary to install force / torque sensors or tactile sensors on the robot's end effector, or to use data gloves or exoskeletons worn by the user. These methods have significant drawbacks: 1. Relying on dedicated sensors at the end point or object end results in high hardware costs and complex installation and calibration processes, which is not conducive to lightweight, portable, and plug-and-play deployment, and makes it difficult to achieve large-scale popularization in real unstructured scenarios such as home and office. 2. The sensing device is strongly coupled with the robot end effector or the object being manipulated, resulting in poor versatility. Re-adaptation is required when changing the robot platform or the object being manipulated. Sensors are susceptible to the influence of object material, shape, and installation location, resulting in limited stability and adaptability to different scenarios. 3. It is difficult to accurately capture the gesture logic, joint movement patterns and force exertion mechanisms in the real human operation process, and it is impossible to fully obtain the core information of human operation, making it difficult to support the learning and replication of fine operation of robot-like human hands.

[0025] Currently, there is no existing technology that can completely eliminate the need for robotic end-effectors and wearable devices such as human hand exoskeletons and data gloves, and achieve precise perception of finger joint angles and operational force through bare hands alone, enabling multimodal data acquisition and processing.

[0026] In summary, at least the following three key issues need to be addressed: 1. Existing UMI data acquisition solutions for robots rely on deploying dedicated force / tactile sensors on the end effector. This not only results in high hardware costs but also complex installation, debugging, and calibration processes. Furthermore, it places high demands on the deployment environment, making it difficult to adapt to real-world unstructured scenarios such as homes and offices. Consequently, it cannot achieve large-scale, low-cost operational data acquisition, which hinders the widespread adoption and practical application of the interface.

[0027] 2. The existing perception module is deeply bound to the robot platform and end effector. The perception parameters and calibration logic all depend on specific hardware configurations. When changing the platform, end effector type or operation object, recalibration and adaptation are required. The process is cumbersome and inefficient, resulting in poor interface universality and insufficient scalability, making it difficult to flexibly adapt to multiple types of robot operation scenarios.

[0028] 3. Existing solutions mostly focus on visual modal gesture and joint angle recognition, and have not formed a complete framework of "multimodal acquisition - hierarchical algorithm processing - standardized data output". It is difficult to synchronously and stably acquire information on gesture operation type, finger joint angle and continuous operation force. It is impossible to fully reproduce the gesture logic, joint movement law and force mechanism of human operation, which makes it difficult for robots to learn human-like hand operation logic, unable to achieve the fine operation replication at the level of dexterous hand, and unable to provide standardized and complete training data for the large-scale embodied intelligence model of robots.

[0029] To address the aforementioned issues, the main solution of this application is as follows: acquiring bioelectrical impedance signals collected by a wearable device and image data collected by a head-mounted device; determining finger joint angles based on bioelectrical impedance signals and / or image data, and determining hand manipulation force based on finger joint angles and bioelectrical impedance signals; and obtaining an embodied dataset based on image data, finger joint angles, and hand manipulation force.

[0030] This application utilizes bioelectrical impedance signals collected by a wearable device and image data aligned with the operator's line of sight collected by a head-mounted device to determine finger joint angles and hand manipulation force, ultimately obtaining an embodied dataset. This eliminates the need for the operator to wear a hand-worn device or rely on a specific robotic platform for remote operation. The wearable device only collects bioelectrical impedance signals without touching the hand, allowing the hand to remain in a natural, bare state, completely undisturbed by existing tactile perception and operational habits. In determining finger joint angles, bioelectrical impedance signals reflect the muscle activity controlling finger movement, while image data provides visual information of the hand aligned with the operator's line of sight. Both can be used selectively or complement each other as needed to accurately reconstruct joint angles. Furthermore, the inherent correlation between the determined joint angles and bioelectrical impedance signals is used to calculate the manipulation force, fully recording the fine posture and mechanical characteristics during natural operation. The entire data acquisition process does not apply any additional devices or constraints to the hands, thus avoiding motion distortion and calibration errors caused by external contact. The resulting embodied dataset is fundamentally guaranteed in terms of motion authenticity and information richness, thereby achieving the acquisition of high-quality embodied data without changing the operator's natural bare-hand operation state.

[0031] Based on this, this application proposes a first embodiment of a embodied data acquisition method. In this embodiment, the embodied data acquisition method is applied to an embodied data acquisition system, which includes a wearable device and a head-mounted device. The image acquisition direction of the head-mounted device is consistent with the wearer's line of sight. The embodied data acquisition method includes the following steps S10~S30: Step S10: Acquire bioelectrical impedance signals collected by the wearable device and image data collected by the head-mounted device; Wearable devices refer to bioelectrical impedance signal acquisition devices worn on parts of the body other than the hands, and are distinct from hand-worn devices.

[0032] Head-mounted devices, such as head-mounted cameras, AR (Augmented Reality) glasses, or VR (Virtual Reality) glasses, ensure that the optical axis for image acquisition is aligned with the wearer's line of sight, thereby guaranteeing that the acquired image is always focused on the operating area currently being viewed by the operator and recording the visual context of the operating scene.

[0033] The system acquires and records bioelectrical impedance signals and image data in real time through hardware synchronization signals or software timestamps. It can further perform baseline filtering and noise reduction preprocessing as needed to provide clean and aligned raw data for subsequent steps.

[0034] Step S20: Determine the finger joint angle based on bioelectrical impedance signals and / or image data, and determine the hand operation force based on the finger joint angle and bioelectrical impedance signals; Finger joint angles specifically refer to the bending angles of the proximal interphalangeal joints, distal interphalangeal joints, and metacarpophalangeal joints of each finger, encompassing multiple degrees of freedom.

[0035] When using bioelectrical impedance signals alone, a pre-trained temporal neural network model, such as TCN (Temporal Convolutional Network) or LSTM (Long Short-Term Memory), can be used to map multi-channel bioelectrical impedance signals to finger joint angle values, which is suitable for scenarios with severe visual occlusion.

[0036] When using image data alone, hand pose estimation networks (such as MediaPipe Hands, Google's hand keypoint detection framework) can be used to extract 2D or 3D finger joint keypoints from the image, and then calculate the angles of each joint.

[0037] When both are used simultaneously, a multimodal fusion architecture can be designed to combine the temporal resolution of bioelectrical impedance signals with the spatial accuracy of vision, thereby improving the noise resistance and accuracy of angle estimation in dynamic operations.

[0038] After obtaining the finger joint angle, the contact force or operational force applied by the finger is calculated using a force regression model in conjunction with the bioelectrical impedance signal at the same time. This force can be quantified as fingertip pressure, twisting force, pinching force or joint driving torque, and output as the force value at the current moment.

[0039] Step S30: Based on image data, finger joint angles, and hand manipulation force, obtain embodied dataset.

[0040] The generated raw and derived information is structured and integrated: image data provides visual information about the operation scene, used to record the spatial layout and dynamic changes of the operation object and the environment; finger joint angles can be described in the form of quaternions or Euler angles to describe the hand movement trajectory and posture changes; the force of hand operation is stored in Newton or relative dimensions to preserve the mechanical characteristics of the interaction process.

[0041] The system can precisely align visual information, joint angles, and force data at the frame level according to a unified time reference. After alignment, the data can be encapsulated in a universal storage format, supporting random access and streaming, and can be directly used in multiple scenarios for robot embodied learning. For example, this dataset can be used for: Vision-Language-Action (VLA) pre-training, helping the model learn the movement patterns and force mechanisms of human bare hand operations; action retargeting, mapping finger joint angles and operational forces in the dataset to the robot's dexterous hand to achieve the replication of human-like fine-grained operations; policy learning, training the robot's general operation strategies based on the operation logic in the dataset to improve its ability to manipulate everyday objects; and model fine-tuning, used to optimize the robot's robustness and adaptability in unstructured scenarios.

[0042] Based on the first embodiment of this application, in the second embodiment of this application, the content that is the same as or similar to that in the first embodiment described above can be referred to the above description and will not be repeated hereafter. Based on this, the step of determining the hand manipulation force based on the finger joint angle and bioelectrical impedance signal includes: Step A10: Recognize the gesture operation type based on the finger joint angle; Finger joint angles reflect the changes in hand posture and movement patterns when performing different operations. By performing time-domain or frequency-domain analysis on finger joint angles in consecutive frames, features such as joint angle thresholds, rates of change, and cooperative motion relationships can be extracted. Pre-trained classifiers (such as support vector machines, decision trees, or lightweight neural networks) can then be used to identify the gesture type of the current operation. Common gesture types include, but are not limited to: pressing (e.g., applying vertical force with a single finger or the base of the palm), pinching (e.g., pinching a small object with fingertips), grasping (e.g., grasping with the entire hand in a wrapping motion), and twisting (e.g., applying force by rotating the thumb and forefinger relative to each other).

[0043] Step A20: Match the corresponding force regression model based on the gesture operation type, where different gesture operation types correspond to different force regression models; Different types of hand gestures (pressing, pinching, grasping, twisting) involve different muscle group coordination patterns and force transmission paths, thus requiring the establishment of targeted force mapping relationships. The system has multiple preset force regression models, corresponding to the pressing model, pinching model, grasping model, and twisting model, respectively. These models can be simplified physical models based on biomechanical principles, or regressors (such as linear regression, random forest, or shallow neural networks) trained after offline collection and calibration data.

[0044] Based on the identified gesture type, a matching force regression model is dynamically selected to ensure the accuracy of force estimation.

[0045] Step A30: Input the bioelectrical impedance signal into the matched force regression model and output the hand operation force.

[0046] After determining the force regression model corresponding to the current gesture operation type, the bioelectrical impedance signal collected at the same time is used as input and fed into the selected model for calculation. Based on the force mapping relationship specific to the gesture type, the model outputs the hand operation force at the current operation moment. This force can be expressed as fingertip or palm heel pressure for pressing, fingertip pinching force for pinching, gripping force for grasping, and twisting torque for twisting.

[0047] This embodiment identifies the current gesture operation type based on the finger joint angle. Since different gesture operation types correspond to different muscle coordination modes and force transmission paths, the system presets independent force regression models for each type of gesture and dynamically matches and selects the appropriate model based on the recognition results. The bioelectrical impedance signal is input into the force regression model matching the gesture, making the mapping relationship from bioelectrical impedance signal to mechanical output more consistent with the actual force exertion mechanism of the current gesture. This method avoids the bias introduced by using a single general model for aliasing estimation of different gestures, thereby effectively improving the estimation accuracy and specificity of hand operation force, and making the final output force value more realistically reflect the operator's actual force application.

[0048] In one possible implementation, prior to the step of recognizing the gesture operation type based on the finger joint angle, the method further includes: Step B10: Monitor the change in finger joint angles within a preset time period; During continuous data acquisition, the system calculates the change in finger joint angles in real time within a preset time period (e.g., 0.5 seconds or 1 second). This change can be expressed as the sum of the absolute values ​​of the differences between the finger joint angle values ​​within that time period, the maximum change in a single joint, or the variance of the joint angle sequence, among other statistical indicators.

[0049] The preset duration can be adjusted according to the speed of the operation. For example, a longer time window can be set for delicate and slow operations, and a shorter time window can be set for fast operations.

[0050] Step B20: If the change is less than or equal to a preset threshold, then execute the step of recognizing the gesture operation type based on the finger joint angle. When the change in finger joint angle within a preset time period does not exceed a preset threshold, it indicates that the hand posture is in a relatively stable stage. At this time, the operator is continuously performing a certain gesture operation (such as maintaining a pressing, grasping, or twisting state), rather than being in a motion switching process. In this stable state, the system determines that it has favorable conditions for performing gesture type recognition and force estimation, and therefore proceeds to the next step, identifying the gesture operation type based on the currently stable finger joint angle, and further performing force regression.

[0051] Step B30: If the change is greater than a preset threshold, return to the step of acquiring the bioelectrical impedance signal collected by the wearable device and the image data collected by the head-mounted device.

[0052] When the change in finger joint angle exceeds a preset threshold within a preset time period, it indicates that the hand is undergoing significant movement or a rapid shift in posture (e.g., from pinching to grasping, or finger extension followed by flexion). During this dynamic transition phase, the gesture type is not yet stable, and directly performing type recognition and force estimation is prone to errors. Therefore, the system does not perform gesture recognition and force calculation, but instead returns to step S10 to continue acquiring new bioelectrical impedance signals and image data, restarting the monitoring of changes in finger joint angle until a stable state is detected before proceeding with subsequent processing.

[0053] This cyclical mechanism effectively reduces unreliable identification results during the dynamic transition phase, improving the robustness of the entire acquisition process.

[0054] In one possible implementation, after the step of recognizing the gesture operation type based on the finger joint angle, the method further includes: Step C10: If the gesture operation type belongs to the preset operation type, then execute the step of matching the corresponding force regression model based on the gesture operation type. The preset operation type includes one or more of pressing, pinching, grasping and twisting. When the identified gesture type falls into a preset set of interest types (such as pressing, pinching, grasping, twisting, etc., which are operation types with high value for embodied data collection), the system determines that the current operation is suitable for subsequent force estimation. At this time, the system continues to execute subsequent steps, matching the corresponding force regression model according to the specific gesture type to ensure the relevance of subsequent force output.

[0055] Step C20: If the gesture operation type is not a preset operation type, return to the step of acquiring the bioelectrical impedance signal collected by the wearable device and the image data collected by the head-mounted device.

[0056] When the identified gesture type is not in the preset set (e.g., meaningless stretching, relaxing, or other atypical gestures), in order to avoid invalid data consuming processing resources or introducing noise, the system does not perform subsequent force regression, but returns to step S10 to continue acquiring new bioelectrical impedance signals and image data, and restarts the acquisition process.

[0057] This filtering mechanism improves collection efficiency and ensures that all data fragments in the final embodied dataset come from preset valid operation types.

[0058] Based on the first and / or second embodiments of this application, in the third embodiment of this application, the content that is the same as or similar to the first and second embodiments described above can be referred to the above description and will not be repeated hereafter. Based on this, the step of determining the finger joint angle based on bioelectrical impedance signals and / or image data includes: Step D10: Extract features from the bioelectrical impedance signal to obtain impedance signal features, input the impedance signal features into the pre-trained first angle regression model, and output the finger joint angle. The acquired bioelectrical impedance signals are preprocessed, including filtering, denoising, and sliding window segmentation. Then, time-domain features (such as root mean square and envelope) and frequency-domain features (such as median frequency and average power frequency) are extracted to construct the impedance signal feature vector.

[0059] The feature vector is input into a pre-trained first angle regression model, which can employ a temporal neural network or a fully connected regression network to output the angle values ​​of each finger joint at the current moment. This method is suitable for scenarios with visual occlusion or poor lighting conditions, utilizing bioelectrical impedance signals to achieve continuous angle estimation.

[0060] Alternatively, in step D20, visual features are extracted from the image data, and the visual features are input into the pre-trained second angle regression model to output the finger joint angle. The hand region can be extracted from acquired images (such as RGB images or depth maps), and then visual features are obtained through a visual feature extraction network, such as a convolutional neural network or a hand pose estimation network. Extracted features may include hand keypoint heatmaps, relative joint positions, etc. The extracted visual features are then input into a pre-trained second angle regression model, which is based on fully connected layers, graph convolutional networks, or other regression structures, and outputs the angle values ​​of each finger joint.

[0061] This method is suitable for scenarios where bioelectrical impedance signals are interfered with by motion artifacts or electrode displacement, and uses spatial information from vision to obtain accurate joint angles.

[0062] Alternatively, in step D30, visual features and electrical impedance signal features are extracted from the image data and the bioelectrical impedance signal respectively. The visual features and electrical impedance signal features are fused to obtain fused features. The fused features are input into a pre-trained third-angle regression model to output the finger joint angle.

[0063] When using image data and bioelectrical impedance signals simultaneously, visual features and bioelectrical impedance signal features are first extracted according to steps D20 and D10, respectively. Then, the two types of features are fused at the feature level. The fusion method can employ feature concatenation, weighted summation, or adaptive fusion based on an attention mechanism. The resulting fused features are input into a pre-trained third-angle regression model, which is typically designed as a multimodal fusion architecture (e.g., a multi-branch neural network) to combine the spatial accuracy of vision with the temporal resolution of bioelectrical impedance, outputting more robust and accurate finger joint angles during dynamic operations.

[0064] This method is suitable for acquisition scenarios that require high precision and high stability, and can complement the advantages of the two modes.

[0065] In one possible implementation, the step of extracting features from the bioelectrical impedance signal to obtain impedance signal features includes: Step F10: Bandpass filtering and notch filtering are performed sequentially on the bioelectrical impedance signal to obtain the preprocessed bioelectrical impedance signal. The raw bioelectrical impedance signal was bandpass filtered, with the passband range set according to the effective frequency domain range of impedance modulation caused by hand muscle movements. This preserved the impedance response components related to changes in hand movements while suppressing low-frequency motion artifacts and high-frequency electromagnetic spurious noise. Next, notch filtering was performed, with the notch center frequency set according to the local power frequency interference frequency to filter out periodic fixed-frequency interference introduced by mains equipment. The signal-to-noise ratio of the bioelectrical impedance signal after these processing steps was significantly improved, providing a high-purity signal base for subsequent feature extraction.

[0066] Step F20: Feature extraction is performed on the preprocessed bioelectrical impedance signal to obtain the impedance signal features.

[0067] After obtaining the preprocessed bioelectrical impedance signal, a sliding time window is used to segment the continuous impedance time-series signal into frames. The window length and sliding step size are set according to the model requirements. Impedance features characterizing the degree of hand muscle deformation and movement state are extracted within each time window. The extracted impedance signal features include time-domain and frequency-domain features. Time-domain features can include root mean square, energy envelope, waveform length, zero-crossing rate, etc., while frequency-domain features can include median frequency, average power frequency, etc. Higher-order statistics and autoregressive model coefficients can also be selected as supplementary features. The extracted multi-dimensional impedance signal features are used as the joint input of the angle regression model and the force regression model.

[0068] In one possible implementation, the step of extracting visual features from image data includes: Step G10: Denoising, hand region segmentation and normalization are performed on the image data in sequence to obtain preprocessed image data; The raw images acquired by the head-mounted device are denoised using methods such as median filtering, Gaussian filtering, or bilateral filtering to suppress sensor noise and environmental interference. Then, hand region segmentation is performed, using skin color detection, background subtraction, or deep learning-based segmentation networks (such as U-Net) to extract the pixel region containing the operator's hand and remove irrelevant background information. Finally, the segmented hand region is normalized, including size normalization (scaling to a uniform resolution) and pixel value normalization (e.g., scaling to the 0-1 range or zero-mean standardization), resulting in preprocessed image data that provides standardized input for subsequent feature extraction.

[0069] Step G20: Extract visual features from the preprocessed image data.

[0070] After obtaining the preprocessed hand region image, a visual feature extraction network is used to extract discriminative visual features. These features may include hand keypoint heatmaps (e.g., probability distributions of finger joint and fingertip positions), hand pose encodings (e.g., joint relative positions or angle embeddings), or deep feature maps learned through a convolutional neural network. The extracted visual features will serve as input to the angle regression model in subsequent steps to determine finger joint angles. The specific network structure for feature extraction can be selected based on actual needs, such as the lightweight MobileNet or high-precision ResNet variants.

[0071] In one possible implementation, the step of obtaining an embodied dataset based on image data, finger joint angles, and hand manipulation force includes: Step H10: Perform time-series alignment processing on the image data, finger joint angles, and hand operation force, and encapsulate the time-series aligned image data, finger joint angles, and hand operation force to obtain the embodied dataset. During the data acquisition process, image data, finger joint angles, and hand manipulation force may originate from different sensors (head-mounted cameras, bioelectrical impedance analysis modules, etc.), and their sampling rates and time bases may differ. First, the three data points are time-aligned according to a unified timeline, which can be achieved using hardware synchronization signals or software interpolation methods, ensuring that the visual information, joint angles, and force values ​​at each moment are precisely correlated in time.

[0072] The aligned multimodal data is further encapsulated, for example, organized into a frame sequence structure, where each frame contains a synchronized image, angle vectors of each finger joint, and a force scalar or vector at the current moment. Finally, the encapsulated data structure is stored as a standardized embodied dataset, which can be directly used for model training later.

[0073] Alternatively, in step H20, the hand posture and gesture operation type are determined based on the finger joint angles. The image data, finger joint angles, hand operation force, hand posture, and gesture operation type are time-aligned. The time-aligned image data, finger joint angles, hand operation force, hand posture, and gesture operation type are formatted and encapsulated to obtain the embodied dataset.

[0074] Higher-level semantic information can be deduced from the determined finger joint angles. Based on the angle values ​​of each finger joint, a complete hand posture (such as outstretched, clenched fist, pinched, etc.) can be reconstructed by combining a simplified human hand skeleton model; at the same time, the specific gesture operation type (such as pressing, pinching, grasping, twisting, etc.) can be identified based on the joint angle sequence.

[0075] Subsequently, the original acquired image data, finger joint angles, hand operation force, and derived hand posture and gesture operation type are all subjected to time alignment processing to ensure that all information corresponds to each other on the same time reference.

[0076] Finally, the data is packaged into a single sample unit, combining the image data, finger joint angles, hand manipulation force, hand posture, and gesture type at each moment, forming a richer embodied dataset. This dataset not only contains low-level sensor signals but also integrates high-level behavioral semantics, providing more comprehensive supervision information for embodied intelligent models. Furthermore, the reconstructed complete hand posture can be directly used for robot operation replication.

[0077] Furthermore, embodiments of this application also propose an embodied data acquisition system, which includes: Wearable devices used to collect bioelectrical impedance signals; A head-mounted device used to acquire image data, and the image acquisition direction of the head-mounted device is consistent with the wearer's line of sight; The processing unit is used to execute the steps of the embodied data acquisition method described above.

[0078] For example, in one specific embodiment, refer to Figure 2 As shown, the embodied data acquisition system includes a hardware system, an algorithm inference module mounted on the processing unit, and an embodied intelligent dataset produced by the system. The hardware system consists of a head-mounted device and a wearable device, specifically a forearm bioelectrical impedance signal acquisition device. The head-mounted device is equipped with a camera, and the two work together to collect multimodal sensory signals during human bare-hand operations. The algorithm inference module is responsible for receiving, processing, and parsing the various sensor data collected by the hardware system, ultimately outputting an embodied dataset that can be directly used for robot training.

[0079] The specific implementation of the hardware system is as follows: The forearm bioelectrical impedance signal acquisition device employs an N-channel acquisition architecture and is worn on the operator's wrist or forearm (position can be flexibly adjusted). The sampling frequency is set to F Hz, and the acquired raw bioelectrical impedance signals are transmitted wirelessly (such as via Bluetooth or WiFi) to the algorithm processing module. Furthermore, the device can be equipped with an IMU sensor to assist in acquiring forearm posture information or to analyze wrist spatial pose by combining visual information; the IMU's sampling rate, accuracy, and other parameters can be flexibly configured according to actual needs.

[0080] The head-mounted vision device uses a lightweight head-mounted carrier (which can be a standalone headband, AR glasses, VR glasses with a built-in camera, or other head-mounted devices; a lightweight headband is used as an example in this case). It is equipped with one or more cameras, with a resolution of W×H and a frame rate of S fps. The lens focal length can be adjusted to a suitable range for capturing hand gestures. The device is worn on the operator's head, with the lenses facing the bare hand's operating area, acquiring real-time images of the hand and scene information. The acquired image signals are transmitted to the algorithm processing module via wired or wireless means.

[0081] Time synchronization alignment is used to ensure the temporal consistency of signals acquired by the forearm bioelectrical impedance signal acquisition device and the head-mounted vision device, providing reliable input for subsequent multimodal fusion algorithms, and ensuring that the final output embodied intelligence dataset is accurately aligned on the time axis. Time alignment can be achieved using hard synchronization or soft synchronization: hard synchronization typically achieves simultaneous acquisition by both devices by adding an independent synchronization trigger module; soft synchronization achieves signal timing alignment through software timestamp matching.

[0082] The algorithm inference module comprises three sub-modules: a finger joint angle tracking module, a gesture operation type classification module based on finger joint angles, and a force regression module based on specific gesture operation types. The specific machine learning / deep learning models, data preprocessing, and feature extraction schemes used in each module can be adjusted according to actual needs.

[0083] During the model training phase, the aforementioned hardware system collects multimodal data (including bioelectrical impedance signals and image signals) from different test subjects and under different bare-hand operation scenarios. At the same time, a high-precision camera is used to label the corresponding finger joint angles and gesture operation types, and a pressure sensor is used to label the actual operating force. In this way, a standard dataset is constructed for model training and testing.

[0084] Data preprocessing: Denoising, hand region segmentation, and normalization are performed sequentially on the image signal; bandpass filtering and notch filtering are performed sequentially on the bioelectrical impedance signal to obtain the preprocessed multimodal signal.

[0085] The core function of the finger joint angle tracking module is to achieve accurate tracking of finger joint angles using bioelectrical impedance signal features, visual features, or a combination of both. The algorithm block diagram of this module is shown below. Figures 3 to 5 As shown. Specifically: as Figure 3 As shown, the visual features of the preprocessed image signal 101 can be extracted separately by the visual feature extraction module 102, and the angle regression module 103 outputs the angles of each finger joint; as shown Figure 4 As shown, the feature vectors corresponding to the bioelectrical impedance signal 201 can also be extracted separately through the bioelectrical impedance signal feature extraction module 202, and the angles of each finger joint can be output by the angle regression module 103; for example... Figure 5 As shown, visual features and bioelectrical impedance signal features can also be fused using a feature stitching method. The fused features 301 are then input into the angle regression module 103, which outputs the angles of each finger joint. After the angle regression module 103 outputs the joint angles, it can be combined with a simplified human hand skeleton model to reverse-engineer the complete hand posture 104 for robot operation replication. The specific type of hand model and the kinematic constraints are not limited; only accurate output of joint angles and optional hand shape restoration are required.

[0086] The core function of the gesture operation type classification module based on finger joint angles is to identify different gesture operation types using finger joint angle data as input. The implementation process of this module is as follows: Figure 6 As shown: Step S11: Obtain finger joint angle information, i.e., receive data output by the angle tracking module.

[0087] Step S12: Feature Extraction. Time-domain or frequency-domain features are extracted from the finger joint angle sequence, including joint angle thresholds, rate of change, and cooperative motion relationships, to obtain classification feature vectors.

[0088] Step S13: Gesture Classification. A pre-trained classifier is used to identify the feature vector and determine the current gesture category.

[0089] Step S141: If the target operation type is identified (belonging to preset operation types such as pressing, pinching, grasping, twisting, etc.), then proceed to static condition judgment.

[0090] Step S142: If the operation is identified as an irrelevant type (not belonging to the preset operation type), the process terminates and does not trigger subsequent force regression.

[0091] Step S15: Determine whether the static condition for finger joint angle is met. Monitor the change in finger joint angle within a preset time period: if the change is ≤ preset threshold, the static condition is met; if the change is > preset threshold, the static condition is not met.

[0092] Step S16: Trigger the force regression algorithm when the static conditions are met; otherwise, do not trigger it and return to S11 to reacquire angle information.

[0093] The core function of the force regression module based on gesture operation type is to match the output of the gesture operation type classification module with the corresponding force regression model, and then output the operation force value. For example... Figure 7 As shown, after triggering force regression, the following steps are executed: Step S21: Gesture type input, input the recognized target gesture operation type into this module.

[0094] Step S22: Force regression model matching. Based on the gesture type, select a matching force regression model from multiple preset models (corresponding to pressing, pinching, grasping, twisting, etc.).

[0095] Step S23: Obtain the original bioelectrical impedance signal or feature vector, that is, extract the multi-channel original bioelectrical impedance signal or its time-frequency features from the preprocessed bioelectrical impedance signal.

[0096] Step S24: Output the operational force. Input the bioelectrical impedance features into the matched force regression model to calculate the operational force (such as fingertip pressure, pinching force, gripping force, or twisting torque) corresponding to the current gesture.

[0097] Embody Intelligence Dataset Output: Through the collaborative operation of the aforementioned multimodal perception hardware system and algorithm inference module, the system ultimately outputs an embodied intelligence dataset. In the specific implementation process, the hardware system simultaneously acquires bioelectrical impedance signals and visual images of bare-hand operations; the algorithm inference module preprocesses the acquired signals, extracts features, performs multimodal fusion inference, and force regression to obtain hand posture, gesture operation type, and operation force data corresponding to each frame of operation; finally, the system uniformly performs temporal alignment and format encapsulation to form a standardized embodied intelligence dataset that can be directly used for training large-scale embodied intelligence models of robots.

[0098] This dataset can be used in various robot embodied learning scenarios. Example applications include: ① VLA (Vision-Language-Motion) pre-training: Inputting the dataset as pre-training data into the robot's embodied learning model helps the model learn the movement patterns and force mechanisms of human bare hand operations; ② Action retargeting: Mapping finger joint angles and operational force data from the dataset to the robot's dexterous hand to achieve the replication of human-like fine-grained operations; ③ Policy learning: Training the robot's general operation strategies based on the operation logic in the dataset to improve its ability to manipulate various everyday objects; ④ Model optimization: Using the dataset to fine-tune the robot's operation model to improve the model's robustness and adaptability in unstructured scenarios.

[0099] The embodied data acquisition system provided in this application, employing the embodied data acquisition method described in the above embodiments, solves the technical problem of how to achieve high-quality embodied data collection without altering the operator's natural bare-handed operating state. Compared with the prior art, the beneficial effects of the embodied data acquisition system provided in this application are the same as those of the embodied data acquisition method provided in the above embodiments, and other technical features of this embodied data acquisition system are the same as those disclosed in the previous embodiment method, and will not be repeated here.

[0100] It should be understood that the various parts disclosed in this application can be implemented using hardware, software, firmware, or a combination thereof. In the description of the above embodiments, specific features, structures, materials, or characteristics can be combined in any suitable manner in one or more embodiments or examples.

[0101] The above are merely specific embodiments of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

[0102] In addition, to achieve the above objectives, embodiments of this application also provide a readable storage medium having computer-readable program instructions (i.e., a computer program) stored thereon, the computer-readable program instructions being used to execute the embodied data acquisition method in the above embodiments.

[0103] The computer-readable storage medium provided in this application embodiment may be, for example, a USB flash drive, but is not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to: electrical connections with one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, system, or device. The program code contained on the computer-readable storage medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (Radio Frequency), etc., or any suitable combination thereof.

[0104] The aforementioned computer-readable storage medium may be included in the embodied data acquisition system; or it may exist independently and not be assembled into the embodied data acquisition system.

[0105] The aforementioned computer-readable storage medium carries one or more programs, which, when executed by the embodied data acquisition system, cause the embodied data acquisition system to perform the process steps of any embodiment of the embodied data acquisition method.

[0106] Computer program code for performing the operations of this application can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, and C++, and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a Local Area Network (LAN) or a Wide Area Network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0107] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0108] The modules described in the embodiments of this application can be implemented in software or hardware. The names of the modules do not necessarily limit the functionality of the modules themselves.

[0109] The readable storage medium provided in this application is a computer-readable storage medium that stores computer-readable program instructions (i.e., a computer program) for executing the above-described embodied data acquisition method. This solves the technical problem of how to achieve high-quality embodied data acquisition without altering the operator's natural bare-handed operating state. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided in this application are the same as those of the embodied data acquisition method provided in the above embodiments, and will not be elaborated upon here.

[0110] Furthermore, embodiments of this application also propose a computer program product, including a computer program that, when executed by a processor, implements the steps of the embodied data acquisition method as described above.

[0111] The specific implementation of the computer program product in this application is basically the same as the embodiments of the above-described embodied data acquisition method, and will not be repeated here.

[0112] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or system that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or system. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or system that includes that element.

[0113] The sequence numbers of the embodiments in this application are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0114] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software sensor. This computer software sensor is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) as described above, and includes several instructions to cause a head-mounted device (which may be a mobile phone, computer, server, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0115] The above are merely preferred embodiments of this application and do not limit the patent scope of this application. Any equivalent structural or procedural transformations made using the content of this application's specification and drawings, or direct or indirect applications in other related technical fields, are similarly included within the patent protection scope of this application.

Claims

1. A method for acquiring embodied data, characterized in that, The method for acquiring embodied data includes a wearable device and a head-mounted device, wherein the image acquisition direction of the head-mounted device is consistent with the wearer's line of sight, and the embodied data acquisition method comprises the following steps: Acquire the bioelectrical impedance signal collected by the wearable device and the image data collected by the head-mounted device; The finger joint angle is determined based on the bioelectrical impedance signal and / or the image data, and the hand operation force is determined based on the finger joint angle and the bioelectrical impedance signal. Based on the image data, the finger joint angles, and the force of the hand operation, a embodied dataset is obtained.

2. The embodied data acquisition method as described in claim 1, characterized in that, The step of determining the hand manipulation force based on the finger joint angle and the bioelectrical impedance signal includes: The gesture operation type is identified based on the finger joint angle. The force regression model is matched based on the gesture operation type, wherein different gesture operation types correspond to different force regression models; The bioelectrical impedance signal is input into the matched force regression model, and the hand operation force is output.

3. The embodied data acquisition method as described in claim 2, characterized in that, Before the step of recognizing the gesture operation type based on the finger joint angle, the method further includes: Monitor the change in the angle of the finger joints within a preset time period; If the change is less than or equal to a preset threshold, then the step of recognizing the gesture operation type based on the finger joint angle is executed; If the change is greater than a preset threshold, then return to the step of acquiring the bioelectrical impedance signal collected by the wearable device and the image data collected by the head-mounted device.

4. The embodied data acquisition method as described in claim 2, characterized in that, Following the step of recognizing the gesture operation type based on the finger joint angle, the method further includes: If the gesture operation type belongs to a preset operation type, then the step of matching the corresponding force regression model based on the gesture operation type is executed, wherein the preset operation type includes one or more of pressing, pinching, grasping and twisting; If the gesture operation type does not belong to the preset operation type, then return to the step of obtaining the bioelectrical impedance signal collected by the wearable device and the image data collected by the head-mounted device.

5. The embodied data acquisition method as described in claim 1, characterized in that, The step of determining the finger joint angle based on the bioelectrical impedance signal and / or the image data includes: Feature extraction is performed on the bioelectrical impedance signal to obtain an impedance signal feature vector. This impedance signal feature vector is then input into a pre-trained first angle regression model, which outputs the finger joint angle; or... Feature extraction is performed on the image data to obtain visual features. These visual features are then input into a pre-trained second angle regression model, which outputs the finger joint angles; or, Visual features and electrical impedance signal features are extracted from the image data and the bioelectrical impedance signal respectively. The visual features and electrical impedance signal features are fused to obtain fused features. The fused features are input into a pre-trained third-angle regression model to output the finger joint angle.

6. The embodied data acquisition method as described in claim 5, characterized in that, The step of extracting features from the bioelectrical impedance signal to obtain impedance signal features includes: The bioelectrical impedance signal is sequentially subjected to bandpass filtering and notch filtering to obtain the preprocessed bioelectrical impedance signal. The bioelectrical impedance signal is preprocessed to extract its features.

7. The embodied data acquisition method as described in claim 5, characterized in that, The step of extracting visual features from the image data includes: The image data is subjected to denoising, hand region segmentation and normalization processes in sequence to obtain the preprocessed image data; Visual features are obtained by feature extraction from the preprocessed image data.

8. The method for acquiring embodied data as described in any one of claims 1 to 7, characterized in that, The step of obtaining the embodied dataset based on the image data, the finger joint angles, and the hand manipulation force includes: The image data, finger joint angles, and hand manipulation force are time-aligned, and the time-aligned image data, finger joint angles, and hand manipulation force are formatted and encapsulated to obtain an embodied dataset; or... Based on the finger joint angles, the hand posture and gesture operation type are determined. The image data, finger joint angles, hand operation force, hand posture, and gesture operation type are time-aligned. The time-aligned image data, finger joint angles, hand operation force, hand posture, and gesture operation type are formatted and encapsulated to obtain the embodied dataset.

9. An embodied data acquisition system, characterized in that, The embodied data acquisition system includes: Wearable devices used to collect bioelectrical impedance signals; A head-mounted device for acquiring image data, wherein the image acquisition direction of the head-mounted device is consistent with the wearer's line of sight; A processing unit is configured to perform the steps of the embodied data acquisition method as described in any one of claims 1 to 8.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores an embodied data acquisition program, which, when executed by a processor, implements the steps of the embodied data acquisition method as described in any one of claims 1 to 8.

Images