A multi-modal human operation information acquisition and fusion platform and method for robot skill learning

Abstract

The application relates to a multi-modal human operation information acquisition and fusion platform and method for robot skill learning, and belongs to the field of robot teaching and multi-modal perception. The platform comprises a visual acquisition module, a force sensation acquisition module, an electromyography acquisition module, a unified time reference module, a space calibration module and a preprocessing and fusion module. Synchronous acquisition of each module is triggered by a unified starting event, a public time axis is constructed according to a conversation starting time, a device timestamp and an electromyography sample serial number, and time sequence alignment of different frequency data is realized. After space calibration, filtering and feature extraction, human body posture, contact force and electromyography features are fused into a unified state vector, and a multi-modal teaching data set with a time sequence label is output. The application solves the alignment problem caused by different sampling frequencies and independent device clocks of multi-source heterogeneous data, and provides a standardized data basis for robot imitation learning, grasping learning and operation strategy learning.

Description

Technical Field

[0001] This invention belongs to the field of robot teaching and multimodal perception, specifically relating to a multimodal human operation information acquisition and fusion platform and method for robot skill learning. Background Technology

[0002] With the development of robotics, artificial intelligence, and intelligent manufacturing technologies, training robots using human teaching data has become an important technical means to improve robots' environmental adaptability and ability to perform complex tasks. Existing teaching schemes typically use RGB cameras or depth cameras to acquire human motion trajectories, joint poses, or target object positional relationships, and construct robot learning samples based on this.

[0003] While a single visual modality can describe the overall movement trends and spatial trajectories of the human body, it is difficult to accurately depict the force state, contact changes, and slippage trends during the contact process between the fingers and the target object. In contact-based tasks such as fine grasping, assembly, twisting, and tool use, relying solely on visual information often makes it difficult to determine whether contact has been established, whether the grasp is stable, and when the operator begins to apply force.

[0004] Force sensory information can directly reflect the local contact state during the interaction between a human hand and an object, but relying solely on force sensory information is insufficient to fully describe the overall movement trajectory and operational context of the human body. Surface electromyography (EMG) signals can reflect muscle activation state, movement preparation, and force exertion trends, but their sampling frequency is usually significantly higher than that of visual and force sensory data. Furthermore, different devices have independent clocks, buffers, and communication delays, making it difficult to directly align multimodal data.

[0005] Therefore, a platform is needed that can perform modular initialization, unified startup, timestamp recording, asynchronous multi-frequency data alignment, and quality monitoring of visual, force, and electromyographic data under a unified acquisition interface, in order to obtain multimodal human operation teaching data with clear time correspondence. Summary of the Invention

[0006] The purpose of this invention is to provide a multimodal human operation information acquisition and fusion platform and method for robot skill learning, in order to solve the problems of inconsistent start-up times of different devices, large differences in sampling frequencies, different timestamp systems of devices, data queue delays, and difficulty in accurately aligning cross-modal data in existing multimodal acquisition systems.

[0007] To achieve the above objectives, the technical solution of the present invention is: a multimodal human operation information acquisition and fusion platform for robot skill learning, comprising:

[0008] The visual acquisition module is used to acquire color images, depth images, and key points or pose trajectories of human joints during the operation of the operator to obtain visual data.

[0009] The force sensing acquisition module is used to collect multi-axis contact force data during the process of the operator's hand coming into contact with the target object;

[0010] The electromyography (EMG) acquisition module is used to acquire multi-channel surface electromyography (EMG) signal data of the target muscle group of the operator.

[0011] The unified time reference module is used to send a unified start event to each acquisition module, record the session start time, obtain the frame timestamp of visual data, the sampling timestamp of force data and the sample sequence number of electromyographic signal data respectively, and establish a common time axis based on the session start time, device timestamp, sample sequence number and corresponding sampling frequency.

[0012] The spatial calibration module is used to establish the rigid body transformation relationship between the visual sensor coordinate system and the operating area reference coordinate system, the axial mapping relationship between the force sensor coordinate system and the hand or finger coordinate system, and the correspondence between the electromyography channel number and the target muscle group name, body side and attachment position; the camera coordinate transformation can be determined by the calibration plate, known reference point or standard posture; the axial mapping of the force sensor can be determined according to the installation posture and calibration load;

[0013] The preprocessing and fusion module is used to map visual data, force data, and electromyographic signal data to a common time axis, perform time alignment processing, and extract features to generate multimodal teaching data;

[0014] The data output module is used to output a dataset that includes multimodal teaching data.

[0015] Furthermore, the visual acquisition module includes a depth camera, which is configured to acquire color images and depth images at a preset frame rate, and associate each frame image with a frame number, a camera hardware timestamp, and a host receiving timestamp.

[0016] Furthermore, the depth camera can be an Intel RealSense D455, or other depth-sensing devices with equivalent functionality.

[0017] Furthermore, the force sensing acquisition module includes a force or tactile sensor arranged at the fingertip of at least one of the operator's fingers to acquire force components Fx, Fy, and Fz in three orthogonal directions, as well as torque components Mx, My, and Mz in three directions.

[0018] Furthermore, the electromyography (EMG) acquisition module includes multiple EMG sensors arranged on the surface of the muscle groups of the operator's forearm, hand, upper arm, or shoulder. Each EMG sensor is configured to record the channel number and sample number (and also record the reading status and voltage value, which can be uniformly converted to microvolts). The system establishes a mapping table between the EMG channel number and the target muscle group name, body side, and attachment position.

[0019] Furthermore, the unified time reference module includes a session control unit, multiple device acquisition threads, a device readiness event, and a unified start event. Each device acquisition thread enters a waiting state after initialization; once all enabled devices are in the ready state, the session control unit records the session start time T_0 and sends a unified start event. Camera frames and force data use the same host clock to record nanosecond-level timestamps; electromyography data utilizes sample sequence numbers and sampling frequencies to reconstruct the uniform sampling timing within the channel.

[0020] Furthermore, the unified time reference module determines the relative sampling time of each EMG sample within the same channel based on the sample sequence number difference and the EMG sampling frequency, and calculates the estimated sampling time of the k-th EMG sample according to the formula t_e(k)=T_0+(n_k−n_0) / f_e, where T_0 is the session start time or the reference time of the first valid EMG sample, n_k is the sample sequence number of the k-th EMG sample, n_0 is the sequence number of the first valid sample, and f_e is the EMG sampling frequency. Camera frames and force data are mapped to the same common time axis according to their device timestamps or host timestamps, and device clock deviations can be corrected through first frame offset, synchronization events, or timestamp regression.

[0021] Furthermore, the unified time reference module includes multiple independent acquisition threads. Each acquisition thread enters a waiting state after completing the initialization of the corresponding device, and starts data acquisition in response to the unified start event. It also records the timestamp and the elapsed time relative to the start time of the session under the same host clock.

[0022] Furthermore, the preprocessing and fusion module is used to perform preprocessing, temporal alignment, and fusion on the synchronized multimodal data. Visual data can undergo keypoint smoothing, missing point compensation, pose estimation, and trajectory reconstruction; force data can undergo zero-point correction, low-pass filtering, contact event detection, and slip detection; electromyographic signal data can undergo 20–450 Hz bandpass filtering, power frequency suppression, full-wave rectification, envelope extraction, normalization, and sliding window feature extraction. The above parameters are only preferred embodiments and do not constitute limitations.

[0023] Furthermore, the preprocessing and fusion module is configured to: select a visual frame moment, a fixed frequency moment, or a preset event moment as the target time axis node; perform nearest neighbor matching or interpolation on the force data; extract the root mean square value, average absolute value, waveform length, or frequency domain features within the time window corresponding to the target moment from the electromyographic signal data; and combine the human pose features, force sensation features, and electromyographic features into a unified state vector.

[0024] For time-series alignment, a camera frame moment, a fixed frequency moment, or a contact event moment can be selected as the target time axis. For each target moment, the nearest force data is matched or linear interpolation is performed on adjacent force data. For electromyography (EMG) data, a preset window is selected before and after the target moment, and the root mean square value, mean absolute value, waveform length, zero-crossing rate, or frequency domain features are calculated. This forms a unified state vector Z_m=[P_m,F_m,E_m], where P_m represents human pose features, F_m represents multi-axis contact force features, and E_m represents EMG features.

[0025] Furthermore, it also includes a data quality monitoring module, which is configured to record the actual sampling rate, adjacent sample time interval, number of dropped frames or samples, buffer queue length, and device connection status of each modality of data, and output prompts or status flags when data anomalies are detected. Specifically, when a device is not connected, the unified acquisition interface can still be started and other modules can be allowed to acquire data independently; when a device malfunctions or data is interrupted, a status flag is added to the corresponding data segment.

[0026] Furthermore, it also includes a task segmentation module, which is configured to divide the continuous operation process into different skill stages based on at least one of human pose change, contact force threshold, contact event, or electromyographic activation change, and add stage labels to the data.

[0027] Furthermore, the data output module saves multimodal data separately for each session, and records at least the session identifier, unified timestamp, relative time, modality type, frame or sample sequence number, channel number, device status, raw data, and processed features. The output can be in the form of CSV, binary recording file, image sequence, point cloud file, or database, and can be further converted into state-action pairs, sequence samples, or stage label data required by the robot learning model.

[0028] This invention also provides a method for collecting and fusing multimodal human manipulation information for robot skill learning based on the platform described above, comprising the following steps:

[0029] S1. Initialize the visual acquisition module, force acquisition module, and electromyography acquisition module, and create the corresponding data acquisition threads;

[0030] S2. After each acquisition thread completes initialization, a unified start event is sent and the session start time is recorded;

[0031] S3. Parallel acquisition of visual data, force data, and electromyographic signal data, and recording of the corresponding frame timestamp, sampling timestamp, and electromyographic sample number respectively;

[0032] S4. Construct a common timeline based on the session start time, device timestamp, sample number, and sampling frequency;

[0033] S5. Perform filtering, resampling, and time window matching processing on each modal data;

[0034] S6. Extract the feature parameters of each modality and fuse the feature parameters into a unified multimodal representation;

[0035] S7. Output multimodal teaching data for robot skill learning.

[0036] Furthermore, step S4 specifically includes:

[0037] The reception time of visual frames and force data is recorded using a unified clock on the host computer.

[0038] The sampling timing sequence within the electromyographic channel is reconstructed based on the electromyographic sample number and the electromyographic sampling frequency.

[0039] Based on the timestamp of the first valid electromyography sample, the offset of the electromyography timing relative to the host unified clock is calculated.

[0040] Compared with the prior art, the present invention has the following beneficial effects:

[0041] First, by using a unified startup event, a unified host clock, device timestamps, and EMG sample serial numbers to construct a common timeline, a clear temporal correspondence can be maintained even when visual, force, and EMG sampling frequencies differ.

[0042] Second, visual information, local contact force, and muscle activation information can be used to analyze overall movement, local contact, and the body's force exertion. Figure 3 The multiple layers jointly represent the operational process, improving the completeness of teaching data for complex contact tasks.

[0043] Third, by controlling device readiness, monitoring queue buffers, recording actual sampling rates, and marking abnormal states, the stability and traceability of long-term multi-device data acquisition processes can be improved.

[0044] Fourth, the platform adopts a modular acquisition interface, which can still start and acquire other modes when some devices are offline, facilitating system debugging, patent implementation and subsequent sensor expansion. Attached Figure Description

[0045] Figure 1 This is a schematic diagram of the multimodal human operation information collection and fusion process.

[0046] Figure 2 This is an example of the electromyography (EMG) signal acquisition results.

[0047] Figure 3 Example image for capturing key points of the human body using a depth camera.

[0048] Figure 4This is an example image of the results of triaxial fingertip force acquisition.

[0049] Figure 5 A schematic diagram of the interface for unified and synchronous collection of multimodal human operation information. Detailed Implementation

[0050] The technical solution of the present invention will be further described below with reference to the accompanying drawings. The following embodiments are used to illustrate the present invention, and not to limit the scope of protection. Equivalent substitutions for device models, sampling parameters, communication methods, data formats, or fusion algorithms can be applied to the present invention without departing from the core concept of the present invention.

[0051] This invention provides a multimodal human manipulation information acquisition and fusion platform for robot skill learning, comprising:

[0052] The visual acquisition module is used to acquire color images, depth images, and key points or pose trajectories of human joints during the operation of the operator to obtain visual data.

[0053] The force sensing acquisition module is used to collect multi-axis contact force data during the process of the operator's hand coming into contact with the target object;

[0054] The electromyography (EMG) acquisition module is used to acquire multi-channel surface electromyography (EMG) signal data of the target muscle group of the operator.

[0055] The unified time reference module is used to send a unified start event to each acquisition module, record the session start time, obtain the frame timestamp of visual data, the sampling timestamp of force data and the sample sequence number of electromyographic signal data respectively, and establish a common time axis based on the session start time, device timestamp, sample sequence number and corresponding sampling frequency.

[0056] The spatial calibration module is used to establish the rigid body transformation relationship between the visual sensor coordinate system and the operating area reference coordinate system, the axial mapping relationship between the force sensor coordinate system and the hand or finger coordinate system, and the correspondence between the electromyography channel number and the target muscle group name, body side and attachment position; the camera coordinate transformation can be determined by the calibration plate, known reference point or standard posture; the axial mapping of the force sensor can be determined according to the installation posture and calibration load;

[0057] The preprocessing and fusion module is used to map visual data, force data, and electromyographic signal data to a common time axis, perform time alignment processing, and extract features to generate multimodal teaching data;

[0058] The data output module is used to output a dataset that includes multimodal teaching data.

[0059] This invention also provides a method for collecting and fusing multimodal human manipulation information for robot skill learning based on the platform described above, comprising the following steps:

[0060] S1. Initialize the visual acquisition module, force acquisition module, and electromyography acquisition module, and create the corresponding data acquisition threads;

[0061] S2. After each acquisition thread completes initialization, a unified start event is sent and the session start time is recorded;

[0062] S3. Parallel acquisition of visual data, force data, and electromyographic signal data, and recording of the corresponding frame timestamp, sampling timestamp, and electromyographic sample number respectively;

[0063] S4. Construct a common timeline based on the session start time, device timestamp, sample number, and sampling frequency;

[0064] S5. Perform filtering, resampling, and time window matching processing on each modal data;

[0065] S6. Extract the feature parameters of each modality and fuse the feature parameters into a unified multimodal representation;

[0066] S7. Output multimodal teaching data for robot skill learning.

[0067] The following are specific implementation examples of the present invention.

[0068] Example 1: Overall Platform Structure

[0069] like Figure 1 and Figure 5 As shown, this embodiment provides a multimodal acquisition platform consisting of a visual acquisition module, a force acquisition module, an electromyography (EMG) acquisition module, a unified time reference module, a spatial calibration module, a preprocessing and fusion module, a data quality monitoring module, and a data output module. The unified acquisition interface is used to configure the session name, output directory, acquisition duration, device activation status, serial port number, channel range, camera resolution, and frame rate, and displays the device connection status, unified clock, image preview, triaxial force curve, and multi-channel EMG curve.

[0070] The platform adopts a modular operation approach. It does not force the initialization of all devices upon interface startup; instead, it initializes a device only when the user selects it and performs connection checks or begins data acquisition. Therefore, even if any of the electromyography (EMG) devices, force sensors, or depth cameras are temporarily unavailable, the acquisition interface can still be opened independently and used for interface demonstrations, parameter configuration, or other modal acquisition.

[0071] Example 2: Unified Time Base and Synchronous Data Acquisition

[0072] In this embodiment, the depth camera acquires color and depth images at 30 frames per second, the force sensor acquires Fx, Fy, and Fz at approximately 50 Hz, and the electromyography (EMG) system acquires multiple channels at 1000 Hz. Since the three devices have different sampling frequencies, and the camera and EMG devices have internal buffers, the actual sampling order cannot be directly determined by the order in which the program retrieves data from the queue.

[0073] Before data acquisition begins, each device thread first completes port opening, sensor enumeration, camera stream configuration or channel configuration, and sets a ready flag. The session control unit records a unified start time T_0 after all enabled threads are ready, and then sends a unified start event. Force sensor data and camera data record the host absolute timestamp and the elapsed time relative to T_0 each time valid data is acquired; camera data simultaneously saves the hardware timestamp and frame sequence number.

[0074] Multiple channels of the electromyography (EMG) system are sampled synchronously by the same acquisition system, with channel alignment based on the same sample sequence number. The program records a timestamp when reading data from each channel queue, but this timestamp is only used to estimate queue latency; the true timing within and between channels is primarily recovered based on the EMG sample sequence number and sampling frequency. EMG samples with the same sample sequence number in different channels are considered data obtained at the same sampling time.

[0075] When alignment with external force and visual data is required, the relative sampling time of EMG is added to the session baseline time or the host baseline time of the first valid EMG sample to obtain the estimated absolute sampling time. Then, nearest neighbor matching, linear interpolation, or sliding window aggregation are performed based on the target timeline. The platform saves the original timestamp, estimated sampling time, and aligned time for subsequent evaluation of synchronization errors and time drift.

[0076] Example 3: Data Preprocessing and Fusion

[0077] The visual data is first registered in color and depth frames, and then key points such as shoulders, elbows, wrists, fingers, or other human body features are extracted. The key point sequence can be smoothed and missing points compensated using moving average, Kalman filtering, or spline interpolation, and the three-dimensional coordinates are recovered from the depth values.

[0078] Force data is first subjected to no-load zero-point correction. For force data of approximately 50 Hz, a low-pass filter with a cutoff frequency of 5–15 Hz can be used to suppress high-frequency noise, and the resultant force modulus, force change rate, contact threshold exceedance, and slip correlation characteristics are calculated. Figure 4 The acquisition results of Fx, Fy, and Fz in three independent coordinate axes are shown.

[0079] Electromyography data can be processed using a 20–450 Hz bandpass filter and a 50 Hz power frequency suppression. The filtered signal is then rectified using full-wave rectification, and the root mean square value, mean absolute value, waveform length, or median frequency in the frequency domain is calculated using a 50–200 ms time window. Figure 2 The original voltage change of an electromyography channel during operation is shown; multi-channel data can be stored in a long table format to record channel number and sample number, and converted into a wide table with one column per channel according to sample number during analysis.

[0080] Taking the m-th target time as an example, the platform obtains the human pose feature P_m, the triaxial force feature F_m, and the electromyography window feature E_m, and constructs a multimodal state vector Z_m=[P_m,F_m,E_m]. The state vector can be combined with the operator's action, the robot's target action, or the skill stage label to form a state-action sample, which can be used for supervised learning, imitation learning, offline reinforcement learning, or temporal model training.

[0081] Example 4: Capture Task Scenario

[0082] In grasping and handling tasks, the vision acquisition module acquires the spatial trajectory of the operator's hand and upper limb, the force acquisition module acquires Fx, Fy, and Fz during the contact process between the fingers and the object, and the electromyography (EMG) acquisition module acquires the activation changes of the target muscle group in the forearm. The platform can divide the task into approach, contact, grasping stabilization, handling, and release stages based on the hand approach speed, contact force threshold, and EMG envelope changes.

[0083] The fused data not only indicates when the operator approaches the target object and what trajectory is used to complete the grasp, but also the contact establishment time, grasp stability, force direction, and changes in force application intention, providing multimodal basis for the robot to learn precise grasping strategies.

[0084] Example 5: Assembly and Tooling Operation Scenario

[0085] In tasks involving insertion, twisting, pressing, or tool manipulation, the vision module represents the spatial relationship between the hand, tool, and target location; the force module represents changes in contact pressure and direction; and the electromyography module represents the operator's force intensity and readiness state at different assembly stages. After alignment along a common timeline, events such as contact establishment, increased resistance, direction correction, and operation completion can be identified, generating teaching sequences suitable for the robot's fine assembly learning.

[0086] This invention is not limited to the Intel RealSense D455, specific force sensors, or BTS electromyography devices; other devices capable of outputting images, depth, force, torque, or surface electromyography signals can also be used. The sampling rate, filtering parameters, time windows, and fusion algorithms for each modality can be adjusted according to task requirements. Synchronization methods can also be further combined with external hardware triggering, photoelectric synchronization signals, or network clock protocols to improve synchronization accuracy.

[0087] The fusion method is not limited to feature concatenation, and may employ weighted fusion, dynamic time warping, recurrent neural networks, temporal convolutional networks, Transformer, or other temporal modeling methods. Any equivalent substitutions made using the same or similar concepts as this invention, such as unified startup, timestamp recording, sample sequence number recovery, asynchronous multi-frequency alignment, and multimodal output, fall within the scope of this invention.

[0088] The above are preferred embodiments of the present invention. Any changes made to the technical solution of the present invention that do not exceed the scope of the technical solution of the present invention shall fall within the protection scope of the present invention.

Claims

1. A multimodal human operation information acquisition and fusion platform for robot skill learning, characterized in that, include: The visual acquisition module is used to acquire color images, depth images, and key points or pose trajectories of human joints during the operation of the operator to obtain visual data. The force sensing acquisition module is used to collect multi-axis contact force data during the process of the operator's hand coming into contact with the target object; The electromyography (EMG) acquisition module is used to acquire multi-channel surface electromyography (EMG) signal data of the target muscle group of the operator. The unified time reference module is used to send a unified start event to each acquisition module, record the session start time, obtain the frame timestamp of visual data, the sampling timestamp of force data and the sample sequence number of electromyographic signal data respectively, and establish a common time axis based on the session start time, device timestamp, sample sequence number and corresponding sampling frequency. The spatial calibration module is used to establish the rigid body transformation relationship between the visual sensor coordinate system and the operating area reference coordinate system, the axial mapping relationship between the force sensor coordinate system and the hand or finger coordinate system, and the correspondence between the electromyography channel number and the target muscle group name, body side and attachment position. The preprocessing and fusion module is used to map visual data, force data, and electromyographic signal data to a common time axis, perform time alignment processing, and extract features to generate multimodal teaching data; The data output module is used to output a dataset that includes multimodal teaching data.

2. The multimodal human operation information acquisition and fusion platform for robot skill learning according to claim 1, characterized in that, The visual acquisition module includes a depth camera, which is configured to acquire color images and depth images at a preset frame rate, and associate each frame image with a frame number, a camera hardware timestamp, and a host receiving timestamp.

3. The multimodal human operation information acquisition and fusion platform for robot skill learning according to claim 1, characterized in that, The force sensing acquisition module includes a force or tactile sensor arranged at the fingertip of at least one of the operator's fingers, which acquires force components Fx, Fy and Fz in three orthogonal directions, as well as torque components Mx, My and Mz in three directions.

4. The multimodal human operation information acquisition and fusion platform for robot skill learning according to claim 1, characterized in that, The electromyography (EMG) acquisition module includes multiple EMG sensors arranged on the surface of the muscles of the operator's forearm, hand, upper arm, or shoulder. Each EMG sensor is configured to record a channel number and a sample number. The unified time reference module determines the relative sampling time of each EMG sample in the same channel based on the sample number difference and the EMG sampling frequency, and calculates the estimated sampling time of the k-th EMG sample according to the formula t_e(k)=T_0+(n_k−n_0) / f_e, where T_0 is the session start time or the reference time of the first valid EMG sample, n_k is the sample number of the k-th EMG sample, n_0 is the first valid sample number, and f_e is the EMG sampling frequency.

5. A multimodal human operation information acquisition and fusion platform for robot skill learning according to claim 1, characterized in that, The unified time reference module includes multiple independent acquisition threads. Each acquisition thread enters a waiting state after completing the initialization of the corresponding device, and starts data acquisition in response to the unified start event. It records the timestamp and the elapsed time relative to the start time of the session under the same host clock.

6. The multimodal human operation information acquisition and fusion platform for robot skill learning according to claim 1, characterized in that, The preprocessing and fusion module is configured to: select a visual frame moment, a fixed frequency moment, or a preset event moment as a target time axis node; perform nearest neighbor matching or interpolation on the force data; and extract the root mean square value, mean absolute value, waveform length, or frequency domain features of the electromyographic signal data within the time window corresponding to the target moment. Human body posture features, force sensation features, and electromyographic features are combined into a unified state vector.

7. A multimodal human operation information acquisition and fusion platform for robot skill learning according to claim 1, characterized in that, It also includes a data quality monitoring module, which is configured to record the actual sampling rate of each modality data, the time interval between adjacent samples, the number of frames or samples lost, the length of the buffer queue and the device connection status, and output prompts or status flags when data abnormalities are detected.

8. A multimodal human operation information acquisition and fusion platform for robot skill learning according to claim 1, characterized in that, It also includes a task segmentation module, which is configured to divide a continuous operation process into different skill stages based on at least one of human pose change, contact force threshold, contact event, or electromyographic activation change, and add stage labels to the data.

9. A method for acquiring and fusing multimodal human operation information for robot skill learning based on the platform described in any one of claims 1-8, characterized in that, Includes the following steps: S1. Initialize the visual acquisition module, force acquisition module, and electromyography acquisition module, and create the corresponding data acquisition threads; S2. After each acquisition thread completes initialization, a unified start event is sent and the session start time is recorded; S3. Parallel acquisition of visual data, force data, and electromyographic signal data, and recording of the corresponding frame timestamp, sampling timestamp, and electromyographic sample number respectively; S4. Construct a common timeline based on the session start time, device timestamp, sample number, and sampling frequency; S5. Perform filtering, resampling, and time window matching processing on each modal data; S6. Extract the feature parameters of each modality and fuse the feature parameters into a unified multimodal representation; S7. Output multimodal teaching data for robot skill learning.

10. The method for acquiring and fusing multimodal human operation information for robot skill learning according to claim 1, characterized in that, Step S4 specifically includes: The reception time of visual frames and force data is recorded using a unified clock on the host computer. The sampling timing sequence within the electromyographic channel is reconstructed based on the electromyographic sample number and the electromyographic sampling frequency. Based on the timestamp of the first valid electromyography sample, the offset of the electromyography timing relative to the host unified clock is calculated.

Images