Data processing method and device for humanoid robot and robot

Abstract

The application discloses a data processing method and device for a humanoid robot and the robot, and relates to the field of data collection. The method creates a collection session and obtains a session context. After the collection session is started, raw input data from at least one teleoperation input device is received, the raw input data is analyzed and standardized to generate a unified input state. The control target is solved by inverse kinematics and safety constraints are applied to generate executable control instructions. The executable control instructions are sent to the executable components for execution, and multi-modal data is synchronously collected according to a unified collection period. The multi-modal data is associated with a unified timestamp and frame number to generate a multi-modal data record of the current frame. In response to a stop instruction or a session end event, the collection session is terminated, and the multi-modal data records of all frames are saved as a data set. The application can improve the compatibility, synchronization and data quality of teleoperation data collection of the humanoid robot.

Description

Technical Field

[0001] This application relates to the field of data acquisition, and in particular to a data processing method, apparatus and robot for humanoid robots. Background Technology

[0002] With the widespread application of humanoid robots in industrial and service sectors, how to efficiently acquire high-quality robot operation data has become an urgent problem to be solved. Teleoperation data acquisition is one of the important ways to obtain robot training data.

[0003] Several teleoperation data acquisition schemes exist in the current technology. One common scheme is based on a single input device and a single robot. This type of scheme uses a fixed type of input device to control the robot and generates control commands according to the data format of the device, while recording camera images, joint states, and motion data during operation. This type of scheme is deeply tied to the specific input device, resulting in poor scalability when different devices need to be replaced or combined. Another scheme mainly targets teleoperation control of a single arm or partial degrees of freedom. This type of scheme is difficult to cover the multi-degree-of-freedom collaborative control requirements of humanoid robots, including the head, both arms, both hands, and the entire body's joints.

[0004] Regarding data acquisition, existing solutions primarily store basic data such as camera images and joint states. The data dimensions are limited, lacking crucial information like raw input device data and unified timestamps, leading to insufficient data for subsequent multimodal data alignment and anomaly tracing. In terms of acquisition process management, existing solutions typically employ continuous recording, which struggles to effectively handle interruptions such as pauses and device malfunctions, easily generating invalid data fragments or disrupting data continuity. Regarding data quality control, existing solutions rely heavily on manual inspection, which is inefficient and prone to overlooking problems. Summary of the Invention

[0005] This application provides a data processing method, apparatus, and robot for humanoid robots, which can solve the problems of poor input device compatibility, difficulty in multi-degree-of-freedom collaborative control, difficulty in synchronizing and aligning multimodal data, lack of effective management of the acquisition process, and uncontrollable data quality in existing teleoperation data acquisition schemes. The technical solution is as follows: In a first aspect, embodiments of this application provide a data processing method for humanoid robots, including: In response to a user-triggered acquisition configuration command, an acquisition session is created and a session context is obtained. Following this step, the process further includes: in response to a user-triggered pause command, pausing the writing of new data to the multimodal data record of the current frame while maintaining the session context and all hardware connections; or in response to a user-triggered resume command, resuming data writing and continuing to append new multimodal data records using the current session identifier and existing data files; or in response to a detected device anomaly or communication timeout event, automatically performing an abnormal termination operation and saving all acquired data. The session context includes at least input device configuration information, robot configuration information, and control mode information. After the acquisition session is started, raw input data from at least one remotely operated input device is received. The raw input data is parsed and standardized according to the configuration information of the input device to generate a unified input state. Based on the session context and the unified input state, a control objective is generated, and inverse kinematics is solved on the control objective and safety constraints are applied to generate executable control instructions. The executable control instructions are sent to each execution component of the robot for execution, and multimodal data is collected according to a unified global synchronization time source. The multimodal data is associated with a unified timestamp and frame number to generate a multimodal data record for the current frame. The multimodal data includes the executable control instructions. In response to a user-triggered stop command or session end event, the acquisition session is terminated, and the multimodal data records of all frames are saved as a dataset.

[0006] Secondly, embodiments of this application provide a data processing device for humanoid robots, the device comprising: The acquisition module is used to create an acquisition session and acquire the session context in response to a user-triggered acquisition configuration command. After creating the acquisition session and acquiring the session context in response to the user-triggered acquisition configuration command, the module further includes: pausing the writing of new data to the multimodal data record of the current frame in response to a user-triggered pause command, while maintaining the session context and all hardware connections; or resuming data writing in response to a user-triggered resume command, continuing to append new multimodal data records using the current session identifier and existing data files; or automatically performing an abnormal termination operation and saving all acquired data in response to a detected device anomaly or communication timeout event. The session context includes at least input device configuration information, robot configuration information, and control mode information. The generation module is used to receive raw input data from at least one remotely operated input device after the acquisition session is started, and to parse and standardize the raw input data according to the configuration information of the input device to generate a unified input state; The constraint module is used to generate control objectives based on the session context and the unified input state, perform inverse kinematics solution on the control objectives and apply safety constraints to generate executable control instructions; The acquisition module is used to send the executable control instructions to each execution component of the robot for execution, and to acquire multimodal data according to a unified global synchronization time source, associate the multimodal data with a unified timestamp and frame number, and generate a multimodal data record for the current frame, wherein the multimodal data includes the executable control instructions; The recording module is used to terminate the acquisition session in response to a user-triggered stop command or session end event, and save the multimodal data of all frames as a dataset.

[0007] Thirdly, embodiments of this application provide a computer storage medium storing a plurality of instructions adapted for loading by a processor and executing the above-described method steps.

[0008] Fourthly, embodiments of this application provide a robot that may include: a processor and a memory; wherein the memory stores a computer program adapted to be loaded by the processor and to execute the above-described method steps.

[0009] The beneficial effects of the technical solutions provided in some embodiments of this application include at least the following: This application creates a session context by acquiring configuration instructions, enabling the processor to determine the input device type and robot configuration before acquisition begins. Combined with the parsing and standardization of the raw input data, it converts data from different devices into a uniformly formatted input state. Consequently, subsequent control processes are no longer tied to specific devices; when replacing or adding devices, only parsing logic needs to be added without modifying the overall process, significantly improving the system's scalability and compatibility.

[0010] This application generates control objectives adapted to the current robot configuration based on control mode information, and coordinates the movements of the head, arms, and all joints through inverse kinematics, converting the operator's intentions into executable control commands covering multiple degrees of freedom. Simultaneous application of safety constraints ensures that the commands meet joint limits and velocity restrictions, thereby achieving full-body coordinated teleoperation of the humanoid robot.

[0011] This application collects multimodal data at a unified acquisition cycle simultaneously with the issuance of instructions, and establishes a correlation using a unified timestamp and frame number. Because acquisition and execution are performed synchronously, a precise time correspondence is established between images, inputs, control commands, and feedback states, providing a high-quality synchronous data foundation for subsequent model training.

[0012] This application organizes the entire process on a session-by-session basis, forming a complete closed loop of state management from session creation to session termination, avoiding the generation of invalid data fragments, while maintaining the temporal continuity and traceability of data for the same task.

[0013] This application establishes a time-series index for each frame of data by using frame numbering. When an anomaly occurs, the input and control information at the corresponding time can be quickly located, providing a basic guarantee for data quality inspection.

[0014] In summary, this application can solve the problems of poor input device compatibility, difficulty in multi-degree-of-freedom collaborative control, difficulty in aligning multimodal data, lack of management in the acquisition process, and uncontrollable data quality in the prior art. Attached Figure Description

[0015] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0016] Figure 1 This is a scenario architecture diagram provided in the embodiments of this application; Figure 2 This is a flowchart illustrating a data processing method for humanoid robots provided in an embodiment of this application; Figure 3 This is a schematic diagram of the structure of a data processing device for humanoid robots provided in this application; Figure 4 This is a schematic diagram of a computer storage medium provided in this application; Figure 5 This is a structural schematic diagram of a robot provided in this application. Detailed Implementation

[0017] To make the objectives, technical solutions, and advantages of this application clearer, the embodiments of this application will be described in further detail below with reference to the accompanying drawings.

[0018] Figure 1 This is a scenario architecture diagram provided for an embodiment of this application. (Refer to...) Figure 1The scenario architecture includes at least one teleoperated input device 1 and a humanoid robot 2. The teleoperated input device 1 and the humanoid robot 2 can establish a communication connection through a wired network or a wireless network, and the two cooperate to execute the data processing method for humanoid robots provided in this application.

[0019] Teleoperation input device 1 is used to collect the operator's control intentions. Teleoperation input device 1 can be various types of human-computer interaction devices, including but not limited to VR headsets, VR controllers, motion capture devices, exoskeletons, data gloves, master-slave robotic arms, SpaceMouse, game controllers, keyboards and mice, or mobile terminals. The operator generates raw input data by manipulating the aforementioned devices. This raw input data contains the operator's motion information and control commands. Teleoperation input device 1 transmits the generated raw input data to humanoid robot 2 via a wired or wireless network.

[0020] Humanoid robot 2 is used to execute the data processing method provided in this application. Humanoid robot 2 integrates a processor, which serves as the main executor of the method and possesses computational, communication, storage, and control functions. The processor of humanoid robot 2 establishes communication connections with the teleoperation input device 1 and each of the robot's own execution components, including joint actuators, dexterous hand or gripper controllers, and mobile chassis controllers. The processor receives raw input data from the teleoperation input device 1, parses and standardizes the raw input data to generate a unified input state, then generates a control target based on the unified input state, performs inverse kinematics solving on the control target, applies safety constraints, and generates executable control instructions. The processor sends the executable control instructions to each execution component for execution. During the control instruction issuance process, the processor collects multimodal data according to a unified global synchronization time source. The multimodal data includes raw input data, unified input state, control target, executable control instructions, feedback state of humanoid robot 2, and multi-channel visual data. The processor associates the above multimodal data with a unified timestamp and frame number to generate multimodal data records, which are organized and saved in units of acquisition sessions.

[0021] The communication connection between the remote input device 1 and the humanoid robot 2, established via a wired or wireless network, can utilize various communication protocols for data transmission, including WebSocket, HTTP (Hypertext Transfer Protocol), ROS (Robot Operating System), ROS2, DDS (Data Distribution Service), UDP (User Datagram Protocol), or TCP (Transmission Control Protocol). When using wireless network communication, the humanoid robot 2 can receive control commands from a remote operator, suitable for remote data acquisition scenarios across geographical locations. The scenario architecture provided in this application supports multiple deployment methods, including local area networks, wireless networks, and public network cross-regional communication, adapting to different application environments and network conditions. Figure 1 The scenario architecture shown is an exemplary application environment for the method provided in this application and should not limit the scope of protection of this application.

[0022] Please see Figure 2 This is a flowchart illustrating a data processing method for humanoid robots, as provided in this application embodiment. Figure 2 As shown, the method in this application embodiment may include the following steps: S1. In response to the user-triggered collection configuration command, create a collection session and obtain the session context.

[0023] The robot receives user-inputted data acquisition configuration commands through an interactive interface. This interface can be a touchscreen display on the robot itself, a mobile terminal application wirelessly connected to the robot, or a graphical user interface on a remote computer. Users select or confirm the configuration parameters used for this data acquisition through this interface.

[0024] The data acquisition configuration command must include at least the input device configuration information, robot configuration information, and control mode information. The input device configuration information indicates the type, model, and communication parameters of one or more teleoperated input devices used in this data acquisition. The robot configuration information indicates the robot's model, number of joint degrees of freedom, kinematic parameters, and end effector type. This information is pre-stored in the robot's non-volatile memory, and the user only needs to select the corresponding model or configuration number for automatic loading. The control mode information indicates the control method used in this data acquisition, such as whole-body control mode, dual-arm control mode, or single-arm control mode.

[0025] The system loads the corresponding configuration file and parameters based on the acquisition configuration instructions. In one implementation, multiple preset configuration files are stored internally, each corresponding to a specific input device or robot configuration. After the user selects a configuration file, the system directly invokes the appropriate file, eliminating the need for the user to manually input complex parameters.

[0026] After creating a data acquisition session, a self-check of the current system status will be performed. This self-check includes: whether the teleoperated input device has established a normal communication connection with the robot via wireless network or Bluetooth; whether the robot's joint actuators are powered on and communicating normally; whether all camera devices are turned on; whether local storage space is sufficient; and whether necessary calibration has been completed. When all self-check items pass, a ready signal is sent to the interactive interface, prompting the user to begin data acquisition. If any self-check item fails, a corresponding prompt message will be displayed on the interactive interface, and the data acquisition session may be prevented from proceeding to the next step until the problem is resolved.

[0027] In one example, taking a typical humanoid robot teleoperation data acquisition task as an example, the user selects a VR headset and controllers as the input device type through the graphical interface displayed on the robot's chest touchscreen, selects Model A dual-arm humanoid robot as the current robot configuration, and selects full-body control mode as the control mode. Based on the user's selection, the communication configuration parameters of the VR device and the URDF model file and kinematic parameters of Model A robot are loaded. Subsequently, the system automatically checks whether the VR device is connected via wireless network, whether all joints of Model A robot are powered on and communicating normally, whether the head camera and wrist camera are turned on, and whether the local solid-state drive has more than 100GB of free space. After all checks pass, the "Start Acquisition" button on the interactive interface changes from grayed out to clickable.

[0028] S2. After the acquisition session is started, receive raw input data from at least one remotely operated input device, parse and standardize the raw input data according to the input device configuration information, and generate a unified input status.

[0029] This involves continuously receiving raw input data from teleoperated input devices. The format of the raw input data varies depending on the device type. For pose-based input devices, the raw input data typically includes the device's position coordinates and attitude angles or quaternions in three-dimensional space. For joint-based input devices, the raw input data typically includes the angle values ​​of each joint in the human body or the encoder readings of each joint of an exoskeleton. For button-based input devices, the raw input data typically includes the pressed or released state of a button, the offset of a joystick, or the travel value of a trigger.

[0030] S3. Based on the session context and unified input state, generate control objectives, perform inverse kinematics solution on the control objectives and apply safety constraints to generate executable control instructions.

[0031] Based on the control mode information and robot configuration information in the session context, the components in the unified input state are mapped to control objectives for different parts of the robot. The control objective is the desired motion quantity expressed in task space, which has not yet been kinematically solved, such as the desired pose of the end effector or the desired attitude angle of the head gimbal. The mapping process needs to consider the robot's physical configuration, number of joints, end effector type, and controllable degrees of freedom.

[0032] In one possible implementation, the robot configuration information includes a Unified Robot Description Format (UDF) model file. The UDF is an XML-based robot model description specification used to describe information such as the robot's links, joints, geometry, kinematic structure, joint constraints, and collision profile. This model file records parameters such as the number of joints, the degree of freedom type of each joint, link length and mass, joint range of motion, and the collision bounding body of each link using a standardized data format. Since humanoid robots of different configurations can all have their physical structure and kinematic parameters described using the UDF model file, simply loading the corresponding robot's model file automatically retrieves all of the robot's structural information.

[0033] The inverse kinematics solution for the control target is built upon a unified robot description format model file. Specifically, the link and joint definitions recorded in the model file are parsed, and the robot's kinematic tree structure is automatically constructed based on the type and connection relationship of each joint. Based on this kinematic tree structure, forward kinematic functions are established from the robot base to each end effector, describing the mapping relationship between the joint angles and the end effector poses. When solving the inverse kinematics, the forward kinematic functions are used as a basis, combined with the joint limit parameters recorded in the model file, to perform the inverse kinematics solution for the control target. Since the differences in kinematic structure between different robots are all reflected in the model file, there is no need to develop separate kinematic solution logic for different robot models; simply loading the corresponding model file automatically adapts to different robot configurations.

[0034] Collision detection within safety constraints is also based on a unified robot description format model file. Specifically, the model file contains collision model descriptions for each link, using basic geometry or triangular meshes to approximate the spatial occupancy of each link. The collision model definitions in the model file are parsed to construct the collision bounding volumes for each link. During collision detection, the distance between the collision bounding volumes of each link is calculated based on the position and orientation of each link in the current candidate joint state. When this distance is less than a preset safety distance threshold, a collision risk is identified. Since the differences in collision shapes between different robots are all reflected in the model file, there is no need to configure collision detection parameters separately for different robot models; simply loading the corresponding model file automatically adapts to robots with different configurations.

[0035] S4. Send executable control commands to each execution component of the robot for execution, and collect multimodal data according to a unified global synchronization time source. Associate the multimodal data with a unified timestamp and frame number to generate a multimodal data record for the current frame.

[0036] The robot's processor sends executable control commands to each actuator via an internal bus or communication interface. Actuators include joint actuators, dexterous hand or gripper controllers, and a motion chassis controller. The format of the executable control commands varies depending on the type of actuator. For joint actuators, executable control commands are in the form of joint position commands, containing the target angle value for each joint; for velocity-based control, executable control commands are in the form of joint velocity commands. For dexterous hand or gripper controllers, executable control commands contain the target angle of each finger joint or the target opening / closing amount of the gripper. For motion chassis controllers, executable control commands contain the desired linear velocity and desired angular velocity of the chassis. In multimodal data, all data streams originate from a common clock source, but each data stream generates an independent timestamp at its respective acquisition time; therefore, the timestamp values ​​are different between different frames and between different data sources within the same frame.

[0037] Multimodal data alignment is performed during the offline processing phase after data acquisition. The timestamps of each frame recorded in the structured dataset are read, and using a unified timestamp as a baseline, the data frame closest to the current frame's timestamp on the timeline is found for each data source. Specifically, for a given time point to be aligned, the frame with the smallest distance to that time point is found in the timestamp sequence of each data source, and the corresponding data value is extracted as the alignment data for that data source at that time point. Once the nearest neighbor frame has been found for each data source, the alignment data from all data sources are combined into a complete alignment record.

[0038] S5. In response to a user-triggered stop command or session end event, terminate the acquisition session and save the multimodal data of all frames as a dataset.

[0039] Throughout the acquisition process, the system continuously monitors for stop commands and session end events. Stop commands are triggered by the user through the interactive interface. Session end events include, but are not limited to: the acquisition time reaching the preset maximum duration, storage space nearing exhaustion, communication with the remote input device being lost, or the user closing the acquisition application.

[0040] When a user-triggered stop command or session end event is detected, the system stops receiving new raw input data and control commands, and ceases writing new data to the multimodal data record of the current frame. Simultaneously, it continues writing all currently received but not yet fully written multimodal data to the storage medium until all received data frames are confirmed to be completely saved. After all received data has been written, all currently open file handles are closed, and the session status is marked as ended. The system also writes session metadata information to a separate configuration file, including session identifier, acquisition start time, acquisition end time, total number of frames, input device list, robot model, control mode, task name, and operator remarks. Through this method, the system ensures that all received data is completely written when acquisition stops, avoiding data loss or incompleteness caused by immediate interruption.

[0041] When a user-triggered stop command, session end event, or device malfunction is detected, a safe stop operation is executed first, followed by session termination or malfunction handling procedures. The safe stop operation sends corresponding safe stop commands to each actuator according to the current control mode. In position control mode, a current position hold command is sent to each joint actuator to maintain the robot's actuators at their current feedback position at the time of stop, without generating additional displacement. In speed control mode, a zero-speed command is issued to each joint actuator and chassis controller to stop the movement of each actuator. After the safe stop command is sent, new control inputs from teleoperation input devices are no longer accepted, and no further parsing, control mapping, or data recording processing is performed on the newly added raw input data. Through this method, this system ensures the robot can safely stop when a session terminates or an malfunction occurs, effectively preventing the risk of robot loss of control due to command interruption or data loss, and ensuring the safety of the equipment and operator.

[0042] After the session terminates, all multimodal data records from all frames are organized into a complete dataset. The dataset uses the session identifier as its root directory name, which contains multiple subdirectories and files. Structured data files, raw log files, image files, video files, and metadata configuration files are all stored in this root directory.

[0043] After the dataset is saved, the raw data in the dataset is immediately subjected to quality verification. Quality verification includes at least image quality verification and acquisition frequency verification. Image quality verification is used to detect whether there are issues such as image freezing, blurring, or abnormal brightness during the acquisition process. Specifically, this includes calculating the structural similarity between adjacent frames to determine if an image is frozen, calculating the high-frequency component energy of the image to assess blur, and statistically analyzing the mean and standard deviation of image pixel brightness to determine whether the brightness range is within the normal range. Acquisition frequency verification is used to detect the sampling stability of each data source. Specifically, this includes calculating the variance of the actual acquisition frequency of each data source. When the variance of the acquisition frequency of a data source exceeds the preset tolerance range, it is determined that the data source may have frame dropping or acquisition instability issues. Based on the verification results, a raw data quality report is generated, and when anomalies are found, the user is prompted with the corresponding fault type and suggested handling measures through the interactive interface.

[0044] In one possible implementation, the raw input data is parsed and standardized based on the input device configuration information to generate a unified input state, including: S21. Determine the analytical function and calibration parameters corresponding to the remote operation input device based on the input device configuration information. The calibration parameters include at least one of the coordinate system transformation matrix, zero offset, scaling factor, and filtering parameters.

[0045] The robot is equipped with multiple device adapters, each corresponding to a type of teleoperated input device. Each device adapter encapsulates the communication protocol, data format definition, and parsing logic for that type of device. When a data acquisition session begins, the robot reads the input device configuration information from the session context. This configuration information includes the device type and model selected by the user.

[0046] The system locates the corresponding device adapter based on the input device configuration information and extracts the parsing function from the adapter. The parsing function is an executable piece of parsing logic used to convert the numerical format of the raw input data into a unified data structure within the system. The raw input data formats differ between different device types. Pose-based devices typically output position coordinates and attitude angles or quaternions; joint-based devices typically output encoder values ​​or angle values ​​for each joint; and button-based devices typically output digital signals or analog values. The parsing function's role is to unify these diverse data formats into a standard format that the processor can recognize and process.

[0047] Calibration parameters are a set of values ​​obtained beforehand through a calibration process, used to eliminate systematic differences between different devices. The specific content of the calibration parameters depends on the type of equipment and control requirements.

[0048] A coordinate transformation matrix is ​​used to convert position and attitude data from the input device's coordinate system to a unified teleoperation coordinate system. In a teleoperation system, the space in which the operator resides and the space in which the robot resides typically have different coordinate system origins and axis directions. The coordinate transformation matrix establishes a mapping relationship between the two spaces, enabling the operator's actions to be accurately reproduced in the robot's space. The coordinate transformation matrix is ​​typically obtained through two methods: manual calibration and automatic calibration. In manual calibration, the operator places the input device in a preset calibration pose, records the device readings, and calculates the transformation matrix. In automatic calibration, the transformation matrix is ​​automatically solved using multiple sets of corresponding point pairs.

[0049] Zero-point offset is used to eliminate the initial zero-point offset of the input device. Most input devices do not have zero readings when not worn or operated, but rather have a fixed offset value. Before acquisition begins, device readings over a period of time are recorded, and the average value is calculated as the zero-point offset. In subsequent processing, this zero-point offset is subtracted from all readings, ensuring the device's output is zero in its initial state.

[0050] The scaling factor is used to map the operator's range of motion to the robot's range of motion. Because of dimensional differences between the operator and the robot in areas such as arm length and joint travel, the operator's range of motion needs to be scaled to fit the robot's physical structure. The scaling factor is typically a positive value, and its value can be determined based on the ratio of the operator's height and arm length to the robot's arm length, or it can be adjusted according to task requirements. For example, in fine manipulation tasks, a smaller scaling factor can be used to reduce the robot's movements, thereby improving the control resolution of fine manipulation.

[0051] Filtering parameters are used to smooth the input signal. During the acquisition process, remotely operated input devices are inevitably affected by factors such as hand tremors, sensor noise, and communication interference, resulting in high-frequency noise components in the input signal. A low-pass filter is used to filter the input signal, removing high-frequency noise and making control commands smoother. Filtering parameters include at least the filter's cutoff frequency and time constant. The cutoff frequency determines the range of signal frequencies the filter retains; signal components below the cutoff frequency are retained, while those above are attenuated. The time constant determines the filter's response speed to signal changes; a larger time constant results in smoother filtering, but also a greater response delay.

[0052] S22. Using analytical functions and calibration parameters, the original input data is converted into a standardized input state.

[0053] Raw input data from the teleoperated input device is received during each frame acquisition cycle. The raw input data is encapsulated in a format defined by the device manufacturer and contains all sensor readings of the device at the current moment. The parsing function determined in step S21 is called to extract the values ​​of each field from the raw data packet according to the device's data format definition.

[0054] For pose data, the following method is used. Let T_h,t be the raw pose data read from the input device at a certain moment, representing the position and orientation of the input device in its own coordinate system. First, the pose in quaternion form is converted into a rotation matrix. Then, the position coordinates and the rotation matrix are combined into a 4x4 homogeneous transformation matrix. The pose in the input device coordinate system is transformed to a unified teleoperation coordinate system using the coordinate system transformation matrix. The transformed pose is calculated as follows: T u,t =T base ·T calib ·T h,t .

[0055] Where T h,t T represents the homogeneous pose matrix of the teleoperation input device relative to the device tracking coordinate system or the device reference coordinate system at time t. calib T represents the calibration transformation matrix from the input device coordinate system to the unified teleoperation coordinate system. base This represents the transformation matrix from the unified teleoperation coordinate system to the robot's base coordinate system. It serves as a common reference coordinate system between different teleoperation input devices, enabling pose data output from different input devices to be transformed into the same spatial representation. For head-mounted devices, the transformed pose T... u,t This represents the target pose in the transformed robot base coordinate system. For handheld devices, the transformed pose represents the equivalent position and orientation of the operator's hand in robot space.

[0056] In absolute pose control mode, the raw input data transmitted by the teleoperated input device includes pose data, which contains position and attitude information. Position information is in three-dimensional spatial coordinates, and attitude information is in quaternions or Euler angles. After acquiring the pose data, the position information is first scaled according to a pre-configured scaling factor, which maps the operator's movement range to the robot's movement range. The scaled position information is then combined with calibration parameters to generate a unified input state.

[0057] When generating a control target based on a unified input state, if there is a deviation between the target pose indicated by the control target and the robot's current actual pose, an interpolation method is used to control the robot to gradually move from the current pose to the target pose. Specifically, the difference between the target pose and the current actual pose is calculated, and an intermediate pose sequence is generated according to a preset interpolation step size or interpolation speed, which is then sent to each execution unit frame by frame. During each frame control cycle, the difference between the current pose input by the teleoperation input device and the robot's current actual pose is continuously monitored. When the difference is less than a preset threshold, the interpolation process is exited, and the robot resumes normal follow-up control mode. This method effectively avoids motion jumps caused by inconsistencies between the teleoperation input device and the robot's pose, making the robot's motion transitions smoother and more natural.

[0058] For joint data, it is processed as follows. Let θ be the raw joint angle read from the input device at a certain moment. t First, subtract the zero-position offset θ0, then multiply by the scaling factor S to obtain the standardized joint control value. The standardization process is calculated as follows: u joint,t =θ0+S·(θ t -θ0).

[0059] When the original input data is in velocity form, the velocity value is fed into a low-pass filter for smoothing. Let the original velocity value be v. raw,t The velocity value v after filtering filtered,t Calculated as follows: v filtered,t =α·v raw,t +(1-α)·v filtered,t-1 .

[0060] Where α is the filter coefficient, and its value is determined by the cutoff frequency and the acquisition period in the filter parameters.

[0061] For button and joystick data, the raw values ​​are read directly and normalized. Button states are typically Boolean values, 1 for pressed and 0 for released, which are converted into semantic commands within the system. Joystick values ​​are typically analog quantities, ranging from -1 to +1, which are directly used as the scaling factor for speed commands.

[0062] After the above processing, all categories of input data are transformed into a unified data structure. This data structure contains multiple fields such as head target pose, left hand target pose, right hand target pose, finger bending amount, button semantic commands, and joystick control amount. The value of each field is within a preset standard range.

[0063] S23. When multiple remote operation input devices participate in control at the same time, multiple standardized input states are fused according to control mode information to generate a unified input state; the unified input state includes at least one of head target state, hand target state, finger or gripper state, button semantic state, joystick semantic state and speed command state.

[0064] In complex teleoperation tasks, operators often use multiple input devices simultaneously to express different control intentions. For example, an operator might use a VR headset to control the robot's head orientation and viewpoint, a data glove to control the robot's finger movements, and a controller to control the robot's chassis movement. In such multi-device collaborative scenarios, it is necessary to integrate standardized input states from multiple devices into a unified control state according to certain rules for use by subsequent control mapping modules.

[0065] The fusion method is determined based on the control mode information in the session context. This control mode information is set by the user during the data acquisition configuration phase and describes the master-slave relationship and control allocation among multiple devices in this data acquisition. Different control modes correspond to different fusion strategies.

[0066] In master-slave fusion mode, the input from one device is used as the master control variable, and the input from the other device is used as the auxiliary control variable. The master control variable has higher priority, and the auxiliary control variable only takes effect when the master control variable is not activated or is within its dead zone. Master-slave fusion mode is suitable for scenarios that require a combination of fine-grained operation and coarse-grained movement.

[0067] In the partitioned fusion mode, control parameters from different devices are allocated to different parts of the robot. Each device independently controls one or more parts of the robot, with no overlap or competition for control between parts. The partitioned fusion mode is suitable for scenarios requiring full-body coordinated control, such as using a VR headset to control the head, using data gloves to control the hands, and using an exoskeleton to control the torso and legs.

[0068] In weighted fusion mode, the same control input from multiple devices is weighted and averaged. Weighted fusion mode is suitable for scenarios where multiple operators collaboratively control the same robot, or where a single operator uses multiple redundant devices to provide the same control signal. Weighted fusion improves the robustness and accuracy of control.

[0069] In priority fusion mode, the control input of a device is selected according to a preset priority order. When the control input of a higher priority device is valid, the control input of a lower priority device is ignored.

[0070] Regardless of the fusion method used, the final unified input state has a fixed data structure. Subsequent control mapping modules only need to read this data structure to obtain all the operator's control intentions, without needing to know which specific device generated it.

[0071] The unified input state contains standardized values ​​for each control channel of the operator. The head target state describes the desired orientation of the operator's head, typically expressed as quaternions or Euler angles. The hand target state describes the desired position and posture of the operator's left and right hands, typically expressed as three-dimensional position coordinates and three-dimensional posture angles. The finger or gripper state describes the desired degree of flexion of each finger or the desired opening / closing of the gripper, typically expressed as normalized values ​​ranging from 0 to 1. The button semantic state describes the control command corresponding to the button pressed by the operator. The joystick semantic state describes the speed or position command corresponding to the joystick movement by the operator. The speed command state describes the operator's desired movement or rotation speed.

[0072] In one possible implementation, the inverse kinematics of the control objective are solved and safety constraints are applied to generate executable control instructions, including: S31. Determine the set of joints and the set of frozen joints to participate in inverse kinematics optimization based on the robot configuration information, and keep the joints in the set of frozen joints at preset values ​​or feedback values ​​from the previous frame.

[0073] The robot's physical structure consists of multiple joints and links, each joint having one or more degrees of freedom. During inverse kinematics solving, not all joints need to participate in the optimization calculation. Based on the current control mode and task requirements, all joints of the robot are divided into two sets: the set of joints participating in the optimization and the set of frozen joints.

[0074] Optionally, this application can generate task space control targets or joint space control targets based on a unified input state; when the control target is the task space pose, inverse kinematics solution is performed; when the control target is the joint state, end effector state, or moving chassis speed, corresponding control mapping and safety constraint processing are performed.

[0075] The set of joints involved in the optimization is the target joint that the inverse kinematics solver needs to calculate. The joint angle values ​​of these joints are updated in real time according to changes in operator input, used to drive the robot's end effector or specific parts to reach the desired position and orientation. Typically, the joints involved in the optimization include the degrees of freedom joints of the arm and the head.

[0076] Joints in the frozen joint set are not involved in the inverse kinematics solution. The angle values ​​of these joints are fixed to preset values ​​or feedback values ​​from the previous frame and do not change with operator input. The selection of the frozen joint set is determined based on control mode information. In different control modes, the same joint may be assigned to different sets.

[0077] For example, in full-leg locked mode, all joints of the robot's legs are frozen in the current standing posture, allowing the robot to maintain stable support when performing arm operations. In chassis-moving mode, the leg joints may be frozen in a fixed standing posture, with the moving chassis handling the motion. In full-body control mode, all joints participate in optimization, and the set of frozen joints is empty.

[0078] There are two ways to maintain a frozen joint. The first is to maintain it at a preset value, which is usually the joint angle when the robot is in its initial standing posture. The second is to maintain it at the feedback value of the previous frame. That is, in each frame acquisition cycle, the actual joint angle of the robot at the current moment is read, and this angle is used as the fixed value of the joint in the next frame solution. Maintaining it at the feedback value of the previous frame allows the robot to maintain its current actual posture at the frozen joint, avoiding motion jumps caused by sudden changes in joint values.

[0079] S32. Construct an inverse kinematics optimization objective function, which includes an end-effector pose tracking error term and a joint change smoothing term between adjacent frames.

[0080] The essence of inverse kinematics is an optimization problem: finding a set of optimal joint angle values ​​that, under various constraints, make the pose of the robot's end effector as close as possible to the desired control objective. The constructed objective function serves as a quantitative standard for evaluating the quality of the solution.

[0081] The objective function is composed of a weighted sum of several sub-terms. The first sub-term is the end-effector pose tracking error term, which evaluates the deviation between the actual pose of the robot's end effector and the control target under the current joint configuration. The specific form of the end-effector pose tracking error term depends on the type of control target. When the control target is end-effector position, the tracking error term is typically expressed as the sum of squares of the position error vector. When the control target is end-effector pose, the tracking error term is typically expressed as a measure of pose error. When multiple end-effector control targets exist simultaneously, the end-effector pose tracking error term is a weighted sum of the individual end-effector error terms.

[0082] The second sub-item is the adjacent frame joint change smoothing term, which limits the variation between the current joint configuration and the actual joint configuration in the previous frame. This sub-item is introduced to make the robot's movements smoother and more consistent, avoiding jitter or abrupt changes caused by unstable operator input or solver numerical fluctuations. The adjacent frame joint change smoothing term is typically expressed as the sum of the squares of the differences between the joint angle values ​​in the current frame and the joint angle values ​​in the previous frame, multiplied by a smoothing weight coefficient.

[0083] In actual construction, the two sub-items mentioned above are weighted and combined according to preset weight coefficients to form a complete optimization objective function. The value of the weight coefficients affects the characteristics of the solution result. The larger the weight coefficient of the end-effector pose tracking error term, the higher the tracking accuracy of the solution result, but it may sacrifice motion smoothness. The larger the weight coefficient of the joint change smoothness term between adjacent frames, the smoother the motion of the solution result, but it may sacrifice tracking accuracy. A set of debugged default weight coefficients are pre-stored in the system, which can be applied to most common task scenarios.

[0084] S33. Solve the objective function and apply joint limit constraints and joint velocity limit constraints during the solution process to obtain the candidate joint states.

[0085] After constructing the objective function, a pre-defined numerical optimization solver is invoked to solve it. The solver iteratively calculates and finds the joint angle configuration that minimizes the objective function. Since the objective function is non-linear, the solution process typically employs iterative optimization algorithms, such as gradient descent, Newton's method, the Levenberg-Marquardt algorithm, or sequential quadratic programming. The solver iterates from an initial guess, calculating the objective function value and gradient direction for the current joint configuration in each iteration, and then updating the joint angle values ​​along the gradient descent direction until the objective function converges or the maximum number of iterations is reached.

[0086] During the solution process, two types of constraints are applied simultaneously. The first type is joint limit constraints, used to ensure that the angle value of each joint obtained does not exceed the mechanical limit range of that joint. The limit range of each joint is determined by the robot's physical structure, including negative and positive limit values. When the solver attempts to adjust the angle value of a joint beyond its limit range, the constraint function restricts the angle value of that joint to the limit boundary. The second type is joint speed limit constraints, used to ensure that the change in joint angle between adjacent frames does not exceed the maximum allowable speed value of that joint. The maximum allowable speed value of each joint is determined by the robot's motor performance and reducer specifications, and is measured in degrees per second or radians per second. Under a given acquisition period, the maximum allowable speed value is converted into the maximum allowable change within a single frame. When the solver attempts to allow a joint to change too much within a single frame, the constraint function restricts the change of that joint to within the maximum allowable change range.

[0087] After the above solution and constraint application process, a set of joint angle values ​​that satisfy all constraints is obtained. This set of values ​​is the candidate joint state. The candidate joint state contains the target angle values ​​of all joints involved in the optimization.

[0088] S34. Perform safety checks on the candidate joint status. Safety checks include joint boundary crossing detection, speed over-limit detection, and collision risk detection.

[0089] Although the candidate joint states are already constrained during the solution process, due to the approximation of numerical solutions and potential boundary conditions during constraint application, the solution results may still have slight out-of-bounds limits or be in an unsafe state. Before converting the candidate joint states into executable control commands, an independent safety check is performed to ensure that the commands issued to each execution component are absolutely safe.

[0090] Joint out-of-bounds detection involves checking each joint angle value in the candidate joint state one by one to verify whether each joint angle value is still within the corresponding limit range. Unlike the constraint application during the solution process, the purpose of this detection is to discover any possible out-of-bounds situations, providing a basis for the selection of subsequent processing strategies.

[0091] Speed ​​over-limit detection compares the candidate joint state with the actual joint state of the previous frame, calculates the angle change of each joint, and then checks whether the change exceeds the maximum allowable change in a single frame.

[0092] Collision risk detection is used to assess whether there is a risk of self-collision between the links of the robot in candidate joint states, and whether there is a risk of collision between the robot and its surrounding environment. Self-collision detection is implemented through a robot collision model, which defines the geometry, relative positional relationships, and joint range of motion of each link. Based on the candidate joint states, the position and orientation of each link in three-dimensional space are calculated. Then, it checks whether there is overlap or excessive proximity between the bounding volumes of each link. If the distance is less than a preset safety threshold, a risk of self-collision is determined to exist.

[0093] Environmental collision detection is used to assess whether a robot's links are in contact with or too close to obstacles in the environment. In one implementation, environmental information is acquired using a camera, and the acquired image data is used to generate a point cloud model of the environment. The bounding volumes of each robot link are projected onto this point cloud model, and the distances between each link and any point in the point cloud are calculated. If the distance between a link and any point in the point cloud is less than a preset safety threshold, that link is considered to have a risk of collision with the environment. In another implementation, collision detection can also be performed based on a pre-established environmental geometric model, that is, by performing intersection tests or distance calculations between the bounding volumes of each robot link and the environmental geometric model. These methods combine visual perception and preset models to achieve accurate detection of environmental collision risks.

[0094] Collision models can be approximated based on simple geometry, such as using spheres, cylinders, or cuboid bounding boxes to approximate the robot's links. Simple geometry models are computationally fast and suitable for real-time detection scenarios.

[0095] S35. When a joint out of bounds is detected, the first processing strategy is executed. The first processing strategy is to reject the current solution result and retain the safety instruction of the previous frame.

[0096] After the safety verification is completed, a corresponding response is made based on the verification results. If the verification results show that a joint has exceeded its limits or the speed has exceeded its limits, it indicates that the solution result for the current frame has reached or exceeded the robot's physical limits. In this case, issuing control commands for this frame may cause robot joint overload, motor stalling, mechanism jamming, or structural damage.

[0097] The approach to handling such serious security issues is to implement the first processing strategy. Under this strategy, the candidate joint states of the current frame are discarded and not issued as executable control commands. Instead, the security commands that were actually issued to each execution component in the previous frame and confirmed for execution are resent to each execution component.

[0098] In this way, the robot continues to operate in the safe state of the previous frame, without taking dangerous actions due to abnormal commands in a single frame. Although this approach causes the robot to stop responding to operator input in that frame, the brief delay or pause is acceptable compared to the potential consequences of issuing dangerous commands.

[0099] S36. When a collision risk is detected, select to execute the first processing strategy or the second processing strategy. The second processing strategy is to apply a penalty term to the optimization objective function and then solve it again.

[0100] When the safety check only detects a collision risk but not a joint overrun or speed exceeding the limit, there are two possible response strategies. The first is to execute the first processing strategy, which rejects the current solution and retains the safety instructions from the previous frame. The second is to execute the second processing strategy, which adds a collision penalty term to the optimization objective function and then resolves the problem.

[0101] The collision penalty term is a function that varies with joint configuration. Its design principle is that the penalty term is zero or close to zero when the joint configuration is far from the collision state, and increases sharply when the joint configuration is close to the collision state. By adding a collision penalty term to the objective function, the solver is guided to move away from the collision region in subsequent iterations, ultimately finding a set of joint angle values ​​that satisfy both tracking accuracy requirements and avoid collisions.

[0102] In the specific implementation of the second processing strategy, a collision penalty term is first added to the original optimization objective function. Let the original optimization objective function be E. ik Add a smoothing term, and then add a collision penalty term C to it. penalty A new optimization objective function is obtained. The solver is invoked to resolve the new objective function. All constraints in the solution process are the same as in step S33, including joint limit constraints and velocity limit constraints. The resolved joint states are then subjected to safety checks again. If the new candidate joint state passes the collision risk detection, it is issued as an executable control command for the current frame. If the new candidate joint state still has a collision risk, the second processing strategy can be executed again for a new round of solving, or the first processing strategy can be reverted to retain the command from the previous frame.

[0103] The decision-making criteria for choosing which strategy to implement may include preset priority rules, the severity of collision risk, the real-time requirements of the current task, or the user's preset preferences.

[0104] In one possible implementation, the process of generating executable control instructions in step S3 further includes collision detection and re-solving based on the detection results. A collision model pre-stored in memory is acquired, which describes the geometry of each link of the robot and its spatial occupancy. The collision model can be approximated using basic geometry, such as using a sphere, cylinder, or cuboid bounding box to cover each link. Based on the solved candidate joint states, the position and orientation of each link in three-dimensional space are calculated using forward kinematics, and then self-collision detection and environmental collision detection are performed based on the collision model. Self-collision detection checks whether there is mutual penetration or excessive proximity between the robot's links. Environmental collision detection checks whether the robot's links have come into contact with or are too close to obstacles in the environment. During collision detection, a safe distance threshold is set; when the minimum distance between two objects is less than this threshold, a collision risk is considered to exist. A typical safe distance threshold is between 0.02 meters and 0.05 meters, with the specific value determined based on the robot's size and the task scenario.

[0105] When a collision risk detection passes, the current candidate joint state is directly output as an executable control command to each actuator. When a collision risk is detected, either a first processing strategy or a second processing strategy is selected based on preset decision logic. If the first processing strategy is selected, the current candidate joint state is rejected and the safety command from the previous frame is maintained. If the second processing strategy is selected, the optimization objective function is modified by adding a collision penalty term to the original objective function. The collision penalty term is a function that varies with the joint configuration. In a candidate joint state, when the distance between two links of the robot is less than the safe distance threshold, the collision penalty term outputs a positive penalty value, and this value increases sharply as the distance decreases; when the distance between all links is greater than the safe distance threshold, the collision penalty term outputs a value of zero.

[0106] This application also adds an optimization objective function for the collision penalty term, which is resubmitted to the solver for solving. During the process of minimizing the optimization objective function, the solver adjusts the joint angles to move the links further apart to reduce the cost of the collision penalty term. This ensures that the new solution meets both end-effector pose tracking requirements and collision safety requirements. After the re-solution is completed, safety checks are performed again on the new candidate joint states. These checks include joint boundary crossing detection, velocity over-limit detection, and collision risk detection. If all checks pass, the new candidate joint states are issued as executable control commands for the current frame to each execution component.

[0107] If the speed limit exceedance detection fails, the current solution is not discarded directly. Instead, an interpolation transition method is used to gradually transition the current joint state to a candidate joint state over multiple frames. Specifically, based on the difference between the candidate joint state and the current joint state, a set of intermediate joint sequences is generated in conjunction with a preset number of transition frames. These sequences are then sent to each execution component frame by frame, allowing the robot to smoothly reach the target position over multiple frames. This avoids motion interruption caused by completely rejecting the command for a single frame due to speed exceeding the limit.

[0108] If the collision risk detection fails, the target joint directly related to the collision risk among the joints involved in the optimization is first located. Based on this, the target joint is removed from the joint set and assigned to the frozen joint set. The frozen target joint retains the feedback value from the previous frame and does not participate in subsequent inverse kinematics optimization. After freezing one or more target joints, the remaining joints continue to participate in solving the objective function according to steps S32 to S33, and a safety check is performed after the solution is completed. Through this method, when a collision risk occurs, only the joints directly related to the collision are frozen, while other joints can still respond normally to the operator's control input. This allows the robot to stop the movement of dangerous joints in a timely manner under unsafe motion conditions, while maintaining the overall task execution as much as possible through the continuous movement of other joints. This reduces the overall operation interruption caused by a single joint collision and improves the continuity and stability of the teleoperation process.

[0109] The final executable control commands include various types, depending on the robot's configuration and the current control mode. For articulated robots, the executable control commands are joint position commands or joint velocity commands, i.e., target angle values ​​or target angular velocity values ​​for each joint. For robots equipped with dexterous hands or grippers, the executable control commands also include the target angles of each joint of the dexterous hand or the target opening / closing amount of the gripper. For robots equipped with a mobile chassis, the executable control commands also include the desired linear velocity and desired angular velocity of the chassis.

[0110] Full-body control encompasses not only arm movements but also synchronized control of body posture. When the operator performs forward leaning, backward leaning, or squatting movements, corresponding body control targets are generated based on a unified input state, ensuring the robot's body posture moves synchronously with the operator, thus achieving coordinated control of the torso and arms. During the inverse kinematics solution process, a center-of-mass position function is constructed based on a unified robot description format model file. This function describes the mapping relationship between each joint angle and the overall center-of-mass position of the robot. A center-of-mass constraint is also applied during the solution process, ensuring that the projection of the robot's center of mass onto the horizontal plane always falls within the supporting polygon. The supporting polygon is enclosed by the contact points between the robot's feet and the ground. When the robot is in a single-leg support state, the supporting polygon shrinks to the area of ​​the single foot's base; when the robot is in a two-leg support state, the supporting polygon expands to the area enclosed by both feet. In each iteration of the solution, the robot's center of mass projection position is first calculated based on the current candidate joint state. Then, it is determined whether this projection position is within the supporting polygon. If it is within the supporting polygon, the candidate joint state is allowed to proceed; if it is outside the supporting polygon, the solver is guided to converge towards the direction that satisfies the center of mass constraint by adding a penalty term or rejecting the candidate joint state. Through the above center of mass constraint mechanism, this method ensures the overall balance of the robot when performing actions such as tilting forward, tilting backward, and extending both arms, preventing the risk of tipping over due to the center of mass shifting beyond the support boundary, and ensuring that the whole-body teleoperation remains stable during dynamic movements. Through the above processing, it is ensured that the final control commands, while tracking the operator's intentions, meet multiple safety requirements such as joint limits, speed limits, collision avoidance, and anti-tipping, providing a safe and reliable command foundation for the synchronous acquisition of multimodal data.

[0111] In one possible implementation, while the robot's various actuators execute control commands, multimodal data is synchronously acquired and recorded according to a uniform acquisition cycle. The acquisition cycle is a pre-set fixed time interval, typically 30 Hz, 60 Hz, or 100 Hz, with the specific value determined based on the frame rate of the vision sensor and the real-time requirements of the control system.

[0112] At the start of each acquisition cycle, the unified timestamp and frame number of the current frame are first obtained. The unified timestamp comes from an internally maintained global clock with millisecond or microsecond precision and is used to identify the global time position of the current acquisition moment. The frame number is an incrementing sequence number starting from the start of the acquisition session and proceeding through the acquisition cycle, with the initial frame number being 1, and the frame number incremented by 1 after each frame of data is acquired.

[0113] The system acquires the raw input data from the remote operation input device at the current moment, as well as the unified input state obtained after processing in step S2. The raw input data is read directly from the device driver layer, maintaining the original format defined by the device manufacturer. The unified input state is read from the memory, which contains the control quantities of all operator control channels that have undergone parsing, standardization, and fusion processing.

[0114] The control target for the current frame and the executable control commands actually issued to each actuator are retrieved from memory. The control target is the desired end-effector pose or desired joint angle that has not yet been solved by inverse kinematics. The executable control commands are the command values ​​actually sent to the joint actuator, dexterous hand controller, or chassis controller after optimization and safety verification in step S3. Both are recorded simultaneously, allowing for subsequent comparison of the operator's original intent with the robot's actual executed commands.

[0115] The processor reads the robot's feedback status via an internal bus or communication interface. The feedback status includes at least the actual angle, actual velocity, and output torque reported by each joint actuator. For robots equipped with force sensors, the feedback status also includes contact force or torque information on the end effector. For robots equipped with collision detection, the feedback status also includes collision flags and collision severity levels. For robots equipped with a mobile chassis, the feedback status also includes the chassis's actual position and actual velocity in the global coordinate system. The processor parses the feedback data packets according to the communication protocols defined by each actuator manufacturer, extracting the values ​​of each field.

[0116] Multiple visual data streams are acquired from the camera system. This system includes cameras mounted on the robot's head, on its left and right wrists, on its chest, and positioned in the external environment. Each camera can be either a color camera or a depth camera; color cameras output RGB (Red, Green, Blue) images, while depth cameras output depth images. Image frames from each camera at the current moment are read through the camera data acquisition interface; each frame contains its own acquisition timestamp. For depth cameras, the data read is the depth value corresponding to each pixel. All image data is temporarily stored in a memory buffer, and the system decides whether to store the raw images directly or after compression, typically using JPEG (Joint Photographic Experts Group) or PNG (Portable Network Graphics) formats.

[0117] All the above data are combined into a complete data record using a unified timestamp and frame number as the association index. For data records containing large image data, the image data is stored as a separate image file, and referenced within the data record using the file path and filename. The complete record of the associated frame is then written to a non-volatile storage medium. This record contains the complete information chain from operator input to robot feedback to environmental perception, providing a synchronized data foundation for subsequent training, analysis, and playback.

[0118] In one possible implementation, step S4 further includes a hierarchical storage process. During the acquisition process, two parallel data write paths are maintained simultaneously: one for storing the raw data logs and the other for storing the structured dataset.

[0119] During the raw data log storage process, raw input data received from the remotely operated input device is directly written to the raw data log file without any parsing, conversion, or processing. The raw input data retains the original format defined by the device manufacturer, including raw binary data packets, raw communication frames, and unprocessed sensor readings. It is appended sequentially according to the acquisition cycle, with each raw data record marked with its corresponding frame number and reception timestamp. This method of preserving the original format provides the most fundamental basis for subsequent data playback, system debugging, and anomaly tracing. When data anomalies occur during processing, developers can trace back to the raw data log to check whether the anomaly originates from a signal problem in the input device itself, a transmission problem in the communication link, or a problem with the processor's internal processing logic, thereby accurately locating the cause of the fault.

[0120] During the storage of the structured dataset, the multimodal data records of each frame are organized into a structured dataset according to a unified data schema. The unified data schema predefines the name, data type, numerical range, and storage format of each field in each frame. Numerical data in the multimodal data records is organized in tabular form, with one row per frame and one field per column, such as frame number, timestamp, joint angles, end-effector pose, and button status. Image data is named according to frame number and camera identifier and stored as independent image files, referenced in the table by the file path field. Video stream data is stored as video files according to time segments, referenced in the table by the start and end frame numbers. Structured data is appended in real-time after each frame is acquired, and all frames' data form a complete dataset after acquisition.

[0121] After being organized in a unified data model, structured datasets are easier for subsequent automated quality inspection scripts to read and process, easier for visualization tools to load and display, and easier to convert to various robot training data formats, such as HDF5 (Hierarchical Data Format version 5) or LeRobot (LeRobot, the open-source library for the Hugging Face robot) format, through format conversion tools.

[0122] The raw data log and structured dataset are associated using the same frame number and session identifier. The frame number is strictly consistent across both data paths; the same frame number corresponds to one raw input record in the raw data log and one record containing complete multimodal information in the structured dataset. The session identifier is generated when the acquisition session is created and persists throughout the entire acquisition process, identifying all data files generated within the same acquisition session. When it is necessary to trace the source of anomalies in a frame of structured data, the corresponding raw input data can be quickly located in the raw data log using the frame number of that frame and the session identifier of the current session, achieving bidirectional traceability from processing results to raw data. After acquisition, a folder is created using the session identifier as the root directory name, storing both the raw data log and the structured dataset in this folder for easy data management and retrieval. Through this hierarchical storage mechanism, this method balances the convenience of data traceability with the efficiency of data use.

[0123] In one possible implementation, after step S1 is completed and before step S5 begins, the robot's processor also executes a session control step to manage the running state of the acquisition process.

[0124] In response to a user-triggered pause command, a pause operation is executed. During the pause state, writing new data frames to the multimodal data record ceases, and subsequent received input and control data is no longer saved to the dataset. Simultaneously, the session context and all hardware connections remain uninterrupted. The session context includes input device configuration information, robot configuration information, and control mode information. Hardware connections include communication links with teleoperated input devices, control links with each robot actuator, and data links with the camera. During the pause, the robot is controlled to maintain a safe state according to the current control mode. In position control mode, a current position hold command is sent to each actuator, keeping each joint at the command position of the last frame before the pause. In speed control mode, a zero-speed command is sent to each actuator, stopping the movement of each joint and the chassis. Through these methods, the robot is ensured to remain safely stationary during the pause, while avoiding safety risks caused by continuous chassis movement or joint creep.

[0125] This application also supports incremental control mode. In incremental control mode, the absolute pose of the teleoperated input device is not used as the control target, but rather the change in the teleoperated input device relative to the reference pose is used as the control basis. Specifically, when incremental control mode is activated, the current pose of the teleoperated input device is recorded as the anchor point, and the actual pose of the robot at the current moment is recorded as the reference pose. In subsequent control processes, the current pose of the teleoperated input device is acquired in real time, the change in the current pose relative to the pose at the anchor point is calculated, and this change is superimposed on the reference pose to generate the control target for the current frame.

[0126] The process of generating control objectives under incremental control mode can be expressed as follows: .

[0127] T goal (t) represents the control target to be generated at the current time t, i.e., the desired pose of the robot's end effector at the current time. This value will be sent to the inverse kinematics solver as the input to the solution target.

[0128] T robot (t0) represents the robot's actual pose at the moment t0 when incremental control is activated. This value serves as a reference for subsequent incremental control, ensuring that the robot does not jump at the moment incremental control is activated and smoothly continues the current pose.

[0129] T device (t0) represents the pose of the teleoperated input device at the moment t0 when incremental control is activated. This value serves as the anchor point reference for incremental control on the device side, and all subsequent device motion is calculated relative to this anchor point.

[0130] T device (t0) -1 This represents the inverse matrix of the pose of the teleoperated input device at the anchor point. Its function is to convert the absolute pose of the device at time t into a relative change relative to the anchor point. device (t0) -1 ·T device (t) represents the pose change of the device from the anchor point to the current time.

[0131] T device (t) represents the real-time pose of the teleoperated input device at the current time t. This value is reported in real time by the teleoperated input device and reflects the operator's action state at the current moment. Through the above incremental control method, when the operator needs to readjust their posture or grip position during data acquisition, the incremental control mode can be temporarily activated. Without interrupting data acquisition, the current posture is redefined as a new anchor point, keeping the robot in the current position. Subsequent operator actions can still be collected and recorded normally, thereby improving the flexibility and continuity of data acquisition.

[0132] In response to a user-triggered recovery command, a recovery operation is executed. During recovery, new data is resumed being written to the multimodal data records. After recovery, the current session identifier and existing data files are used; no new sessions or files are created. Instead, new data frames are appended directly to the end of the existing data files. The frame number of the first frame written after recovery is the frame number of the last frame before the pause, plus 1, ensuring the continuity and incrementing nature of the frame numbers. Through the coordination of pause and recovery, the operator can skip invalid segments in a continuous task process. The final generated dataset maintains continuity at the frame numbering level, and all data belongs to the same session identifier, facilitating subsequent data management and traceability.

[0133] In response to detected device anomalies or communication timeout events, an automatic termination operation is executed. Device anomalies include remote input device disconnection, communication interruption with any actuator, unresponsive camera device, or storage media write failure. Communication timeout events include failure to receive any data packets from the remote input device within a preset timeout period, typically ranging from 500 to 2000 milliseconds. When the above anomalies are detected, data writing is immediately stopped, all currently open file handles are closed, and all acquired data is completely saved to non-volatile storage media. Simultaneously, the session state is marked as abnormally terminated, and the cause and time of the anomaly are recorded. This abnormal termination operation ensures that even if a failure occurs during data acquisition, the acquired data will not be lost, providing a complete data foundation for subsequent troubleshooting.

[0134] In one possible implementation, after step S5 is completed, the robot also performs a data post-processing step, automatically checking the quality and converting the format of the saved dataset.

[0135] First, the dataset undergoes integrity verification. Integrity verification includes frame rate verification, frame count consistency verification, and image resolution verification. Frame rate verification calculates the actual acquisition frame rate of the dataset, which is the ratio of the total number of frames to the acquisition time, and compares this ratio with the preset expected frame rate. A deviation exceeding ±5% is considered a failure. Frame count consistency verification checks whether the total number of frames recorded across different data sources is consistent. For example, it checks whether the number of frames recorded in structured data matches the number of images stored in the image folder. A difference exceeding a preset tolerance is considered a failure. Image resolution verification checks whether the image resolution acquired by each camera matches the resolution set in the acquisition configuration.

[0136] Then, motion quality verification is performed on the dataset. Motion quality verification includes joint value range verification, joint distribution rationality verification, motion trajectory continuity verification, and timestamp continuity verification. Joint value range verification checks whether the angle value of each joint is within its corresponding mechanical limit range. Joint distribution rationality verification checks whether the angle value distribution of each joint conforms to the normal operating range, such as whether there are instances of prolonged stays at the limit boundaries. Motion trajectory continuity verification checks whether the joint angle changes between adjacent frames are smooth, and whether there are any jumps or abrupt changes; when the change between adjacent frames exceeds a preset threshold, it is considered abnormal. Timestamp continuity verification checks whether the timestamp interval between adjacent frames is stable, and whether there are any frame drops or timestamp rollbacks.

[0137] A quality inspection report is generated based on the results of integrity and action quality checks. The report is output in structured text format and contains basic information about the dataset, including session identifier, total number of frames, acquisition duration, and expected frame rate. The report lists the name, expected value, actual value, and result of each check, indicating a pass or fail. For failed checks, the report also includes suggested corrective actions.

[0138] A format conversion operation is performed on the dataset that passes all validation items. The format conversion transforms the structured dataset from its original storage format to the target training format, including HDF5 and LeRobot formats. HDF5 is a hierarchical data storage format suitable for storing large-scale array data and metadata. LeRobot format is a dataset format for robot imitation learning, containing a standardized organizational structure for observation and motion data. During the conversion process, numerical and image data from the structured dataset are read, reorganized and encoded according to the target format definition, generating a data file that can be directly used for model training. After conversion, the quality inspection report and the converted dataset are output to the specified output directory for subsequent model training and data reuse. Through the above post-processing steps, this method automatically completes data quality inspection and format standardization after data collection, reducing manual screening costs and improving data reuse efficiency.

[0139] The following are embodiments of the apparatus described in this application, which can be used to execute the embodiments of the method described in this application. For details not disclosed in the apparatus embodiments of this application, please refer to the embodiments of the method described in this application.

[0140] Please see Figure 3 This illustration shows a schematic diagram of a data processing apparatus for a humanoid robot provided in an exemplary embodiment of this application, hereinafter referred to as apparatus 3. Apparatus 3 can be implemented as all or part of a robot through software, hardware, or a combination of both. Apparatus 3 includes: The acquisition module 301 is used to create an acquisition session and acquire the session context in response to a user-triggered acquisition configuration command. The session context includes at least input device configuration information, robot configuration information, and control mode information. The generation module 302 is used to receive raw input data from at least one remotely operated input device after the acquisition session is started, and to parse and standardize the raw input data according to the configuration information of the input device to generate a unified input state; The constraint module 303 is used to generate a control target based on the session context and the unified input state, and to perform inverse kinematics solution on the control target and apply safety constraints to generate executable control instructions. The acquisition module 304 is used to send the executable control instructions to each execution component of the robot for execution, and to acquire multimodal data according to a unified global synchronization time source, associate the multimodal data with a unified timestamp and frame number, and generate a multimodal data record for the current frame, wherein the multimodal data includes the executable control instructions; The recording module 305 is used to terminate the acquisition session in response to a user-triggered stop command or session end event, and save the multimodal data of all frames as a dataset.

[0141] For further details regarding the implementation of the above-mentioned technical solutions by each module in the data processing device for humanoid robots, please refer to the description in the data processing method for humanoid robots provided in the above-mentioned embodiments of the invention, which will not be repeated here.

[0142] It should be noted that the above-described embodiment of the device 3, when executing the data processing method for humanoid robots, only illustrates the division of the above functional modules. In practical applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the above functions. Furthermore, the data processing device for humanoid robots and the data processing method embodiment for humanoid robots provided in the above embodiments belong to the same concept, and their implementation process is detailed in the method embodiment, which will not be repeated here.

[0143] The sequence numbers of the embodiments in this application are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0144] See Figure 4 The diagram shown is a schematic of a computer storage medium provided in an embodiment of this application. The computer storage medium 4 can store multiple instructions (i.e., Figure 4 The computer program shown above), the instructions are adapted to be loaded and executed by a processor as described above. Figure 2The method steps of the illustrated embodiment can be found in the following documentation for detailed execution. Figure 2 The specific details of the illustrated embodiments will not be elaborated here.

[0145] This application also provides a computer program product that stores at least one instruction, which is loaded and executed by the processor to implement the data processing method for humanoid robots as described in the above embodiments.

[0146] Please see Figure 5 This provides a schematic diagram of the robot's structure as an embodiment of this application. Figure 5 As shown, the robot 500 may include: at least one processor 501, at least one network interface 504, user interface 503, memory 505, and at least one communication bus 502.

[0147] The communication bus 502 is used to enable communication between these components.

[0148] The user interface 503 may include input units such as a mouse and keyboard.

[0149] The network interface 504 may optionally include a standard wired interface or a wireless interface (such as a Wi-Fi interface).

[0150] The processor 501 may include one or more processing cores. The processor 501 connects to various parts within the robot 500 using various interfaces and lines, and performs various functions and processes data of the robot 500 by running or executing instructions, programs, code sets, or instruction sets stored in the memory 505, and by calling data stored in the memory 505. Optionally, the processor 501 may be implemented using at least one of the following hardware forms: Digital Signal Processor (DSP), Field-Programmable Gate Array (FPGA), and Programmable Logic Array (PLA). The processor 501 may integrate one or a combination of several of the following: Central Processing Unit (CPU), Graphics Processing Unit (GPU), and modem. The CPU primarily handles the operating system, user interface, and applications; the GPU is responsible for rendering and drawing the content to be displayed on the screen; and the modem handles wireless communication. It is understood that the modem may also not be integrated into the processor 501 and may be implemented as a separate chip.

[0151] The memory 505 may include random access memory (RAM) or read-only memory. Optionally, the memory 505 may include a non-transitory computer-readable storage medium. The memory 505 can be used to store instructions, programs, code, code sets, or instruction sets. The memory 505 may include a program storage area and a data storage area, wherein the program storage area may store instructions for implementing an operating system, instructions for at least one function (such as touch function, sound playback function, image playback function, etc.), instructions for implementing the above-described method embodiments, etc.; the data storage area may store data involved in the above-described method embodiments, etc. Optionally, the memory 505 may also be at least one storage device located remotely from the aforementioned processor 501. Figure 5 As shown, the memory 505, which serves as a computer storage medium, may include an operating system, a network communication module, a user interface module, and application programs.

[0152] exist Figure 5 In the robot 500 shown, the user interface 503 is mainly used to provide an input interface for the user and to obtain the user's input data; while the processor 501 can be used to call the application program stored in the memory 505 and specifically execute, such as Figure 2 The method shown can be referred to for details. Figure 2 As shown, it will not be elaborated further here.

[0153] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The program can be stored in a computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. The storage medium can be a magnetic disk, optical disk, read-only memory, or random access memory, etc.

[0154] The above-disclosed embodiments are merely preferred embodiments of this application and should not be construed as limiting the scope of this application. Therefore, any equivalent variations made in accordance with the claims of this application shall still fall within the scope of this application.

Claims

1. A data processing method for humanoid robots, characterized in that, The processor applied to a robot, the method comprising: In response to a user-triggered acquisition configuration command, an acquisition session is created and a session context is obtained. Following this step, the process further includes: in response to a user-triggered pause command, pausing the writing of new data to the multimodal data record of the current frame while maintaining the session context and all hardware connections; or in response to a user-triggered resume command, resuming data writing and continuing to append new multimodal data records using the current session identifier and existing data files; or in response to a detected device anomaly or communication timeout event, automatically performing an abnormal termination operation and saving all acquired data. The session context includes at least input device configuration information, robot configuration information, and control mode information. After the acquisition session is started, raw input data from at least one remotely operated input device is received. The raw input data is parsed and standardized according to the configuration information of the input device to generate a unified input state. Based on the session context and the unified input state, a control objective is generated, and inverse kinematics is solved on the control objective and safety constraints are applied to generate executable control instructions. The executable control instructions are sent to each execution component of the robot for execution, and multimodal data is collected according to a unified global synchronization time source. The multimodal data is associated with a unified timestamp and frame number to generate a multimodal data record for the current frame. The multimodal data includes the executable control instructions. In response to a user-triggered stop command or session end event, the acquisition session is terminated, and the multimodal data records of all frames are saved as a dataset.

2. The data processing method for humanoid robots according to claim 1, characterized in that, The step of parsing and standardizing the original input data according to the input device configuration information to generate a unified input state includes: The analytical function and calibration parameters corresponding to the remote operation input device are determined based on the input device configuration information. The calibration parameters include at least one of coordinate system transformation matrix, zero offset, scaling factor and filtering parameters. Using the analytical function and the calibration parameters, the original input data is converted into a standardized input state; When multiple remote operation input devices participate in control simultaneously, the multiple standardized input states are fused according to the control mode information to generate the unified input state; The unified input state includes at least one of the following: head target state, hand target state, finger or gripper state, button semantic state, joystick semantic state, and speed command state.

3. The data processing method for humanoid robots according to claim 1, characterized in that, The robot configuration information includes the robot's unified robot description format model file. The inverse kinematics solution of the control target and the collision detection in the safety constraints are both constructed based on the unified robot description format model file.

4. The data processing method for humanoid robots according to claim 1, characterized in that, The step of performing inverse kinematics solution on the control target and applying safety constraints to generate executable control instructions includes: Based on the robot configuration information, determine the set of joints participating in inverse kinematics optimization and the set of frozen joints, and keep the joints in the set of frozen joints at a preset value or the feedback value of the previous frame. Construct an inverse kinematics optimization objective function, which includes an end-effector pose tracking error term and a joint change smoothing term between adjacent frames; The optimization objective function is solved, and joint limit constraints and joint velocity limit constraints are applied during the solution process to obtain candidate joint states; The candidate joint status is subjected to safety verification, which includes joint boundary crossing detection, speed exceeding limit detection, and collision risk detection. When a joint out of bounds is detected, the first processing strategy is executed, which is to reject the current solution result and retain the safety instructions of the previous frame. When a collision risk is detected, either the first processing strategy or the second processing strategy is selected to be executed. The second processing strategy is to apply a penalty term to the optimization objective function and then solve it again.

5. The data processing method for humanoid robots according to claim 4, characterized in that, The method further includes: Obtain the collision model of the robot, and perform self-collision detection and environmental collision detection on the solved joint states; When a collision risk is detected and the second processing strategy is selected, a collision penalty term is added to the optimization objective function, and the optimization problem with the collision penalty term is solved again. The executable control commands include at least one of joint position commands or joint speed commands, dexterous hand or gripper commands, and moving chassis speed commands.

6. The data processing method for humanoid robots according to claim 1, characterized in that, The generation of the multimodal data record specifically includes: Obtain the unified timestamp and the frame number of the current frame; Collect the raw input data and unified input status of the remote operation input device; Obtain the control target and the issued executable control instructions; Read the robot's feedback state, which includes at least one of joint feedback state, end effector feedback state, collision state, and chassis state; Acquire multi-channel visual data, wherein the multi-channel visual data includes at least one of head camera images, wrist camera images, chest camera images, and external camera images; All the above data are associated with the unified timestamp and the frame number and encapsulated into a multimodal data record for the current frame.

7. The data processing method for humanoid robots according to claim 4, characterized in that, The method further includes: The original input data is saved as an original data log, which is used for data playback, system debugging, and anomaly tracing. The multimodal data records of each frame are organized into a structured dataset according to a unified data pattern. The structured dataset is used for data quality inspection, visualization, and training format conversion. The raw data log and the structured dataset are associated with the same frame number and session identifier.

8. The data processing method for humanoid robots according to claim 1, characterized in that, After terminating the acquisition session in response to a user-triggered stop command or session end event, and saving the multimodal data records of all frames as a dataset, the method further includes: The dataset is subjected to integrity verification, which includes verifying at least one of frame rate, frame count consistency, and image resolution. The dataset is subjected to motion quality verification, which includes verifying at least one of the following: joint value range, joint distribution rationality, motion trajectory continuity, and timestamp continuity. A quality inspection report is generated based on the verification results. The dataset that passes the verification is converted into the target training format and output. The target training format includes HDF5 format or LeRobot format.

9. A data processing device for humanoid robots, characterized in that, include: The acquisition module is used to create an acquisition session and acquire the session context in response to a user-triggered acquisition configuration command. After creating the acquisition session and acquiring the session context in response to the user-triggered acquisition configuration command, the module further includes: pausing the writing of new data to the multimodal data record of the current frame in response to a user-triggered pause command, while maintaining the session context and all hardware connections; or resuming data writing in response to a user-triggered resume command, continuing to append new multimodal data records using the current session identifier and existing data files; or automatically performing an abnormal termination operation and saving all acquired data in response to a detected device anomaly or communication timeout event. The session context includes at least input device configuration information, robot configuration information, and control mode information. The generation module is used to receive raw input data from at least one remotely operated input device after the acquisition session is started, and to parse and standardize the raw input data according to the configuration information of the input device to generate a unified input state; The constraint module is used to generate control objectives based on the session context and the unified input state, perform inverse kinematics solution on the control objectives and apply safety constraints to generate executable control instructions; The acquisition module is used to send the executable control instructions to each execution component of the robot for execution, and to acquire multimodal data according to a unified global synchronization time source, associate the multimodal data with a unified timestamp and frame number, and generate a multimodal data record for the current frame, wherein the multimodal data includes the executable control instructions; The recording module is used to terminate the acquisition session in response to a user-triggered stop command or session end event, and save the multimodal data of all frames as a dataset.

10. A robot, characterized in that, include: A processor and a memory; wherein the memory stores a computer program adapted to be loaded by the processor and to execute the steps of the data processing method for humanoid robots as described in any one of claims 1 to 8.

Images