Processing method, device and storage medium of control information of robot

Abstract

The application discloses a processing method, device and storage medium of control information of a robot. The method is applied to an electronic device, and the method comprises the following steps: acquiring action information of a user detected by at least one sensor in a wearing module; determining wearing information based on the action information, wherein the wearing information comprises arm orientation information, arm position information and finger joint information of the user; mapping the wearing information to a reference system of an execution module of a robot to obtain initial instructions at a current time; performing smoothing processing on the initial instructions at the current time to obtain target execution instructions, wherein the target execution instructions are used for instructing the robot to execute corresponding actions according to the action information of the user, and the target execution instructions are obtained by correcting the initial instructions according to a time change trend of historical execution instructions; and controlling the robot to execute the target execution instructions. The application can realize accurate and smooth mapping of robot instructions.

Description

Technical Field

[0001] This application belongs to the field of computer technology, and more specifically, relates to a method, apparatus and storage medium for processing control information of a robot. Background Technology

[0002] Dual-arm dexterous robots are a type of dexterous robot that integrates humanoid arms and dexterous hands. They can perform the same or corresponding robotic actions based on the operator's movements. Dual-arm dexterous robots are designed to handle complex and delicate tasks and represent an important direction in current robotics technology development. A dual-arm dexterous robot system has dozens of degrees of freedom. Accurately and in real-time mapping the operator's arm and hand movements onto the robot, while avoiding collisions and instability points, is a complex technical challenge. For example, existing dexterous robot control processes may suffer from any of the following problems: existing absolute attitude mapping methods may cause deviations in robot movement direction due to different initial operator orientations; existing direct smoothing methods may mask systematic tracking errors; and existing manual intervention methods that directly switch commands may cause mechanical shocks. Summary of the Invention

[0003] The purpose of this application is to provide a method, apparatus, and storage medium for processing robot control information, which can improve the stability of robot movements and avoid erroneous collisions.

[0004] In a first aspect, embodiments of this application provide a method for processing control information of a robot, applied to an electronic device. The method includes: acquiring user motion information detected by at least one sensor in a wearing module; determining wearing information based on the motion information, the wearing information including the user's arm orientation information, arm position information, and finger joint information; mapping the wearing information to a reference frame of the robot's execution module to obtain an initial instruction at the current moment; smoothing the initial instruction at the current moment to obtain a target execution instruction, the target execution instruction being used to instruct the robot to perform a corresponding action according to the user's motion information, the target execution instruction being obtained by correcting the initial instruction based on the trend of historical execution instructions changing over time, the historical execution instructions being instructions executed by the robot at moments prior to the current moment; and controlling the robot to execute the target execution instruction.

[0005] The method provided in this application acquires user action information detected by the wearable module's sensors, determines wearing information including arm orientation, position, and finger joints, maps this information to the robot's execution module's reference frame to obtain initial commands, and then smooths the initial commands to obtain target execution commands. This supports lightweight pose sensing devices, reducing the cost and deployment complexity of teleoperation hardware. The smoothing process effectively suppresses sensor noise and communication jitter, ensuring stable and jitter-free robot end-effector movement. The reference frame mapping corrects the coordinate semantic differences between the operator's space and the robot's space, ensuring accurate and intuitive motion mapping and achieving intuitive control. Furthermore, this method is not limited to specific sensor types and algorithms, exhibiting good versatility and platform scalability.

[0006] In a possible implementation, mapping the wearing information to the reference frame of the execution module to obtain the initial instruction of the execution module at the current moment includes: performing continuous constraint motion mapping on the orientation information and / or position information based on the reference frame of the execution module to obtain a first instruction of the execution module, wherein the first instruction represents the target orientation that the robot needs to reach based on the orientation information and / or the target position that it needs to reach based on the position information; performing finger mapping on the finger joint information based on the reference frame of the execution module to obtain a second instruction of the execution module, wherein the second instruction represents the target finger joint action that the robot performs based on the finger joint information; and using the first instruction and the second instruction as the initial instruction.

[0007] This application embodiment processes the robotic arm motion and the dexterous hand motion separately, fully considering the differences in kinematic structure between the two arms (including the robotic arm) and the dexterous hand, making the mapping more accurate; at the same time, the continuity constraint ensures a smooth transition of instructions between adjacent moments, avoiding joint command jumps caused by independent frame-by-frame solving, and improving the stability and safety of robot motion.

[0008] In a possible implementation, the step of performing continuous constraint motion mapping on the orientation information and / or position information based on the reference frame of the execution module to obtain the first instruction of the execution module includes: determining the position difference between the position information at the previous moment and the position information at the current moment in the reference frame; determining the manipulator position information mapping information based on the position difference, the abstract positional relationship information between the manipulator and the execution module, the motion scale between the manipulator and the execution module, and the reference position information of the execution module, wherein the manipulator position mapping information is the mapping information of the robot's arm position in the robot's reference frame; determining the manipulator orientation information mapping information based on the reference orientation information of the execution module, the inverse matrix of the orientation information at the previous moment, and the orientation information at the current moment, wherein the manipulator orientation information mapping information is the mapping information of the robot's manipulator in the robot's reference frame; and obtaining the first instruction based on the manipulator position information mapping information and the manipulator orientation information mapping information.

[0009] In this embodiment, when mapping orientation information and / or position information to a first command, the arm position mapping information is determined based on the position difference between the previous and current moments, the abstract position relationship matrix, the motion scale, and the reference position information. The arm orientation mapping information is determined based on the reference orientation information, the inverse matrix of the orientation information from the previous moment, and the orientation information from the current moment. The first command is then obtained by combining these information. The independent mapping of position and orientation ensures that the operator's initial orientation does not change the semantic direction of the translational motion, effectively avoiding deviations in the motion direction. By introducing the state from the previous moment as a continuity constraint, the temporal consistency of the mapping results is guaranteed, preventing false translations and jitter. The abstract position relationship matrix can be flexibly calibrated according to different deployment scenarios, improving the system's adaptability to different robot platforms.

[0010] In a possible implementation, the step of mapping the finger joint information based on the reference frame of the execution module to obtain the second instruction of the execution module includes: obtaining finger joint mapping information in the reference frame based on the robot arm position mapping information, robot arm orientation mapping information and the finger joint information at the previous moment, wherein the finger joint mapping information represents the finger joint movement of the robot's fingers in the robot's reference frame; and obtaining the second instruction of the execution module based on the finger joint mapping information.

[0011] In this embodiment, when performing finger mapping, the robotic arm position mapping information, robotic arm orientation mapping information, and the finger joint information from the previous moment are used as input to obtain finger joint mapping information in the reference frame, and then a second command is obtained. The finger mapping is associated with the overall motion state of the arm, so that the dexterous hand's movements can adapt to changes in the arm's posture, realizing coordinated control between the arm and the dexterous hand. At the same time, the finger state from the previous moment is introduced as a continuous reference to avoid abrupt changes in finger joint commands, making the dexterous hand's grasping and releasing actions smoother and more natural, and improving the success rate of fine operations.

[0012] In a possible implementation, the step of smoothing the initial instruction of the execution module at the current moment to obtain the target execution instruction includes: obtaining the weighted value of the initial instruction at the current moment and the smoothed instruction weighted value of the instruction at the previous moment based on the initial instruction of the execution module at the previous moment, a preset weight coefficient, and the initial instruction at the current moment; and obtaining the execution instruction based on the sum of the initial instruction weighted value and the smoothed instruction weighted value.

[0013] In this embodiment, when smoothing the initial command, a weighted value and a smoothing command weighted value are obtained based on the initial command from the previous moment, a preset weighting coefficient, and the initial command from the current moment, respectively. The execution command is then obtained based on the sum of the two. The smoothing process is achieved using an exponentially weighted moving average method, which has low computational complexity and high real-time performance, making it suitable for the stringent low-latency requirements of teleoperation systems. By adjusting the weighting coefficient, a flexible trade-off can be made between response speed and smoothness, adapting to different sensor noise levels and task requirements. It effectively suppresses high-frequency tracking noise common in lightweight sensing devices, ensuring the quality of commands received by the robot from the source, and achieving a stable control experience without the need for expensive anti-interference hardware.

[0014] In a possible implementation, before smoothing the initial instruction of the execution module to obtain the execution instruction, the method further includes: obtaining deviation information of the instruction at the current moment based on the finger joint information at the current moment, the finger joint target information at the current moment, and the reciprocal value of the finger joint target information at the current moment; correcting the initial instruction at the current moment based on the deviation information to obtain a corrected initial instruction; and using the corrected initial instruction to generate the execution instruction.

[0015] Before smoothing the initial command, this embodiment first obtains deviation information based on the current finger joint information, the finger joint target information, and the reciprocal of the target information. Then, the deviation is used to correct the current initial command, and the corrected initial command is used to generate the execution command. The processing order of correction before smoothing can suppress noise without masking systematic tracking errors, effectively solving the problem that simple smoothing may cause the command to deviate from the target, and improving the accuracy of finger control. Through the deviation feedback mechanism, the dexterous hand can more accurately reproduce the operator's finger movements, which is especially suitable for tasks such as grasping and pinching that require fine force control, ensuring the faithful transmission of the operation intention.

[0016] In a possible implementation, the method further includes: receiving a takeover switching instruction, the takeover instruction instructing the robot to switch from an automatically executing action state to an operator-controlled state; generating a takeover instruction based on the takeover switching instruction, the execution instructions prior to the takeover time, and the operation information at the takeover time, the takeover instruction being used to instruct the robot to perform an action corresponding to the operator based on the takeover instruction.

[0017] In response to a received takeover switching command, this embodiment generates a takeover command based on the execution command prior to the takeover moment and the operation information at the takeover moment. This command instructs the robot to execute the target action. Instead of directly using the operator's absolute posture, the robot's current command is used as the starting reference for manual takeover. This ensures that the robot will not experience command jumps or shocks due to differences in posture between the operator and the robot during the takeover moment, achieving a smooth takeover without sudden jumps. This mechanism is applicable to both emergency manual recovery when strategy execution fails and regular switching scenarios where the operator actively intervenes, significantly improving the system's human-machine collaboration safety and user experience.

[0018] In a possible implementation, generating the takeover instruction at the takeover time based on the takeover switching instruction, the execution information before the takeover time, and the operation information at the takeover time includes: generating an initial takeover instruction at the takeover time based on the takeover switching instruction, the operation information at the takeover time, and a preset coordination coefficient; and smoothing the initial takeover instruction at the takeover time and the execution instructions before the takeover time to obtain the takeover instruction at the takeover time.

[0019] In this embodiment, when generating a takeover command, an initial takeover command is generated based on the takeover command, the operation information at the takeover time, and a preset coordination coefficient. Then, the initial takeover command and the execution commands before the takeover time are smoothed to obtain the final takeover command. The coordination coefficient allows the operator or system to flexibly adjust the sensitivity of the operation after takeover according to the task requirements. The coordination coefficient can be reduced to obtain more precise control, and the coordination coefficient can be increased to obtain a faster response in large-scale movements. Smoothing is introduced again during the takeover process to ensure that the entire transition from automatic control to manual control is smooth and shock-free. At the same time, the coordination coefficient can also adapt to the habits of different operators, further improving the adaptability and comfort of the manual takeover process.

[0020] Secondly, embodiments of this application also provide a robot control information processing device, applied to an electronic device, such as a dual-arm dexterous robot. The device includes: a motion information acquisition module, used to acquire motion information of a user detected by at least one sensor in a wearing module; a wearing information module, used to determine wearing information based on the motion information, the wearing information including the user's arm orientation information, arm position information, and finger joint information; an initial instruction module, used to map the wearing information to the reference frame of the robot's execution module to obtain an initial instruction at the current moment; a target instruction acquisition module, used to smooth the initial instruction at the current moment to obtain a target execution instruction, the target execution instruction being used to instruct the robot to perform a corresponding action according to the user's motion information, the target execution instruction being obtained by correcting the initial instruction after adjusting the historical execution instruction according to the trend of change over time; after generating the target execution instruction, the robot's control module controls the robot to execute the target execution instruction.

[0021] Thirdly, embodiments of this application provide a robot control information processing device, including a memory and a processor, wherein the memory is used to store data or information required by the processor to process data, and the processor is used to execute the method provided in any embodiment of this application.

[0022] Fourthly, embodiments of this application provide a non-transitory computer-readable storage medium storing computer instructions, wherein the computer instructions are used to cause the computer to perform steps performed by the apparatus provided in any embodiment of this application.

[0023] Fifthly, embodiments of this application provide a computer program product, which includes a computer program that, when executed, implements the method as described in the first aspect. Attached Figure Description

[0024] To more clearly illustrate the technical solutions in the embodiments of this application, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0025] Figure 1 This application provides a schematic diagram of the main structure of a robot. Figure 2 This is a schematic diagram of a method process provided in an embodiment of this application; Figure 3 This is a schematic diagram of another method flow according to an embodiment of this application; Figure 4 This is a schematic diagram illustrating an example of a robot control information processing method according to this application; Figure 5 This is a schematic diagram of a robot control information processing device according to an embodiment of this application. Detailed Implementation

[0026] To make the technical problems, technical solutions, and beneficial effects to be solved by this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and are not intended to limit the scope of this application.

[0027] A dual-arm dexterous robot, which may also be simply referred to as a robot in this embodiment, has at least one robotic arm, and the robotic arm is equipped with an actuator. This actuator can be a multi-finger gripper or a dexterous hand. The dexterous hand has multiple independently actuable fingers. The multi-finger gripper or dexterous hand can simulate the grasping and manipulation capabilities of a human hand, and can be used to replace or assist humans in working in complex environments, especially in scenarios requiring fine motor coordination, such as assembling precision parts, laboratory operations, and household chores. The robotic arm can simulate the movement functions of a human arm; for example, the robotic arm can have a joint structure similar to that of a human arm. The robotic arm can also be referred to as a mechanical arm.

[0028] Among the possible implementations, refer to Figure 1As shown, the dual-arm dexterous robot includes a robotic arm system 11, a dexterous hand 12, a sensing system 13, and a control and communication system 14. The robotic arm system 11 is the main body for performing tasks and typically includes at least one high-degree-of-freedom robotic arm. Each robotic arm is designed to mimic the human upper limb, incorporating structures that simulate the arm, shoulder joint, elbow joint, and wrist joint. To extend the working range, in some cases, the robotic arm is integrated into a mobile platform, such as a chassis, which may be equipped with one or more wheels to assist in the robotic arm's movement. The dexterous hand 12 is located at the end of the robotic arm system 11 and is used to simulate the human hand for fine manipulation. The dexterous hand 12 can be configured with multiple fingers, each with 12 to 15 degrees of freedom, enabling human-like movements such as pinching and twisting. The sensing system 13 may include at least one sensor, such as a vision sensor, a pressure sensor, or a motion sensor. The sensors of the sensing system 13 can be used to detect the operator's arm movements. The control and communication system 14 may include a controller, a memory, etc., for sending instructions to the dexterous hand 12, the robotic arm system 11, or the sensing system 13, such as sending instructions to the robotic arm system 11 to grasp an object and place it in a designated location. The control and communication system 14 can also be used to transmit instructions or data between the dexterous hand 12, the robotic arm system 11, or the sensing system 13.

[0029] In other possible implementations, the aforementioned sensing system 13 may include a control and communication system communication and timing processing module, a signal smoothing and control point correction module, a coordinate semantic transformation module, a dual-arm and dexterous hand mapping module, a motion sequence connection module, a compliant safety control module, a status monitoring module, and a manual intervention and recording module, used to process the information received by the sensing system 13. The commands received by the robotic arm system 11 and the dexterous hand 12 can come from the operator, or from a preset strategy model or other task modules.

[0030] The operation of a dual-arm dexterous robot can include stages such as teleoperation, motion mapping, compliant execution, and human-in-the-loop takeover. Through these stages, the dual-arm dexterous robot can safely and stably mimic human fine motor skills and can be smoothly taken over by an operator when necessary. Specifically, the teleoperation stage transmits the operator's intentions to the remote robotic arm system via master-slave devices. The motion mapping stage establishes the kinematic correspondence between the human arm and the robotic arm system, translating captured human joint angles into robot joint commands. The compliant execution stage allows the robotic arm system to actively control contact forces when interacting with the environment or objects, preventing damage due to collisions. When the robotic arm system's autonomous strategy encounters a situation it cannot handle, the human operator can intervene and take over control, entering the human-in-the-loop takeover stage.

[0031] In the possible implementations, Figure 1Based on the structure shown, the dual-arm dexterous robot can also be equipped with a lightweight pose sensing device, namely... Figure 1 The lightweight device of the sensing system 13 shown is a lightweight form factor device. A lightweight pose sensing device can refer to a small, lightweight, low-power, and easily deployed device used to measure and track the position (X, Y, Z coordinates) and attitude (pitch, yaw, roll angles) of an object in three-dimensional space, providing information in six degrees of freedom. Lightweight pose sensing devices facilitate rapid deployment (i.e., rapid connection with other robot components) and natural operation. However, the operator's original signals may be affected by communication jitter, tracking noise, control point offset, and coordinate semantic differences. Directly mapping the original signals to the robot may result in end-effector jitter, spurious translation, or deviations in motion direction.

[0032] In view of this, embodiments of this application provide a method for processing control information of a robot, applied to a dual-arm dexterous robot. The control information can be used to control all information of the robot during teleoperation, including motion data from the operator and execution instructions sent to the robot. Further, the control information may include wearing information and / or execution information. The dual-arm dexterous robot includes a wearing module and an execution module. The wearing module is used to receive wearing information, generate execution information based on the wearing information, and send the execution information to the execution module. The execution module is used to perform actions based on the execution information. In possible implementations, the wearing module may include… Figure 1 The sensing system 13 shown (the wearable module may include, in addition to sensors, mechanical structures such as a fixing strap for wearing) and the execution module may include Figure 1 The robotic arm system 11 and / or dexterous hand 12 are shown. The method includes, as... Figure 2 Steps S21 to S24 are shown.

[0033] Step S21: Obtain user action information detected by at least one sensor in the wearable module.

[0034] In possible implementations, motion information may include information about the actions performed by the user (i.e., the operator) after wearing the device. For example, motion information may be information generated when the user moves objects, assembles products, or makes items. In possible implementations, motion information may include: the operator's limb joint angles and angular velocities; the movement trajectory of the operator's hands or arms; the operator's torso posture and head orientation; and information about the force or torque applied by the operator's hands.

[0035] Step S22: Based on the wearer's motion information detected by the sensors of the wearing module, wear information is generated; the wear information includes the user's orientation information, position information, and finger joint information.

[0036] In possible implementations, the information to be worn may include motion data collected by a sensing system worn on the operator, including positional information, orientation information, finger bending degree, and finger joint information of various body parts.

[0037] The operator wears a sensing system, whose various sensors detect the operator's body movements in real time. The system aggregates and formats this raw sensor data to generate structured wearing information. For example, the sensing system may include data gloves, an IMU (Inertial Measurement Unit) tracker, etc. Generating wearing information based on the wearer's movement information detected by the sensors of the wearing module can include: generating a digital signal from the analog signals detected by the sensors to obtain wearing information. Alternatively, generating wearing information based on the wearer's movement information detected by the sensors of the wearing module can include: generating a digital signal from the analog signals detected by the sensors, and then standardizing the digital signal to obtain standardized wearing information.

[0038] In this embodiment, orientation information can be used to describe the direction in which a part of the operator's body points in three-dimensional space, i.e., its rotational posture. Further, orientation information may include arm orientation information, palm orientation information, or wrist orientation information. Position information can be used to describe the spatial coordinates of a part of the operator's body in three-dimensional space. Finger joint information can be used to describe the degree of bending or angle state of each joint of the operator's fingers. Finger joint information can be a multi-dimensional vector containing the position and / or angle values ​​of all tracked finger joints.

[0039] Step S23: Map the wearing information to the reference frame of the robot's execution module to obtain the initial command at the current moment.

[0040] In possible implementations, the reference frame of the execution module can refer to the reference coordinate system of the robotic arm or the overall reference coordinate system of the robot, in order to unify the operator space and the robot space and make the mapping accurate.

[0041] In a possible implementation, the operator's movements are defined in a human coordinate system, while the robot has its own coordinate system and kinematic structure. Step S22 converts the wear information corresponding to the operator's movements into instructions that the robot can understand and execute. Further, mapping the wear information to the execution module's reference frame to obtain the execution module's current instruction may include mapping the wear information to the execution module's reference frame to obtain the execution module's theoretical target position in the reference frame, then determining the displacement required for the execution module to move to the theoretical target position based on the theoretical target position and the execution module's current position, and finally determining the execution module's current instruction based on the required displacement.

[0042] The initial instruction of the execution module at the current moment can refer to the instruction that, theoretically, the robot's control and communication system or other control modules should send to the execution module. Furthermore, the instruction of the execution module at the current moment can refer to the instruction executed by the robot to perform an action corresponding to the operator's information.

[0043] Step S24: Smooth the initial instruction at the current moment to obtain the target execution instruction. The target execution instruction is used to instruct the robot to perform the corresponding action according to the user's action information. The target execution instruction is obtained by correcting the initial instruction with the trend of historical execution instructions changing over time. The historical execution instructions are the instructions executed by the robot at times before the current moment.

[0044] Target execution instructions refer to the final control instructions issued to the robot's execution module after processing, used to drive the robot's execution module to complete corresponding actions. Target execution instructions include at least one of the following forms: the target pose of the robot's end effector, the amount of change in joint angles, the magnitude and direction of the grasping force, and a global motion trajectory including obstacle avoidance paths. Target execution instructions are converted into drive signals by the execution module configured on the robot to control the robot to complete the set task.

[0045] Due to sensor noise, communication jitter, and other factors, the current command from the execution module may contain high-frequency fluctuations, which can cause end effector jitter if sent directly to the robot. Therefore, in step S23, the initial command from the execution module at the current moment is smoothed to make the command changes smoother and more continuous, ultimately obtaining a stable and executable target command. Smoothing can refer to low-pass filtering or weighted averaging of the time-domain signal to suppress high-frequency components and retain low-frequency motion trends. When the robot is equipped with lightweight sensing devices, although convenient, the signal quality is poor. Direct execution without processing can cause high-frequency jitter at the end effector, overheating of the execution module, or even damage. This embodiment achieves a trade-off between response speed and motion smoothness through smoothing, reducing jitter and preventing damage to the execution module.

[0046] The method provided in this application can acquire orientation, position, and finger joint information based on the sensors of the wearable module, supporting the access of lightweight pose sensing devices. This allows the system to break free from the dependence on fixed motion capture sites and complex donning procedures required by traditional teleoperation, significantly reducing hardware costs and deployment complexity. Operators can control the robot naturally and without constraints without professional training, improving the ease of use and comfort of teleoperation. Before obtaining execution information, the current instructions of the execution module are smoothed, which can effectively suppress the inherent measurement noise of lightweight sensors, communication jitter, and high-frequency fluctuations caused by wireless transmission anomalies. This makes the instructions issued to the robot smoother and more continuous, and the robot end effector will not experience visible micro-shaking or stuttering when performing tasks. The execution action is more stable, extending the life of the robot joints and transmission mechanisms, and also improving operational safety.

[0047] In one implementation, the step of mapping the wearing information to the reference frame of the execution module to obtain the initial instruction of the execution module at the current moment is referenced. Figure 3 As shown, it includes the following steps S31 to S33.

[0048] Step S31: Based on the reference frame of the execution module, perform continuous constraint motion mapping on the orientation information and / or position information to obtain the first instruction of the execution module. The first instruction represents the target orientation that the robot needs to reach based on the orientation information and / or the target position that it needs to reach based on the position information.

[0049] Step S32: Based on the reference frame of the execution module, perform finger mapping on the finger joint information to obtain the second instruction of the execution module. The second instruction represents the target finger joint action performed by the robot based on the finger joint information.

[0050] Step S33: Use the first instruction and the second instruction as the initial instruction.

[0051] In this application, the terms "first instruction" and "second instruction" are used only to distinguish different instructions or signals and do not constitute any limitation on the number of instructions, execution order, or priority. The first instruction and second instruction may refer to the same or different types of control signals, data packets, or program code, and their specific meaning depends on their technical context. Unless otherwise expressly stated, there is no logical dependency or temporal order between the first instruction and the second instruction.

[0052] In this embodiment, continuous constraint motion mapping refers to the smooth mapping of orientation and position information between the previous and current moments. Through continuous constraint mapping, when mapping the operator's arm movement to the robot, the continuity between the previous and current moments is considered, avoiding abrupt changes in joint commands. Specifically, in continuous constraint motion mapping, the joint angles of the previous moment can be used as constraints. These joint angles include arm joint angles and / or finger joint angles. If, based on the initial command, the joint angles at the current moment have changed significantly compared to the joint angles at the previous moment, then the joint angles at the current moment need to be smoothed out.

[0053] The first instruction can refer to the joint movement instructions of the robotic arm (mechanical arm) obtained by mapping the orientation and position information of the operator's arm. Finger mapping is used to convert the bending state of the operator's finger joints into drive instructions for each joint of the dexterous hand, and can include the correspondence between different hand structures. The second instruction can refer to the dexterous hand drive instructions obtained by mapping the operator's finger information.

[0054] Since the robot consists of two parts—a robotic arm (performing large-range movements) and a dexterous hand (performing precise grasping)—the mapping needs to be processed separately. During the conversion from human movement to robot commands, it's also necessary to simultaneously handle differences in robot structure, joint constraints, and the dexterous hand's driving relationships. If solved independently frame by frame or lacking continuous constraints, tracking fluctuations will be further amplified into joint command jumps. In this embodiment, the robotic arm part obtains the first command through continuous constraint motion mapping, and the finger part obtains the second command through finger mapping. Together, these two parts constitute the complete command for the current moment. Through smooth mapping, this adapts to the differences in mapping logic between the arms and the dexterous hand.

[0055] In one embodiment, the step of performing continuous constraint motion mapping on the orientation information and / or position information based on the reference frame of the execution module to obtain a first instruction from the execution module includes: determining the position difference between the position information at the previous moment and the position information at the current moment in the reference frame; determining manipulator position information mapping information based on the position difference, abstract positional relationship information between the manipulator and the execution module, motion scale between the manipulator and the execution module, and reference position information of the execution module, wherein the manipulator position mapping information is the mapping information of the robot's arm position in the robot's reference frame; determining manipulator orientation information mapping information based on the reference orientation information of the execution module, the inverse matrix of the orientation information at the previous moment, and the orientation information at the current moment, wherein the manipulator orientation information mapping information is the mapping information of the robot's manipulator in the robot's reference frame; and obtaining the first instruction based on the manipulator position information mapping information and the manipulator orientation information mapping information.

[0056] In possible implementations, the arm position information mapping information may include the result of mapping the translational changes of the operator's arm to the positional changes of the robot's end effector, including scale information and coordinate transformation information. The arm orientation information mapping information includes the result of mapping the rotational changes of the operator's arm to the orientation changes of the robot's end effector.

[0057] The position mapping is based on the operator's position change, and after scaling and coordinate transformation, it is superimposed on the robot's reference position; the orientation mapping is based on the operator's orientation change, and then superimposed on the robot's reference orientation, ensuring that the operator's initial orientation does not affect the semantic direction of the translational motion, thus avoiding the problem of motion direction deviation.

[0058] In one embodiment, the step of mapping the finger joint information based on the reference frame of the execution module to obtain the second instruction of the execution module includes: obtaining finger joint mapping information in the reference frame based on the robot arm position mapping information, the robot arm orientation mapping information, and the finger joint information at the previous moment, wherein the finger joint mapping information represents the finger joint movement of the robot's fingers in the robot's reference frame; and obtaining the second instruction of the execution module based on the finger joint mapping information.

[0059] Finger joint mapping information can include target values ​​for each joint of the robot's dexterous hand, converted from raw readings (such as bending angles) of the operator's finger sensors. In a possible implementation, the operator information can first be stabilized, then mapped to obtain the current command of the execution module, and finally smoothed to make the command more compliant, thus obtaining the execution information.

[0060] In a possible example, the system maps the operator's position and orientation information to the robot's reference frame to obtain the first command, avoiding changes in translation semantics due to the initial orientation. The relationship between the first command and the position and orientation information can be abstractly represented as: p t= p r o+s×A×(p w t p w o), R t=R r o×(R w o) -1 ×R w t.

[0061] Where p t and R t is the first instruction of the execution module at time t, p t represents the position information in the first instruction at time t, R t represents the orientation information in the first instruction at time t. p w R w For operator information, p w For the location information in the operator information, R w This refers to the orientation information within the operator's information. r o、R r o represents the reference state of the execution module, corresponding to the historical execution instructions in the aforementioned embodiments. A represents an abstract coordinate relationship, including at least one preset matrix. s represents the motion scale, which may include at least one preset scalar. By analyzing the position information p... w t and orientation information R w The t method uses independent relationships to decouple the mapping between position and orientation, thus avoiding mutual interference.

[0062] The dual-arm target obtains joint commands through robot motion mapping with continuous constraints, and the finger joint information is used to obtain the second command: q through hand mapping. t=IK(p t, R t; q t-1 ), m t=G(ht).

[0063] Where IK represents the pre-defined mathematical mapping of the abstract robotic arm, and G represents the pre-defined mathematical mapping of the abstract dexterous hand. The specific solver, limit switches, regularization terms, finger channels, and calibration parameters are determined based on the robot platform. t represents the second command corresponding to the robot arm angle at time t, m t represents the second command of the finger joint at time t, and ht represents the finger joint information at time t.

[0064] In one implementation, the step of smoothing the initial instruction of the execution module at the current moment to obtain the target execution instruction includes: obtaining the weighted value of the initial instruction at the current moment and the smoothed instruction weighted value of the instruction at the previous moment based on the initial instruction of the execution module at the previous moment, a preset weight coefficient, and the initial instruction at the current moment; and obtaining the target execution instruction based on the sum of the initial instruction weighted value and the smoothed instruction weighted value.

[0065] In a possible implementation, the preset weighting coefficient can be a coefficient used in the smoothing process, with a value ranging from 0 to 1, used to control the proportion of historical and current instructions in the smoothing result. For example, a smaller preset weighting coefficient results in a stronger smoothing effect but a slower response; a larger preset weighting coefficient results in a faster response but a weaker smoothing effect. The initial instruction from the previous moment can refer to the smoothed instruction that was finally executed in the previous control cycle, i.e., the execution information from the previous moment, serving as a reference for the current smoothing. Alternatively, the instruction from the previous moment can also refer to the initial instruction before the smoothing process from the previous moment.

[0066] The method provided in this application embodiment can suppress end jitter caused by high-frequency noise through effective time-domain smoothing processing. At the same time, it has low computational load and good real-time performance, making it suitable for lightweight teleoperation systems.

[0067] When a new sequence of actions replaces the currently executing sequence, the system smoothly combines the execution information from the previous moment and the execution information from the current moment during the transition using the following formula: u t = (1 w t )×u_old,t+ w t ×u_new,t.

[0068] Among them, w t The weights, which vary with the transition process, can be preset and can be calculated using a preset weight function with time t as the independent variable. t The length of the specific transition process and the curve of the weighting function are not limited. The smoothed command is also subject to robot safety constraints. t The instructions to be executed at time t include instructions regarding orientation and position. u_old,t represents the instructions to be executed at the time preceding the takeover time, and u_new,t represents the operation information at the takeover time.

[0069] In one embodiment, before smoothing the initial instruction at the current moment to obtain the target execution instruction, the method further includes: obtaining deviation information of the instruction at the current moment based on the finger joint information at the current moment, the finger joint target information at the current moment, and the reciprocal value of the finger joint target information at the current moment; correcting the initial instruction at the current moment based on the deviation information to obtain a corrected initial instruction; and using the corrected initial instruction to generate the execution instruction.

[0070] The robot employs a compliant execution method to absorb prediction errors and contact deviations, obtaining deviation information. The relationship can be simplified to: τ = Kp(q) t qo) K_d×Dqt.

[0071] This formula only indicates that the target error and motion damping jointly determine the execution action. Kp and K_d are preset empirical values, and qo is the target value of the robotic arm at time t. The empirical values ​​and target values ​​can be pre-configured on the robot platform based on the specific control mode. τ represents the deviation information between the second command and the target value, Dqt is the derivative of qt, and q t represents the second command corresponding to the robot arm angle at time t.

[0072] In this embodiment, the finger joint target information at the current moment can refer to the expected finger joint angle target value at the current moment, used to calculate the deviation between the current command and the target. The reciprocal value of the finger joint target information at the current moment can be used to normalize or scale the deviation, thereby correcting the deviation of the command. The deviation information can be the difference between the current actual finger joint state and the target state, used to reversely correct the current command.

[0073] The method provided in this application first calculates the deviation between the current state and the target state of the finger joint, uses this deviation to correct the original command at the current moment, obtains the corrected command, and then performs exponential smoothing on the corrected command. This method of first correcting and then smoothing can reduce tracking errors, prevent smoothing from masking systematic control deviations, and improve the accuracy of finger mapping.

[0074] In one embodiment, the method for processing robot control information further includes: receiving a takeover switching instruction, the takeover instruction instructing the robot to switch from an automatic execution state to an operator takeover control state; generating a takeover instruction based on the takeover switching instruction, the execution instructions prior to the takeover time, and the operation information at the takeover time, the takeover instruction being used to instruct the robot to perform an action corresponding to the operator based on the takeover instruction.

[0075] In this embodiment, the takeover switching command can refer to a control command issued by the operator or the status monitoring module, requesting a switch from automatic / policy control to manual control. The takeover moment can refer to the time when the takeover command takes effect; the system needs to record the robot's state and the operator's state at that moment. The target execution command before the takeover moment can refer to the last safety command the robot is executing at the moment of takeover, serving as the starting reference point for manual takeover. The operational information at the takeover moment can refer to the operator's current posture and position state at the moment of takeover, used to calculate subsequent relative changes.

[0076] During robot operation, inconsistencies arise in the connection between old and new action sequences, and the operator's absolute posture differs from the robot's current posture when switching from strategy control to manual takeover. Rigid execution in contact tasks can also amplify prediction errors. In this embodiment, when strategy execution fails or the operator intervenes, the system can determine that a switch from automatic control to manual control is necessary. Instead of directly using the operator's absolute posture to generate commands, the system records the robot's current command at the moment of takeover as a baseline. Based on this, the system superimposes the operator's subsequent relative motion changes, ensuring that the robot does not experience command jumps due to the difference in posture between the operator and the robot at the moment of takeover, thus achieving seamless manual recovery.

[0077] In one implementation, generating a takeover instruction at the takeover time based on the takeover switching instruction, the target execution instruction before the takeover time, and the operation information at the takeover time includes: generating an initial takeover instruction at the takeover time based on the takeover switching instruction, the operation information at the takeover time, and a preset coordination coefficient; and smoothing the initial takeover instruction at the takeover time and the execution instructions before the takeover time to obtain the takeover instruction at the takeover time.

[0078] In this embodiment, a preset coordination coefficient can be used to adjust the amplitude ratio of the operator's relative motion change mapped to the robot's command change, thereby achieving sensitivity control or gain control. The coordination coefficient allows the operator to adjust the sensitivity of control after taking over, adapting to the accuracy requirements of different tasks.

[0079] When the status monitoring or operator determines that takeover is necessary, the robot's system records the takeover moment, and the robot executes the takeover command at time t0. _t0, and the operation information x_t0 at takeover time t0. After takeover, the operator's absolute command is not used directly, but its relative change is superimposed on the robot's current command: u t=u _t0+β×[H(xt) H(x_t0)], where H is the abstract mapping from operator state to robot command, β is the coordination coefficient, and u t represents the execution instructions generated based on operator information starting at the takeover time, and xt represents the operation information at time t. The takeover start time satisfies: u _t0=u _t0.

[0080] In this embodiment, the command will not change due to different absolute postures of the operator at the moment of takeover. The system can perform a consistency check before takeover and record the robot's observations, task information, and actual executed commands during takeover to form a recovery trajectory.

[0081] In one implementation, generating wearing information based on the wearer's motion information detected by the sensors of the wearing module may include stabilizing the motion information and then correcting the deviation of the stabilized motion information to obtain the wearing information.

[0082] Motion information can include the operator's hand pose, finger status, and necessary task input detected by sensors. The communication and timing processing module synchronizes the signal and protects against anomalies, while the smoothing module reduces high-frequency fluctuations. Its basic relationship can be abstracted as: x' t =(1 α)×x' t-1 +α×x t Where x t Display operation information, x' t This represents the stabilized operational information corresponding to the current time t, where α is the smoothing coefficient. α can also be determined using other equivalent filtering, prediction, or constraint methods.

[0083] When the tracking control point is inconsistent with the operator's actual rotation reference, the system uses an abstract offset r to correct the control point: p_w = p_c + R_c × r. Here, p_c and R_c represent the robot's position and orientation before correction, and p_w represents the robot's reference position after correction. The offset r (an empirical value) can be obtained through calibration or other estimation methods.

[0084] In one example of this application, the method for processing robot control information includes, as follows: Figure 4 The steps are shown.

[0085] Step S41: Configure the operator sensing devices, robot execution platform, and necessary communication and security relationships.

[0086] Step S42: Receive the operator's original information and perform timing, exception and noise processing to obtain the operator information.

[0087] Step S43: Correct the control reference and coordinate semantics, and map the operator information to the initial instructions of the execution module at the current moment.

[0088] Step S44: Smoothly connect the instructions of the execution module in the previous moment with the instructions in the current moment and process them through compliant safety control to obtain the target execution instructions.

[0089] Step S45: Monitor the robot's task execution status and trigger manual takeover in case of abnormality or failure.

[0090] Step S46: Using the robot's current target execution instruction as a reference, apply relative changes based on operator information.

[0091] Step S47: Record the recovery process and output it to the external data or policy update module.

[0092] In possible implementations, operator perception can be achieved using a handle, hand tracker, data glove, or device with equivalent output. Signal stabilization can employ various filtering, prediction, interpolation, and anomaly protection methods. Control point and coordinate semantic correction can be achieved through offline calibration, online estimation, or based on known geometric relationships. The mapping of the robotic arm and dexterous hand can utilize different kinematic, optimization, or learning methods.

[0093] Compliant execution can employ impedance, admittance, or other contact-compliant control methods. Operator takeover can be triggered manually, by a condition detector, or both.

[0094] The method provided in this application can reduce the dependence on external motion capture, exoskeletons and master-slave devices, making it easier to deploy in a lightweight manner. It can also reduce the impact of signal noise, transmission anomalies and control point deviations on robot commands. It improves the consistency of commands between the two arms and the dexterous hand through coordinate semantic correction and continuous mapping. It reduces command switching and contact shock through smooth and compliant execution of action sequences. It avoids abrupt changes in control switching through relative takeover of the current command anchor. It uniformly records the manual recovery process, which is convenient for subsequent evaluation, data analysis or strategy updates.

[0095] This application also provides a robot control information processing device, referring to... Figure 5 As shown, the device includes: a motion information acquisition module for acquiring motion information of a user detected by at least one sensor in the wearing module; a wearing information module for determining wearing information based on the motion information, the wearing information including the user's arm orientation information, arm position information, and finger joint information; an initial instruction module for mapping the wearing information to the reference frame of the robot's execution module to obtain an initial instruction at the current moment; a target instruction acquisition module for smoothing the initial instruction at the current moment to obtain a target execution instruction, the target execution instruction being used to instruct the robot to perform a corresponding action according to the user's motion information, the target execution instruction being obtained by correcting the initial instruction based on the trend of historical execution instructions changing over time; after generating the target execution instruction, the robot's control module controls the robot to execute the target execution instruction.

[0096] This application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, can implement the steps in the above-described method embodiments.

[0097] This application provides a computer program product, including a computer program, which, when run, causes the steps in the above-described method embodiments to be executed.

[0098] The embodiments of the present invention described above are combinations of elements and features of the invention. Unless otherwise stated, elements or features may be considered optional. Individual elements or features may be practiced without combination with other elements or features. Furthermore, embodiments of the invention may be constructed by combining some elements and / or features. The order of operations described in the embodiments of the invention may be rearranged. Some constructions of any embodiment may be included in another embodiment and may be replaced by corresponding constructions of another embodiment. It will be apparent to those skilled in the art that claims that are not explicitly referenced in the appended claims may be combined to form embodiments of the invention, or may be included as new claims in modifications made after the submission of this invention.

[0099] In firmware or software configuration, the embodiments of the present invention can be implemented in the form of modules, processes, functions, etc. Software code can be stored in a memory unit and executed by a processor. The memory unit is located inside or outside the processor and can send data to and receive data from the processor via various known means.

[0100] The various aspects of the systems and methods described herein can be implemented as functions programmable into any type of circuit, including programmable logic devices (PLDs) such as field-programmable gate arrays (FPGAs), programmable array logic (PAL) devices, electronically programmable logic and memory devices, standard cell-based devices, and application-specific integrated circuits (ASICs). Other possibilities for implementing these aspects of the system include: microcontrollers with memory, such as electronically erasable programmable read-only memory (EEPROM), embedded microprocessors, firmware, software, etc. Furthermore, these aspects of the system can be embodied in microprocessors with software-based circuit emulation, discrete logic (sequential and combinational), custom devices, fuzzy (neural) logic, quantum devices, and any combination of the various device types described above. Of course, underlying device technologies can be provided in various component types, such as metal-oxide-semiconductor field-effect transistor (MOSFET) technologies such as complementary metal-oxide-semiconductor (CMOS), bipolar technologies such as emitter-coupled logic (ECL), polymer technologies (e.g., silicon conjugated polymers and metal conjugated polymer metal structures), hybrid analog and digital, etc.

[0101] The various functions or processes disclosed herein can be described, based on their behavior, register transfers, logic components, transistors, geometric layouts, and / or other characteristics, as data and / or instructions embodied in various computer-readable media. Computer-readable media that may contain such formatted data and / or instructions include, but are not limited to, various forms of non-volatile storage media (e.g., optical, magnetic, or semiconductor storage media) and carrier waves, which can be used to transmit such formatted data and / or instructions via wireless, optical, or wired signal media, or any combination thereof. Such data and / or instructions can be processed by a processing entity (e.g., one or more processors) upon receipt by any of various circuits (e.g., a computer).

[0102] The above description of the illustrated embodiments of the systems and methods is not intended to be exhaustive or to limit the systems and methods to the precise forms disclosed. While specific embodiments and examples of system components and methods have been described herein for illustrative purposes, those skilled in the art will understand that various equivalent modifications can be made within the scope of the systems, components, and methods. The teachings of the systems and methods provided herein can be applied to other processing systems and methods, and are not limited to those described above.

[0103] Those skilled in the art will understand that various changes and / or modifications can be made to the invention illustrated in particular embodiments without departing from the spirit or scope of the broad description of the invention. Therefore, these embodiments are to be considered illustrative rather than restrictive in all respects. Furthermore, the invention includes any combination of features described with respect to different embodiments (including those in the abstract section), even if such feature or combination of features is not expressly specified in the claims or the detailed description of these embodiments.

[0104] Generally, the terminology used in the following claims should not be construed as limiting the systems and methods to the specific embodiments disclosed in the specification and claims, but should be interpreted as encompassing all processing systems operating under the claims. Therefore, the systems and methods are not limited by this disclosure, but their scope is determined entirely by the claims.

[0105] Unless the context explicitly requires otherwise, throughout the specification and claims, the words “comprising,” “including,” and “including” should be interpreted in a encompassing sense, not in an exclusive or exhaustive sense; that is, in the sense of “including but not limited to.” The use of singular or plural words also includes both singular and plural, respectively. Furthermore, “this article,” “in the following,” “above,” “below,” and words with similar meanings refer to the application as a whole, and not to any particular part of the application. When the word “or” is used to refer to a list of two or more items, the word “or” includes all of the following interpretations: any item in the list, all items in the list, and any combination of items in the list.

[0106] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of this application, "multiple" means two or more, unless otherwise explicitly specified.

[0107] The above are merely preferred embodiments of this application and are not intended to limit this application. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this application should be included within the protection scope of this application.

Claims

1. A method for processing control information of a robot, applied to electronic devices, characterized in that, The method includes: Acquire user action information detected by at least one sensor in the wearable module; Wearing information is determined based on the motion information, including the user's arm orientation, arm position, and finger joint information; The wearing information is mapped to the reference frame of the robot's execution module to obtain the initial command at the current moment; The initial instruction at the current moment is smoothed to obtain the target execution instruction. The target execution instruction is used to instruct the robot to perform the corresponding action according to the user's action information. The target execution instruction is obtained by correcting the initial instruction with the trend of historical execution instructions changing over time. The historical execution instructions are the instructions executed by the robot at times before the current moment. Control the robot to execute the target execution command.

2. The method according to claim 1, characterized in that, The step of mapping the wearing information to the reference frame of the execution module to obtain the initial instruction of the execution module at the current moment includes: Based on the reference frame of the execution module, continuous constraint motion mapping is performed on the orientation information and / or position information to obtain the first instruction of the execution module. The first instruction represents the target orientation that the robot needs to reach based on the orientation information and / or the target position that it needs to reach based on the position information. Based on the reference frame of the execution module, the finger joint information is mapped to obtain the second instruction of the execution module. The second instruction represents the target finger joint action performed by the robot based on the finger joint information. The first instruction and the second instruction are used as the initial instruction.

3. The method according to claim 2, characterized in that, Based on the reference frame of the execution module, the orientation information and / or position information are subjected to continuous constraint motion mapping to obtain the first instruction of the execution module, including: Determine the position difference between the position information at the previous moment and the position information at the current moment in the reference frame; Based on the position difference, the abstract positional relationship information between the robotic arm and the execution module, the motion scale between the robotic arm and the execution module, and the reference position information of the execution module, the robotic arm position information mapping information is determined. The robotic arm position mapping information is the mapping information of the position of the robot's arm in the robot's reference frame. Based on the reference orientation information of the execution module, the inverse matrix of the orientation information at the previous moment, and the orientation information at the current moment, the orientation information mapping information of the robotic arm is determined. The orientation information mapping information of the robotic arm is the mapping information of the robotic arm of the robot in the robot reference frame. The first instruction is obtained based on the arm position information mapping information and the arm orientation information mapping information.

4. The method according to claim 3, characterized in that, The second instruction of the execution module is obtained by mapping the finger joint information based on the reference frame of the execution module, including: Based on the robotic arm position mapping information, robotic arm orientation mapping information, and the finger joint information of the previous moment, the finger joint mapping information in the reference frame is obtained. The finger joint mapping information represents the finger joint movement of the robot's fingers in the robot's reference frame. Based on the finger joint mapping information, the second instruction of the execution module is obtained.

5. The method according to claim 1, characterized in that, The step of smoothing the initial instruction at the current moment to obtain the target execution instruction includes: Based on the initial instruction of the execution module at the previous moment, the preset weight coefficient, and the initial instruction at the current moment, the weighted value of the initial instruction at the current moment and the smoothed instruction weighted value of the instruction at the previous moment are obtained. The target execution instruction is obtained based on the sum of the initial instruction weighting value and the smooth instruction weighting value.

6. The method according to any one of claims 1-5, characterized in that, Before smoothing the initial instruction at the current moment to obtain the target execution instruction, the method further includes: Based on the finger joint information at the current moment, the finger joint target information at the current moment, and the reciprocal value of the finger joint target information at the current moment, the deviation information of the instruction at the current moment is obtained; Based on the deviation information, the initial instruction at the current moment is corrected to obtain the corrected initial instruction; the corrected initial instruction is used to generate the execution instruction.

7. The method according to any one of claims 1-5, characterized in that, The method further includes: Receive a takeover switching command, the takeover command instructing the robot to switch from an automatic action execution state to an operator takeover control state; Based on the takeover switching instruction, the execution instruction before the takeover time, and the operation information at the takeover time, a takeover instruction is generated. The takeover instruction is used to instruct the robot to perform an action corresponding to the operator based on the takeover instruction.

8. The method according to claim 7, characterized in that, The process of generating a takeover instruction based on the takeover switching instruction, the execution information prior to the takeover time, and the operation information at the takeover time includes: Based on the takeover switching command, the operation information at the takeover time, and the preset coordination coefficient, an initial takeover command at the takeover time is generated; The initial takeover instruction at the takeover moment and the execution instructions prior to the takeover moment are smoothed to obtain the takeover instruction at the takeover moment.

9. A device for processing control information of a robot, characterized in that, It includes a processor and a memory, the memory being used to store a computer program, and the processor being used to execute the computer program to implement the method as described in any one of claims 1-8.

10. A non-transitory computer-readable storage medium storing computer instructions, wherein, The computer instructions are used to cause the computer to perform the method according to any one of claims 1-8.

Images