Data processing method, device and equipment applied to robot teleoperation and medium

Abstract

This invention provides a data processing method for robot teleoperation, applicable to the field of artificial intelligence. The method includes: responding to teleoperation data of the operator's current target action, controlling the robot's end effector to execute a target spatial action matching the current target action; during the execution of the target spatial action, acquiring global pose data and joint motion data of the target spatial action in a global coordinate system; generating a geometric consistency factor, a dynamic damping factor, and an active gating factor based on the global pose data and joint motion data; and generating a continuous multidimensional intent factor based on the nonlinear fusion of the geometric consistency factor, dynamic damping factor, and active gating factor, used to optimize the execution process of the target spatial action. This invention also provides a data processing device, apparatus, storage medium, and program product for robot teleoperation.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, specifically to the field of teleoperation technology, and more specifically to a data processing method, apparatus, equipment, medium, and product for robot teleoperation. Background Technology

[0002] Teleoperation technology is a key approach for humanoid robots to achieve complex manipulation and acquire imitation learning teaching data. Its core task is to accurately map the operator's motion intentions to the robot's execution end through a human-machine interface. However, in high-precision tasks such as precision assembly (e.g., sub-millimeter pin hole assembly) and dexterous manipulation, the sensing accuracy and sampling frequency of low-cost teleoperation systems are limited. Furthermore, the operator's inherent physiological tremors (usually concentrated at 8-12Hz) and unconscious corrective movements become bottlenecks restricting system performance. This results in the inability to accurately perceive the operator's true task intentions, leading to semantic blindness, inaccurate operation, and poor stability. Summary of the Invention

[0003] In view of at least one of the above-mentioned problems, embodiments of the present invention aim to provide a data processing method, apparatus, device, medium, and product for a humanoid robot or robotic arm teleoperation system that can perceive and quantify the operator's intention in real time. This can solve a series of problems caused by the inability to accurately perceive the operator's intention in existing teleoperation technologies. Specifically, it can infer the operator's true task intention (e.g., whether in a state of free exploration, precise alignment, or intention to withdraw) in real time and accurately from teleoperation signals mixed with physiological tremors and unconscious corrections, and quantify it continuously and normally, thereby overcoming the problems of "semantic blindness" and discontinuous control in traditional methods.

[0004] One aspect of the present invention provides a data processing method for robot teleoperation, comprising: controlling the robot's end effector to perform a target spatial action matching the current target action in response to teleoperation data of the operator's current target action; acquiring global pose data and joint motion data of the target spatial action in a global coordinate system during the execution of the target spatial action; generating a geometric consistency factor, a dynamic damping factor, and an active gating factor based on the global pose data and joint motion data; and generating a continuous multidimensional intent factor based on the nonlinear fusion of the geometric consistency factor, the dynamic damping factor, and the active gating factor, for optimizing the execution process of the target spatial action.

[0005] According to an embodiment of the present invention, in the process of executing a target space action, acquiring global pose data and joint motion data of the target space action in the global coordinate system includes: acquiring the current position data and current posture data of the end effector in the global coordinate system of the target space action through a preset sensing device as global pose data; and acquiring the instantaneous velocity data of each joint of the end effector as joint motion data through the underlying sensor or joint encoder of the end effector.

[0006] According to an embodiment of the present invention, the generation of geometric consistency factor, dynamic damping factor and active gating factor based on global pose data and joint motion data includes: generating geometric consistency factor based on the pose error screw of the robot in the local coordinate system corresponding to the global pose data, combined with a weight allocation matrix matching the six degrees of freedom of the end effector.

[0007] According to an embodiment of the present invention, before generating the geometric consistency factor based on the pose error screw of the robot in the local coordinate system corresponding to the global pose data and the weight allocation matrix matching the six degrees of freedom of the end effector, the method further includes: transforming the global pose data in the global coordinate system to the local coordinate system of the robot to generate the pose error screw; and generating the weight allocation matrix by normalizing the task tolerance matrix matching the six degrees of freedom of the end effector.

[0008] According to an embodiment of the present invention, in generating the geometric consistency factor, the dynamic damping factor and the active gating factor based on global pose data and joint motion data, the method further includes: generating the dynamic damping factor by using the velocity screw weight norm of the robot's end effector according to a preset Fitts law rule.

[0009] According to an embodiment of the present invention, before generating the dynamic damping factor by using the velocity screw weight norm of the robot's end effector according to a preset Fitts law rule, the method further includes: generating the instantaneous velocity screw of the end effector by mapping the joint motion data obtained by the end effector at the current sampling time of the joint motion data to the Lie group space through a preset geometric Jacobian matrix; and generating the velocity screw weight norm by performing quadratic weighting on the six-dimensional vector of the instantaneous velocity screw according to the velocity weight matrices corresponding to the translational velocity component and the rotational velocity component of the end effector.

[0010] According to an embodiment of the present invention, in generating the geometric consistency factor, the dynamic damping factor, and the active gating factor based on global pose data and joint motion data, the method further includes: generating the active gating factor by using the cosine similarity between the pose error screw corresponding to the geometric consistency factor and the instantaneous velocity screw corresponding to the dynamic damping factor.

[0011] According to an embodiment of the present invention, in the process of generating a continuous multidimensional intent factor based on the nonlinear fusion of geometric consistency factor, dynamic damping factor and active gating factor to optimize the execution of target space actions, the method includes: performing nonlinear fusion processing on geometric consistency factor, dynamic damping factor and active gating factor based on preset multiplication gating rules and through preset logical activation function to generate a continuous multidimensional intent factor.

[0012] Another aspect of the present invention provides a data processing device for robot teleoperation, comprising an action execution module, a data acquisition module, a factor generation module, and a factor fusion module. The action execution module controls the robot's end effector to execute a target space action matching the current target action in response to teleoperation data of the operator's current target action. The data acquisition module acquires global pose data and joint motion data of the target space action in a global coordinate system during the execution of the target space action. The factor generation module generates a geometric consistency factor, a dynamic damping factor, and an active gating factor based on the global pose data and joint motion data. The factor fusion module generates a continuous multidimensional intent factor based on the nonlinear fusion of the geometric consistency factor, the dynamic damping factor, and the active gating factor, for optimizing the execution process of the target space action.

[0013] Another aspect of the present invention provides an electronic device including one or more processors and a memory for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors perform the data processing method described above for robot teleoperation.

[0014] Another aspect of the present invention provides a computer-readable storage medium having executable instructions stored thereon, which, when executed by a processor, cause the processor to perform the data processing method described above for robot teleoperation.

[0015] Another aspect of the present invention provides a computer program product including a computer program that, when executed by a processor, implements the above-described data processing method for robot teleoperation.

[0016] The data processing method for robot teleoperation provided in this invention can at least partially solve the problems of low accuracy and poor stability in precision assembly tasks in related teleoperation technologies, and therefore can achieve at least one of the following technical effects:

[0017] (1) It realizes the precise quantification of the operation intention: It can innovatively transform the operator's vague and inherent operation intention into a precise, continuous and normalized mathematical scalar α, so that the robot system can "understand" the human operation intention, fundamentally solving the "semantic blindness" problem of traditional methods.

[0018] (2) Improved robustness of intention reasoning: By integrating features of three dimensions—geometry, dynamics, and direction—it avoids misjudgment that may be caused by a single feature (such as relying solely on distance judgment), and can robustly distinguish between easily confused complex scenarios such as "high-precision alignment" and "rapid passing," significantly improving the accuracy of intention reasoning.

[0019] (3) Provides high-quality semantic input for downstream control modules: The generated intention factor α provides key and unambiguous semantic information for subsequent advanced functions such as adaptive filtering and shared control. It is the basis for solving the "causal confusion" problem at the source of the signal, realizing high-quality teleoperation and generating high-quality teaching data.

[0020] It should be understood that the above general description and the following specific embodiments are merely exemplary and illustrative, and do not limit the scope of the invention. Attached Figure Description

[0021] The above-described features, other objects, and advantages of the present invention will become clearer from the following description of embodiments of the invention with reference to the accompanying drawings, in which:

[0022] Figure 1 The illustrations schematically depict application scenarios of data processing methods and / or execution optimization methods, apparatuses, devices, media, and program products for robot teleoperation according to embodiments of the present invention.

[0023] Figure 2A A flowchart illustrating a data processing method for robot teleoperation according to an embodiment of the present invention is shown schematically.

[0024] Figure 2B A flowchart illustrating an execution optimization method for robot teleoperation according to another embodiment of the present invention is shown schematically;

[0025] Figure 3 This illustration schematically depicts an application scenario of a data processing method for robot teleoperation and / or an execution optimization method for robot teleoperation according to embodiments of the present invention.

[0026] Figure 4 This diagram illustrates another application scenario of the data processing method for robot teleoperation and / or the execution optimization method for robot teleoperation according to embodiments of the present invention.

[0027] Figure 5A This schematic diagram illustrates a structural block diagram of a data processing apparatus for robot teleoperation according to an embodiment of the present invention.

[0028] Figure 5B A schematic diagram illustrates a structural block diagram of an execution optimization device for robot teleoperation according to another embodiment of the present invention; and

[0029] Figure 6 A block diagram of an electronic device suitable for implementing a data processing method for robot teleoperation and / or an execution optimization method for robot teleoperation is illustrated according to embodiments of the present invention.

[0030] The accompanying drawings mentioned above are part of the specification of embodiments of the present invention, illustrating exemplary embodiments of the invention. The drawings, together with the description in the specification, serve to illustrate the principles of the embodiments of the present invention. It should be understood that the above general description with reference to the drawings and the following detailed description are merely exemplary and illustrative, and do not limit the scope of the invention. Detailed Implementation

[0031] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the spirit of the contents disclosed in the present invention will be clearly explained below with reference to the accompanying drawings and detailed description. After understanding the embodiments of the present invention, any person skilled in the art can make changes and modifications based on the technology taught in the present invention without departing from the spirit and scope of the present invention.

[0032] The illustrative embodiments and descriptions of the present invention are used to explain the invention, but are not intended to limit the invention. Furthermore, elements / components using the same or similar reference numerals in the drawings and embodiments are used to represent the same or similar parts.

[0033] The terms "first," "second," etc., used in this invention do not specifically refer to any order or sequence, nor are they intended to limit the invention; they are merely used to distinguish elements or operations described using the same technical terms.

[0034] The directional terms used in this invention, such as up, down, left, right, front, or back, are merely for reference to the accompanying drawings. Therefore, the directional terms used are for illustrative purposes and not for limiting the scope of this invention.

[0035] The terms “comprising,” “including,” “having,” “containing,” etc., used in this invention are all open-ended terms, meaning that they include but are not limited to.

[0036] The term "and / or" as used in this invention includes any or all combinations of the things mentioned.

[0037] In this invention, "multiple" includes "two" and "more than two"; in this invention, "multiple groups" includes "two groups" and "more than two groups".

[0038] The terms "approximately," "about," etc., used in this invention are intended to modify any quantity or error that may vary slightly, but these slight variations or errors do not change the essence of the quantity or error. Generally, the range of slight variations or errors modified by such terms may be 20% in some embodiments, 10% in some embodiments, 5% in some embodiments, or other values. Those skilled in the art should understand that the aforementioned values ​​can be adjusted according to actual needs and are not limited thereto.

[0039] All terms used herein (including technical and scientific terms) have the meanings commonly understood by those skilled in the art, unless otherwise defined. It should be noted that the terms used herein are to be interpreted in a manner consistent with the context of this specification, and not in an idealized or overly rigid way.

[0040] When expressions such as "at least one of A, B, and C" are used, they should generally be interpreted in accordance with the meaning commonly understood by a person skilled in the art (e.g., "a system having at least one of A, B, and C" should include, but is not limited to, systems having A alone, having B alone, having C alone, having A and B, having A and C, having B and C, and / or having A, B, and C, etc.). When expressions such as "at least one of A, B, or C" are used, they should generally be interpreted in accordance with the meaning commonly understood by a person skilled in the art (e.g., "a system having at least one of A, B, or C" should include, but is not limited to, systems having A alone, having B alone, having C alone, having A and B, having A and C, having B and C, and / or having A, B, and C, etc.). A person skilled in the art should also understand that any conjunction and / or phrase that substantially arbitrarily indicates two or more optional items, whether in the specification, claims, or drawings, should be understood to indicate the possibility of including one of these items, either of these items, or both items. For example, the phrase “A or B” should be understood as including the possibility of “A” or “B”, or “A and B”.

[0041] Figure 1 The illustration schematically depicts application scenarios of data processing methods and / or execution optimization methods, apparatuses, devices, media, and program products for robot teleoperation according to embodiments of the present invention.

[0042] like Figure 1As shown, application scenario 100 according to this embodiment may include terminal devices 101, 102, and 103, network 104, and server 105. Network 104 is used as a medium to provide a communication link between terminal devices 101, 102, and 103 and server 105. Network 104 may include various connection types, such as wired or wireless communication links or fiber optic cables, etc.

[0043] Users can use terminal devices 101, 102, and 103 to interact with server 105 via network 104 to receive or send messages, etc. Various communication client applications can be installed on terminal devices 101, 102, and 103, such as shopping applications, web browser applications, search applications, instant messaging tools, email clients, social media platform software, etc. (for example only).

[0044] Terminal devices 101, 102, and 103 can be various electronic devices with displays and web browsing capabilities, including but not limited to smartphones, tablets, laptops, and desktop computers.

[0045] Server 105 can be a server that provides various services, such as a backend management server that supports websites browsed by users using terminal devices 101, 102, and 103 (for example only). The backend management server can analyze and process data such as received user requests, and feed back the processing results (such as web pages, information, or data obtained or generated according to user requests) to the terminal devices.

[0046] It should be noted that the data processing method and / or execution optimization method for robot teleoperation provided in the embodiments of the present invention can generally be executed by server 105. Correspondingly, the data processing device and / or execution optimization device for robot teleoperation provided in the embodiments of the present invention can generally be located in server 105. The data processing method and / or execution optimization method for robot teleoperation provided in the embodiments of the present invention can also be executed by a server or server cluster that is different from server 105 and capable of communicating with terminal devices 101, 102, 103 and / or server 105. Correspondingly, the data processing device and / or execution optimization device for robot teleoperation provided in the embodiments of the present invention can also be located in a server or server cluster that is different from server 105 and capable of communicating with terminal devices 101, 102, 103 and / or server 105.

[0047] It should be understood that Figure 1 The number of terminal devices, networks, and servers shown is merely illustrative. Depending on implementation needs, any number of terminal devices, networks, and servers can be included.

[0048] The following will be based on Figure 1 The described scene, through Figures 2A to 4 The data processing method for robot teleoperation of Embodiment 1 and / or the execution optimization method for robot teleoperation of Embodiment 2 are described in detail.

[0049] Example 1

[0050] Teleoperation is a key technology for humanoid robots to achieve complex manipulation and acquire imitation learning teaching data. Its core task is to accurately map the operator's motion intentions to the robot's execution end through a human-machine interface. However, in high-precision tasks such as precision assembly (e.g., sub-millimeter pin hole assembly) and dexterous manipulation, the sensing accuracy and sampling frequency of low-cost teleoperation systems are limited, and the operator's inherent physiological tremors (usually concentrated at 8-12 Hz) and unconscious corrective movements become bottlenecks restricting system performance.

[0051] Existing teleoperation-assisted control methods mainly rely on techniques such as time-domain filtering or virtual fixtures. Traditional time-domain filters (such as 1-Euro filters) suppress noise by adjusting the cutoff frequency, but they often exhibit severe "semantic blindness" in the high-precision alignment stage, meaning they cannot distinguish between task-related effective motion and physiological tremors orthogonal to the task direction, and are prone to introducing phase lag, affecting the real-time performance and intuitiveness of the operation. Guidance methods based on Artificial Potential Field (APF) or virtual fixtures provide spatial constraints, but under low-frequency control loops, their internal "virtual springs" are prone to destructive coupling with human muscle tremors, inducing high-frequency oscillations. In addition, discrete switching schemes based on Finite State Machines (FSMs) generate acceleration jerks at the capture boundary, which disrupt the continuity of motion and give the operator a distinct "virtual wall" impact sensation.

[0052] Furthermore, existing teleoperation frameworks struggle to effectively identify and filter operator trial-and-error jitter at the signal source, resulting in these task-irrelevant cluttered signals being recorded into the teaching trajectory. This can lead to "causal confusion" in downstream imitation learning tasks, where generative policies (such as diffusion policies) may mistake these jittered signals for necessary control strategies, triggering unsafe behaviors during autonomous execution. Therefore, current technologies lack an effective means to accurately and robustly perceive and infer the operator's true intentions from raw signals.

[0053] In view of at least one of the above-mentioned problems, embodiments of the present invention aim to provide a data processing method, apparatus, device, medium, and product for a humanoid robot or robotic arm teleoperation system that can perceive and quantify the operator's intention in real time. This can solve a series of problems caused by the inability to accurately perceive the operator's intention in existing teleoperation technologies. Specifically, it can infer the operator's true task intention (e.g., whether in a state of free exploration, precise alignment, or intention to withdraw) in real time and accurately from teleoperation signals mixed with physiological tremors and unconscious corrections, and quantify it continuously and normally, thereby overcoming the problems of "semantic blindness" and discontinuous control in traditional methods.

[0054] The following will be based on Figure 1 The described scene, through Figures 2A to 4 The data processing method for robot teleoperation according to the disclosed embodiments is described in detail.

[0055] like Figure 2A As shown, one aspect of an embodiment of the present invention provides a data processing method for robot teleoperation, comprising operations S211 to S214.

[0056] In operation S211, in response to the teleoperation data of the operator's current target action, the robot's end effector is controlled to perform a target space action that matches the current target action;

[0057] In operation S212, during the execution of the target space action, global pose data and joint motion data of the target space action in the global coordinate system are acquired;

[0058] In operation S213, a geometric consistency factor, a dynamic damping factor, and an active gating factor are generated based on global pose data and joint motion data; and

[0059] In operation S214, a continuous multidimensional intent factor is generated based on the nonlinear fusion of geometric consistency factor, dynamic damping factor and active gating factor, which is used to optimize the execution process of target space action.

[0060] An end effector refers to a gripper or dexterous hand connected to the end of a robotic arm via a flange and connectors. In this case, a dexterous hand is used as the end effector.

[0061] In the embodiments of the present invention (including Embodiment 2 described below), the so-called robot can be an intelligent machine device with mechanical actuators that has autonomous execution capability or is controlled to execute, including single mechanical devices with only mechanical operating parts (such as single-controlled robotic arms, single robotic hands or single dexterous hands, etc.), humanoid devices with double robotic arms and double robotic legs, etc., and there is no specific limitation.

[0062] Teleoperation technology primarily enables real-time control of remote machines or systems by humans or embodied intelligence (i.e., operators) through interactive devices. Its core lies in synchronizing the operator's movements with the remote execution device. Operators typically wear motion sensing devices (such as handles or sensor-operated robotic arms). These devices detect motion data (such as position and posture) of the operator's arms or hands, generating teleoperation data at the operator's end. The current target action refers to the operator's body movements during the teleoperation data detection process conducted by the sensing device at the current moment.

[0063] An end effector can be a robot that performs a motion task in response to teleoperation data from an operator's current target action, such as a dexterous hand or a robotic arm linked to a dexterous hand. Target space motion can be an action that the end effector imitates by referencing the current target action.

[0064] Global pose data can be the motion data such as the position and posture of the robot's end effector in the global coordinate system during the execution of the action in the target space. In addition, joint motion data can be the motion data of each joint of the end effector corresponding to the global pose data, such as joint rotation speed and joint turning.

[0065] In this embodiment of the invention, the geometric consistency factor can be a parameter that quantifies the spatial proximity of the end effector and the target pose by calculating the error screw between them and weighting it in conjunction with the task tolerance. The dynamic damping factor can be a parameter based on Fitts' law that senses the speed of the operator's movements by evaluating the velocity screw norm of the end effector, identifying the operator's intention to actively decelerate for fine-tuning operations. The master gate chair is a parameter that calculates the cosine similarity between the current direction of motion of the end effector and the direction toward the target to determine whether the operator is actively approaching the target or intending to withdraw (used to distinguish between convergence and withdrawal intentions).

[0066] For the nonlinear intent fusion based on the geometric consistency factor, dynamic damping factor and active gating factor mentioned above, a nonlinear function with multiplicative gating can be used to fuse the three independent feature factor parameters into a final intent factor (i.e., a continuous multidimensional intent factor) that changes continuously and smoothly within the interval. This can be used to accurately quantify the system's confidence that the operator is performing a high-precision alignment task.

[0067] Therefore, by utilizing the data processing method for robot teleoperation described above in this embodiment of the invention, the continuous optimization of target space actions through the aforementioned continuous multidimensional intent factors achieves precise quantification of operational intent. Specifically, it innovatively transforms the operator's vague and intrinsic operational intent into a precise, continuous, and normalized mathematical scalar α, enabling the robot system to "understand" the human's operational intent and fundamentally solving the "semantic blindness" problem of traditional methods. Furthermore, this also enhances the robustness of intent reasoning. Specifically, by integrating features from three dimensions—geometric, dynamic, and directional—it avoids misjudgments that may arise from a single feature (such as relying solely on distance judgment), and can robustly distinguish between easily confused complex scenarios such as "high-precision alignment" and "rapid glide," significantly improving the accuracy of intent reasoning. Finally, it provides high-quality semantic input for downstream control modules (as described in Embodiment 2 below). Specifically, the generated continuous multidimensional intent factor α provides crucial and unambiguous semantic information for subsequent advanced functions such as adaptive filtering and shared control, forming the basis for solving the "causal confusion" problem at the signal source, achieving high-quality teleoperation, and generating high-quality teaching data.

[0068] like Figures 2A-4 As shown, according to an embodiment of the present invention, during the execution of the action in the target space in operation S212, the acquisition of global pose data and joint motion data of the target space action in the global coordinate system includes:

[0069] The current position and attitude data of the end effector in the global coordinate system are obtained by pre-set sensing devices as global pose data;

[0070] The instantaneous velocity data of each joint of the end effector is obtained as joint motion data through the underlying sensors or joint encoders of the end effector.

[0071] Global pose data can include the current position and current orientation data of the end effector during the execution of actions in the target space. The current position data can be sensor data related to the execution position of the end effector in the global coordinate system, while the current orientation data can be sensor data related to the execution orientation (such as direction) of the corresponding end effector. Specifically, global pose data can be acquired in real time using visual sensing devices (i.e., pre-defined perception devices) such as depth cameras (e.g., Intel D435i). For example, these pre-defined perception devices can be the "eyes" at the head position of a humanoid robot. Therefore, the target space action (which can be understood as the target task) in the Lie group space can be acquired using pre-defined perception devices. Global pose data in The target task refers to the process by which a robotic arm's end effector converges from a free motion state to a pre-defined low-dimensional task manifold in the operational space SE(3), and ultimately reconstructs a specific spatial topological relationship under manifold constraints. The low-dimensional task manifold (Monifold Constrained path) is defined by the target pose. The geometric features and operational semantics are jointly defined.

[0072] The target task includes, but is not limited to, several features such as phased features, constraint features, and accuracy features. Phased features can be understood as the task comprising two semantic phases: free space exploration and manifold convergence. Constraint features can be understood as the need to identify the operator's intent and dynamically apply anisotropic spatial constraints during task execution to suppress motion deviations in non-task directions. Accuracy features can be understood as the task objective typically involves sub-millimeter-level spatial alignment or precision assembly, with alignment requirements exceeding the original accuracy level of the operational sensing source.

[0073] To further illustrate the broad applicability of the above definition, the target tasks of the embodiments of the present invention include, but are not limited to, the following types: (1) Precision assembly of holes and shafts: such as cylindrical pin hole assembly (5 degrees of freedom constraint), USB interface plugging and unplugging (6 degrees of freedom constraint), etc.; (2) Path / surface constraint tasks: such as industrial grinding, precision gluing or surgical suturing. The common feature of these tasks is that the end of the robotic arm needs to move along a specific spatial curve or surface, and the curve or surface constitutes the low-dimensional task manifold.

[0074] In this embodiment of the invention, physiological tremor noise typically refers to involuntary, quasi-periodic limb shaking or oscillating movements generated by the human neuromuscular system during teleoperation tasks. At the signal processing level, this noise possesses physical, spectral, and spatial characteristics. The physical characteristics manifest as minute, high-frequency pose fluctuations superimposed on the operator's subjective operational intention trajectory. The spectral characteristics, particularly the power spectral density (PSD), are typically concentrated in the high-frequency range of 8-12 Hz and exhibit a high-entropy distribution. The spatial characteristics can be understood as, within a task-manifold-based control framework, this noise is defined as a non-task-related motion component orthogonal to the target task trajectory. Therefore, this noise is amplified in low-cost sensing devices or low-frequency servo control, causing the end-effector trajectory to deviate from the preset task constraint space and increasing the system's cognitive load and control entropy. High entropy is used here to qualitatively describe the disorder and unpredictability of operational commands. Due to sensor noise from low-cost exoskeletons and the operator's physiological tremors, the input original commands... It contains a large number of random fluctuations that are unrelated to the mission objective.

[0075] In an embodiment of the present invention, for example, during the hole-shaft assembly process, the position and orientation of the target hole are obtained through a vision sensing module. The task manifold constraint is as follows: the movement of the "shaft" to be assembled has full degrees of freedom in all directions before it comes into contact with the hole, and after the "shaft" comes into contact with the hole, it has only a single degree of freedom to move along the depth direction of the hole.

[0076] Therefore, in this embodiment of the invention, a Lie group-based method is constructed. The task manifold. The fundamental definition of a Lie group specifically satisfies the following two aspects: group properties and smoothness. Group properties state that elements within a set satisfy closure, associativity, existence of an identity element, and an inverse element. Smoothness defines the group multiplication operation. Sum of inverse operations They are all smooth (infinitely differentiable) mappings.

[0077] In this embodiment of the invention, a special Euclidean group can be used. To describe the state of the robot's end effector or task target. It is composed of rotation matrix (Special Orthogonal Group) Translation Vector The semi-direct product formed:

[0078] (1)

[0079] in, Represents a three-dimensional rotation group, satisfying and , where I is the identity matrix.

[0080] It should be noted that the state of the task target can be, for example, the object's posture (such as orientation) and position in a task of grasping an object. Specifically, this posture and position information can be defined using Euler angles and quaternions.

[0081] Joint motion data can include instantaneous joint velocity information from the underlying sensors of the end effector or joint encoders, that is, the instantaneous velocity of each joint is obtained through encoders or sensors at the bottom of the robotic arm. As the result of raw motion data acquisition.

[0082] In embodiments of the present invention, such as Figure 2A Operation S213, as shown, can calculate the intent factor in real time by extracting the geometric consistency, dynamic damping, and active gating features of the robotic arm's end effector. This enables the construction of a multi-dimensional intention reasoning network. Specifically, Lie algebra logarithmic mapping can be used to calculate the pose error screw, combined with the end-effector velocity screw norm, and a nonlinear fusion algorithm can be used to quantify the deterministic convergence of the system state (robotic arm state vector) to the task manifold. Task manifold convergence can be understood as the manifold being the form in which the state vector exists in the state space; it is a mathematical abstraction. The task manifold is the geometric constraint of the task. For example, in a vertical hole-shaft assembly task, before the end-effector TCP (Tool Coordinate System) touches the hole, the geometric constraint is a funnel-like cone-shaped Cartesian space; after the TCP touches the hole, it becomes a Cartesian space parallel to the depth direction of the hole. The deterministic convergence of the task manifold can be understood as "determining which stage the current state is in. For example, if it is aligning empty spaces, a calculated 'a' value close to 1 is high, which is equivalent to making a judgment: the probability that the current state is converging towards the task manifold is very high."

[0083] Finally, regarding multidimensional intent factors The calculation can be performed by acquiring the pose error, velocity screw weight norm, and operational trend of the robotic arm's end effector in real time. Specifically, the calculation involves the cosine similarity between the direction vector between the end effector and the target point and the actual motion direction vector being executed by the end effector. The geometric consistency factor is then calculated first. Dynamic damping factor and active gating factor The velocity screw weight norm is used to calculate the dynamic damping term, and the velocity screw is used to calculate the direction gating term. The so-called operational trend can be understood as the cosine similarity between the direction vector between the TCP point of the end effector and the target point, and the actual motion direction vector being executed by the end effector.

[0084] Furthermore, the end-effector spinor norm can be a scalar that dynamically changes over time, used to quantify the instantaneous motion intensity of the robotic arm's end effector in three-dimensional space in real time. The linear and angular velocities of the end effector can be weighted and fused as a dynamic characteristic for the system to identify when the operator enters the precision alignment phase. According to Fitts's law, this norm naturally decreases as the operator approaches the target and begins precision alignment, thereby driving the system to generate a virtual sense of contact and enhance motion constraints.

[0085] Furthermore, the variables and operators involved in the end-effector velocity screw norm can include the end-effector velocity screw and velocity weight matrix, as well as matrix transpose calculation. The end-effector velocity screw can be a six-dimensional vector containing the instantaneous linear velocity and instantaneous angular velocity of the end-effector at the current moment. The velocity weight matrix is ​​usually a positive definite diagonal matrix, which serves to balance the different dimensions of linear velocity and angular velocity and adjust the system's sensitivity to different motion directions according to task requirements. The matrix transpose notation can be used to convert the velocity screw from a column vector to a row vector for inner product operations.

[0086] like Figures 2A-4 As shown, according to an embodiment of the present invention, in operation S213, generating a geometric consistency factor, a dynamic damping factor, and an active gating factor based on global pose data and joint motion data, the following is included:

[0087] Based on the pose error spinor in the robot's local coordinate system corresponding to the global pose data, and combined with the weight allocation matrix that matches the six degrees of freedom of the end effector, a geometric consistency factor is generated.

[0088] Geometric consistency factor Spatial proximity (i.e., pose error spinor) can be evaluated using exponential mapping. ), where the weight matrix and the task tolerance Mapping association. Spatial proximity can be understood as the spatial distance between the endpoint location and the target location.

[0089] In this embodiment of the invention, the task tolerance matrix is ​​utilized. Spatial proximity can be determined through exponential mapping; therefore, the geometric consistency factor can be:

[0090] (2)

[0091] Among them, the quadratic term The weighted square norm in a Lie algebra space can be defined geometrically. Similar to the Mahalanobis distance structure in statistics, it can be defined using a weight matrix. The errors of different degrees of freedom (such as translation and rotation) are standardized in terms of dimensions and weighted accordingly. Specifically, the weight matrix... This can be understood as the weight allocation matrix.

[0092] like Figures 2A-4 As shown, according to an embodiment of the present invention, before generating the geometric consistency factor based on the pose error screw of the robot in the local coordinate system corresponding to the global pose data, combined with the weight allocation matrix matching the six degrees of freedom of the end effector, the method further includes:

[0093] The global pose data in the global coordinate system is transformed into the robot's local coordinate system to generate the pose error screw.

[0094] A weight allocation matrix is ​​generated by normalizing the task tolerance matrix that matches the six degrees of freedom of the end effector.

[0095] In this embodiment of the invention, in conjunction with the foregoing description, the pose error spinor It can be represented as:

[0096] (3)

[0097] Wherein, the global target pose is ; It can represent the transformation from the robot coordinate system (the coordinate system with the robot as the origin) to the global coordinate system (the coordinate system with the robot and its environment). This indicates a transformation "from the global coordinate system to the robot coordinate system". Wherein, Its function is to set the global target pose. The coordinates are transformed to the robot's own local coordinate system to calculate the relative error with respect to the current position, which reflects the magnitude of the robotic arm's pose deviation. It is the vee operator.

[0098] Pose error screw It can be represented in Lie algebra space as describing the motion of a robotic arm's end effector from its current pose to its target pose. The required minimum helical motion characteristic vector. This vector is a six-dimensional vector composed of three-dimensional linear velocity components and three-dimensional rotational angular velocity components, which completely characterizes the instantaneous deviation of the end effector in three-dimensional space.

[0099] Weighting matrix It is The positive definite diagonal matrix, whose diagonal elements correspond to the weight distribution of the end effector in the six degrees of freedom (three translational dimensions and three rotational dimensions). In formula (2), the weight distribution matrix... Located at the core of the exponential term, it acts as a quadratic matrix on the error spinor. Therefore, this weighting matrix can act as a "scaling ruler," meaning that when the weight of a certain dimension increases, even a small error in that dimension will lead to an increase in the geometric consistency factor. The rapid decrease demonstrates the system's high sensitivity to the accuracy of this dimension.

[0100] In addition, the task tolerance matrix This represents the geometric constraints of the target task (such as an assembly task), reflecting the maximum allowable physical deviation in hole-shaft assembly. The weight allocation matrix... It can be based on the task tolerance matrix. The operator obtained after normalization mapping is used to adjust the sensitivity of the system to pose errors in different directions in formula (2).

[0101] Among them, the weight allocation matrix and task tolerance matrix The mapping relationship between the two satisfies:

[0102] (4)

[0103] in, It is the task tolerance matrix The diagonal elements, i.e. In a physical sense, This represents the maximum allowable error boundary for the task manifold across six degrees of freedom (3-axis translation and 3-axis rotation), and its specific value can be determined by the geometric constraints of the assembly task. For example, in a shaft-hole assembly task, and Let it be the difference between the hole diameter and the shaft diameter (fit tolerance); and for the unconstrained direction, its value can be set to be larger.

[0104] Furthermore, in the mapping relation, It is a dimensionless adjustable gain, which can be referred to as the "sensitivity scaling factor" or "uniformity intensity coefficient" in the embodiments of this invention. Wherein, when When the value is large, the system is extremely sensitive to behavior that deviates from the tolerance range; geometric consistency factor It will collapse rapidly, triggering strong conflict detection logic. When When the value is small, the system exhibits a stronger "soft constraint" characteristic, allowing the operator to make deviations within a certain range without immediately cutting off the virtual boot.

[0105] like Figures 2A-4 As shown, according to an embodiment of the present invention, in operation S213, which generates the geometric consistency factor, dynamic damping factor, and active gating factor based on global pose data and joint motion data, the following further step is taken:

[0106] According to the pre-defined Fitts' law, the dynamic damping factor is generated by using the velocity screw weight norm of the robot's end effector.

[0107] Dynamic damping factor Virtual contact resistance can be simulated during the alignment phase using the velocity screw weight norm. The alignment phase can be understood as the stage in a target task (such as a hole-shaft assembly task) where the TCP point of the robot's dexterous hand aligns with the target object. Furthermore, virtual contact resistance can refer to the simulated damping force generated in real-time by the control algorithm based on the current task state (such as pose error or intent factor) within a human-robot collaborative or shared control framework. Virtual contact resistance does not originate from physical contact; rather, it is altered through noise covariance scheduling in the control law of an Adaptive Kalman Filter (AKF) to change the dynamic behavior of the robotic arm, making the operator experience a resistance similar to that of passing through a viscous fluid (such as honey or water).

[0108] Therefore, virtual contact resistance refers to the simulated damping force generated in real time by the control algorithm based on the current task state (such as pose error or intention factor) under the human-machine collaboration or shared control framework. It does not come from physical contact, but changes the dynamic behavior of the robotic arm through noise covariance scheduling in the AKF control law, so that the operator feels a resistance similar to passing through a viscous fluid (such as honey or water) during the operation.

[0109] In this embodiment of the invention, the dynamic damping factor Based on the pre-defined Fitts' law, the velocity screw weight norm of the end effector can be utilized. To determine this, virtual contact perception is achieved. That is, when the operator "slows down," the end effector is usually considered to be performing fine operations. This leads to an increase in the dynamic damping term to make its total intention factor higher and closer to 1, thereby attempting to make the Kalman filter stronger.

[0110] The dynamic damping factor can be expressed as follows:

[0111] (5)

[0112] in, This can be the velocity screw weight norm, specifically at each sampling time. The norm is obtained through the system.

[0113] like Figures 2A-4 As shown, according to an embodiment of the present invention, before generating the dynamic damping factor based on the velocity screw weight norm of the robot's end effector according to a preset Fitts's law rule, the method further includes:

[0114] Based on the joint motion data obtained by the end effector at the current sampling time of the joint motion data, the instantaneous velocity spinor of the end effector is generated by mapping it to the Lie group space through a preset geometric Jacobian matrix.

[0115] Based on the velocity weight matrices corresponding to the translational and rotational velocity components of the end effector, the six-dimensional vector of the instantaneous velocity screw is subjected to quadratic weighting to generate the velocity screw weight norm.

[0116] As mentioned above, during the raw data acquisition process, the instantaneous speed of each joint can be obtained through encoders or sensors at the bottom of the robotic arm. These are used as joint motion data.

[0117] Instantaneous velocity spinor This could describe the robotic arm's end effector at the current sampling moment. A six-dimensional vector representing an instantaneous state of motion can be defined in Lie algebra space. In the middle. This vector is obtained by acquiring the instantaneous velocity of each joint of the robotic arm. And utilize the pre-defined geometric Jacobian matrix of the end effector. Mapping to the operation space achieves spatial mapping, that is:

[0118] (6)

[0119] in, It includes three-dimensional instantaneous linear velocity and three-dimensional instantaneous angular velocity.

[0120] Instantaneous velocity spinor It includes three-dimensional instantaneous linear velocity components and three-dimensional instantaneous angular velocity components, which can be used to characterize the actual motion trend generated by the operator driving the end effector of the robotic arm. The system's Jacobian matrix... It can be obtained in real time through the kinematic parameter model provided by the robot itself (such as the URDF description file).

[0121] Introduce a preset velocity weight matrix The above six-dimensional vectors are subjected to quadratic weighting operations to achieve weighted fusion, and finally the velocity spinor weight norm in scalar form is extracted. Among them, the preset velocity weight matrix It can satisfy:

[0122] (7)

[0123] in, The translational velocity weight at the actuator end is the component in the x-direction. The translational velocity weight at the actuator end is the component in the y-direction. The component of the translational velocity weight at the actuator end in the z-direction; The rotational velocity weight of the actuator end joint is the component in the x-direction. The component of the rotational velocity weight of the actuator end joint in the y-direction; Let (x, y, z) be the z-component of the rotational velocity weight of the actuator end joint. Here, (x, y, z) are the coordinates in the global coordinate system.

[0124] The specific values ​​of the velocity weight matrix M need to be considered to ensure that the contributions of linear velocity and angular velocity to the norm are equal, and to satisfy the condition that "the weight of the axis to which the system is expected to respond faster should be increased".

[0125] In summary, the velocity spinor weighting norm of the embodiments of the present invention satisfy:

[0126] (8)

[0127] in, As the translation speed weight, The rotational speed weights are used for translational speed weights and rotational speed weights. The values ​​for these weights can be derived from the following logic:

[0128] Characteristic length method: To ensure that the contributions of linear velocity and angular velocity to the norm are equal, the following value is typically taken:

[0129] (9)

[0130] in, This can be the characteristic length of the end effector tool of the robotic arm (e.g., the length of a gripper or dexterous hand). This can be used to reflect the offset of the translational speed of the end effector and the offset of the rotational speed of the end effector joint.

[0131] like Figures 2A-4 As shown, according to an embodiment of the present invention, in operation S213, which generates the geometric consistency factor, dynamic damping factor, and active gating factor based on global pose data and joint motion data, the following further step is taken:

[0132] An active gating factor is generated by using the cosine similarity between the pose error screw corresponding to the geometric consistency factor and the instantaneous velocity screw corresponding to the dynamic damping factor.

[0133] In this embodiment of the invention, for the active gating factor The cosine similarity between instantaneous velocity screw and pose error screw can be used to distinguish between convergence and withdrawal intentions, thus effectively preventing trajectory lock-in. Trajectory lock-in can be understood as a phenomenon that occurs in shared control systems when the system only weights parameters based on "distance from the target." For example, imagine an operator pulling a robotic arm through a hole with a rubber band. As the distance decreases, the pulling force increases. If the operator suddenly realizes they've inserted it incorrectly and tries to pull it back, the algorithm-generated attraction (or constraint) is extremely strong due to the proximity to the target, making it impossible to pull back. The robotic arm is locked onto the trajectory towards the target, as if attracted by a magnet.

[0134] Instantaneous velocity spinor It describes the end effector of the robotic arm at the current sampling time. A six-dimensional vector representing the instantaneous motion state is defined in Lie algebra space. This vector can be used to obtain the instantaneous velocity of each joint of the robotic arm. And using the geometric Jacobian matrix Mapped to the operation space, we obtain equation (6). Instantaneous velocity spinor It includes three-dimensional instantaneous linear velocity components and three-dimensional instantaneous angular velocity components, which characterize the actual motion trend generated by the operator driving the end effector of the robotic arm.

[0135] Pose error screw It can be a six-dimensional vector describing the geometric deviation between the current pose of the robotic arm's end effector and the target pose. Pose error screw. By analyzing the current pose matrix With the target pose matrix The relative transformation is used to perform a logarithmic map and utilize... The operator is extracted, i.e., formula (3). Pose error screw. The vector represents, mathematically, the minimum spiral motion required to move the end effector from its current pose to the target pose, with its direction pointing toward the mission objective.

[0136] Active gating factor in embodiments of the present invention The cosine similarity between instantaneous velocity screw and pose error screw can be used to evaluate trend consistency, which can be used to distinguish between active convergence intention and withdrawal intention. Among them, the core logic of active gating factor is to use cosine similarity to determine whether the "direction to go" and the "direction of the target" are consistent. (1) Convergence intention: When the instantaneous velocity screw Rotation with pose error When the included angle is acute (cosine similarity greater than 0), it means the operator is driving the robotic arm closer to the target. At this point, the system determines it as a convergence intention, and the active gating factor is activated. (2) Evacuation intention: When the included angle is obtuse (cosine similarity less than 0), it means that the operator is driving the robotic arm away from the target (for example, if an alignment error is found and it needs to be restarted). At this time, the system judges it as an evacuation intention and the active gating factor is activated. The system quickly demotes the user's privileges, automatically removes the constraints, and completely returns control to the operator.

[0137] The active gating factor in this embodiment of the invention satisfies:

[0138] (10)

[0139] in, The preset, extremely small positive constant is used as a numerical smoothing factor. Its technical significance lies in: calculating the active gating factor. At the same time, it can prevent the rotation of the end effector speed of the robotic arm. or error spinor The division-to-zero anomaly caused by the parameter approaching zero ensures the numerical stability of the directional consistency detection process under all operating conditions, avoiding numerical mutations in control commands at the start or end of the task.

[0140] like Figures 2A-4 As shown, according to an embodiment of the present invention, in the process of generating a continuous multidimensional intent factor based on the nonlinear fusion of geometric consistency factor, dynamic damping factor and active gating factor in operation S214 to optimize the execution of target space actions, the following steps are included:

[0141] Based on preset multiplication gating rules, a nonlinear fusion process is performed on geometric consistency factors, dynamic damping factors, and active gating factors through a preset logical activation function to generate continuous multidimensional intent factors.

[0142] In this embodiment of the invention, nonlinear intent fusion is used to generate intent factors. Specifically, a preset logical activation function such as the Sigmoid function can be used, and a preset multiplicative gating mechanism is used to fuse the above-mentioned multidimensional features (geometric consistency factor, dynamic damping factor, and active gating factor) into a continuous intent factor. The Sigmoid function, also known as the sigmoid function or logistic function, is an activation function that can be applied to neural networks and machine learning. In this embodiment of the invention, the main characteristic of the preset Sigmoid function is that it can map any real number to a value between 0 and 1.

[0143] Therefore, the continuous multidimensional intent factor in the embodiments of the present invention This can be expressed as:

[0144] (11)

[0145] in, This is a preset logic activation function; Preset geometric weighting coefficients; Preset speed weighting coefficient; This is the preset gating index.

[0146] The data processing method for robot teleoperation in this invention primarily involves constructing a complete mapping mechanism from raw sensor data to advanced semantic intent. By nonlinearly fusing features across three orthogonal dimensions—geometric consistency, dynamic damping, and active gating—the resulting intent factor α is no longer a simple threshold judgment of distance or speed, but a continuous variable that robustly and comprehensively reflects the operator's true intent. This technical solution paves the way for fundamentally resolving the contradiction between jitter suppression and operational flexibility in teleoperation, as well as the "causal confusion" problem in teaching data, and is a key element in achieving a high-precision, highly natural human-computer interaction system.

[0147] Therefore, the data processing method for robot teleoperation in this embodiment of the invention provides a robot teleoperation intent reasoning method based on multi-dimensional feature perception. This method can extract and fuse features of three dimensions—geometric, dynamic, and directional—in parallel to calculate a continuously changing "intent factor" α in real time.

[0148] Specifically, firstly, a task manifold based on the Lie group SE(3) can be constructed. The pose state of the robot end effector and the task target is described by using a special Euclidean group SE(3). The target pose is obtained by using external sensing devices, providing a unified mathematical framework and geometric reference for intent reasoning. Furthermore, three core feature factors are computed in parallel through a multidimensional intent reasoning network to fully capture the operator's intent: (1) Geometric consistency factor The pose error screw between the end effector and the target pose is calculated and weighted in combination with the task tolerance to quantify the spatial proximity between the two; (2) dynamic damping factor Based on Fitts's law, the speed of the operator's movement can be sensed by evaluating the velocity screw norm of the end effector, and the intention of actively decelerating for fine operation can be identified; (3) Active gating factor By calculating the cosine similarity between the current direction of motion of the end effector and the direction toward the target, it is determined whether the operator is actively approaching the target or intending to withdraw, thus distinguishing between convergence and withdrawal intentions. Then, nonlinear intention fusion is performed: a multiplicative-gated nonlinear function can be used to fuse the three independent feature factors into a final intention factor α that changes continuously and smoothly within an interval. This factor precisely quantifies the system's confidence that the operator is performing a high-precision alignment task.

[0149] Based on the above-described data processing method for robot teleoperation, this invention also provides a data processing device for robot teleoperation. The following will be combined with... Figure 5A The device is described in detail.

[0150] Figure 5A A schematic block diagram of a data processing apparatus 501 for robot teleoperation according to an embodiment of the present invention is shown.

[0151] like Figure 5A As shown, the data processing device 501 for robot teleoperation in this embodiment includes an action execution module 511, a data acquisition module 512, a factor generation module 513, and a factor fusion module 514.

[0152] The motion execution module 511 is used to control the robot's end effector to perform a target space motion that matches the current target motion in response to the operator's teleoperation data for the current target motion. In one embodiment, it can be used to perform the operation S211 described above, which will not be repeated here.

[0153] The data acquisition module 512 is used to acquire global pose data and joint motion data of the target space action in the global coordinate system during the execution of the target space action. In one embodiment, it can be used to perform the operation S212 described above, which will not be repeated here.

[0154] The factor generation module 513 is used to generate geometric consistency factors, dynamic damping factors, and active gating factors based on global pose data and joint motion data. In one embodiment, it can be used to perform the operation S213 described above, which will not be repeated here.

[0155] The factor fusion module 514 is used to generate continuous multidimensional intent factors based on the nonlinear fusion of geometric consistency factors, dynamic damping factors, and active gating factors, which are used to optimize the execution process of target space actions. In one embodiment, it can be used to perform the operation S214 described above, which will not be repeated here.

[0156] According to embodiments of the present invention, any plurality of modules among the action execution module 511, data acquisition module 512, factor generation module 513, and factor fusion module 514 can be combined into one module, or any one of these modules can be split into multiple modules. Alternatively, at least part of the functionality of one or more of these modules can be combined with at least part of the functionality of other modules and implemented in one module. According to embodiments of the present invention, at least one of the action execution module 511, data acquisition module 512, factor generation module 513, and factor fusion module 514 can be at least partially implemented as hardware circuitry, such as a field-programmable gate array (FPGA), a programmable logic array (PLA), a system-on-a-chip, a system-on-a-substrate, a system-on-package, an application-specific integrated circuit (ASIC), or any other reasonable means of integrating or packaging the circuitry, or implemented in hardware or firmware, or in any one of software, hardware, and firmware implementations, or in a suitable combination of any of these. Alternatively, at least one of the action execution module 511, data acquisition module 512, factor generation module 513, and factor fusion module 514 can be at least partially implemented as a computer program module, which can perform corresponding functions when the computer program module is run.

[0157] To further verify the role of the aforementioned continuous multidimensional intent factor α, the following embodiment 2 is provided in detail in this invention.

[0158] Example 2

[0159] In robot teleoperation, especially in precision assembly tasks, a core technical challenge lies in handling the relationship between operator intent and robot control execution. Existing technologies typically employ a signal representing the operator's intent (or obtain it through related methods); however, effectively utilizing this intent signal to optimize the control system remains a significant challenge.

[0160] Existing filtering control methods often have the following limitations when utilizing the intended signal:

[0161] (1) The inherent contradiction between filter smoothness and operation delay: Traditional fixed-parameter filters, such as 1-Euro filters, cannot dynamically adjust their filtering strength according to the task context. In order to filter out physiological tremors during fine operations, high-intensity filtering must be used, but this will introduce unacceptable phase delays in the free movement phase where rapid response is required, resulting in sluggish and unresponsive operation. Conversely, if the filtering strength is reduced to ensure real-time performance, high-frequency jitter cannot be effectively suppressed.

[0162] (2) The conflict between the rigidity of virtual constraints and operational flexibility: Although methods based on virtual fixtures (APF) or finite state machines (FSM) can provide task constraints, the way these constraints are applied is "rigid". When the operator's movement touches the constraint boundary, they will feel a harsh impact from the "virtual wall", which disrupts the continuity of movement. More seriously, when human and machine intentions conflict (for example, the operator discovers an alignment error and tries to withdraw), this rigid constraint will produce a "track lock" phenomenon, which will prevent the operator from correcting the error in time and pose a safety hazard.

[0163] (3) Inconsistency between high-level task constraints and low-level physical execution: Traditional methods usually perform constraint planning in the task space (Cartesian space), but in the robot's low-level joint space controller, there is a lack of direct, physical representation of the task manifold. This layered approach leads to unsatisfactory constraint execution and fails to suppress jitter in non-task directions at the root (i.e., at the joint drive level).

[0164] Therefore, there is an urgent need for a new control method that can receive an input signal representing an intention and use it to dynamically and flexibly apply physically consistent constraints, thereby fundamentally resolving the aforementioned existing contradictions.

[0165] In view of at least one of the above problems, embodiments of the present invention aim to provide an execution optimization method, apparatus, device, medium, and product for a teleoperation system for humanoid robots or robotic arms, which can dynamically adjust control characteristics to generate high-quality motion trajectories based on real-time changing operational intentions in precision operation tasks. This allows for the construction of an advanced teleoperation filtering control system, based on known operator intentions. Specifically, it utilizes a real-time intention factor α to dynamically and physically consistently resolve the contradiction between filtering smoothness and operational delay, the conflict between the rigidity of virtual constraints and operational flexibility, and the inconsistency between high-level task constraints and low-level physical execution.

[0166] The following will be based on Figure 1 The described scene, through Figures 2A to 4 The execution optimization method for robot teleoperation according to the disclosed embodiments is described in detail.

[0167] like Figure 2B As shown, one aspect of an embodiment of the present invention provides an execution optimization method for robot teleoperation, which includes operations S221 to S224.

[0168] In operation S221, a continuous multi-dimensional intent factor is generated based on the teleoperation data of the current target action of the operator received;

[0169] In operation S222, the process noise covariance matrix corresponding to the teleoperation data is generated based on the continuous multidimensional intent factor.

[0170] In operation S223, an initial information matrix corresponding to the observation noise covariance matrix is ​​generated based on the continuous multidimensional intention factor;

[0171] In operation S224, based on the preset adaptive Kalman filter, physical regularization is performed on the process noise covariance matrix and the effective information matrix after preprocessing by the collision detector, and the low-entropy expert trajectory is extracted as a regularization control command to achieve remote operation execution optimization.

[0172] The process of generating continuous multidimensional intent factors based on the teleoperation data of the current target action of the operator received in operation S221 can be referred to the content of the aforementioned embodiment 1, and will not be repeated here.

[0173] In this embodiment of the invention, process noise Q can be represented by a random matrix, primarily used to describe the uncertainty of the robotic arm's state-space model. Specifically, during task execution, process noise Q characterizes the random deviations of the robotic arm's actual motion state relative to its ideal physical model. These deviations include control fluctuations caused by physiological tremors, unknown environmental disturbances, and unmodeled terms in the dynamics model. Correspondingly, measurement noise R characterizes errors introduced by factors such as sensor detection, which will not be elaborated upon further.

[0174] In this embodiment of the invention, a process noise covariance matrix Q_k(α) that dynamically changes with the continuous multidimensional intention factor α can be constructed through a covariance anisotropic scheduling process.

[0175] Therefore, the execution optimization method for robot teleoperation in this embodiment of the invention can directly and physically inject abstract task manifold geometric constraints into the state transition model of the underlying filter. This allows the Kalman filter to "know" in which direction of motion strong correction should be performed and in which direction greater degrees of freedom should be allowed at different stages of the task. Specifically, when performing precise alignment (α→1), the process noise component in the constrained direction will decrease sharply, thereby enhancing the filter's correction capability; while in the unconstrained direction or during free movement (α→0), a higher noise margin is allowed, ensuring operational flexibility. This mechanism enables the system to adaptively allocate control weights in different dimensions according to the operational intent, thereby eliminating physiological tremors while preserving the naturalness and efficiency of the operation to the greatest extent.

[0176] like Figures 2B-4As shown, according to an embodiment of the present invention, in operation S222, generating the process noise covariance matrix corresponding to the teleoperation data based on continuous multidimensional intent factors, the following is included:

[0177] Based on continuous multidimensional intent factors Corresponding process noise anisotropy variance And Jacobi pseudo-inverse operator The process noise matrix is ​​generated through joint space transformation. ;

[0178] For the process noise matrix Discrete-time kinematics integration is performed to generate the process noise covariance matrix. .

[0179] In this embodiment of the invention, the joint space transformation process can satisfy the following:

[0180] (12)

[0181] Formula (12) is mainly used to characterize the process of mapping task space constraints to joint velocity space, where joint velocity space can be understood as the spatial representation of inverse kinematics.

[0182] Therefore, Equation (12) can characterize the covariance pull-back mapping from the task space to the joint space. Through this linear mapping, the system can generate task constraints with anisotropic characteristics (process noise anisotropic variance). Projected onto the joint velocity space, an adaptive process noise matrix is ​​generated. This allows high-level task geometry constraints to be directly injected into the robot's underlying motor control loop, thereby achieving targeted suppression of physiological tremors at the physical level. For example, in precision assembly (such as shaft-hole alignment), constraints exist around the end effector. For instance, it can move faster along the axis of the hole (Z-axis), but must move very steadily perpendicular to the axis (XY plane). This requirement of anisotropy is the task space constraint.

[0183] Furthermore, this can be achieved by analyzing the process noise matrix. Perform discrete-time kinematic integration to construct the full-order process noise covariance matrix. Complete the construction of the time-domain integral.

[0184] like Figures 2B-4 As shown, according to an embodiment of the present invention, before generating the process noise matrix through joint space transformation based on the process noise anisotropic variance and Jacobian pseudo-inverse operator corresponding to the continuous multidimensional intention factor, the method further includes:

[0185] The process noise anisotropic variance is generated by using continuous multidimensional intention factors and preset automatic spatial motion covariance and preset constrained spatial motion covariance.

[0186] Based on the transpose of the preset geometric Jacobian matrix, the Jacobian pseudo-inverse operator is generated by the damped least squares method.

[0187] The modeling process for task space variance can be performed in the task slicing space. (As mentioned above, the Lie algebra space) defines the process noise Q, anisotropic variance. as follows:

[0188] (13)

[0189] The anisotropic variance of the process noise can define the dynamic scheduling logic of the task space covariance. The pre-defined free-space motion covariance is used to characterize the motion flexibility under low-constraint conditions. The pre-defined constraint space motion covariance is used to characterize the high-precision constraint requirements during the precision alignment stage. Continuous multidimensional intention factor. As weights, they enable a smooth transition for the robotic arm from compliant movement to precision filtering.

[0190] Furthermore, the Jacobi pseudo-inverse can be calculated using the damped least squares (DLS) method to handle kinematic singularities and complete the damped least squares projection process. Specifically, the Jacobi pseudo-inverse operator is:

[0191] (14)

[0192] in, It is a Jacobi pseudo-inverse operator with singularity resistance. Among them, Let be the transpose of the Jacobian matrix. This is the damping factor in the damped least squares method, used to introduce an identity matrix when the robotic arm is in a singular configuration. The regularization term ensures the stability of numerical calculations and the continuity of control commands.

[0193] In this embodiment of the invention, a kinematic singularity can be understood as a critical state in which the robotic arm, under a specific configuration (such as when the joint axes coincide or when it is at the edge of the workspace), results in a Jacobian matrix rank deficit. Near a singularity, traditional inverse kinematics solutions can produce physically inaccessible instantaneous joint velocities due to excessive gain.

[0194] The method employs damped least squares (DLS) to construct the Jacobian pseudo-inverse matrix. Its core lies in reconstructing the singular values ​​of the Jacobian matrix by introducing a non-negative damping factor, lambda. Near singular points, the damping term lambda becomes dominant, ensuring the matrix remains invertible and thus limiting the calculated joint speed to within the physical limits of the motor. Therefore, this mechanism allows the system to proactively sacrifice a small portion of path tracking accuracy near singular configurations in exchange for the continuity and numerical stability of control commands. This effectively avoids the risk of motor overcurrent and ensures the robustness of the robotic arm's motion under complex paths (such as rotational adjustments during precise alignment of shafts and holes).

[0195] like Figures 2B-4 As shown, according to an embodiment of the present invention, before generating the process noise matrix through joint space transformation based on the process noise anisotropic variance and Jacobian pseudo-inverse operator corresponding to the continuous multidimensional intention factor, the method further includes:

[0196] Based on the state transition matrix corresponding to the preset motion model, the system state vector that follows the discrete-time kinematic integral is generated through the process noise covariance matrix.

[0197] In this embodiment of the invention, it is possible to [do something] at discrete time steps. The kinematic state evolution of the robotic arm follows a discrete-time kinematic integral equation, which can be expressed as the system state vector as follows:

[0198] (15)

[0199] in, This is the state transition matrix corresponding to the preset motion model; The covariance matrix can be the anisotropy matrix in formula (12). (used to characterize process noise) The system state vector can also be expressed as:

[0200] (16)

[0201] This includes the current time. joint angle and joint velocity .

[0202] In this embodiment of the invention, the preset motion model can be a uniform kinematic model, whose state transition matrix... Specifically, it can be expressed as:

[0203] (17)

[0204] in, The sampling period is the reciprocal of the clock frequency of the control system. It is an identity matrix.

[0205] like Figures 2B-4 As shown, according to an embodiment of the present invention, in operation S223, generating an effective information matrix corresponding to the observation noise covariance matrix based on continuous multidimensional intention factors, the following steps are included:

[0206] An initial information matrix is ​​generated based on continuous multidimensional intention factors, a preset human operation information matrix, a preset virtual constraint observation matrix, and a preset task tolerance to generate the corresponding observation noise covariance matrix.

[0207] Preprocessing based on a conflict detector is performed on the initial information matrix to generate an effective information matrix that completes the adaptive constraints of conflict detection.

[0208] In this embodiment of the invention, the initial information matrix can be expressed as follows:

[0209] (18)

[0210] in, This can be understood as a parameter that can be scaled based on the intensity of intent and the collision detection value; Representing human operations, in the task manifold of this embodiment of the invention, it corresponds to the dimension that humans can freely control; Representing virtual constraint terms, in the task manifold of this embodiment, they correspond to the dimensions in which the system imposes geometric constraints and limits fluctuations; basic terms. (and The human operation observation matrix is ​​mainly used to extract the observation dimensions directly related to the operator's control from the system's full state space. A pre-defined human operation information matrix can describe the determinism of the operator's input signals; continuous multi-dimensional intent factors. As a moderating weight, it can reflect the system's level of trust in human intentions; constraint terms (and It can be a preset virtual constraint observation matrix, used to define the normal space of the task manifold, that is, which degrees of freedom are subject to physical or geometric constraints (such as radial displacement during assembly). The pre-defined virtual constraint information matrix can serve as the source of the system's "coercive force". This is the preset task tolerance, used to determine the stringency of the constraint. The smaller the preset task tolerance, the better. The greater the strength. Among them, the composite term... That is, the effective information matrix, which serves as the initial information matrix, can be a joint information metric synthesized through an information filter, used to determine the gain allocation of the final fusion control law.

[0211] In this embodiment of the invention, information filtering is a dual form of Kalman filtering, the core of which lies in using an information matrix instead of a covariance matrix. Specifically, in this embodiment, the information matrix can be defined... To observe the noise covariance matrix The inverse is as follows:

[0212] (19)

[0213] Among them, refer to the aforementioned formula (18). It can represent the determinism of human-operated signals; This can represent the noise covariance of the operator's input; This represents the determinism of virtual constraints.

[0214] Therefore, in embodiments of the present invention, an information filtering architecture can be employed at the computational layer to transform traditional measurement uncertainty into an information density metric by defining the above formula (19). In formula (18), by utilizing the additivity of the information matrix in a linear subspace, seamless decoupling and fusion of the human intention space and the virtual constraint space are achieved. In the system architecture diagram description, observation noise is retained. The covariance matrix is ​​used to characterize the statistical properties of task tolerance and physiological tremor, ensuring a high degree of unity between the simplicity of mathematical operations and the intuitiveness of physical definitions.

[0215] It should be noted that in the system architecture diagram of state estimation, Q can be used to represent process noise and R to represent measurement noise (i.e., observation noise), which is a common language in control engineering. In this embodiment of the invention, Y is used to reflect the logical advantages of the computation process (i.e., the additive characteristics of information fusion).

[0216] In an embodiment of the present invention, a collision detector It can quantify the deviation between the operator's velocity direction and the manifold unit tangent direction in real time. When the deviation exceeds a set threshold... When this occurs, the system can identify non-task intents and adaptively reduce the virtual guidance strength to ensure the operator's permission to override constraints.

[0217] Introducing a collision detector Real-time quantification of command conflicts detects whether two information sources conflict based on Mahalanobis distance. Its core logic is to calculate the human command vector. With virtual guiding vector The distance between them, specifically, the collision detector can satisfy:

[0218] (20)

[0219] in, A preset human operation information matrix can be provided. ; A preset virtual constraint information matrix can be provided. Therefore, if the human instruction vector With virtual guiding vector If the directions are completely opposite (e.g., a person wants to move left, but the algorithm wants to align to the right), then this human instruction vector... With virtual guiding vector The distance between them will increase dramatically.

[0220] The results calculated using the above formula (20) can reflect the degree of inconsistency between the two information sources. When the operator's intention to deviate from the manifold is detected, the guidance intensity is adaptively reduced to ensure that the operator has the authority to cover the physical constraints. Information filtering is the dual form of Kalman filtering and is consistent with the function represented by R. The so-called deviation from the manifold can be understood as deviating from the task manifold constraints. For example, the vision originally guides the operator to move the end effector towards the task target, but because the operator wants to fine-tune it himself in the final approach stage, he can "break free" from the constraints brought by the geometric and velocity terms. The so-called guidance intensity can be understood as the value of a. Adaptive reduction means that if the direction term in the multiplication gate deviates a lot in an instant, even if the distance and velocity terms bring a large component, the total a can be reduced in an instant to allow the operator to "break free" from the sense of obstruction. The quantization command conflict can be understood as: for example, sometimes the detected distance and direction terms want to "close closer", but the operator himself gives a large deviation command. The system will listen to the operator. The quantization method is through the fusion of information matrices.

[0221] In this embodiment of the invention, the semantic arbitration module is developed based on information filtering. Specifically, an effective information matrix can be constructed using information filtering theory. This system employs soft arbitration to bridge human remote control commands and virtual geometric guidance. "Soft arbitration" refers to a dynamic decision-making mechanism based on probability weights. Unlike traditional logic gate switching, "soft arbitration" does not force the selection of a single control source. Instead, it utilizes intent factors to adjust the confidence weights of each control component in real time, achieving seamless integration of human and machine commands.

[0222] It should be noted that, within the information filtering framework, human teleoperation commands (representing the operator's subjective motivation) and virtual geometric guidance (representing prior constraints of the task) are uniformly modeled as "observational inputs" to the system state. Both have equivalent mathematical descriptive forms within the information space, thus presenting a parallel relationship and jointly determining the magnitude of the correction to the predicted state.

[0223] In this embodiment of the invention, the effective information matrix can be the inverse of the state covariance matrix. This effective information matrix can be used to transform "uncertainty" into "information" for calculation. By superimposing and mapping the human instruction information matrix with the geometric guidance information matrix, the system can automatically suppress human physiological tremors under high-precision requirements (where guidance information accounts for a large proportion), and maintain operational flexibility during free movement (where human instruction information accounts for a large proportion), thereby achieving optimal control allocation at the mathematical level.

[0224] Therefore, the "soft arbitration" mentioned in this case refers to a dynamic decision-making mechanism based on probability weights. Unlike traditional logic gate switching, it does not forcibly select a single control source, but instead uses intent factors to adjust the confidence weights of each control component in real time, achieving seamless integration of human and machine commands. Specifically, within the information filtering framework, human teleoperation commands (representing the operator's subjective motivation) and virtual geometric guidance (representing prior task constraints) are uniformly modeled as "observed inputs" to the system state.

[0225] Physical regularization of the effective information matrix after collision detection can improve the system's performance during the precision assembly stage. The variance collapse of the subspace orthogonal to the task manifold occurs, which suppresses unintended transient biases by imparting virtual inertia to the system, and outputs the final regularized control command. The so-called physical collapse of subspace variance can be understood as: during the precision assembly stage ( Due to process noise and observation noise The anisotropic scheduling allows the system to achieve variance collapse within a subspace orthogonal to the task manifold. Physically, this process manifests as anisotropic dissipation of 8-12Hz physiological tremor noise, resulting in improved state estimation. Forced convergence to a lower-dimensional task manifold.

[0226] In this embodiment of the invention, the precision assembly stage can be a task requiring sub-millimeter precision with extremely small tolerance margins (such as the insertion process of shaft-hole assembly). In this stage, the system exhibits characteristics such as enhanced spatial constraints, high error sensitivity, and focused intent. Enhanced spatial constraints mean that the degrees of freedom allowed for movement are highly compressed; high error sensitivity means that even small positional or orientation deviations can lead to assembly failure or excessive contact force; and focused intent means that the operator's target is clearly defined, at which point any movement deviating from the task manifold is highly likely to be noise rather than a valid intent.

[0227] Among them, unintentional transient deviations (i.e., physiological tremors) are manifested in the signal spectrum as high-frequency components superimposed on low-frequency intentional signals (usually in the range of 8-12Hz), and in the spatial trajectory as subtle jitters or glitches perpendicular to the direction of the task manifold.

[0228] like Figures 2B-4 As shown, according to an embodiment of the present invention, in operation S224, based on a preset adaptive Kalman filter, physical regularization processing is performed on the process noise covariance matrix and the effective information matrix after preprocessing by the collision detector to extract the low-entropy expert trajectory as a regularization control command, including:

[0229] In the prediction stage of the preset adaptive Kalman filter, the prior error covariance matrix is ​​generated by driving the process noise covariance matrix.

[0230] In the gain phase of the preset adaptive Kalman filter, low-entropy expert trajectories are extracted based on the prior error covariance matrix and the observation noise covariance matrix.

[0231] For the generation and implementation of physical regularization data, a book-based adaptive control mechanism can be used to generate high-quality action execution data in real time during human-computer interaction. Firstly, in each sampling period... Inside, the system acquires the original teleoperation command. Based on the aforementioned real-time full-order process noise covariance matrix and the aforementioned effective information matrix The system performs recursive state updates and state estimations using the prediction and correction equations of a pre-defined adaptive Kalman filter, and outputs the final regularized control command. .

[0232] like Figures 2B-4 As shown, according to an embodiment of the present invention, in the gain stage of a preset adaptive Kalman filter, the extraction of low-entropy expert trajectories based on the prior error covariance matrix and the observation noise covariance matrix includes:

[0233] The Kalman gain for the gain stage is generated by using the prior error covariance matrix and the observation noise covariance matrix.

[0234] The system state vector generated based on the process noise covariance matrix and the system state estimation vector generated by the Kalman gain are used to complete the physical regularization process and obtain a smooth state sequence as a low-entropy expert trajectory.

[0235] Process noise is analyzed during the prediction phase in this embodiment of the invention. The introduction of this method, through the full-order process noise covariance matrix Real-time driving prior error covariance matrix :

[0236] (twenty one)

[0237] Furthermore, in the gain stage of this embodiment of the invention, the prior error covariance matrix is ​​used... and observation noise The weight allocation determines the Kalman gain of the preset adaptive Kalman filter. satisfy:

[0238] (twenty two)

[0239] Among them, based on the aforementioned system state vector The system state vector estimation vector corresponding to the Kalman gain in the above embodiments of the present invention satisfies:

[0240] (twenty three)

[0241] in, This is one of the smoothed state sequences after physical regularization, 1≤ ≤ and It is a positive integer.

[0242] The smoothed state sequence after the above physical regularization process The low-entropy expert trajectory is extracted to generate regularized control commands. Save the data. The robotic arm can then calculate its joint motion parameters, such as position and velocity, based on this low-entropy expert trajectory. Therefore, this trajectory not only eliminates the random noise of human operation but also strictly adheres to the Lie group manifold geometric constraints defined earlier.

[0243] In this embodiment of the invention, the parameter 'a', composed of the collision detector (i.e., the direction consistency term), velocity term, distance term, and direction term, is used to control the parameters of the adaptive Kalman filter. The state is updated through two steps: the Kalman filter prediction step and the update step, which are the "regularized" joint angle control commands, i.e., regularized control commands. .

[0244] Furthermore, the generated low-entropy expert trajectories are used as label data to construct a training dataset for training downstream generative imitation learning models, such as Diffusion Policy or Action Chunking with Transformers (ACT), ensuring that the model learns physically consistent and task-relevant optimal decision features.

[0245] To address the aforementioned technical problems, the present invention discloses an execution optimization method for robot teleoperation based on intent-driven and manifold constraints. This method receives a real-time changing continuous multidimensional intent factor α∈ as input and uses it as the core driving signal to construct an adaptive Kalman filter (AKF).

[0246] Specifically, firstly, anisotropic scheduling of process noise is implemented: An anisotropic variance matrix is ​​dynamically constructed within the task space (Lie group tangent space se(3)) using the input intention factor α, and then "pulled back" to the robot's joint velocity space using the Jacobian pseudo-inverse matrix, thereby generating an adaptive process noise matrix Q(α). This allows high-level task geometric constraints to be directly injected into the physical model of the low-level filter. The purpose of this anisotropic scheduling process is to transform high-level task geometric constraints, through the intention factor α, into physical parameters that the low-level controller can understand and execute.

[0247] Subsequently, a semantic arbitration module based on information filtering was developed. This module primarily models human teleoperation commands and virtual geometric guidance as a unified information source and utilizes an information filtering framework (a dual form of Kalman filtering) for "soft arbitration." During this process, a conflict detector is introduced. When a conflict between human and machine commands is detected, the system can adaptively reduce the weight of the virtual guidance, seamlessly and smoothly returning control to the operator to prevent trajectory locking.

[0248] Finally, physically regularized data generation is implemented: by integrating the above mechanisms into a complete adaptive Kalman filter loop, through recursive prediction and updating, the final output is a physically regularized state estimation sequence. This output trajectory is not only smooth and jitter-free, but its motion also strictly follows the anisotropic task manifold constraints defined by the intention α, ensuring the physical consistency and high quality of the data.

[0249] Based on the above-described execution optimization method for robot teleoperation, this invention also provides an execution optimization device for robot teleoperation. The following will be combined with... Figure 5B The device is described in detail.

[0250] Figure 5B The diagram illustrates the structure of an execution optimization device 502 for robot teleoperation according to an embodiment of the present invention.

[0251] like Figure 5B As shown, the execution optimization device 502 for robot teleoperation in this embodiment includes an intent generation module 521, a process noise matrix generation module 522, an initial information matrix generation module 523, and a trajectory instruction extraction module 524.

[0252] The intent generation module 521 is used to generate continuous multi-dimensional intent factors based on the teleoperation data of the current target action of the received operator. In one embodiment, it can be used to perform the operation S221 described above, which will not be repeated here.

[0253] The process noise matrix generation module 522 is used to generate a process noise covariance matrix corresponding to the teleoperation data based on continuous multidimensional intent factors. In one embodiment, it can be used to perform the operation S222 described above, which will not be repeated here.

[0254] The initial information matrix generation module 523 is used to generate an initial information matrix corresponding to the observation noise covariance matrix based on continuous multidimensional intention factors. In one embodiment, it can be used to perform the operation S223 described above, which will not be repeated here.

[0255] The trajectory instruction extraction module 524 is used to perform physical regularization processing on the process noise covariance matrix and the effective information matrix after preprocessing by the collision detector based on a preset adaptive Kalman filter, and extract low-entropy expert trajectories as regularized control instructions for remote operation execution optimization. In one embodiment, it can be used to execute the operation S224 described above, which will not be repeated here.

[0256] According to embodiments of the present invention, any plurality of modules among the intent generation module 521, process noise matrix generation module 522, initial information matrix generation module 523, and trajectory instruction extraction module 524 can be combined into one module, or any one of these modules can be split into multiple modules. Alternatively, at least part of the functionality of one or more of these modules can be combined with at least part of the functionality of other modules and implemented in one module. According to embodiments of the present invention, at least one of the intent generation module 521, process noise matrix generation module 522, initial information matrix generation module 523, and trajectory instruction extraction module 524 can be at least partially implemented as hardware circuitry, such as a field-programmable gate array (FPGA), a programmable logic array (PLA), a system-on-a-chip, a system-on-a-substrate, a system-on-package, an application-specific integrated circuit (ASIC), or any other reasonable means of integrating or packaging the circuitry, or implemented in hardware or firmware, or in any one of software, hardware, and firmware implementations, or in a suitable combination of any of these. Alternatively, at least one of the intent generation module 521, process noise matrix generation module 522, initial information matrix generation module 523, and trajectory instruction extraction module 524 can be at least partially implemented as a computer program module, which can perform corresponding functions when the computer program module is run.

[0257] To further illustrate the data processing method and / or execution optimization method for robot teleoperation described in the embodiments of the present invention, and to enable those skilled in the art to fully understand the innovation of the methods described in the embodiments of the present invention, in conjunction with the aforementioned embodiments 1 and 2... Figure 3 and Figure 4 Further embodiment 3 is provided below:

[0258] like Figure 3 As shown, Embodiment 3 of the present invention uses a low-cost humanoid robot performing a precision assembly task with a gap of 0.3 mm as an example for illustration. The VIST framework designed for this system mainly consists of a hardware layer, a perception layer, an intent reasoning layer, and an adaptive control layer.

[0259] In terms of hardware configuration, the system can use a low-cost joint encoder exoskeleton as the master input device, and the slave actuator is a LinkerArm-A7 robotic arm with 7 degrees of freedom, and the end effector does not have any force sensors installed.

[0260] In terms of control system parameter settings, the system servo frequency is set to 80Hz. An adaptive Kalman filter (AKF) is established as the system state vector within joint space. This vector is used to accurately describe the position and velocity information of a 7-DOF robotic arm.

[0261] In the perception layer implementation, the target pose is obtained by combining an Intel RealSense D435i camera with ArUco markers. To eliminate abrupt noise during the perception process, the system smooths the visual data using 50-frame mean filtering, thereby providing a stable geometric reference for subsequent control.

[0262] Intent Factor As the core semantic bridge of the system, it is used to quantify the confidence level of the system state converging to a low-dimensional task manifold. Its specific mathematical implementation logic includes the calculation of geometric, dynamic, and trend terms, as well as the final nonlinear fusion.

[0263] Geometric term of intention factor Using the Lie algebra-logarithmic mapping, the pose error spinor between the current end effector pose and the target pose is calculated as follows:

[0264] (twenty four)

[0265] Furthermore, the spatial proximity can be calculated using the exponential mapping function:

[0266] (25)

[0267] in, This is a preset task tolerance weight matrix used to adjust the system's sensitivity to geometric deviations.

[0268] Dynamics of the intention factor Based on Fitts' Law, the velocity spinor norm is defined as follows:

[0269] (26)

[0270] The velocity spinor norm is used to assess the operator's motion state.

[0271] The damping factor is calculated using the following formula:

[0272] (27)

[0273] Therefore, when the robotic arm approaches the target and actively decelerates, the factor approaches 1, thereby enabling the system to perceive the virtual contact state.

[0274] Trend term of intention factor The following is obtained by calculating the cosine similarity between the velocity screw and the error screw:

[0275] (28)

[0276] Among them, the trend term of the intention factor This is used to determine whether the operator's real-time intention is to actively converge to the task manifold or to attempt to withdraw from the task area, providing a basis for system permission allocation.

[0277] The final intention factor fusion strategy adopts a non-linear formula:

[0278] (29)

[0279] in, The sigmoid function is used. This multiplicative fusion mechanism effectively prevents the system from misinterpreting the operator's hesitation as alignment intent, thus enhancing the robustness of inference.

[0280] The core of Embodiment 3 of this invention lies in the anisotropic covariance pullback mapping (process noise Q-scheduling), which aims to project the physical constraints of the task space onto the joint space. Firstly, in the task slicing space... Define arbitrary graph factors Anisotropic variance of evolution:

[0281] (30)

[0282] in For the variance of free movement, This represents low variance in the precise constraint stage. Specifically, the variance of free motion... It can be defined as Precisely constrained variance It can be .

[0283] To handle the robot's kinematic singularities, the system employs damped least squares (DLS) to calculate the Jacobian pseudo-inverse matrix:

[0284] (31)

[0285] in, The damping factor is set to . .

[0286] The mapping formula is pulled back using covariance:

[0287] (32)

[0288] Project the geometric constraints of the task space onto the joint velocity space.

[0289] Therefore, the system constructs the full-order process noise covariance matrix through kinematic integration. When the intention factor At this time, the system undergoes subspace variance collapse in a direction orthogonal to the task manifold. This mechanism endows the system with virtual inertia, dissipating the high-frequency physiological tremors generated by the operator into viscous feedback, enabling the robotic arm to exhibit extremely high physical stability in the constraint direction.

[0290] To achieve a smooth transition of control permissions, the system constructs an effective information matrix:

[0291] (33)

[0292] Among them, a collision detector is used. The deviation of the quantified operator's speed from the manifold tangent is used to automatically decay the virtual information matrix when non-task intentions are detected. The weights are assigned to ensure that operators have the authority to override physical constraints.

[0293] Based on the experimental verification results of Embodiment 3 above, in precision assembly tasks with a gap of 0.3 mm, the success rate of the VIST method (Ours) of this invention was increased from approximately 33% of the original solution to 95%, demonstrating a success rate exceeding expectations. Regarding motion smoothness, VIST significantly reduced the average Jerk index of the end effector and did not exhibit the boundary steps or high-frequency oscillations commonly seen in traditional solutions.

[0294]

[0295] Table 1

[0296] Table 1 shows the quantitative analysis results of task success rate and task time. The task completion time is started from when the end effector of the robotic arm reaches a horizontal distance of 25cm from the task target and stops when the task is completed and the shaft is inserted.

[0297]

[0298] Table 2

[0299]

[0300] Table 3

[0301] Table 2 presents the quantitative analysis results of RMS Jerk, and Table 3 presents the quantitative results of principal component analysis. Regarding the evaluation of teaching data quality, principal component analysis (PCA) energy analysis shows that the energy proportion of the trajectory processed by VIST in the task principal axis direction increased to 88.5%. This demonstrates that VIST can effectively solve the causal confusion problem in data acquisition, reducing the variance in non-task directions through anisotropy, and providing high-quality physically consistent data support for subsequent diffusion model training.

[0302] Figure 6 A block diagram of an electronic device suitable for implementing the above-described method according to an embodiment of the present invention is shown schematically.

[0303] The electronic device provided in the embodiments of the present invention includes one or more processors and a memory, wherein the memory is used to store one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors perform the above-described method.

[0304] like Figure 6 As shown, an electronic device 600 according to an embodiment of the present invention includes a processor 601, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 602 or a program loaded from a storage portion 608 into a random access memory (RAM) 603. The processor 601 may include, for example, a general-purpose microprocessor (e.g., a CPU), an instruction set processor and / or an associated chipset and / or a special-purpose microprocessor (e.g., an application-specific integrated circuit (ASIC)), etc. The processor 601 may also include onboard memory for caching purposes. The processor 601 may include a single processing unit or multiple processing units for performing different actions of the method flow according to an embodiment of the present invention.

[0305] RAM 603 stores various programs and data required for the operation of electronic device 600. Processor 601, ROM 602, and RAM 603 are interconnected via bus 604. Processor 601 executes various operations of the method flow according to embodiments of the present invention by executing programs in ROM 602 and / or RAM 603. It should be noted that the programs may also be stored in one or more memories other than ROM 602 and RAM 603. Processor 601 may also execute various operations of the method flow according to embodiments of the present invention by executing programs stored in said one or more memories.

[0306] According to an embodiment of the present invention, the electronic device 600 may further include an input / output (I / O) interface 605, which is also connected to a bus 604. The electronic device 600 may also include one or more of the following components connected to the I / O interface 605: an input section 606 including a keyboard, mouse, etc.; an output section 607 including a cathode ray tube (CRT), liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 608 including a hard disk, etc.; and a communication section 609 including a network interface card such as a LAN card, modem, etc. The communication section 609 performs communication processing via a network such as the Internet. A drive 610 is also connected to the I / O interface 605 as needed. A removable medium 611, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., is installed on the drive 610 as needed so that computer programs read from it can be installed into the storage section 608 as needed.

[0307] The present invention also provides a computer-readable storage medium having executable instructions stored thereon, which, when executed by a processor, cause the processor to perform the above-described method.

[0308] The computer-readable storage medium may be included in the device / apparatus / system described in the above embodiments; or it may exist independently and not assembled into the device / apparatus / system. The computer-readable storage medium carries one or more programs, which, when executed, implement the method according to the embodiments of the present invention.

[0309] According to embodiments of the present invention, a computer-readable storage medium may be a non-volatile computer-readable storage medium, such as including, but not limited to: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In the present invention, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. For example, according to embodiments of the present invention, a computer-readable storage medium may include ROM 602 and / or RAM 603 and / or one or more memories other than ROM 602 and RAM 603 described above.

[0310] Embodiments of the present invention also include a computer program product comprising a computer program that, when executed by a processor, implements the above-described method.

[0311] The computer program includes program code for performing the methods shown in the flowchart. When the computer program product is run on a computer system, the program code is used to enable the computer system to implement the methods provided in the embodiments of the present invention.

[0312] When the computer program is executed by the processor 601, it performs the functions defined in the system / apparatus of this invention. According to embodiments of the invention, the systems, apparatuses, modules, units, etc., described above can be implemented by computer program modules.

[0313] In one embodiment, the computer program may rely on a tangible storage medium such as an optical storage device or a magnetic storage device. In another embodiment, the computer program may also be transmitted and distributed in the form of signals over a network medium, and downloaded and installed via the communication section 609, and / or installed from the removable medium 611. The program code contained in the computer program can be transmitted using any suitable network medium, including but not limited to: wireless, wired, etc., or any suitable combination thereof.

[0314] In such an embodiment, the computer program can be downloaded and installed from a network via the communication section 609, and / or installed from the removable medium 611. When the computer program is executed by the processor 601, it performs the functions defined in the system of this embodiment of the invention. According to embodiments of the invention, the systems, devices, apparatuses, modules, units, etc., described above can be implemented by computer program modules.

[0315] According to embodiments of the present invention, program code for executing the computer programs provided in the embodiments of the present invention can be written in any combination of one or more programming languages. Specifically, these computational programs can be implemented using high-level procedural and / or object-oriented programming languages, and / or assembly / machine languages. Programming languages ​​include, but are not limited to, languages ​​such as Java, C++, Python, "C", or similar programming languages. The program code can be executed entirely on the user's computing device, partially on the user's device, partially on a remote computing device, or entirely on a remote computing device or server. In cases involving remote computing devices, the remote computing device can be connected to the user's computing device via any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computing device (e.g., via the Internet using an Internet service provider).

[0316] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram or flowchart, and combinations of blocks in a block diagram or flowchart, may be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0317] Furthermore, all actions involving the acquisition of information, signals, or data in this invention are carried out in compliance with the relevant data protection laws, regulations, and policies of the country where the invention is located, and with the authorization granted by the owner of the corresponding device.

[0318] Those skilled in the art will understand that the features described in the various embodiments and / or claims of the present invention can be combined or combined in various ways, even if such combinations or combinations are not explicitly described in the present invention. In particular, the features described in the various embodiments and / or claims of the present invention can be combined or combined in various ways without departing from the spirit and teachings of the present invention. All such combinations and / or combinations fall within the scope of the present invention.

[0319] The embodiments of the present invention have been described above. However, these embodiments are merely illustrative and not intended to limit the scope of the invention. Although various embodiments have been described above, this does not mean that the measures in the various embodiments cannot be used advantageously in combination. The scope of the invention is defined by the appended claims and their equivalents. Various substitutions and modifications can be made by those skilled in the art without departing from the scope of the invention, and all such substitutions and modifications should fall within the scope of the invention.

Claims

1. A data processing method for robot teleoperation, characterized in that, include: In response to teleoperation data of the operator's current target action, the robot's end effector is controlled to perform a target space action that matches the current target action; During the execution of the target space action, global pose data and joint motion data of the target space action in the global coordinate system are acquired; Generate a geometric consistency factor, a dynamic damping factor, and an active gating factor based on the global pose data and joint motion data; as well as Based on the nonlinear fusion of the geometric consistency factor, dynamic damping factor, and active gating factor, a continuous multidimensional intent factor is generated to optimize the execution process of the target space action.

2. The method according to claim 1, characterized in that, During the execution of the target space action, acquiring the global pose data and joint motion data of the target space action in the global coordinate system includes: The current position and current posture data of the end effector in the global coordinate system of the target space action are obtained by a preset sensing device and used as the global pose data. The instantaneous velocity data of each joint of the end effector is obtained through the underlying sensors or joint encoders of the end effector as the joint motion data.

3. The method according to claim 1, characterized in that, The generation of the geometric consistency factor, dynamic damping factor, and active gating factor based on the global pose data and joint motion data includes: The geometric consistency factor is generated based on the pose error spinor in the robot's local coordinate system corresponding to the global pose data, combined with the weight allocation matrix that matches the six degrees of freedom of the end effector.

4. The method according to claim 3, characterized in that, Before generating the geometric consistency factor by combining the pose error spinor in the robot's local coordinate system corresponding to the global pose data with the weight allocation matrix matching the six degrees of freedom of the end effector, the method further includes: The global pose data in the global coordinate system is transformed into the local coordinate system of the robot to generate the pose error spinor; The weight allocation matrix is ​​generated by normalizing the task tolerance matrix that matches the six degrees of freedom of the end effector.

5. The method according to claim 2, characterized in that, The process of generating the geometric consistency factor, dynamic damping factor, and active gating factor based on the global pose data and joint motion data further includes: According to the preset Fitts' law rule, the dynamic damping factor is generated by the velocity spinor weight norm of the robot's end effector.

6. The method according to claim 5, characterized in that, Before generating the dynamic damping factor based on the velocity screw weight norm of the robot's end effector according to the preset Fitts law rule, the method further includes: Based on the joint motion data acquired by the end effector at the current sampling time of the joint motion data, the instantaneous velocity spinor of the end effector is generated by mapping it to the Lie group space through a preset geometric Jacobian matrix. Based on the velocity weight matrices corresponding to the translational and rotational velocity components of the end effector, the six-dimensional vector of the instantaneous velocity spinor is subjected to quadratic weighting to generate the velocity spinor weight norm.

7. The method according to claim 1, characterized in that, The process of generating the geometric consistency factor, dynamic damping factor, and active gating factor based on the global pose data and joint motion data further includes: The active gating factor is generated by the cosine similarity between the pose error screw corresponding to the geometric consistency factor and the instantaneous velocity screw corresponding to the dynamic damping factor.

8. The method according to claim 1, characterized in that, The process of generating a continuous multidimensional intent factor based on the nonlinear fusion of the geometric consistency factor, dynamic damping factor, and active gating factor to optimize the execution of the target space action includes: Based on preset multiplication gating rules, the geometric consistency factor, dynamic damping factor, and active gating factor are subjected to nonlinear fusion processing through a preset logical activation function to generate the continuous multidimensional intent factor.

9. A data processing device for robot teleoperation, characterized in that, include: The motion execution module is used to control the robot's end effector to perform a target spatial motion that matches the current target motion in response to the teleoperation data of the operator's current target motion. The data acquisition module is used to acquire global pose data and joint motion data of the target space action in the global coordinate system during the execution of the target space action; The factor generation module is used to generate geometric consistency factors, dynamic damping factors, and active gating factors based on the global pose data and joint motion data. as well as The factor fusion module is used to generate continuous multidimensional intent factors based on the nonlinear fusion of the geometric consistency factor, dynamic damping factor and active gating factor, which are used to optimize the execution process of the target space action.

10. An electronic device, comprising: One or more processors; Memory, used to store one or more programs. Wherein, when the one or more programs are executed by the one or more processors, the one or more processors perform the method of any one of claims 1 to 8.

11. A computer-readable storage medium having executable instructions stored thereon, which, when executed by a processor, cause the processor to perform the method of any one of claims 1 to 8.

12. A computer program product comprising a computer program that, when executed by a processor, implements the method of any one of claims 1 to 8.

Images