Adaptive teleoperation method, computer equipment and storage medium for narrow scenes

Abstract

The present invention provides an adaptive teleoperation method, computer device, and storage medium for narrow scenarios. The method includes motion data processing, including reading data of a first motion trajectory of a human hand, filtering the data, redirecting the first motion trajectory to convert it into a second motion trajectory in a robotic arm workspace, and interpolating and smoothing the data of the second motion trajectory after performing joint motion calculation to obtain the required motion trajectory data, which is then output to the robotic arm. The method also includes solving the robotic arm joint motion problem, including calculating inverse kinematics to convert the motion trajectory in the robotic arm workspace into multiple joint configurations in the joint space, and then screening and optimizing candidate joint configurations to obtain joint angle information for ultimately controlling the robotic arm. The present invention can achieve autonomous obstacle avoidance during teleoperation.

Description

Technical Field

[0001] The present invention belongs to the field of robotics technology, and in particular relates to an adaptive teleoperation method, computer equipment and storage medium for narrow scenes. Background Art

[0002] With the rapid development of modern robotics and artificial intelligence technologies, the application prospects of various robots, including collaborative robotic arms and humanoid robots, in production and daily life are highly anticipated. Robots should not be limited to performing simple, programmed actions, such as those performed by traditional industrial robotic arms on assembly lines. They should also be capable of taking on more complex tasks, such as performing daily tasks in the home or completing difficult tasks in dangerous and unknown environments. Unlike the hardware challenges faced by robots in traditional repetitive tasks, such as mechanical and motor-assisted tasks, the current requirements for robots are focused on the automation and intelligence level of their software.

[0003] Many of the aforementioned application scenarios stem from the demand for robots to replace humans in on-site tasks. While recent progress in robot cognition and behavior has been made through integration with artificial intelligence methods, the current state of intelligent autonomous technology is far from sufficient to build a fully autonomous robotic system capable of sensing and maneuvering. Teleoperated robots, acting as human avatars on-site, have become a viable solution. In environments such as construction sites, chemical plants, contaminated areas, and space, teleoperated robots can avoid placing humans in dangerous situations, making research on robot teleoperation highly valuable.

[0004] The emergence of virtual reality (VR) devices has brought an immersive experience and excellent interactivity, features crucial for robotic teleoperation. VR-based robotic teleoperation systems primarily consist of two components: a VR head-mounted display (HMD), which provides visual and auditory feedback to the operator; and a handheld or wearable VR device, which collects signals from key points of human movement. Most VR-based teleoperation systems utilize commercial VR headsets, which offer stable data acquisition, robust underlying implementation, and ease of use.

[0005] However, existing teleoperation technologies have the following obvious problems or defects:

[0006] 1. Collision risk: remote operation is prone to collisions when faced with a narrow environment. In robot teleoperation, since the arm structure of the teleoperated robot is different from that of the operator, there is no way to accurately reflect the movements of the human body, which brings difficulties to actual teleoperation. This is particularly prominent when faced with a narrow environment. In a narrow environment, the flexibility of the human body is inconsistent with that of the robot, which can easily cause the robot to collide with obstacles in the environment such as walls and cabinets during movement, making the operation task impossible to complete. In particular, during teleoperation, since the human body does not have an accurate obstacle reference, it is more difficult to control the robotic arm in a relatively narrow space.

[0007] ② The diversity of robot structures poses challenges to teleoperation. The field of robotics is developing rapidly, and the resulting robots have diverse configurations. Migrating algorithms for teleoperation across different robots is difficult. Furthermore, many existing teleoperation manipulators are specially designed, often sacrificing flexibility to improve teleoperation performance. This reduces adaptability and operational freedom compared to traditional manipulators.

[0008] ③ During teleoperation, selecting the correct robot configuration is a challenging task. Many currently used robots utilize arms with redundant degrees of freedom. While this improves flexibility, it also presents control challenges. In teleoperation, this issue results in multiple solutions for the same end-effector pose, some of which can affect subsequent operations, such as approaching singular poses and making obstacle avoidance difficult.

[0009] In summary, the problem of robot teleoperation in confined environments is a shared control problem within teleoperation. When motion acquisition equipment is limited, this problem becomes even more of an information-deficient shared control problem. Directly applying existing teleoperation solutions often makes it difficult to accomplish complex operations. Therefore, further research is needed to effectively address this issue. Summary of the Invention

[0010] In response to the above problems, the present invention provides an adaptive teleoperation method, a computer device and a storage medium for narrow scenarios.

[0011] The adaptive teleoperation method for narrow scenes provided by the present invention comprises the following steps:

[0012] S1. Motion data processing, including: reading data of a first motion trajectory of a human hand, filtering the data, redirecting the first motion trajectory to convert it into a second motion trajectory in the workspace of the robotic arm, and interpolating and smoothing the data of the second motion trajectory after performing joint motion settlement to obtain the required motion trajectory data, which is then output to the robotic arm;

[0013] S2. Robot arm joint motion solution, including: calculating inverse kinematics, thereby converting the motion trajectory in the robot arm workspace into multiple sets of joint configurations in the joint space, and then screening and optimizing the candidate joint configurations to obtain the joint angle information for ultimately controlling the robot arm, wherein a model predictive path integral control algorithm is used to simultaneously process multiple motion trajectories.

[0014] further,

[0015] The step S1 comprises the steps of:

[0016] A tracker is used as a human motion signal collector, and the tracker cooperates with a corresponding base station to read the current coordinates of the tracker;

[0017] The trajectory of the hand is recorded as The data of the hand tracking by the tracker is processed to obtain the collected hand movement trajectory. ;

[0018] The hand motion trajectory in the hand workspace Perform redirection calculation and convert it into motion trajectory in the robot workspace .

[0019] further,

[0020] The step S2 comprises the steps of:

[0021] The motion trajectory of the robot in the workspace A batch of inverse kinematics solutions are obtained through processing and calculation, which are recorded as ,in represents a set of joint angles, corresponding to a configuration of the robotic arm, B To process the motion trajectory of the robot in the workspace The total number of batches, N is the number of inverse kinematic solutions generated,

[0022] In the inverse kinematics solution Determine the two moments before and after t -1 and t The angle difference between different groups of joint configurations, t is a positive integer, record the time t No. i Group joint configuration and timing t -1st j The angular difference between the group joint configurations is , denoted as ndof degrees of freedom of the manipulator, considering that each set of joints is configured as an array of ndof dimensions, then It is also an array of ndof dimensions, where Wei is , then At the moment t， The said j Group joint configuration and i The sum of the joint angle differences between the group joint configurations:

[0023] ,

[0024] And the said j Group joint configuration and i Maximum joint angle difference between group joint configurations:

[0025] ,

[0026] In the above formula, the right side of the equal sign represents the array from the ndof dimension Select The maximum value of , where max|·| is the maximum value function.

[0027] further,

[0028] The step S2 further comprises the steps of:

[0029] Apply greedy algorithm to sample the path Afterwards, multiple paths are sampled by selecting different initial joint angle configurations to obtain , where M is the number of sampled paths and is a positive integer, specifically including the steps:

[0030] First, all the initial joint angle configurations are evaluated to obtain the value of each initial joint angle configuration relative to the current joint angle of the manipulator. The set of all inverse kinematic solutions of , including: the sum of joint angle differences, i.e. Moment , and the difference between the maximum joint angle is Moment :

[0031] ,

[0032] ,

[0033] Afterwards, from the collection Extract or select the initial joint angle , that is, the set of joint configurations of M groups at the initial moment, with Represents the mth group at the first moment, i.e. The joint configuration at the moment, m is an integer, The expression is:

[0034]

[0035] The above formula represents the set Select M elements from Indicates that the selected element is in the collection , where argmin is the argmin function and ‌ refers to the value of the variable when the objective function reaches its minimum. Specifically, ‌argmin f(x)‌ represents the value of the variable x when the function f(x) reaches its minimum.

[0036] Then greedy sampling calculation is performed, and the full set selected each time is the solution set at time t solved by calculating inverse kinematics:

[0037] ,

[0038] in, Refers to the moment The index value corresponding to the inverse kinematics solution selected in the mth trajectory obtained by sampling calculation,

[0039] Gradually sample all trajectories at all times After that, the initial joint angle is extracted By splicing in chronological order, we can obtain M motion trajectories with t joint configurations at a time, that is, .

[0040] further,

[0041] The step S2 further comprises the steps of:

[0042] The model predictive path integral control algorithm is used to Each track in Optimize separately: First, according to the path Calculate the approximate joint angular velocity based on the motion acquisition interval , and then the approximate joint angular velocity The cost function and the robot system model are input into the model prediction path integral control algorithm to obtain the optimal system speed , the optimal system speed Integrating over time yields Right now , and then The M trajectories obtained after optimization are evaluated separately, and the one with the best effect is selected.

[0043] in,

[0044] The approximate joint angular velocity Initial control value for the model predictive path integral control algorithm , the noise defined in the model prediction path integral control algorithm Applying it to obtain the system control quantity of the model predictive path integral control algorithm

[0045] ,

[0046] In the above formula, is the covariance matrix of the given system noise, Represents 0 as the mean, is the normally distributed noise with variance, R Represents the number of noise samples;

[0047] The joint angle of the robot arm is X , then the model equation of the robotic arm system is:

[0048] ,

[0049] in, Indicates the system control time interval;

[0050] By adding noise, it is equivalent to changing the control quantity of the system Emitted R Different control quantities , and then through the robotic arm system model we get R Different system trajectories ,

[0051] The cost function includes: process cost and terminal cost, the process cost and terminal cost are for the In terms of any one trajectory.

[0052] Furthermore, the process cost at time t

[0053] ,

[0054] in,

[0055] ,

[0056] ,

[0057] is a constraint item, and the weight W applied to the constraint item ranges from 6000 to 14000; , , is a Boolean value, and represents whether the robot arm collides with the environment, the robot arm collides with itself, and exceeds the joint limit, respectively. If the above situation occurs, the corresponding Boolean value is true, otherwise it is false; is an error term used to ensure that the robot arm tracks the operator's hand;

[0058] The forward kinematics of the manipulator, used to calculate the position and pose of the end effector under a set of joint configurations; is the coordinate of the point in the trajectory obtained by tracking the human body and redirecting it at time t,

[0059] The terminal cost

[0060] ,

[0061] In the above formula, Indicates the The joint configuration at time t for any trajectory in .

[0062] further,

[0063] Considering the noise term , the calculation formula is:

[0064]

[0065] In the above formula, T represents the matrix transpose, for The inverse matrix of represents the noise applied to the calculated trajectory,

[0066] Then for the For any trajectory, add the noise term The trajectory cost

[0067] ,

[0068] Further, we obtain the The weight corresponding to any trajectory in:

[0069] and

[0070] , ,

[0071] in, It is a hyperparameter in the model prediction path integral control algorithm, which is used to adjust the noise control performance of the model prediction path integral control algorithm. represent Middle The trajectory cost value of the trajectory, Representative In each track The minimum value of

[0072] Further, calculate the Among the M trajectories obtained after optimization, m Optimized control volume for each trajectory

[0073] ,

[0074] Then the optimized trajectory is obtained through the robot system model equation , in the above formula For the Middle r Group control quantity,

[0075] At this point, the M trajectories are optimized separately to obtain the optimized trajectory set .

[0076] further,

[0077] For the trajectory set Evaluate each trajectory in the :

[0078] ,

[0079] in

[0080] ,

[0081] ;

[0082] The weight W is set to 10,000.

[0083] The present invention also provides a computer device, which includes a memory, a first processor, and a first computer program stored in the memory and executable on the first processor. When the first computer program is executed by the first processor, the above-mentioned adaptive teleoperation method for narrow scenarios is implemented.

[0084] The present invention also provides a computer-readable storage medium for storing a second computer program. The second computer program can be executed by at least one second processor to enable the at least one second processor to perform the above-mentioned adaptive teleoperation method for narrow scenarios.

[0085] The adaptive teleoperation method for narrow scenarios provided by the present invention solves possible joint configurations by calculating inverse kinematics during teleoperation, and introduces a cost function through the MPPI method to optimize the joint configuration, thereby obtaining a smooth robotic arm trajectory that can avoid obstacles. Simple equipment can be used to smoothly teleoperate the robotic arm, achieving autonomous obstacle avoidance during teleoperation, and real-time teleoperation of the robotic arm in narrow and confined environments can be performed more conveniently to efficiently complete the operation task.

[0086] Other features and advantages of the present invention will be described in the following description, and in part will become apparent from the description, or will be understood by practicing the present invention. The purpose and other advantages of the present invention can be realized and obtained by the structures pointed out in the description, claims and drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0087] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0088] Figure 1 A flow chart of an adaptive teleoperation method for narrow scenarios according to an embodiment of the present invention is shown. DETAILED DESCRIPTION

[0089] To make the objectives, technical solutions, and advantages of the embodiments of the present invention more clear, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts shall fall within the scope of protection of the present invention.

[0090] Unless otherwise defined, all technical and scientific terms used herein have the same meanings as commonly understood by those skilled in the art to which this application belongs. The terms used in the specification and description of the application are for the purpose of describing specific embodiments only and are not intended to limit this application. The terms "including" and "having," as well as any variations thereof, in the specification and claims of this application and the accompanying drawings are intended to cover non-exclusive inclusions. The terms "first," "second," "third," etc., in the specification and claims of this application and the accompanying drawings are used to distinguish different objects, not to describe a particular order or a primary-secondary relationship. The term "plurality" as used in this application refers to two or more (including two).

[0091] References herein to "embodiments" mean that a particular feature, structure, or characteristic described in connection with the embodiments may be included in at least one embodiment of the present application. The appearance of this phrase in various places in the specification does not necessarily refer to the same embodiment, nor does it constitute an independent or alternative embodiment that is mutually exclusive of other embodiments. It is understood, both explicitly and implicitly, by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0092] In mission scenarios, when teleoperating a robotic arm, there are obstacles between the robotic arm and the object or target location that need to be avoided. For example, if a robotic arm needs to retrieve an object from deep within a drawer, it must open the drawer and then extend the end effector into the drawer to retrieve the object. The environment inside the drawer is quite confined for the robotic arm.

[0093] During teleoperation, the operator and the robotic arm are structurally different, and the operator lacks a realistic reference to environmental factors. This makes smooth control of the robotic arm's movements challenging. To address this, the robotic arm's control method needs to provide certain obstacle avoidance capabilities. To achieve this, the robotic arm must rapidly calculate collision-avoiding joint angle configurations based on the teleoperator's terminal trajectory during operation.

[0094] Figure 1 The flowchart of the adaptive teleoperation method for narrow scenes provided by the present invention includes the following steps.

[0095] S1. Motion data processing.

[0096] Motion data processing first uses a VR tracker device to read the data of the first motion trajectory of the human hand. After basic filtering of the data, the first motion trajectory is redirected and converted into a second motion trajectory in the robotic arm workspace. After the joint motion is settled, the data of the second motion trajectory is interpolated and smoothed to obtain the required motion trajectory data and then output it to the robotic arm.

[0097] Specifically, the present invention uses a tracker, such as a VR tracker, as a human motion signal collector. This tracker, in conjunction with a corresponding base station, can read the VR tracker's current coordinates. During use, the VR tracker can be attached to the back of a worker's hand or a corresponding handheld device as needed.

[0098] Then, the motion trajectory of the hand is defined as The data tracked by the VR tracker can be obtained through the relevant Python interface of SteamVR. After basic filtering and other data processing, the collected human hand motion trajectory is obtained as follows: The purpose of data processing is to eliminate the effects of the tracker's own noise and slight hand shaking on teleoperation. The interpolation and smoothing of data after joint motion calculation is to provide the robot arm with a smoother trajectory, making the robot arm move more smoothly.

[0099] In addition, since the working space of the robot is different from that of the human body, the Perform redirection calculation and convert it into motion trajectory in the robot workspace The essential goal of redirection is to fit the motion coordinate system of the human hand to the coordinate system of the robotic arm. In this invention, the coordinates of the human hand are obtained by the VR tracker. Therefore, the coordinate system of the robotic arm can be converted to the VR tracker, and then the coordinates of the human hand can be transformed to solve the redirection problem.

[0100] The present invention sets the redirection process as a part of the calibration process, and the specific method is as follows. First, the approximate size of the working space of the robotic arm is obtained through the known information of the robotic arm. The known information can be obtained from the configuration files such as the Unified Robot Description Format (URDF) of the robotic arm; then, in the calibration process, the staff assumes a fixed posture and reads the corresponding coordinates, wherein the posture of the robotic arm corresponding to the coordinates is pre-set, that is, the initial position where the robotic arm should be when the human body assumes the fixed posture. In this way, the position of the origin of the robotic arm's motion space in the tracker coordinate system can be deduced, thereby converting the robotic arm coordinate system to the VR tracker coordinate system. Further, after fitting, the redirection conversion relationship matrix can be obtained. , and then In the present invention, the fixed posture mainly constrains the arms, keeping the arms as wide as the shoulders and the forearms raised to be perpendicular to the upper arms.

[0101] S2. Robotic arm joint motion solution.

[0102] The robot arm joint motion solution first uses methods such as IKflow to calculate inverse kinematics, thereby converting the motion trajectory in the robot arm's workspace into multiple sets of joint configurations in the joint space. Finally, the model prediction path integral control (MPPI) algorithm is used to screen and optimize the candidate joint configurations to obtain the joint angle information that ultimately controls the robot arm.

[0103] Specifically, after obtaining the motion trajectory in the robot workspace, it is necessary to perform inverse kinematics solution before handing it over to the robotic arm for execution. In order to quickly solve the problem while minimizing the restrictions on the flexibility of the robotic arm, the present invention uses a parallel high-speed inverse kinematics solver IKflow, which uses a graphics processing unit (GPU) to obtain a large number of inverse kinematics solutions in a short time. A batch of inverse kinematics solutions can be obtained by processing and calculating, which are recorded as ,in Represents a set of joint angles, corresponding to a configuration of the robotic arm. The processing calculation needs to be performed in batches. B To process motion trajectories The time length of each batch, that is, the number of sampling moments (abbreviated as moments) of each batch, is a positive integer. N The number of inverse kinematic solutions generated when using IKflow, a positive integer.

[0104] The solutions generated directly using IKflow are different from other inverse kinematics solvers that use the Jacobian matrix. The continuity between the generated solutions cannot be naturally guaranteed. Therefore, the generated solution set needs to be screened. At the same time, in this process, environmental obstacles and other factors are taken into consideration to select the most suitable trajectory for execution.

[0105] The problem addressed by the present invention is how to reasonably avoid obstacles in real-time tracking. From a methodological point of view, if accurate tracking is to be maintained while avoiding obstacles, the obstacle avoidance performance of the robot arm will be greatly tested, and in most cases, such a method is completely unfeasible. Therefore, the present invention considers finding a feasible solution around the target being tracked in real time to avoid obstacles, that is, to plan the trajectory in a manner similar to sampling around the tracking point. Based on this, the screening control method selected by the present invention is MPPI. The basic principle of this method is to apply noise to the control quantity to obtain different control sequences, and gradually diverge on the state of the controlled system (i.e., the robot arm system) through path integration, thereby obtaining different trajectories. According to the set cost function, the weight of each trajectory can be calculated, and then a weighted average is performed to obtain a statistically optimal solution. Specifically, the present invention selects the control quantity as the instantaneous velocity of each joint of the robot arm, and the state quantity as the angle of each joint of the robot arm.

[0106] In this paper, due to the real-time limitations of MPPI, the torch version of MPPI is used and improved to enable it to process multiple trajectories simultaneously. Compared with the original method of sequential processing, the screening efficiency is greatly improved, thereby being able to meet the basic frequency requirements of teleoperation.

[0107] In the implementation of the method, we first need to ensure the continuity of the robot arm's motion trajectory. Determine the two moments before and after t -1 and t The angle difference between different groups of joint configurations, including the sum of the joint angle differences Difference from the maximum joint angle , t Is a positive integer. t No. i Group joint configuration and timing t -1st j The angular difference between the group joint configurations is , let ndof be the degree of freedom of the robot arm, considering that each set of joints is configured as an array of ndof dimensions, then It is also an array of ndof dimensions, where Wei is , then At the moment t， No. j Group joint configuration and i The sum of the joint angle differences between the group joint configurations:

[0108]

[0109] and j Group joint configuration and i Maximum joint angle difference between group joint configurations:

[0110] .

[0111] In the above formula, the right side of the equal sign represents the array from the ndof dimension Select The maximum value of , where max|·| is the maximum value function.

[0112] In addition, a greedy algorithm is used to sample a relatively continuous path Afterwards, in order to increase the divergence of the trajectory in the MPPI algorithm, multiple paths can be sampled by selecting different initial joint angle configurations to obtain , where M is the number of sampled paths and is a positive integer. In order to avoid the initial joint angle configuration being too bad, in this process, all initial joint angle configurations are first evaluated to obtain the relative value of each initial joint angle configuration to the current joint angle of the manipulator. The set of all inverse kinematic solutions of , including: the sum of joint angle differences (i.e. Moment ) and the maximum joint angle difference (i.e. Moment ):

[0113] ,

[0114] .

[0115] Afterwards, from the collection Extract or select the initial joint angle , that is, the set of joint configurations of M groups at the initial moment, represents the joint configuration of the mth (m is an integer) group at the first moment, The expression is:

[0116]

[0117] The above formula represents the set Select M elements from Indicates that the selected element is in the collection The index value in . argmin is the argmin function, and ‌ refers to the value of the variable when the objective function reaches its minimum value. Specifically, ‌argmin f(x)‌ represents the value of the variable x when the function f(x) reaches its minimum value.

[0118] Then, greedy sampling calculation is performed. For time t, a set is selected from the solution set of time t solved by the inverse kinematics calculated by IKflow:

[0119] ,

[0120] in, Refers to the moment The index value corresponding to the inverse kinematics solution selected in the mth trajectory obtained by sampling calculation.

[0121] Gradually sample all trajectories at all times After that, the initial joint angle is extracted By splicing them in chronological order, we can obtain M motion trajectories with t joint configurations at a time, that is, .

[0122] Next, we use the MPPI algorithm to Each track in Optimize separately: First, according to the path Calculate the approximate joint angular velocity based on the motion acquisition interval . Then approximate the joint angular velocity The cost function defined in the present invention and the manipulator system model are input into the MPPI algorithm to obtain the optimal system speed. , the optimal system speed Integrating over time yields Right now . Then The M trajectories obtained after optimization are evaluated separately, and the one with the best effect is selected.

[0123] The calculated approximate joint angular velocity The role of the MPPI algorithm is to serve as the initial control value in the algorithm , the noise defined in the MPPI algorithm Apply it on it to get the system control quantity

[0124] ,

[0125] In the above formula For a given system noise covariance matrix, it determines the size of the system noise. Represents 0 as the mean, is the normally distributed noise with variance, R Represents the number of noise samples.

[0126] The main function of the robot system model is to update the system state and define how the control quantity changes the system state. In this invention, the system state is the robot joint angle, which is defined as X , then the model equation of the robotic arm system is:

[0127] ,

[0128] in, Indicates the system control time interval.

[0129] By adding noise, it is equivalent to changing the control quantity of the system Emitted R Different control quantities , and then we get the R Different system trajectories .

[0130] Note that, similar to model predictive control (MPC), for the final optimized trajectory, only the joint configurations of the expected number of moments, such as the previous one or several moments, are taken and handed over to the robot for execution, so as to increase the continuity of the control trajectory and the robustness of the system.

[0131] The evaluation of joint trajectory smoothness and automatic obstacle avoidance is achieved by setting a cost function. By adding the collision detection results and the size of the robot arm speed to the cost function, the divergent Each trajectory is evaluated and finally optimized. .

[0132] In specific implementation, the cost function adopted by the present invention mainly includes two parts, process cost and terminal cost. The process cost mainly considers the collision of the robot arm at each moment and the error between the end of the robot and the tracking. Specifically, the process cost at time t is

[0133] ,

[0134] in,

[0135] ,

[0136] ,

[0137] In the process cost, the collision situation of the robot arm is calculated by curobo to determine whether a collision has occurred. This part, the self-collision situation of the robot arm and the situation of exceeding the joint limit are put into the constraint item. Considered in the above formula, as a hard constraint of the robot arm, this part will impose a higher weight W, and the value range of W is 6000-14000, preferably 10000. , , are Boolean values, representing whether the robot arm has encountered some unexpected situation. Specifically, , , In turn, they represent whether the robot arm collides with the environment, the robot arm collides with itself, and exceeds the joint limit. If the above situations occur, the corresponding Boolean value is true, otherwise it is false. If the robot arm collides with the environment, then If it is 1, it is true, otherwise it is 0, which is false. In addition, since the various factors considered in the process cost will try to get the minimum value after optimization, the error term Taking this into account can ensure that the error is as small as possible, so the error term It can be used to ensure that the robot arm tracks the operator's hand. For the forward kinematics of the robotic arm, it is used to calculate the position and posture of the end effector corresponding to a certain joint configuration. The coordinates of the points in the trajectory obtained by tracking the human body and redirecting it at time t are also the coordinates of the points when using an IK solver such as IKflow to solve and calculate inverse kinematics.

[0138] The terminal cost mainly considers the smoothness of the entire trajectory. In order for the actual robot arm to perform operations, the smoothness of the entire trajectory needs to be guaranteed. In this invention, the trajectory smoothness is measured using the smoothness item. To reflect or illustrate, its components include the above mentioned and , but the slight difference here is that the entire trajectory needs to be calculated, that is, the terminal cost

[0139] ,

[0140] In the above formula, represents the joint configuration at time t in the entire trajectory.

[0141] In addition, the MPPI algorithm also evaluates the noise level applied to the system and defines the result as the noise term , the calculation formula is:

[0142]

[0143] In the above formula, T represents the matrix transpose, for The inverse matrix of Represents the noise applied to the calculated trajectory.

[0144] for Add a noise term to any trajectory The trajectory cost

[0145] Further we can get the The weight corresponding to any trajectory in:

[0146] and

[0147] ,

[0148] In the above formula, It is a hyperparameter in the MPPI algorithm, used to adjust the noise control performance of the MPPI algorithm. It is set by technicians according to the working conditions. Representative Middle The trajectory cost value of the trajectory, Representative In each track The minimum value of .

[0149] Further, we can calculate Among the M trajectories obtained after optimization, m Optimized control volume for each trajectory

[0150] ,

[0151] Then the optimized trajectory is obtained through the robot system model equation , in the above formula For the Middle Group control quantity.

[0152] At this point, the M trajectories are optimized separately to obtain the optimized trajectory set Finally, each trajectory is evaluated and the optimal trajectory is selected. :

[0153] ,

[0154] in

[0155] ,

[0156] .

[0157] Finally, the joint configuration of the expected number of moments, such as the previous one or several moments, is returned. According to the above process, the steps of updating the system state to returning the joint configuration can be repeated during the robot arm teleoperation process to obtain a real-time control trajectory.

[0158] Finally, before applying it to the robotic arm, the trajectory needs to be interpolated to provide a smoother trajectory for the robotic arm.

[0159] The present invention uses IKflow to calculate inverse kinematics to solve possible joint configurations in teleoperation, and introduces a cost function through the MPPI method to optimize the joint configuration, thereby obtaining a smooth robotic arm trajectory that can avoid obstacles. This allows the present invention to use simple equipment to perform smooth teleoperation of the robotic arm, while achieving a certain obstacle avoidance effect during the teleoperation process.

[0160] The present invention also provides a computer device, which may include a memory, a first processor, and a first computer program stored in the memory and executable on the first processor. When the first computer program is executed by the first processor, the above-mentioned adaptive teleoperation method for narrow scenarios is implemented.

[0161] The present invention also provides a computer-readable storage medium for storing a second computer program. The second computer program can be executed by at least one second processor to enable the at least one second processor to perform the above-mentioned adaptive teleoperation method for narrow scenarios.

[0162] The memory may be a volatile memory, such as a random-access memory (RAM), or an external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), RAMbus direct RAM (RDRAM), direct RAM bus dynamic RAM (DRDRAM), and RAMbus dynamic RAM (RDRAM); or a non-volatile memory, such as a read-only memory (ROM), a programmable ROM (PROM), an electrically programmable ROM (EPROM), an electrically erasable programmable ROM (EEPROM), flash memory, a hard disk drive (HDD), or a solid-state drive (SSD); or a combination of the above types of memory.

[0163] The first and second processors may be at least one of an application-specific integrated circuit (ASIC), a digital signal processor (DSP), a digital signal processing device (DSPD), a programmable logic device (PLD), a field programmable gate array (FPGA), a central processing unit (CPU), a controller, a microcontroller, and a microprocessor. It is understood that for different devices, the electronic components used to implement the functions of the processors may be other, and this is not specifically limited in the embodiments of the present invention.

[0164] The adaptive teleoperation method for narrow scenarios provided by the present invention introduces an efficient robot inverse kinematics solver IKFlow and a model predictive path integral control algorithm, thereby realizing autonomous obstacle avoidance during teleoperation. It can more conveniently perform real-time teleoperation of the robotic arm in narrow and confined environments to efficiently complete the operation task.

[0165] According to an embodiment of the present invention, the method flow according to an embodiment of the present invention can be implemented as a computer software program. For example, an embodiment of the present invention includes a computer program product, which includes a computer program carried on a computer-readable storage medium, and the computer program includes program code for executing the method shown in the flowchart. According to an embodiment of the present invention, the electronic devices, devices, apparatuses, modules, units, etc. described above can be implemented by computer program modules.

[0166] The present invention also provides a computer-readable storage medium, which may be included in the device / apparatus / system described in the above embodiments, or may exist independently and not incorporated into the device / apparatus / system. The computer-readable storage medium carries one or more programs, which, when executed, implement the method according to the embodiments of the present invention.

[0167] According to an embodiment of the present invention, a computer-readable storage medium may be a non-volatile computer-readable storage medium, such as, but not limited to, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In the present invention, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device.

[0168] The flowcharts and block diagrams in the accompanying drawings illustrate the possible implementation architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present invention. In this regard, each box in the flowchart or block diagram can represent a module, program segment, or a part of code, and the above-mentioned module, program segment, or a part of code contains one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in an order different from that marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flowchart, and the combination of boxes in the block diagram or flowchart, can be implemented with a dedicated hardware-based system that performs the specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0169] Those skilled in the art will appreciate that various combinations and / or combinations of features described in the various embodiments and / or claims of the present invention may be made, even if such combinations and / or combinations are not explicitly described in the present invention. In particular, various combinations and / or combinations of features described in the various embodiments and / or claims of the present invention may be made, without departing from the spirit and teachings of the present invention. All such combinations and / or combinations fall within the scope of the present invention.

[0170] The above describes embodiments of the present invention. However, these embodiments are for illustrative purposes only and are not intended to limit the scope of the present invention. Although each embodiment has been described separately above, this does not mean that the measures in each embodiment cannot be advantageously used in combination. The scope of the present invention is defined by the appended claims and their equivalents. Without departing from the scope of the present invention, those skilled in the art may make various substitutions and modifications, which are intended to fall within the scope of the present invention.

Claims

1. Adaptive teleoperation method for narrow scenes, characterized by: Including steps: S1. Motion data processing, including: reading data of a first motion trajectory of a human hand, filtering the data, redirecting the first motion trajectory to convert it into a second motion trajectory in the workspace of the robotic arm, and interpolating and smoothing the data of the second motion trajectory after performing joint motion settlement to obtain the required motion trajectory data, which is then output to the robotic arm; S2. Robot arm joint motion solution, including: calculating inverse kinematics to transform the motion trajectory in the robot arm workspace into multiple sets of joint configurations in the joint space, and then screening and optimizing the candidate joint configurations to obtain the final joint angle information for controlling the robot arm, wherein the model predictive path integral control algorithm is used to simultaneously process multiple motion trajectories. The step S1 comprises the steps of: A tracker is used as a human motion signal collector, and the tracker cooperates with a corresponding base station to read the current coordinates of the tracker; The trajectory of the hand is recorded as The data of the hand tracking by the tracker is processed to obtain the collected hand movement trajectory. ; The hand motion trajectory in the hand workspace Perform redirection calculation and convert it into motion trajectory in the robot workspace ; The step S2 comprises the steps of: The motion trajectory of the robot in the workspace A batch of inverse kinematics solutions are obtained through processing and calculation, which are recorded as ,in represents a set of joint angles, corresponding to a configuration of the robotic arm, B To process the motion trajectory of the robot in the workspace The total number of batches, N is the number of inverse kinematic solutions generated, In the inverse kinematics solution Determine the two moments before and after t -1 and t The angle difference between different groups of joint configurations, t is a positive integer, record the time t No. i Group joint configuration and timing t -1st j The angular difference between the group joint configurations is , denoted as ndof degrees of freedom of the manipulator, considering that each set of joints is configured as an array of ndof dimensions, then It is also an array of ndof dimensions, where Wei is , then At the moment t， The said j Group joint configuration and i The sum of the joint angle differences between the group joint configurations: ， And the maximum joint angle difference between the j-th joint configuration and the i-th joint configuration: ， In the above formula, the right side of the equal sign represents the array of ndof dimensions. Select The maximum value of To obtain the maximum value function, The step S2 further includes the steps of: Apply greedy algorithm to sample the path Afterwards, multiple paths are sampled by selecting different initial joint angle configurations to obtain , where M is the number of sampled paths and is a positive integer. The specific steps include: First, all the initial joint angle configurations are evaluated to obtain the value of each initial joint angle configuration relative to the current joint angle of the manipulator. The set of all inverse kinematic solutions of , including: the sum of joint angle differences, i.e., the sum of the joint angle differences at time t=1 , and the maximum joint angle difference, that is, the time t=1 : ， ， Afterwards, from the collection Extract or select the initial joint angle , that is, the set of joint configurations of M groups at the initial moment, with represents the joint configuration of the mth group at the first moment, i.e., t=1, where m is an integer. The expression is: , The above formula represents the set Select elements, Indicates that the selected element is in the collection , where argmin is the argmin function and ‌ refers to the value of the variable when the objective function reaches its minimum. Specifically, ‌argmin f(x)‌ represents the value of the variable x when the function f(x) reaches its minimum. Then greedy sampling calculation is performed, and the full set selected each time is the solution set at time t solved by calculating inverse kinematics: , in, Refers to the first sampling calculation at time t-1 The index value corresponding to the inverse kinematics solution selected in the trajectory, Gradually sample all trajectories at all times After that, the initial joint angle is extracted By splicing in chronological order, we can obtain M motion trajectories with t joint configurations at a time, that is, , The step S2 further comprises the steps of: The model predictive path integral control algorithm is used to Each track in Optimize separately: First, according to the path Calculate the approximate joint angular velocity based on the motion acquisition interval , and then the approximate joint angular velocity The cost function and the robot system model are input into the model prediction path integral control algorithm to obtain the optimal system speed , the optimal system speed Integrating over time yields Right now , then the The M trajectories obtained after optimization are evaluated separately, and the one with the best effect is selected. in, The approximate joint angular velocity Initial control value for the model predictive path integral control algorithm , the noise defined in the model prediction path integral control algorithm Applying it to obtain the system control quantity of the model predictive path integral control algorithm , In the above formula, is the covariance matrix of the given system noise, Represents 0 as the mean, is the normally distributed noise with variance, R Represents the number of noise samples; , in, Indicates the system control time interval; By adding noise, it is equivalent to changing the control quantity of the system Emitted R Different control quantities , and then through the robotic arm system model we get R Different system trajectories , The cost function includes: process cost and terminal cost, the process cost and terminal cost are for the For any of the trajectories, The process cost at time t , in, , , is a constraint item, and the weight W applied to the constraint item ranges from 6000 to 14000; , , is a Boolean value, and represents whether the robot arm collides with the environment, the robot arm collides with itself, and exceeds the joint limit, respectively. If the above situation occurs, the corresponding Boolean value is true, otherwise it is false; is an error term used to ensure that the robot arm tracks the operator's hand; The forward kinematics of the manipulator, used to calculate the position and pose of the end effector under a set of joint configurations; is the coordinate of the point in the trajectory obtained by tracking the human body and redirecting it at time t, The terminal cost , In the above formula, Indicates the The joint configuration of any trajectory at time t, Considering the noise term , the calculation formula is: , In the above formula, T represents the matrix transpose, for The inverse matrix of represents the noise applied to the calculated trajectory, Then for the For any trajectory, add the noise term The trajectory cost , Further, we obtain the The weight corresponding to any trajectory in: , in, It is a hyperparameter in the model prediction path integral control algorithm, which is used to adjust the noise control performance of the model prediction path integral control algorithm. Representative Middle The trajectory cost value of the trajectory, Representative In each track The minimum value of Further, calculate the Among the M trajectories obtained after optimization, m Optimized control volume for each trajectory , Then the optimized trajectory is obtained through the robot system model equation , in the above formula For the Middle Group control quantity, At this point, the M trajectories are optimized separately to obtain the optimized trajectory set .

2. The adaptive teleoperation method for narrow scenes according to claim 1 is characterized in that: For the trajectory set Evaluate each trajectory in the : ， in ， ； The weight W is set to 10,000.

3. A computer device, characterized in that: The invention comprises a memory, a first processor and a first computer program stored in the memory and executable on the first processor, wherein the first computer program, when executed by the first processor, implements the adaptive teleoperation method for narrow scenes according to any one of claims 1 to 2.

4. A computer-readable storage medium, characterized in that The computer-readable storage medium is used to store a second computer program, and the second computer program can be executed by at least one second processor, so that the at least one second processor executes the adaptive teleoperation method for narrow scenes according to any one of claims 1-2.

Images