Adaptive teleoperation method for narrow scene, computer equipment and storage medium

Abstract

The invention provides an adaptive teleoperation method for a narrow scene, computer equipment and a storage medium, the adaptive teleoperation method comprises the following steps: motion data processing: reading data of a first motion track of a human hand, filtering the data, and then redirecting the first motion track to obtain motion data; the second motion trail is converted into a second motion trail in the working space of the mechanical arm, interpolation smoothing processing is carried out on data of the second motion trail after joint motion settlement is carried out, and needed motion trail data are obtained and then output to the mechanical arm; and the mechanical arm joint motion calculation comprises the steps that inverse kinematics is calculated, so that the motion trail of the mechanical arm in the working space is converted into multiple sets of joint configurations in the joint space, and then the candidate joint configurations are screened and optimized to obtain joint angle information for finally controlling the mechanical arm. According to the invention, autonomous obstacle avoidance in the teleoperation process can be realized.

Description

Technical Field

[0001] The invention belongs to the technical field of robots, and in particular relates to an adaptive teleoperation method, a computer device and a storage medium for narrow scenes. Background Art

[0002] With the rapid development of modern robotics and artificial intelligence technology, the application prospects of various robots such as collaborative robotic arms and humanoid robots in production and life are highly anticipated. Robots should not be limited to performing simple, programmed actions, such as the tasks of traditional industrial robotic arms on assembly lines. They should also be able to undertake more complex tasks, such as performing daily tasks in a home environment, or completing difficult work in dangerous and unknown environments. Different from the mechanical, motor and other hardware tests faced by robots under traditional repetitive task conditions, the current higher requirements for robots are concentrated on the automation and intelligence level of their software.

[0003] Many of the application scenarios mentioned above stem from people's demand for robots to replace humans in performing tasks on site. Although much progress has been made in robot cognition and behavior in recent years after combining with artificial intelligence methods, the current level of intelligent autonomous technology is far from enough to build a robot system with fully autonomous perception and action capabilities. Teleoperated robots acting as human avatars on site have become a reasonable solution. In environments such as construction sites, chemical plants, contaminated areas, and space, teleoperated robots can avoid putting humans in dangerous environments, so research on robot teleoperation is very valuable.

[0004] The emergence of virtual reality (VR) devices has brought immersive experience and excellent interactivity, which are crucial for robot teleoperation. The VR-based robot teleoperation system mainly consists of two parts in terms of the equipment used. One part is the VR head-mounted display (HMD), which is mainly used to provide visual and auditory feedback to the operator; the other part is the VR handheld or wearable device part, which is responsible for collecting signals from key points of human movement. Most VR-based teleoperation systems use commercial VR headsets, which have stable data acquisition, good underlying implementation, and are easy to use.

[0005] However, the existing teleoperation technology has the following obvious problems or defects: ①. Collision risk, that is, remote operation is prone to collision when facing a narrow environment. In robot teleoperation, since the arm structure of the remote-operated robot is different from that of the operator, there is no way to reflect the movements of the human body one-to-one, which brings difficulties to the actual teleoperation, especially when facing a narrow environment. In a narrow environment, the flexibility of the human body is inconsistent with that of the robot, which can easily cause the robot to collide with obstacles in the environment such as walls and cabinets during movement, making the operation task impossible to complete. In particular, during the teleoperation process, since the human body does not have an accurate obstacle reference, it is difficult to control the robotic arm in a relatively narrow space.

[0006] ② The diversity of robot structures brings challenges to teleoperation. The current field of robotics is developing rapidly, and the robot configurations that appear are different. It is difficult to migrate related algorithms when teleoperating different robots. In addition, many existing teleoperation manipulators are specially designed. In order to improve teleoperation performance, some flexibility is often sacrificed. Compared with traditional manipulators, their adaptability and operational freedom are reduced.

[0007] ③. In the process of teleoperation, it is a difficult task to reasonably select the configuration of the robot. Many robots currently in widespread use use manipulators with redundant degrees of freedom. Although this improves the flexibility of the manipulator, it also brings challenges to control. In teleoperation, this problem results in multiple solutions for the same end effector posture, but some of them will affect subsequent operations, such as approaching singular postures and difficulty in avoiding obstacles.

[0008] In summary, the problem of robot teleoperation in a narrow environment is a shared control problem in teleoperation. In the case of limited motion acquisition equipment, this is even more of an information-deficient shared control problem. Directly applying existing teleoperation solutions often makes it difficult to complete complex operations. Therefore, how to effectively solve this problem deserves further discussion. Summary of the invention

[0009] In view of the above problems, the present invention provides an adaptive teleoperation method, a computer device and a storage medium for narrow scenes.

[0010] The adaptive teleoperation method for narrow scenes provided by the present invention comprises the following steps: S1, motion data processing, including: reading data of a first motion trajectory of a human hand, filtering the data, redirecting the first motion trajectory to convert it into a second motion trajectory in the workspace of the robot arm, interpolating and smoothing the data of the second motion trajectory after performing joint motion settlement, obtaining the required motion trajectory data and then outputting it to the robot arm; S2. Robot arm joint motion solution, including: calculating inverse kinematics, thereby converting the motion trajectory in the robot arm workspace into multiple sets of joint configurations in the joint space, and then screening and optimizing the candidate joint configurations to obtain the joint angle information for ultimately controlling the robot arm, wherein a model prediction path integral control algorithm is used to simultaneously process multiple motion trajectories.

[0011] further, The step S1 comprises the steps of: A tracker is used as a human motion signal collector, and the tracker cooperates with a corresponding base station to read the current coordinates of the tracker; The motion trajectory of the hand is recorded as The data of the hand tracking by the tracker is processed to obtain the collected hand movement trajectory. ; The hand motion trajectory in the hand workspace Perform redirection calculation and convert it into motion trajectory in the robot workspace .

[0012] further, The step S2 comprises the steps of: The motion trajectory of the robot in the workspace After processing and calculation, a batch of inverse kinematics solutions are obtained, denoted as ,in represents a set of joint angles, corresponding to a configuration of the robotic arm, B To process the motion trajectory of the robot in the workspace The total number of batches, N is the number of inverse kinematic solutions generated, In the inverse kinematics solution Determine the two moments before and after t -1 and t The angle difference between different sets of joint configurations, t is a positive integer, record the time t No. i Group joint configuration and timing t -1st j The angular difference between the group joint configurations is , let ndof be the degree of freedom of the robot, considering that each set of joints is configured as an array of ndof dimensions, then It is also an array of ndof dimensions, where Wei is , then At the moment t， The said jGroup joint configuration and i The sum of the joint angle differences between the group joint configurations: ,

[0013] And the j Group joint configuration and i Maximum joint angle difference between group joint configurations: , In the above formula, the right side of the equal sign represents the array from the ndof dimension Select The maximum value of , where max|·| is the maximum value function.

[0014] further, The step S2 further comprises the steps of: Apply the greedy algorithm to sample the path Afterwards, multiple paths are sampled by selecting different initial joint angle configurations to obtain , where M is the number of sampled paths and is a positive integer, specifically including the steps: First, all the initial joint angle configurations are evaluated to obtain the relative value of each initial joint angle configuration to the current joint angle of the robot arm. The set of all inverse kinematic solutions of , including: the sum of joint angle differences, i.e. Moment , and the maximum joint angle difference is Moment : , , Afterwards, from the collection Extract or select the initial joint angle , that is, the set of joint configurations at the initial moment of the M group, Represents that the mth group is at the first moment, The joint configuration at the moment, m is an integer, The expression is:

[0015] The above formula represents the Select M elements from Indicates that the selected element is in the set , where argmin is the argmin function and ‌ refers to the value of the variable when the objective function reaches the minimum value. Specifically, ‌argmin f(x)‌ represents the value of the variable x when the function f(x) reaches the minimum value. Then greedy sampling calculation is performed, and the full set selected each time is the solution set at time t solved by calculating inverse kinematics: , in, Refers to the moment The index value corresponding to the inverse kinematics solution selected in the mth trajectory obtained by sampling calculation, Step-by-step sampling to obtain all trajectories at all times After that, the initial joint angle is extracted By splicing in chronological order, we can obtain M motion trajectories with t joint configurations at a time, that is, .

[0016] further, The step S2 further comprises the steps of: The model predictive path integral control algorithm is used to Each track in Separate optimization: First, according to the path Calculate the approximate joint angular velocity based on the action acquisition interval , and then the approximate joint angular velocity The cost function and the robot system model are input into the model prediction path integral control algorithm to obtain the optimal system speed , the optimal system speed Integrating over time gives Right now , and then The M trajectories obtained after optimization are evaluated separately, and the one with the best effect is selected. in, The approximate joint angular velocity The control initial value for the model predictive path integral control algorithm , the noise defined in the model prediction path integral control algorithm Applying it to obtain the system control quantity of the model predictive path integral control algorithm , In the above formula, is the covariance matrix of the given system noise, Represents 0 as the mean and is the normally distributed noise with variance, R Represents the number of noise samples; The joint angle of the robot arm is X , then the model equation of the robotic arm system is: , in, Indicates the system control time interval; By adding noise, the control quantity of the system is equivalent to Scattered out R Different control quantities , and then through the robot system model, we get R Different system trajectories , The cost function includes: process cost and terminal cost, wherein the process cost and terminal cost are for In terms of any one of the trajectories.

[0017] Furthermore, the process cost at time t is , in, , , is a constraint item, and the weight W applied to the constraint item ranges from 6000 to 14000; , , is a Boolean value, and represents respectively whether the robot arm collides with the environment, the robot arm collides with itself, and exceeds the joint limit. If the above situation occurs, the corresponding Boolean value is true, otherwise it is false; is an error term, used to ensure the tracking of the operator's hand by the robot arm; The forward kinematics of the robot arm is used to calculate the position and posture of the end effector corresponding to a set of joint configurations; is the coordinate of the point in the trajectory obtained by tracking the human body and redirecting it at time t, The terminal cost , In the above formula, Indicates the The joint configuration at time t for any trajectory in .

[0018] further, Consider the noise term , the calculation formula is:

[0019] In the above formula, T represents the matrix transpose, for The inverse matrix of represents the noise applied to the calculated trajectory, Then for the Any track in The trajectory cost , Further, we obtain the The weight corresponding to any trajectory in: and , , in, It is a hyperparameter in the model predictive path integral control algorithm, which is used to adjust the noise control performance of the model predictive path integral control algorithm. represent Middle The trajectory cost value of the trajectory, Representative In each track The minimum value of Further, calculate the Among the M trajectories obtained after optimization, m Optimized control volume for each trajectory , Then, the optimized trajectory is obtained through the robot system model equation , in the above formula For the Middle r Group control quantity, At this point, the M trajectories are optimized separately to obtain the optimized trajectory set .

[0020] further, For the trajectory set Evaluate each trajectory in and select the best trajectory : , in , ; The weight W is set to 10,000.

[0021] The present invention also provides a computer device, which includes a memory, a first processor, and a first computer program stored in the memory and executable on the first processor, wherein the first computer program implements the above-mentioned adaptive remote operation method for narrow scenes when executed by the first processor.

[0022] The present invention also provides a computer-readable storage medium, which is used to store a second computer program. The second computer program can be executed by at least one second processor to enable the at least one second processor to execute the above-mentioned adaptive teleoperation method for narrow scenes.

[0023] The adaptive teleoperation method for narrow scenes provided by the present invention solves possible joint configurations by calculating inverse kinematics during teleoperation, and introduces a cost function through the MPPI method to optimize the joint configuration, thereby obtaining a smooth robot arm trajectory that can avoid obstacles. The robot arm can be smoothly teleoperated using simple equipment to achieve autonomous obstacle avoidance during teleoperation, and the robot arm can be teleoperated in real time in a narrow and restricted environment more conveniently to efficiently complete the operation task.

[0024] Other features and advantages of the present invention will be described in the following description, and partly become apparent from the description, or understood by practicing the present invention. The purpose and other advantages of the present invention can be realized and obtained by the structures pointed out in the description, claims and drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0026] Figure 1 A flow chart of an adaptive teleoperation method for narrow scenes according to an embodiment of the present invention is shown. DETAILED DESCRIPTION

[0027] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0028] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by technicians in the technical field of the present application; the terms used in the specification of the application herein are only for the purpose of describing specific embodiments and are not intended to limit the present application; the terms "including" and "having" in the specification and claims of the present application and the above-mentioned drawings and any variations thereof are intended to cover non-exclusive inclusions. The terms "first", "second", "third", etc. in the specification and claims of the present application or the above-mentioned drawings are used to distinguish different objects, not to describe a specific order or a primary and secondary relationship. "Multiple" appearing in this application refers to more than two (including two).

[0029] Reference to "embodiments" herein means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0030] In the task scenario, when using teleoperation to control the robot arm to perform tasks, there are obstacles that need to be avoided between the robot arm and the object or target position. For example, the robot arm needs to take out an object from the deep inside of a drawer. This requires the robot arm to open the drawer and extend the end effector into the drawer to take out the object. The environment inside the drawer is a very narrow environment for the robot arm.

[0031] During teleoperation, the operator and the robot have different structures, and the operator also lacks a realistic reference to factors such as the environment, so it is difficult to smoothly control the movement of the robot. To solve this problem, the robot control method needs to provide a certain obstacle avoidance performance. In order to achieve the obstacle avoidance performance, the robot needs to quickly calculate the joint angle configuration that can avoid collisions during operation according to the terminal trajectory given by teleoperation.

[0032] Figure 1 The flowchart of the adaptive teleoperation method for narrow scenes provided by the present invention includes the following steps.

[0033] S1. Motion data processing.

[0034] The motion data processing first uses the VR tracker device to read the data of the first motion trajectory of the human hand. After basic filtering of the data, the first motion trajectory is redirected and converted into a second motion trajectory in the workspace of the robotic arm. After the joint motion settlement, the data of the second motion trajectory is interpolated and smoothed to obtain the required motion trajectory data and then output it to the robotic arm.

[0035] Specifically, firstly, the present invention uses a tracker such as a VR tracker as a human motion signal collector, which can read the current coordinates of the VR tracker in cooperation with a corresponding base station. During use, the VR tracker can be fixed on the back of the staff's hand or a corresponding handheld device as needed.

[0036] Then, the motion trajectory of the hand is defined as The data tracked by the VR tracker can be obtained through the relevant Python interface of SteamVR. After basic filtering and other data processing, the collected hand movement trajectory is obtained as follows: The purpose of data processing is to eliminate the noise of the tracker itself and the influence of slight shaking of the human hand on the teleoperation. The data processing after the joint motion solution, that is, interpolation and smoothing, is to provide a smoother trajectory for the robot arm, making the robot arm move more smoothly.

[0037] In addition, since the robot's workspace is somewhat different from the human body's workspace, the human hand's workspace Perform redirection calculation and convert it into motion trajectory in the robot workspace The essential goal of redirection is to fit the motion coordinate system of the human hand to the coordinate system of the robotic arm. In the present invention, the coordinates of the human hand are obtained by the VR tracker. Therefore, the coordinate system of the robotic arm can be converted to the VR tracker, and then the coordinates of the human hand can be transformed to solve the redirection problem.

[0038] The present invention sets the redirection process as a part of the calibration process, and the specific method is as follows. First, the approximate size of the working space of the robotic arm is obtained through the known information of the robotic arm, and the known information can be obtained from the configuration files such as the Unified Robot Description Format (URDF) of the robotic arm; then, in the calibration process, the staff poses a fixed posture and reads the corresponding coordinates, wherein the posture of the robotic arm corresponding to the coordinates is pre-set, that is, when the human body poses the fixed posture, the robotic arm should be in the initial position. In this way, the position of the origin of the robotic arm's motion space in the tracker coordinate system can be deduced, thereby converting the robotic arm coordinate system to the VR tracker coordinate system. Further, the redirection transformation relationship matrix can be obtained after fitting. , and then In the present invention, the fixed posture mainly constrains the arms, keeps the arms as wide as the shoulders, and raises the forearms to be perpendicular to the upper arms.

[0039] S2, robot arm joint motion solution.

[0040] The solution of the robot arm joint motion first uses the IKflow method to calculate the inverse kinematics, thereby converting the motion trajectory in the robot arm workspace into multiple sets of joint configurations in the joint space. Finally, the model prediction path integral control (MPPI) algorithm is used to screen and optimize the candidate joint configurations to obtain the joint angle information that ultimately controls the robot arm.

[0041] Specifically, after obtaining the motion trajectory in the robot workspace, it is necessary to perform inverse kinematics solution before handing it over to the robot arm for execution. In order to quickly solve the problem while minimizing the restrictions on the flexibility of the robot arm, the present invention uses a parallel high-speed inverse kinematics solver IKflow, which uses a graphics processing unit (GPU) to obtain a large number of inverse kinematics solutions in a short time. After processing and calculation, a batch of inverse kinematics solutions can be obtained, denoted as ,in Represents a set of joint angles, corresponding to a configuration of the robot arm. The processing and calculation need to be performed in batches. B To process motion trajectories The time length of each batch, that is, the number of sampling moments (abbreviated as moments) of each batch, is a positive integer. N The number of inverse kinematic solutions generated when using IKflow, a positive integer.

[0042] The solutions generated directly using IKflow are different from other inverse kinematics solvers that use the Jacobi matrix. The continuity between the generated solutions cannot be naturally guaranteed. Therefore, the generated solution set needs to be screened. At the same time, in this process, environmental obstacles and other conditions are taken into account to screen out the most suitable trajectory for execution.

[0043] The problem addressed by the present invention is how to reasonably avoid obstacles in real-time tracking. From a methodological point of view, if accurate tracking is to be maintained while avoiding obstacles, the obstacle avoidance performance of the robot arm will be greatly tested, and even in most cases such a method is completely unfeasible. Therefore, the present invention considers finding a feasible solution around the target of real-time tracking to avoid obstacles, that is, a method similar to sampling around the tracking point to plan the trajectory. Based on this, the screening control method selected by the present invention is MPPI. The basic principle of this method is to apply noise to the control quantity to obtain different control sequences, and gradually diverge on the state of the controlled system (i.e., the robot arm system) through path integration, thereby obtaining different trajectories. According to the set cost function, the weight of each trajectory can be calculated, and then a weighted average is performed to obtain a statistically optimal solution. Specifically, the present invention selects the control quantity as the instantaneous velocity of each joint of the robot arm, and the state quantity as the angle of each joint of the robot arm.

[0044] In the present invention, due to the real-time limitation of MPPI, the torch version of MPPI is used and improved to a certain extent, so that it can process multiple trajectories at the same time. Compared with the original method of sequential processing, the screening efficiency is greatly improved, thereby being able to meet the basic frequency requirements of remote operation.

[0045] In the implementation of the method, we first need to ensure the continuity of the robot's motion trajectory. Determine the two moments before and after t -1 and t The angle difference between different groups of joint configurations, including the sum of joint angle differences Difference from the maximum joint angle , t is a positive integer. t No. i Group joint configuration and timing t -1st j The angular difference between the group joint configurations is , let ndof be the degree of freedom of the robot, considering that each set of joints is configured as an array of ndof dimensions, then It is also an array of ndof dimensions, where Wei is , then At the moment t，No. j Group joint configuration and i The sum of the joint angle differences between the group joint configurations:

[0046] and j Group joint configuration and i Maximum joint angle difference between group joint configurations: .

[0047] In the above formula, the right side of the equal sign represents the array from the ndof dimension Select The maximum value of , where max|·| is the maximum value function.

[0048] In addition, a greedy algorithm is used to sample a relatively continuous path Afterwards, in order to increase the divergence of the trajectory in the MPPI algorithm, multiple paths can be sampled by selecting different initial joint angle configurations to obtain , where M is the number of sampled paths and is a positive integer. In order to avoid the initial joint angle configuration being too bad, in this process, all initial joint angle configurations are first evaluated to obtain the relative value of each initial joint angle configuration to the current joint angle of the robot arm. The set of all inverse kinematic solutions of , including: the sum of joint angle differences (i.e. Moment ) and the maximum joint angle difference (i.e. Moment ): , .

[0049] Afterwards, from the collection Extract or select the initial joint angle , that is, the set of joint configurations at the initial moment of the M group, represents the joint configuration of the mth (m is an integer) group at the first moment, The expression is:

[0050] The above formula represents the Select M elements from Indicates that the selected element is in the collection The index value in . Among them, argmin is the argmin function, ‌ refers to the value of the variable when the objective function reaches the minimum value. Specifically, ‌argmin f(x)‌ represents the value of the variable x when the function f(x) reaches the minimum value.

[0051] Then, greedy sampling calculation is performed. For time t, a set is selected from the solution set of time t solved by IKflow calculation of inverse kinematics: , in, Refers to the moment The index value corresponding to the inverse kinematics solution selected in the mth trajectory obtained by sampling calculation.

[0052] Step-by-step sampling to obtain all trajectories at all times After that, the initial joint angle is extracted By splicing them in chronological order, we can obtain M motion trajectories with t joint configurations at each moment, that is, .

[0053] Next, we use the MPPI algorithm to Each track in Optimize separately: First, according to the path Calculate the approximate joint angular velocity based on the action acquisition interval . Then approximate the joint angular velocity The cost function defined in the present invention and the robot system model are input into the MPPI algorithm to obtain the optimal system speed. , the optimal system speed Integrating over time gives Right now . Then The M trajectories obtained after optimization are evaluated separately, and the one with the best effect is selected.

[0054] The calculated approximate joint angular velocity The role of the MPPI algorithm is to serve as the initial control value in the algorithm. , the noise defined in the MPPI algorithm Apply it on it to get the system control quantity , In the above formula is the covariance matrix of the given system noise, which determines the size of the system noise. Represents 0 as the mean, is the normally distributed noise with variance, R Represents the number of noise samples.

[0055] The main function of the robot system model is to update the system state and define how the control quantity changes the system state. In this invention, the system state is the robot joint angle, which is defined as X , then the model equation of the robotic arm system is: , in, Indicates the system control time interval.

[0056] By adding noise, the control quantity of the system is equivalent to Scattered out R Different control quantities , and then through the robot system model we get R Different system trajectories .

[0057] Note that, similar to model predictive control (MPC), for the final optimized trajectory, only the joint configurations of the expected number of moments, such as the previous one or several moments, are taken and given to the robot to execute, so as to increase the continuity of the control trajectory and the robustness of the system.

[0058] The evaluation of joint trajectory smoothness and automatic obstacle avoidance is achieved by setting a cost function. By adding the collision detection results and the size of the robot arm speed to the cost function, the divergent Each trajectory in is evaluated, and the final optimization is obtained .

[0059] In specific implementation, the cost function used in the present invention mainly includes two parts, process cost and terminal cost. The process cost mainly considers the collision of the robot arm at each moment and the error between the end of the robot and the tracking. Specifically, the process cost at time t , in, , , In the process cost, the collision situation of the robot arm is calculated by curobo to determine whether there is a collision. This part, the self-collision situation of the robot arm and the situation of exceeding the joint limit are put into the constraint item. As a hard constraint of the robot arm, this part will impose a higher weight W, and the value range of W is 6000-14000, preferably 10000. In the above formula, , , are Boolean values, representing whether the robot arm has encountered some unexpected situation. Specifically, , , In turn, they represent whether the robot arm collides with the environment, the robot arm collides with itself, and exceeds the joint limit. If the above situations occur, the corresponding Boolean value is true, otherwise it is false. If the robot arm collides with the environment, then If it is 1, it is true, otherwise it is 0, which is false. In addition, since all the factors considered in the process cost will try to minimize the value after optimization, the error term Taking this into account can ensure that the error is as small as possible, so that the error term It can be used to ensure that the robot arm tracks the operator's hand. Forward kinematics of the robot arm, used to calculate the position and posture of the end effector corresponding to a certain joint configuration. The coordinates of the points in the trajectory obtained by tracking the human body and redirecting it at time t are also the coordinates of the points when calculating the inverse kinematics using an IK solver such as IKflow.

[0060] The terminal cost mainly considers the smoothness of the entire trajectory. In order for the actual robot arm to perform operations, the smoothness of the entire trajectory needs to be guaranteed. In the present invention, the trajectory smoothness is measured using the smoothness item To reflect or illustrate, its components include the above mentioned and , but the slight difference here is that it needs to be calculated for the entire trajectory, that is, the terminal cost , In the above formula, represents the joint configuration at time t in the entire trajectory.

[0061] In addition, the MPPI algorithm also evaluates the noise level applied to the system and defines the result as the noise term , the calculation formula is:

[0062] In the above formula, T represents the matrix transpose, for The inverse matrix of Represents the noise applied to the calculated trajectory.

[0063] for Add a noise term to any trajectory The trajectory cost

[0064] Further we can get the The weight corresponding to any trajectory in: and , In the above formula, It is a hyperparameter in the MPPI algorithm, which is used to adjust the noise control performance of the MPPI algorithm. It is set by technicians according to the working conditions. Representative Middle The trajectory cost value of the trajectory, Representative In each track The minimum value of .

[0065] Further, we can calculate Among the M trajectories obtained after optimization, m Optimized control volume for each trajectory , Then the optimized trajectory is obtained through the robot system model equation , in the above formula For the Middle Group control quantity.

[0066] At this point, the M trajectories are optimized separately to obtain the optimized trajectory set Finally, each trajectory is evaluated and the best trajectory is selected. : , in , .

[0067] Finally, the joint configuration of the expected number of moments, such as the previous one or several moments, is returned. According to the above process, the steps of updating the system state to returning the joint configuration can be repeated during the robot teleoperation process to obtain a real-time control trajectory.

[0068] Finally, before applying it to the robot arm, some interpolation is required on the trajectory to provide a smoother trajectory for the robot arm.

[0069] The present invention uses IKflow to calculate inverse kinematics to solve possible joint configurations during teleoperation, and introduces a cost function through the MPPI method to optimize the joint configuration, thereby obtaining a smooth robot arm trajectory that can avoid obstacles. This allows the present invention to use simple equipment to perform smooth teleoperation of the robot arm, while having a certain obstacle avoidance effect during the teleoperation process.

[0070] The present invention also provides a computer device, which may include a memory, a first processor, and a first computer program stored in the memory and executable on the first processor, wherein the first computer program implements the above-mentioned adaptive remote operation method for narrow scenes when executed by the first processor.

[0071] The present invention also provides a computer-readable storage medium, which is used to store a second computer program. The second computer program can be executed by at least one second processor to enable the at least one second processor to execute the above-mentioned adaptive teleoperation method for narrow scenes.

[0072] The memory may be a volatile memory, such as a random access memory (RAM), or an external cache memory. By way of illustration and not limitation, RAM is available in a variety of forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), RAMbus direct RAM (RDRAM), direct RAM bus dynamic RAM (DRDRAM), and RAMbus dynamic RAM (RDRAM); or a non-volatile memory, such as a read-only memory (ROM), a programmable ROM (PROM), an electrically programmable ROM (EPROM), an electrically erasable programmable ROM (EEPROM), a flash memory, a hard disk drive (HDD) or a solid-state drive (SSD); or a combination of the above types of memory.

[0073] The first processor and the second processor may be at least one of an application-specific integrated circuit (ASIC), a digital signal processor (DSP), a digital signal processing device (DSPD), a programmable logic device (PLD), a field programmable gate array (FPGA), a central processing unit (CPU), a controller, a microcontroller, and a microprocessor. It is understandable that for different devices, the electronic device used to implement the functions of the processor may also be other, and the embodiment of the present invention does not specifically limit it.

[0074] The adaptive teleoperation method for narrow scenes provided by the present invention introduces an efficient robot inverse kinematics solver IKFlow and a model prediction path integral control algorithm, thereby realizing autonomous obstacle avoidance during teleoperation. It is more convenient to perform real-time teleoperation of the robotic arm in a narrow and restricted environment to efficiently complete the operation task.

[0075] According to an embodiment of the present invention, the method flow according to an embodiment of the present invention can be implemented as a computer software program. For example, an embodiment of the present invention includes a computer program product, which includes a computer program carried on a computer-readable storage medium, and the computer program includes a program code for executing the method shown in the flowchart. According to an embodiment of the present invention, the electronic device, device, apparatus, module, unit, etc. described above can be implemented by a computer program module.

[0076] The present invention also provides a computer-readable storage medium, which may be included in the device / apparatus / system described in the above embodiment; or may exist independently without being assembled into the device / apparatus / system. The above computer-readable storage medium carries one or more programs, and when the above one or more programs are executed, the method according to the embodiment of the present invention is implemented.

[0077] According to an embodiment of the present invention, the computer-readable storage medium may be a non-volatile computer-readable storage medium, for example, may include but is not limited to: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In the present invention, the computer-readable storage medium may be any tangible medium containing or storing a program, which may be used by or in combination with an instruction execution system, apparatus or device.

[0078] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present invention. In this regard, each box in the flow chart or block diagram can represent a module, a program segment, or a part of a code, and the above-mentioned module, program segment, or a part of a code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flow chart, and the combination of the boxes in the block diagram or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0079] It will be appreciated by those skilled in the art that the features described in the various embodiments and / or claims of the present invention may be combined and / or combined in various ways, even if such combinations and / or combinations are not explicitly described in the present invention. In particular, the features described in the various embodiments and / or claims of the present invention may be combined and / or combined in various ways without departing from the spirit and teachings of the present invention. All of these combinations and / or combinations fall within the scope of the present invention.

[0080] The embodiments of the present invention are described above. However, these embodiments are only for the purpose of illustration, and are not intended to limit the scope of the present invention. Although each embodiment is described above, it does not mean that the measures in each embodiment cannot be used in combination. The scope of the present invention is defined by the attached claims and their equivalents. Without departing from the scope of the present invention, those skilled in the art may make various substitutions and modifications, which should all fall within the scope of the present invention.

Claims

1. Adaptive teleoperation method for narrow scenes, characterized by: Includes steps: S1, motion data processing, including: reading data of a first motion trajectory of a human hand, filtering the data, redirecting the first motion trajectory to convert it into a second motion trajectory in the workspace of the robot arm, interpolating and smoothing the data of the second motion trajectory after performing joint motion settlement, obtaining the required motion trajectory data and then outputting it to the robot arm; S2. Robot arm joint motion solution, including: calculating inverse kinematics, thereby converting the motion trajectory in the robot arm workspace into multiple sets of joint configurations in the joint space, and then screening and optimizing the candidate joint configurations to obtain the joint angle information for ultimately controlling the robot arm, wherein a model prediction path integral control algorithm is used to simultaneously process multiple motion trajectories.

2. The adaptive teleoperation method for narrow scenes according to claim 1, characterized in that: The step S1 comprises the steps of: A tracker is used as a human motion signal collector, and the tracker cooperates with a corresponding base station to read the current coordinates of the tracker; The motion trajectory of the hand is recorded as The data of the hand tracking by the tracker is processed to obtain the collected hand movement trajectory. ; The hand motion trajectory in the hand workspace Perform redirection calculation and convert it into motion trajectory in the robot workspace .

3. The adaptive teleoperation method for narrow scenes according to claim 2 is characterized in that: The step S2 comprises the steps of: The motion trajectory of the robot in the workspace After processing and calculation, a batch of inverse kinematics solutions are obtained, denoted as ,in represents a set of joint angles, corresponding to a configuration of the robotic arm, B To process the motion trajectory of the robot in the workspace The total number of batches, N is the number of inverse kinematic solutions generated, In the inverse kinematics solution Determine the two moments before and after t -1 and t The angle difference between different sets of joint configurations, t is a positive integer, record the time t No. i Group joint configuration and timing t -1st j The angular difference between the group joint configurations is , let ndof be the degree of freedom of the robot, considering that each set of joints is configured as an array of ndof dimensions, then It is also an array of ndof dimensions, where Wei is , then At the moment t， The said j Group joint configuration and i The sum of the joint angle differences between the group joint configurations: ， And the j Group joint configuration and i Maximum joint angle difference between group joint configurations: ， In the above formula, the right side of the equal sign represents the array from the ndof dimension Select The maximum value of , where max|·| is the maximum value function.

4. The adaptive teleoperation method for narrow scenes according to claim 3 is characterized in that: The step S2 further comprises the steps of: Apply the greedy algorithm to sample the path Afterwards, multiple paths are sampled by selecting different initial joint angle configurations to obtain , where M is the number of sampled paths and is a positive integer, specifically including the steps: First, all the initial joint angle configurations are evaluated to obtain the relative value of each initial joint angle configuration to the current joint angle of the robot arm. The set of all inverse kinematic solutions of , including: the sum of joint angle differences, i.e. Moment , and the maximum joint angle difference is Moment : ， ， Afterwards, from the collection Extract or select the initial joint angle , that is, the set of M groups of joint configurations at the initial moment, Represents that the mth group is at the first moment, The joint configuration at the moment, m is an integer, The expression is: ， The above formula represents the Select elements, Indicates that the selected element is in the set , where argmin is the argmin function and ‌ refers to the value of the variable when the objective function reaches the minimum value. Specifically, ‌argmin f(x)‌ represents the value of the variable x when the function f(x) reaches the minimum value. Then greedy sampling calculation is performed, and the full set selected each time is the moment solved by calculating inverse kinematics The solution set: ， in, Refers to the moment The sampling calculation results The index value corresponding to the inverse kinematics solution selected in the trajectory, Step-by-step sampling to obtain all trajectories at all times After that, the initial joint angle is extracted By splicing in chronological order, we can obtain M motion trajectories with t joint configurations at a time, that is, .

5. The adaptive teleoperation method for narrow scenes according to claim 4, characterized in that: The step S2 further comprises the steps of: The model predictive path integral control algorithm is used to Each track in Separate optimization: First, according to the path Calculate the approximate joint angular velocity based on the action acquisition interval , and then the approximate joint angular velocity The cost function and the robot system model are input into the model prediction path integral control algorithm to obtain the optimal system speed , the optimal system speed Integrating over time gives Right now , and then The M trajectories obtained after optimization are evaluated separately, and the one with the best effect is selected. in, The approximate joint angular velocity The control initial value for the model predictive path integral control algorithm , the noise defined in the model prediction path integral control algorithm Applying it to obtain the system control quantity of the model predictive path integral control algorithm ， In the above formula, Given the covariance matrix of the system noise, Represents 0 as the mean, is the normally distributed noise with variance, R Represents the number of noise samples; The joint angle of the robot arm is X , then the model equation of the robotic arm system is: ， in, Indicates the system control time interval; By adding noise, the control quantity of the system is equivalent to Scattered out R Different control quantities , and then through the robot system model, we get R Different system trajectories , The cost function includes: process cost and terminal cost, wherein the process cost and terminal cost are for In terms of any one of the trajectories.

6. The adaptive teleoperation method for narrow scenes according to claim 5, characterized in that: time The cost of the process ， in, ， ， is a constraint item, and the weight W applied to the constraint item ranges from 6000 to 14000; , , is a Boolean value, and represents respectively whether the robot arm collides with the environment, the robot arm collides with itself, and exceeds the joint limit. If the above situation occurs, the corresponding Boolean value is true, otherwise it is false; is an error term, used to ensure the tracking of the operator's hand by the robot arm; The forward kinematics of the robot arm is used to calculate the position and posture of the end effector corresponding to a set of joint configurations; For at the moment The coordinates of the points in the trajectory obtained by tracking the human body and redirecting it, The terminal cost ， In the above formula, Indicates the The joint configuration at time t for any trajectory in .

7. The adaptive teleoperation method for narrow scenes according to claim 6, characterized in that: Consider the noise term , the calculation formula is: ， In the above formula, represents the matrix transpose, for The inverse matrix of represents the noise applied to the calculated trajectory, Then for the Any track in The trajectory cost ， Further, we obtain the The weight corresponding to any trajectory in: and ， in, It is a hyperparameter in the model predictive path integral control algorithm, which is used to adjust the noise control performance of the model predictive path integral control algorithm. Representative Middle The trajectory cost value of the trajectory, Representative In each track The minimum value of Further, calculate the Among the M trajectories obtained after optimization, m Optimized control volume for each trajectory ， Then, the optimized trajectory is obtained through the robot system model equation , in the above formula For the Middle r Group control quantity, At this point, the M trajectories are optimized separately to obtain the optimized trajectory set .

8. The adaptive teleoperation method for narrow scenes according to claim 7, characterized in that: For the trajectory set Evaluate each trajectory in and select the best trajectory : ， in ， ， The weight W is set to 10,000.

9. A computer device, characterized in that: The invention comprises a memory, a first processor and a first computer program stored in the memory and executable on the first processor, wherein the first computer program, when executed by the first processor, implements the adaptive teleoperation method for narrow scenes as described in any one of claims 1 to 8.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium is used to store a second computer program, and the second computer program can be executed by at least one second processor, so that the at least one second processor executes the adaptive teleoperation method for narrow scenes according to any one of claims 1-8.

Images