A teleoperation tracking method, apparatus, device, system, and storage medium

Abstract

The embodiment of the application belongs to the technical field of remote operation, and relates to a remote operation tracking method, which comprises the following steps: when it is judged that delay occurs in robot tracking, interrupting original remote operation tracking of the robot; mapping a current target posture of the robot based on current motion data of an operator; generating a motion prediction instruction based on the current target posture, so as to instruct the robot to move to the current target posture in a predicted track; and generating specific visual information of the predicted track. The application also provides a remote operation tracking device, equipment, system and storage medium. The application can help the operator to better understand the motion condition of the robot in the delay process.

Description

Technical Field

[0001] This application relates to the field of teleoperation technology, and in particular to a teleoperation tracking method, apparatus, device, system and storage medium. Background Technology

[0002] With the development of technology, the application of robots is becoming increasingly widespread. In some complex and dangerous environments, robots are required to be more flexible and have a higher degree of human-like work ability. For this reason, teleoperated robots have emerged. Teleoperated robots are operated by installing attitude sensors, such as inertial measurement units (IMUs), on the operator. The operator performs the target task in another real or virtual scenario. The IMU captures the operator's movements during the operation and collects corresponding motion feedback data, which is sent to the controller. The controller generates corresponding motion control commands based on the feedback data to control the slave robot, thereby achieving the purpose of teleoperation.

[0003] Currently, during teleoperation, delays often occur when the robot tracks the operator's movements. Once a delay occurs, the operator cannot understand what is happening during the delay, which may lead to a loss of control in subsequent teleoperations. Summary of the Invention

[0004] The purpose of this application is to provide a teleoperation tracking method, apparatus, device, system, and storage medium to help operators better understand the robot's movement during the delay process.

[0005] Firstly, embodiments of this application provide a teleoperation tracking method, which employs the following technical solution:

[0006] A teleoperation tracking method includes the following steps:

[0007] If a delay is detected in robot tracking, the original teleoperation tracking of the robot is interrupted;

[0008] The current target pose of the robot is mapped based on the operator's current pose;

[0009] Based on the current target posture, a motion prediction instruction is generated to instruct the robot to move to the current target posture along a predicted trajectory;

[0010] Generate specific visualization information for the predicted trajectory; the specific visualization information refers to visualization information that is distinguishable from other visualization information.

[0011] Furthermore, generating specific visualization information for the predicted trajectory includes the following steps:

[0012] Obtain multiple motion postures of the robot under the motion prediction command;

[0013] Each of the aforementioned motion postures is converted to display coordinates under the image displayed on the monitor;

[0014] The specific visualization information is generated based on the display coordinates.

[0015] Furthermore, generating the specific visualization information based on the display coordinates includes the following steps:

[0016] Based on the displayed coordinates of the robot joints, the specific visualization information is generated; or

[0017] Connect the display coordinates of two adjacent robot joints using a specific visual connector;

[0018] The specific visualization information is generated based on the display coordinates of the robot joints and the connector.

[0019] Furthermore, the method also includes the following steps:

[0020] Obtain the teleoperation trajectory of the robot during the original teleoperation tracking process;

[0021] Generate visualization information of the teleoperation trajectory.

[0022] Furthermore, before interrupting the robot's original teleoperation tracking when a delay in robot tracking is determined, the following steps are also included:

[0023] Obtain the robot's target pose and its corresponding first timestamp;

[0024] Obtain the robot's pose and its corresponding second timestamp;

[0025] Determine whether the robot's pose is the same as the target pose; if they are not the same, repeat the step of obtaining the robot's pose and the corresponding second timestamp.

[0026] If they are the same, obtain the second timestamp corresponding to when the robot's posture is the same as the target posture;

[0027] Based on the time difference between the first timestamp and the second timestamp, it is determined whether a delay has occurred in robot tracking.

[0028] Furthermore, after generating visualization information based on the multiple motion postures, the process further includes the following steps:

[0029] Send the visualization information to the display so that the visualization information can be displayed on the display; and / or

[0030] After generating motion prediction instructions based on the current target pose, the method further includes:

[0031] Continue the original remote operation tracking of the robot.

[0032] Furthermore, the process of mapping the robot's current target pose based on the operator's current pose includes the following steps:

[0033] Acquire current motion data of key parts of the operator in the current posture, collected and transmitted by the attitude sensor;

[0034] Convert the current motion data into the robot's current target pose; or

[0035] Acquire the current operator image captured by the image sensor;

[0036] The operator's current posture is identified based on the operator's image;

[0037] The current pose is converted into the robot's current target pose.

[0038] Furthermore, the step of generating motion prediction instructions based on the current target pose includes the following steps:

[0039] Trajectory planning is performed based on the current target pose to generate the motion prediction command; or

[0040] Obtain the preset model;

[0041] Based on the current target pose, trajectory planning is performed using the preset model to generate the motion prediction command; or

[0042] Obtain the preset motion prediction model;

[0043] Based on the current target posture, the motion prediction command is output through the motion prediction model.

[0044] Secondly, embodiments of this application provide a teleoperated tracking device, the device comprising: a delay tracking module; the delay tracking module comprising:

[0045] The tracking interruption submodule is used to interrupt the robot's original teleoperation tracking when a delay is detected in the robot's tracking.

[0046] The posture mapping submodule is used to map the robot's current target posture based on the operator's current posture.

[0047] The instruction generation submodule is used to generate motion prediction instructions based on the current target posture, so as to instruct the robot to move to the current target posture along a predicted trajectory;

[0048] The information visualization submodule is used to generate specific visualization information of the predicted trajectory; the specific visualization information refers to visualization information that is distinguishable from other visualization information.

[0049] Thirdly, embodiments of this application provide a remote operating system, the system comprising: an attitude sensor, a robot, a display, and a controller;

[0050] The attitude sensor, robot, and display are respectively communicatively connected to the controller;

[0051] The posture sensor is used to collect the operator's motion data and send the motion data to the controller;

[0052] The controller is configured to generate motion commands based on the motion data to instruct the robot to perform the original telescopic tracking; interrupt the original telescopic tracking of the robot when it is determined that the robot tracking is delayed; map the current target posture of the robot based on the operator's current posture; generate motion prediction commands based on the current target posture to instruct the robot to move to the current target posture along a predicted trajectory; and generate specific visualization information of the predicted trajectory; the specific visualization information refers to visualization information that is distinguishable from other visualization information.

[0053] The display is used to show the specific visual information.

[0054] Fourthly, embodiments of this application provide a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps of the teleoperation tracking method as described in any of the above claims.

[0055] Fifthly, embodiments of this application provide a computer-readable storage medium, characterized in that the computer-readable storage medium stores a computer program, which, when executed by a processor, implements the steps of the teleoperation tracking method as described in any of the above claims.

[0056] Compared with the prior art, the embodiments of this application have the following main advantages:

[0057] This application embodiment autonomously generates a predicted trajectory for the robot under time delay and generates specific visualization information for the predicted trajectory, which can help the operator to more intuitively understand the robot's movement under time delay, thereby helping the operator to better perform remote operation control of the robot. Attached Figure Description

[0058] To more clearly illustrate the solutions in this application, the accompanying drawings used in the description of the embodiments of this application will be briefly introduced below. Obviously, the accompanying drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0059] Figure 1 This is an exemplary system architecture diagram to which this application can be applied;

[0060] Figure 2 This is a schematic diagram of an embodiment of a display screen showing specific visual information, as described in this application.

[0061] Figure 3 This is a flowchart illustrating an embodiment of the teleoperation tracking method of this application;

[0062] Figure 4 This is a schematic diagram of one embodiment of the teleoperated tracking device of this application;

[0063] Figure 5 This is a basic structural block diagram of an embodiment of the computer device of this application. Detailed Implementation

[0064] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application pertains; the terminology used herein in the specification of the application is for the purpose of describing particular embodiments only and is not intended to be limiting of the application; the terms "comprising" and "having," and any variations thereof, in the specification, claims, and foregoing drawings of this application, are intended to cover non-exclusive inclusion. The terms "first," "second," etc., in the specification, claims, or foregoing drawings of this application are used to distinguish different objects, not to describe a particular order.

[0065] In this document, the term "embodiment" means that a particular feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment mutually exclusive with other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.

[0066] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings.

[0067] like Figure 1 As shown, Figure 1This is an exemplary system architecture diagram in which this application can be applied.

[0068] This application provides a remote operating system 100, which includes: an attitude sensor 110, a robot 120, a display 130, and a controller 140.

[0069] The attitude sensor 110, robot 120 and display 130 are respectively connected to the controller 140 via wired or wireless means.

[0070] It should be noted that the aforementioned wireless connection methods may include, but are not limited to, 3G / 4G / 5G connections, WiFi connections, Bluetooth connections, WiMAX connections, Zigbee connections, UWB (ultra wideband) connections, and other currently known or future wireless connection methods.

[0071] The attitude sensor 110 is used to collect the operator's motion data.

[0072] The aforementioned attitude sensor 110 can be any existing or future attitude sensor capable of collecting operator motion data, such as: IMU110 (e.g.) Figure 1 (As shown in the figure), image sensor (figure omitted), etc. For ease of understanding, this application mainly uses the attitude sensor 110 as an example for detailed description.

[0073] Specifically, the motion data mentioned above varies depending on the attitude sensor. For example, when the sensor is an IMU (Inertial Measurement Unit), it is used to measure motion data related to the target object, including three-dimensional acceleration and three-dimensional rotation angle. Or, when the attitude sensor is an image sensor, the motion data can be image data including the operator or key parts of the operator. The controller then analyzes and processes this image data to map the robot's attitude information, and so on.

[0074] Specifically, the posture sensor 110 can be directly fixed to a preset position such as the operator's joint, or fixed to a preset position of the operator through a wearable device. For example, the posture sensor can be pre-fixed to an arm exoskeleton, and the arm exoskeleton can be worn on the operator's arm, thereby fixing the posture sensor to the operator's arm.

[0075] For example, the IMU110 continuously collects motion data of the operator 200 during the execution of a target task in a real or virtual environment at a preset frequency, and sends the motion data to the controller 140.

[0076] Robot 120 is used to perform teleoperation tracking based on the indication of motion commands sent by controller 140, and, in the case of a delay, to move along a predicted trajectory based on the indication of motion prediction commands sent by controller 140.

[0077] The aforementioned robot 120 can refer to a humanoid robot, a robotic arm, etc. It can refer to the entire robot or a part of a robot based on teleoperation control, such as the upper body or the gripper portion. Taking robotic arm 120 as an example, the robot end effector described in the following embodiments can refer to the output end of the end joint of the robotic arm, such as the center of the flange at the output end of the sixth joint of a 6-axis robotic arm.

[0078] Display 130 is used to display specific visual information sent by controller 140.

[0079] In one embodiment, the display can show not only the visualization information of the predicted trajectory, but also the visualization information of the robot's tracking trajectory.

[0080] Specifically, the display may be, but is not limited to, a screen or an AR / VR virtual display device.

[0081] In one embodiment, the present application may further include a slave sensor 150.

[0082] The slave sensor 150 is used to collect various feedback data related to the slave robot 120, such as images, joint movements, and forces.

[0083] Furthermore, in one embodiment, the slave sensor 150 is communicatively connected to the controller 140 to send feedback data collected by the slave sensor 150 to the controller 140, and the controller 140 sends the feedback data or the data processed from the feedback data to the display 130 for display.

[0084] The controller 140 is used to execute the teleoperation tracking method described in the embodiments of this application, etc.

[0085] In one embodiment, the controller generates motion commands based on motion data collected by the attitude sensor to instruct the robot to perform original telescopic tracking; when it is determined that the robot tracking is delayed, the original telescopic tracking of the robot is interrupted; the current target posture of the robot is mapped based on the operator's current posture; motion prediction commands are generated based on the current target posture to instruct the robot to move to the current target posture along the predicted trajectory; and specific visualization information of the predicted trajectory is generated.

[0086] Specifically, the teleoperation tracking method provided in this application embodiment can be applied to computer terminals (PCs); industrial personal computers (IPCs); mobile terminals; servers; systems including terminals and servers, implemented through interaction between the terminals and servers; programmable logic controllers (PLCs); field-programmable gate arrays (FPGAs); digital signal processors (DSPs) or microcontroller units (MCUs) and similar controllers. The controller generates program instructions based on a pre-fixed program and data collected by external attitude sensors 110, etc. Specific limitations of the controller can be found in the limitations of the teleoperation tracking method in the following embodiments.

[0087] Specifically, it can be applied to, for example Figure 5 The computer device shown can be a terminal or a server. The server can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN (Content Delivery Network), and big data and artificial intelligence platforms. The terminal can be a smartphone, tablet, laptop, desktop computer, smart speaker, smartwatch, etc., but is not limited to these. The terminal and server can be directly or indirectly connected via wired or wireless communication, which is not limited herein.

[0088] It should be noted that the teleoperation tracking method provided in this application embodiment is generally executed by the controller 140, and correspondingly, the device for teleoperation tracking is generally disposed in the controller 140.

[0089] like Figure 3 As shown, Figure 3 This is a flowchart illustrating an embodiment of the teleoperation tracking method of this application.

[0090] In one embodiment, the teleoperation tracking method may include the following steps:

[0091] Step 210: When it is determined that there is a delay in robot tracking, the original teleoperation tracking of the robot is interrupted.

[0092] like Figure 4 As shown, Figure 4 This is a schematic diagram of one embodiment of the teleoperation tracking device of this application. In this embodiment, on one hand, the teleoperation tracking module 310 of the controller 140 is used to generate motion commands to instruct the robot to perform original teleoperation tracking; on the other hand, the auxiliary tracking module 320 of the controller 140 is used to send an interrupt command to the teleoperation tracking module 310 to interrupt the robot's original teleoperation tracking when it is determined that the robot tracking has been delayed during the original teleoperation tracking process.

[0093] In one embodiment, the teleoperation tracking method for the robot described above includes the following steps:

[0094] Step 250 generates motion commands based on the operator's motion data acquired through the attitude sensor to instruct the robot to perform remote operation tracking.

[0095] In one embodiment, the controller's teleoperation tracking module 310 obtains the operator's motion data collected by the attitude sensor from the memory or server according to a preset address, maps the motion data to the robot's joint motion information, and generates motion commands based on this motion information to instruct the robot to perform the original teleoperation trajectory tracking.

[0096] Step 220: Obtain the robot's current target pose based on the operator's current pose mapping.

[0097] In one embodiment, step 220 may specifically include the following method steps:

[0098] Step 221: Obtain current motion data of the operator's key parts in the current posture, collected and transmitted by the attitude sensor.

[0099] Step 223 converts the current motion data into the robot's current target pose.

[0100] Furthermore, in one embodiment, the controller converts the current IMU data of the operator's key parts (e.g., wrist) in the current posture into the current posture of the robot joint (e.g., the end effector of the robotic arm) after the IMU collects and sends the current IMU data in the current posture, and then converts it into the current posture of the robot joint (e.g., the end effector of the robotic arm) after the mapping steps such as filtering and coordinate system transformation, and uses the current posture as the target posture of the robot.

[0101] In one embodiment, step 220 may specifically include the following method steps:

[0102] Step 222: Acquire the current operator image captured by the image sensor.

[0103] Step 224 identifies the operator's current pose based on the operator's image.

[0104] Step 226 converts the current pose to the robot's current target pose.

[0105] Furthermore, in one embodiment, the controller uses an image of the operator captured by an image sensor to identify the operator's current posture based on the image, and then converts the current posture into the target posture of the robot joints after coordinate system transformation.

[0106] For example, the controller identifies the current posture of the operator's wrist in the image of the operator based on the image sensor that is acquired and sent by the image sensor. Based on the preset calibration relationship between the image sensor and the robot, the current posture of the wrist is converted into the current target posture of the robot end effector.

[0107] Step 230 generates motion prediction instructions based on the current target pose to instruct the robot to move to the current target pose along the predicted trajectory.

[0108] In one embodiment, the controller can autonomously generate motion prediction instructions based on trajectory planning or artificial intelligence methods to instruct the robot to move along the predicted trajectory to the current target posture, which will be further described in detail in later embodiments.

[0109] In the event of a delay in teleoperation tracking, this embodiment interrupts teleoperation tracking and instructs the robot to assist in teleoperation tracking according to a self-generated predicted trajectory. This helps to improve the speed of teleoperation tracking, making the entire teleoperation tracking process more coherent and responsive. In addition, it can reduce the impact of network latency, thereby improving the stability of robot operation.

[0110] In one embodiment, after step 230, the following method steps may also be included:

[0111] Step 260 continues the original remote operation tracking of the robot.

[0112] like Figure 4 As shown in this embodiment, after the robot executes the motion prediction command, the auxiliary tracking module 320 of the controller 140 sends a start command to the teleoperation tracking module 310 of the controller 140 again. In response to the start command, the teleoperation tracking module 310 continues to execute the original teleoperation tracking method steps until it receives an interrupt command sent by the auxiliary tracking module 320 again.

[0113] Step 240 generates specific visualization information for the predicted trajectory.

[0114] Specifically, specific visualization information refers to visualization information that is distinct from other visualization information. For example, if a display can also show non-delayed visualization information of the actuator in its non-delayed state, then the specific visualization information for predicting the trajectory must be distinguished from the non-delayed visualization information.

[0115] In one embodiment, the controller sends the motion prediction instruction generated in step 220 to the robot to instruct the robot to execute a predicted motion trajectory based on the motion prediction instruction. The predicted motion trajectory is composed of multiple sets of posture information in sequence. The controller obtains the posture information of each set of motion postures under the execution of the predicted trajectory and represents each set of posture information with visualization information, so that a motion ghost image of the robot executing the predicted trajectory can be formed on the display screen.

[0116] like Figure 2 As shown, Figure 2 This is a schematic diagram of an embodiment of a display screen showing specific visual information. For example, the visual information in a delayed state can be identified by dashed lines or hollow dots (M); while the non-delayed visual information can be identified by solid lines or solid dots (M'), thereby creating a motion ghosting effect on the displayed screen of the robot executing the predicted trajectory M.

[0117] This application embodiment generates specific visualization information by estimating the predicted motion under delayed conditions, which can help the operator to more intuitively understand the robot's motion under delayed conditions, thereby providing a basis for better implementation of the entire teleoperation tracking.

[0118] In an optional embodiment, after step 240 generates specific visualization information of the predicted trajectory, the following steps may also be included:

[0119] Step 260 sends specific visualization information to the display so that the specific visualization information can be displayed on the display.

[0120] This application embodiment sends specific visual information to a display, and the display can display the specific visual information.

[0121] In an optional embodiment, step 240, generating specific visualization information of the predicted trajectory, may specifically include the following method steps:

[0122] Step 241: Obtain multiple motion postures under the motion prediction command executed by the robot.

[0123] Step 242 converts each motion pose to display coordinates under the image displayed on the monitor.

[0124] Step 243 generates specific visualization information based on the display coordinates.

[0125] In one embodiment, the controller can obtain the robot's preset position (e.g., the robot's end effector) posture based on the operator's remote control operation mapping, and based on coordinate system transformation relationships, can convert the robot's end effector posture to the corresponding display coordinates under the display image; and assign specific visual marker information to the location of the display coordinates.

[0126] Furthermore, in one embodiment, step 243, which generates specific visualization information based on display coordinates, may include the following method steps:

[0127] Step 2431 generates specific visualization information based on the display coordinates of the robot joints.

[0128] This application embodiment transforms each motion posture into display coordinates under the image displayed on the monitor, and generates specific visualization information based on the display coordinates, thereby facilitating a more accurate visualization of the robot's predicted motion trajectory characteristics.

[0129] like Figure 2 As shown, in one embodiment, the controller can obtain the robot joint posture based on the operator's remote control operation mapping, and based on coordinate system transformation relationships, etc., can transform the robot joint posture to the corresponding display coordinates under the display image; and connect the display coordinates of two adjacent joints with a specific visual connector; and assign specific visual marking information to the location of the display coordinates and the connector.

[0130] Furthermore, in one embodiment, step 243 may include the following method steps:

[0131] Step 2432 connects the display coordinates of two adjacent robot joints using a specific visual connector.

[0132] Step 2434 generates specific visualization information based on the display coordinates and connectors of the robot joints.

[0133] The embodiments of this application are based on a preset visualization mapping model, which can convert the posture of robot joints into the corresponding display coordinates on the monitor; and use a specific visualization connector to connect the display coordinates of two adjacent joints, thereby facilitating more accurate visualization of the robot's predicted motion trajectory.

[0134] In an optional embodiment, the teleoperation tracking method described in this application may further include the following steps:

[0135] Step 270: Obtain the teleoperation trajectory of the robot during the original teleoperation tracking process.

[0136] Step 280 generates visualization information of the teleoperation trajectory.

[0137] In this embodiment of the application, in addition to displaying the visualization information of the predicted trajectory on the display, the visualization information of the teleoperation trajectory can also be displayed.

[0138] Furthermore, in an optional embodiment, after step 280, the following steps are also included:

[0139] The visualization information of the teleoperation trajectory is sent to the display so that the visualization information of the teleoperation trajectory can be displayed on the display.

[0140] This application embodiment provides reference personnel with more comprehensive visual information by comparing and displaying the teleoperation trajectory and the predicted trajectory on the same display, thereby helping operators to better perform other control operations in the future.

[0141] In an optional embodiment, before interrupting the original teleoperation tracking of the robot when it is determined that a delay has occurred in robot tracking, step 210 may further include the following steps:

[0142] Step 290 determines whether there is a delay in robot tracking.

[0143] Specifically, the controller can determine whether the robot is experiencing a delay based on various existing or future methods and procedures.

[0144] In one embodiment, step 290, determining whether robot tracking is delayed, may specifically include the following method steps:

[0145] Step 291: Obtain the robot target pose and the corresponding first timestamp based on the operator's pose mapping.

[0146] For example, taking a robot as a robotic arm, the human-robotic arm motion mapping module can obtain human joint information based on images collected and sent by image sensors, and draw the human skeleton; using any robotic arm as the object of teleoperation control, the pose of the end of the human arm and the rotation angle of each joint of the human arm are mapped to the end of the robotic arm and each joint of the robotic arm, respectively. The two mappings are combined together to finally obtain the target posture of the robotic arm.

[0147] Step 292: Obtain the robot's pose and the corresponding second timestamp.

[0148] In one embodiment, the controller retrieves the robot's posture from a memory or server according to a preset storage address.

[0149] Specifically, the aforementioned "robot's posture" can refer to the posture of the robot's end effector, the posture of various key parts of the robot (such as joints), or the posture of the entire robot represented by the position of the origin of a preset robot coordinate system, etc.

[0150] In one embodiment, the following method steps may be included before step 292:

[0151] In one embodiment, the controller can identify the robot's posture (e.g., the posture of each joint of the robot) based on images acquired by an image sensor; or it can determine the posture of the robot's end effector based on the motion of the robot joints acquired by an encoder and kinematic equations.

[0152] Step 293 determines whether the robot's pose is the same as the target pose; if they are not the same, repeat step 292 above to obtain the robot's pose and the corresponding second timestamp.

[0153] In one embodiment, the controller compares the target pose acquired at a first moment with the robot poses acquired sequentially in step 292 after the first moment until a robot pose that is the same as the target pose is obtained (for example, when the deviation between the two poses is less than a preset threshold, they are considered to be the same pose).

[0154] If step 294 is the same, obtain the second timestamp corresponding to when the robot's pose is the same as the target pose.

[0155] Step 295 determines whether a delay has occurred in robot tracking based on the time difference between the first and second timestamps.

[0156] For example, a time difference between the first and second timestamps can be set to be greater than or equal to a preset threshold (e.g., 10mm) as a delay in robot tracking.

[0157] In this embodiment, trajectory prediction is only performed when the delay reaches a certain preset requirement. This reduces unnecessary trajectory prediction and makes the trajectory prediction more targeted and effective.

[0158] In an optional embodiment, step 230 above, which generates motion prediction instructions based on the current target pose, may specifically include the following steps:

[0159] Step 231 performs trajectory planning based on the current target attitude to generate motion prediction instructions.

[0160] In one embodiment, the controller acquires the robot's current target pose, performs trajectory planning based on the current target pose, and generates motion prediction instructions.

[0161] For example, taking a robot as a manipulator, the current posture of the manipulator's end effector can be calculated based on kinematic equations. Then, with the current target posture as the target, a trajectory is planned to generate continuous motion prediction commands such as displacement / angular displacement, velocity / angular velocity, acceleration / angular acceleration, etc. of each joint of the robot. Thus, the manipulator is instructed to move according to the predicted trajectory through motion prediction commands.

[0162] In another optional embodiment, step 230 above, which generates motion prediction instructions based on the current target pose, may specifically include the following steps:

[0163] Step 232: Obtain the preset model;

[0164] Step 233: Based on the current target attitude, trajectory planning is performed in conjunction with a preset model to generate motion prediction instructions.

[0165] Specifically, the aforementioned model can be a pre-built simulated scenario in a teleoperation environment. Based on the simulated scenario, obstacles in the scenario can be obtained, thereby providing a reference for trajectory planning prediction and thus making teleoperation safer.

[0166] This application embodiment combines a preset model for trajectory planning and prediction, which can pre-analyze potential obstacles in the environment, thereby enhancing the safety of remote operation.

[0167] In an optional embodiment, the step 230 above, which generates motion prediction instructions based on the current target pose, may specifically include the following steps:

[0168] Step 234: Obtain the preset motion prediction model;

[0169] Step 235, based on the current target pose, outputs motion prediction instructions through the motion prediction model.

[0170] The embodiments of this application are based on artificial intelligence. Based on the input of the robot's current target posture and current posture, the above-mentioned motion prediction instructions can be directly output, thereby improving the generalization and robustness of trajectory prediction in various situations.

[0171] It should be noted that the motion prediction model described in this application embodiment can include any network composed of neurons capable of achieving the above functions, such as: Feed-Forward Networks, RNN, LSTM, Transformer, GNN, GAN, AE, MLP, Convolutional Neural Network (CNN). Common CNN models may include, but are not limited to: LeNet, AlexNet, ZFNet, VGG, GoogLeNet, Residual Net, DenseNet, R-CNN, SPP-NET, Fast-RCNN, Faster-RCNN, FCN, Mask-RCNN, YOLO, SSD, GCN, and other network model structures that are currently known or will be developed in the future.

[0172] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. This computer program can be stored in a computer-readable storage medium, and when executed, it can include the processes of the embodiments of the methods described above. The aforementioned storage medium can be a non-volatile storage medium such as a magnetic disk, optical disk, or read-only memory (ROM), or random access memory (RAM).

[0173] It should be understood that although the steps in the flowcharts of the accompanying figures are shown sequentially as indicated by the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the accompanying figures may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times, and their execution order is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the sub-steps or stages of other steps.

[0174] Further reference Figure 4 As a response to the above Figure 3 The implementation of the method shown in this application provides an embodiment of a teleoperated tracking device, which is similar to... Figure 3 Corresponding to the method embodiments shown, this device can be specifically applied to various controllers.

[0175] like Figure 4 As shown, the teleoperation tracking device 300 of this embodiment includes: a delay tracking module 320; the delay tracking module 320 includes:

[0176] The tracking interruption submodule 321 is used to interrupt the robot's original teleoperation tracking when it is determined that a delay has occurred in robot tracking;

[0177] The posture mapping submodule 322 is used to map the robot's current target posture based on the operator's current posture.

[0178] The instruction generation submodule 323 is used to generate motion prediction instructions based on the current target posture, so as to instruct the robot to move to the current target posture along the predicted trajectory;

[0179] The information visualization submodule 324 is used to generate specific visualization information for the predicted trajectory; specific visualization information refers to visualization information that is distinguishable from other visualization information.

[0180] In one embodiment, the teleoperation tracking device 300 further includes a teleoperation tracking submodule 310.

[0181] The teleoperation tracking submodule 310 is used to generate motion commands to instruct the robot to perform the original teleoperation tracking.

[0182] In an optional embodiment, the visualization submodule 324 includes:

[0183] The attitude acquisition unit is used to acquire multiple motion attitudes of the robot under the motion prediction command;

[0184] An image conversion unit is used to convert each motion posture to display coordinates under the image displayed on the monitor;

[0185] Visual generation unit, used to generate specific visual information based on display coordinates.

[0186] Furthermore, in an optional embodiment, the visual generation unit includes:

[0187] The first generation subunit is used to generate specific visualization information based on the display coordinates of the robot joints.

[0188] or

[0189] A connection generation subunit is used to connect the display coordinates of two adjacent robot joints using a specific visual connector;

[0190] The second generation subunit is used to generate specific visualization information based on the display coordinates and connectors of the robot joints.

[0191] In an optional embodiment, the teleoperation tracking device 300 further includes:

[0192] The trajectory acquisition submodule is used to acquire the teleoperation trajectory of the robot during the original teleoperation tracking process;

[0193] The visual generation submodule is used to generate visualization information of teleoperation trajectories.

[0194] In an optional embodiment, the teleoperation tracking device 300 further includes an attitude determination submodule.

[0195] Furthermore, in an optional embodiment, the attitude determination submodule includes:

[0196] The first acquisition unit is used to acquire the robot's target pose and the corresponding first timestamp;

[0197] The second acquisition unit is used to acquire the robot's posture and the corresponding second timestamp;

[0198] The posture determination unit is used to determine whether the robot's posture is the same as the target posture; if they are not the same, the step of obtaining the robot's posture and the corresponding second timestamp is repeated.

[0199] The third acquisition unit is used to acquire the second timestamp corresponding to the robot's posture being the same as the target posture if they are identical.

[0200] The delay judgment unit is used to determine whether a delay has occurred in robot tracking based on the time difference between the first timestamp and the second timestamp.

[0201] In an optional embodiment, the teleoperation tracking device 300 further includes:

[0202] An image sending submodule is used to send visualization information to a display for display of the visualization information; and / or

[0203] The tracking continuation submodule is used to continue the original teleoperation tracking of the robot.

[0204] In an optional embodiment, the attitude mapping submodule 322 includes:

[0205] The current acquisition unit is used to acquire the current motion data of the operator's key parts in the current posture, which are collected and transmitted by the attitude sensor.

[0206] The current conversion unit is used to convert the current motion data into the robot's current target pose.

[0207] In another alternative embodiment, the attitude mapping submodule 322 includes:

[0208] The image acquisition unit is used to acquire the current operator image captured by the image sensor;

[0209] An attitude recognition unit is used to recognize the operator's current attitude based on an image of the operator.

[0210] The attitude conversion unit is used to convert the current attitude into the robot's current target attitude.

[0211] In an optional embodiment, the instruction generation submodule 323 includes:

[0212] The first generation unit is used to perform trajectory planning based on the current target attitude in order to generate motion prediction instructions.

[0213] In another alternative embodiment, the instruction generation submodule 323 includes:

[0214] The first acquisition unit is used to acquire a preset model;

[0215] The second generation unit is used to perform trajectory planning based on the current target posture and a preset model to generate motion prediction instructions.

[0216] In another alternative embodiment, the instruction generation submodule 323 includes:

[0217] The second acquisition unit is used to acquire a preset motion prediction model;

[0218] The result output unit is used to output motion prediction instructions based on the current target pose and the motion prediction model.

[0219] To address the aforementioned technical problems, embodiments of this application also provide a computer device. Please refer to [link / reference needed]. Figure 5 , Figure 5 This is a basic structural block diagram of one embodiment of the computer device of this application.

[0220] The computer device 6 includes a memory 61, a processor 62, and a network interface 63 that are interconnected via a system bus. It should be noted that only the computer device 6 with components 61-63 is shown in the figure; however, it should be understood that it is not required to implement all the shown components, and more or fewer components can be implemented alternatively. Those skilled in the art will understand that the computer device described here is a device capable of automatically performing numerical calculations and / or information processing according to pre-set or stored instructions, and its hardware includes, but is not limited to, microprocessors, application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), digital signal processors (DSPs), embedded devices, etc.

[0221] The computer device can be a desktop computer, laptop, handheld computer, or cloud server, etc. The computer device can interact with the user via a keyboard, mouse, remote control, touchpad, or voice control.

[0222] The memory 61 includes at least one type of readable storage medium, including flash memory, hard disk, multimedia card, card-type memory (e.g., SD or DX memory), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, magnetic disk, optical disk, etc. In some embodiments, the memory 61 may be an internal storage unit of the computer device 6, such as the hard disk or memory of the computer device 6. In other embodiments, the memory 61 may also be an external storage device of the computer device 6, such as a plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, etc., equipped on the computer device 6. Of course, the memory 61 may include both the internal storage unit and its external storage device of the computer device 6. In this embodiment, the memory 61 is typically used to store the operating system and various application software installed on the computer device 6, such as the program code of the teleoperation tracking method. In addition, the memory 61 can also be used to temporarily store various types of data that have been output or will be output.

[0223] In some embodiments, the processor 62 may be a central processing unit (CPU), a controller, a microcontroller, a microprocessor, or other data processing chip. The processor 62 is typically used to control the overall operation of the computer device 6. In this embodiment, the processor 62 is used to run program code stored in the memory 61 or process data, for example, to run the program code of the teleoperation tracking method.

[0224] The network interface 63 may include a wireless network interface or a wired network interface, which is typically used to establish communication connections between the computer device 6 and other electronic devices.

[0225] This application also provides another embodiment, namely, a computer-readable storage medium storing a teleoperable tracking program, which can be executed by at least one processor to cause the at least one processor to perform the steps of the teleoperable tracking method as described above.

[0226] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes several instructions to cause a terminal device (which may be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0227] Obviously, the embodiments described above are only some embodiments of this application, not all embodiments. The accompanying drawings show preferred embodiments of this application, but do not limit the patent scope of this application. This application can be implemented in many different forms; rather, the purpose of providing these embodiments is to provide a more thorough and comprehensive understanding of the disclosure of this application. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing specific embodiments, or make equivalent substitutions for some of the technical features. Any equivalent structures made using the content of this application's specification and drawings, directly or indirectly applied to other related technical fields, are similarly within the scope of patent protection of this application.

Claims

1. A teleoperation tracking method, characterized in that, Includes the following steps: Obtain the robot's target pose and its corresponding first timestamp; Obtain the robot's pose and its corresponding second timestamp; Determine whether the robot's pose is the same as the target pose; if they are not the same, repeat the step of obtaining the robot's pose and the corresponding second timestamp. If they are the same, obtain the second timestamp corresponding to when the robot's posture is the same as the target posture; Based on the time difference between the first timestamp and the second timestamp, determine whether a delay has occurred in robot tracking; If a delay is detected in robot tracking, the original teleoperation tracking of the robot is interrupted; The current target pose of the robot is mapped based on the operator's current pose; Based on the current target posture, a motion prediction instruction is generated to instruct the robot to move to the current target posture along a predicted trajectory; Generate specific visualization information of the predicted trajectory; The specific visualization information refers to visualization information that is distinguishable from other visualization information.

2. The teleoperation tracking method according to claim 1, characterized in that, The specific visualization information for generating the predicted trajectory includes the following steps: Obtain multiple motion postures of the robot under the motion prediction command; Each of the aforementioned motion postures is converted to display coordinates under the image displayed on the monitor; The specific visualization information is generated based on the display coordinates.

3. The teleoperation tracking method according to claim 2, characterized in that, Generating the specific visualization information based on the display coordinates includes the following steps: Based on the displayed coordinates of the robot joints, the specific visualization information is generated; or Connect the display coordinates of two adjacent robot joints using a specific visual connector; The specific visualization information is generated based on the display coordinates of the robot joints and the connector.

4. The teleoperation tracking method according to claim 1 or 2, characterized in that, The method further includes the following steps: Obtain the teleoperation trajectory of the robot during the original teleoperation tracking process; Generate visualization information of the teleoperation trajectory.

5. The teleoperation tracking method according to claim 2, characterized in that, It also includes the following steps: The specific visualization information is sent to the display so that the specific visualization information can be displayed on the display. and / or After generating motion prediction instructions based on the current target pose, the method further includes: Continue the original remote operation tracking of the robot.

6. The teleoperation tracking method according to claim 1 or 2, characterized in that, The process of mapping the robot's current target pose based on the operator's current pose includes the following steps: Acquire current motion data of key parts of the operator in the current posture, collected and transmitted by the attitude sensor; The current motion data is converted into the robot's current target pose; or Acquire the current operator image captured by the image sensor; The operator's current posture is identified based on the operator's image; The current pose is converted into the robot's current target pose.

7. The teleoperation tracking method according to claim 1 or 2, characterized in that, The process of generating motion prediction instructions based on the current target posture includes the following steps: Trajectory planning is performed based on the current target posture to generate the motion prediction command; or Obtain the preset model; Based on the current target posture, trajectory planning is performed using the preset model to generate the motion prediction command; or Obtain the preset motion prediction model; Based on the current target posture, the motion prediction command is output through the motion prediction model.

8. A remotely operated tracking device, characterized in that, The device includes: an attitude determination submodule and a delay tracking module; The attitude determination submodule includes: The first acquisition unit is used to acquire the robot's target pose and the corresponding first timestamp; The second acquisition unit is used to acquire the robot's posture and the corresponding second timestamp; The posture determination unit is used to determine whether the robot's posture is the same as the target posture; if they are not the same, the step of obtaining the robot's posture and the corresponding second timestamp is repeated. The third acquisition unit is used to acquire the second timestamp corresponding to the robot's posture being the same as the target posture if they are identical. The delay judgment unit is used to determine whether a delay has occurred in robot tracking based on the time difference between the first timestamp and the second timestamp. The delay tracking module includes: The tracking interruption submodule is used to interrupt the robot's original teleoperation tracking when a delay is detected in the robot's tracking. The posture mapping submodule is used to map the robot's current target posture based on the operator's current posture. The instruction generation submodule is used to generate motion prediction instructions based on the current target posture, so as to instruct the robot to move to the current target posture along a predicted trajectory; The information visualization submodule is used to generate specific visualization information of the predicted trajectory; the specific visualization information refers to visualization information that is distinguishable from other visualization information.

9. A teleoperation system, characterized in that, The system includes: an attitude sensor, a robot, a display, and a controller; The attitude sensor, robot, and display are respectively communicatively connected to the controller; The posture sensor is used to collect the operator's motion data and send the motion data to the controller; The controller is used to generate motion commands based on the motion data to instruct the robot to perform remote tracking. The system is used to perform the following steps: acquiring the target pose of the robot and its corresponding first timestamp; acquiring the pose of the robot and its corresponding second timestamp; determining whether the robot's pose is the same as the target pose; if they are different, repeating the step of acquiring the robot's pose and its corresponding second timestamp; if they are the same, acquiring the second timestamp corresponding to when the robot's pose is the same as the target pose; and determining whether robot tracking is delayed based on the time difference between the first timestamp and the second timestamp. When a delay is detected in the robot tracking, the original teleoperation tracking of the robot is interrupted; the current target posture of the robot is mapped based on the operator's current posture; a motion prediction command is generated based on the current target posture to instruct the robot to move to the current target posture along a predicted trajectory; specific visualization information of the predicted trajectory is generated; the specific visualization information refers to visualization information that is distinguishable from other visualization information. The display is used to show the specific visual information.

10. A computer device comprising a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps of the teleoperation tracking method as described in any one of claims 1 to 7.

11. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the steps of the teleoperation tracking method as described in any one of claims 1 to 7.

Images