Robot teleoperation control method and system, medium, equipment and program product

Abstract

The embodiment of the invention provides a robot teleoperation control method and system, a medium, equipment and a program product. The method comprises the steps that multi-degree-of-freedom control input information of a plurality of operable parts from teleoperation equipment is received; based on the multi-degree-of-freedom control input information, generating a control instruction for controlling each degree of freedom in the controllable component; sending the control instruction to at least one robot so as to control the corresponding degree of freedom of a controllable part of the robot to execute an operation corresponding to the control instruction; wherein the number of the robots is multiple, and a plurality of operable parts of the teleoperation equipment are operated by one operator; or the number of the robot is one, and a plurality of operable parts of the teleoperation equipment are operated by a plurality of operators. The number of the robots and the number of the operators can have a one-to-many or many-to-one corresponding relation, the requirement for a single operator is lowered, and the flexibility of teleoperation is improved.

Description

Technical Field

[0001] The embodiments of the present application relate to the technical field of robot teleoperation, and in particular to a robot teleoperation control method, system, medium, device, and program product. Background Art

[0002] Robot teleoperation technology involves an operator remotely controlling a robot to perform tasks in real time, and can be applied to fields such as industrial manufacturing, medical treatment, aerospace, and nuclear energy. Through human-computer interaction, data transmission, and control algorithms, the operator can precisely operate the robot in complex or dangerous environments, improving operation safety and efficiency. However, the related robot teleoperation control methods have high requirements for the operator and lack flexibility.

[0003] Based on this, the embodiments of the present application provide a robot teleoperation control method, system, medium, device, and program product to improve the related technology. Summary of the Invention

[0004] The purpose of the embodiments of the present application is to provide a robot teleoperation control method, system, medium, device, and program product, which reduces the requirements for the operator and improves the flexibility of teleoperation.

[0005] The purpose of the embodiments of the present application is achieved by adopting the following technical solutions:

[0006] In a first aspect, the embodiments of the present application provide a robot teleoperation control method, the method including: receiving multi-degree-of-freedom control input information of a plurality of operable components from a teleoperation device, the plurality of operable components being used to collect one or more of the operator's attitude information, position information, action information, and control input instructions, and the plurality of operable components being operated by one or more operators; generating control instructions for controlling each degree of freedom of the controllable components based on the multi-degree-of-freedom control input information; sending the control instructions to the at least one robot to control the corresponding degrees of freedom of the controllable components of the robot to perform operations corresponding to the control instructions; wherein, the robot includes one or more of a physical robot and a virtual robot in a simulation environment, the plurality of controllable components includes one or more of a head, a waist, a robotic arm, a chassis, and an end effector; the number of robots is multiple, and the plurality of operable components of the teleoperation device are operated by one operator; or, the number of robots is one, and the plurality of operable components of the teleoperation device are operated by multiple operators.

[0007] In some possible implementation manners, generating control instructions for controlling each degree of freedom in the controllable component based on the multi-degree-of-freedom control input information includes: generating control instructions for controlling each degree of freedom in the controllable component based on the one-to-one mapping relationship between the control input information for each degree of freedom in the multi-degree-of-freedom control input information and each degree of freedom of multiple controllable components in at least one robot.

[0008] In some possible implementation manners, the mapping relationship corresponding to each degree of freedom of the head is represented by a linear formula; or, the mapping relationships corresponding to the degrees of freedom of the waist, the robotic arm, the chassis, and the end effector are represented by pose transformation matrices.

[0009] In some possible implementation manners, the method further includes: recording the control instructions and their corresponding status data, and storing them as historical control data; performing state reconstruction based on the historical control data to generate a robot motion trajectory for graphical display and storing it; when a playback instruction is received, playing back one or more of the control instructions, status data, and robot motion trajectory in the historical control data.

[0010] In some possible implementation manners, the method further includes: receiving a user instruction through a human-machine interface, and controlling the robot to perform one or more operations of reset, self-check, parameter reset, and running log clearing.

[0011] In some possible implementation manners, the method further includes: collecting the operating state of the robot; when the operating state is an abnormal state or an emergency stop instruction is received, controlling the robot to stop running.

[0012] In some possible implementation manners, the method further includes: setting a specified parameter value range for at least one operating parameter; when the parameter value of the operating parameter is not within the corresponding parameter value range, performing an alarm operation.

[0013] In some possible implementation manners, the method further includes: receiving and fusing environmental perception data of multiple perception types in the environment where the robot is located; the multiple perception types include at least two of vision, audition, and touch; performing three-dimensional reconstruction and monitoring of the environment where the robot is located based on the environmental perception data of the multiple perception types; providing one or more of the three-dimensional reconstruction results and monitoring results to an operator, or using them as inputs to an automatic decision-making model.

[0014] In some possible implementation manners, the method further includes: in the case where a control instruction is lost or the communication link between the teleoperation device and the robot is interrupted, performing one or more of the following processes: initializing the communication link; retransmitting uncompleted control instructions; reconstructing the context.

[0015] In a second aspect, an embodiment of the present application provides a robot teleoperation control system, the system comprising: at least one robot, the robot including one or more of a physical robot and a virtual robot in a simulation environment, the robot including one or more controllable components such as a head, a waist, a robotic arm, a chassis, and an end effector; a teleoperation device, including a plurality of operable components, the plurality of operable components being configured to collect one or more of the posture information, position information, motion information, and control input instructions of an operator, and the plurality of operable components being operated by one or more operators; a control device, configured to perform the following processing: receiving multi-degree-of-freedom control input information from the plurality of operable components of the teleoperation device; generating control instructions for controlling each degree of freedom of the controllable components based on the one-to-one mapping relationship between the control input information for each degree of freedom in the multi-degree-of-freedom control input information and each degree of freedom of the plurality of controllable components in at least one robot; and sending the control instructions to the at least one robot to control the corresponding degrees of freedom of the controllable components of the robot to perform operations corresponding to the control instructions.

[0016] In some possible implementation manners, the teleoperation device includes one or more of a virtual reality device, a multi-degree-of-freedom handle, and a mobile terminal device.

[0017] In some possible implementation manners, the teleoperation device further includes a plurality of environmental perception simulation components of multiple perception types, the multiple perception types including at least two of vision, audition, and touch; the environmental perception simulation components are configured to receive environmental perception data of corresponding perception types in the environment where the robot is located, and provide corresponding perception feedback to the operator to simulate visual images, sound information, or tactile feedback in the environment where the robot is located.

[0018] In a third aspect, an embodiment of the present application provides a computer-readable storage medium storing a computer program, and when the computer program is executed by a processor, the method according to any one of the above is implemented.

[0019] In a fourth aspect, an embodiment of the present application provides a computer device, the computer device including a memory and a processor, the memory storing a computer program, and when the processor executes the computer program, the method according to any one of the above is implemented.

[0020] In a fifth aspect, an embodiment of the present application provides a computer program product including a computer program, and when the computer program is executed by a processor, the method according to any one of the above is implemented.

[0021] Embodiments of the present application provide a robot teleoperation control method, system, medium, device, and program product. The method includes: receiving multi-degree-of-freedom control input information from multiple operable components of a teleoperation device, where the multiple operable components are used to collect one or more of the operator's posture information, position information, action information, and control input instructions, and the multiple operable components are operated by one or more operators; generating control instructions for controlling each degree of freedom in the controllable components based on the multi-degree-of-freedom control input information; sending the control instructions to at least one robot to control the corresponding degrees of freedom of the controllable components of the robot to perform operations corresponding to the control instructions; where the robot includes one or more of a physical robot and a virtual robot in a simulation environment, and the multiple controllable components include one or more of a head, waist, robotic arm, chassis, and end effector; the number of robots is multiple, and the multiple operable components of the teleoperation device are operated by one operator; or, the number of robots is one, and the multiple operable components of the teleoperation device are operated by multiple operators. Since the operable components can be operated by one or more operators, and the number of robots to be controlled can be one or more, the number of robots and the number of operators can have a one-to-many or many-to-one correspondence. For example, when there are more degrees of freedom to be controlled or the control is more complex, a working mode of multiple operators controlling a single robot can be adopted, and when the operator can operate proficiently, a working mode of a single operator controlling multiple robots can be adopted, reducing the requirements for a single operator and improving the flexibility of teleoperation. Since non-one-to-one scenarios can be supported, the collaboration efficiency of the operator and / or the operation efficiency of the robot can be improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] The embodiments of the present application will be further described below in conjunction with the accompanying drawings of the specification and the specific implementation manners.

[0023] Figure 1 is a flowchart showing a robot teleoperation control method provided by an embodiment of the present application.

[0024] Figure 2 is a flowchart showing another robot teleoperation control method provided by an embodiment of the present application.

[0025] Figure 3 is a flowchart showing another robot teleoperation control method provided by an embodiment of the present application.

[0026] Figure 4 is a flowchart showing another robot teleoperation control method provided by an embodiment of the present application.

[0027] Figure 5 is a flowchart showing another robot teleoperation control method provided by an embodiment of the present application.

[0028] Figure 6 It is a schematic flowchart of another robot teleoperation control method provided by an embodiment of the present application.

[0029] Figure 7 It is a schematic flowchart of another robot teleoperation control method provided by an embodiment of the present application.

[0030] Figure 8 It is a schematic diagram of modules of a robot teleoperation control system provided by an embodiment of the present application.

[0031] Figure 9 It is a structural block diagram of a computer device provided by an embodiment of the present application. Detailed implementation manners

[0032] Next, the technical solutions in the embodiments of the present application will be clearly and completely described in conjunction with the accompanying drawings in the embodiments of the present application. Obviously, the described implementation manners are only a part of the implementation manners of the embodiments of the present application, rather than all the implementation manners. Based on the implementation manners in the embodiments of the present application, all other implementation manners obtained by those skilled in the art without creative efforts belong to the scope protected by the embodiments of the present application.

[0033] In the description of the embodiments of the present application, it should be understood that the terms "first" and "second" are only used for description purposes, and cannot be construed as indicating or implying relative importance or implicitly specifying the quantity of the indicated technical features. Thus, the features defined with "first" and "second" may explicitly or implicitly include one or more of the described features. In the description of the embodiments of the present application, "a plurality" means two or more, unless otherwise specifically defined.

[0034] Through in-depth analysis of the relevant robot teleoperation control methods, the applicant found the following defects: 1) The problem of non-matching degree of freedom mapping: The relevant control methods usually adopt a simplified mapping model (such as a three-degree-of-freedom handle controlling a seven-degree-of-freedom robotic arm), while the number and type of degrees of freedom of the teleoperation device (such as a three-degree-of-freedom handle) and the robot (such as a seven-degree-of-freedom robotic arm) do not match, and there is a lack of a dynamic mapping strategy, resulting in the operator needing to manually switch modes to control different joints, and the action mapping error rate is relatively high; 2) Single-operator limitation: The existing teleoperation control methods usually only support a one-to-one scenario (such as a single operator can only control a single robot), and it is difficult to support non-one-to-one scenarios (such as multiple operators collaborating to control a single robot, a single operator controlling multiple robots, etc.).

[0035] Therefore, in order to reduce the above-mentioned problems of the mismatch of the degree-of-freedom mapping and the single-operator limitation, the embodiments of the present application provide a robot teleoperation control method. By receiving multi-degree-of-freedom control input information of multiple operable components from a teleoperation device, the multi-degree-of-freedom control input information includes control input information for each degree of freedom of the controllable components of the robot; using the one-to-one mapping relationship between the control input information and each degree of freedom of the multiple controllable components in at least one robot, that is, the control input information for each degree of freedom is independently mapped to the respective degrees of freedom of the controllable components, thereby reducing the drawback that the mapping relationship in related methods is single and cannot achieve refined cooperative control, realizing independent and precise control of the multi-degree-of-freedom components of the robot, and improving the operation flexibility (such as the coordinated movement of the robotic arm and the waist); in addition, since the above-mentioned operable components can be operated by one or more operators, and the number of robots to be controlled can be one or more, therefore, this method can also support non-one-to-one scenarios (such as multiple operators cooperatively controlling one robot, one operator controlling multiple robots, etc.), improving the cooperation efficiency of the operators and / or the operation efficiency of the robots.

[0036] See Figure 1 , Figure 1 is a schematic flowchart of a robot teleoperation control method provided by the embodiments of the present application.

[0037] Regarding the problem of the mismatch of the degree-of-freedom mapping and the above-mentioned single-operator limitation existing in the related teleoperation control method, the present application proposes a robot teleoperation control method. See Figure 1 , and this method includes steps S100 to S300.

[0038] Step S100: Receive multi-degree-of-freedom control input information of multiple operable components from a teleoperation device. Among them, the multiple operable components are used to collect one or more of the operator's attitude information, position information, action information, and control input instructions, and the multiple operable components are operated by one or more operators.

[0039] Step S200: Generate control instructions for controlling each degree of freedom in the controllable components based on the multi-degree-of-freedom control input information.

[0040] Step S300: Send the control instructions to at least one robot to control the corresponding degrees of freedom of the controllable components of the robot to perform the operations corresponding to the control instructions. Among them, the robot includes one or more of a physical robot and a virtual robot in a simulation environment, and the multiple controllable components include one or more of a head, a waist, a robotic arm, a chassis, and an end effector.

[0041] Wherein, the robot includes one or more of a physical robot and a virtual robot in a simulation environment, and the multiple controllable components include one or more of a head, a waist, a robotic arm, a chassis, and an end effector.

[0042] The number of the robots is multiple, and multiple operable components of the teleoperation device are operated by one operator; or, the number of the robots is one, and multiple operable components of the teleoperation device are operated by multiple operators.

[0043] In some embodiments, generating control instructions for controlling each degree of freedom of the controllable components based on the multi-degree-of-freedom control input information may include: generating control instructions for controlling each degree of freedom of the controllable components based on a one-to-one mapping relationship between the control input information for each degree of freedom in the multi-degree-of-freedom control input information and each degree of freedom of multiple controllable components in at least one robot.

[0044] In the above step S100, the teleoperation device may include one or more of a virtual reality device, a multi-degree-of-freedom handle, and a mobile terminal device. The teleoperation device includes multiple operable components. The multiple operable components are used to collect one or more of the posture information, position information, motion information, and control input instructions of the operator. The operable components may include one or more of corresponding sensors, physical buttons, and / or touch pads, etc., which are not specifically limited herein.

[0045] It can be understood that a virtual reality device refers to a device that generates a three-dimensional virtual environment through computer technology and enables users to immerse themselves in and interact with it using hardware devices such as sensors and display devices. The virtual reality device includes one or more of a head-mounted display, a projection system (such as a CAVE projection system, a large-scale virtual reality projection system), an interaction device (such as a data glove, a motion capture device), a sound device, and a modeling tracking system, etc. For example, a head-mounted display collects the operator's three-axis rotation data (pitch / yaw / roll), three-dimensional spatial displacement, and motion acceleration through its inertial measurement unit, etc., and the head-mounted display can collect the operator's voice commands, etc. through its audio input device.

[0046] It can be understood that a multi-degree-of-freedom handle, for example, refers to an input device that can be used in virtual reality, augmented reality, or mixed reality systems, capable of detecting movements in multiple directions, enabling users to perform precise interactive operations in three-dimensional space. For example, it can be a six-degree-of-freedom handle, etc. For example, the physical buttons of the multi-degree-of-freedom handle can collect the pressing state, pressure value, and triggering depth of the operator, the touchpad / touch panel of the multi-degree-of-freedom handle can collect the finger position, sliding direction, and touch state of the operator, the joystick of the multi-degree-of-freedom handle can collect the two-dimensional axial offset and pressing state of the operator, and the gesture / finger tracking sensor of the multi-degree-of-freedom handle can collect the states of the operator's finger bending, spreading, pinching, etc.

[0047] It can be understood that a mobile terminal device refers to a computer device that can be used while moving. For example, mobile terminal devices can include smartphones, tablets, laptops, smart terminals, in-vehicle computers, wearable devices, etc. For example, mobile terminal devices collect the touch coordinates, multi-touch trajectories, etc. of the operator through a touch display screen. Mobile terminal devices can establish a communication channel with a robot through their applications. Mobile terminal devices can collect one or more of the operator's posture information, position information, motion information, and control input instructions through their applications.

[0048] In some embodiments, a teleoperation device (such as a mobile terminal device) can have the function of connecting to and disconnecting from the robot body, and at the same time can display in real time the images captured by the robot's head and the received audio information, etc., facilitating the operator to remotely monitor the working state of the robot.

[0049] In some embodiments, a teleoperation device (such as a mobile terminal device) can be a key component of a robot teleoperation control system and have a variety of practical functions. For example, the mobile terminal device can be securely and stably connected to the robot body through a mobile application (APP) or other means, enabling the operator to remotely monitor the working state of the robot at any time and place through the mobile terminal device. At the same time, it can display in real time the images captured by the robot and the received audio information, etc., providing intuitive and comprehensive on-site feedback for the operator.

[0050] In some embodiments, the operator can realize the basic control and status monitoring of the robot through the human-machine interaction interface (such as collecting simple gesture, button operation information, etc.) on the teleoperation device (such as a mobile terminal device).

[0051] In some embodiments, the teleoperation device (such as a mobile terminal device) can also support multi-device connection, allowing multiple operators to connect to and monitor the same robot simultaneously, improving the collaboration efficiency.

[0052] In some embodiments, a teleoperation device (such as a mobile terminal device) may have a task recording and playback function, capable of recording key data and videos during the operation process, facilitating the operator to conduct post-event analysis and summary. In case of sudden dangerous situations, the operator can quickly trigger the emergency stop function through the human-machine interface (HMI layer) of the robot teleoperation control system or the human-machine interaction interface on the teleoperation device (such as an application on the mobile terminal), causing the robot to immediately stop all actions, effectively reducing the probability of safety accidents and ensuring the safety of personnel and equipment. When the robot has an abnormality or a fault, the operator can send a reset command through the human-machine interaction interface on the teleoperation device, causing the robot to re-initialize and return to the initial state, ensuring the stability and reliability of the system and providing guarantee for subsequent operations.

[0053] In the above step S100, the multi-degree-of-freedom control input information is collected by the operable components of the teleoperation device. The multi-degree-of-freedom control input information refers to a data set that can represent the operator's motion intention in the three-dimensional space or the control intention for the robot. The multi-degree-of-freedom control input information includes one or more of the operator's posture information, position information, motion information, and control input instructions.

[0054] Among them, the posture information is used to describe the angular orientation of the operator's body parts (such as the head, hand) or the corresponding control device (such as the teleoperation device) in the three-dimensional space. The posture information can be represented in the form of Euler angles (pitch, yaw, roll), etc.

[0055] The position information is used to characterize the absolute coordinates or relative coordinates of the operator's body parts or the corresponding control device (such as the teleoperation device) in the three-dimensional space. The motion information can represent the change in the motion state of the operator or the corresponding control device (such as the teleoperation device).

[0056] The motion information refers to the human motion data that can be parsed into the control instructions of the robot. The motion information has the following characteristics: spatio-temporal continuity and multimodality. Spatio-temporal continuity means that it contains the pose changes in the time series (such as the movement trajectory of the handle). Multimodality means that it can include discrete trigger instructions (such as button presses) and can also include continuous pose signals (such as hand rotation). The motion information can include basic motion data (such as the position of the handle, the rotation of the handle, the bending degree of the fingers, etc.). The motion information can also include high-order motion semantic data (such as the grasping intention, etc.). The motion information can be collected by corresponding sensors, etc.

[0057] A control input instruction refers to a discrete command triggered through a physical or virtual interface (such as button triggering, voice commands, gesture recognition). For example, a teleoperation device can integrate an operable component with intelligent voice recognition, and identify the operator's voice input information through this operable component, taking the voice input information as a control input instruction. The teleoperation device can also be provided with buttons with specified instruction functions. When the above buttons are pressed, corresponding control input instructions are triggered. The teleoperation device can also identify the operator's gestures through relevant operable components, taking the gestures as control input instructions. That is to say, the operator can control the basic actions and task execution of the robot through control input instructions (such as voice commands), improving the convenience and efficiency of operation. The robot teleoperation control system (such as a robot) can also feedback the state information and environmental data of the robot to the operator in the form of voice, realizing seamless communication between humans and machines.

[0058] In the above step S100, the multi-degree-of-freedom control input information can be received by the control layer or control device of the robot, etc., and no specific limitation is made here.

[0059] In the above step S100, the above-mentioned multiple operable components can come from one teleoperation device or multiple teleoperation devices. That is to say, an operator can operate one teleoperation device, or an operator can operate multiple teleoperation devices. It is also possible for multiple operators to operate multiple operable components of one teleoperation device, or multiple operators to operate multiple operable components of multiple teleoperation devices.

[0060] In the above embodiment, the mapping relationship can be pre-constructed. The mapping relationship can refer to the deterministic correspondence rule between the control input information for each degree of freedom and the responses of each degree of freedom of multiple controllable components in at least one robot. Through the above mapping relationship, the N-degree-of-freedom control quantities (such as the above-mentioned "control input information for each degree of freedom") of multiple operable components from the teleoperation device can be converted into control instructions or responses of K degrees of freedom of M controllable components of at least one robot. Among them, N, M, and K are positive integers.

[0061] Among them, the control input information for each degree of freedom of the head of one or more robots can be in one-to-one correspondence with the responses of each degree of freedom (such as the two degrees of freedom of pitch angle and yaw angle) of the head of one or more robots. The control input information for each degree of freedom of the waist of one or more robots can be in one-to-one correspondence with the responses of each degree of freedom of the waist of one or more robots, and so on.

[0062] When the above mapping relationship has been pre - constructed, when determining the control input information for the corresponding degree of freedom, the target parameters (such as target position or attitude parameters, etc.) of the corresponding degrees of freedom of one or more controllable components in the robot can be determined.

[0063] For example, after obtaining the pitch angle and yaw angle of the robot's head in the multi - degree - of - freedom control input information, the target pitch angle and target yaw angle of the robot's head can be obtained by using the mapping relationship corresponding to each degree of freedom of the head. Another example is that after obtaining the control input information for each degree of freedom of the robotic arm in the multi - degree - of - freedom control input information, the target position / attitude parameters of each degree of freedom of the robotic arm can be obtained by using the mapping relationship corresponding to each degree of freedom of the robotic arm. After that, the target parameters (such as target position or attitude parameters, etc.) can be converted into angle or displacement parameters in the joint space by means of inverse kinematics solution algorithms and other methods. Based on the above mapping relationship, control instructions for controlling each degree of freedom in the controllable components can be generated using existing related technologies. The specific generation process of the control instructions will not be elaborated here.

[0064] It can be understood that the above mapping relationship accurately converts the multi - degree - of - freedom control input information from the teleoperation device into control instructions for each degree of freedom in each controllable component of the robot.

[0065] In some embodiments, the "one - to - one mapping relationship" may not be a simple linear proportional relationship. The "one - to - one mapping relationship" means, for example, that the multi - degree - of - freedom control input information from multiple operable components provides control input information for N degrees of freedom, and the above - mentioned N degrees of freedom of control input information are used to control N degrees of freedom of the controllable components of the robot. Here, N is a positive integer. For example, if the operable components (buttons, knobs) can provide a total of 10 types of control input information, then these 10 types of control input information can control 10 degrees of freedom of the controllable components of the robot. For example, a head - mounted display can collect two attitude angles, and these two attitude angles can be used as two control input information to control two attitude angles of the robot's head in a one - to - one mapping manner. The "one - to - one mapping relationship" can also include the following meaning: kinematic decoupling (such as mapping only the translational input of a six - degree - of - freedom handle to the end - effector position of the robotic arm and mapping only the rotational input of the six - degree - of - freedom handle to the wrist of the robotic arm). The "one - to - one mapping relationship" can also include the following meaning: dynamic parameter adaptation (automatically adjusting the sensitivity of the mapping according to task requirements, such as reducing the displacement gain during fine operations).

[0066] In the above step S300, the virtual robot in the simulation environment refers to, for example, a virtual entity constructed by high-precision digital modeling and real-time simulation technology and having the same kinematic and dynamic characteristics as the above-mentioned physical robot. The control instruction can be sent to one robot or multiple robots. The specific process of "controlling the corresponding degrees of freedom of the controllable components of the robot to execute the operations corresponding to the control instructions" can directly adopt the existing technologies in the art, so it will not be elaborated here.

[0067] It can be seen that the above control method first receives multi-degree-of-freedom control input information from a teleoperation device (such as a head-mounted display), generates control instructions for controlling each degree of freedom in the controllable components, and drives the robot (such as a physical robot and / or a virtual robot) to execute operations through these control instructions. Since the mapping relationship between the above control input information for each degree of freedom and each degree of freedom of multiple controllable components in at least one robot is one-to-one, that is, the control input information for each degree of freedom is independently mapped to each degree of freedom of the controllable components respectively, thus reducing the drawback that the mapping relationship is single and refined cooperative control cannot be achieved in related methods, realizing independent and precise control of the multi-degree-of-freedom components of the robot, and improving operation flexibility (such as the coordinated movement of the robotic arm and the waist); in addition, since the above multiple operable components can be operated by one or more operators, the number of robots controlled by the operator can be one or more, that is, the above multi-degree-of-freedom control input information can establish a mapping relationship with the controllable components in one robot, or can establish a mapping relationship with the multiple controllable components in multiple robots. Therefore, this method can not only support a one-to-one scenario, but also support a non-one-to-one scenario (such as multiple operators cooperatively controlling one robot, one operator controlling multiple robots, etc.), improving the cooperation efficiency of the operator and / or the operation efficiency of the robot.

[0068] In some embodiments, when the number of robots is one, the multiple operable components of the teleoperation device can be operated by one operator.

[0069] In some embodiments, when the number of robots is multiple, the multiple operable components of the teleoperation device can be operated by multiple operators.

[0070] When the related robot teleoperation control method is applied to collaborative tasks that require centralized control (such as multiple robotic arms in a factory collaborating to carry large components), generally, an independent operator needs to be assigned to each robot, resulting in high labor costs and low collaboration efficiency. Therefore, in some embodiments, in the above step S100, when the number of robots is multiple, multiple operable components of the teleoperation device can be operated by one operator, that is, the multi-robot single-operator mode. For example, one operator can control multiple robots to execute corresponding action processes. For another example, one operator controls robot A to grasp the left side of a box and controls robot B to grasp the right side of the box at a preset offset to form collaborative handling. The multi-robot single-operator mode can also be implemented in other ways. It can be seen that if the above multi-robot single-operator mode is adopted, this method can significantly reduce labor costs and has high collaboration efficiency.

[0071] When the related robot teleoperation control method is applied to complex tasks that require multi-role collaboration (such as a surgeon controlling a robotic arm and an assistant controlling an endoscope in a surgical robot), it is impossible to achieve real-time fusion of multiple operation instructions and it is difficult to support multi-person collaborative operation of complex tasks. Therefore, in some embodiments, in the above step S100, when the number of robots is one, multiple operable components of the teleoperation device can be operated by multiple operators, that is, the single-robot multi-operator mode. For example, if the total number of degrees of freedom of multiple controllable components in a robot is a first quantity (such as 10), then the degrees of freedom of the first quantity (such as 10) of controllable components can be controlled by a second quantity (such as 2) of operators. For example, 2 operators respectively control 5 degrees of freedom. It can be seen that if the above single-robot multi-operator mode is adopted, this method can support multi-person collaborative operation of complex tasks (such as the division of labor between a doctor and a nurse in a medical robot).

[0072] The following uses several specific embodiments to illustrate the application of this control method in different scenarios.

[0073] For example, in an automobile manufacturing workshop, two operators need to cooperate to control a heavy robotic arm (physical robot) and a virtual robot (such as a digital twin) to complete precision assembly operations. Multiple operators can send multi-degree-of-freedom control input information to the robot through teleoperation devices. For example, operator A wears a head-mounted display and a six-degree-of-freedom handle. The viewing angle of the camera of the robotic arm is controlled by the head pose collected by the head-mounted display, and the motion information collected by the six-degree-of-freedom handle is mapped to the spatial pose of the end effector of the robotic arm; operator B sends control commands (such as gripper opening and closing, assembly path confirmation) through a tablet computer or intelligent terminal, and the control commands are transmitted to the corresponding control system of the robot through the same communication protocol. The virtual robot can synchronously receive the multi-degree-of-freedom control input information sent by operator B. After converting the multi-degree-of-freedom control input information into corresponding control commands, it can preview the assembly process according to the generated relevant control commands in the simulation environment, reducing the collision risk of the physical robot. The robotic arm of the physical robot can perform actual assembly actions, and the virtual robot can display the assembly effect in real time. When operator A triggers an emergency stop, the motion states of the physical robot and the virtual robot are frozen synchronously.

[0074] For another example, a doctor controls a physical robot (robotic arm + end surgical instrument) and a virtual robot (simulated patient tissue) locally to perform a remote operation on a patient in a remote area. The doctor controls the movement of each degree of freedom of the robotic arm of the physical robot through a force feedback handle, and adjusts the tissue parameters (such as blood flow simulation) of the virtual robot through voice commands; at the same time, relevant algorithms can be used to automatically filter out hand tremor noise. The pose of the end effector of the physical robot is strictly synchronized with that of the virtual robot. The tissue deformation data in the virtual environment is fed back to the force feedback handle in real time, and the doctor perceives the "virtual resistance". When the network delay is greater than a preset threshold (such as 100 ms), the virtual robot predicts the state for a certain period of time in the future (such as 0.1 second), and the physical robot moves in advance according to the prediction result. If the torque sensor of the physical robot detects abnormal resistance, a red warning in the virtual environment is immediately triggered.

[0075] For another example, an operator (such as a student) can use a virtual robot to perform archaeological excavation operations in a simulation environment to reduce the frequency of the operator's direct contact with cultural relics. Among them, the virtual robot can include a virtual robotic arm and a virtual brush head (simulating real archaeological tools). The operator can use a multi-degree-of-freedom handle to control the brush head of the virtual robot to contact different objects. When the brush head of the virtual robot contacts different materials (such as pottery / bronze), the vibration frequency of the multi-degree-of-freedom handle can change (for example, the vibration frequency is 50 Hz when contacting pottery and 100 Hz when contacting bronze) to achieve tactile feedback. The scratching sound emitted when the brush head contacts different materials can be synthesized in real time through the material acoustic library of the virtual robot to achieve auditory feedback. The virtual robot can also use lidar to perform three-dimensional reconstruction of the archaeological site where the virtual robot is located (such as simulating the three-dimensional reconstruction of stratified soil). State reconstruction can be performed based on historical control data to generate a robot motion trajectory for graphical display. When a playback instruction is received, the robot motion trajectory, control instructions, and state data in the historical control data can be played back to review the process of archaeological excavation and analyze operation errors. When the clamping force collected by the end effector (such as a gripper) of the virtual robot exceeds a preset threshold, a warning can be triggered to prevent the end effector from damaging cultural relics.

[0076] It can be seen that in some embodiments, the present control method can support one operator to control multiple robots simultaneously, or support multiple operators to jointly control a single robot, thereby improving the multi-robot collaboration efficiency (such as industrial assembly line division of labor) and supporting multiple people to cooperate in operating complex tasks. That is to say, the present method can not only support one-to-one scenarios, but also support non-one-to-one scenarios (such as multiple operators jointly controlling a single robot, one operator controlling multiple robots, etc.), improving the collaboration efficiency of operators and / or the operation efficiency of robots, and expanding the applicable scope of the present method.

[0077] Since the motion characteristics of the controllable components of different robots vary greatly, a single mapping method cannot meet the accuracy requirements. Therefore, in some embodiments, the mapping relationship corresponding to each degree of freedom of the head can be represented by a linear formula; or, the mapping relationship corresponding to each degree of freedom of the waist, robotic arm, chassis, and end effector can be represented by a pose transformation matrix.

[0078] In some embodiments, the attitude angle of the operator's head can be obtained in real time through a remote control device (such as a virtual reality device). For example, the attitude angle may include a pitch angle and a yaw angle; the head of the robot also has two degrees of freedom, pitch angle and yaw angle. It is understandable that the above attitude angle may also include a roll angle, and the head of the robot may also increase the degree of freedom of the roll angle accordingly. For example, in order to achieve precise control, the attitude angle of the operator's head obtained in real time by the remote control device may be mapped to the pitch angle and yaw angle corresponding to the robot's head. The linear formulas for the two degrees of freedom of the pitch angle and yaw angle of the head may be expressed as follows:

[0079] θ robot_pitch =k pitch ·θ VR_pitch +b pitch ;

[0080] θ robot_yaw =k yaw ·θ VR_yaw +b yaw ;

[0081] Among them, θ robot_pitch represents the target pitch angle of the robot head; θ VR_pitch represents the pitch angle obtained in real time by the teleoperation device (such as virtual reality device), k pitch represents θ VR_pitch The mapping gain coefficient is used to adjust θ robot_pitch and θ VR_pitch The sensitivity of the mapping between pitch represents the first offset, which is used to compensate for the difference between the initial pitch angle of the teleoperation device and the actual initial pitch angle of the robot head; θ robot_yaw represents the target yaw angle of the robot head, θ VR_yaw represents the yaw angle obtained in real time by the teleoperation device (such as virtual reality device), k yaw represents θ VR_yaw The mapping gain coefficient is used to adjust θ robot_yaw and θ VR_yaw The sensitivity of the mapping between yaw represents the second offset, which is used to compensate for the difference between the initial yaw angle of the teleoperated device and the actual initial yaw angle of the robot head. The robot can dynamically adjust the above k according to the actual scene requirements (such as the end load). pitch and k yaw For example, if the actual application scenario requires the robot head to move more sensitively, the above k can be appropriately increased. pitch and k yaw; If the actual application scenario requires the movement of the robot's head to be more stable, the above k can be appropriately reduced pitch and k yaw , so that the mapping relationships corresponding to each degree of freedom of the head can adapt to the requirements of different task scenarios (for example, high sensitivity is required for fine operations, and low sensitivity is required for fast movement). Those skilled in the art can determine the above b pitch and b yaw in advance (such as when initializing the corresponding control system) through calibration operations. For example, when the virtual reality device and the robot's head are both in the initial posture, record the differences between the pitch angle and yaw angle of the teleoperation device and the actual pitch angle and actual yaw angle of the robot's head at this time, and then calculate the corresponding above b pitch and b yaw .

[0082] It can be understood that the mapping relationships corresponding to each degree of freedom of the above head can also be represented by other forms of linear formulas, and no specific limitation is made here.

[0083] After the mapping relationship is pre-constructed, control instructions for controlling each degree of freedom of the robot's head can be generated based on the mapping relationships corresponding to each degree of freedom of the above head and the control input information. For example, the specific process of generating control instructions for controlling each degree of freedom (such as pitch angle and yaw angle) of the robot's head can be: 1) Data acquisition: Obtain real-time attitude information (such as pitch angle and yaw angle) through a teleoperation device (such as a virtual reality device); 2) Mapping calculation: Use the above linear formula and the above attitude information to obtain the target attitude information of the robot's head; Use the target attitude information to generate control instructions for controlling each degree of freedom (such as pitch angle and yaw angle) in the robot's head, so that the pitch angle and yaw angle of the robot's head are respectively equal to the above target pitch angle and target yaw angle; 3) Angle publishing: Transmit the above control instructions to the robot through the corresponding transmission path (such as the communication interface of the control system), and drive the robot's head to perform corresponding movements.

[0084] It can be seen that in some embodiments, by constructing an accurate and complex mapping relationship between the teleoperation device and the robot, comprehensively considering the robot kinematic model, dynamic characteristics and the operation degrees of freedom of the teleoperation device, the operator's subtle movements can be accurately converted into control instructions for the corresponding joints or components of the robot. In addition, through the above mapping relationship, an accurate mapping from the attitude information (such as pitch angle and yaw angle) obtained in real time from the teleoperation device (such as a virtual reality device) to each degree of freedom (such as pitch angle and yaw angle) in the robot's head can be realized, providing an intuitive and accurate control experience for the operator.

[0085] In some embodiments, the mapping relationships corresponding to the degrees of freedom of the robotic arm can be represented by a pose transformation matrix. The expression of this pose transformation matrix can be:

[0086] P robot_target = P robot_initial ·ΔP VR ;

[0087] where P robot_target represents the target pose data of the robotic arm, P robot_initial represents the initial pose data of the robotic arm, and ΔP VR represents the pose change data of the teleoperation device (such as a multi-degree-of-freedom handle). The pose change data can be obtained from the control input information for each degree of freedom of the robotic arm. The pose data can include one or more data among the three translational degrees of freedom and the three rotational degrees of freedom. And and P VR_initial represents the initial pose data of the teleoperation device (such as a multi-degree-of-freedom handle), and P VR_current represents the current pose data of the teleoperation device (such as a multi-degree-of-freedom handle).

[0088] In some embodiments, the mapping relationships corresponding to the degrees of freedom of the waist, chassis, and end effector can also be represented by a pose transformation matrix. That is to say, for multi-degree-of-freedom components such as the waist, chassis, and end effector (such as a gripper) of the robot, the mapping relationships of the operable components corresponding to the teleoperation device can be established respectively to ensure precise control of each degree of freedom. The specific form or acquisition method of the pose transformation matrix here can also refer to the specific form or acquisition method of the pose transformation matrix of the above-mentioned robotic arm, which will not be elaborated here.

[0089] It can be seen that in some embodiments, this control method can represent the mapping relationships corresponding to the degrees of freedom of the head by a linear formula, thereby making the control of the head simple and efficient, suitable for quick response, and significantly reducing the delay of head rotation; while for other complex controllable components (such as the waist and robotic arm), a pose transformation matrix can be used to achieve precise mapping of spatial motion, significantly reducing the risk of motion overrun of other complex controllable components.

[0090] The related teleoperation control methods also have problems of lack of post-analysis ability and insufficient real-time monitoring granularity; among them, the problem of lack of post-analysis ability means that the related methods usually only store the original control instructions (such as joint angle sequences), and it is difficult to intuitively restore the full state of the robot during the operation process (such as end effector speed, environmental interaction force); the problem of insufficient real-time monitoring granularity means that the operation interface usually only displays the current frame state, lacking continuous visualization of the historical trajectory, resulting in difficult traceability of misoperations. For the problems of lack of post-analysis ability and insufficient real-time monitoring granularity existing in the related teleoperation control methods, in some embodiments, seeFigure 2 The above method may include steps S100 to S400.

[0091] Step S400 may include: recording the control instruction and its corresponding status data, and storing them as historical control data; performing status reconstruction based on the historical control data to generate a robot motion trajectory for graphical display and storing it; when a playback instruction is received, playing back one or more of the control instruction, status data, and robot motion trajectory in the historical control data.

[0092] In some embodiments, in step S400, the above corresponding status data may include one or more of multi-modal data such as robot joint status, motor current, collision flag, etc. Those skilled in the art can also determine the specific categories of the above status data according to the actual application scenario. In addition, existing technologies (such as time series interpolation, forward dynamics calculation, and visualization rendering, etc.) can be used to perform status reconstruction based on the above historical control data to generate a robot motion trajectory for graphical display and store it. The playback instruction may come from a teleoperation device (such as a human-computer interaction interface on a mobile terminal, etc.). One or more of the control instruction, status data, and robot motion trajectory in the historical control data can be displayed by a teleoperation device (such as a head-mounted display) or other devices.

[0093] It can be seen that in some embodiments, the present control method can record the control instruction and its corresponding status data, store them as historical control data, generate a robot motion trajectory for graphical display based on the historical data, support playing back one or more of the control instruction, status data, and robot motion trajectory, thereby facilitating operation review and error analysis (such as industrial accident tracing), and providing data support for algorithm optimization.

[0094] Since there are problems of cumbersome operation and slow response when manual intervention is required for abnormal robot states in related teleoperation control methods, therefore, in some embodiments, referring to Figure 3 The above method may include steps S100 to S300 and step S500.

[0095] Step S500 may include: receiving a user instruction through a human-machine interface, and controlling the robot to perform one or more operations of reset, self-check, parameter reset, and running log clearing.

[0096] In some embodiments, in step S500, the above-mentioned human-machine interface may be a human-machine interaction interface on a teleoperation device. For example, it may be a human-machine interaction interface on a mobile terminal or a head-mounted display. The user instructions can be input in various forms. For example, they can be input by collected voice (such as "Execute emergency reset"), gestures (such as long-pressing the menu key of a six-degree-of-freedom handle), or clicking a button. The above-mentioned human-machine interface can also be a human-machine interface of a robot teleoperation control system. The above-mentioned reset operation can be used for one or more of the operations such as cutting off the motor power supply, restarting the control layer, reloading default parameters, and slowly returning the robotic arm to zero according to a preset trajectory. Those skilled in the art can also determine the specific content of the above-mentioned reset operation by themselves. The above-mentioned self-check operation can be used to automatically detect and evaluate the key components and functional states of itself through built-in hardware diagnosis and / or software algorithms. The core goal of the above-mentioned self-check operation is to ensure the reliability of the robot before performing tasks and to monitor potential faults in real time during operation. Those skilled in the art can also determine the specific content of the above-mentioned self-check operation by themselves. The above-mentioned parameter reset operation can be used to load the corresponding configuration from a preset parameter library according to the current task of the robot (such as "precision assembly" or "rapid handling"). Those skilled in the art can also determine the specific content of the above-mentioned parameter reset operation by themselves. The above-mentioned operation of clearing the operation log can be used to optimize the management of system resources, ensure compliance, and control security risks by selectively deleting or archiving historical log data. Those skilled in the art can also determine the specific content of the above-mentioned operation of clearing the operation log by themselves.

[0097] In some embodiments, the human-machine interface (HMI layer) may include a status display module. The status display module can be used for one or more of joint status display, position information display, power status display, and environmental monitoring display. For example, joint status display may refer to the real-time collection and display of the precise angles of each joint of the robot, enabling the operator to intuitively understand the posture and motion state of the robot. Position information display may refer to the real-time update and display of the precise position of the robot in the global coordinate system and the position information relative to the target object, ensuring that the operator has a comprehensive understanding of the position of the robot. Power status display may refer to the intuitive presentation of the motor speed of the robot to reflect its power output, helping the operator judge the operating power and load capacity of the robot. Environmental monitoring display can be used to effectively display the working environment parameters of the robot, such as temperature, humidity, air pressure, etc., enabling the operator to timely understand the changes in the environment where the robot is located.

[0098] In some embodiments, the human-machine interface (HMI layer) may include an interaction control module. The interaction control module may be used for one or more functions among historical record query, alarm threshold setting, system reset function, emergency stop function, system restart function, and operation pause function. Among them, the historical record query is used to provide the viewing function of the operation history. The operator can retrieve and view the past operation data at any time, which is convenient for summarizing experience and analyzing problems. The alarm threshold setting is used to allow the operator to set various alarm thresholds according to actual needs, such as high-temperature alarm, over-load alarm, etc. When the running state of the robot exceeds the set range, the system automatically triggers an alarm to remind the operator to take measures in time. The system reset function is used when the robot has an abnormality or a fault. The operator can send a reset command through the HMI layer to re-initialize the robot and restore it to the initial state, ensuring the stability and safety of the system. The emergency stop function may refer to that when the operator encounters an unexpected dangerous situation, he can quickly press the emergency stop button to make the robot immediately stop all actions, reducing the probability of safety accidents. The system restart function is used to support the restart operation of the robot system. The operator can restart the robot at any time according to needs to solve possible software failures or update system configurations. The operation pause function is used to allow the operator to pause the actions of the robot at any time during the operation to adjust or check the operation, ensuring the accuracy and safety of the operation.

[0099] It can be seen that in some embodiments, this control method receives user instructions through the human-machine interface and performs operations such as reset, self-check, and parameter reset, which can quickly restore system stability (such as reset when the joint is stuck) and reduce operation and maintenance costs (such as remotely clearing logs).

[0100] [[ID=,6]]Since the relevant teleoperation control method has the problems that in an emergency, the relevant emergency stop mechanism depends on a physical button, with a high response delay and a high false trigger rate, therefore, in some embodiments, referring to Figure 4 , the above method may include steps S100 to S300 and step S600.

[0101] Step S600 may include: collecting the running state of the robot; when the running state is an abnormal state or an emergency stop instruction is received, controlling the robot to stop running.

[0102] In some embodiments, in step S600, dynamic parameters and other data of the robot body and the environment can be collected in real time to form a state evaluation vector. The state evaluation vector can be used to characterize the operating state of the robot. Those skilled in the art can determine the abnormal determination criteria for the above state evaluation vector through relevant algorithms or by themselves. After the abnormal determination criteria are determined, the operating state of the robot can be directly determined according to the abnormal determination criteria to see if it is an abnormal state. For another example, an emergency stop instruction can come from a corresponding operable component of the teleoperation device (such as a microphone array receiving a voice instruction). Those skilled in the art can also determine the specific content of the above operating state by themselves. For example, it can include the basic operating state, abnormal state, collaborative robot state, network and communication state, etc.

[0103] In some embodiments, in step S600, the operating state of the robot can be collected by the hardware abstraction layer (HAL layer) of the robot teleoperation control system. That is to say, the hardware abstraction layer (HAL layer) can implement emergency stop services and recovery services to ensure that the robot can stop quickly when encountering sudden dangerous situations and resume normal operation quickly after the faults are cleared.

[0104] In some embodiments, in step S600, an emergency stop instruction can be issued by the human-machine interface of the teleoperation device (such as an application on a mobile terminal). An emergency stop instruction can also be issued by the human-machine interface (HMI layer) of the robot teleoperation control system. The emergency stop instruction can be received by the control layer of the robot teleoperation control system. That is to say, both the human-machine interface (HMI layer) of the robot teleoperation control system and the human-machine interface of the teleoperation device can implement the emergency stop function. When facing sudden dangers, the operator can also trigger this function in time to make the robot immediately stop the relevant actions, thereby reducing the probability of safety accidents. In addition, the control layer of the robot teleoperation control system can also have its own collision detection function, which can prevent collisions between different components of the robot itself, further enhancing the safety and stability of the robot in complex movements and providing all-round protection for the safe operation of the entire system.

[0105] It can be seen that in some embodiments, this control method can quickly cut off the power at the hardware layer by collecting the operating state in real time and immediately stopping the robot in the abnormal state or when an emergency stop instruction is received, reducing the collision risk and supporting software-triggered emergency stops.

[0106] Since the related teleoperation control method has the problem of lacking an active warning mechanism when the operating parameters of the robot exceed the limit, therefore, in some embodiments, referring to Figure 5 , the above method can include steps S100 to S300 and step S700.

[0107] Step S700 may include: setting a specified parameter value range for at least one operating parameter; and performing an alarm operation when the parameter value of the operating parameter is not within the corresponding parameter value range.

[0108] The purpose of the above "setting a specified parameter value range for at least one operating parameter" may be to establish a dynamically adjustable legal range for the relevant operating parameter. The operating parameter may include the joint angle of the robotic arm, the current / temperature range of the motor, etc., and no specific limitation is made here. Those skilled in the art can determine the "specified parameter value range of the operating parameter" and the "alarm operation" according to the actual application scenario. For example, the alarm operation may be an audible and visual prompt, or an emergency stop or speed limit at the control layer, etc.

[0109] It can be seen that in some embodiments, the present control method can set a specified parameter value range for at least one operating parameter (such as temperature, current), and trigger an alarm (such as an audible and visual prompt or speed limit at the control layer) when the parameter value of the operating parameter is not within the corresponding parameter value range, thereby achieving preventive maintenance (such as reducing the frequency before the motor overheats) and improving safety (such as protecting the grasping force threshold of the dexterous hand).

[0110] Due to the problem that a single sensor data in the related teleoperation control method cannot comprehensively perceive a complex environment (such as a nuclear power plant cannot identify radiation only by a camera), therefore, in some embodiments, referring to Figure 6 , the above method may include steps S100 to S300 and step S800.

[0111] Step S800 may include: receiving and fusing environmental perception data of multiple perception types of the environment where the robot is located; the multiple perception types include at least two of vision, audition, and touch; based on the environmental perception data of the multiple perception types, performing three-dimensional reconstruction and monitoring of the environment where the robot is located; and providing one or more of the three-dimensional reconstruction result and the monitoring result to the operator, or using them as inputs to an automatic decision-making model.

[0112] In some embodiments, in step S800, visual environmental perception data can be collected by a camera on a teleoperation device or a robot, etc. Auditory environmental perception data can be collected by a microphone array on a teleoperation device or a robot, etc. Tactile environmental perception data can be collected by a tactile sensor on a teleoperation device or a robot. Existing relevant fusion algorithms (such as multi-modal data fusion algorithms, etc.) can be used to fuse environmental perception data of multiple perception types in the environment where the robot is located, and then an environmental digital twin including attributes such as vision, audition, and touch can be constructed. Existing technologies (such as dynamic environment modeling, etc.) can be used to perform three-dimensional reconstruction and monitoring on the environment where the robot is located based on the above environmental perception data, and the specific process thereof will not be elaborated here. One or more of the three-dimensional reconstruction results and monitoring results can be provided to the operator through a display screen or a human-machine interface of the teleoperation device, etc. One or more of the three-dimensional reconstruction results and monitoring results can also be used as inputs to an automatic decision-making model. The automatic decision-making model can generate corresponding behavior strategies, etc. based on multi-source perception data (such as three-dimensional reconstruction results, monitoring results).

[0113] It can be seen that in some embodiments, this control method can fuse multi-modal data such as vision, audition, and touch, reconstruct a three-dimensional environmental model based on the fused data, and monitor the relevant results. The above results can be used for operator decision-making or automatic model input, thereby enhancing the immersion (such as tactile feedback combined with stereoscopic vision) and improving the reliability of environmental perception (such as multi-sensor redundancy in a noisy scenario).

[0114] Since there is a problem of high risk of robot out-of-control when the communication is interrupted in the relevant teleoperation control method, therefore, in some embodiments, referring to Figure 7 , the above method can include steps S100 to S300 and step S900.

[0115] Step S900 can include: in the case where the control instruction is lost or the communication link between the teleoperation device and the robot is interrupted, perform one or more of the following processes: initialize the communication link; resend the uncompleted control instructions; reconstruct the context.

[0116] In some embodiments, in step S900, for the above-mentioned initialization of the communication link, it can be performed by switching to a backup communication channel. For the above-mentioned retransmission of uncompleted control instructions, a certain number of the most recent instructions can be stored in the buffer at the sending end; when it is necessary to retransmit uncompleted control instructions, the control instructions to be retransmitted are taken out from the buffer at the sending end and retransmitted. Reconstructing the context may refer to dynamically reconstructing the complete spatio-temporal scenario during the operation of the robot (especially during a period before and after the loss of control instructions or the interruption of the communication link) by fusing historical control data, real-time sensor feedback, and environmental status information. Those skilled in the art can determine the specific ways of initializing the communication link, retransmitting uncompleted control instructions, and reconstructing the context by themselves. It is also possible to use existing technologies to implement the initialization of the communication link, the retransmission of uncompleted control instructions, and the reconstruction of the context.

[0117] In some embodiments, in step S900, when a control instruction is lost or the relevant communication link is interrupted, the "reconstructing the context" can be achieved through the following state prediction and compensation mechanism. For example, the state of the robot at the next moment can be predicted by a trained long short-term memory network (LSTM) model to reduce the impact of the delay caused by the loss of control instructions or the interruption of the communication link on the control accuracy of the robot. For example, assuming that the state sequence (such as head angle, arm joint angle, waist position and angle) of the robot at time t is s(t), and the length of its state sequence is T, then the expression of the historical state sequence input to the long short-term memory network model can be: S(t) = [s(t - T), s(t - T + 1),..., s(t - 1)], and the target predicted state output by the long short-term memory network model is s^(t). The expression of s^(t) can be: s^(t) = f(S(t), W, b); where S(t) represents the historical state sequence, f

[0118] (·) represents the long short-term memory network model, and W and b represent the weights and biases of the long short-term memory network model. W and b can be obtained through training and learning. The state sequence length T represents the size of the historical state window used to predict the current state, that is, the number of time steps traced back from the current moment t, or the length of the historical state sequence considered by the long short-term memory network model each time it makes a prediction. The state sequence length T determines how "far" back in history the long short-term memory network model can see for prediction.

[0119] In some embodiments, before using the above short-term memory network model, corresponding data preparation can be performed. The data preparation may include several steps such as data collection, preprocessing, normalization processing, and constructing a time series. Among them, data collection may include: collecting historical state data of the robot under different operations (such as state data corresponding to different control instructions). For example, the historical state data may include one or more of data such as the angle of the head, the angle of the arm joints, the position and angle of the waist, etc. Those skilled in the art can also determine the specific data categories included in the historical state data according to the actual application scenario. Preprocessing may include: preprocessing the collected historical state data (such as cleaning, etc.) to obtain the preprocessed historical state data to remove outliers and noise. Normalization processing may include: normalizing or standardizing the preprocessed historical state data to obtain the normalized or standardized historical state data to ensure its stability. Constructing a time series may include: converting the normalized or standardized historical state data into a time series form for training the long short-term memory network model.

[0120] In some embodiments, after the corresponding data preparation, the construction and training of the short-term memory network model can be carried out. The construction of the short-term memory network model may include: using the historical state sequence S(t) as the input of the short-term memory network model and using the target prediction state s^(t) as the output of the short-term memory network model. The target prediction state s^(t) is the predicted value of the future state of the robot. The training phase of the short-term memory network model may include: training the initial short-term memory network model using the historical state data obtained in the data preparation phase to obtain the trained short-term memory network model. Since the training process of the short-term memory network model belongs to the prior art in this field, it will not be elaborated here. After obtaining the trained short-term memory network model, the state data of the robot at the next moment can be predicted according to the current state data of the robot and the trained short-term memory network model.

[0121] In some embodiments, when the control instruction is lost or the communication link between the teleoperation device and the robot is interrupted (such as there is a delay in data transmission), the trained short-term memory network model can be used to predict the state data of the robot at the next moment, so as to make a control decision and compensate for the impact of the delay. For example, under normal circumstances, the robot executes according to the actual state; when the control instruction is lost or the communication link is interrupted, the state prediction compensation mechanism is started, that is, the prediction result (such as the state data at the next moment mentioned above) is obtained through the short-term memory network model. The prediction result can be used to control the robot instead of the actual state data. The state prediction compensation mechanism can handle the situation of lost control instructions or interrupted communication links, and achieve real-time compensation and state prediction. The robot can use the prediction result obtained by the short-term memory network model as a reference value to formulate corresponding control strategies in advance, and reduce the degradation of system performance caused by the delay.

[0122] It can be seen that in some embodiments, this control method can reconstruct and restore the control state by initializing the communication link, retransmitting uncompleted instructions, and / or reconstructing the context, so as to reduce the impact of communication delay / interruption and ensure task continuity (such as signal loss handling in space teleoperation).

[0123] It can be seen that in some embodiments, when the control instruction is lost or the relevant communication link is interrupted, this control method can estimate the possible state of the robot during the delay period based on the current state data of the robot and the above short-term memory network model, and then make corresponding control decisions in advance to reduce the negative impact of the delay on the system performance.

[0124] It can be seen that in some embodiments, this control method can establish an accurate and complex mapping relationship between the teleoperation device and the robot at the control logic level, and can map the corresponding control input information of the operator's head, waist, hand, etc. to the corresponding degrees of freedom of the controllable components of the robot. By comprehensively considering the kinematic model of the robot, dynamic characteristics and the operation degrees of freedom of the teleoperation device, the subtle movements of the operator can be accurately converted into control instructions for the corresponding controllable components (such as joints) of the robot, solving the deficiencies in the mapping relationship of related robot teleoperation control methods, especially the limitations in terms of degrees of freedom, and realizing more refined and flexible control of the robot.

[0125] It can be seen that in some embodiments, this control method can construct a highly immersive and intuitive operation environment for the operator through advanced virtual reality (VR) technology. The operator can control the robot precisely and in real time as if on the spot through the teleoperation device.

[0126] It can be seen that in some embodiments, at the execution feedback level, the robot in this control method can execute according to the control instructions output by the control layer or the control device and give corresponding feedback. The hardware abstraction layer (HAL layer) can obtain the state data of the robot's motors, temperature, joint angles, etc., and feedback it to the control layer or the control device to form a closed-loop control loop, which can ensure the accuracy and coherence of the entire operation process.

[0127] The above is the description of a robot teleoperation control method. Refer to Figure 8 , embodiments of the present application also provide a robot teleoperation control system. The above system includes at least one robot 100, a teleoperation device 200, and a control device 300. The robot 100 includes one or more of a physical robot and a virtual robot in a simulation environment. The robot 100 includes one or more controllable components such as a head, a waist, a robotic arm, a chassis, and an end effector. The teleoperation device 200 includes a plurality of operable components. The plurality of operable components are used to collect one or more of the posture information, position information, motion information, and control input instructions of the operator, and the plurality of operable components are operated by one or more operators. The control device 300 is used to perform the following processing: receiving multi-degree-of-freedom control input information from the plurality of operable components of the teleoperation device 200; generating control instructions for controlling each degree of freedom of the controllable components based on the one-to-one mapping relationship between the control input information for each degree of freedom in the multi-degree-of-freedom control input information and each degree of freedom of the plurality of controllable components in at least one robot 100; sending the control instructions to at least one robot 100 to control the corresponding degrees of freedom of the controllable components of the robot 100 to execute the operations corresponding to the control instructions.

[0128] The above physical robot serves as the core execution terminal of the teleoperation system. The physical robot can adopt a modular design concept, integrate a multi-degree-of-freedom motion mechanism and a high-precision sensing system, and have the ability to adapt to full-scenario tasks. The physical robot can include one or more controllable components such as a head, a waist, a robotic arm, a chassis, and an end effector. The head can support multiple degrees of freedom (such as pitching and yawing, etc.). The waist can also support multiple degrees of freedom. For example, the waist can support lifting and pitching adjustments. The chassis can move freely. The end effector can include an adaptive gripper or a biomimetic dexterous hand, etc. The physical robot is also equipped with a variety of precision actuators. For example, motors are used to drive the precise rotation of joints, thereby endowing the robot with the ability to complete fine operations such as grasping objects and moving in complex terrains. This design ensures that the physical robot can efficiently and accurately execute remote operation instructions and achieve the expected goals in diverse application scenarios.

[0129] In some embodiments, the present control system can also achieve precise control of the teleoperation device to the robot in the following ways. For example, it can subscribe to the data sent by the teleoperation device through multiple nodes (such as the position of the multi-degree-of-freedom handle, the state of the trigger and buttons, etc.). The calculation and publication of joint angles can be carried out. For example, according to the subscribed data above, combined with the mapping relationship and control algorithm, the corresponding joint angles are calculated and published to achieve real-time control of each controllable component of the robot. For example, precise control of the opening and closing action of the gripper to meet the operation requirements of grasping and releasing objects. For example, control of the body part of the robot: achieving comprehensive control of the body part of the robot, including adjustments in aspects such as position and posture, to adapt to different task scenarios.

[0130] In some embodiments, the head, waist, robotic arm, chassis, and end effector can belong to the execution layer of the present control system.

[0131] In some embodiments, the robotic arm can have seven degrees of freedom. For example, it can cover degrees of freedom such as abduction, adduction, and rotation of the shoulder, flexion and extension of the elbow, and flexion and extension and rotation of the wrist. Each joint of the robotic arm can be driven by a motor to accurately map the multi-degree-of-freedom control input information from the teleoperation device, realizing the generation and execution of complex spatial trajectories, and being suitable for high-precision operations in scenarios such as industrial assembly and medical surgery.

[0132] In some embodiments, the end effector can include a gripper and / or a dexterous hand. The gripper can be driven by a motor to achieve the opening and closing action. The dexterous hand can have the ability of multi-finger coordination and can imitate actions such as grasping and operating of the human hand. The dexterous hand can also support multiple operation modes, such as force control and position control, to meet different task requirements.

[0133] In some embodiments, the chassis can be used as the mobile platform of the robot. The chassis can support actions such as forward, backward, and rotation. The function of the chassis can be similar to that of an automated guided vehicle. The chassis can achieve precise movement control by driving the wheels with a motor. The chassis can also have path planning and obstacle avoidance functions to ensure the autonomous movement ability of the robot in a complex environment.

[0134] In some embodiments, the waist of the robot can have the functions of vertical lifting and pitch adjustment. The above-mentioned waist can be driven by a motor to achieve the overall height adjustment and bending action of the robot, providing flexibility for the robot's operation at different heights and angles.

[0135] In some embodiments, the head of the robot can have two degrees of freedom. The head of the robot can support actions such as looking up, looking down, and rotating left and right, and achieve omnidirectional vision adjustment through motor drive to ensure that the robot can monitor and observe the surrounding environment in real time.

[0136] In some embodiments, during the execution of control instructions, the robot does not merely passively complete the instructions but has the ability to monitor its own motion state in real time. The robot can obtain detailed execution feedback information through various built-in sensors, such as using joint encoders to measure joint angles and motor current sensors to sense the load conditions. On the one hand, this feedback information is used for the robot's own safety protection mechanism, such as adjusting actions in a timely manner or triggering an alarm when encountering abnormal resistance or overload. On the other hand, the feedback information can be promptly transmitted back to the control layer or control device to provide a basis for optimizing the control instructions issued by the control layer, forming a closed-loop control circuit to ensure the accuracy and coherence of the entire operation process.

[0137] In some embodiments, this control system may also include a Hardware Abstraction Layer (HAL layer). The Hardware Abstraction Layer can be tightly connected to the system bottom layer and the robot's hardware. The Hardware Abstraction Layer plays a fundamental and crucial role. The Hardware Abstraction Layer can be electrically connected to various sensors of the robot, and thus can collect relevant data of various sensors in real time and stably. For example, the Hardware Abstraction Layer can communicate with the motor driver to obtain operating parameters such as the rotational speed and current of the motor, so as to accurately grasp the power output state of the robot. The Hardware Abstraction Layer can collect relevant data of the temperature sensor and monitor the temperature changes of the key components of the robot to prevent overheating failures. The Hardware Abstraction Layer can collect relevant information of the joint angle sensor to determine the precise positions of the robot's joints. After preliminary sorting and format conversion, these status data are accurately fed back to the control layer or control device through a reliable communication interface, providing solid data support for the control layer or control device to comprehensively understand the operating conditions of the robot and formulate subsequent control strategies.

[0138] In some embodiments, the Hardware Abstraction Layer can publish joint states. For example, by setting up multiple joint state publishers, after configuration, they are responsible for publishing joint state information of components such as the robotic arm and end effector, covering key parameters such as joint angles, speeds, and torques, ensuring that the control layer obtains accurate joint operation data in real time.

[0139] [[ID=!0]]In some embodiments, the Hardware Abstraction Layer can publish other states. For example, by creating dedicated publishers, it is used to publish all-round information such as the state of the robot's neck, waist, chassis, battery, and overall state, enabling the control layer to comprehensively understand the operating conditions of each part of the robot and providing data support for comprehensive decision-making.

[0140] In some embodiments, the hardware abstraction layer can implement the acquisition and update of status data and the publication of real-time data. For example, it can regularly and frequently collect the status data of each device, covering sensor readings, device operation parameters, etc., and update them in a timely manner to the corresponding message objects. Subsequently, through an efficient communication mechanism, the newly collected status data can be quickly published to ensure the timeliness and accuracy of the information received by the control layer.

[0141] In some embodiments, the hardware abstraction layer can implement the reception and execution of control instructions. For example, it can receive control instructions from the control layer or a control device for components such as a robotic arm, an end effector, a head, a waist, etc. The received control instructions are accurately parsed and converted into specific control signals, which then drive the corresponding devices to accurately execute the specified actions, ensuring that the robot operates in accordance with the requirements of the control strategy.

[0142] In some embodiments, the hardware abstraction layer can implement an emergency control function. For example, when an emergency stop instruction is received, it immediately triggers the emergency stop mechanism of each device, quickly cuts off the power output, ensures that the robot stops safely within the shortest time, and avoids potential dangers. The hardware abstraction layer can also provide a recovery service to clear the emergency stop state, re-initialize the parameters and control logic of each device, and enable the device to resume normal operation and be able to accept control instructions and resume work again.

[0143] In some embodiments, the hardware abstraction layer can implement a fault clearing and recovery function. For example, it implements a fault clearing service to detect and clear the fault information of the corresponding device, enabling the device to work normally again. After the fault is cleared, the relevant devices and systems are restarted automatically or manually to restore their normal operating states and ensure the continuity of the robot operation.

[0144] In some embodiments, the hardware abstraction layer can implement start-up and stop services. For example, it provides start-up and stop services to control the start-up and stop of devices. During start-up, each device is initialized in a preset order to ensure that it smoothly enters the working state; during stop, the power and control connections of the corresponding devices are cut off in an orderly manner to avoid the impact and damage caused by sudden stops and extend the service life of the corresponding devices.

[0145] It should be noted that the relevant content of the robot and the teleoperation device here has been discussed in the previous robot teleoperation control method. The specific processes executed by the control device are also basically the same as the content of steps S100 to S300 in the previous text, so they will not be elaborated here. The technical effects that this robot teleoperation control system can achieve are also basically the same as those of the previous robot teleoperation control method.

[0146] In some embodiments, the teleoperation device can include one or more of a virtual reality device, a multi-degree-of-freedom handle, and a mobile terminal device.

[0147] In some embodiments, a virtual reality device can be used to receive and display visual images, audio information, etc. transmitted by a robot, providing an immersive visual experience for an operator. For example, the virtual reality device can include a head-mounted display (HMD). The head-mounted display can serve as the visual and auditory interface of the operator, capable of receiving and displaying high-resolution visual images and high-fidelity audio information transmitted by the robot, providing an immersive visual experience for the operator. The head-mounted display can be equipped with a high-resolution display screen and high-fidelity speakers to ensure that the operator can clearly see the details of the environment where the robot is located and hear the sounds in the environment. In addition, the head-mounted display can also support real-time video stream transmission to ensure the smoothness and low latency of the images, enabling the operator to respond to environmental changes in a timely manner. The head-mounted display can also have an environmental perception function, capable of identifying the lighting conditions and obstacle distribution in the environment where the robot is located, providing additional environmental information for the operator to assist in decision-making.

[0148] In some embodiments, a multi-degree-of-freedom handle can be used to capture one or more of the operator's posture information, position information, motion information, and control input commands, and transmit this data to a control device to achieve precise control of the robot. For example, a six-degree-of-freedom handle can be used to precisely capture one or more of the operator's posture information, position information, motion information, and control input commands, use this data as multi-degree-of-freedom control input information, and transmit it to the control device in real time to achieve precise control of the robot. The six-degree-of-freedom handle can adopt advanced sensor technology, capable of capturing the minute motions of the operator's hand with high precision and high frequency to ensure the accuracy and real-time nature of the subsequent generated control commands. The six-degree-of-freedom handle can also have a force feedback function, capable of transmitting the resistance and / or feedback force encountered by the robot during task execution to the operator, enhancing the intuitiveness and immersion of the operation. In addition, the six-degree-of-freedom handle also supports custom control modes, and the operator can customize the handle buttons and control parameters according to personal habits and task requirements to improve the flexibility and adaptability of the operation.

[0149] In some embodiments, the teleoperation device can also include environmental perception simulation components of multiple perception types, and the multiple perception types can include at least two of vision, hearing, and touch; the environmental perception simulation components are used to receive environmental perception data of the corresponding perception type in the environment where the robot is located and provide corresponding perception feedback to the operator to simulate visual images, sound information, or tactile feedback in the environment where the robot is located. For example, in the scenario of nuclear power plant inspection, the above visual images can be relevant images of radiation field visualization, and the above sound information can be an alarm prompt sound related to the abnormal state of the explosion-proof door.

[0150] It should be noted that the specific process of "receiving environmental perception data corresponding to the perception type in the environment where the robot is located and providing corresponding perception feedback to the operator to simulate visual images, sound information, or tactile feedback in the environment where the robot is located" has been discussed in the previous robot teleoperation control method, so it will not be elaborated here.

[0151] It can be seen that in some embodiments, the teleoperation device can integrate various environmental perception data such as vision, hearing, and touch, providing more comprehensive and rich environmental perception information to the operator to assist in decision-making and operation. By integrating various environmental perception data, the robot teleoperation control system can achieve three-dimensional reconstruction and real-time monitoring of the environment where the robot is located, further improving the safety and reliability of operation.

[0152] In some embodiments, the robot teleoperation control system may further include a data transmission layer. The data transmission layer is used to transmit the data acquired by the teleoperation device to the control device. The data transmission layer can exist in a virtual form.

[0153] In some embodiments, the data transmission layer may include a data acquisition module and a data preprocessing module. The data acquisition module is used to continuously acquire real-time relevant data (such as one or more of the above-mentioned attitude information, position information, action information, and control input instructions) from the teleoperation device. The data acquisition module can support multiple data input devices to ensure compatibility and expandability. The data preprocessing module can be used to preprocess the relevant data acquired by the data acquisition module to remove the noise therein and ensure the accuracy and integrity of the data. The data preprocessing module can support data encryption to ensure data security.

[0154] In some embodiments, the data preprocessing module can combine the sliding average filtering algorithm and the Kalman filtering algorithm for preprocessing to effectively remove noise and ensure the accuracy and integrity of the data. The following is an introduction to them in combination with the relevant expressions of these two filtering algorithms.

[0155] The basic principle of the sliding average filtering algorithm is: to smooth the data by averaging multiple consecutive data points. The sliding average filtering algorithm can be used to remove random noise. The data y i after filtering using the sliding average filtering algorithm can be expressed as:

[0156]

[0157] where, x1, x2, …, x n represents the data sequence input to the data preprocessing module, and N represents the window size.

[0158] The Kalman filter algorithm is a recursive minimum variance estimation algorithm. The Kalman filter algorithm can be used for state estimation of dynamic systems and can effectively remove Gaussian noise. The core formulas of the Kalman filter algorithm include two steps: prediction and update. Among them, the expression of the prediction step can be: The expression of the update step can be: Among them, is the state estimate at time k, P k|k is the covariance of the estimation error, K k is the Kalman gain, F k is the state transition matrix, B k is the control input matrix, u k is the control input, Q k is the covariance matrix of the process noise, H k is the observation matrix, R k is the covariance matrix of the observation noise, z k is the observation value.

[0159] Specifically, the data preprocessing module can first perform preliminary smoothing on the original data (such as one or more of the above-mentioned attitude information, position information, motion information, and control input commands) using moving average filtering to remove high-frequency noise; then use the result of the moving average filtering as the input of the Kalman filter, and utilize the dynamic characteristics of the Kalman filter to further optimize the estimated value, adapt to the dynamic changes of the system, and remove the remaining noise. For example, first perform moving average filtering on the original data x1, x2, …, x n to obtain the preliminary smoothed data y1, y2, …, y n ; then, use the data y i after moving average filtering as the observation value z k of the Kalman filter, perform Kalman filter processing, and obtain the finally filtered data By combining moving average filtering and the Kalman filter, the advantages of the two filtering algorithms can be fully utilized to achieve more effective noise removal and data smoothing, and improve the performance and reliability of the robot teleoperation control system.

[0160] In some embodiments, the data transmission layer can stably and efficiently send the processed data to the control device according to a pre-established communication protocol. The data transmission layer can monitor the transmission status in real time, automatically adjust the transmission parameters, and optimize the transmission effect. That is to say, the data transmission layer can use a variety of data processing algorithms and technologies to preprocess the raw data. For example, it can filter out noise interference to ensure the accuracy and integrity of the data; it can perform data compression to adapt to the limited transmission bandwidth and reduce the transmission delay. Then, the processed data is sent to the control layer or the control device stably and efficiently according to the pre-established communication protocol, ensuring that the control instructions can reach the subsequent processing links in a timely and accurate manner.

[0161] It can be seen that in some embodiments, the data transmission layer of this control system can adopt a variety of advanced technical means, such as an efficient inverse kinematics (IK) solution algorithm, a high-speed data transmission protocol, and a filtering algorithm for filtering out noise interference. The application of these algorithms ensures that the raw data can be preprocessed quickly and accurately, and is stably and efficiently transmitted to the control layer or the control device according to the pre-established communication protocol, greatly reducing the transmission delay and ensuring the timely and accurate arrival of the control instructions, thus significantly improving the real-time performance and accuracy of the operation.

[0162] It can be seen that in some embodiments, this control system can not only ensure the real-time transmission of control signals, but also optimize the operation process with the help of relevant intelligent algorithms, greatly expanding the application potential of robots in complex and dangerous environments, bringing an innovative technological breakthrough to the field of robot operation, and can be widely applied to multiple fields such as industry, military, aerospace, and medical, especially suitable for complex and dangerous environments that require remote precise operation.

[0163] It can be seen that in some embodiments, this control system can use virtual reality technology to create a highly immersive and intuitive operation environment for the operator. Through the teleoperation device, the operator can precisely and real-time control the remote robot. This control system can cover multiple-layer architectures such as data transmission, control logic, and execution feedback, and each layer works together to ensure the real-time transmission of control signals.

[0164] The above is the description of the robot teleoperation control system. The embodiments of the present application also provide a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, it implements any one of the robot teleoperation control methods in the above embodiments.

[0165] The embodiments of the present application also provide a computer program product, which includes a computer program. When the computer program is executed by a processor, it implements any one of the robot teleoperation control methods in the above embodiments.

[0166] A computer program product may be a portable compact disc read-only memory (CD-ROM) and include program code, and may run on a terminal device, such as a personal computer. However, the computer program product of the embodiments of the present application is not limited thereto, and the computer program product may adopt any combination of one or more computer-readable media.

[0167] See Figure 9 , Figure 9 which is a structural block diagram of a computer device provided by an embodiment of the present application.

[0168] The embodiments of the present application also provide a computer device. The computer device includes a memory and a processor. The memory stores a computer program, and when the processor executes the computer program, the above-mentioned robot teleoperation control method of any one of the embodiments is implemented.

[0169] The embodiments of the present application do not limit the computer device, and it may be, for example, a local computer device, a cloud computer device, a distributed computer device, etc.

[0170] The computer device may include: a memory 110, a processor 120, and a communication interface 130. Among them, the memory 110, the processor 120, and the communication interface 130 are connected through an internal connection path.

[0171] The memory 110 is used to store a computer program. In some implementation manners, the computer program may include code for implementing the method of the embodiments of the present application.

[0172] The processor 120 is used to execute the computer program stored in the memory 110 to control the communication interface 130 to receive input data and information and output operation result data, etc. In some implementation manners, when implementing the solution of the embodiments of the present application through software or firmware, the computer program for implementing the solution of the embodiments of the present application may be stored in the processor 120 and executed by the processor 120.

[0173] The memory 110 can be a volatile memory, a non-volatile memory, or can include both volatile and non-volatile memories. Among them, the non-volatile memory can be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory can be a random access memory (RAM). It should be noted that the memory 110 described herein is intended to include, but is not limited to, any of these and other suitable types of memories. As an example, the memory 110 includes a random access memory (RAM), a cache memory, and a read-only memory (ROM). Among them, the memory 110 stores a computer program, and the computer program can be executed by the processor 120, so that the processor 120 implements the steps of any of the above methods.

[0174] The processor 120 can be a central processing unit (CPU), and the processor 120 can also be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor, or, the processor 120 can also be any conventional processor, etc.

[0175] In the implementation process, each step of the above method can be completed by the integrated logic circuit of the hardware in the processor 120 or the instructions in the form of software. The method disclosed in combination with the embodiments of the present application can be directly embodied as being executed and completed by a hardware processor, or executed and completed by a combination of the hardware and software modules in the processor 120. The software module can be located in a mature storage medium in the art such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory, or an electrically erasable programmable memory, a register, etc. This storage medium is located in the memory 110, and the processor 120 reads the information in the memory 110 and combines its hardware to complete the steps of the above method. To avoid repetition, it will not be described in detail here.

[0176] In some implementations, in addition to the hardware units described above, a computer device may further include software modules. Among them, the software modules may be, for example, an operating system, a Basic Input Output System (BIOS), application software, etc.

[0177] The operating system is used to manage the hardware and / or software resources of a computer device and is the core and foundation of the computer device. The operating system needs to handle basic tasks such as managing and configuring memory, determining the priority order of system resource supply and demand, controlling input and output devices, operating the network, and managing the file system. To facilitate user operation, most operating systems provide an operation interface for users to interact with the system.

[0178] The BIOS is used to perform hardware initialization during the power-on boot phase and provide runtime services for the operating system and application programs. In some implementations, the BIOS can also monitor the temperature of the display processor and perform functions such as adjusting the temperature protection strategy.

[0179] Application software, also known as an application program, can be understood as software written for a specific application purpose of users and is one of the main classifications of computer software. For example, the application software can be a program for achieving purposes such as power control and temperature management.

[0180] It can be understood that the specific examples in this application are only to help those skilled in the art better understand the implementation manners of this application, rather than limiting the protection scope of this application.

[0181] It can be understood that in various implementation manners of this application, the magnitudes of the sequence numbers of each process do not mean the sequence of execution. The execution sequence of each process should be determined by its function and internal logic, and should not constitute any limitation to the implementation process of this application.

[0182] It can be understood that the various implementation manners described in this application can be implemented alone or in combination, and this application does not limit this.

[0183] Unless otherwise specified, all technical and scientific terms used in this application have the same meaning as commonly understood by those skilled in the technical field of this application. The terms used in this application are only for the purpose of describing specific implementation manners and are not intended to limit the scope of this application. The singular forms of "a", "above", and "the" used in this application and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise.

[0184] Those of ordinary skill in the art will appreciate that the units and algorithm steps of the examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware or in a combination of computer software and electronic hardware. Whether these functions are executed in hardware or software depends on the specific application and design constraints of the technical solution. A person skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered to exceed the scope of this application.

[0185] Those skilled in the art can clearly understand that for the sake of convenience and brevity of description, the specific working processes of the above-described embodiments can refer to the corresponding processes in other embodiments and will not be elaborated herein.

[0186] In the embodiments provided in this application, it should be understood that the disclosed systems, devices, and methods can be implemented in other ways. For example, the device embodiments described above are merely illustrative. For example, the division of units is only a logical function division, and there can be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the displayed or discussed couplings or direct couplings or communication connections to each other can be through some interfaces. The indirect couplings or communication connections of devices or units can be electrical, mechanical, or other forms.

[0187] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they can be located in one place or distributed to multiple network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the technical solution of this application.

[0188] In addition, the functional units in various embodiments of this application can be integrated into one processing unit, or each unit can exist physically alone, or two or more units can be integrated into one unit.

[0189] If a function is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the related technology, or part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in various embodiments of this application. The aforementioned storage medium includes: various media such as USB flash drives, mobile hard disks, read-only memories (ROMs), random access memories (RAMs), magnetic disks, or optical discs that can store program codes.

[0190] The above are only specific embodiments of this application, but the protection scope of this application is not limited thereto. Any person skilled in the art within the technical scope disclosed by this application can easily think of changes or substitutions, which should all be covered within the protection scope of this application. Therefore, the protection scope of this application should be subject to the protection scope of the claims.

Claims

1. A robot teleoperation control method, characterized in that, The method includes: Receiving multi - degree - of - freedom control input information from multiple operable components of a teleoperation device, where the multiple operable components are used to collect one or more of the operator's posture information, position information, motion information, and control input instructions, and the multiple operable components are operated by one or more operators; Generating control instructions for controlling each degree of freedom in the controllable components based on the multi - degree - of - freedom control input information; Sending the control instructions to the at least one robot to control the corresponding degrees of freedom of the controllable components of the robot to perform operations corresponding to the control instructions; Wherein, the robot includes one or more of a physical robot and a virtual robot in a simulation environment, and the multiple controllable components include one or more of a head, a waist, a robotic arm, a chassis, and an end effector; The number of robots is multiple, and the multiple operable components of the teleoperation device are operated by one operator; or, the number of robots is one, and the multiple operable components of the teleoperation device are operated by multiple operators.

2. The robot teleoperation control method according to claim 1, characterized in that, The generating control instructions for controlling each degree of freedom in the controllable components based on the multi - degree - of - freedom control input information includes: Generating control instructions for controlling each degree of freedom in the controllable components based on the one - to - one mapping relationship between the control input information for each degree of freedom in the multi - degree - of - freedom control input information and each degree of freedom of the multiple controllable components in at least one robot.

3. The robot teleoperation control method according to claim 1, wherein The mapping relationship corresponding to each degree of freedom of the head is represented by a linear formula; or, the mapping relationships corresponding to each degree of freedom of the waist, the robotic arm, the chassis, and the end effector are represented by pose transformation matrices.

4. The robot teleoperation control method according to claim 1, wherein The method further includes: Recording the control instructions and their corresponding status data and storing them as historical control data; Performing state reconstruction based on the historical control data to generate a robot motion trajectory for graphical display and storing it; When a playback instruction is received, playing back one or more of the control instructions, status data, and robot motion trajectory in the historical control data.

5. The robot teleoperation control method according to claim 1, characterized in that, The method further includes: Receiving a user instruction through a human - machine interface and controlling the robot to perform one or more operations such as reset, self - check, parameter reset, and running log clearing.

6. The robot teleoperation control method according to claim 1, wherein The method further includes: Collecting the operating state of the robot; When the operating state is an abnormal state or an emergency stop instruction is received, controlling the robot to stop running.

7. The robot teleoperation control method according to claim 1, wherein The method further includes: Setting a specified parameter value range for at least one operating parameter; When the parameter value of the operating parameter is not within the corresponding parameter value range, performing an alarm operation.

8. The robot teleoperation control method according to claim 1, characterized in that, The method further includes: Receiving and fusing environmental perception data of multiple perception types in the environment where the robot is located; the multiple perception types include at least two of vision, audition, and touch; Based on the environmental perception data of the multiple perception types, performing three - dimensional reconstruction and monitoring of the environment where the robot is located; Providing one or more of the three - dimensional reconstruction results and monitoring results to the operator, or using them as inputs to an automatic decision - making model.

9. The robot teleoperation control method according to claim 1, characterized in that, The method further includes: In the case of loss of control instructions or interruption of the communication link between the teleoperation device and the robot, perform one or more of the following processes: initialize the communication link; resend uncompleted control instructions; reconstruct the context.

10. A robot teleoperation control system, characterized in that, The system includes: At least one robot, which includes one or more of an entity robot and a virtual robot in a simulation environment, and the robot includes one or more controllable components such as a head, a waist, a robotic arm, a chassis, and an end effector; A teleoperation device, including a plurality of operable components, the plurality of operable components being used to collect one or more of the posture information, position information, action information, and control input instructions of an operator, and the plurality of operable components being operated by one or more operators; A control device, configured to perform the following processes: Receive multi-degree-of-freedom control input information from a plurality of operable components of the teleoperation device; Generate control instructions for controlling each degree of freedom of the controllable components based on the one-to-one mapping relationship between the control input information for each degree of freedom in the multi-degree-of-freedom control input information and each degree of freedom of a plurality of controllable components in at least one robot; Send the control instructions to the at least one robot to control the corresponding degrees of freedom of the controllable components of the robot to perform the operations corresponding to the control instructions.

11. The robot teleoperation control system according to claim 10, wherein The teleoperation device includes one or more of a virtual reality device, a multi-degree-of-freedom handle, and a mobile terminal device.

12. The robot teleoperation control system according to claim 10, wherein The teleoperation device further includes a plurality of environmental perception simulation components of multiple perception types, and the multiple perception types include at least two of vision, hearing, and touch; The environmental perception simulation components are used to receive environmental perception data of the corresponding perception type in the environment where the robot is located and provide corresponding perception feedback to the operator to simulate visual images, sound information, or tactile feedback in the environment where the robot is located.

13. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 9 is implemented.

14. A computer device, characterized in that, The computer device includes a memory and a processor, the memory stores a computer program, and when the processor executes the computer program, the method according to any one of claims 1 to 9 is implemented.

15. A computer program product, characterized in that, The computer program product includes a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 9 is implemented.

Images