Robot teleoperation method and system based on virtual reality and digital twinning

Abstract

The invention provides a robot teleoperation method and system based on virtual reality and digital twinning, and particularly relates to the technical field of man-machine collaborative intelligent manufacturing and assemblation.The scheme comprises the steps that an industrial robot and wearable equipment are started; establishing a two-way communication connection between the virtual environment of the wearable device and the industrial robot, and rendering the virtual robot of the wearable device in the virtual environment; and inputting the target pose operation instruction into the virtual robot, responding to a received target motion track planned by the wearable device according to the target pose operation instruction, and controlling the industrial robot to execute an operation task according to the motion track. According to the scheme, information interaction between an operator and the robot can be achieved, so that immersive real-time robot teleoperation is achieved through instant and accurate operation commands and field feedback, and the teleoperation precision of complex tasks and flexible operation tasks can be remarkably improved.

Description

Technical Field

[0001] The present invention relates to the technical field of human-machine collaborative intelligent manufacturing and assembly, and in particular to a robot teleoperation method and system based on virtual reality and digital twins. Background Art

[0002] As an important supporting technology for modern manufacturing, industrial robots are widely used in various production links, such as machining, welding and handling. As the complexity of production scenarios and the flexibility of tasks increase, robots are unable to complete tasks in all scenarios independently, and robot teleoperation technology has emerged.

[0003] At present, more and more researchers are trying to introduce virtual reality and digital twins into human-computer interaction and robot teleoperation. Virtual reality technology can provide operators with an immersive remote presence environment, and can expand and combine multimodal interaction methods, so that operators can obtain information about robots, operating platforms and their surrounding environment in a more intuitive way, enhancing people's perception and cognitive ability of the on-site environment. Digital twin technology can use digital technology to describe and model the characteristics, behaviors, formation processes and performance of physical objects. However, the action planning and execution in human-computer interaction and robot teleoperation based on virtual reality and digital twins require operators to have professional cognitive and operational capabilities. Otherwise, it is difficult for operators to obtain environmental information at the remote site, and its visual feedback makes it difficult for operators to build effective environmental understanding. It is also difficult for operators to intuitively operate robots and their end effectors to perform more complex and precise operations, which reduces the accuracy of teleoperation.

[0004] Therefore, there is an urgent need for a robot teleoperation method that can provide low cognitive burden and low learning cost in the field of robot teleoperation in production and manufacturing scenarios, so as to improve the teleoperation accuracy of complex tasks and flexible operation tasks. Summary of the invention

[0005] In view of the above-mentioned deficiencies in the prior art, the object of the present invention is to provide a robot remote operation method and system based on virtual reality and digital twins, aiming to solve the problem of low remote operation accuracy for complex tasks and flexible operation tasks existing in the prior art.

[0006] In order to achieve the above-mentioned object, the first aspect of the present invention provides a robot teleoperation method based on virtual reality and digital twin, comprising:

[0007] Launching industrial robots and wearable devices;

[0008] Establishing a two-way communication connection between the virtual environment of the wearable device and the industrial robot, and rendering the virtual robot of the wearable device in the virtual environment;

[0009] The target posture operation instruction is input into the virtual robot, and in response to receiving the target motion trajectory planned by the wearable device according to the target posture operation instruction, the industrial robot is controlled to perform the working task according to the target motion trajectory.

[0010] Optionally, establishing a two-way communication connection between the virtual environment of the wearable device and the industrial robot includes:

[0011] Setting a communication address of a communication port of the industrial robot;

[0012] Based on the communication address, configuring the communication port of the virtual environment;

[0013] Based on the communication port, a two-way communication connection is established between the virtual environment of the wearable device and the industrial robot.

[0014] Optionally, the process of generating the target motion trajectory includes:

[0015] Determining the spatial target position and posture of the virtual robot based on the target position and posture operation instruction;

[0016] Based on the spatial target posture, the wearable device is used to generate a target motion trajectory using a preset path planning algorithm.

[0017] Optionally, the generating the target motion trajectory based on the spatial target posture by using the wearable device and a preset path planning algorithm includes:

[0018] Collecting the working environment information and motion information of the industrial robot in real time;

[0019] Transmitting the working environment information and the motion information back and rendering them into the virtual environment, and updating the rendering effect of the virtual robot;

[0020] Based on the rendering effect, the wearable device is controlled to generate a target motion trajectory using a path planning algorithm.

[0021] Optionally, based on the rendering effect, controlling the wearable device to generate a target motion trajectory using a path planning algorithm includes:

[0022] Based on the rendering effect, controlling the virtual robot to align with the industrial robot to obtain a current motion posture of the virtual robot;

[0023] Based on the current motion posture of the virtual robot, the wearable device is controlled to generate a target motion trajectory using a path planning algorithm.

[0024] Optionally, the industrial robot includes a mechanical arm and a mechanical hand, and controlling the industrial robot to perform a working task according to the target motion trajectory includes:

[0025] According to the motion trajectory, control the handle controller of the industrial robot to adjust the end position of the robotic arm, and control the data glove controller of the industrial robot to adjust the opening and closing state of the fingers of the robotic hand;

[0026] Based on the end position of the robotic arm and the opening and closing state of the fingers, the operation task is performed. A second aspect of the present invention provides a robot teleoperation system based on virtual reality and digital twins, the system comprising:

[0027] Initialization module, used to start industrial robots and wearable devices;

[0028] A two-way communication establishment module, used to establish a two-way communication connection between the virtual environment of the wearable device and the industrial robot, and render the virtual robot of the wearable device in the virtual environment;

[0029] The operation execution module is used to input the target posture operation instruction into the virtual robot, and in response to receiving the target motion trajectory planned by the wearable device according to the target posture operation instruction, control the industrial robot to perform the operation task according to the target motion trajectory.

[0030] Optionally, it also includes a field vision and safety feedback module, which is used to control the field vision and safety feedback module of the industrial robot to collect the working environment information and motion information of the industrial robot in real time; and control the field vision and safety feedback module to transmit the working environment information and the motion information back and render them into the virtual environment.

[0031] The third aspect of the present invention provides an intelligent terminal, which includes a memory, a processor, and a robot remote operation program based on virtual reality and digital twins stored in the memory and executable on the processor. When the robot remote operation program based on virtual reality and digital twins is executed by the processor, any step of the above-mentioned robot remote operation method based on virtual reality and digital twins is implemented.

[0032] A fourth aspect of the present invention provides a computer-readable storage medium, on which is stored a robot remote operation program based on virtual reality and digital twins, which, when executed by a processor, implements any step of the above-mentioned robot remote operation method based on virtual reality and digital twins.

[0033] Compared with the prior art, the beneficial effects of this solution are as follows:

[0034] The present invention establishes a two-way communication connection between the virtual environment of the wearable device and the industrial robot, and renders the virtual robot of the wearable device in the virtual environment; the virtual reality device can provide an input method that conforms to the human operation intuition, so that the operator can learn to send immediate operation instructions in a relatively short time, reducing the learning cost while improving the operation efficiency; the wearable device is used to plan the target motion trajectory according to the target posture operation instruction, and the industrial robot is controlled to perform the operation task according to the motion trajectory, which can realize information interaction between the operator and the robot, thereby realizing immersive real-time robot teleoperation through instant and accurate operation commands and on-site feedback, and can significantly improve the teleoperation accuracy of complex tasks and flexible operation tasks. At the same time, the present invention uses a modular method to decouple the operation end and the execution end, so that each module can meet the customization requirements of the actual task to meet various complex flexible production requirements. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0036] Figure 1 This is a flowchart of the robot teleoperation method based on virtual reality and digital twins of the present invention;

[0037] Figure 2 It is a detailed flow chart of the robot teleoperation method based on virtual reality and digital twin of the present invention;

[0038] Figure 3 It is a robot teleoperation system module and data interaction block diagram based on virtual reality and digital twin of the present invention;

[0039] Figure 4 It is a schematic diagram of a robot teleoperation system module based on virtual reality and digital twins of the present invention;

[0040] Figure 5 It is a schematic diagram of the structure of the intelligent terminal of the present invention. DETAILED DESCRIPTION

[0041] In the following description, specific details such as specific system structures, technologies, etc. are provided for the purpose of illustration rather than limitation, so as to provide a thorough understanding of the embodiments of the present invention. However, it should be clear to those skilled in the art that the present invention may be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to prevent unnecessary details from obstructing the description of the present invention.

[0042] It should be understood that when used in this specification and the appended claims, the term "comprising" indicates the presence of described features, integers, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or groups thereof.

[0043] It should also be understood that the terms used in the present specification are only for the purpose of describing specific embodiments and are not intended to limit the present invention. As used in the present specification and the appended claims, the singular forms "a", "an" and "the" are intended to include plural forms unless the context clearly indicates otherwise.

[0044] It should be further understood that the term "and / or" used in the present description and the appended claims refers to and includes any and all possible combinations of one or more of the associated listed items.

[0045] As used in this specification and the appended claims, the term "if" may be interpreted as "when" or "upon" or "in response to determining" or "in response to detecting," depending on the context. Similarly, the phrases "if it is determined" or "if [described condition or event] is detected" may be interpreted as meaning "upon determination" or "in response to determining" or "upon detection of [described condition or event]" or "in response to detecting [described condition or event]," depending on the context.

[0046] The following is a clear and complete description of the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0047] In the following description, many specific details are set forth to facilitate a full understanding of the present invention, but the present invention may also be implemented in other ways different from those described herein, and those skilled in the art may make similar generalizations without violating the connotation of the present invention. Therefore, the present invention is not limited to the specific embodiments disclosed below.

[0048] In the field of human-machine collaborative intelligent manufacturing assembly in manufacturing scenarios, as the flexibility of production scenarios and tasks increases, traditional operation methods based on displays and key input will continue to increase the cognitive and operational burden of operators. In the traditional robot teleoperation mode, it is difficult for operators to obtain environmental information at the remote site, and its visual feedback makes it difficult for operators to build an effective understanding of the environment. In addition, it is difficult for operators to intuitively operate the robot and its end effector to perform more complex and precise operations, which reduces the efficiency of teleoperation.

[0049] It can be seen that in the field of robot teleoperation in production and manufacturing scenarios, there is a lack of robot teleoperation systems that can provide low cognitive burden and low learning cost. Based on this, this paper proposes a robot teleoperation method based on virtual reality and digital twins. It mainly uses virtual reality technology to provide an immersive remote on-site environment for operators, so that operators can obtain information about robots, operating platforms and their surrounding environment in a more intuitive way, enhancing people's perception and cognitive ability of the on-site environment; through virtual reality technology combined with multimodal data provided by data glove controllers and ergonomic handles, the operation logic is more intuitive, reducing the cognitive burden and learning cost of operators, and realizing safe, natural and efficient real-time two-way communication. It can be seen that for operators, this scheme can enhance people's cognitive ability of the production environment and reduce their learning cost and cognitive burden through the assistance of virtual reality technology; for industrial robots, it can receive the operator's action instructions in real time and perform corresponding operations, which can significantly improve the teleoperation accuracy of complex tasks and flexible operation tasks, and provide comprehensive feedback to operators through multimodal data.

[0050] The embodiment of the present invention provides a robot teleoperation method based on virtual reality and digital twins, which is deployed on electronic devices such as computers and servers and applied to robot production and manufacturing scenarios, targeting the situation of robot teleoperation based on virtual reality and digital twins. Figure 1-Figure 3 As shown, the steps of the method in this embodiment include:

[0051] Step S100: starting the industrial robot and the wearable device;

[0052] Specifically, in this embodiment, the operator wears a wearable device to operate the on-site operation robot (i.e., industrial robot). Therefore, it is necessary to first start the industrial robot control module, load the initialization control parameters of the control system, and calibrate the initial position through the sensor to ensure that the robot arm is in a ready-to-work state, and start monitoring equipment such as on-site cameras and infrared detectors to provide initialization information and communication ports for the subsequent establishment of two-way communication between the wearable device and the industrial robot. Start the wearable device, and start the virtual environment module in the wearable device, load the virtual model of the physical device and the working area and render it to the wearable device to build and render a virtual environment, and provide virtual environment data and communication ports for the subsequent establishment of two-way communication between them.

[0053] This embodiment does not specifically limit the types of industrial robots and wearable devices. The industrial robot may be an arm-type collaborative robot, a handling robot, a welding robot, etc. The wearable device may be smart glasses, smart watches, smart clothing, etc. worn on one or more parts of the body. For example, the wearable device may be a composite wearable device consisting of a virtual reality head-mounted display device and a virtual reality wearable input device, so as to immersively experience the surrounding environment through the virtual reality head-mounted display device, and accurately simulate virtual actions and adjust virtual action postures through the virtual reality wearable input device, so as to control the movement of the industrial robot.

[0054] Furthermore, in order to ensure the safety of production operations, a field vision and safety feedback module is set in the industrial robot. The field vision and safety feedback module is started to collect field environmental information to monitor potential collisions, extrusions and other dangerous actions that may exist during the operation process. If a potential dangerous situation is found, early warning information is promptly fed back so that the operator can adjust the control action of the industrial robot in time.

[0055] Step S200: establishing a two-way communication connection between the virtual environment of the wearable device and the industrial robot, and rendering the virtual robot of the wearable device in the virtual environment;

[0056] Specifically, before establishing a two-way communication connection between the virtual environment of the wearable device and the industrial robot, a communication protocol, data format, instruction set and other adaptive software interfaces are designed for the wearable device and the industrial robot in advance to ensure that they can correctly understand the data and instructions sent to each other. Then, a hardware connection relationship between the wearable device and the industrial robot is established through wireless communication (such as Wi-Fi, Bluetooth, etc.) or wired communication to facilitate data exchange and instruction transmission between the two. Then, based on the hardware connection and software interface, two-way communication can be achieved between the wearable device and the industrial robot, and data, instructions and status information can be sent and received to each other. For example, the wearable device can send instructions to the industrial robot, requiring it to perform a specific task; at the same time, the industrial robot can also feedback the task progress, error information, etc. to the wearable device.

[0057] Then, the virtual robot of the wearable device is rendered in the virtual environment, specifically including: first, using virtual reality (VR), augmented reality (AR) or mixed reality (MR) and other technologies to build a virtual environment including various virtual objects, scenes and interactive elements to provide an immersive remote presence environment for operators, so that operators can obtain information about the robot, operating platform and its surrounding environment in a more intuitive way, and enhance people's perception and cognitive ability of the on-site environment. Then, in the virtual environment, according to the characteristics and requirements of the wearable device, the corresponding virtual robot is designed, and it is ensured that the virtual robot can meet its interactivity, operability and visualization effect in the virtual environment. The virtual robot is equivalent to a digital industrial robot, which has similar functions and appearance to industrial robots. Finally, a graphics rendering engine (such as Unity, Unreal Engine, etc.) is used to render the virtual robot into the virtual environment to convert the 3D model of the virtual robot into an image that can be displayed on the screen, and ensure that the virtual robot has a realistic appearance and dynamic effects in the virtual environment.

[0058] In one embodiment, establishing a two-way communication connection between the virtual environment of the wearable device and the industrial robot in step S200 includes:

[0059] Step S210: setting the communication address of the communication port of the industrial robot;

[0060] Specifically, determine the communication port of the industrial robot, which can be a network interface (such as an Ethernet interface or a Wi-Fi module) or a serial communication interface (such as RS-232, RS-485 or USB), and then assign a unique communication address to the communication port of the industrial robot, which can be an IP address (for a network interface) or a serial port address (for a serial communication interface). Set the robot arm control and safety parameters through the controller of the industrial robot, and set the initial posture of the industrial robot. Start the network communication port of the industrial robot, set the communication address and wait for the virtual environment communication port to connect.

[0061] Step S220: configuring the communication port of the virtual environment based on the communication address;

[0062] Specifically, in the virtual environment of the wearable device, configure a communication port corresponding to the communication port of the industrial robot. If the industrial robot uses a network interface, then a network interface also needs to be configured in the virtual environment, and the same subnet and gateway as the industrial robot need to be set to ensure that both are in the same network. If the industrial robot uses a serial communication interface, then a virtual serial port (such as a serial port simulated by software) may be required in the virtual environment to communicate with the industrial robot. After configuring the communication port, start the network communication module of the wearable device and match the communication address of the communication port of the virtual environment with the communication port of the host computer of the industrial robot.

[0063] Step S230: Based on the communication port, a two-way communication connection is established between the virtual environment of the wearable device and the industrial robot.

[0064] Specifically, the communication port of the wearable device and the communication port of the host computer of the industrial robot are connected, and a two-way communication connection is established between the virtual environment of the wearable device and the industrial robot through the same communication protocol and data format. That is, real-time synchronization of information interaction is achieved through the digital twin communication mechanism, laying a good foundation for subsequent data exchange and information exchange in the process of executing work tasks.

[0065] In this embodiment, according to the type and communication address of the communication port of the industrial robot, the corresponding communication port and communication address are configured in the virtual robot, so as to simply and quickly establish a two-way communication relationship between the virtual and real robots, which is conducive to improving the timeliness of data and information transmission between the two.

[0066] Step S300: inputting a target posture operation instruction into the virtual robot, and in response to receiving a target motion trajectory planned by the wearable device according to the target posture operation instruction, controlling the industrial robot to perform a work task according to the target motion trajectory.

[0067] Specifically, the operator generates target posture operation instructions through gestures, buttons, voice and other operation methods, and inputs the target posture operation instructions for controlling the virtual robot in the virtual environment. Among them, the target posture operation instructions include but are not limited to the target position (coordinates) and posture (direction, angle, etc.) that the end effector (such as gripper, welding gun, etc.) of the industrial robot needs to achieve. It can be seen that virtual reality equipment can provide an input method that conforms to human operation intuition, so that operators can learn to send immediate operation instructions in a relatively short time, which reduces the learning cost and is conducive to improving operation efficiency.

[0068] After receiving the target posture operation instruction, the virtual robot uses the path planning algorithm to plan the target motion trajectory from the current position to the target position, wherein the target motion trajectory is composed of a series of points or path segments, each of which contains information such as the robot's position, velocity and acceleration. Then, the target motion trajectory is sent to the wearable device through the wearable device. After receiving the target motion trajectory, the industrial robot uses its own control system and driver to execute the target motion trajectory. In the process of performing the task in the effective human-machine working area, the industrial robot combines the obtained target motion trajectory information, real-time posture information, pre-marked robot joint points and other multi-modal input information, uses the deep reinforcement learning algorithm to generate the running trajectory of the end effector, and calculates the joint space combination sequence required by the industrial robot through the inverse kinematics solver. Among them, the process of calculating the joint space combination sequence required by the industrial robot through the inverse kinematics solver includes: establishing an industrial robot model, defining the target position and posture sequence, using the inverse kinematics principle to calculate the joint angle corresponding to each joint point, and generating the joint space combination sequence to ensure that the industrial robot can accurately execute the target motion trajectory. At the same time, mapping the joint space combination sequence to the joint space of the virtual robot through the network communication module is beneficial for the operator to adjust the motion trajectory in real time according to the feedback of the joint space combination sequence.

[0069] In this embodiment, by designing a new type of robot remote operation mode, the flexibility and adaptability of the operator and the reliability and accuracy of the robot are fully combined, so that the operator can remotely control the industrial robot and its end effector in an area far away from the dangerous scene. This not only enables the industrial robot to perform more complex operations more flexibly, thereby significantly improving the remote operation accuracy and operation efficiency of complex tasks and flexible operation tasks, but also ensures the safety of the operator.

[0070] The step S320 of generating a target motion trajectory based on the spatial target posture by using the wearable device and a preset path planning algorithm includes:

[0071] Step S321: collecting the working environment information and motion information of the industrial robot in real time;

[0072] Specifically, the operator controls the on-site vision and safety feedback module of the industrial robot to collect the working environment information and motion information of the industrial robot in real time, wherein the on-site vision and safety feedback module includes an on-site vision unit and a safety feedback unit. The on-site vision unit is used to capture various images and video information of the robot's working site through high-precision cameras and other visual sensors, and quickly and accurately analyze the captured information to provide intuitive visual feedback to the operator. Among them, the image and video information include the motion state of the industrial robot itself, such as position, speed, posture, etc., as well as obstacles and operating objects in the working environment. The safety feedback unit is used to use built-in multiple sensors to monitor the interaction between the industrial robot and the environment in real time to ensure the safety and reliability of the industrial robot during operation. Among them, the sensors include force sensors, temperature sensors, pressure sensors, etc., which monitor the interaction between the robot and the environment in real time. Once potential safety hazards are detected, such as the risk of collision between the robot and obstacles, overheating or overpressure, etc., the module will immediately trigger a safety alarm and warn the operator through various methods such as sound, light and electricity. At the same time, the module can also automatically adjust the robot's motion parameters or perform emergency shutdown operations according to the preset safety strategy to prevent accidents. It can be seen that the on-site vision and safety feedback module provides comprehensive and accurate data support for operators by collecting and analyzing the working environment information and motion information of the industrial robot in real time. It can not only improve the robot's operating efficiency and accuracy, but also greatly enhance the safety and reliability of the operating process.

[0073] Step S322: transmitting the working environment information and the motion information back and rendering them into the virtual environment, and updating the rendering effect of the virtual robot;

[0074] Specifically, the operator controls the on-site vision and safety feedback module to transmit the working environment information and motion information back to the virtual environment. The wearable device performs coordinate conversion, scale adjustment, physical property matching and other operations on the received working environment information and motion information, dynamically adjusts the rendering effect of the virtual environment and virtual robot, and feeds back the real-time rendering results and analysis results in the virtual environment to the operator, so that the operator can more intuitively understand the dynamic changes of the working environment and motion state of the industrial robot, and make more accurate and efficient decisions.

[0075] Step S323: Based on the rendering effect, control the wearable device to generate a target motion trajectory using a path planning algorithm.

[0076] Specifically, based on the rendering effect, the virtual robot is controlled to align with the industrial robot, that is, the collected industrial robot information is synchronized with the virtual robot to ensure the consistency of the two in time and space, so as to obtain the current motion posture of the virtual robot, including the robot's position, orientation, joint angle, etc. If the virtual robot and the industrial robot are not synchronized, restart the virtual robot and the industrial robot and perform the alignment operation.

[0077] Based on the current motion posture of the virtual robot, the kinematic constraints (such as joint restrictions, speed restrictions, etc.) and dynamic characteristics (such as inertia, friction, etc.) of the wearable device are considered, as well as obstacles and obstacle avoidance strategies in the working environment, and the wearable device is controlled to use a path planning algorithm to generate the target motion trajectory, and the generated target motion trajectory is converted into a sequence of instructions that the wearable device can understand and execute. It is easy to understand that in the process of executing the task, the state of the equipment and the changes in the environment will be continuously monitored, and the trajectory will be corrected or replanned when necessary.

[0078] In this embodiment, by collecting the working environment information and motion information of the industrial robot in real time, comprehensive and accurate data support is provided to the operator to ensure that the robot can perform tasks safely and efficiently; by aligning the virtual robot with the industrial robot, the accuracy and effectiveness of the planned target motion trajectory can be improved.

[0079] In one embodiment, if the industrial robot includes a robotic arm and a robotic hand, controlling the industrial robot to perform the task according to the target motion trajectory in step S300 includes:

[0080] Step S330: according to the motion trajectory, controlling the handle controller of the industrial robot to adjust the end position of the robotic arm, and controlling the data glove controller of the industrial robot to adjust the opening and closing state of the fingers of the robotic hand;

[0081] Specifically, the wearable device in this embodiment includes a head-mounted virtual reality device and a wearable device. The head-mounted virtual reality device is used to immersively experience the surrounding environment of the working space, and the wearable device is used to accurately simulate virtual actions and adjust the virtual action postures, and perform input operations of action instructions, etc. Among them, the wearable device includes a robotic arm and a five-finger robotic hand, which can change the posture of the end of the robotic arm by pressing the handle controller button, including adjusting the direction and movement speed of the wrist, and changing the opening and closing of each finger of the five-finger robotic hand through the data glove controller, including adjusting the opening and closing state of the fingers of the robotic hand. In actual operation, it is necessary to turn on the data glove controller and the handle controller in advance and connect them to the virtual reality engine, and render their corresponding models in the head-mounted display device. The robot receives the operator's instructions and attempts to move to the target position.

[0082] Step S340: Execute the operation task based on the end position of the robot arm and the opening and closing state of the fingers.

[0083] Specifically, the robot arm is controlled to move along the target motion trajectory to the target posture; the robot motor setting is initialized, and the target motion trajectory is transmitted to the robot motor to drive the robot motor to make each robot finger move to its designated position, so as to realize the flexible control of the robot in the workspace with six degrees of freedom in real time by using the data glove controller and the handle controller to complete various flexible operations. At the same time, the robot's motion and on-site environmental information are captured in real time by the visual and safety feedback module, and are transmitted back to the virtual environment to be rendered as a dynamic image for the operator to monitor; the virtual engine renders the input controller model in real time so that the operator can confirm the input operation; the infrared detector continuously detects whether the robot enters the restricted area. If the robot arm moves to the restricted area, the safety detection module issues a stop command, the robot will interrupt the motion operation and return to the initial position, and re-execute steps S330 and S340 until the robot arm and the five-finger robot of the industrial robot move to the target posture, stop moving and wait for new operation commands in a loop.

[0084] In this embodiment, the accuracy of performing work tasks along the target motion trajectory is improved through the mutual coordination of the head-mounted virtual reality device and the wearable device; by providing a robotic arm and a five-finger robotic hand for the wearable device, the flexibility and accuracy of the movement can be effectively improved.

[0085] Furthermore, in order to solve the problem of converting the operator's instructions into actual robot operations, the motion planning of the robotic arm and the five-finger manipulator (i.e., the end effector) in step S330 specifically includes: first, the target space coordinates input by the operator through the handle controller are transformed into the target point position in the industrial robot coordinate system. Next, the spatial attitude quaternion data of each inertial sensor of the data glove controller is read at a set frequency, and the angle formed by each fingertip sensor and the wrist sensor is calculated through the angle calculation module, and the angle is converted into the corresponding manipulator finger posture input format. Finally, through the established two-way communication relationship, these data are sent from the operating end to the on-site robot host computer and input into the robot and end effector drive.

[0086] In summary, the beneficial effects brought about by the method of the present invention include:

[0087] The present invention can obtain the operator's action command input, robot status, and work site environment information in real time, so that the operator and the robot can achieve real-time information exchange, thereby realizing immersive real-time robot remote operation through instant and accurate operation commands and on-site feedback. Virtual reality equipment provides an input method that conforms to human operation intuition, allowing operators to learn to send instant operation instructions in a relatively short time, reducing learning costs while improving operation efficiency. Industrial robots and virtual robots achieve real-time synchronization through the digital twin communication mechanism, and at the same time, visual information is transmitted back to the operation end in real time through visual sensors on site, ensuring that personnel obtain instant feedback during the operation process.

[0088] The system proposed by the present invention has the characteristics of modularity, and therefore has a high degree of adaptability and scalability. The system places the operator in a safe position, while the robot is located in the on-site working environment to perform operations. Therefore, the on-site operating environment can be flexibly changed under the premise of ensuring the necessary environmental requirements for the operation of the robot. Since the present invention uses a modular approach to decouple the operating end from the execution end, each module can be customized according to actual task requirements, such as replacing the end effector of the robot arm, changing the input method of the operator's wearable device, etc. At the same time, functional modules can be added based on the existing system according to task requirements to meet various complex flexible production requirements.

[0089] like Figure 4 As shown, corresponding to the above-mentioned robot teleoperation method based on virtual reality and digital twin, an embodiment of the present invention further provides a robot teleoperation system based on virtual reality and digital twin, and the above-mentioned robot teleoperation system based on virtual reality and digital twin includes:

[0090] An initialization module 410 is used to start the industrial robot and the wearable device;

[0091] A two-way communication establishing module 420, configured to establish a two-way communication connection between the virtual environment of the wearable device and the industrial robot, and render the virtual robot of the wearable device in the virtual environment;

[0092] The operation execution module 430 is used to input the target posture operation instruction into the virtual robot, and in response to receiving the target motion trajectory planned by the wearable device according to the target posture operation instruction, control the industrial robot to perform the operation task according to the target motion trajectory.

[0093] Furthermore, the above-mentioned system also includes an on-site vision and safety feedback module, which is used to control the on-site vision and safety feedback module of the industrial robot to collect the working environment information and motion information of the industrial robot in real time; and control the on-site vision and safety feedback module to transmit the working environment information and the motion information back and render them into the virtual environment.

[0094] Specifically, in this embodiment, the specific functions of the above-mentioned robot remote operation system based on virtual reality and digital twins can also refer to the corresponding description in the above-mentioned robot remote operation method based on virtual reality and digital twins, and will not be repeated here.

[0095] Based on the above embodiments, the present invention further provides an intelligent terminal, whose principle block diagram can be shown as follows: Figure 5 As shown. The above-mentioned intelligent terminal includes a processor, a memory, a network interface and a display screen connected via a system bus. Among them, the processor of the intelligent terminal is used to provide computing and control capabilities. The memory of the intelligent terminal includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a robot teleoperation program based on virtual reality and digital twins. The internal memory provides an environment for the operation of the operating system in the non-volatile storage medium and the robot teleoperation program based on virtual reality and digital twins. The network interface of the intelligent terminal is used to communicate with an external terminal through a network connection. When the robot teleoperation program based on virtual reality and digital twins is executed by the processor, the steps of any one of the above-mentioned robot teleoperation methods based on virtual reality and digital twins are implemented. The display screen of the intelligent terminal can be a liquid crystal display screen or an electronic ink display screen.

[0096] Those skilled in the art will understand that Figure 5 The principle block diagram shown in the figure is only a block diagram of a partial structure related to the scheme of the present invention, and does not constitute a limitation on the smart terminal to which the scheme of the present invention is applied. The specific smart terminal may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0097] In one embodiment, a smart terminal is provided, which includes a memory, a processor, and a robot teleoperation program based on virtual reality and digital twins stored in the memory and executable on the processor. When the robot teleoperation program based on virtual reality and digital twins is executed by the processor, the steps of any one of the robot teleoperation methods based on virtual reality and digital twins provided in an embodiment of the present invention are implemented.

[0098] An embodiment of the present invention also provides a computer-readable storage medium, on which is stored a robot remote operation program based on virtual reality and digital twins. When the robot remote operation program based on virtual reality and digital twins is executed by a processor, the steps of any one of the robot remote operation methods based on virtual reality and digital twins provided in an embodiment of the present invention are implemented.

[0099] It should be understood that the sequence numbers of the steps in the above embodiments do not imply a sequence of execution, and the execution sequence of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0100] Those skilled in the art can clearly understand that, for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example for illustration. In practical applications, the above-mentioned function allocation can be completed by different functional units and modules as needed, that is, the internal structure of the above-mentioned device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiment can be integrated in a processing unit, or each unit can exist physically separately, or two or more units can be integrated in one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units. In addition, the specific names of the functional units and modules are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of the present invention. The specific working process of the units and modules in the above-mentioned system can refer to the corresponding process in the aforementioned method embodiment, which will not be repeated here.

[0101] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0102] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.

[0103] In the embodiments provided by the present invention, it should be understood that the disclosed apparatus / terminal equipment and method can be implemented in other ways. For example, the apparatus / terminal equipment embodiments described above are only illustrative, for example, the division of the above modules or units is only a logical function division, and in actual implementation, other division methods can be used, for example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed.

[0104] The embodiments described above are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, it should be understood by those skilled in the art that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features therein may be replaced by equivalents. However, these modifications or replacements do not deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should be included in the protection scope of the present invention.

Claims

1. A robot teleoperation method based on virtual reality and digital twin, characterized in that: The following steps are involved: Launching industrial robots and wearable devices; Establishing a two-way communication connection between the virtual environment of the wearable device and the industrial robot, and rendering the virtual robot of the wearable device in the virtual environment; The target posture operation instruction is input into the virtual robot, and in response to receiving the target motion trajectory planned by the wearable device according to the target posture operation instruction, the industrial robot is controlled to perform the working task according to the target motion trajectory.

2. The robot teleoperation method based on virtual reality and digital twin according to claim 1, characterized in that: The establishing of a two-way communication connection between the virtual environment of the wearable device and the industrial robot comprises: Setting a communication address of a communication port of the industrial robot; Based on the communication address, configuring the communication port of the virtual environment; Based on the communication port, a two-way communication connection is established between the virtual environment of the wearable device and the industrial robot.

3. The robot teleoperation method based on virtual reality and digital twin according to claim 1, characterized in that: The process of generating the target motion trajectory includes: Determining the spatial target position and posture of the virtual robot based on the target position and posture operation instruction; Based on the spatial target posture, the target motion trajectory is generated by using the wearable device using a preset path planning algorithm.

4. The robot teleoperation method based on virtual reality and digital twin according to claim 3 is characterized in that: The step of generating a target motion trajectory based on the spatial target posture and using a preset path planning algorithm by using the wearable device includes: Collecting the working environment information and motion information of the industrial robot in real time; Transmitting the working environment information and the motion information back and rendering them into the virtual environment, and updating the rendering effect of the virtual robot; Based on the rendering effect, the wearable device is controlled to generate a target motion trajectory using a path planning algorithm.

5. The robot teleoperation method based on virtual reality and digital twin according to claim 4, characterized in that: The step of controlling the wearable device to generate a target motion trajectory by using a path planning algorithm based on the rendering effect includes: Based on the rendering effect, controlling the virtual robot to align with the industrial robot to obtain a current motion posture of the virtual robot; Based on the current motion posture of the virtual robot, the wearable device is controlled to generate a target motion trajectory using a path planning algorithm.

6. The robot teleoperation method based on virtual reality and digital twin according to claim 1, characterized in that: The industrial robot includes a mechanical arm and a mechanical hand, and the step of controlling the industrial robot to perform a working task according to the target motion trajectory includes: According to the motion trajectory, control the handle controller of the industrial robot to adjust the end position of the robotic arm, and control the data glove controller of the industrial robot to adjust the opening and closing state of the fingers of the robotic hand; Based on the end position of the robot arm and the opening and closing state of the fingers, the operation task is performed.

7. A robot teleoperation system based on virtual reality and digital twin, characterized in that: The system comprises: Initialization module, used to start industrial robots and wearable devices; A two-way communication establishment module, used to establish a two-way communication connection between the virtual environment of the wearable device and the industrial robot, and render the virtual robot of the wearable device in the virtual environment; The operation execution module is used to input the target posture operation instruction into the virtual robot, and in response to receiving the target motion trajectory planned by the wearable device according to the target posture operation instruction, control the industrial robot to perform the operation task according to the target motion trajectory.

8. The robot teleoperation system based on virtual reality and digital twin according to claim 7, characterized in that: It also includes an on-site vision and safety feedback module, which is used to control the on-site vision and safety feedback module of the industrial robot to collect the working environment information and motion information of the industrial robot in real time; and control the on-site vision and safety feedback module to transmit the working environment information and the motion information back and render them into the virtual environment.

9. An intelligent terminal, characterized in that: The intelligent terminal includes a memory, a processor, and a robot remote operation program based on virtual reality and digital twins stored in the memory and executable on the processor. When the robot remote operation program based on virtual reality and digital twins is executed by the processor, the steps of the robot remote operation method based on virtual reality and digital twins as described in any one of claims 1 to 6 are implemented.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a robot teleoperation program based on virtual reality and digital twins. When the robot teleoperation program based on virtual reality and digital twins is executed by the processor, the steps of the robot teleoperation method based on virtual reality and digital twins as described in any one of claims 1 to 6 are implemented.

Images