Human-machine-data three-element integrated robot teleoperation and data acquisition system and method

Abstract

The invention belongs to the technical field of robots, and discloses a man-machine-data three-element integrated robot teleoperation and data acquisition system and method, and the system comprises a man-machine interaction module, a physical robot, a prediction simulation module, a digital robot, a virtual environment and a data management module. Wherein the prediction simulation module is used for aligning information of a natural person to execution action information of the physical robot, predicting an action instruction sequence and correcting the state of the digital robot in real time according to state information fed back by the physical robot; the digital robot is used for executing a corresponding action in the virtual environment according to the combined action instruction sequence, if a safety problem exists in the execution process, execution is stopped, and otherwise, the combined action instruction sequence is sent to the physical robot; the data management module is used for collecting data in the system operation process. The technical problems of operation delay and lack of humanoid robot training data in a traditional teleoperation system are solved.

Description

Technical Field

[0001] The present invention belongs to the field of robot technology, and more specifically, relates to a "man-machine-data" three-in-one robot teleoperation and data acquisition system and method. Background Art

[0002] With the rapid development of robotics technology, teleoperated robots are increasingly used in complex and dangerous environments and in data acquisition. In particular, humanoid robots have high flexibility and adaptability when performing tasks similar to humans.

[0003] However, traditional teleoperation methods face challenges such as operation lag and insufficient information feedback. In traditional "human-machine" interactions, operators are often unable to keep pace with physical robots directly, resulting in operation lag or misoperation, which increases the risk of damage to the robot. In addition, in order for robots to imitate human actions to perform a series of tasks, a large amount of virtual and real data is usually required for training, but such data is often scarce. The significant structural differences between physical robots and natural humans often cause errors in existing systems when performing spatial mapping, which affects operation accuracy and makes direct teleoperation difficult. In traditional teleoperation systems, there are still certain deficiencies in the effective management and collection of data and the combined application with intelligent enhancement modules, which makes it difficult to meet the needs of large-scale training data and the generalized execution of complex tasks. How to efficiently collect large-scale, human-like data through more advanced teleoperation methods has also become a major problem in the current field.

[0004] Therefore, problems such as operation delays, insufficient data feedback, lack of humanoid robot training data, and unnatural human-computer interaction in traditional remote operation systems have become technical challenges in this field. Summary of the invention

[0005] In response to the above defects or improvement needs of the prior art, the present invention provides a "human-machine-data" three-in-one robot remote operation and data acquisition system and method, thereby solving the technical problems of operation delays in traditional remote operation systems and lack of humanoid robot training data.

[0006] To achieve the above-mentioned purpose, according to one aspect of the present invention, a "human-machine-digital" three-in-one robot teleoperation and data acquisition system is provided, comprising a human-machine interaction module, a physical robot to be teleoperated, and a digital simulation module located in the cloud and closer to the physical robot than the human-machine interaction module, wherein the digital simulation module comprises a prediction simulation module, a digital robot, a virtual environment and a data management module; wherein,

[0007] The human-machine interaction module is used to transmit the information of the natural person to the prediction simulation module, and to feed back the status information fed back by the digital robot and / or the physical robot to the natural person;

[0008] The prediction simulation module is used to align the information of a natural person with the execution action information of the physical robot, that is, to obtain human-machine alignment data, convert the human-machine alignment data to obtain the rotation angle of the joint motor of the physical robot, that is, to obtain the action instruction sequence of the digital robot, and predict the subsequent several action instructions of the action instruction sequence in real time according to the action instruction sequence and the feedback information of the physical robot, and merge the action instruction sequence and the subsequent several action instructions and input them into the digital robot; it is also used to correct the state of the digital robot in real time according to the state information fed back by the physical robot, so that it is always consistent with the state of the physical robot;

[0009] The digital robot is used to perform corresponding actions in the virtual environment according to the combined action instruction sequence. If there is a safety problem during the execution, the execution is stopped; otherwise, the combined action instruction sequence is sent to the physical robot;

[0010] The physical robot is used to interact with the physical environment according to the combined action instruction sequence, and to feed back state information to the prediction simulation module and the human-computer interaction module in real time; the virtual environment and the physical environment are twin environments;

[0011] The data management module is used to collect and manage data during the operation of the human-computer interaction module, the physical robot and / or the digital simulation module.

[0012] Preferably, the data collected by the data management module includes the information of the natural person, digital simulation information, physical robot data and / or AI synthesized data; the information of the natural person includes the natural person's motion information, motion timing information and / or voice information; the digital simulation information includes the data generated by the predictive simulation module and the data generated by the movement or training of the digital robot in a virtual scene; the physical robot data includes the data generated by the physical robot during remote operation; the AI ​​synthesized data is data synthesized by using AI to obtain the information of the natural person, the digital simulation information and / or the physical robot data.

[0013] Preferably, the digital simulation module further includes an intelligent enhancement submodule, which includes a prediction training unit, a decision intelligence unit, an operation intelligence unit and a mobile intelligence unit, wherein:

[0014] The prediction training unit is used to use the data collected by the data management module as a training set to train the prediction simulation module to make the prediction result more accurate;

[0015] The decision-making intelligent unit is used to perform training based on the collected information of the natural person and the physical robot data, so that after training, it becomes a digital brain that enables the digital robot to imitate the timing of human actions to plan a designated task;

[0016] The operation intelligent unit is used to perform training according to the collected information of the natural person and the physical robot data, and after training, it can enable the digital robot of the specified form to generalize and perform tasks, and form an operation skill library of the digital robot;

[0017] The mobile intelligent unit is used to perform training based on the walking gait and trajectory data collected from the natural person's motion information and the physical robot data. After training, it can enable the physical robot aligned with the digital robot's motion to have the most energy-saving and natural gait.

[0018] Preferably, the digital simulation module also includes a digital simulated person, which is a twin simulated person of a natural person, and is used to superimpose the virtual image of the natural person into the physical environment according to the information of the natural person, perform corresponding remote operation actions, and display it to the natural person for viewing through the interactive device in the human-computer interaction module.

[0019] Preferably, the human-computer interaction module includes a perception submodule and a feedback submodule.

[0020] The perception submodule is used to perceive the information of the natural person by adopting homogeneous perception, heterogeneous perception and / or natural perception, and transmit the information to the prediction simulation module;

[0021] The feedback submodule is used to feed back the status information fed back by the digital robot and / or the physical robot to the natural person through visual feedback, force feedback and / or auditory feedback.

[0022] Preferably, the physical robot includes a robot joint control module and a physical robot mechanical body, and the robot joint control module is used to convert the merged motion instruction sequence into torque or current information and send it to the joint motor in the physical robot mechanical body to realize physical interaction between the physical robot and the physical environment.

[0023] As another aspect of the present invention, the following technical solution is also provided:

[0024] A "human-machine-digital" three-in-one robot teleoperation method based on the above system, inputting the information of a natural person into a trained decision-making intelligent unit, which plans the corresponding teleoperation task and outputs the planning result to the operation intelligent unit and the mobile intelligent unit;

[0025] The operation intelligent unit selects skills that match the planning results from the skill library, and the mobile intelligent unit plans the walking path and gait of the digital robot similar to human walking based on the planning results. The digital robot performs corresponding actions based on the skills selected by the operation intelligent unit and the walking path and gait planned by the mobile intelligent unit, and aligns the actions to the physical robot.

[0026] As another aspect of the present invention, the following technical solution is also provided:

[0027] A "human-machine-digital" three-in-one robot remote operation and data acquisition method based on the above system captures the operator's whole body movements through the human-machine interaction module, drives the digital robot to run in the virtual environment through the predictive simulation module, executes the operator's movements, and drives the physical robot to interact with the physical environment after judging that there is no safety problem. The status of the physical robot and the digital robot is displayed to the operator through the human-machine interaction module in a virtual-real fusion manner; during the whole process, the data generated by the human-machine interaction module, the physical robot and / or the digital simulation module are collected through the data management module.

[0028] As another aspect of the present invention, the following technical solution is also provided:

[0029] Based on the above system, a "human-machine-digital" three-in-one robot remote operation and data collection method is proposed. Natural persons perceive and control digital robots to run in a virtual environment and perform designated tasks through human-machine interaction modules and predictive simulation modules, and collect digital robot operation data through data management modules.

[0030] As another aspect of the present invention, the following technical solution is also provided:

[0031] A "human-machine-digital" three-in-one robot teleoperation and data acquisition method based on the above system captures the operator's whole body movements through the human-machine interaction module, drives the digital robot to run in the virtual environment through the prediction simulation module, executes the operator's movements, and drives the physical robot to interact with the physical environment after judging that there is no safety problem;

[0032] The operator's whole body movements are transmitted to the digital simulator through the human-computer interaction module, and the digital simulator is superimposed on the physical environment to execute the operator's movements;

[0033] The states of the physical robot, digital robot and digital simulated human are displayed to the operator through the human-computer interaction module in a virtual-real fusion manner;

[0034] During the whole process, the data generated by the human-computer interaction module, the physical robot and / or the digital simulation module are collected through the data management module.

[0035] In general, compared with the prior art, the technical solution conceived by the present invention has the following beneficial effects:

[0036] 1. There are significant differences between natural humans and physical robots in terms of time, space, and logic. In terms of time, physical robots rely on precise clock calculations and lack human subjective time perception; in terms of space, robots can only represent space through discrete data and symbols, and cannot perform comprehensive perception like humans; in terms of logic, physical robots are limited to preset rules and it is difficult to achieve dynamic and complex planning like humans. The "human-machine-number" ternary system of the present invention is different from the "two-body" relationship of digital twins. It innovatively introduces the unique element of "digital robot", which significantly improves the complexity and functionality of the system.

[0037] The "human-machine-digital" three-in-one robot teleoperation and data acquisition system of the present invention designs the digital robot as an intelligent bridge between natural humans and physical robots, achieving precise alignment in three dimensions. In the time dimension, the digital human simulates the human operation instructions in advance through the prediction simulation module, and transmits them to the physical robot after verifying the safety in the virtual environment, reducing operation delays and improving real-time and safety; in the spatial dimension, the digital robot can perform spatial mapping in the virtual environment, accurately mapping the human operation space to the robot's workspace, overcoming spatial differences and achieving precise control; in the logical dimension, the digital human optimizes and adjusts the human operation instructions through the digital simulation module to make them conform to the motion logic and constraints of the physical robot, ensuring the executability and rationality of the operation instructions. In addition, it also has data management capabilities to support continuous optimization of the system.

[0038] The present invention provides a "human-machine-digital" three-in-one robot remote operation and data acquisition system, which introduces a digital simulation module. Through predictive simulation and safety verification, it greatly reduces the delay problem in remote operation and the risk of physical robot damage caused by operational errors, thereby ensuring the safety of operation; through the parallel data acquisition mode, the system can significantly improve the data generation efficiency and meet the robot training tasks with large-scale data requirements; by organically combining the human-machine interaction module (natural person), physical robot and digital robot, a closed-loop operation process is constructed, which promotes the development of robot remote operation system in terms of multi-information fusion, safe and agile operation, etc., and has broad application prospects.

[0039] 2. The present invention provides a "human-machine-digital" three-in-one robot remote operation and data acquisition system, which provides a means for a physical robot to learn human movements. Since the physical properties and joint structures of physical robots are completely different from those of humans, simple position replication is difficult for the physical robot to maintain balance. The present invention first uses simulation data to train the digital robot so that it can maintain balance autonomously, and then remaps human movements to the robot so that it can both imitate human movements and maintain its own balance. On this basis, the physical robot imitates human movements to complete a series of operation tasks, and collects anthropomorphic operation data that conforms to the characteristics of the physical robot, which can provide high-quality training data for subsequent physical robots to complete similar tasks autonomously.

[0040] 3. The present invention provides a "man-machine-digital" three-in-one robot teleoperation and data acquisition system. By introducing a digital robot, the next action of the physical robot is first predicted and simulated in the simulation to avoid the problem of physical robots colliding in the real environment. At the same time, by using a large amount of physical robot operation data, the digital robot can predict the robot actions of the next several steps in advance according to the received action sequence, effectively eliminating the time delay caused by remote teleoperation. In order to correct the errors generated after the movement of the digital robot and the physical robot, the present invention can automatically correct the errors of the digital robot and the physical robot after each operation to ensure the reliability of the simulation. At the same time, the operator can clearly observe the virtual and real images of the digital robot and the physical robot in the MR helmet, providing a more friendly teleoperation experience for the operator (natural person).

[0041] 4. The present invention provides a "man-machine-number" three-in-one robot remote operation and data acquisition system, in which the intelligent enhancement submodule has decision-making intelligence, operational intelligence and mobile intelligence, which can continuously expand the skill library and realize the generalization ability of robot tasks.

[0042] 5. The present invention provides a "man-machine-number" three-in-one robot remote operation and data acquisition system, whose multimodal perception and feedback mechanism integrates visual, auditory, force and other information, and can feedback the robot's status to the operator in real time, providing a more intuitive and immersive interactive experience.

[0043] 6. The present invention provides a "human-machine-digital" three-in-one robot remote operation method. After the intelligent enhancement sub-module is trained, the decision-making intelligent unit can directly plan the task, and the operation intelligent unit and the mobile intelligent unit can directly select the operation skills of the digital robot and plan the walking path and gait of the digital robot in the skill library, so that the digital robot can perform corresponding actions and align the actions to the physical robot without going through the prediction simulation module, continuously optimizing the system, and improving the intelligence, accuracy and convenience of remote operation.

[0044] 7. The present invention provides a "man-machine-number" three-in-one robot remote operation method, which provides multiple operation modes, promotes the development of robot remote operation systems in multi-information fusion, intelligence enhancement and autonomous decision-making, and has broad application prospects. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] Figure 1 It is a schematic diagram of the relationship between the three main objects of a "man-machine-data" three-in-one robot teleoperation and data acquisition system in an embodiment of the present invention;

[0046] Figure 2 It is a schematic diagram of the overall framework of a "man-machine-data" three-in-one robot teleoperation and data acquisition system in an embodiment of the present invention;

[0047] Figure 3 is a schematic diagram of the structure of a human-computer interaction module in an embodiment of the present invention;

[0048] Figure 4 It is a schematic diagram of the structure of the prediction simulation module in an embodiment of the present invention;

[0049] Figure 5 is a schematic diagram of the structure of a data management and intelligence enhancement module in an embodiment of the present invention;

[0050] Figure 6 It is a data collection mode flow chart of a "man-machine-data" three-in-one robot teleoperation and data collection system in an embodiment of the present invention;

[0051] Figure 7 This is a data collection mode 2 flow chart of a "man-machine-data" three-in-one robot teleoperation and data collection system in an embodiment of the present invention;

[0052] Figure 8 It is a three-flow chart of data collection mode of a "man-machine-data" three-in-one robot teleoperation and data collection system in an embodiment of the present invention.

[0053] Throughout the drawings, the same reference numerals are used to denote the same elements or structures, wherein:

[0054] 1. Natural person; 2. Physical robot; 3. Digital human; 4. Virtual environment; 5. Physical environment; 6. Network; 7. Digital simulation module; 11. Human-computer interaction module; 21. Robot joint control module; 22. Physical robot mechanical body; 3-1. Digital simulation human; 3-2. Digital robot; 31. Prediction simulation module; 32. Data management and intelligent enhancement module; 111. Perception module; 112. Feedback module; 321. Data management module; 322. Intelligent enhancement module. DETAILED DESCRIPTION

[0055] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention. In addition, the technical features involved in the various embodiments of the present invention described below can be combined with each other as long as they do not conflict with each other.

[0056] The present invention provides a "man-machine-digital" three-in-one robot teleoperation and data acquisition system that combines teleoperation, data acquisition and intelligent enhancement. By introducing the "man-machine-digital" three-in-one collaborative architecture and using digital humans as intermediaries between natural humans and physical robots, a high degree of collaboration between natural humans, physical robots and digital humans, as well as efficient data acquisition and precise operation control are achieved. At the same time, with the help of a virtual environment to achieve prediction and simulation of operational behaviors, the delay in teleoperation can be significantly reduced, ensuring the safety of the operation of physical robots. At the same time, the system can effectively improve data acquisition efficiency, provide solid data support for robot autonomous learning and intelligent enhancement, and further enhance the robot's autonomous learning and intelligent enhancement capabilities.

[0057] like Figure 1 As shown, a "human-machine-digital" three-in-one robot teleoperation and data acquisition system provided by an embodiment of the present invention includes three main objects, namely a natural person 1, a physical robot 2 and a digital person 3. The natural person 1 refers to a real human on earth; the physical robot 2 refers to a real robot that can interact with the real physical environment, mainly a humanoid robot here, but can also be a robot entity of other shapes. The digital person 3 is a digital person that is simulated in a computer and is consistent with the shape of a human or a robot. The interaction and alignment between the human and the physical robot are completed through the digital person.

[0058] like Figure 2 As shown, the overall framework of the "human-machine-digital" three-in-one robot teleoperation and data acquisition system of the present invention includes a natural person 1, a physical robot 2, a digital human 3, a virtual environment 4, a physical environment 5, a network 6 and a digital simulation module 7. The physical robot 2 includes a robot joint control module 21 and a physical robot mechanical body 22. The robot joint control module 21 can receive instructions to drive the physical robot mechanical body 22 to move. At the same time, when it determines that the current state is dangerous during the execution of the instruction movement, it stops the movement of the robot or resets it to protect the physical robot mechanical body 22 from damage in special, emergency and dangerous situations.

[0059] The digital simulation module 7 includes a prediction simulation module 31, a data management and intelligence enhancement submodule 32, a digital human 3, and a virtual environment 4. The digital human 3 includes a digital simulation human 3-1 and a digital robot 3-2. The digital simulation human 3-1 is a twin simulation human of a natural person, and the digital robot 3-2 is a twin simulation robot of the physical robot 2. The prediction simulation module 31 and the data management and intelligence enhancement module 32 are the brains of the digital human 3. The virtual environment 4 and the physical environment 5 are twin environments, and there is a time-space mapping relationship between them.

[0060] like Figure 3 As shown, the human-machine interaction module 11 includes a perception submodule 111 and a feedback submodule 112, wherein the perception submodule 111 can digitize human information into digital information that can be understood by the digital robot, and transmit it to the prediction simulation module 31, which is a process from human to machine; the feedback submodule 112 can feedback the robot's information to humans in a form that can be perceived by human senses, which is a process from machine to human. Among them, the perception submodule 111 includes three modes, namely isomorphic perception, heterogeneous perception and natural perception. Isomorphic perception refers to the isomorphism of the perception device joints and the robot joints, such as exoskeleton, which can directly use the information of the human operation perception device for physical robot control without processing; heterogeneous perception refers to the different configurations of the perception device and the robot joints, and the perception information needs to be processed (such as inverse kinematics solution, etc.) before it can be used for the control of the physical robot, such as hand controllers, etc.; natural perception refers to capturing the language information of the human body and remapping it to the control of the physical robot in a specific way, such as motion capture equipment, VR equipment, microphones, etc.

[0061] The feedback submodule 112 includes three modes. One is visual feedback information. This feedback mode may include RGB images, AR (augmented reality) and MR (mixed reality) of virtual and real fusion as needed. The operator can intuitively see the operating status of the physical robot and the digital robot. The second feedback mode is force feedback. This feedback information needs to use related equipment to transmit the force felt by the physical robot and the digital robot to the human body. The third feedback mode is auditory feedback. The physical robot feeds back relevant information to the human body through voice. These perception and feedback modes are not mutually exclusive, and several perception and feedback modes are usually included in the operation process.

[0062] like Figure 4As shown, the human-machine interaction module 11 transmits the motion information of the natural person 1 to the prediction simulation module 31. The prediction simulation module 31 has two main functions, one is human-machine alignment, and the other is prediction simulation. Due to the large difference between the natural person 1 and the physical robot 2, it is impossible to directly perform remote operation. Therefore, it is necessary to first perform human-machine alignment in the digital simulation module 7, that is, the alignment of the human motion information and the physical robot motion information in data (alignment of the digital simulation person with the digital robot). The human-machine alignment data is converted into the spatial position of the physical robot joint, so as to obtain the action instruction sequence 1, 2...M of the digital robot, that is, the rotation angle of the physical robot joint motor; if the interactive device and the robot are of isomorphic type, the human-machine alignment data does not need to be converted too much, and like natural interaction or heterogeneous interaction, it is necessary to first remap the human-machine joints, and use inverse kinematics to solve the robot joint spatial position to drive the robot movement. Since teleoperation may have time delay or unstable connection, there will be some delay or danger in directly mapping the joint position of a person to the robot. Therefore, the process is to first verify the human body information instructions on the digital robot 3-2. In this process, the digital robot 3-2 will take the action instruction sequence 1, 2...M of the previous period and the feedback information of the physical robot 2 as input, and predict the subsequent action instructions M+1, M+2..., M+N in real time, where the value of N depends on the specific time delay. The action instruction sequence 1, 2...M+N is input into the digital robot 3- 2 makes it perform corresponding actions in the virtual environment 4. If there are safety issues such as collision during the execution, the execution is stopped and the action instruction sequence is not sent to the physical robot 2. Accordingly, the physical robot 2 does not execute the action instruction sequence. If there are no safety issues during the execution, the operation safety of the digital robot 3-2 is verified, and the action instruction sequence 1, 2...M+N is sent to the robot joint control module 21 of the physical robot 2. The action instruction information is converted into torque or current information and sent to the joint motor in the mechanical body 22 of the physical robot 2 to realize the physical interaction between the physical robot 2 and the physical environment 5. After the physical robot 2 interacts physically with the physical environment 5, the relevant state information will be fed back to the prediction simulation module 31. Due to the difference between the digital robot 3-2 and the physical robot 2 in the simulation, the state information will be used to correct and update the state of the digital robot 3-2 in real time, so that it is always consistent with the state of the physical robot 2. At the same time, the feedback information will be fed back to the human through the human-computer interaction module 11 in a virtual-real fusion manner to complete the closed loop. The entire process will be communicated and interacted by the robot-specific network 6. The human motion data, digital human information and physical robot information generated in this process will be stored in the data management and intelligent enhancement module 32 according to the needs of data set production, and will be used for subsequent update training of the general robot basic model or other purposes. After training, the model can be used to automatically operate other types of robots with similar configurations to complete similar tasks.

[0063] The prediction simulation module 31 predicts and verifies the interactive information between the digital human 3 and the virtual environment 4 in advance, and sends it to the physical robot 2 after verification, which will reduce the delay of teleoperation and the risk of misoperation. Since the state of the physical robot 2 after each operation will deviate from that of the digital robot 3-2, the digital robot 3-2 will correct and update its own state in real time according to the current state information of the physical robot 2, so that it is always consistent with the state of the physical robot 2, reducing the movement deviation between the digital robot 3-2 and the physical robot 2.

[0064] like Figure 5As shown, the data management and intelligent enhancement module 32 includes a data management submodule 321 and an intelligent enhancement submodule 322. The data management submodule 321 is mainly used to collect and store human language and motion information, motion timing information data, robot simulation data, physical robot data and AI synthetic data. Among them, human motion data is mainly collected from the movement of human teleoperated robots, including the movement and timing information of human motion; robot simulation data includes two parts, one is the predicted simulation data of digital people, and the other is the data generated by people directly controlling the movement of digital people in virtual scenes or training digital robots in virtual environments; physical robot data comes from the data generated by the teleoperated robot of the ternary system of the present invention; synthetic data mainly uses AI to synthesize more similar data through the several types of data collected above. Among these data, since the data of physical robots and human movements are difficult to collect in large quantities, most of them are simulation and synthetic data, but the quality of these data is poor compared with human motion data and physical robots. The intelligence enhancement submodule 322 includes a prediction training unit, a decision-making intelligence unit, an operation intelligence unit and a mobile intelligence unit, wherein the prediction training unit is used to use the collected data as a training set to continuously iterate the prediction simulation module to make the prediction result more accurate; the decision-making intelligence unit is mainly trained according to the human language and motion information, motion timing information data and physical robot data collected by the system, so that after training, it becomes a digital brain that enables the digital robot to imitate the motion timing of people to plan the specified task, which can not only understand human language behavior, but also plan and make decisions on tasks. The decision-making intelligence unit does not change with the morphological changes of the physical robot 2, that is, its trained model can be used to automatically operate other types of robots with similar configurations to complete similar tasks. The operation intelligence unit is mainly trained according to the human language and motion information, physical robot data and simulation and synthetic data collected by the system for specific tasks, so that after training, the digital robot of a specific form can generalize and perform similar tasks, and finally form an operation skill library of the digital robot, which contains various skills possessed by the trained digital robots and is updated in real time according to the expansion of the database. The mobile intelligent unit mainly includes two aspects. On the one hand, it is gait control similar to human walking, and on the other hand, it is planning of human walking paths. The walking gait and trajectory data in the human motion information and the physical robot data are used to train the mobile intelligent unit, so that after training, the physical robot aligned with the digital robot's movements has the most energy-saving and natural gait.

[0065] After each unit of the intelligent enhancement submodule 322 is trained, human language or motor action information is input into the decision-making intelligent unit of the intelligent enhancement submodule 322, and the decision-making intelligent unit can plan the task and output the task plan to the operation intelligent unit and the mobile intelligent unit. The operation intelligent unit selects skills that match the task plan from the skill library according to the task plan, and the mobile intelligent unit plans the walking path and gait of the digital robot similar to human walking according to the task plan. The digital robot performs corresponding actions according to the skills selected by the operation intelligent unit and the walking path and gait planned by the mobile intelligent unit, and aligns the actions to the physical robot.

[0066] The "man-machine-data" three-in-one robot teleoperation and data acquisition system proposed in the present invention has three modes. Mode 1 is a single "man-machine-data" teleoperation data acquisition. Figure 6 As shown, the operator uses AR, MR and other devices to capture the operator's whole body movements, and then drives the digital robot 3-2 to run in the virtual environment through the predictive simulation module. After judging that it is safe and there is no collision, the physical robot 2 is driven to interact with the physical environment. The states of the physical robot 2 and the digital robot will be displayed in the display unit (such as a helmet) of the human-computer interaction module through the fusion of virtual and real. In this process, the relevant motion data of "human, machine, and number" will be collected to provide data samples for the training of subsequent strategies.

[0067] Mode 2 is parallel “man-machine-data” teleoperation data collection. Since training a robot to achieve a task often requires a large amount of data, using a single physical robot for data collection is often inefficient, so this system can be used for parallel data collection. Figure 7 As shown, a teaching commander is added on the basis of a single "man-machine-digital" teleoperation data collection, and the operator can be expanded as needed. At this time, the digital human 3 has two forms. One is the virtual human form, namely the digital analog human 3-1. This form of the digital human 3 does not participate in the specific teleoperation, but superimposes its virtual image on the physical environment. Through the interactive device worn by the operator (such as VR glasses), the digital analog human 3-1 can be seen demonstrating the actions of the teaching commander in the physical environment, so that all operators can intuitively see the actions of the teaching commander. The other is the form of the digital robot 3-2, which is used to align the operator with the physical robot 2, and can provide the operator with mode 1 physical robot data collection.

[0068] Mode 3 is virtual teleoperation data collection. Figure 8As shown, a virtual environment and digital robots are generated through computer graphics and other technologies. People use VR, AR and other devices to perceive and control digital robots to complete specific tasks in the virtual environment, and at the same time collect digital robot operation data. This mode can enhance the immersive experience of human remote operation. Since no physical robots are used, there is no need to worry about damage to the physical robot due to operational errors.

[0069] There are significant differences between natural person 1 and physical robot 2 in terms of time, space and logic. In terms of time, physical robot 2 relies on precise clock calculations and lacks the subjective time perception of natural person 1; in terms of space, physical robot 2 can only represent space through discrete data and symbols, and cannot perform comprehensive perception like natural person 1; in terms of logic, physical robot 2 is limited to preset rules and it is difficult to achieve the same dynamic and complex planning as natural person 1. The "human-machine-digital" ternary system is different from the "two-body" relationship of digital twins. It innovatively introduces the unique element of "digital human", which significantly improves the complexity and functionality of the system.

[0070] The present invention is different from the traditional digital twin system. First of all, the subject matter of the present invention includes three objects: man, machine, and number. There are differences between the three and they are not twins. At the same time, the traditional digital twin is two systems running simultaneously, but in the present invention, the digital human is the information transmission medium between man and machine, the physical robot is the embodiment of the digital human, the digital human is superior to the physical robot, and the physical robot is controlled by the digital human.

[0071] The "human-machine-digital" three-in-one robot teleoperation and data acquisition system proposed in the present invention designs the digital human 3 as an intelligent bridge between the natural human 1 and the physical robot 2, achieving precise alignment in three dimensions. In the time dimension, the digital robot simulates the human operation instructions in advance through the prediction simulation module, and transmits them to the physical robot 2 after verifying the safety in the virtual environment, reducing the operation delay and improving the real-time performance and safety; in the spatial dimension, the prediction simulation module performs spatial mapping and coordinate conversion in the virtual environment, accurately maps the natural human's operation space to the digital robot's workspace, overcomes the spatial difference, and achieves precise control; in the logical dimension, the intelligent enhancement submodule optimizes and adjusts the human operation instructions to make them conform to the motion logic and constraints of the physical robot 2, ensuring the executability and rationality of the operation instructions. In addition, the digital simulation module also has data management capabilities, which can collect, store and analyze operation data to support continuous optimization of the system.

[0072] In the system of the present invention, natural persons interact with the system through the human-computer interaction module, and the physical robot interacts physically with the real environment. The digital human serves as a bridge between the human and the physical robot. It has a large amount of historical data and prior knowledge, and is a digital incarnation of the fusion of human and robot. It performs predictive simulation and safety verification through a virtual environment to ensure the safety and efficiency of remote operation. The data management and intelligent enhancement module in the system collects and manages the human motion data, simulation data and physical robot data generated during the operation for subsequent intelligent training and task execution. The system supports remote operation data acquisition in single mode, parallel mode and virtual mode, which improves the efficiency of data acquisition, enhances the remote operation experience and reduces the risk of damage to the physical robot.

[0073] It will be easily understood by those skilled in the art that the above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A "man-machine-data" three-in-one robot teleoperation and data acquisition system, characterized in that: It includes a human-computer interaction module, a physical robot being remotely operated, and a digital simulation module located in the cloud and closer to the physical robot than the human-computer interaction module, wherein the digital simulation module includes a prediction simulation module, a digital robot, a virtual environment, and a data management module; wherein, The human-machine interaction module is used to transmit the information of the natural person to the prediction simulation module, and to feed back the status information fed back by the digital robot and / or the physical robot to the natural person; The prediction simulation module is used to align the information of a natural person with the execution action information of the physical robot, that is, to obtain human-machine alignment data, convert the human-machine alignment data to obtain the rotation angle of the joint motor of the physical robot, that is, to obtain the action instruction sequence of the digital robot, and predict the subsequent several action instructions of the action instruction sequence in real time according to the action instruction sequence and the feedback information of the physical robot, and merge the action instruction sequence and the subsequent several action instructions and input them into the digital robot; it is also used to correct the state of the digital robot in real time according to the state information fed back by the physical robot, so that it is always consistent with the state of the physical robot; The digital robot is used to perform corresponding actions in the virtual environment according to the combined action instruction sequence. If there is a safety problem during the execution, the execution is stopped; otherwise, the combined action instruction sequence is sent to the physical robot; The physical robot is used to interact with the physical environment according to the combined action instruction sequence, and to feed back state information to the prediction simulation module and the human-computer interaction module in real time; the virtual environment and the physical environment are twin environments; The data management module is used to collect and manage data during the operation of the human-computer interaction module, the physical robot and / or the digital simulation module.

2. A "man-machine-data" three-in-one robot teleoperation and data acquisition system as claimed in claim 1, characterized in that: The data collected by the data management module includes the information of the natural person, digital simulation information, physical robot data and / or AI synthesized data. The information of the natural person includes the natural person's motion information, motion timing information and / or voice information. The digital simulation information includes the data generated by the predictive simulation module and the data generated by the digital robot moving or training in a virtual scene; the physical robot data includes the data generated by the physical robot during remote operation; the AI ​​synthesized data is data synthesized by using AI to obtain the information of the natural person, the digital simulation information and / or the physical robot data.

3. A "man-machine-data" three-in-one robot teleoperation and data acquisition system as claimed in claim 2, characterized in that: The digital simulation module also includes an intelligent enhancement submodule, which includes a prediction training unit, a decision intelligence unit, an operation intelligence unit and a mobile intelligence unit, wherein: The prediction training unit is used to use the data collected by the data management module as a training set to train the prediction simulation module to make the prediction result more accurate; The decision-making intelligent unit is used to perform training based on the collected information of the natural person and the physical robot data, so that after training, it becomes a digital brain that enables the digital robot to imitate the timing of human actions to plan a designated task; The operation intelligent unit is used to perform training according to the collected information of the natural person and the physical robot data, and after training, it can enable the digital robot of the specified form to generalize and perform tasks, and form an operation skill library of the digital robot; The mobile intelligent unit is used to perform training based on the walking gait and trajectory data collected from the natural person's motion information and the physical robot data. After training, it can enable the physical robot aligned with the digital robot's motion to have the most energy-saving and natural gait.

4. The "man-machine-data" three-in-one robot teleoperation and data acquisition system as claimed in claim 1, characterized in that: The digital simulation module also includes a digital simulated person, which is a twin simulated person of a natural person, and is used to superimpose the virtual image of the natural person into the physical environment according to the information of the natural person, perform corresponding remote operation actions, and display it to the natural person for viewing through the interactive device in the human-computer interaction module.

5. The "man-machine-data" three-in-one robot teleoperation and data acquisition system as claimed in claim 1, characterized in that: The human-computer interaction module includes a perception submodule and a feedback submodule. The perception submodule is used to perceive the information of the natural person by adopting homogeneous perception, heterogeneous perception and / or natural perception, and transmit the information to the prediction simulation module; The feedback submodule is used to feed back the status information fed back by the digital robot and / or the physical robot to the natural person through visual feedback, force feedback and / or auditory feedback.

6. The "man-machine-data" three-in-one robot teleoperation and data acquisition system as claimed in claim 1, characterized in that: The physical robot includes a robot joint control module and a physical robot mechanical body. The robot joint control module is used to convert the combined action instruction sequence into torque or current information and send it to the joint motor in the physical robot mechanical body to achieve physical interaction between the physical robot and the physical environment.

7. A "man-machine-number" three-in-one robot teleoperation method based on the system of claim 3, characterized in that: The information of the natural person is input into the trained decision-making intelligent unit, and the decision-making intelligent unit plans the corresponding teleoperation task and outputs the planning result to the operation intelligent unit and the mobile intelligent unit; The operation intelligent unit selects skills that match the planning results from the skill library, and the mobile intelligent unit plans the walking path and gait of the digital robot similar to human walking based on the planning results. The digital robot performs corresponding actions based on the skills selected by the operation intelligent unit and the walking path and gait planned by the mobile intelligent unit, and aligns the actions to the physical robot.

8. A "man-machine-data" three-in-one robot teleoperation and data acquisition method based on the system according to any one of claims 1 to 6, characterized in that: The operator's whole-body movements are captured through the human-computer interaction module, and the digital robot is driven to run in the virtual environment through the predictive simulation module to execute the operator's movements. After it is determined that there are no safety issues, the physical robot is driven to interact with the physical environment. The states of the physical robot and the digital robot are displayed to the operator through the human-computer interaction module in a virtual-real fusion manner. During the whole process, the data generated by the human-computer interaction module, the physical robot and / or the digital simulation module are collected through the data management module.

9. A "man-machine-data" three-in-one robot teleoperation and data acquisition method based on the system according to any one of claims 1 to 6, characterized in that: Natural persons perceive and control digital robots to run in a virtual environment and perform designated tasks through the human-computer interaction module and the predictive simulation module, and collect digital robot operation data through the data management module.

10. A "man-machine-data" three-in-one robot teleoperation and data acquisition method based on the system of claim 4, characterized in that: The human-machine interaction module captures the operator's full-body movements, and the predictive simulation module drives the digital robot to run in the virtual environment, execute the operator's movements, and drive the physical robot to interact with the physical environment after determining that there are no safety issues; The operator's whole body movements are transmitted to the digital simulator through the human-computer interaction module, and the digital simulator is superimposed on the physical environment to execute the operator's movements; The states of the physical robot, digital robot and digital simulated human are displayed to the operator through the human-computer interaction module in a virtual-real fusion manner; During the whole process, the data generated by the human-computer interaction module, the physical robot and / or the digital simulation module are collected through the data management module.

Images