A "human-machine-data" three-in-one robot teleoperation and data acquisition system and method

Abstract

This invention belongs to the field of robotics and discloses a "human-machine-digital" three-in-one robot teleoperation and data acquisition system and method. The system includes a human-machine interaction module, a physical robot, a predictive simulation module, a digital robot, a virtual environment, and a data management module. The predictive simulation module is used to align a person's information with the physical robot's execution action information, predict the action instruction sequence, and correct the digital robot's state in real time based on the status information fed back by the physical robot. The digital robot is used to execute corresponding actions in the virtual environment according to the combined action instruction sequence. If there is a safety issue during execution, execution is stopped; otherwise, the combined action instruction sequence is sent to the physical robot. The data management module is used to collect data during the system's operation. This invention solves the technical problems of operation delays and the lack of humanoid robot training data in traditional teleoperation systems.

Description

Technical Field

[0001] The present invention belongs to the field of robotics technology, and more specifically, relates to a "man-machine-data" three-in-one robot teleoperation and data acquisition system and method. Background Art

[0002] With the rapid development of robotics technology, teleoperated robots are increasingly used in complex and dangerous environments and in data acquisition. In particular, humanoid robots have high flexibility and adaptability when performing tasks similar to humans.

[0003] However, traditional teleoperation methods face challenges such as operational lag and insufficient information feedback. In traditional "human-machine" interactions, operators are often unable to directly synchronize with the physical robot, resulting in operational lag or misoperation, which increases the risk of damage to the robot. In addition, in order for robots to imitate human movements to perform a series of tasks, a large amount of virtual and real data is usually required for training, but such data is often scarce. The significant structural differences between physical robots and natural humans often lead to errors in spatial mapping in existing systems, affecting operational accuracy and making direct teleoperation difficult. In traditional teleoperation systems, the effective management and collection of data and the combined application of intelligent enhancement modules still have certain shortcomings, making it difficult to meet the needs of large-scale training data and the generalized execution of complex tasks. How to efficiently collect large-scale, human-like data through more advanced teleoperation methods has also become a major challenge in the current field.

[0004] Therefore, problems such as operation delays, insufficient data feedback, lack of humanoid robot training data, and unnatural human-computer interaction in traditional teleoperation systems have become technical difficulties in this field. Summary of the Invention

[0005] In response to the above-mentioned defects or improvement needs of the existing technology, the present invention provides a "human-machine-data" three-in-one robot remote operation and data acquisition system and method, thereby solving the technical problems of operation delays and lack of humanoid robot training data in traditional remote operation systems.

[0006] To achieve the above objectives, according to one aspect of the present invention, a "human-machine-digital" three-in-one robot teleoperation and data acquisition system is provided, comprising a human-machine interaction module, a physical robot being teleoperated, and a digital simulation module located in the cloud and closer to the physical robot than the human-machine interaction module. The digital simulation module comprises a prediction simulation module, a digital robot, a virtual environment, and a data management module; wherein:

[0007] The human-computer interaction module is used to transmit the natural person's information to the prediction simulation module, and to feed back the status information fed back by the digital robot and / or the physical robot to the natural person;

[0008] The prediction simulation module is used to align the information of a natural person with the execution action information of the physical robot, that is, to obtain human-machine alignment data, convert the human-machine alignment data to obtain the rotation angle of the physical robot joint motor, that is, to obtain the action instruction sequence of the digital robot, and predict the subsequent several action instructions of the action instruction sequence in real time based on the action instruction sequence and the feedback information of the physical robot, and combine the action instruction sequence and the subsequent several action instructions and input them into the digital robot; it is also used to correct the state of the digital robot in real time based on the state information fed back by the physical robot, so that it is always consistent with the state of the physical robot;

[0009] The digital robot is used to perform corresponding actions in the virtual environment according to the combined action instruction sequence. If there is a safety problem during the execution, the execution is stopped; otherwise, the combined action instruction sequence is sent to the physical robot;

[0010] The physical robot is used to interact with the physical environment according to the combined action instruction sequence, and to feed back state information to the prediction simulation module and the human-computer interaction module in real time; the virtual environment and the physical environment are twin environments;

[0011] The data management module is used to collect and manage data during the operation of the human-computer interaction module, the physical robot and / or the digital simulation module.

[0012] Preferably, the data collected by the data management module includes the information of the natural person, digital simulation information, physical robot data and / or AI synthesized data, the information of the natural person includes the natural person's motion information, motion timing information and / or voice information, the digital simulation information includes the data generated by the predictive simulation module and the data generated by the digital robot moving or training in the virtual scene; the physical robot data includes the data generated by the physical robot during the remote operation process; the AI ​​synthesized data is data synthesized by AI using the information of the natural person, the digital simulation information and / or the physical robot data.

[0013] Preferably, the digital simulation module further includes an intelligent enhancement submodule, which includes a prediction training unit, a decision intelligence unit, an operation intelligence unit and a mobile intelligence unit, wherein:

[0014] The prediction training unit is used to use the data collected by the data management module as a training set to train the prediction simulation module to make the prediction result more accurate;

[0015] The decision-making intelligent unit is used to train based on the collected information of the natural person and the physical robot data, so that after training, it becomes a digital brain that enables the digital robot to imitate the timing of human movements to perform designated tasks;

[0016] The operation intelligence unit is used to perform training based on the collected information of the natural person and the physical robot data, so that after training, it can enable the digital robot of a specified form to generalize and perform tasks, and form an operation skill library of the digital robot;

[0017] The mobile intelligent unit is used to perform training based on the walking gait and trajectory data collected from the natural person's motion information and the physical robot data. After training, it can enable the physical robot aligned with the digital robot's motion to have the most energy-efficient and natural gait.

[0018] Preferably, the digital simulation module also includes a digital simulated person, which is a twin simulated person of a natural person, and is used to superimpose the virtual image of the natural person into the physical environment according to the information of the natural person, perform corresponding remote operation actions, and display it to the natural person for viewing through the interactive device in the human-computer interaction module.

[0019] Preferably, the human-computer interaction module includes a perception submodule and a feedback submodule.

[0020] The perception submodule is used to perceive the information of the natural person by adopting homogeneous perception, heterogeneous perception and / or natural perception, and transmit the information to the prediction simulation module;

[0021] The feedback submodule is used to feed back the status information fed back by the digital robot and / or the physical robot to the natural person through visual feedback, force feedback and / or auditory feedback.

[0022] Preferably, the physical robot includes a robot joint control module and a physical robot mechanical body, and the robot joint control module is used to convert the merged motion instruction sequence into torque or current information and send it to the joint motor in the physical robot mechanical body to realize physical interaction between the physical robot and the physical environment.

[0023] As another aspect of the present invention, the following technical solutions are also provided:

[0024] A "human-machine-data" three-in-one robot teleoperation method based on the above system inputs the natural person's information into a trained decision-making intelligence unit, which plans the corresponding teleoperation task and outputs the planning results to the operation intelligence unit and the movement intelligence unit;

[0025] The operation intelligent unit selects skills that match the planning results from the skill library, and the mobile intelligent unit plans the walking path and gait of the digital robot similar to human walking based on the planning results. The digital robot performs corresponding actions based on the skills selected by the operation intelligent unit and the walking path and gait planned by the mobile intelligent unit, and aligns the actions to the physical robot.

[0026] As another aspect of the present invention, the following technical solutions are also provided:

[0027] A "human-machine-digital" three-in-one robot teleoperation and data acquisition method based on the above system captures the operator's whole-body movements through the human-machine interaction module, drives the digital robot to run in a virtual environment through the predictive simulation module to execute the operator's movements, and drives the physical robot to interact with the physical environment after determining that there are no safety issues. The status of the physical robot and the digital robot are displayed to the operator through the human-machine interaction module in a virtual-real fusion manner; throughout the entire process, the data generated by the human-machine interaction module, the physical robot and / or the digital simulation module are collected through the data management module.

[0028] As another aspect of the present invention, the following technical solutions are also provided:

[0029] Based on the above system, a "human-machine-digital" three-in-one robot remote operation and data collection method is proposed. Natural persons perceive and control digital robots to run in a virtual environment and perform designated tasks through the human-computer interaction module and the predictive simulation module, and at the same time collect digital robot operation data through the data management module.

[0030] As another aspect of the present invention, the following technical solutions are also provided:

[0031] A "human-machine-digital" three-in-one robot teleoperation and data acquisition method based on the above system captures the operator's full-body movements through the human-machine interaction module, drives the digital robot to operate in a virtual environment through the predictive simulation module, executes the operator's movements, and drives the physical robot to interact with the physical environment after determining that there are no safety issues;

[0032] The operator's whole-body movements are transmitted to the digital mannequin through the human-computer interaction module, and the digital mannequin is superimposed on the physical environment to execute the operator's movements;

[0033] The status of the physical robot, digital robot and digital simulated human are displayed to the operator through the human-computer interaction module in a virtual-real fusion manner;

[0034] During the entire process, the data generated by the human-computer interaction module, the physical robot and / or the digital simulation module is collected through the data management module.

[0035] In general, the above technical solutions conceived by the present invention have the following beneficial effects compared with the prior art:

[0036] 1. Natural humans and physical robots differ significantly in terms of time, space, and logic. In terms of time, physical robots rely on precise clock calculations and lack the subjective time perception of humans. In terms of space, robots can only represent space through discrete data and symbols, lacking the integrated perception of humans. In terms of logic, physical robots are limited to preset rules and struggle to achieve the dynamic and complex planning that humans achieve. The present invention's "human-machine-digital" ternary system differs from the "two-body" relationship of digital twins by innovatively introducing the unique element of "digital robots," significantly enhancing the system's complexity and functionality.

[0037] The present invention's "human-machine-digital" three-in-one robotic teleoperation and data acquisition system designs a digital robot as an intelligent bridge between a natural human and a physical robot, achieving precise alignment in three dimensions. In the temporal dimension, the digital human, through a predictive simulation module, simulates human operating instructions in advance, verifying their safety in a virtual environment before transmitting them to the physical robot. This reduces operational delays and improves real-time performance and safety. In the spatial dimension, the digital robot can perform spatial mapping within the virtual environment, accurately mapping the human operating space to the robot's workspace, overcoming spatial differences and achieving precise control. In the logical dimension, the digital human, through a digital simulation module, optimizes and adjusts human operating instructions to align them with the motion logic and constraints of the physical robot, ensuring their executability and rationality. Furthermore, the system possesses data management capabilities to support continuous system optimization.

[0038] The present invention provides a "human-machine-digital" three-in-one robot teleoperation and data acquisition system, which introduces a digital simulation module. Through predictive simulation and safety verification, it greatly reduces the delay problem in teleoperation and the risk of damage to the physical robot caused by operational errors, thereby ensuring the safety of operation; through the parallel data acquisition mode, the system can significantly improve the efficiency of data generation and meet the robot training tasks with large-scale data requirements; by organically combining the human-machine interaction module (natural person), physical robot and digital robot, a closed-loop operation process is constructed, which promotes the development of robot teleoperation systems in terms of multi-information fusion, safe and agile operation, etc., and has broad application prospects.

[0039] 2. The present invention provides a "human-machine-digital" three-in-one robot remote operation and data acquisition system, which provides a means for physical robots to learn human movements. Since the physical properties and joint structures of physical robots are completely different from those of humans, simple position replication makes it difficult for physical robots to maintain balance. The present invention first uses simulation data to train digital robots so that they can maintain balance autonomously. Then, by remapping human movements onto the robot, it can both imitate human movements and maintain its own balance. On this basis, the physical robot imitates human movements to complete a series of operation tasks, and collects anthropomorphic operation data that conforms to the characteristics of the physical robot, which can provide high-quality training data for subsequent physical robots to complete similar tasks autonomously.

[0040] 3. The present invention provides a "human-machine-digital" three-in-one robot teleoperation and data acquisition system. By introducing a digital robot, the next action of the physical robot is first predicted and simulated in the simulation to avoid the problem of the physical robot colliding in the real environment. At the same time, by utilizing a large amount of physical robot operation data, the digital robot can predict the robot action of the next several steps in advance based on the received action sequence, effectively eliminating the time delay caused by remote teleoperation. To correct the errors generated by the movement of the digital robot and the physical robot, the present invention can automatically correct the errors of the digital robot and the physical robot after each operation to ensure the reliability of the simulation. At the same time, the operator can clearly observe the virtual and real images of the digital robot and the physical robot in the MR helmet, providing the operator (natural person) with a more friendly teleoperation experience.

[0041] 4. The present invention provides a "human-machine-data" three-in-one robot remote operation and data acquisition system. The intelligent enhancement submodule has decision-making intelligence, operational intelligence and mobile intelligence, which can continuously expand the skill library and realize the generalization capability of robot tasks.

[0042] 5. The present invention provides a "human-machine-data" three-in-one robot teleoperation and data acquisition system. Its multimodal perception and feedback mechanism integrates visual, auditory, force and other information, and can provide real-time feedback on the robot's status to the operator, providing a more intuitive and immersive interactive experience.

[0043] 6. The present invention provides a "human-machine-digital" three-in-one robot remote operation method. After the intelligent enhancement sub-module is trained, the decision-making intelligent unit can directly plan tasks. The operation intelligent unit and the mobile intelligent unit can directly select the operation skills of the digital robot and plan the walking path and gait of the digital robot from the skill library, so that the digital robot can perform corresponding actions and align the actions with the physical robot without going through the predictive simulation module. The system is continuously optimized and the intelligence, accuracy and convenience of remote operation are improved.

[0044] 7. The present invention provides a "man-machine-number" three-in-one robot remote operation method, which provides multiple operation modes, promotes the development of robot remote operation systems in multi-information fusion, intelligence enhancement and autonomous decision-making, and has broad application prospects. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] Figure 1 Schematic diagram of the relationship between the three main objects of a "man-machine-data" three-in-one robot teleoperation and data acquisition system in an embodiment of the present invention;

[0046] Figure 2 Schematic diagram of the overall framework of a "man-machine-data" three-in-one robot teleoperation and data acquisition system in an embodiment of the present invention;

[0047] Figure 3 Schematic diagram of the structure of the human-computer interaction module in an embodiment of the present invention;

[0048] Figure 4 2 is a schematic diagram of the structure of the prediction simulation module in an embodiment of the present invention;

[0049] Figure 5 This is a schematic diagram of the structure of the data management and intelligence enhancement module in an embodiment of the present invention;

[0050] Figure 6 This is a flow chart of a data acquisition mode of a "man-machine-data" three-in-one robot teleoperation and data acquisition system in an embodiment of the present invention;

[0051] Figure 7 This is a flow chart of data acquisition mode 2 of a "man-machine-data" three-in-one robot teleoperation and data acquisition system in an embodiment of the present invention;

[0052] Figure 8 This is a three-flow chart of the data acquisition mode of a "man-machine-data" three-in-one robot teleoperation and data acquisition system in an embodiment of the present invention.

[0053] Throughout the drawings, the same reference numerals are used to denote the same elements or structures, wherein:

[0054] 1. Natural person; 2. Physical robot; 3. Digital human; 4. Virtual environment; 5. Physical environment; 6. Network; 7. Digital simulation module; 11. Human-computer interaction module; 21. Robot joint control module; 22. Physical robot mechanical body; 3-1. Digital simulated human; 3-2. Digital robot; 31. Prediction simulation module; 32. Data management and intelligent enhancement module; 111. Perception module; 112. Feedback module; 321. Data management module; 322. Intelligent enhancement module. DETAILED DESCRIPTION

[0055] In order to make the objectives, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely for the purpose of explaining the present invention and are not intended to limit the present invention. In addition, the technical features involved in the various embodiments of the present invention described below may be combined with each other as long as they do not conflict with each other.

[0056] The present invention provides a "human-machine-digital" three-in-one robot teleoperation and data acquisition system that combines teleoperation, data acquisition, and intelligence enhancement. By introducing a "human-machine-digital" three-in-one collaborative architecture and utilizing a digital human as an intermediary between a natural person and a physical robot, a high degree of collaboration between the natural person, physical robot, and digital human is achieved, as well as efficient data acquisition and precise operational control. At the same time, by using a virtual environment to predict and simulate operational behavior, it is possible to significantly reduce delays in teleoperation and ensure the safety of physical robot operation. At the same time, the system can effectively improve data acquisition efficiency, provide solid data support for the robot's autonomous learning and intelligence enhancement, and further enhance the robot's autonomous learning and intelligence enhancement capabilities.

[0057] like Figure 1 As shown, an embodiment of the present invention provides a "human-machine-digital" three-in-one robotic teleoperation and data acquisition system comprising three main objects: a natural person 1, a physical robot 2, and a digital human 3. Natural person 1 refers to a real human on Earth; physical robot 2 refers to a real robot that can interact with the real physical environment, primarily a humanoid robot, but also robotic entities of other shapes. Digital human 3 is a digitized human simulated in a computer with the same physical form as a human or robot. Interaction and alignment between the human and the physical robot are achieved through the digital human.

[0058] like Figure 2 As shown, the overall framework of the "human-machine-digital" three-in-one robot teleoperation and data acquisition system of the present invention includes a natural person 1, a physical robot 2, a digital human 3, a virtual environment 4, a physical environment 5, a network 6, and a digital simulation module 7. The physical robot 2 includes a robot joint control module 21 and a physical robot mechanical body 22. The robot joint control module 21 can receive instructions to drive the physical robot mechanical body 22 to move. At the same time, if it determines that the current state is dangerous during the execution of the instruction movement, it terminates the robot movement or resets it to protect the physical robot mechanical body 22 from damage in special, emergency, or dangerous situations.

[0059] The digital simulation module 7 includes a prediction and simulation module 31, a data management and intelligence enhancement module 32, a digital human 3, and a virtual environment 4. The digital human 3 includes a digital simulated human 3-1 and a digital robot 3-2. Digital simulated human 3-1 is a human twin, while digital robot 3-2 is a physical robot 2 twin. The prediction and simulation module 31 and the data management and intelligence enhancement module 32 serve as the brains of the digital human 3. The virtual environment 4 and the physical environment 5 are twin environments, with a time-space mapping relationship.

[0060] like Figure 3 As shown, the human-computer interaction module 11 includes a perception submodule 111 and a feedback submodule 112. The perception submodule 111 can digitize human information into digital information that can be understood by the digital robot and transmit it to the prediction simulation module 31, which is a process from human to machine. The feedback submodule 112 can feedback the robot's information to the human in a form that can be perceived by human senses, which is a process from machine to human. Among them, the perception submodule 111 includes three modes: homogeneous perception, heterogeneous perception, and natural perception. Homogeneous perception refers to the fact that the joints of the perception device and the robot joints are homogeneous, such as an exoskeleton, and the information of the human operating the perception device can be directly used for physical robot control without processing; heterogeneous perception refers to the fact that the perception device and the robot joint configuration are different, and the perception information needs to be processed (such as inverse kinematics solution, etc.) before it can be used for physical robot control, such as hand controllers; natural perception refers to capturing the human body's language information and remapping it in a specific way to the control of the physical robot, such as motion capture equipment, VR equipment, microphones, etc.

[0061] The feedback submodule 112 includes three methods: visual feedback information, which can include RGB images, virtual and real fusion AR (augmented reality), MR (mixed reality), and other methods as needed. The operator can intuitively see the operating status of the physical robot and the digital robot. The second feedback method is force feedback, which requires the use of relevant equipment to transmit the force felt by the physical and digital robots to the operator. The third feedback method is auditory feedback, in which the physical robot provides relevant information to the operator through voice feedback. These perception and feedback methods are not mutually exclusive and are usually included in the operation process.

[0062] like Figure 4As shown, the human-computer interaction module 11 transmits the motion information of the natural person 1 to the prediction simulation module 31. The prediction simulation module 31 has two main functions: one is human-computer alignment, and the other is prediction simulation. Due to the large difference between the natural person 1 and the physical robot 2, direct remote operation is not possible. Therefore, it is necessary to first perform human-computer alignment in the digital simulation module 7, that is, the alignment of the human's motion information with the physical robot's motion information in data (alignment of the digital simulation person with the digital robot). The human-computer alignment data is converted into the spatial position of the physical robot joint, thereby obtaining the motion instruction sequence 1, 2...M of the digital robot, that is, the rotation angle of the physical robot joint motor; if the interactive device and the robot are of the same type, the human-computer alignment data does not need to be converted too much, but like natural interaction or heterogeneous interaction, the human-computer joints need to be remapped first, and the inverse kinematics solution is used to calculate the spatial position of the robot joint to drive the robot movement. Since teleoperation may have more or less time delay or unstable connection problems, there will be a certain lag or danger in directly mapping the joint position of the human to the robot. Therefore, the process is to first verify the human body information instructions on the digital robot 3-2. During this process, the digital robot 3-2 will take the action instruction sequence 1, 2...M of the previous period and the feedback information of the physical robot 2 as input, and predict the subsequent several action instructions M+1, M+2..., M+N in real time, where the value of N depends on the specific time delay; the action instruction sequence 1, 2...M+N is input into the digital robot 3- 2 causes it to perform the corresponding actions in the virtual environment 4. If there are safety issues such as collisions during execution, execution is halted and the action command sequence is not sent to the physical robot 2. Accordingly, the physical robot 2 does not execute the action command sequence. If there are no safety issues during execution, the digital robot 3-2 is verified to be safe to operate and the action command sequence 1, 2…M+N is sent to the robot joint control module 21 of the physical robot 2. The action command information is converted into torque or current information and sent to the joint motors in the mechanical body 22 of the physical robot 2, thus achieving physical interaction between the physical robot 2 and the physical environment 5. After the physical robot 2 interacts with the physical environment 5, it will feedback relevant state information to the prediction simulation module 31. Due to the differences between the digital robot 3-2 and the physical robot 2 in the simulation, this state information is used to correct and update the state of the digital robot 3-2 in real time to ensure that it remains consistent with the state of the physical robot 2. At the same time, this feedback information is fed back to the human through the human-machine interaction module 11 in a virtual-realistic manner, completing the closed loop. The entire process will be communicated and interacted by the robot-specific network 6. The human motion data, digital human information and physical robot information generated in this process will be stored in the data management and intelligence enhancement module 32 according to the needs of data set production, and used for subsequent update training of the general robot basic model or other purposes. After training, the model can be used to automatically operate other types of robots with similar configurations to complete similar tasks.

[0063] Predictive simulation module 31 predicts and verifies the interaction information between digital human 3 and virtual environment 4 in advance, and sends it to physical robot 2 after verification. This reduces the delay and risk of misoperation during teleoperation. Because the state of physical robot 2 after each operation may differ from that of digital robot 3-2, digital robot 3-2 will correct and update its own state in real time based on the current state information of physical robot 2, ensuring that it remains consistent with the state of physical robot 2, reducing the movement deviation between digital robot 3-2 and physical robot 2.

[0064] like Figure 5As shown, the data management and intelligence enhancement module 32 includes a data management submodule 321 and an intelligence enhancement submodule 322. The data management submodule 321 is primarily used to collect and store human language and motion information, motion timing information data, robot simulation data, physical robot data, and AI-synthesized data. Human motion data is primarily collected from the motion of a remotely operated robot, including motion and timing information of human motion. Robot simulation data consists of two parts: one is the predicted simulation data of a digital human, and the other is data generated by a human directly controlling the motion of a digital human in a virtual scene or by training a digital robot in a virtual environment. Physical robot data comes from data generated by remotely operating a robot using the ternary system of the present invention. Synthesized data is primarily generated by using AI to synthesize more similar data from the previously collected data. Since data on physical robots and human motion is difficult to collect in large quantities, most of this data is simulated and synthesized data, but the quality of this data is inferior to that of human motion data and physical robots. The intelligence enhancement submodule 322 includes a prediction training unit, a decision-making intelligence unit, an operation intelligence unit, and a motion intelligence unit. The prediction training unit uses collected data as a training set to continuously iterate the prediction simulation module, ensuring more accurate predictions. The decision-making intelligence unit is primarily trained based on human language and motion information, motion timing information, and physical robot data collected by the system. After training, it becomes a digital brain that enables the digital robot to mimic human motion timing and perform designated tasks. It can not only understand human language behavior, but also plan and make decisions about tasks. The decision-making intelligence unit does not change with changes in the physical robot 2's morphology; that is, its trained model can be used to automatically operate other robots of similar configurations to complete similar tasks. The operation intelligence unit is primarily trained based on human language and motion information, physical robot data, and simulated and synthesized data collected by the system for specific tasks. After training, the digital robot of a specific configuration can generalize and perform similar tasks. This ultimately forms a digital robot operation skill library containing various trained digital robot skills, which is updated in real time as the database expands. The mobile intelligent unit mainly includes two aspects: one is gait control similar to human walking, and the other is planning of human walking paths; the walking gait and trajectory data in human motion information and physical robot data are used to train the mobile intelligent unit, so that after training, the physical robot aligned with the digital robot's movements has the most energy-saving and natural gait.

[0065] After each unit of the intelligent enhancement sub-module 322 is trained, human language or movement information is input into the decision-making intelligent unit of the intelligent enhancement sub-module 322. The decision-making intelligent unit can plan the task and output the task plan to the operation intelligent unit and the mobile intelligent unit. The operation intelligent unit selects skills that match the task plan from the skill library according to the task plan. The mobile intelligent unit plans the walking path and gait of the digital robot similar to human walking according to the task plan. The digital robot performs corresponding actions according to the skills selected by the operation intelligent unit and the walking path and gait planned by the mobile intelligent unit, and aligns the actions to the physical robot.

[0066] The "man-machine-data" three-in-one robot teleoperation and data acquisition system proposed in this invention has three modes. Mode 1 is a single "man-machine-data" teleoperation data acquisition. Figure 6 As shown, the operator uses AR, MR and other devices to capture the operator's whole-body movements, and then drives the digital robot 3-2 to run in the virtual environment through the predictive simulation module. After judging that it is safe and there is no collision, the physical robot 2 is driven to interact with the physical environment. The status of the physical robot 2 and the digital robot will be displayed in the display unit (such as a helmet) of the human-computer interaction module through the fusion of virtual and real. In this process, the relevant motion data of "human, machine, and digital" will be collected to provide data samples for the training of subsequent strategies.

[0067] Mode 2 is parallel “human-machine-data” teleoperation data collection. Since training a robot to achieve a task often requires a large amount of data, using a single physical robot for data collection is often inefficient. Therefore, this system can be used for parallel data collection. Figure 7 As shown, a teaching commander is added to the single "human-machine-digital" teleoperation data collection system, and the number of operators can be expanded as needed. At this point, Digital Human 3 has two forms. One is a virtual human form, namely Digital Simulator 3-1. This form of Digital Human 3 does not participate in the actual teleoperation, but instead superimposes its virtual image on the physical environment. Through the interactive device worn by the operator (such as VR glasses), Digital Simulator 3-1 can be seen demonstrating the teaching commander's movements in the physical environment, allowing all operators to intuitively observe the teaching commander's movements. The other form is the form of Digital Robot 3-2, which is used to align the operator with the physical robot 2 and provide the operator with Mode 1 physical robot data collection.

[0068] Mode 3 is virtual teleoperation data collection. Figure 8As shown, a virtual environment and digital robots are generated through computer graphics and other technologies. People use VR, AR and other devices to perceive and control digital robots to complete specific tasks in the virtual environment, and at the same time collect digital robot operation data. This mode can enhance the immersive experience of human remote operation. Since physical robots are not used, there is no need to worry about damage to the physical robot due to operational errors.

[0069] There are significant differences between natural person 1 and physical robot 2 in terms of time, space, and logic. In the temporal dimension, physical robot 2 relies on precise clock calculations and lacks the subjective time perception of natural person 1. In the spatial dimension, physical robot 2 can only represent space through discrete data and symbols, lacking the comprehensive perception of natural person 1. In the logical dimension, physical robot 2 is limited to preset rules and cannot achieve the same dynamic and complex planning as natural person 1. This "human-machine-digital" three-in-one system, unlike the "two-body" relationship of digital twins, innovatively introduces the unique element of "digital human," significantly enhancing the system's complexity and functionality.

[0070] The present invention is different from the traditional digital twin system. First of all, the subject of the present invention includes three objects: human, machine, and number. There are differences between the three and they are not twins. At the same time, the traditional digital twin is two systems running simultaneously, but in the present invention, the digital human is the information transmission medium between human and machine, and the physical robot is the embodiment of the digital human. The digital human is superior to the physical robot, and the physical robot is controlled by the digital human.

[0071] The proposed "human-machine-digital" three-in-one robotic teleoperation and data acquisition system utilizes a digital human (3) as an intelligent bridge between a physical human (1) and a physical robot (2), achieving precise alignment in three dimensions. In the temporal dimension, the digital robot uses a predictive simulation module to simulate human operating instructions in advance, verifying their safety in a virtual environment before transmitting them to the physical robot (2). This reduces operational latency and improves real-time performance and safety. In the spatial dimension, the predictive simulation module performs spatial mapping and coordinate transformation within the virtual environment, accurately mapping the human's operating space to the digital robot's workspace, overcoming spatial differences and achieving precise control. In the logical dimension, an intelligent enhancement submodule optimizes and adjusts human operating instructions to align them with the motion logic and constraints of the physical robot (2), ensuring their executability and rationality. Furthermore, the digital simulation module possesses data management capabilities, collecting, storing, and analyzing operational data to support continuous system optimization.

[0072] In the system of the present invention, natural humans interact with the system through the human-computer interaction module, while physical robots physically interact with the real environment. The digital human, acting as a bridge between humans and physical robots, possesses a wealth of historical data and prior knowledge, serving as a digital avatar of the fusion of humans and robots. It performs predictive simulation and safety verification in a virtual environment to ensure the safety and efficiency of teleoperation. The system's data management and intelligence enhancement module collects and manages the human motion data, simulation data, and physical robot data generated during the operation for subsequent intelligent training and task execution. The system supports teleoperation data acquisition in single, parallel, and virtual modes, improving data acquisition efficiency, enhancing the teleoperation experience, and reducing the risk of damage to the physical robot.

[0073] It will be easily understood by those skilled in the art that the above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A "man-machine-data" three-in-one robot teleoperation and data acquisition system, characterized by: It includes a human-computer interaction module, a physical robot being remotely operated, and a digital simulation module located in the cloud and closer to the physical robot than the human-computer interaction module. The digital simulation module includes a prediction simulation module, a digital robot, a virtual environment, and a data management module. The human-computer interaction module is used to transmit the natural person's information to the prediction simulation module, and to feed back the status information fed back by the digital robot and / or the physical robot to the natural person; The prediction simulation module is used to align the information of a natural person with the execution action information of the physical robot, that is, to obtain human-machine alignment data, convert the human-machine alignment data to obtain the rotation angle of the physical robot joint motor, that is, to obtain the action instruction sequence of the digital robot, and predict the subsequent several action instructions of the action instruction sequence in real time based on the action instruction sequence and the feedback information of the physical robot, and combine the action instruction sequence and the subsequent several action instructions and input them into the digital robot; it is also used to correct the state of the digital robot in real time based on the state information fed back by the physical robot, so that it is always consistent with the state of the physical robot; The digital robot is used to perform corresponding actions in the virtual environment according to the combined action instruction sequence. If there is a safety problem during the execution, the execution is stopped; otherwise, the combined action instruction sequence is sent to the physical robot; The physical robot is used to interact with the physical environment according to the combined action instruction sequence, and to feed back state information to the prediction simulation module and the human-computer interaction module in real time; the virtual environment and the physical environment are twin environments; The data management module is used to collect and manage data during the operation of the human-computer interaction module, the physical robot and / or the digital simulation module.

2. The "man-machine-data" three-in-one robot teleoperation and data acquisition system according to claim 1, characterized in that: The data collected by the data management module includes the information of the natural person, digital simulation information, physical robot data and / or AI synthesized data. The information of the natural person includes the natural person's motion information, motion timing information and / or voice information. The digital simulation information includes the data generated by the predictive simulation module and the data generated by the digital robot moving or training in the virtual scene; the physical robot data includes the data generated by the physical robot during the remote operation process; the AI ​​synthesized data is data synthesized by AI using the information of the natural person, the digital simulation information and / or the physical robot data.

3. The "man-machine-data" three-in-one robot teleoperation and data acquisition system according to claim 2, characterized in that: The digital simulation module also includes an intelligent enhancement submodule, which includes a prediction training unit, a decision intelligence unit, an operation intelligence unit and a mobile intelligence unit, wherein: The prediction training unit is used to use the data collected by the data management module as a training set to train the prediction simulation module to make the prediction result more accurate; The decision-making intelligent unit is used to train based on the collected information of the natural person and the physical robot data, so that after training, it becomes a digital brain that enables the digital robot to imitate the timing of human movements to perform designated tasks; The operation intelligence unit is used to perform training based on the collected information of the natural person and the physical robot data, so that after training, it can enable the digital robot of a specified form to generalize and perform tasks, and form an operation skill library of the digital robot; The mobile intelligent unit is used to perform training based on the walking gait and trajectory data collected from the natural person's motion information and the physical robot data. After training, it can enable the physical robot aligned with the digital robot's motion to have the most energy-efficient and natural gait.

4. The "man-machine-data" three-in-one robot teleoperation and data acquisition system according to claim 1, characterized in that: The digital simulation module also includes a digital simulated person, which is a twin simulated person of a natural person, and is used to superimpose the virtual image of the natural person into the physical environment according to the information of the natural person, perform corresponding remote operation actions, and display it to the natural person for viewing through the interactive device in the human-computer interaction module.

5. The "man-machine-data" three-in-one robot teleoperation and data acquisition system according to claim 1, characterized in that: The human-computer interaction module includes a perception submodule and a feedback submodule. The perception submodule is used to perceive the information of the natural person by adopting homogeneous perception, heterogeneous perception and / or natural perception, and transmit the information to the prediction simulation module; The feedback submodule is used to feed back the status information fed back by the digital robot and / or the physical robot to the natural person through visual feedback, force feedback and / or auditory feedback.

6. The "man-machine-data" three-in-one robot teleoperation and data acquisition system according to claim 1, characterized in that: The physical robot includes a robot joint control module and a physical robot mechanical body. The robot joint control module is used to convert the combined motion instruction sequence into torque or current information and send it to the joint motor in the physical robot mechanical body to achieve physical interaction between the physical robot and the physical environment.

7. A "man-machine-data" three-in-one robot teleoperation method based on the system of claim 3, characterized in that: The natural person's information is input into the trained decision-making intelligent unit, which plans the corresponding teleoperation task and outputs the planning results to the operation intelligent unit and the mobile intelligent unit; The operation intelligent unit selects skills that match the planning results from the skill library, and the mobile intelligent unit plans the walking path and gait of the digital robot similar to human walking based on the planning results. The digital robot performs corresponding actions based on the skills selected by the operation intelligent unit and the walking path and gait planned by the mobile intelligent unit, and aligns the actions to the physical robot.

8. A "man-machine-data" three-in-one robot teleoperation and data acquisition method based on the system according to any one of claims 1 to 6, characterized in that: The operator's whole-body movements are captured through the human-computer interaction module, and the digital robot is driven to run in the virtual environment through the predictive simulation module to execute the operator's movements. After it is determined that there are no safety issues, the physical robot is driven to interact with the physical environment. The status of the physical robot and the digital robot are displayed to the operator through the human-computer interaction module in a virtual-real fusion manner; during the entire process, the data generated by the human-computer interaction module, the physical robot and / or the digital simulation module are collected through the data management module.

9. A "man-machine-data" three-in-one robot teleoperation and data acquisition method based on the system according to any one of claims 1 to 6, characterized in that: Natural persons perceive and control digital robots to run in a virtual environment and perform designated tasks through the human-computer interaction module and the predictive simulation module, and collect digital robot operation data through the data management module.

10. A "man-machine-data" three-in-one robot teleoperation and data acquisition method based on the system of claim 4, characterized in that: The human-computer interaction module captures the operator's full-body movements, and the predictive simulation module drives the digital robot to run in the virtual environment, execute the operator's movements, and drive the physical robot to interact with the physical environment after determining that there are no safety issues; The operator's whole-body movements are transmitted to the digital mannequin through the human-computer interaction module, and the digital mannequin is superimposed on the physical environment to execute the operator's movements; The status of the physical robot, digital robot and digital simulated human are displayed to the operator through the human-computer interaction module in a virtual-real fusion manner; During the entire process, the data generated by the human-computer interaction module, the physical robot and / or the digital simulation module is collected through the data management module.

Images