Human-robot collaborative training system and training method based on virtual reality and haptic feedback apparatus

Abstract

A human-robot collaborative training system and training method based on virtual reality and a haptic feedback apparatus. The training system comprises a physical simulation platform, a virtual-reality device and a wearable haptic feedback apparatus, wherein a computing center of the physical simulation platform computes formation state information and motion guidance information, and collaboratively controls a human-robot formation by means of a formation organization algorithm; the virtual-reality device comprises a virtual-reality head-mounted display and a virtual-reality handle; a user wears the virtual-reality head-mounted display to acquire a first-perspective image of a virtual agent, and uses the virtual-reality handle to control the motion and interaction of the human-robot formation; and the wearable haptic feedback apparatus performs haptic coding on the formation state information and motion guidance information that are obtained by the computing center, so as to provide a haptic feedback for the user. A human-robot collaborative training system uses a training method in which an immersive experience is combined with haptic feedback, thereby implementing multi-modal visual and haptic feedback, and better improving the training effect of human-robot collaboration.

Description

A human-machine collaborative training system and training method based on virtual reality and tactile feedback device Technical Field

[0001] The present invention belongs to the field of computer technology, and in particular relates to a human-computer collaborative training system based on virtual reality and tactile feedback devices. Background Art

[0002] Human-robot teaming, through the complementary capabilities and coordinated actions of individual robots, can often accomplish tasks that would be difficult for a single robot, improving the overall system's efficiency. Furthermore, human-robot teaming offers advantages such as high redundancy and robustness. Consequently, human-robot teaming has found widespread application in various fields. In the military, it is widely used in border patrols, geographic resource surveys, and reconnaissance and rescue operations. In disaster relief, it can significantly improve the efficiency of post-disaster search and rescue operations.

[0003] The application scenarios of human-machine teaming are often complex and changeable. However, in order to better improve the collaborative capabilities of human-machine teaming and train human-machine teaming to complete tasks in actual environments, there are still many shortcomings: setting up venues wastes manpower and material resources, and cannot meet the training needs in various special environments, such as extremely low visibility environments.

[0004] At the same time, in complex and dynamic environments, due to factors such as limited visual channels, overloaded visual channels, or non-line-of-sight, people's situational awareness capabilities are often weak. Feedback channels other than visual channels are needed to improve people's timely and sensitive perception of the status of robot teammates, environmental threats, and mission objectives.

[0005] Summary of the Invention

[0006] To solve the above problems, the present invention discloses a human-machine collaborative training system based on virtual reality and tactile feedback devices. It combines immersive experience with tactile feedback, does not require actual space, overcomes the limitations of weak situational awareness caused by limited vision and excessive load, realizes visual and tactile multimodal feedback, better improves the effect of human-machine collaboration, and can also meet the requirements of human-machine formation collaborative training in different special environments or different task requirements, with better results.

[0007] To achieve the above object, the technical solution of the present invention is as follows:

[0008] A human-machine collaborative training system based on virtual reality and tactile feedback device, comprising: a physical simulation platform, a virtual reality device, and a wearable tactile feedback device;

[0009] The physical simulation platform includes a human-machine formation, a physical scene and an interface. The human-machine formation moves in the physical scene and performs set tasks; the physical scene includes multiple interactive objects and obstacles; the human-machine formation includes a virtual agent, several robots and a computing center; the computing center uses robot sensor information and scene information to calculate formation status information and motion guidance information for task objectives, and collaboratively controls the movement and transformation of the human-machine formation through a formation organization algorithm; the interface can provide a way to modify the physical scene, formation organization algorithm, etc.

[0010] The virtual reality device includes a virtual reality headset and a virtual reality controller; the user wears the virtual reality headset to obtain a first-person perspective image of the virtual agent, and uses the virtual reality controller to control the movement and transformation of the human-machine formation and interact with interactive objects in the physical scene;

[0011] The wearable tactile feedback device has multiple tactile feedback modes, and tactilely encodes the formation status information and movement guidance information obtained by the computing center to provide tactile feedback to the user.

[0012] Preferably, the physical simulation platform, physical scene and human-machine formation model are physically built in the simulation engine Gazebo, and the computing center is coded and implemented in the Robot Operating System (ROS).

[0013] Preferably, the scene information includes a static map and a mission target point. The static map is obtained by simultaneous localization and mapping (SLAM). The mission target point is dynamically set by the remote device via 5G network communication.

[0014] Preferably, the robot is equipped with multiple sensors including: lidar, depth camera, inertial sensor, and communication module.

[0015] Preferably, the computing center collects robot sensor information and collaboratively controls the movement of the human-robot formation according to different formation organization algorithms and transformation instructions issued by the user.

[0016] Preferably, the computing center generates a planned path for the next mission target point based on the static map using the A* path planning algorithm; obtains the positioning of the human-machine formation using the adaptive Monte Carlo (AMCL) positioning algorithm based on the robot's sensor information, and then generates real-time motion guidance information.

[0017] Preferably, the computing center tracks the movement and transformation of the human-machine formation in real time, and updates the current formation status information in real time.

[0018] Preferably, the virtual reality device includes a virtual reality head display configured with a gyroscope and an accelerometer, and a handle configured with a gyroscope, an accelerometer and a vibration unit.

[0019] Preferably, the virtual reality device communicates with ROS via Unity3D, inputs control instructions into the human-machine formation, and transmits back the first-person perspective image of the virtual agent.

[0020] Preferably, the wearable tactile feedback device has multiple tactile feedback modes, and tactilely encodes and feeds back the formation status information and movement guidance information.

[0021] The beneficial effects of the present invention are:

[0022] (1) The use of virtual reality equipment allows users to obtain an immersive experience comparable to that of real scenes, making the human-machine team collaboration training effect closer to the real scene; at the same time, the use of tactile feedback channels to make up for the lack of visual feedback makes the human-machine team collaboration effect better, reduces the burden on the user's visual channel, and improves the stability of human-machine team collaboration and the efficiency of task completion.

[0023] (2) The physical scene and formation organization algorithm can be adjusted through the interface to meet the requirements of human-machine teaming collaborative training in different special environments or different mission requirements;

[0024] (3) It does not require the use of actual space, which greatly saves the time and resources required to build the scene. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] FIG1 is a schematic diagram of a human-machine collaborative training system based on virtual reality and tactile feedback devices according to an embodiment of the present invention. DETAILED DESCRIPTION

[0026] The present invention will be further explained below with reference to the accompanying drawings and specific embodiments. It should be understood that the following specific embodiments are only used to illustrate the present invention and are not used to limit the scope of the present invention.

[0027] 1 , a human-machine collaborative training system based on virtual reality and tactile feedback device according to the present invention comprises: a physical simulation platform, a virtual reality device, and a wearable tactile feedback device;

[0028] The physical simulation platform includes a human-machine formation, a physical scene and an interface. The human-machine formation moves in coordination in the physical scene and performs the tasks set by the trainer. The physical scene includes multiple interactive objects and obstacles. The human-machine formation includes a virtual agent, several robots and a computing center. The computing center uses robot lidar, depth camera, inertial sensor information and static map, task target point coordinates, and calculates formation status information including formation formation transformation, size transformation, surrounding obstacles, and motion guidance information for the task target, and collaboratively controls the movement and transformation of the human-machine formation through the formation organization algorithm - the leader-follower algorithm. The interface can provide a way to modify the physical scene, formation organization algorithm, etc.

[0029] The virtual reality device includes a virtual reality headset and a virtual reality handle; the user wears the virtual reality headset to obtain a first-person perspective image of the virtual agent, and uses the virtual reality handle to control the movement of the virtual agent in the human-machine formation, the formation formation change, the formation size change, and the interaction with the interactive objects in the physical scene.

[0030] The wearable tactile feedback device includes three modes: vibration feedback, squeeze feedback, and shear feedback. The formation status information and movement guidance information obtained by the computing center are tactilely encoded to provide tactile feedback to the user.

[0031] In practice, the trainer first adjusts the physical scene to the training requirements, sets the formation organization algorithm to a leader-follower algorithm, and sets the initial mission target point in the physical scene. The coordinates of the mission target point are stored as scene information in the human-machine formation computing center, and its visual characteristics are communicated to the user via voice. During the training process, the trainer can still dynamically set the mission target point, requiring the user to explore or interact.

[0032] Users use VR controllers to control the movements of a virtual agent in the physical scene. Through a VR headset, users receive first-person images of the virtual agent, ambient sounds, and the trainer's voice, providing an immersive and clear understanding of the surrounding environment and the situation of other team members. Users can adjust the team's formation and size as the surrounding environment changes. For example, when encountering a narrow area, the human-machine formation can be tightened.

[0033] The human-machine formation computing center collects robot sensor information and uses the adaptive Monte Carlo (AMCL) positioning algorithm to obtain the robot's current position. Based on the formation change and size change instructions issued by the user, it calculates the robot's expected position under the leader-follower algorithm, and uses the PID algorithm and artificial potential field method to jointly adjust the robot's control speed, thereby achieving coordinated control of each robot.

[0034] The A* algorithm is used to generate a planned path from the currently explored mission target point to the next mission target point using the static grid map of the scene obtained by SLAM technology and the coordinate information of the set mission target points. The planned path generates sub-targets according to the curvature algorithm and combines the current position of the navigator to generate real-time updated motion direction guidance information.

[0035] By real-time tracking of user transformation instructions and the robot's current position, the formation status information including the current formation's formation transformation, size transformation, surrounding obstacles, etc. is updated in real time.

[0036] The user wears a wearable tactile feedback device. The computing center transmits the real-time updated formation status information and movement guidance information to the main control part of the wearable tactile feedback device via the wireless serial port.

[0037] The formation change situation and surrounding obstacle information will be mapped to the vibration feedback part. The vibration feedback part uses the principle of apparent motion to convey information to the operator through the regular vibration of the vibration motor array. For example, when the formation change is completed, the motor array will vibrate according to the corresponding formation shape, prompting the user that the formation change has been completed, and the user no longer needs to divert attention to observe the completion status.

[0038] The size change will be mapped to the squeeze feedback part, which conveys information to the operator by controlling the different degrees of tightness and looseness of the arm strap.

[0039] The motion direction guidance will be mapped to the shear feedback part. The shear feedback transmits direction information to the operator through the shear skin of the motor end rotating body distributed at fixed angle intervals, prompting the operator to go in the direction to the task target point.

[0040] It should be noted that the above content merely illustrates the technical idea of ​​the present invention and cannot be used to limit the scope of protection of the present invention. For ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present invention. These improvements and modifications all fall within the scope of protection of the claims of the present invention.

Claims

1. A human-machine collaborative training system based on virtual reality and tactile feedback device, characterized in that: include: Physical simulation platform, virtual reality equipment, wearable tactile feedback device; The physical simulation platform includes a human-machine formation, a physical scene and an interface. The human-machine formation moves in the physical scene and performs a set task. The physical scene includes multiple interactive objects and obstacles. The human-machine formation includes a virtual agent, a robot and a computing center. The computing center uses robot sensor information and scene information to calculate formation state information and motion guidance information for task objectives, and coordinates the movement and transformation of the human-machine formation through a formation organization algorithm. The interface provides a way to modify the physical scene and the formation organization algorithm. The virtual reality device includes a virtual reality head display and a virtual reality handle; the user wears the virtual reality head display to obtain the first-person perspective image of the virtual agent, and uses the virtual reality handle to control the movement and transformation of the human-machine formation and the interaction with the interactive objects in the physical scene; The wearable tactile feedback device has multiple tactile feedback modes, and tactilely encodes the formation status information and movement guidance information obtained by the computing center to provide tactile feedback to the user.

2. The training method of the human-machine collaborative training system based on virtual reality and tactile feedback device according to claim 1, characterized in that: The training setter adjusts the physical scene according to the training needs, sets the formation collaborative control method, and sets the task target point in the physical simulation platform. The user wears the wearable tactile feedback device on the arm, uses the virtual reality handle to control the movement of the virtual agent in the physical simulation platform, obtains the virtual agent's first-person image, environmental sound and the training setter's voice through the virtual reality headset, and immersively understands the surrounding environment, the situation of other members of the formation, and the task target point. In the blind spot of the viewpoint, the user relies on the tactile force generated by the device on the surface of the arm skin to perceive the formation state and the surrounding environment, and goes to the task target point according to the guidance of the tactile force; the computing center of the human-machine formation collects the robot sensor information, and then uses the positioning algorithm to obtain the current position of the robot. According to the formation change and size change instructions issued by the user, it calculates the expected position of the robot under the leader-follower algorithm, and uses the PID algorithm and the artificial potential field method to jointly adjust the robot control speed, thereby realizing the collaborative control of each robot.

3. The human-machine collaborative training system based on virtual reality and tactile feedback device according to claim 1, characterized in that: The physical simulation platform, physical scene and human-machine formation model are built in the physical simulation engine Gazebo, and the computing center is implemented in the robot operating system coding.

4. The human-machine collaborative training system based on virtual reality and tactile feedback device according to claim 1, characterized in that: The scene information includes a static map and a task target point; the static map has been obtained by real-time positioning and map building technology; The mission target point is dynamically set by the remote device through 5G network communication.

5. The human-machine collaborative training system based on virtual reality and tactile feedback device according to claim 1, characterized in that: The robot is equipped with a variety of sensors including: laser radar, depth camera, inertial sensor, and communication module.

6. The human-machine collaborative training system based on virtual reality and tactile feedback device according to claim 1, characterized in that: The computing center collects robot sensor information and collaboratively controls the movement of the human-machine formation according to different formation organization algorithms and transformation instructions issued by the user.

7. The human-machine collaborative training system based on virtual reality and tactile feedback device according to claim 1, characterized in that: The computing center generates a planned path for the next mission target point based on the static map using the A* path planning algorithm; obtains the positioning of the human-machine formation using the adaptive Monte Carlo positioning algorithm based on the robot's sensor information, and then generates real-time motion guidance information.

8. The human-machine collaborative training system based on virtual reality and tactile feedback device according to claim 1, characterized in that: The computing center tracks the movement and transformation of the human-machine formation in real time, and updates the current formation status information in real time.

9. The human-machine collaborative training system based on virtual reality and tactile feedback device according to claim 1, characterized in that: The virtual reality device includes a virtual reality head display equipped with a gyroscope and an accelerometer, and a handle equipped with a gyroscope, an accelerometer and a vibration unit; the virtual reality device communicates with ROS through Unity3D, inputs control instructions into the human-machine formation, and transmits back the first-person perspective image of the virtual agent.

10. The human-machine collaborative training system based on virtual reality and tactile feedback device according to claim 1, characterized in that: The wearable tactile feedback device has a squeeze force feedback mode, a shear force feedback mode and a vibration feedback mode, and performs tactile encoding and feedback on formation state information and motion guidance information.

Images