VR multi-person interaction system and method based on body tracking

Abstract

The invention discloses a VR multi-user interaction system and method based on body tracking, and the system comprises an image collection module, a body tracking unit, a multi-user interaction unit and VR equipment, and the VR equipment is used for presenting a virtual reality scene to a user; the image acquisition module is used for acquiring and tracking motion data of a user wearing the VR equipment and acquiring an actual environment where the user is located; the body tracking unit is used for processing data acquired by the image acquisition module and obtaining body action posture data, the output end of the body tracking unit is connected with the multi-user interaction unit, and the multi-user interaction unit is used for storing the synchronous state of each user in a scene, receiving and updating synchronous information, issuing synchronous data and dynamically generating a multi-user model; and the server is connected with VR equipment worn by the user and sends the virtual reality scene data processed by the multi-user interaction unit to the VR equipment. Multi-person interaction of the virtual scene is carried out by utilizing VR and body tracking, so that the interaction depth is improved, and the user experience immersion is improved.

Description

Technical Field

[0001] The present invention relates to the field of virtual reality interactive control, and in particular to a VR multi-person interactive system and method based on body tracking. Background Art

[0002] Virtual reality (VR) technology is a completely virtual three-dimensional environment generated by computers to enhance users' perception and interactive experience of the virtual world. It mainly relies on computer graphics, sensor technology and human-computer interaction technology, and interacts with the virtual world through head-mounted displays (HMDs) and other devices (handles, gloves). In existing VR application scenarios, users often wear HMD devices and handles to achieve some interactions and operations in the scene. Although HMD can provide users with a visual immersive experience, the interaction method is far from real natural interaction. This type of interaction lacks immersion and authenticity, resulting in an unnatural user experience.

[0003] VR devices with existing technology can only be experienced and controlled by one person, and cannot meet the needs of real interaction between multiple people. The traditional method of using handles for user virtual interaction has defects such as low interaction efficiency, accuracy, and complex control, and cannot meet the requirements of authenticity, participation, and immersive interaction. Summary of the invention

[0004] The purpose of the present invention is to overcome the shortcomings of the prior art and provide a VR multi-person interaction system and method based on body tracking. By integrating high-precision body tracking with the VR system, users can interact with the virtual environment directly through body movements (such as gestures, steps, etc.), thereby enhancing the naturalness and immersion of the interaction and achieving the purpose of multi-person interaction.

[0005] In order to achieve the above-mentioned purpose, the technical solution adopted by the present invention is: a VR multi-person interaction system based on body tracking, comprising an image acquisition module, a body tracking unit, a multi-person interaction unit and a VR device, wherein the user wears the VR device, and the VR device is used to present a virtual reality scene to the user; the image acquisition module is used to collect and track the motion data of the user wearing the VR device and to collect the actual environment in which the user is located, and its output end is connected to the body tracking unit; the body tracking unit is used to process the data collected by the image acquisition module and obtain body motion posture data, and its output end is connected to the multi-person interaction unit, and the multi-person interaction unit is used to store the synchronization status of each user in the scene, receive and update synchronization information, send synchronization data and dynamically generate a multi-user model, which is connected to the VR device worn by the user and sends it the virtual reality scene data processed by the multi-person interaction unit.

[0006] The image acquisition module uses a binocular camera to track and collect the actual environment and the target user's actions and movement data.

[0007] The binocular camera is set in an actual scene where the user wears VR, wherein the viewport of the binocular camera includes all movable ranges, collects and tracks actual environmental data within the movable range and motion data of the user's body movements and movements in the current actual environment.

[0008] The VR device includes an HMD head display for presenting a virtual reality scene. Through the VR device, the user can see the picture changes caused by his own movement, the effects of real-time interaction, and the positions, actions, movements of other users in the virtual scene, and the content modules being interacted with.

[0009] The body tracking unit includes a human body detection module, which uses the user image captured by the binocular camera to infer all 2D skeletons and key points through a neural network, and finally extracts a set of skeleton structures that match the user captured by the binocular camera.

[0010] The human tracking unit also includes a human adaptation module, which binds the skeleton structure extracted by the human detection module to a preset character model, and performs stable 3D motion simulation on the character model according to the movements made by the user to drive the task model to follow the movement.

[0011] The multi-person interaction unit includes a server module, which is used to store the synchronization status of each client and receive the updated synchronization information uploaded by the client. After receiving the synchronization information uploaded by a client, the server module will send the synchronization information to other clients. The server module is also used to dynamically generate character model objects. The character model objects of different users contain different user information.

[0012] The multi-person interaction unit also includes a client module, which is used to run a local virtual reality scene, upload the synchronization information of the current client and the operations performed by the client user to the server module, receive the synchronization information of other clients sent by the server module and update the virtual reality scene information in real time.

[0013] The multi-person interaction unit renders and outputs a virtual reality scene through processing, renders the scene generated by user interaction in real time, and outputs it to a VR device for display.

[0014] A VR multi-person interaction method based on body tracking, comprising: after a user wears a VR device, a binocular camera is used to collect the user's motion data and environmental parameter information, and the data is sent to a body tracking unit for processing and sending the data to a multi-person interaction unit; the multi-person interaction unit processes the data uploaded by the body tracking unit to obtain a corresponding virtual reality scene, and sends the scene to the VR device for display.

[0015] The advantages of the present invention are: it allows users to interact with virtual reality scenes through their own movements, allows multiple users to interact with scenes and users in the same virtual reality scene, uses VR and body tracking to perform multi-person interaction in virtual scenes, improves the depth of interaction, enhances the user experience immersion, and strengthens the actual use effect of the virtual scene content itself. The specific technical effects are:

[0016] 1. Improve interaction depth and immersion: The binocular camera tracks the user's movements and spatial position information in real time and accurately projects it into the virtual reality scene, allowing the user to interact directly with the virtual environment through their own movements, greatly increasing the real-time and immersive sense of interaction. Body tracking technology allows users to experience deeper engagement and more natural interaction in the VR experience.

[0017] 2. Achieve efficient multi-person synchronous interaction: The multi-person interaction unit can store, update and synchronize the status information of each user in real time, ensuring that the interactive behaviors of multiple users in the same virtual reality scene can be carried out synchronously, improving the experience quality of multi-person interaction, and expanding the application potential of VR in other fields such as remote collaboration, training and education.

[0018] 3. Improve detection accuracy and adaptability: The body tracking unit uses advanced neural network technology and multiple detection types to accurately infer the user's skeletal structure and key points, thereby accurately capturing and simulating the user's movements. At the same time, the human body adaptation module improves the system's adaptability and detection accuracy for different users by binding a preset character model and simulating the user's 3D movement in real time.

[0019] 4. Increase system flexibility and scalability: The multi-person interaction unit adopts a design that separates the server and the client, supporting both dedicated server mode and host server mode. This mode improves the flexibility and scalability of the system and can adapt to application scenarios of different scales and complexities.

[0020] 5. Optimize user experience and scene content: Through the combination of body tracking and multi-person interaction technology, it can provide users with a richer and more diverse virtual reality interactive experience. At the same time, it also improves the actual use effect of virtual scene content, making the scene content more vivid and realistic, thereby further enhancing user satisfaction. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] The following is a brief description of the contents expressed in the drawings of the present invention and the symbols in the drawings:

[0022] Figure 1 This is a schematic diagram of the principle of the VR multi-person interaction system based on body tracking of the present invention. DETAILED DESCRIPTION

[0023] The specific implementation of the present invention will be further explained in detail below by describing the optimal embodiment with reference to the accompanying drawings.

[0024] The present embodiment discloses a VR multi-person interaction system based on body tracking, including a binocular camera for tracking and collecting actual environment and user motion data; a VR device worn by a user for presenting a virtual reality scene to the user; a body tracking unit for processing motion data and spatial position information data collected by the binocular camera and projecting them into the virtual reality scene; and a multi-person interaction unit for storing the synchronization status of each user, receiving and updating synchronization information, sending synchronization data, and dynamically generating a multi-user model.

[0025] This embodiment is a VR multi-person interaction system based on body tracking, which allows users to interact with virtual reality scenes through their own movements, and allows multiple users to perform scene interaction and user interaction in the same virtual reality scene. It uses VR and body tracking to perform multi-person interaction in virtual scenes, improves the depth of interaction, enhances the user experience immersion, and strengthens the actual use effect of the virtual scene content itself.

[0026] In this embodiment, the VR multi-person interaction method based on body tracking includes collecting the user's motion data and environmental parameter information through the binocular camera after the user wears the VR device and sending it to the body tracking unit for processing the user's motion information and environmental parameter information and sending it to the multi-person interaction unit, and the multi-person interaction unit processes the data uploaded by the body tracking unit to obtain the corresponding virtual reality scene and sends it to the VR device for display. Among them, multiple users wearing VR devices waiting for virtual interaction are in a room with a binocular camera installed or a room specially set up with a binocular camera; after starting the interaction, the VR device will load the virtual reality scene information, and then the binocular camera will collect the action, behavior and other information of the user wearing the VR device in real time, and then analyze and identify the user's action and behavior information collected by the camera, and convert it into the control parameters in the virtual reality scene after analysis and recognition processing, and then control the objects, tasks, lenses, animations, etc. in the virtual reality scene in the VR device, so that the virtual reality screen in the VR device can be updated and changed according to the user's behavior. Each VR device in the same interactive scene is connected to the same interactive system. The cloud platform of the interactive system is used to control the virtual reality scene screen, that is, the VR devices are connected to the same cloud platform and each binocular camera is also connected to the platform. The platform analyzes and processes the movements, environment, behavior and other information of each VR device wearer, and then controls the objects, actions, etc. in the virtual reality scene corresponding to the user's behavior and actions, so that the control information of each VR device user can control the virtual reality screen, thereby realizing interactive control between multiple VR devices in one virtual reality screen.

[0027] like Figure 1 As shown, it is a structural diagram of a VR multi-person interactive control system based on body tracking provided in this embodiment. The VR multi-person interactive system based on body tracking includes:

[0028] Binocular cameras are used to track and collect motion data of the actual environment and the actions and movements of the target user. It uses binocular vision technology to simultaneously obtain two digital images of the object being measured from different angles, and calculates the parallax of corresponding points in the two images, thereby restoring the 3D geometric information of the object. Compared with other 3D vision technologies, such as LiDAR, it is less expensive and can identify richer information data while ensuring accurate collection of 3D geometric data. The collection of actual environmental data is used for the system to more accurately understand the user's spatial environment, including: the undulations of the ground, used to match the actual ground with the ground in the virtual scene; the distance to the wall, used to establish the tracking boundary; the depth information of the target tracking user, used to determine the distance between the user and the camera. The body tracking unit uses the above information data to establish a three-dimensional geometric coordinate system of the actual environment. This data is crucial for accurately calculating the user's body position and posture, and is also the key to improving the accuracy of body tracking. The collection of user actions and movements alone will affect the final interaction effect to a certain extent. There is a situation where the actual environment where the user is located is not an open space with ground or water surface. The lack of data collected from the actual environment will cause the user in the virtual scene to have problems such as position offset, floating, and sinking of characters.

[0029] VR device, which is a head-mounted display (HMD) used to present virtual reality scenes to users. When interacting, users need to wear the VR device;

[0030] The body tracking unit is used to process the motion data and spatial position information data collected by the binocular camera. The spatial position information data refers to the actual environment data and the user's motion data collected by the binocular camera. The body tracking unit processes the data to establish a three-dimensional geometric coordinate system of the actual environment and the three-dimensional coordinates of the user's position. The body tracking unit will send the processed three-dimensional geometric data to the multi-person interaction unit;

[0031] The multi-person interaction unit includes a server module and a client module. There is only one server module, while there can be multiple client modules. The server module is used to store the synchronization status of each user in the scene and receive updated synchronization data. The synchronization status refers to whether each user is in a synchronization state. This state can be understood as the user's online and offline. The synchronization data refers to the three-dimensional position data and interactive operation data of each user. These data are uploaded to the server module by each client module. The server module will also send the synchronization data of other client modules except a certain client module to the client module. The server module and the client module in the multi-person interaction unit will generate character models representing different users in real time and dynamically in the virtual scene according to the synchronization status and synchronization data.

[0032] When multiple people need to interact, the binocular camera of each user's scene is connected to the body tracking unit via wired or wireless means; at the same time, the multi-person interaction unit is connected to the VR device of each participating user via wired or wireless means; in this way, when multiple people interact, the binocular camera can collect the action behavior information of each person's interaction and send the updated virtual reality scene after the interaction to each VR device for display to the user through multi-person interaction.

[0033] In this embodiment, the binocular camera is set up at a fixed position in the actual scene. The actual scene here refers to the scene where the user wearing the VR device is located, which can be a specially set up venue or a room with a corresponding binocular camera installed; the binocular camera viewport includes all movable ranges, collects and tracks actual environmental data within the movable range, and motion data of the user's body movements and movements in the current actual environment.

[0034] VR devices include HMD head displays, which present virtual reality scenes to users. Users can see the changes in the images caused by their own movements, the different effects of real-time interaction, the positions, actions, and movements of other users in the virtual scenes, and the content of the virtual scenes they are interacting with through the devices. VR devices also include a core processor, which receives data from the multi-person interaction unit and then drives the HMD head display to display the corresponding virtual reality scenes.

[0035] The body tracking unit includes a human body detection module and a human body adaptation module. It uses the user image captured by the binocular camera to infer all 2D skeletons and key points through a neural network, and finally extracts a set of skeleton structures that match the user wearing the VR device. This module contains 3 types of body detection, namely Body18, Body34, and Body38. The digital part represents the number of key points contained in the skeleton structure extracted by different detection types. The key points are evenly distributed at each motion point of the extracted skeleton structure. The more key points the skeleton structure contains, the higher the detection accuracy. The human body detection module can detect skeleton structures with different numbers of key points respectively. Neural networks are used to realize functions such as human posture estimation, motion recognition and tracking. They are a type of deep learning. Neural network models such as convolutional neural networks (CNN) realize the inference, collection and detection of 2D skeletons and key points.

[0036] The human body adaptation module will bind the skeletal structure extracted by the human body detection module to the preset character model in the system, and perform stable 3D motion simulation on the character model as the user moves. The character model is a modeled character model used to represent the user in the system, and the modeled model includes a modeling skeleton to ensure that it can be bound to the skeletal structure extracted by the human body adaptation module. The module will also store body data in the form of a structure for character models with different skeletal structures, where the stored data includes a number, i.e. a fixed ID, for identifying objects of different characters; tracking status, i.e. defining whether the current state of the identified object is tracked or lost; operating status, i.e. defining whether the current state of the object is moving or stationary; position, i.e. locating the 3D vector of the character object based on the position information provided by the binocular camera; motion speed, i.e. the movement or motion speed of the character object in space stored in the form of a 3D vector; 2D and 3D bounding boxes, i.e. using four 2D coordinates or eight 3D coordinates define the boundary position of the human object in the image or in space; 2D and 3D key points, that is, the key points provided by the human detection module are stored in the form of 2D or 3D coordinates; 2D and 3D head bounding boxes, that is, four 2D coordinates or eight 3D coordinates are used to define the head limit boundary of the human object; head position, that is, the 3D coordinates that define the center of mass of the human head; joint local position, that is, the local position of each key point in the skeleton structure; joint local direction, that is, the local rotation of each key point in the skeleton structure stored in the form of quaternion; root direction, that is, the global root direction of the body stored in the form of quaternion. The body data stored in this module is captured by the binocular camera for source video and image, and then the deep learning model, that is, the neural network algorithm, detects and processes the human body in the source video and image. The processing process includes inferring key points and extracting skeleton structure by the body tracking module, and estimating the posture of the human body. After processing, the data such as number, tracking status, operation status, position, movement speed, etc. as mentioned above are stored in the form of a structure. These stored data are used to implement human body tracking, which enables real-time tracking of the human body through the human body position, movement posture information, etc. detected in continuous frames, so as to support the user character in the virtual scene to be consistent with the user's actual movements and postures.

[0037] The multi-person interaction unit includes a server module and a client module. The server module is used to store the synchronization status of each client module and receive the updated synchronization data uploaded by the client module. The synchronization status refers to whether each user is in a synchronization state, which can be understood as the user being online or offline. The synchronization data refers to the three-dimensional position data and interactive operation data of each user, which are uploaded to the server module by each client module. The storage of the synchronization status is used to determine the user's active status, determine whether the user is offline, and ensure the real-time maintenance of the user's status in the virtual scene. The storage of synchronization data is used to update the user's movement, operation and interaction in real time.

[0038] After receiving synchronization information uploaded by a client module, the server module will send the synchronization information to other client modules.

[0039] The server module is also used to dynamically generate character model objects. The character model objects of different users contain different user information, and their main identification information comes from the number information of the human body adaptation module.

[0040] The server module is divided into two optional identities, namely dedicated server and host server. The dedicated server is only an instance running as a server, while the host server is a client that carries the server function. It is a single created instance (called a host) that acts as both a server and a client.

[0041] The client module is used to run the local virtual reality scene, upload the synchronization data of the current client module and the operations performed by the user of the client module to the server module, receive the synchronization information of other client modules sent by the server module and update the virtual reality scene information in real time.

[0042] The server module and the client module under the host server identity will send processed data to the VR device. The VR device includes a core processor. The core processor of the VR device is used to receive data sent by the server module and the client module under the host server identity, and then drive the HMD head display to display the corresponding virtual reality scene, and update the virtual reality scene in real time as the data is continuously received.

[0043] The above is a VR multi-person interactive control solution based on body tracking. The above technical solution can be applied in multiple real-life scenarios. The following is an example of the application scenario:

[0044] Scenario 1:

[0045] Digital multi-person education: In a space equipped with binocular cameras, multiple users enter a virtual reality scene by wearing VR devices. The scene content is a preset historical story with educational significance. Users can freely explore the virtual reality scene by walking in the real space, and interact with the scene by raising their hands and touching it, such as helping a pedestrian who has fallen. Users can also interact with other users who have joined the scene, witness historical moments in the historical scene together, and collaborate in digital education.

[0046] Scenario 2:

[0047] The virtual museum pavilion sets up a binocular camera in a fixed area in the real museum pavilion. Users stand in the area and wear VR equipment to enter the virtual museum pavilion scene. Users can walk, touch, grab and other actions to realize the tour in the virtual pavilion and interact with the exhibits and introduction videos in the virtual pavilion in real time. At the same time, it supports multiple tourists to enter the virtual museum pavilion for tours and interactions. The pavilion staff can build services in the host server mode, and enter the virtual museum pavilion scene with tourists to introduce the contents of the pavilion and provide some guidance to tourists.

[0048] Scenario 3:

[0049] Multiplayer interactive VR games, where binocular cameras are set up in a designated area, and multiple users stand in the area wearing VR devices. Game operators open a dedicated server, and players join the dedicated server to start the game. The game can choose different VR scenes, and users can control the corresponding characters in the VR scene to interact with the scene and game content through various actions such as walking, jumping, raising hands to press, grabbing, pushing and pulling. The game can be a multi-user collaborative type or a personal competitive type.

[0050] Obviously, the specific implementation of the present invention is not limited to the above-mentioned methods. As long as various non-substantial improvements are made using the method concept and technical solution of the present invention, they are all within the protection scope of the present invention.

Claims

1. A VR multiplayer interaction system based on body tracking, Features: It includes an image acquisition module, a body tracking unit, a multi-person interaction unit and a VR device, wherein the user wears the VR device, and the VR device is used to present a virtual reality scene to the user; the image acquisition module is used to collect and track the motion data of the user wearing the VR device and to collect the actual environment in which the user is located, and its output end is connected to the body tracking unit; the body tracking unit is used to process the data collected by the image acquisition module and obtain body motion posture data, and its output end is connected to the multi-person interaction unit, and the multi-person interaction unit is used to store the synchronization status of each user in the scene, receive and update synchronization information, send synchronization data and dynamically generate a multi-user model, which is connected to the VR device worn by the user and sends the virtual reality scene data processed by the multi-person interaction unit to it.

2. A VR multi-person interactive system based on body tracking as claimed in claim 1, Features: The image acquisition module uses a binocular camera to track and collect the actual environment and the target user's actions and movement data.

3. A VR multi-person interactive system based on body tracking as claimed in claim 2, Features: The binocular camera is set in an actual scene where the user wears VR, wherein the viewport of the binocular camera includes all movable ranges, collects and tracks actual environmental data within the movable range and motion data of the user's body movements and movements in the current actual environment.

4. A VR multi-person interactive system based on body tracking as claimed in claim 1, Features: The VR device includes an HMD head display for presenting a virtual reality scene. Through the VR device, the user can see the picture changes caused by his own movement, the effects of real-time interaction, and the positions, actions, movements of other users in the virtual scene, and the content modules being interacted with.

5. A VR multi-person interactive system based on body tracking as claimed in claim 1, Features: The body tracking unit includes a human body detection module, which uses the user image captured by the binocular camera to infer all 2D skeletons and key points through a neural network, and finally extracts a set of skeleton structures that match the user captured by the binocular camera.

6. A VR multi-person interactive system based on body tracking as claimed in claim 5, Features: The human tracking unit also includes a human adaptation module, which binds the skeleton structure extracted by the human detection module to a preset character model, and performs stable 3D motion simulation on the character model according to the movements made by the user to drive the task model to follow the movement.

7. A VR multi-person interactive system based on body tracking as described in any one of claims 1 to 6, Features: The multi-person interaction unit includes a server module, which is used to store the synchronization status of each client and receive the updated synchronization information uploaded by the client. After receiving the synchronization information uploaded by a client, the server module will send the synchronization information to other clients. The server module is also used to dynamically generate character model objects. The character model objects of different users contain different user information.

8. A VR multi-person interactive system based on body tracking as claimed in claim 7, Features: The multi-person interaction unit also includes a client module, which is used to run a local virtual reality scene, upload the synchronization information of the current client and the operations performed by the client user to the server module, receive the synchronization information of other clients sent by the server module and update the virtual reality scene information in real time.

9. A VR multi-person interactive system based on body tracking as claimed in claim 7, Features: The multi-person interaction unit renders and outputs a virtual reality scene through processing, renders the scene generated by user interaction in real time, and outputs it to a VR device for display.

10. A VR multi-person interaction method based on body tracking, Features: After the user wears the VR device, the binocular camera collects the user's motion data and environmental parameter information and sends it to the body tracking unit for processing and sending it to the multi-person interaction unit. The multi-person interaction unit processes the data uploaded by the body tracking unit to obtain the corresponding virtual reality scene and sends it to the VR device for display.

Images