Multi-person large-space virtual reality interaction system and method

Abstract

The embodiment of the invention provides a multi-person large-space virtual reality interaction system and method, and the system is characterized in that an inertia sensing module is used for obtaining the first attitude information of each key node except the head of a user, and transmitting the first attitude information to a virtual reality device; the virtual reality equipment is used for acquiring second attitude information of the head of the user, inputting the first attitude information into a pre-constructed attitude prediction model, and determining attitude information of each node of the whole body of the user; and the edge computing server is used for determining a motion capture result of each user according to the second posture information of each user uploaded by each virtual reality device and the posture information of each node of the whole body, generating a target scene picture through rendering according to the motion capture result, and sending the target scene picture to the virtual reality device of the corresponding user. On the basis of an inside-out positioning system mode, inertial sensing modules for capturing the posture of the whole body are added and worn on the waist, the two hands and the ankles, human body actions are predicted in combination with a model, a positioning tracking algorithm is improved, and the accuracy of the inside-out positioning mode is improved.

Description

Technical Field

[0001] The present invention relates to the field of virtual reality technology, and in particular to a multi-person large-space virtual reality interaction system and method. Background Art

[0002] Large-space virtual reality (VR) interaction platforms refer to virtual reality systems designed to operate in wide physical areas, allowing users to move freely in larger spaces than typical room-scale VR settings. This technology enables users to experience a more immersive experience beyond the limitations of a small room, and is ideal for applications that require wider physical interactions, such as training simulations, location-based entertainment, and collaborative environments. Currently, commonly used large-space VR interaction systems are: large-space VR interaction systems based on outside-in positioning, and large-space VR interaction systems based on inside-out positioning.

[0003] Among them, the "outside-in" positioning method is a tracking technology used in VR and augmented reality (AR) systems. It tracks the position and movement of users or devices in space by placing multiple sensors or cameras outside the physical environment, but it has problems such as occlusion, high equipment cost, and complex layout over large areas. The "inside-out" positioning method tracks the position and movement of users in physical space by integrating sensors or cameras on user devices (such as head displays or controllers). This method is the opposite of the "outside-in" positioning method. It does not require the installation of external sensors in the environment, making it more portable and easier to set up. However, it has problems such as positioning accuracy being affected by the environment, occlusion problems, and high computational load requirements. Summary of the invention

[0004] The present invention provides a multi-person large-space virtual reality interaction system and method, which achieves positioning accuracy and reduces system cost.

[0005] In a first aspect, an embodiment of the present invention provides a multi-person large-space virtual reality interactive system, comprising: an edge computing server, at least one virtual reality device, and at least one inertial sensing module, wherein the inertial sensing module is communicatively connected to the virtual reality device, and the virtual reality device is communicatively connected to the edge computing server; wherein:

[0006] The inertial sensing module is used to obtain the first posture information of each key node of the user except the head and send it to the virtual reality device;

[0007] The virtual reality device is used to obtain second posture information of the user's head, input the first posture information into a pre-built posture prediction model, determine the posture information of each node of the user's body, and send the second posture information of the user's head and the posture information of each node of the user's body to the edge computing server;

[0008] The edge computing server is used to determine the motion capture results of each user according to the second posture information of each user and the posture information of each node of the whole body uploaded by each virtual reality device, and to render and generate a target scene picture according to the motion capture result and send it to the virtual reality device corresponding to the user, so as to present the target scene picture to the user through the virtual reality device.

[0009] In a second aspect, an embodiment of the present invention provides a multi-person large space virtual reality interaction method, which is performed by the multi-person large space virtual reality interaction system described in the embodiment of the first aspect, and the method includes:

[0010] The inertial sensing module of the multi-person large-space virtual reality interactive system is used to obtain the first posture information of each key node of the user except the head and send it to the virtual reality device;

[0011] Obtaining second posture information of the user's head through the virtual reality device of the multi-person large-space virtual reality interactive system, inputting the first posture information into a pre-built posture prediction model, determining posture information of each node of the user's body, and sending the second posture information of the user's head and the posture information of each node of the user's body to the edge computing server;

[0012] Through the edge computing server of the multi-person large-space virtual reality interactive system, the motion capture results of each user are determined according to the second posture information of each user and the posture information of each node of the whole body uploaded by each virtual reality device, and the target scene picture is rendered and generated according to the motion capture result and sent to the virtual reality device corresponding to the user, so that the target scene picture is presented to the user through the virtual reality device.

[0013] The embodiment of the present invention provides a multi-person large-space virtual reality interaction system and method, the system includes an edge computing server, at least one virtual reality device and at least one inertial sensing module, the inertial sensing module is communicatively connected to the virtual reality device, and the virtual reality device is communicatively connected to the edge computing server; wherein the inertial sensing module is used to obtain first posture information of each key node of the user except the head and send it to the virtual reality device; the virtual reality device is used to obtain second posture information of the user's head, input the first posture information into a pre-built posture prediction model, determine the posture information of each node of the user's whole body, and send the second posture information of the user's head and the posture information of each node of the user's whole body to the edge computing server; the edge computing server is used to determine the motion capture result of each user according to the second posture information of each user and the posture information of each node of the whole body uploaded by each virtual reality device, and render and generate a target scene screen according to the motion capture result and send it to the virtual reality device corresponding to the user, so as to present the target scene screen to the user through the virtual reality device. Compared with the mainstream outside-in method, the above technical solution does not require external cameras and laser towers, which greatly reduces equipment and installation costs. At the same time, the lack of external sensor equipment saves debugging and calibration steps. Compared with the mainstream inside-out method, multiple inertial sensor modules are added for full-body motion capture, which provides richer interaction methods compared to simple handle control. It simplifies the number of system devices, reduces the difficulty of system construction, reduces the overall cost, and combines with the posture prediction model to improve positioning accuracy and expand the availability of the system.

[0014] It should be understood that the contents described in this section are not intended to identify the key or important features of the embodiments of the present invention, nor are they intended to limit the scope of the present invention. Other features of the present invention will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0016] Figure 1 It is a structural diagram of a large space virtual reality interactive system in the prior art;

[0017] Figure 2 A schematic diagram of the structure of a multi-person large-space virtual reality interactive system provided in Embodiment 1 of the present invention;

[0018] Figure 3 A schematic diagram of the structure of another multi-person large-space virtual reality interactive system provided in the first embodiment of the present invention;

[0019] Figure 4 A structural diagram of another multi-person large-space virtual reality interactive system provided in the first embodiment of the present invention;

[0020] Figure 5 A schematic diagram of a human skeleton model in a multi-person large-space virtual reality interactive system provided in Embodiment 1 of the present invention;

[0021] Figure 6 A schematic diagram of the structure of an initial prediction model in a multi-person large-space virtual reality interactive system provided in the first embodiment of the present invention;

[0022] Figure 7 A flowchart of a multi-person large-space virtual reality interaction method provided in Embodiment 2 of the present invention. DETAILED DESCRIPTION

[0023] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0024] It should be noted that the terms "original", "target", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0025] It should be noted that the large-space VR interactive platform is generally composed of the following parts: 11) Tracking system: used to track the position of characters in the space and provide coordinates for virtual display; 12) VR headset: used to provide users with virtual images and interactive interfaces, and can also provide user positioning information; 13) Control devices: generally have handheld controllers and tactile feedback devices, provide action interaction and physical feedback, and also have motion tracking functions; 14) Computing unit: provides high-performance computing services, is responsible for data processing, and generates virtual environments and interactive images; 15) Physical environment layout: provides site props and security monitoring.

[0026] At present, the commonly used large-space VR interactive systems include: large-space VR interactive systems based on outside-in positioning, and large-space VR interactive systems based on inside-out positioning. For example, Figure 1 It is a structural diagram of a large space virtual reality interactive system in the prior art. Figure 1 As shown in Figure 2, the "outside-in" positioning method is a tracking technology used in virtual reality and augmented reality systems. It tracks the position and movement of users or devices in space by placing multiple sensors or cameras outside the physical environment. The key steps are as follows: 21) Placement of external sensors: In the "outside-in" positioning system, sensors or cameras are installed outside the physical space (such as the ceiling, around the walls, or other fixed locations). These sensors are responsible for capturing the movement of target objects (such as VR helmets, controllers, or human-worn devices). Common sensor types include infrared cameras, laser scanners, RGB cameras, and other optical or electronic sensors with high precision. 22) Markers or trackers on target objects: Tracked objects (such as VR headsets worn by users, handheld controllers, etc.) are usually equipped with special markers or trackers. These markers can be reflective balls, LED light sources, or other elements that can be recognized by external sensors. The sensor calculates the precise position and posture of the target object in three-dimensional space by detecting the position and angle of these markers. 23) Data processing and integration: The location information captured by external sensors is transmitted to a central computing unit (usually a high-performance computer) through data, which integrates and processes the data from multiple sensors to determine the precise location and posture of the object. The system usually uses high-frequency data sampling and real-time processing technology to ensure low latency and high accuracy of tracking, thereby achieving a smooth user experience.

[0027] However, systems based on outside-in positioning have the following problems: 31) Occlusion problem: When the human body or other objects block the line of sight of the camera or laser tower, the optical system cannot correctly capture the action, resulting in data loss or errors. 32) High equipment cost: The camera or laser tower system usually requires multiple high-precision cameras and markers, which has high equipment and maintenance costs and requires professional installation and calibration. 33) Complex layout over large areas: For large-scale motion capture, cameras or laser towers need to deploy multiple devices, which are more complicated to install and synchronize, and have certain space restrictions.

[0028] The "inside-out" positioning method tracks the user's position and movement in physical space by integrating sensors or cameras on the user's device (such as a headset or controller). This method, in contrast to the "outside-in" positioning method, does not require the installation of external sensors in the environment, making it more portable and easier to set up. The key steps are as follows: 41) Built-in sensors: The key to the inside-out positioning system lies in the sensors or cameras built into the device (such as a VR headset or AR glasses). These sensors usually include depth cameras, wide-angle cameras, inertial measurement units, etc., which are responsible for capturing images of the environment around the device and user motion data. The depth camera can capture the environment in front of the device and generate a depth map, which helps the system understand the distance relationship between the user and the environment. The wide-angle camera is used to obtain a larger field of view, thereby covering more environmental information. 42) Environmental perception and position calculation: The built-in camera continuously scans the physical environment around the device and determines the relative position of the device based on fixed points in the environment (such as walls, furniture, ground patterns, etc.). This process is often referred to as "SLAM" (Simultaneous Localization and Mapping) technology. The system uses the SLAM algorithm to analyze environmental images, identify and track feature points in the environment, and then calculate the position and posture of the device in three-dimensional space. In addition, the Inertial Measurement Unit (IMU) sensor (accelerometer, gyroscope, etc.) also provides the device's motion data to further improve the accuracy and response speed of position calculation. 43) Real-time positioning and tracking: The inside-out system continuously updates and calculates the position and posture of the device to provide real-time user position tracking. Since all sensors are integrated into the device itself, this system can move over a large range and maintain good tracking performance without the need for external infrastructure.

[0029] However, systems based on inside-out positioning have the following problems: 51) Positioning accuracy: Although the inside-out system can provide good positioning accuracy in most scenarios, it may encounter challenges in specific environments. For example, in an environment with less texture or insufficient light, it may be difficult for the camera to identify enough feature points, thus affecting the accuracy of positioning. 52) Occlusion problem: If the device's camera is blocked (for example, the user covers the camera with his hand or the device is facing a direction without feature points), the system's positioning may be affected. In this case, the stability and continuity of positioning may be reduced. 53) Computational load: The inside-out system requires the device itself to process a large amount of image data and positioning calculations, which places high demands on the computing power of the device. Especially for VR applications that require high-performance image rendering, processing resources may become a limiting factor.

[0030] Therefore, a multi-person large-space virtual reality interactive system is needed to solve the above problems.

[0031] Embodiment 1

[0032] Figure 2 This is a structural schematic diagram of a multi-person large-space virtual reality interactive system provided in Embodiment 1 of the present invention. The system is applicable to situations where multi-person large-space virtual reality is interactive. The system can be implemented in the form of hardware and / or software.

[0033] In order to more clearly describe the multi-person large-space virtual reality interactive system provided by the present invention, some vocabulary is explained below. Artificial intelligence model refers to a computing program or system used to perform specific tasks, such as language understanding, image recognition, predictive analysis, etc. These models are trained based on data to learn the patterns or logic required to perform tasks. Virtual reality equipment is a hardware tool that simulates a virtual environment through computer technology to immerse users. It is mainly composed of a head-mounted display and a controller. Among them, the head-mounted display is the core component of the virtual reality device, which contains one or two display screens to display images of the 3D virtual world. The head-mounted display is usually also equipped with built-in sensors, such as accelerometers, gyroscopes and magnetometers, for tracking the user's head movement, thereby updating the perspective in the virtual environment in real time. At the same time, it is also a high-performance computing platform that uses powerful computing power to render the virtual world and process various sensor data. An inertial sensing module is a device for measuring the acceleration, angular velocity and magnetic field strength of an object. The inertial sensing module is mainly composed of three sensors: accelerometers, gyroscopes and sometimes magnetometers, which can provide information on the motion state and direction of an object. Inertial sensing modules are widely used in various fields that require accurate measurement of motion states, including but not limited to the following: aerospace, consumer electronics, robotics, virtual reality and augmented reality.

[0034] like Figure 2 As shown, the multi-person large space virtual reality interactive system provided in the first embodiment may specifically include: an edge computing server 10, at least one virtual reality device 20 and at least one inertial sensor module 30, wherein the inertial sensor module 30 is communicatively connected to the virtual reality device 20, and the virtual reality device 20 is communicatively connected to the edge computing server 10; wherein,

[0035] The inertial sensor module 30 is used to obtain the first posture information of each key node of the user except the head and send it to the virtual reality device 20;

[0036] The virtual reality device 20 is used to obtain the second posture information of the user's head, input the first posture information into a pre-built posture prediction model, determine the posture information of each node of the user's body, and send the second posture information of the user's head and the posture information of each node of the user's body to the edge computing server 10;

[0037] The edge computing server 10 is used to determine the motion capture result of each user based on the second posture information of each user and the posture information of each node of the whole body uploaded by each virtual reality device 20, and to render and generate a target scene picture according to the motion capture result and send it to the virtual reality device of the corresponding user, so as to present the target scene picture to the user through the virtual reality device.

[0038] In this embodiment, based on the inside-out positioning system, inertial sensor devices that capture the whole body posture are added and worn on the waist, hands and ankles, and the human body movements are predicted in combination with the artificial intelligence model. The multi-person large-space virtual reality interactive system provided in this embodiment is mainly composed of an edge computing server 10, a virtual reality device 20 and an inertial sensor module 30. In this embodiment, the number of inertial sensor modules 30 can be determined according to actual conditions, and the nodes where the user needs to wear the inertial sensor module 30 can be selected. The inertial sensor module consists of an IMU and an auxiliary circuit as well as a Bluetooth and power supply module. Preferably, each inertial sensor module 30 is worn on the user's waist, wrist and ankle, and the virtual reality device 20 is worn on the user's head.

[0039] Among them, the joint points other than the head that are used to clearly characterize human body movements are recorded as key nodes. The inertial sensor module 30 is worn at the key nodes of the human body other than the head to capture the displacement and rotation information of each key node, which is recorded as the first posture information. Its working principle is the same as the principle of the existing motion capture system based on inertial devices, which will not be repeated here. Exemplarily, the inertial sensor module 30 is worn on the human body's waist, wrists, and ankles to capture the rotational movement of the joints as the first posture information of each key node. After the first posture information of the key node is obtained through the inertial sensor module 30, the first posture information will be sent to the virtual reality device 20. Exemplarily, the inertial sensor module 30 transmits the first posture information to the virtual reality device 20 via Bluetooth transmission.

[0040] In this embodiment, the virtual reality device 20 integrates the function of obtaining head posture information and the function of determining the overall posture of the human body, which can realize the portability of the whole system. The virtual reality device 20 is integrated with a binocular camera and an inertial sensor module. Based on the perception information of the surrounding environment by the binocular camera and the inertial sensor module, based on the Simultaneous Localization And Mapping (SLAM) algorithm, the user's head displacement and rotation information and other motion information can be determined, recorded as the second posture information, and the user's three-dimensional coordinates in space are provided through the SLAM algorithm. It can be understood that compared with inertial sensors, the SLAM algorithm using a camera can provide accurate position and rotation information, and serve as an external precise position reference, thereby eliminating the accumulated error of the inertial sensor module.

[0041] In this embodiment, the virtual reality device 20 collects the information uploaded by the inertial sensor module 30 worn by the user and cleans and synchronizes it, then infers the user's current action based on the pre-trained posture prediction model and the collected information uploaded by the inertial sensor module, and uploads the user's action and SLAM data to the edge computing server 10 for further correction. In this embodiment, an artificial intelligence model is used to train a prediction model that conforms to the principles of dynamics, and the constructed model is recorded as a posture prediction model. By inputting the first posture information of each key node into the posture prediction model, the position and rotation information of other nodes of the human body can be predicted, thereby obtaining the overall posture of the human body.

[0042] Continuing with the above description, the edge computing server 10 receives the human body posture predicted by the model and the second posture information of the user's head determined by the SLAM algorithm uploaded by the virtual reality device 20. After the human body posture is inferred by the posture prediction model, the accurate user head motion information measured by the SLAM algorithm is combined, and the human body posture is corrected using the user head motion information. The posture information of each node of the human body after correction is aligned on the timestamp, and a more accurate action sequence can be obtained as a human motion capture result. The edge computing server 10 can also record and track the current positions of all online users. At the same time, the actual map and virtual map of the entire scene are provided. For each user, the actual map is updated in real time according to the user's current position and the local map generated by the SLAM algorithm on the virtual reality device. The rendering of the virtual scene is provided by the powerful graphics processing unit (GPU) performance, and the corresponding picture is rendered according to the user's current position and eye direction and transmitted to the user's virtual reality device 20 via a wireless network.

[0043] Exemplarily, the execution steps of the multi-person large-space virtual reality interactive system provided by this embodiment can be described as: the user provides the initial position in the actual map through QR code positioning; the edge computing server sends a local actual map of the current position to the user; the edge computing server renders the virtual scene visible to each user and sends it; the user takes action and uploads position and action information; the edge computing server updates and sends the map and rendered scene according to the user's position and action, and triggers events according to human body movements; the edge computing server periodically updates the map and corrects the user's position according to the information uploaded by the user; at the same time, the user's human skeleton information is periodically corrected according to the information uploaded by the user to improve the accuracy of positioning motion capture.

[0044] Compared with the mainstream outside-in method, the above technical solution does not require external cameras and laser towers, which greatly reduces equipment and installation costs. At the same time, the lack of external sensor equipment saves debugging and calibration steps. Compared with the mainstream inside-out method, multiple inertial sensor modules are added for full-body motion capture, which provides richer interaction methods compared to simple handle control. It simplifies the number of system devices, reduces the difficulty of system construction, reduces the overall cost, and combines with the posture prediction model to improve positioning accuracy and expand the availability of the system.

[0045] Figure 3 A structural diagram of another multi-person large space virtual reality interactive system provided in the first embodiment of the present invention is shown in FIG. Figure 3 As shown, as a first optional embodiment of the present invention, based on the above embodiment, this first optional embodiment can optimize the virtual reality device 20, including:

[0046] A head positioning module 21, used to determine the second posture information of the user's head and the local actual map according to the synchronous positioning and map building algorithm;

[0047] The posture reasoning module 22 is used to input the first posture information into the posture prediction model, obtain the posture information of each node other than the key node of the user, and record the first posture information of the key node and the posture information of the other nodes as the posture information of each node of the user's body;

[0048] Specifically, the virtual reality device 20 is integrated with the function of obtaining the motion information of the head and the function of determining the whole body posture. Based on this, the virtual reality device 20 includes a head positioning module 21 and a posture reasoning module 22. Among them, the head positioning module 21 uses a binocular camera and an inertial sensor module to determine the displacement and rotation information of the user's head as the second posture information of the user's head through the SLAM algorithm. At the same time, it can also generate a local map based on the SLAM algorithm to update the actual map in real time. It can be understood that the entire environment scene will be modeled through the SLAM algorithm, image data will be obtained based on the binocular camera sensing the surrounding environment, acceleration, angular velocity and magnetic field data will be obtained based on the inertial sensor module, and the second posture information of the user's head can be obtained based on the fusion processing of the above data.

[0049] It should be noted that, usually, accurate motion capture requires more than ten or even dozens of key node information of the human body. However, if multiple inertial sensing devices are used, the equipment cost is high, so a prediction model based on sparse inertial data is used in this case. Specifically, only the key node information such as the head, wrist, ankle, waist, etc. is used to predict the other joint information of the human body, and then according to the physical property constraints of the human body model, accurate human body movements are finally predicted. Specifically, the posture reasoning module 22, the posture prediction model, predicts the posture information of each other node of the human body except the key node through the first posture information of each key node, and records the first posture information of the key node and the posture information of the other nodes as the posture information of each node of the human body, thereby obtaining the overall posture of the human body.

[0050] The above technical solution specifies that the virtual reality device includes a head positioning module and a posture reasoning module, and uses the SLAM algorithm to obtain accurate movement information of the user's head. At the same time, based on the posture information of key nodes, the posture information of other nodes of the human body is predicted to obtain the overall posture of the human body.

[0051] Further, based on the first optional embodiment, the head positioning module 11 may be optimized, specifically for:

[0052] a1) Acquire first perception data and second perception data of the user's environment through a binocular camera and an inertial sensor module respectively.

[0053] Specifically, the surrounding environment is sensed by a binocular camera, and image data of the human body's environment is obtained as the first perception data. The user's head and the surrounding environment are sensed by an inertial sensor module, and the acceleration, angular velocity and magnetic field data of the user's head in each axis are obtained. By fusing the data of the accelerometer, gyroscope and magnetometer, the posture of the sensor in space (i.e., the tilt angle, rotation angle, etc. of the sensor) can be calculated and recorded as the second perception data.

[0054] b1) Based on the synchronous positioning and map building algorithm, the first perception data and the second perception data are processed to determine the second posture information of the user's head.

[0055] In this embodiment, a SLAM algorithm is used to perform positioning and map construction based on the first perception data and the second perception data, thereby obtaining movement information of the user's head, which is recorded as the second posture information of the user's head.

[0056] The above technical solution provides the system with the displacement and rotation information of the head through the SLAM algorithm. Compared with the inertial sensor, the SLAM algorithm using the camera can provide accurate position and rotation information, and serve as an external precise position reference to eliminate the accumulated error of the inertial module.

[0057] Continue to refer Figure 3 As a second optional embodiment of the present invention, based on the above embodiment, this second optional embodiment can optimize the edge computing server 10, including:

[0058] The motion capture module 11 is used to correct the posture information of each node of the whole body of each user based on the second posture information of the user's head, obtain the corrected posture information of each node of the whole body of the user, and generate the user's motion capture result based on the corrected posture information of each node of the whole body of the user;

[0059] A location tracking module 12, used to record and track the current location of all users;

[0060] The map management module 13 is used to obtain an updated actual map according to the current location of each user and the local actual map sent by the virtual reality device;

[0061] The virtual scene rendering module 14 is used to render and generate a target scene image for each user according to the user's current position, eye orientation information, motion capture results, and an updated actual map.

[0062] In this embodiment, each user wears a virtual reality device 20, and each virtual reality device 20 uploads the second posture information of the corresponding user's head and the posture information of each node of the body to the edge computing server 10. The edge computing server 10 receives the information uploaded by each virtual reality device 20, and renders and generates a virtual scene picture based on the information and feeds it back to the corresponding virtual reality device 20, so that the virtual scene picture is displayed to the user through the virtual reality device 20.

[0063] Among them, the motion capture module 11 uses the movement information of the user's head as a standard to correct the posture information of each node of the whole body, that is, the posture information of each node of the whole body is corrected by the second posture information of the user's head to obtain more accurate posture information of each node of the human body, thereby eliminating the cumulative error of the inertial sensing module.

[0064] In this embodiment, the positioning tracking module 12 records and tracks the current location of all online users. The map management module 13 provides the actual map and virtual map of the entire scene. For each user, the actual map is updated in real time according to the user's current location and the local map generated by the SLAM algorithm on the virtual reality device. The virtual scene rendering module 14 provides virtual scene rendering through powerful GPU performance, renders the corresponding picture according to the user's current location and eye direction, and transmits it to the user's virtual reality device 20 via a wireless network.

[0065] The above technical solution concretizes the execution steps of the edge computing server, uses the movement information of the human head as a standard to correct the posture information of each node of the human body, obtains more accurate posture information of each node of the human body, thereby eliminating the cumulative error of the inertial sensor module, and can obtain the user's accurate action sequence, thereby rendering a more accurate virtual scene image.

[0066] Further, based on the second optional embodiment, the motion capture module 11 may be optimized, specifically for:

[0067] a2) Based on a set data fusion algorithm, the second posture information is fused with the posture information of each node of the user's body to obtain corrected posture information of each node of the user's body.

[0068] In this embodiment, since the inertial sensing module has cumulative errors, and the second posture information of the user's head is based on external information as a reference, the posture information of each node of the whole body is corrected using the movement information of the user's head as a standard to obtain more accurate posture information of each node of the user's whole body, thereby eliminating the cumulative errors of the inertial sensing module. Specifically, the predicted human body posture information is corrected based on the second posture information of the user's head to obtain the corrected posture information of each node of the user's whole body.

[0069] b2) According to the corrected posture information of each node of the user's body, determine the user's action sequence as the motion capture result.

[0070] In this embodiment, timestamp alignment is performed based on the corrected posture information of each node of the user's body, so that a human body motion sequence can be obtained as a human body motion capture result.

[0071] The above technical solution specifies the steps of the motion capture module, and uses the posture information of the user's head to correct the user's overall posture to obtain more accurate user movements. It improves the accuracy of the inertial motion capture system, makes up for the original inertial motion capture accuracy and drift problems after long-term use, and makes the inertial system better applied in scientific research, education, and the gaming industry; at the same time, the virtual reality device integrates the external computing device of the original system, improves the integration and ease of use of the inertial motion capture system, and is more suitable for configuration and use by ordinary users, expanding the user group and market.

[0072] Further, based on the second optional embodiment, the virtual scene rendering module may be optimized, specifically for:

[0073] a3) For each user, a virtual map corresponding to the user is rendered and generated based on the user's current location, eye orientation information, and the updated actual map.

[0074] In this embodiment, the powerful GPU performance is used to render the virtual scene. For each user, according to the user's current position and eye orientation information, it is possible to determine which local map needs to be displayed to the user from the updated actual map, and then render the local map to be displayed to the user to generate a virtual map to be displayed to the user.

[0075] b3) Generate a virtual scene corresponding to the user based on the virtual map corresponding to the user and the motion capture result.

[0076] In this embodiment, for each user, the virtual map corresponding to the user is determined according to the user's current location, and then the virtual scene corresponding to the user is updated and rendered according to the motion capture result of the user. At the same time, corresponding events can also be triggered according to the user's motion capture result, for example, the triggering event can be picking up an object, opening doors and windows, etc.

[0077] The above technical solution specifies the execution steps of the virtual scene rendering module and realizes the rendering of the virtual scene.

[0078] Furthermore, based on the second optional embodiment, the positioning and tracking module can be optimized to generate over-limit warning information according to the current location of each user and a preset safety area.

[0079] In this embodiment, a corresponding safety zone is set for each user, and an alarm service is provided according to the safety zone setting to prevent the user from leaving the safety zone. After the user's current location is obtained, the user's current location can be matched with its corresponding safety zone. If it is determined that the user is close to the boundary of the safety zone, an over-limit warning message is prompted to prevent the user from leaving the safety zone.

[0080] Furthermore, based on this second optional embodiment, the edge computing server 10 can be optimized to also include a model online optimization module, which is used to correct the user position and model information through an extended Kalman filter (EKF) filtering algorithm and a BA least squares algorithm according to the current position information and human posture information sent by the user, so as to improve the positioning accuracy and human motion capture accuracy.

[0081] Figure 4 A structural diagram of another multi-person large space virtual reality interactive system provided in the first embodiment of the present invention is shown in FIG. Figure 4 As shown, as a third optional embodiment of the embodiment of the present invention, based on the above embodiment, this third optional embodiment can optimize the system further including: a cloud server 40, the cloud server 40 includes:

[0082] A model building module 41 is used to build an initial prediction model based on the human body structure and human physical properties;

[0083] The model training module 42 is used to train the initial prediction model based on the posture data sample set to obtain a general prediction model.

[0084] The posture data sample set includes at least one set of posture data sample pairs, posture sample data of key nodes of the posture data sample pairs and posture sample data of other nodes.

[0085] In this embodiment, the system also includes a cloud server 40, which can be considered as a model training platform. The cloud server 40 includes a model building module 41 and a model training module 42. Usually, accurate motion capture requires more than ten or even dozens of key human node information. If multiple inertial sensing modules are used, the equipment cost is high. Therefore, in this embodiment, a prediction model based on sparse inertial data is used. Specifically, only the node information of key nodes such as the head, wrist, ankle, waist, etc. is used to predict other joint information of the human body, and then according to the physical property constraints of the model, accurate human body movements are finally predicted.

[0086] In this embodiment, the model construction module 41 is used to construct a human skeleton model including a hierarchical joint chain of human bones and joints according to the human body structure, construct a human prediction model based on the human skeleton model, and construct a joint torque control model according to the physical properties of the human body, and construct an initial prediction model based on the above models.

[0087] The model training module 42 is used to input the posture sample data of the key nodes into the initial prediction model to obtain the prediction output results of other nodes, substitute the prediction output results of other nodes and the posture sample data of other nodes into the preset loss function to obtain the loss value, adjust the parameters of the initial prediction model based on the loss value, and optimize the preset loss function by adjusting the hyperparameters to obtain a trained prediction model, which is recorded as a general prediction model.

[0088] Further, based on the third optional embodiment, the model building module 41 may be optimized, specifically for:

[0089] a4) Based on the human body structure, a human skeleton model including a hierarchical joint chain of human bones and joints is constructed.

[0090] In this embodiment, the human skeleton model organizes bones and joints in a tree-like hierarchical relationship according to the human body structure, and there is a parent-child hierarchical relationship between bones. Except for the root bone, each bone has a unique parent bone. For example, Figure 5 A schematic diagram of a human skeleton model in a multi-person large space virtual reality interactive system provided in the first embodiment of the present invention, such as Figure 5 As shown in the figure, 0, 1, 2, 3, etc. represent different nodes. The movement of a bone can be regarded as its rotational movement relative to the parent bone. By simplifying the human body into a hierarchical joint chain composed of several joints and bones, the movement of the human body can be described by the translation and rotation of the root bone and the rotational movement of each joint.

[0091] b4) Based on the human skeleton model, a human body prediction model is constructed using the long short-term memory network as the neural network structure.

[0092] In this embodiment, the human body prediction model is based on the skeleton model and predicts the positions of all skeleton points through some key point information. First, the initial state of the skeleton model is preset, and then the rotation, speed, and acceleration information of the head, wrist, ankle, and waist skeleton points are collected, combined with the chain structure of the skeleton model and the long short-term memory network (LSTM) timing prediction structure in the neural network to predict the motion information of each skeleton node.

[0093] c4) Construct a joint torque control model based on the physical properties of the human body.

[0094] In this embodiment, the joint torque control model includes the physical properties of the human body, and after being controlled by the joint torque and having physical effects with the environment, human body movements that conform to real physical laws can be generated.

[0095] d4) Combining the human body prediction model and the joint torque control model as the initial prediction model.

[0096] In this embodiment, the human body prediction model is connected to the joint torque control model, and the joint torque control model is used as a physical constraint of the output result of the human body prediction model, so as to obtain an initial prediction model. Figure 6 A schematic diagram of the structure of an initial prediction model in a multi-person large space virtual reality interactive system provided in the first embodiment of the present invention, such as Figure 6 As shown in the figure, the human kinematics model is based on the skeleton model and predicts the posture of all skeleton points through the information of some key points. First, the initial state of the skeleton model is preset, and then the rotation, speed, and acceleration information of the head, wrist, ankle, and waist skeleton points are collected as input data. The chain structure of the skeleton model and the LSTM timing prediction structure in the neural network are combined. The recurrent neural network adopts the LSTM structure and is connected through a fully connected layer. The output of the human prediction model predicts the motion information of each skeleton node as the output data through the constraints of the joint torque control model. The joint torque control model includes a differential controller and a tracking optimizer.

[0097] The above technical solution concretizes the functions of the model construction module, specifically the human body model includes a human skeleton model, a human body prediction model and a joint torque control model, and realizes the construction of an initial prediction model based on sparse inertial data.

[0098] Further, based on the third optional embodiment, the model training module 42 can be optimized, specifically used to: train the initial prediction model based on the posture data sample set, adjust the hyperparameters, optimize the preset loss function, and obtain a general prediction model.

[0099] Among them, the preset loss function can be set according to actual conditions. In this embodiment, the posture data sample set includes at least one group of posture data sample pairs, posture sample data of key nodes of the posture data sample pair and posture sample data of other nodes. Specifically, the posture sample data of the key nodes are input into the initial prediction model to obtain the prediction output results of other nodes, and the prediction output results of other nodes and the posture sample data of other nodes are substituted into the preset loss function to obtain the loss value, and the parameters of the initial prediction model are adjusted based on the loss value. By adjusting the hyperparameters, the preset loss function is optimized to obtain a trained prediction model, which is recorded as a general prediction model. Since the cloud server GPU has powerful performance and is suitable for large-scale data processing and training, a public data set is used to train the basic model. At the same time, by adjusting the hyperparameters, such as the learning rate, the training batch size, etc., the training is optimized, and the loss function is optimized to finally obtain a basic general prediction model.

[0100] The above technical solution specifies the training steps of the general prediction model.

[0101] Continue to refer Figure 4 Further, based on the second optional embodiment, the edge computing server 10 can be optimized to also include a model optimization module 15, which is specifically used for:

[0102] a5) Obtain a general prediction model.

[0103] The model optimization module 15 in the edge computing server downloads the basic model from the cloud server, and uses the local user data to train the basic general prediction model on the edge computing server through transfer learning to obtain a fine-tuned model. It is understandable that since motion capture has high precision requirements and each person's skeletal model is quite different, it is very difficult to train a general model with good generalization ability. Therefore, for users of different body shapes, the user data is collected, and the general model is retrained to obtain accurate models for different users. Since only the general model is fine-tuned, the scale of training data is small, and training can be performed on the edge computing server, reducing data transmission delays and improving real-time performance. This step is used to obtain a basic posture prediction model.

[0104] b5) Adjust the parameters of the human skeleton model according to the user's body shape.

[0105] It is understandable that the parameters of the human skeleton model corresponding to different user's body shapes are also different. In this step, the parameters of the human skeleton model are adjusted to correspond to the user's body shape. The skeleton model data is adjusted according to the user's body shape, and the user's motion data over a period of time is collected.

[0106] c5) Based on the user's posture data sample set and the adjusted parameters of the human skeleton model, the general prediction model is trained to obtain an adjusted posture prediction model for the user.

[0107] In this embodiment, based on the posture data sample set of the user and the parameters of the adjusted human skeleton model, the basic universal prediction model is trained to obtain the trained universal prediction model as the adjusted posture prediction model for the user.

[0108] It should be noted that when users use the device, a large amount of data will be generated, including the data collected from the inertial sensor module, the data generated by the local model prediction, and the real data of the head posture measured by the virtual device. These data will be regularly transmitted to the cloud platform as Zeng Guang Data to retrain the existing basic model and further improve the performance of the basic model.

[0109] The above technical solution adds a model optimization function, realizes the training of corresponding posture prediction models for different user body types, and provides a basis for subsequent accurate prediction of human body posture.

[0110] Embodiment 2

[0111] Figure 7 This is a flow chart of a multi-person large space virtual reality interaction method provided in the second embodiment of the present invention. The method is applicable to the case where multiple people interact with each other in a large space virtual reality, and the method can be executed by a multi-person large space virtual reality interaction system. The system can be implemented in the form of hardware and / or software. Figure 7 As shown, the multi-person large space virtual reality interaction method provided in the second embodiment may specifically include:

[0112] S201. Obtain first posture information of each key node of the user except the head through the inertial sensing module of the multi-person large-space virtual reality interactive system and send it to the virtual reality device.

[0113] In this embodiment, based on the inside-out positioning system, inertial sensor devices that capture the whole body posture are added and worn on the waist, hands and ankles. The artificial intelligence model is combined to predict human body movements, improve the positioning and tracking algorithm, and improve the accuracy of the inside-out positioning method. The multi-person large-space virtual reality interactive system provided in this embodiment is mainly composed of an edge computing server, a virtual reality device and an inertial sensor module. In this embodiment, the number of inertial sensor modules can be determined according to actual conditions, and the nodes on the human body that need to wear inertial sensor modules can be selected. The inertial sensor module consists of an IMU and an auxiliary circuit as well as a Bluetooth and power supply module. Preferably, each inertial sensor module is worn on the user's waist, wrist and ankle, and the virtual reality device is worn on the user's head.

[0114] Among them, the joint points other than the head that are used to obviously characterize human body movements are recorded as key nodes. The inertial sensor module is worn at the key nodes of the human body other than the head to capture the displacement and rotation information of each key node, which is recorded as the first posture information. Its working principle is the same as the principle of the existing motion capture system based on inertial devices, which will not be repeated here. Exemplarily, the inertial sensor module is worn on the human body's waist, wrists, and ankles to capture the rotational movement of the joints as the first posture information of each key node. After the first posture information of the key node is obtained through the inertial sensor module, the first posture information will be sent to the virtual reality device. Exemplarily, the inertial sensor module transmits the first posture information to the virtual reality device via Bluetooth transmission.

[0115] S202. Obtain second posture information of the user's head through a virtual reality device of a multi-person large-space virtual reality interactive system, input the first posture information into a pre-built posture prediction model, determine posture information of each node of the user's body, and send the second posture information of the user's head and the posture information of each node of the user's body to an edge computing server.

[0116] In this embodiment, the virtual reality device integrates the functions of acquiring head posture information and determining the overall posture of the human body, which can realize the portability of the entire system. The virtual reality device is integrated with a binocular camera and an inertial sensor module. Based on the perception information of the surrounding environment by the binocular camera and the inertial sensor module, based on the SLAM algorithm, the user's head displacement and rotation information and other motion information can be determined, recorded as the second posture information, and the user's three-dimensional coordinates in space are provided through the SLAM algorithm. It can be understood that compared with inertial sensors, the SLAM algorithm using a camera can provide accurate position and rotation information, and serve as an external precise position reference, thereby eliminating the accumulated errors of the inertial sensor module.

[0117] In this embodiment, the virtual reality device collects the information uploaded by the inertial sensor module worn by the user and cleans and synchronizes it, then infers the user's current action based on the pre-trained posture prediction model and the collected information uploaded by the inertial sensor module, and uploads the user's action and SLAM data to the edge computing server for further correction. In this embodiment, an artificial intelligence model is used to train a prediction model that conforms to the principles of dynamics, and the constructed model is recorded as a posture prediction model. By inputting the first posture information of each key node into the posture prediction model, the position and rotation information of other nodes of the human body can be predicted, thereby obtaining the overall posture of the human body.

[0118] S203. Determine the motion capture result of each user through the edge computing server of the multi-person large-space virtual reality interactive system according to the second posture information of each user and the posture information of each node of the whole body uploaded by each virtual reality device, render and generate the target scene picture according to the motion capture result and send it to the virtual reality device of the corresponding user, so as to present the target scene picture to the user through the virtual reality device.

[0119] Continuing with the above description, the edge computing server receives the human body posture predicted by the model and the second posture information of the user's head determined by the SLAM algorithm uploaded by the virtual reality device. After the human body posture is inferred by the posture prediction model, the user's head motion information is used to correct the human body posture in combination with the accurate user head motion information measured by the SLAM algorithm. The posture information of each node of the human body after correction is aligned on the timestamp, and a more accurate action sequence can be obtained as the user's motion capture result. The edge computing server 10 can also record and track the current positions of all online users. At the same time, the actual map and virtual map of the entire scene are provided. For each user, the actual map is updated in real time according to the user's current position and the local map generated by the SLAM algorithm on the virtual reality device. The rendering of the virtual scene is provided by the powerful GPU performance, and the corresponding picture is rendered according to the user's current position and eye direction and transmitted to the user's virtual reality device via a wireless network.

[0120] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps described in the present invention can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solution of the present invention can be achieved, and this document does not limit this.

[0121] The above specific implementations do not constitute a limitation on the protection scope of the present invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A multi-person large space virtual reality interactive system, characterized in that: include: An edge computing server, at least one virtual reality device and at least one inertial sensing module, wherein the inertial sensing module is communicatively connected to the virtual reality device, and the virtual reality device is communicatively connected to the edge computing server; wherein, The inertial sensing module is used to obtain the first posture information of each key node of the user except the head and send it to the virtual reality device; The virtual reality device is used to obtain second posture information of the user's head, input the first posture information into a pre-built posture prediction model, determine the posture information of each node of the user's body, and send the second posture information of the user's head and the posture information of each node of the user's body to the edge computing server; The edge computing server is used to determine the motion capture results of each user according to the second posture information of each user and the posture information of each node of the whole body uploaded by each virtual reality device, and to render and generate a target scene picture according to the motion capture result and send it to the virtual reality device corresponding to the user, so as to present the target scene picture to the user through the virtual reality device.

2. The system according to claim 1, characterized in that The virtual reality device comprises: A head positioning module, used to determine the second posture information of the user's head and a local actual map according to a synchronous positioning and map building algorithm; The posture reasoning module is used to input the first posture information into the posture prediction model, obtain the posture information of each node other than the key node of the user, and record the first posture information of the key node and the posture information of other nodes as the posture information of each node of the user's whole body.

3. The system according to claim 2, characterized in that The head positioning module is specifically used for: Acquire first perception data and second perception data of the user's environment through a binocular camera and an inertial sensor module respectively; Based on the synchronous positioning and mapping algorithm, the first perception data and the second perception data are processed to determine the second posture information of the user's head.

4. The system according to claim 1, characterized in that The edge computing server comprises: A motion capture module, for each user, correcting the posture information of each node of the whole body based on the second posture information of the user's head, obtaining the corrected posture information of each node of the whole body of the user, and generating a motion capture result of the user based on the corrected posture information of each node of the whole body of the user; Location tracking module, used to record and track the current location of all users; A map management module, used to obtain an updated actual map according to the current location of each user and the local actual map sent by the virtual reality device; The virtual scene rendering module is used to render and generate the target scene image for each user according to the user's current position, eye direction information, the motion capture result, and the updated actual map.

5. The system according to claim 4, characterized in that The motion capture module is specifically used for: Based on a set data fusion algorithm, the second posture information is fused with the posture information of each node of the user's body to obtain corrected posture information of each node of the user's body; According to the corrected posture information of each node of the user's body, the user's action sequence is determined as the action capture result.

6. The system according to claim 4, characterized in that The virtual scene rendering module is specifically used for: For each user, based on the user's current location, eye orientation information, and the updated actual map, a virtual map corresponding to the user is rendered and generated; A virtual scene corresponding to the user is generated according to the virtual map corresponding to the user and the motion capture result.

7. The system according to claim 4, characterized in that The positioning tracking module is also used for: Generate over-limit warning information based on the current location of each user and the preset safety area.

8. The system according to claim 1, characterized in that Also included is a cloud server, the cloud server comprising: A model building module is used to build an initial prediction model based on the human body structure and physical properties; A model training module is used to train the initial prediction model based on a posture data sample set to obtain a general prediction model, wherein the posture data sample set includes at least one group of posture data sample pairs, and the posture data sample pairs include posture sample data of key nodes of the human body and posture sample data of other nodes of the human body.

9. The system according to claim 8, characterized in that The model building module is specifically used for: According to the human body structure, a human skeleton model including a hierarchical joint chain of human bones and joints is constructed; Based on the human skeleton model, a human prediction model is constructed using a long short-term memory network as a neural network structure; Constructing a joint torque control model according to the physical properties of the human body; The human body prediction model and the joint torque control model are combined as the initial prediction model.

10. The system according to claim 8, characterized in that The model training module is specifically used for: The initial prediction model is trained based on the posture data sample set, and hyperparameters are adjusted to optimize a preset loss function to obtain the general prediction model.

11. The system according to claim 8, characterized in that The edge computing server also includes a model optimization module, which is specifically used to: Acquire the general prediction model; Adjust the parameters of the human skeleton model according to the user's body shape; Based on the user's posture data sample set and the adjusted parameters of the human skeleton model, the general prediction model is trained to obtain an adjusted posture prediction model for the user.

12. The system according to claim 1, characterized in that The inertial sensor modules are worn on the waist, wrist and ankle of the user, and the virtual reality device is worn on the head of the user.

13. A multi-person large space virtual reality interaction method, characterized in that: The method is performed by the multi-person large-space virtual reality interactive system according to any one of claims 1 to 12, and comprises: The inertial sensing module of the multi-person large-space virtual reality interactive system is used to obtain the first posture information of each key node of the user except the head and send it to the virtual reality device; Obtaining second posture information of the user's head through the virtual reality device of the multi-person large-space virtual reality interactive system, inputting the first posture information into a pre-built posture prediction model, determining posture information of each node of the user's body, and sending the second posture information of the user's head and the posture information of each node of the user's body to the edge computing server; Through the edge computing server of the multi-person large-space virtual reality interactive system, the motion capture results of each user are determined according to the second posture information of each user and the posture information of each node of the whole body uploaded by each virtual reality device, and the target scene picture is rendered and generated according to the motion capture result and sent to the virtual reality device corresponding to the user, so that the target scene picture is presented to the user through the virtual reality device.

Images