Data processing method and device of virtual reality system, server and medium

Abstract

Embodiments of the present application provide a data processing method and device of a virtual reality system, a server and a medium. When packet loss or delay occurs in sensor data of a mobile VR device or a handheld device, the server predicts sensor data of the device at a current collection time according to sensor data of the device at the previous n collection times, and then performs image rendering processing according to the predicted sensor data of the device at the current collection time. Using the predicted sensor data to perform image rendering and other operations enables the server to render continuous texture images, including continuity of changes in viewport position and continuity of position updates of the handheld device in the images, effectively reducing the impact of network disturbances and effectively improving the immersive experience of users.

Description

Technical Field

[0001] This application relates to the field of server technology, and in particular to a data processing method, apparatus, server, and medium for a virtual reality system. Background Technology

[0002] Virtual Reality (VR) technology uses computers to simulate a three-dimensional virtual environment that can realistically simulate visual, auditory, and tactile sensory information. When users interact with VR devices, they can experience a sense of immersion.

[0003] VR systems typically include a head-mounted device, a handheld device, and a server. Taking a haptic feedback scenario as an example, the data processing flow of existing technologies is as follows: The handheld device sends sensor data to the head-mounted device. The head-mounted device processes the sensor data from both the head-mounted device and the handheld device synchronously, packages them together, and sends them to the server for VR image rendering. The server sends the rendered image and haptic feedback results back to the head-mounted device. The head-mounted device performs asynchronous time warp (ATW) or asynchronous space warp (ASW) frame interpolation on the received image based on latency and packet loss, and forwards the received haptic feedback results to the handheld device for haptic feedback. After seeing the image displayed on the head-mounted device, the human eye makes a physiological response, namely, a change in the position of the handheld device and the head-mounted device.

[0004] However, existing solutions are affected by network latency and packet loss, resulting in problems such as discontinuous screen display and handheld device vibration feedback. In particular, with the development of cloud rendering technology, network latency and packet loss will become more severe, and their impact on user experience cannot be ignored. Summary of the Invention

[0005] This application provides a data processing method, apparatus, server, and medium for a virtual reality system, enabling the server to render continuous texture images, effectively reducing the impact of network disturbances, and effectively enhancing the user's immersive experience.

[0006] In a first aspect, embodiments of this application provide a data processing method for a virtual reality system, applied to a server, the method comprising:

[0007] Acquire sensor data from a device, the device including a mobile virtual reality (VR) device, or the mobile VR device and a handheld device for interacting with the mobile VR device;

[0008] When the sensor data is lost or delayed, the sensor data at the current acquisition time is predicted based on the sensor data of the device in the previous n acquisition times, where n is an integer greater than or equal to 1.

[0009] Image rendering processing is performed based on the sensor data of the device obtained from the prediction at the current acquisition time;

[0010] The rendered image is sent to the mobile VR device.

[0011] In some embodiments, the sensor data includes the device’s linear velocity, linear acceleration, angular velocity, angular acceleration, position information, and orientation information.

[0012] In some embodiments, predicting the sensor data of the device at the current acquisition time based on the sensor data of the device at the previous n acquisition times includes:

[0013] Based on the linear velocity, linear acceleration, angular velocity, and angular acceleration of the device at the current acquisition time, determine the linear velocity, linear acceleration, angular velocity, and angular acceleration of the device at the previous n acquisition times;

[0014] Based on the linear velocity and linear acceleration of the device at the current acquisition time, calculate the displacement of the device from the previous acquisition time to the current acquisition time;

[0015] The position information of the device at the current acquisition time is obtained based on the device's position information at the previous acquisition time and the displacement amount;

[0016] Based on the angular velocity of the device at the current acquisition time, calculate the rotation angle of the device from the previous acquisition time to the current acquisition time;

[0017] Based on the orientation information of the device at the previous acquisition time and the rotation angle, the orientation information of the device at the current acquisition time is obtained.

[0018] In some embodiments, determining the linear velocity, linear acceleration, angular velocity, and angular acceleration of the device at the current acquisition moment based on the linear velocity, linear acceleration, angular velocity, and angular acceleration of the device at the previous n acquisition moments includes:

[0019] The linear velocity, linear acceleration, angular velocity, and angular acceleration of the device at the previous acquisition time are determined as the linear velocity, linear acceleration, angular velocity, and angular acceleration of the device at the current acquisition time.

[0020] In some embodiments, determining the linear velocity, linear acceleration, angular velocity, and angular acceleration of the device at the current acquisition moment based on the linear velocity, linear acceleration, angular velocity, and angular acceleration of the device at the previous n acquisition moments includes:

[0021] The average or weighted average of the linear velocity of the device over the previous n acquisition times is determined as the linear velocity of the device at the current acquisition time;

[0022] The average or weighted average of the linear acceleration of the device over the previous n acquisition times is determined as the linear acceleration of the device at the current acquisition time.

[0023] The average or weighted average of the angular velocities of the device over the previous n acquisition times is determined as the angular velocity of the device at the current acquisition time;

[0024] The average or weighted average of the angular acceleration of the device over the previous n acquisition times is determined as the angular acceleration of the device at the current acquisition time.

[0025] In some embodiments, determining the linear velocity, linear acceleration, angular velocity, and angular acceleration of the device at the current acquisition moment based on the linear velocity, linear acceleration, angular velocity, and angular acceleration of the device at the previous n acquisition moments includes:

[0026] The linear velocity, linear acceleration, angular velocity, and angular acceleration of the device at the previous acquisition time are multiplied by a preset adjustment factor to obtain the linear velocity, linear acceleration, angular velocity, and angular acceleration of the device at the current acquisition time.

[0027] In some embodiments, the orientation information is a quaternion, and the rotation angle is an Euler angle;

[0028] The step of obtaining the orientation information of the device at the current acquisition time based on the orientation information of the device at the previous acquisition time and the rotation angle includes:

[0029] Convert the rotation angle into a quaternion;

[0030] The orientation information of the device at the current acquisition time is obtained based on the orientation information of the device at the previous acquisition time and the converted quaternion.

[0031] In some embodiments, acquiring the sensor data of the device includes:

[0032] Receive and package the sensor data of the mobile VR device and the sensor data of the handheld device, which are synchronously transmitted by the mobile VR device.

[0033] In some embodiments, the method further includes:

[0034] Based on the predicted sensor data of the device at the current acquisition time, the tactile feedback result of the handheld device is determined;

[0035] The haptic feedback results are sent to the mobile VR device.

[0036] In some embodiments, the server is a cloud server, and the mobile VR device is a head-mounted device.

[0037] On the other hand, embodiments of this application provide a data processing apparatus for a virtual reality system, the apparatus comprising:

[0038] An acquisition module is used to acquire sensor data of a device, the device including a mobile virtual reality (VR) device, or the mobile VR device and a handheld device for interacting with the mobile VR device;

[0039] The prediction module is used to predict the sensor data of the device at the current acquisition time based on the sensor data of the device in the previous n acquisition times when the sensor data is lost or delayed, where n is an integer greater than or equal to 1;

[0040] The rendering module is used to perform image rendering processing based on the predicted sensor data of the device at the current acquisition time.

[0041] The sending module is used to send the rendered image to the mobile VR device.

[0042] On the other hand, embodiments of this application provide a server, the server comprising: a processor and a memory, the memory for storing computer programs, and the processor for calling and running the computer programs stored in the memory to perform the method as described in any of the preceding claims.

[0043] On the other hand, embodiments of this application provide a computer-readable storage medium for storing a computer program that causes a computer to perform the methods described in any of the foregoing descriptions.

[0044] On the other hand, embodiments of this application provide a computer program product, including a computer program that, when executed by a processor, implements the method described in any of the above-mentioned embodiments.

[0045] The data processing method, apparatus, server, and medium for a virtual reality system provided in this application, when sensor data from a mobile VR device or handheld device experiences packet loss or delay, allow the server to predict the sensor data of the device at the current acquisition time based on the sensor data from the previous n acquisition times. Then, image rendering is performed based on the predicted sensor data at the current acquisition time. Using the predicted sensor data for image rendering and other operations enables the server to render continuous texture images, including continuous changes in viewport position and continuous updates to the handheld device's position within the image. This effectively reduces the impact of network disturbances and significantly enhances the user's immersive experience. Attached Figure Description

[0046] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0047] Figure 1 This is a schematic diagram of the structure of a virtual reality system applicable to the embodiments of this application;

[0048] Figure 2 A diagram showing the data flow relationship between virtual reality and haptic feedback;

[0049] Figure 3 This is a schematic diagram of the data processing flow in an existing virtual reality system;

[0050] Figure 4 This is a schematic diagram of the data processing flow in a virtual reality system provided in an embodiment of this application;

[0051] Figure 5 This is the flow chart of the data processing method for the virtual reality system provided in Embodiment 1 of this application;

[0052] Figure 6 A signaling flowchart of the data processing method for the virtual reality system provided in Embodiment 2 of this application;

[0053] Figure 7 This is a schematic diagram of the structure of the data processing device for the virtual reality system provided in Embodiment 3 of this application;

[0054] Figure 8 This is a schematic diagram of a server provided in Embodiment 4 of this application. Detailed Implementation

[0055] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0056] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or server that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or devices.

[0057] To facilitate understanding of the embodiments of this application, before describing the various embodiments, some concepts involved in all embodiments of this application will be appropriately explained as follows:

[0058] 1) Virtual Reality (VR) is a technology for creating and experiencing virtual worlds. It defines and generates a virtual environment, which is a multi-source information (virtual reality mentioned in this article includes at least visual perception, and may also include auditory perception, tactile perception, motion perception, and even taste perception, olfactory perception, etc.) that realizes the fusion of interactive three-dimensional dynamic visual scenes and simulation of physical behavior in the virtual environment, immersing users in the simulated virtual reality environment, and enabling applications in various virtual environments such as maps, games, videos, education, medical care, simulation, collaborative training, sales, assisted manufacturing, maintenance and repair.

[0059] 2) Virtual reality devices (VR devices) are terminals that realize virtual reality effects. They can usually be provided in the form of glasses, head-mounted displays (HMDs), or contact lenses to realize visual perception and other forms of perception. Of course, the form of virtual reality devices is not limited to these, and they can be further miniaturized or enlarged according to actual needs.

[0060] Optionally, the virtual reality devices described in the embodiments of this application may include, but are not limited to, the following types:

[0061] 2.1) Mobile virtual reality devices support setting up mobile terminals (such as smartphones) in various ways (such as head-mounted displays with dedicated card slots). Through wired or wireless connections with the mobile terminal, the mobile terminal performs relevant calculations for virtual reality functions and outputs data to the mobile virtual reality device, such as watching virtual reality videos through an app on the mobile terminal.

[0062] 2.2) All-in-one virtual reality devices have processors for performing virtual functions, thus having independent virtual reality input and output functions. They do not need to be connected to a PC or mobile terminal, and have a high degree of freedom of use.

[0063] 3) Mixed Reality (MR): This refers to the creation of new environments and visualizations that combine the real and virtual worlds, where physical entities and digital objects coexist and interact in real time to simulate real objects. It blends real, augmented, augmented virtual, and virtual reality technologies. MR is a composite of virtual reality (VR) and augmented reality (AR), an extension of VR technology. By presenting virtual scenes within real-world settings, it enhances the realism of the user experience. The MR field involves computer vision, a science that studies how to enable machines to "see." More specifically, it refers to machine vision, where cameras and computers replace human eyes in tasks such as target recognition, tracking, and measurement, followed by image processing to create images more suitable for human observation or transmission to instruments.

[0064] In other words, MR (Mixed Reality) is a simulated scene that integrates computer-created sensory input (e.g., virtual objects) with sensory input or its representation from a physical setting. In some MR scenes, the computer-created sensory input can adapt to changes in sensory input from the physical setting. Additionally, some electronic systems used to present MR scenes can monitor orientation and / or position relative to the physical setting, enabling virtual objects to interact with real objects (i.e., physical elements or their representations from the physical setting). For example, the system can monitor motion so that virtual plants appear stationary relative to physical buildings.

[0065] Figure 1 This is a schematic diagram of the structure of a virtual reality system applicable to the embodiments of this application, such as... Figure 1 As shown, the virtual reality system includes: a mobile VR device 11, a handheld device 12, and a server 13. The handheld device 12 is used to assist the user in interacting with the mobile VR device 11.

[0066] The mobile VR device 11 can be a head-mounted device (HMD) or glasses, and can move with the user. The mobile VR device 11 can use VR, AR, or MR technologies to provide virtual scenes to the user.

[0067] In this embodiment, the mobile VR device 11 is mainly used for measuring its own position and orientation information, forwarding data from the handheld device 12, and displaying images. Image rendering and feedback data processing are implemented on the server side.

[0068] Accordingly, the mobile VR device 11 includes at least one sensor, allowing the user to acquire the position and orientation information of the mobile VR device 11. This at least one sensor includes, but is not limited to, an accelerometer, a gyroscope, an inertial measurement unit (IMU) compass sensor, or a geomagnetic sensor. The sensor acquires data at a preset acquisition frequency, which only needs to meet the requirements, for example, an acquisition frequency of 400Hz or higher.

[0069] The handheld device 12 can be a glove, game controller, handheld controller, etc., used to enable users to interact with the virtual environment, such as controlling the movement of virtual objects in the virtual environment and providing tactile (i.e. force) feedback.

[0070] The handheld device 12 also has at least one sensor for measuring the position and orientation information of the handheld device 12. The at least one sensor includes, but is not limited to, an accelerometer, a gyroscope, an IMU, a compass sensor or a geomagnetic sensor. The sensor collects data at a preset acquisition frequency and sends it to the mobile VR device 11.

[0071] Optionally, the handheld device 12 can also be used to trigger feedback, applying force to the user through vibration, squeezing, or other means.

[0072] Virtual reality systems achieve haptic interactivity by combining position tracking, handheld controllers, and haptic feedback technology. Figure 2 A diagram showing the data flow relationship between virtual reality and haptic feedback, such as Figure 2 As shown, the virtual reality engine outputs vector force parameters to the force feedback device (such as a game controller, gloves, etc.), the force feedback device responds with actions such as vibration, and then returns updated position information and other parameters to the virtual reality engine, which can be located on a server.

[0073] Server 13 includes a virtual reality engine for image rendering and control logic processing. Server 13 can be a cloud server or a local server; a local server can be a local PC. The cloud server is used for cloud rendering, which is the future trend of rendering technology and is gradually becoming available to ordinary users; for example, cloud rendering is used in cloud gaming. The local server is used for rendering locally; compared to cloud rendering, local rendering has lower network latency and a lower packet loss rate. The control logic processing handles various feedback data from the VR device, including but not limited to haptic feedback data.

[0074] Specifically, server 13 receives sensor data sent by mobile VR device 11 and handheld device 12, and applies it to the virtual environment. For example, it continuously updates the images displayed on the screen of mobile VR device 11 according to the user's gaze, enabling the user to interact with the virtual environment and scene. For example, it continuously updates virtual reality content based on the detected direction of the user's head rotation. It controls sensors and various input / output devices, including obtaining user data (such as actions and voice) and outputting sensory data, such as images, vibrations, temperature, and sounds, which affect the user, the virtual environment, and the real world. It constructs a three-dimensional model of the virtual environment, which may also include various feedback mechanisms such as sound and touch in the three-dimensional model.

[0075] Figure 3 This is a schematic diagram of the data processing flow in an existing virtual reality system. The controller is mainly used for touch operation and haptic feedback, the head-mounted device is mainly used for controller data forwarding and image display, and the cloud server or local PC is used for image data rendering and haptic vibration information transmission.

[0076] like Figure 3 As shown, its data processing flow mainly includes the following aspects:

[0077] (1) The handle sends the sensor data of the handle to the head-mounted device.

[0078] (2) The head-mounted device processes the sensor data from the head-mounted device and the controller in a synchronized manner, and then packages them together and sends them to the cloud server or local PC for VR image rendering.

[0079] (3) The cloud server or local PC sends the rendered image and haptic feedback results to the head-mounted device.

[0080] (4) The head-mounted device performs ATW or ASW frame interpolation on the received images based on the delay and packet loss.

[0081] (5) The head-mounted device forwards the received tactile feedback results to the handle for specific tactile feedback.

[0082] (6) After seeing the image displayed on the head-mounted device, the human eye makes a response based on human function, namely, the change in the position of the handle and the head-mounted device.

[0083] As can be seen from the existing solutions, due to network latency and packet loss, necessary optimizations and supplements have been made to the image data on the head-mounted device side. Specifically, ATW and ASW are used to predict the motion changes of the image and perform timely frame interpolation and frame supplementation to avoid situations such as "black borders" (caused by latency) and "jitter" (caused by packet loss) in the head-mounted device's screen that affect the user's immersive virtual reality experience.

[0084] However, existing solutions do not address the latency and packet loss of sensor data from head-mounted devices and controllers. Virtual reality engines render images based on sensor data from head-mounted devices and controllers. If there is packet loss or latency in the sensor data from head-mounted devices and controllers sent to the cloud server or local PC, the sensor data sent to the virtual reality engine will be discontinuous. The virtual reality engine will be unable to render the current frame, which will lead to the following problems: (1) The virtual reality engine cannot render a continuous viewport image in real time based on the sensor data from head-mounted devices and controllers. For example, the head-mounted image will stutter (caused by discontinuous sensor data from the head-mounted device); (2) The tactile sensations (e.g., vibration) that should occur on the controllers will not occur or there will be discontinuities or interruptions in the tactile sensations or controller position updates (caused by discontinuous controller sensor data); (3) It will affect the user's virtual reality immersion.

[0085] Furthermore, with the development of cloud rendering, network latency and packet loss have become more severe, and their impact on user experience cannot be ignored. Moreover, for virtual reality game scenarios, in addition to visual and auditory senses, major manufacturers are increasingly investing more resources in the "haptic" field to enhance the immersion and realism of virtual reality from another dimension. Therefore, data optimization for touch-enabled devices is indispensable.

[0086] To address the problems of the prior art, embodiments of this application provide a data processing method for a virtual reality system. Figure 4 This is a schematic diagram of the data processing flow in the virtual reality system provided in the embodiments of this application, with reference to... Figure 3 and Figure 4Before rendering images on the cloud server or local PC, the solution in this application determines whether there is delay or packet loss in the sensor data of the head-mounted device and controllers. If there is no delay or packet loss, the existing process is followed. If there is delay or packet loss, the sensor data of the head-mounted device and controllers is predicted based on the sensor data collected at the previous one or more acquisition times. This ensures that the sensor data sent to the virtual reality engine remains continuous, thereby enabling the rendering of continuous images, including continuous changes in viewport position and continuous updates to the controller position in the image.

[0087] Using the predicted sensor data from the head-mounted device and controller at the current acquisition moment, image rendering and haptic feedback logic processing are performed to obtain updated haptic feedback results. This reduces the impact of sensor data discontinuity caused by network disturbances, resulting in smoother rendered images and ensuring no loss of vibration feedback from the controller.

[0088] The data processing method of the virtual reality system provided in the embodiments of this application will be described in detail below with reference to the accompanying drawings.

[0089] Figure 5 This is a flowchart of the data processing method for a virtual reality system provided in Embodiment 1 of this application. It can be applied to... Figure 1 The virtual reality system shown in this embodiment describes the data processing method from the server's perspective. (Refer to...) Figure 1 , Figure 2 , Figure 4 and Figure 5 The data processing method of the virtual reality system in this embodiment includes the following steps.

[0090] S101. Acquire sensor data from a device, which may include a mobile VR device, or a mobile VR device and a handheld device for interacting with the mobile VR device.

[0091] Users typically interact with mobile VR devices through one or more of the following methods: handheld devices (such as gamepads or hand controllers), gesture operation, voice control, or gaze control. This application embodiment does not impose specific limitations on these methods.

[0092] It is understandable that when a mobile VR device does not use handheld device interaction, for example, when the user interacts with the mobile VR device using gestures, voice, or gaze, the server will not obtain sensor data from the handheld device, but only sensor data from the mobile VR device.

[0093] When a mobile VR device interacts with a handheld device, the handheld device can send the collected sensor data to the mobile VR device. The mobile VR device then synchronizes the sensor data it has collected with the handheld device's sensor data and packages it before sending it to the server.

[0094] For example, sensor data includes, but is not limited to: linear velocity, linear acceleration, angular velocity, angular acceleration, position information, and orientation information of the device. Linear velocity and linear acceleration can be triaxial linear velocity and triaxial acceleration, angular velocity and angular acceleration can be triaxial angular velocity and triaxial angular acceleration, position information can be a three-dimensional coordinate, and orientation information can be a quaternion or Euler angles, etc.

[0095] Taking sensors, including IMUs, as an example, an IMU is a very common inertial sensor. The sensor data measured by the IMU can be called inertial data. This inertial data can include triaxial acceleration and triaxial orientation data calculated internally by the sensor. The orientation can be represented by quaternions.

[0096] The virtual reality engine in the server and the head-mounted device usually define a pass-through format for sensor data. It is understandable that different types of virtual reality engines or head-mounted devices define different data pass-through formats.

[0097] Taking the SteamVR platform as an example, the virtual reality engine provides the OpenVR interface to receive sensor data sent by the head-mounted device and render the image texture corresponding to the user's current posture in real time using the sensor data.

[0098] The OpenVR interface will obtain TransQuaternion rotation (orientation information), TransVector3 position (position information), TransVector3 linear Velocity (three-axis linear velocity), TransVector3 linear Acceleration (three-axis linear acceleration), TransVector3 angular Velocity (three-axis angular velocity), and TransVector3 angular Acceleration (three-axis angular acceleration).

[0099] Among them, TransVector3 rotation is a quaternion data, TransVector3 position is a three-dimensional coordinate data, and similarly, triaxial linear velocity, triaxial acceleration, triaxial angular velocity and triaxial angular acceleration are all sensor data represented using three-dimensional coordinate data.

[0100] S102. When sensor data is lost or delayed, predict the device's pose information at the current acquisition time based on the sensor data and pose information of the device in the previous n acquisition times.

[0101] The sensor collects data at a preset frequency. The sensor data includes a timestamp, which can be the time of data collection. After receiving the sensor data, the server determines whether a delay or packet loss has occurred based on the timestamp in the sensor data.

[0102] When there is no packet loss or delay in sensor data, image rendering and other processing are performed based on the sensor data. When network latency or packet loss occurs, the sensor data sent to the virtual reality engine (e.g., SteamVR) becomes discontinuous. This directly leads to discontinuous images rendered by the virtual reality engine, such as stuttering in the head-mounted display (caused by intermittent sensor data from the head-mounted display) or jerking in the position of the controllers in the image (caused by intermittent sensor data from the controllers).

[0103] For example, the server determines the linear velocity, linear acceleration, angular velocity, and angular acceleration of the device at the current acquisition moment based on the linear velocity, linear acceleration, angular velocity, and angular acceleration of the device at the previous n acquisition moments, where n is an integer greater than or equal to 1. Then, based on the linear velocity and linear acceleration of the device at the current acquisition moment, the server calculates the displacement of the device from the previous acquisition moment to the current acquisition moment. Based on the position information and displacement of the device at the previous acquisition moment, the server obtains the position information of the device at the current acquisition moment. Furthermore, based on the angular velocity of the device at the current acquisition moment, the server calculates the rotation angle of the device from the previous acquisition moment to the current acquisition moment. Based on the orientation information and rotation angle of the device at the previous acquisition moment, the server obtains the orientation information of the device at the current acquisition moment.

[0104] Optionally, the linear velocity, linear acceleration, angular velocity, and angular acceleration of the device at the current acquisition moment can be predicted in the following three ways.

[0105] In the first implementation method, the linear velocity, linear acceleration, angular velocity, and angular acceleration of the device at the previous acquisition moment are determined to be the linear velocity, linear acceleration, angular velocity, and angular acceleration of the device at the current acquisition moment.

[0106] That is, the linear velocity, linear acceleration, angular velocity, and angular acceleration of the device at the current acquisition moment are consistent with those at the previous acquisition moment.

[0107] Implementation Method Two: The average or weighted average of the linear velocities of the device over the previous n acquisition times is determined as the linear velocity of the device at the current acquisition time; the average or weighted average of the linear accelerations of the device over the previous n acquisition times is determined as the linear acceleration of the device at the current acquisition time; the average or weighted average of the angular velocities of the device over the previous n acquisition times is determined as the angular velocity of the device at the current acquisition time; and the average or weighted average of the angular accelerations of the device over the previous n acquisition times is determined as the angular acceleration of the device at the current acquisition time.

[0108] For example, n is set to 3, meaning the server calculates the average or weighted average of the linear velocity, linear acceleration, angular velocity, and angular acceleration from the three acquisition times preceding the current acquisition time, thus obtaining the device's linear velocity, linear acceleration, angular velocity, and angular acceleration at the current acquisition time. The weighting factor for the weighted average can be preset.

[0109] The third method involves multiplying the linear velocity, linear acceleration, angular velocity, and angular acceleration of the device at the previous acquisition time by a preset adjustment factor to obtain the linear velocity, linear acceleration, angular velocity, and angular acceleration of the device at the current acquisition time.

[0110] For example, the preset adjustment factor is 0.8, 0.9 or 1.1, etc. Optionally, the adjustment factors corresponding to linear velocity, linear acceleration, angular velocity and angular acceleration can be the same or different. This embodiment does not limit this.

[0111] After determining the device's offline velocity and linear acceleration at the current acquisition moment, the displacement of the device from the previous acquisition moment to the current acquisition moment can be calculated using the following distance formula: S = V0t + 1 / 2at2, where V0 represents the device's linear velocity at the current acquisition moment, a represents the device's linear acceleration at the current acquisition moment, t represents the system time interval at the current frame rate, and S represents the displacement distance.

[0112] The displacement of the device from the previous acquisition time to the current acquisition time is calculated using the above distance formula. This displacement refers to the data increment of the device in the x, y, and z axes, i.e., the movement distance. Furthermore, this displacement is added to the device's position information at the previous acquisition time to obtain the device's position information at the current acquisition time.

[0113] After determining the angular velocity of the device at the current acquisition moment, the rotation angle of the device from the previous acquisition moment to the current acquisition moment can be calculated using the following formula: angular velocity = rotation angle / t, where angular velocity refers to the angular velocity of the device at the current acquisition moment, rotation angle refers to the rotation angle of the device from the previous acquisition moment to the current acquisition moment, and t represents the system time interval at the current frame rate.

[0114] In 3D graphics, the most commonly used rotation representation methods are quaternions and Euler angles, which have the advantages of saving storage space and facilitating interpolation compared to matrices.

[0115] Optionally, if the rotation angle is an Euler angle and the orientation information is a quaternion, then the rotation angle needs to be converted into a quaternion. Based on the orientation information of the device at the previous acquisition time and the converted quaternion, the orientation information of the device at the current acquisition time can be obtained.

[0116] Euler angles are a set of three independent angular parameters used to uniquely determine the position of a rigid body rotating at a fixed point. They consist of the nutation angle θ, precession angle ψ, and rotation angle φ. Euler angles and quaternions are interchangeable; for example, the conversion between Euler angles and quaternions can be achieved using the following formula:

[0117]

[0118] Where θ, ψ, and φ are the rotation angles around the Z-axis, Y-axis, and X-axis, respectively. If represented by Tait-Bryan angle, they are yaw, roll, and pitch, respectively. q0, q1, q2, and q3 are quaternions, corresponding to x, y, z, and w in the Trans Quaternion rotation information.

[0119] It should be noted that when the device includes both mobile VR devices and handheld devices, the sensor data of the mobile VR devices and handheld devices are predicted independently when predicting the sensor data of the mobile VR devices and handheld devices at the current acquisition time.

[0120] S103. Perform image rendering processing based on the predicted sensor data of the device at the current acquisition time.

[0121] The server updates the sensor data based on the predicted sensor data at the current acquisition time, and uses the updated sensor data for image rendering. Taking SteamVR as an example, SteamVR calls the OpenVR interface to import the updated sensor data.

[0122] In this embodiment, when sensor data from the mobile VR device and handheld device experiences delays or packet loss, the sensor data from the previous n acquisition moments of the mobile VR device and handheld device is used to predict the sensor data from the previous n acquisition moments of the mobile VR device and handheld device. The predicted sensor data is then used for image rendering and other operations, enabling the virtual reality engine to render continuous texture images (including continuous changes in viewport position and continuous updates in the position of the handheld device in the image). This effectively reduces the impact of network disturbances and enhances the user's immersive experience.

[0123] In addition, the method in this embodiment uses one-dimensional sensor data for processing, rather than using image-based algorithms such as machine vision (two-dimensional image calculation, or even three-dimensional texture inference). It decomposes the three-dimensional coordinate data and calculates the x, y, and z axes separately. Compared with two-dimensional or even three-dimensional calculation, the processing logic is simpler, does not require additional image acquisition and image recognition systems, and is more efficient.

[0124] S104. Send the rendered image to the mobile VR device.

[0125] Optionally, the server can also determine the haptic feedback result of the handheld device based on the predicted sensor data of the device at the current acquisition time, and send the haptic feedback result to the mobile VR device. The server can package the haptic feedback result and the rendered image and send them to the mobile VR device.

[0126] Mobile VR devices perform ATW or ASW frame interpolation on received images based on latency and packet loss, and forward the received haptic feedback results to the handheld device for specific haptic feedback. The human eye then responds to the images displayed on the mobile VR device.

[0127] In this embodiment, when sensor data from a mobile VR device or handheld device experiences packet loss or delay, the server predicts the sensor data at the current acquisition time based on the sensor data from the previous n acquisition times. Then, image rendering is performed based on the predicted sensor data at the current acquisition time. Using the predicted sensor data for image rendering and other operations enables the server to render continuous texture images, including continuous changes in viewport position and continuous updates to the handheld device's position within the image. This effectively reduces the impact of network disturbances and significantly enhances the user's immersive experience.

[0128] Based on Embodiment 1, Embodiment 2 of this application describes in detail the interaction process between the server, the mobile VR device, and the handheld device from a system perspective. This embodiment uses a cloud server as the server, a head-mounted VR device as the mobile VR device, and a controller as the handheld device for illustration. The same content is described in the foregoing embodiments, and will not be repeated in this embodiment. Figure 6 The signaling flowchart of the data processing method for the virtual reality system provided in Embodiment 2 of this application is as follows: Figure 6 As shown, the method provided in this embodiment includes the following steps.

[0129] S201, The handle sends sensor data from the handle to the head-mounted device.

[0130] The handle collects sensor data at a preset sampling frequency and sends it to the head-mounted device. The handle then transmits the sensor data to the head-mounted device via a network transmission protocol or a proprietary protocol.

[0131] S202, The head-mounted device synchronizes and packages the sensor data from the handle and the sensor data from the head-mounted device.

[0132] The head-mounted device collects sensor data from itself at a preset collection frequency, and synchronizes its own sensor data with the sensor data from the controller. After synchronization, the sensor data from the controller and the head-mounted device are packaged and sent to the cloud server.

[0133] Optionally, in some embodiments, the cloud server can synchronize the sensor data of the handle and the sensor data of the head-mounted device. Accordingly, after the head-mounted device receives the sensor data of the handle, it does not need to synchronize and can directly forward the sensor data of the handle to the cloud server.

[0134] S203, The head-mounted device sends sensor data from the handle and the head-mounted device to the cloud server.

[0135] S204. The cloud server determines whether there is a delay or packet loss in the sensor data.

[0136] Head-mounted devices interact with cloud servers via network communication protocols. Due to network conditions, network latency or packet loss is inevitable, which has a more significant impact on cloud rendering and cloud gaming solutions.

[0137] When sensor data is delayed or lost, proceed to step S205; when sensor data is not delayed or lost, proceed to step S206.

[0138] S205. The cloud server predicts the sensor data at the current acquisition time based on the sensor data from the previous n acquisition times.

[0139] The specific implementation method of this step is described in Example 1, and will not be repeated here.

[0140] S206. The cloud server performs image rendering and touch logic processing based on the sensor data collected at the current time.

[0141] S207, The cloud server packages the rendered image and haptic feedback results and sends them to the head-mounted device.

[0142] S208. The head-mounted device determines whether the rendered image is delayed or has packet loss.

[0143] If the rendered image experiences delay or packet loss, step S209 is executed. If the rendered image does not experience delay or packet loss, the head-mounted device displays the rendered image (not shown in the figure) on the display screen.

[0144] S209. The head-mounted device uses ATW or ASW to perform frame interpolation on the image and displays the interpolated image on the screen.

[0145] After seeing the images displayed on the head-mounted device, the human eye makes a functional response, such as turning the head or operating the control handle.

[0146] S210, the head-mounted device forwards the haptic feedback results to the handle.

[0147] S211, The handle controls the magnitude of the feedback force or the movement of the handle position based on the tactile feedback results.

[0148] In this embodiment, the cloud server calculates and predicts the sensor data of the head-mounted device and controller at the current acquisition time based on the sensor data of the head-mounted device and controller at the previous n acquisition times, and imports the predicted sensor data of the head-mounted device and controller into the virtual reality engine, so that the virtual reality engine can render continuous texture images, effectively reduce the impact of network disturbances, and effectively improve the user's immersive experience.

[0149] To facilitate better implementation of the data processing method for the virtual reality system in the embodiments of this application, the embodiments of this application also provide a data processing device for a virtual reality system. Figure 7 This is a schematic diagram of the structure of the data processing device for the virtual reality system provided in Embodiment 3 of this application, as shown below. Figure 7 As shown, the data processing device 200 of the virtual reality system may include:

[0150] Acquisition module 21 is used to acquire sensor data of the device, the device including a mobile virtual reality (VR) device, or the mobile VR device and a handheld device for interacting with the mobile VR device;

[0151] Prediction module 22 is used to predict the sensor data of the device at the current acquisition time based on the sensor data of the device in the previous n acquisition times when the sensor data is lost or delayed, where n is an integer greater than or equal to 1;

[0152] Rendering module 23 is used to perform image rendering processing based on the predicted sensor data of the device at the current acquisition time;

[0153] The sending module 24 is used to send the rendered image to the mobile VR device.

[0154] In some embodiments, the sensor data includes the device’s linear velocity, linear acceleration, angular velocity, angular acceleration, position information, and orientation information.

[0155] In some embodiments, the prediction module 22 is specifically used for:

[0156] Based on the linear velocity, linear acceleration, angular velocity, and angular acceleration of the device at the current acquisition time, determine the linear velocity, linear acceleration, angular velocity, and angular acceleration of the device at the previous n acquisition times;

[0157] Based on the linear velocity and linear acceleration of the device at the current acquisition time, calculate the displacement of the device from the previous acquisition time to the current acquisition time;

[0158] The position information of the device at the current acquisition time is obtained based on the device's position information at the previous acquisition time and the displacement amount;

[0159] Based on the angular velocity of the device at the current acquisition time, calculate the rotation angle of the device from the previous acquisition time to the current acquisition time;

[0160] Based on the orientation information of the device at the previous acquisition time and the rotation angle, the orientation information of the device at the current acquisition time is obtained.

[0161] In some embodiments, the prediction module 22 is specifically used for:

[0162] The linear velocity, linear acceleration, angular velocity, and angular acceleration of the device at the previous acquisition time are determined as the linear velocity, linear acceleration, angular velocity, and angular acceleration of the device at the current acquisition time.

[0163] In some embodiments, the prediction module 22 is specifically used for:

[0164] The average or weighted average of the linear velocity of the device over the previous n acquisition times is determined as the linear velocity of the device at the current acquisition time;

[0165] The average or weighted average of the linear acceleration of the device over the previous n acquisition times is determined as the linear acceleration of the device at the current acquisition time.

[0166] The average or weighted average of the angular velocities of the device over the previous n acquisition times is determined as the angular velocity of the device at the current acquisition time;

[0167] The average or weighted average of the angular acceleration of the device over the previous n acquisition times is determined as the angular acceleration of the device at the current acquisition time.

[0168] In some embodiments, the prediction module 22 is specifically used for:

[0169] The linear velocity, linear acceleration, angular velocity, and angular acceleration of the device at the previous acquisition time are multiplied by a preset adjustment factor to obtain the linear velocity, linear acceleration, angular velocity, and angular acceleration of the device at the current acquisition time.

[0170] In some embodiments, the orientation information is a quaternion, and the rotation angle is an Euler angle;

[0171] The prediction module 22 is specifically used for:

[0172] Convert the rotation angle into a quaternion;

[0173] The orientation information of the device at the current acquisition time is obtained based on the orientation information of the device at the previous acquisition time and the converted quaternion.

[0174] In some embodiments, the acquisition module 21 is specifically used for:

[0175] Receive and package the sensor data of the mobile VR device and the sensor data of the handheld device, which are synchronously transmitted by the mobile VR device.

[0176] In some embodiments, a processing module is further included for:

[0177] Based on the predicted sensor data of the device at the current acquisition time, the tactile feedback result of the handheld device is determined;

[0178] The sending module 24 is also used to send the haptic feedback result to the mobile VR device.

[0179] In some embodiments, the server is a cloud server, and the mobile VR device is a head-mounted device.

[0180] It should be understood that the device embodiments and method embodiments can correspond to each other, and similar descriptions can be referred to the method embodiments. To avoid repetition, further details will not be provided here.

[0181] The apparatus 200 of this application embodiment has been described above from the perspective of functional modules in conjunction with the accompanying drawings. It should be understood that this functional module can be implemented in hardware, in software instructions, or in a combination of hardware and software modules. Specifically, the steps of the method embodiments in this application can be completed by integrated logic circuits in the processor's hardware and / or by software instructions. The steps of the method disclosed in this application embodiment can be directly embodied as being executed by a hardware decoding processor, or by a combination of hardware and software modules in the decoding processor. Optionally, the software module can be located in a mature storage medium in the art, such as random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, etc. This storage medium is located in memory, and the processor reads information from the memory and, in conjunction with its hardware, completes the steps in the above method embodiments.

[0182] This application also provides a server. Figure 8 This is a schematic diagram of a server structure provided in Embodiment 4 of this application, as shown below. Figure 8 As shown, the server 300 may include:

[0183] The system includes a memory 31 and a processor 32. The memory 31 stores computer programs and transfers the program code to the processor 32. In other words, the processor 32 can retrieve and run the computer programs from the memory 31 to implement the methods described in the embodiments of this application.

[0184] For example, the processor 32 can be used to execute the above-described method embodiments according to instructions in the computer program.

[0185] In some embodiments of this application, the processor 32 may include, but is not limited to:

[0186] General-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc.

[0187] In some embodiments of this application, the memory 31 includes, but is not limited to:

[0188] Volatile memory and / or non-volatile memory. Non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. Volatile memory can be random access memory (RAM), which is used as an external cache. By way of example, but not limitation, many forms of RAM are available, such as Static RAM (SRAM), Dynamic RAM (DRAM), Synchronous DRAM (SDRAM), Double Data Rate SDRAM (DDR SDRAM), Enhanced Synchronous DRAM (ESDRAM), Synchronous Link DRAM (SLDRAM), and Direct Rambus RAM (DR RAM).

[0189] In some embodiments of this application, the computer program may be divided into one or more modules, which are stored in the memory 31 and executed by the processor 32 to perform the method provided in this application. The one or more modules may be a series of computer program instruction segments capable of performing a specific function, which describe the execution process of the computer program in the server.

[0190] like Figure 8 As shown, the server may also include a transceiver 33, which can be connected to the processor 32 or the memory 31.

[0191] The processor 32 can control the transceiver 33 to communicate with other devices; specifically, it can send information or data to other devices or receive information or data sent by other devices. The transceiver 33 may include a transmitter and a receiver. The transceiver 33 may further include antennas, and the number of antennas may be one or more.

[0192] Understandable, although Figure 8As not shown in the diagram, the server 300 may also include a camera module, a Wi-Fi module, a positioning module, a Bluetooth module, a display, a controller, etc., which will not be described in detail here.

[0193] It should be understood that the various components in the server are connected through a bus system, which includes a data bus, a power bus, a control bus, and a status signal bus.

[0194] This application also provides a computer storage medium storing a computer program thereon, which, when executed by a computer, enables the computer to perform the methods of the above-described method embodiments. Alternatively, embodiments of this application also provide a computer program product containing instructions that, when executed by a computer, cause the computer to perform the methods of the above-described method embodiments.

[0195] This application also provides a computer program product, which includes a computer program stored in a computer-readable storage medium. The server's processor reads the computer program from the computer-readable storage medium and executes the computer program, causing the server to perform the corresponding flow in the user position control method in the virtual scene of this application embodiment. For simplicity, further details are omitted here.

[0196] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of modules is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple modules or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or modules may be electrical, mechanical, or other forms.

[0197] The modules described as separate components may or may not be physically separate. The components shown as modules may or may not be physical modules; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. For example, the functional modules in the various embodiments of this application may be integrated into one processing module, or each module may exist physically separately, or two or more modules may be integrated into one module.

[0198] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A data processing method for a virtual reality system, characterized in that, Applied to a server, the method includes: Acquire sensor data from the device, which includes a mobile virtual reality (VR) device, or the mobile VR device and a handheld device for interacting with the mobile VR device; When the sensor data is lost or delayed, the sensor data at the current acquisition time is predicted based on the sensor data of the device in the previous n acquisition times, where n is an integer greater than or equal to 1. Image rendering processing is performed based on the sensor data of the device obtained from the prediction at the current acquisition time; The rendered image is sent to the mobile virtual reality (VR) device; The sensor data includes the device's linear velocity, linear acceleration, angular velocity, angular acceleration, position information, and orientation information; The step of predicting the sensor data of the device at the current acquisition time based on the sensor data of the device in the previous n acquisition times includes: Based on the linear velocity, linear acceleration, angular velocity, and angular acceleration of the device at the current acquisition time, determine the linear velocity, linear acceleration, angular velocity, and angular acceleration of the device at the previous n acquisition times; Based on the linear velocity and linear acceleration of the device at the current acquisition time, calculate the displacement of the device from the previous acquisition time to the current acquisition time; The position information of the device at the current acquisition time is obtained based on the device's position information at the previous acquisition time and the displacement amount; Based on the angular velocity of the device at the current acquisition time, calculate the rotation angle of the device from the previous acquisition time to the current acquisition time; Based on the orientation information of the device at the previous acquisition time and the rotation angle, the orientation information of the device at the current acquisition time is obtained.

2. The method according to claim 1, characterized in that, The step of determining the linear velocity, linear acceleration, angular velocity, and angular acceleration of the device at the current acquisition moment based on the linear velocity, linear acceleration, angular velocity, and angular acceleration of the device at the previous n acquisition moments includes: The linear velocity, linear acceleration, angular velocity, and angular acceleration of the device at the previous acquisition time are determined as the linear velocity, linear acceleration, angular velocity, and angular acceleration of the device at the current acquisition time.

3. The method according to claim 1, characterized in that, The step of determining the linear velocity, linear acceleration, angular velocity, and angular acceleration of the device at the current acquisition moment based on the linear velocity, linear acceleration, angular velocity, and angular acceleration of the device at the previous n acquisition moments includes: The average or weighted average of the linear velocity of the device over the previous n acquisition times is determined as the linear velocity of the device at the current acquisition time; The average or weighted average of the linear acceleration of the device over the previous n acquisition times is determined as the linear acceleration of the device at the current acquisition time. The average or weighted average of the angular velocities of the device over the previous n acquisition times is determined as the angular velocity of the device at the current acquisition time; The average or weighted average of the angular acceleration of the device over the previous n acquisition times is determined as the angular acceleration of the device at the current acquisition time.

4. The method according to claim 1, characterized in that, The step of determining the linear velocity, linear acceleration, angular velocity, and angular acceleration of the device at the current acquisition moment based on the linear velocity, linear acceleration, angular velocity, and angular acceleration of the device at the previous n acquisition moments includes: The linear velocity, linear acceleration, angular velocity, and angular acceleration of the device at the previous acquisition time are multiplied by a preset adjustment factor to obtain the linear velocity, linear acceleration, angular velocity, and angular acceleration of the device at the current acquisition time.

5. The method according to claim 1, characterized in that, The orientation information is a quaternion, and the rotation angle is an Euler angle; The step of obtaining the orientation information of the device at the current acquisition time based on the orientation information of the device at the previous acquisition time and the rotation angle includes: Convert the rotation angle into a quaternion; The orientation information of the device at the current acquisition time is obtained based on the orientation information of the device at the previous acquisition time and the converted quaternion.

6. The method according to any one of claims 1-5, characterized in that, The acquisition of sensor data from the device includes: Receive and package the sensor data of the mobile virtual reality (VR) device and the sensor data of the handheld device, which are simultaneously transmitted by the mobile VR device.

7. The method according to any one of claims 1-5, characterized in that, Also includes: Based on the predicted sensor data of the device at the current acquisition time, the tactile feedback result of the handheld device is determined; The haptic feedback results are sent to the mobile virtual reality (VR) device.

8. The method according to any one of claims 1-5, characterized in that, The server is a cloud server, and the mobile virtual reality (VR) device is a head-mounted device.

9. A data processing device for a virtual reality system, characterized in that, The device includes: An acquisition module is used to acquire sensor data of a device, the device including a mobile virtual reality (VR) device, or the mobile VR device and a handheld device for interacting with the mobile VR device; The prediction module is used to predict the sensor data of the device at the current acquisition time based on the sensor data of the device in the previous n acquisition times when the sensor data is lost or delayed, where n is an integer greater than or equal to 1; The rendering module is used to perform image rendering processing based on the predicted sensor data of the device at the current acquisition time. The sending module is used to send the rendered image to the mobile virtual reality (VR) device; The sensor data includes the device's linear velocity, linear acceleration, angular velocity, angular acceleration, position information, and orientation information; The prediction module is specifically used to determine the linear velocity, linear acceleration, angular velocity, and angular acceleration of the device at the current acquisition time based on the linear velocity, linear acceleration, angular velocity, and angular acceleration of the device at the previous n acquisition times. Based on the linear velocity and linear acceleration of the device at the current acquisition time, calculate the displacement of the device from the previous acquisition time to the current acquisition time; The position information of the device at the current acquisition time is obtained based on the device's position information at the previous acquisition time and the displacement amount; Based on the angular velocity of the device at the current acquisition time, calculate the rotation angle of the device from the previous acquisition time to the current acquisition time; Based on the orientation information of the device at the previous acquisition time and the rotation angle, the orientation information of the device at the current acquisition time is obtained.

10. A server, characterized in that, include: A processor and a memory, the memory being used to store a computer program, the processor being used to invoke and run the computer program stored in the memory to perform the method of any one of claims 1 to 8.

11. A computer-readable storage medium, characterized in that, Used to store a computer program that causes a computer to perform the method as described in any one of claims 1 to 8.

12. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the method as described in any one of claims 1 to 8.

Images