Multi-person VR task unloading method based on multi-path transmission

Abstract

The invention discloses a multi-person VR task unloading method based on multi-path transmission. The method comprises the following steps: S1, a user wears VR glasses to provide an interactive application and add the interactive application into an interactive system; s2, constructing a multi-path transmission network; s3, the user initializes the position and direction of the user, the VR head-mounted display tracks user motion record data, generates a task and packages the task, and sends the task to the edge server according to a multi-path scheduling algorithm; s4, obtaining the number of triangular surfaces and texture pixels of the 3D model rendered in the scene, and obtaining the number of current users from an edge server; s5, the edge server gives a priority scheduling and optimization method, carries out adaptive task scheduling and computing resource allocation, and executes the task; s6, rendering a virtual picture of the camera by the edge server, and returning the virtual picture to the user in a video stream form; and the VR head-mounted display merges and outputs a remote rendering result and a local rendering result. According to the invention, the calculation load of the terminal equipment is reduced, and smoother and consistent VR interaction experience is provided for multiple users.

Description

Technical Field

[0001] The present invention relates to the field of VR task offloading, and particularly to a multi-person VR task offloading method based on multi-path transmission. Background Art

[0002] In many existing solutions for multi-user VR task offloading, the core VR functions and rendering tasks are mainly implemented on the client side, and the server acts as a medium for communication between multiple terminals. This means that the main computing tasks of the entire system are completed by the terminals. These computing tasks mainly include: 3D rendering, information transmission, human-computer interaction, and VR tracking. According to the terminal carriers that the system faces, it can be divided into the following three types:

[0003] 1) PCVR: It needs to be connected to a computer for use. Usually, a high-performance computer is required to drive it to provide a high-quality experience and support full-body tracking.

[0004] 2) Game console VR: It is used in combination with game consoles (such as PS4, PS5, etc.) to provide a gaming experience. Usually, it needs to be connected to the console through a cable.

[0005] 3) Mobile VR: It is divided into mobile phone box VR and VR all-in-one. Mobile phone box VR is to put the mobile phone into the box for use, and the experience effect is affected by the performance of the mobile phone; VR all-in-one is an independent device that does not require external connection and provides a more convenient experience.

[0006] The above three existing carrier solutions each have their own shortcomings:

[0007] 1) PCVR: In addition to purchasing the VR headset itself, it is also necessary to invest in a high-performance computer, which increases the total cost of ownership. Most PCVR devices need to be connected to the computer through a cable, which may limit the user's movement range and increase the risk of falling. The experience quality highly depends on the performance of the computer. If the computer configuration is insufficient, there may be problems such as latency and image quality.

[0008] 2) Game console VR: Usually, it can only be compatible with specific brands of consoles. For example, PlayStation VR is only applicable to PlayStation game consoles. Compared with PCVR, the performance of game consoles may be limited, which may limit the quality of the VR experience. The number of available VR games and applications may not be as rich as that of PCVR. The hardware update cycle of game consoles is relatively long, and it may not be able to keep up with the rapidly developing VR technology.

[0009] 3) Mobile phone box VR: Dependent on the processing power and display screen of smartphones, it usually cannot provide an experience comparable to that of PC VR or game console VR. Most mobile phone box VRs do not support position tracking and can only provide limited rotational tracking. Due to its simple design, discomfort may be felt after long-term use. The available VR content is usually relatively simple and cannot be compared with the games and applications on high-end VR devices.

[0010] 4) All-in-one VR: Although it is stronger than mobile phone box VR, compared with PC VR, the graphics processing power of all-in-one VR is still limited. Due to the integration of the battery and processor, battery life may become a limiting factor, especially when running high-performance applications. Summary of the Invention

[0011] To solve the existing problems, the present invention provides a multi-person VR task offloading method based on multi-path transmission. The specific solution is as follows:

[0012] A multi-person VR task offloading method based on multi-path transmission, comprising the following steps:

[0013] S1, Multiple users wear lightweight VR glasses and provide a multi-person VR interaction application to join the interaction system. Each user has a unique user ID;

[0014] S2, The multi-person VR interaction application establishes a communication link with the edge server based on the QUIC protocol and constructs a multi-path transmission network;

[0015] S3, The user initializes their position and orientation in the VR environment. The VR headset tracks the user's head movement through the built-in sensor, records the corresponding data, generates a task based on this data, packs it, and sends it to the edge server according to the multi-path scheduling algorithm;

[0016] S4, For the 3D models required to be rendered in the scene, obtain the number of triangular faces and texture pixels of the model, and obtain the current number of users from the edge server;

[0017] S5, The edge server gives priority scheduling and optimization methods, performs adaptive task scheduling and computing resource allocation, and executes the task;

[0018] S6, The edge server renders the virtual image of the camera and returns it to the user in the form of a video stream; The VR headset merges and outputs the results of remote rendering and local rendering.

[0019] Preferably, step S3 includes the following steps:

[0020] S31, The user selects their own initialization position information in the VR environment according to the current program prompt, and the VR headset sends this data to the edge server according to the multi-path scheduling algorithm;

[0021] S32: The edge server constructs a completely virtual environment based on the user's initial position, or creates a copy of the virtual scene according to the real-world model and returns it to the user's VR headset for display;

[0022] S33: When the user performs corresponding operations, the VR headset generates corresponding tasks based on the body part transformation information generated by the user's operations, packs them, and sends a task upload request to the edge server;

[0023] S34: The edge server determines the user clients that can upload tasks currently according to the peak shaving strategy based on the current user task request scale, and notifies other user clients to wait;

[0024] S35: The user clients that can upload tasks select appropriate paths according to the multi-path scheduling algorithm to send the packed tasks to the edge server, and other user clients wait to see if they can upload in the next cycle.

[0025] Preferably, step S5 specifically includes the following steps:

[0026] S51. For the tasks arriving at the edge server in the current cycle, the edge server runs a priority-based scheduling algorithm to sort the tasks. The input parameters include the CPU cycles (w n,m ) and GPU cycles (y n,m ) required for the task to be completed, as well as the deadline and importance of the task, and the user ID to which the task belongs;

[0027] S52: This scheduling algorithm sorts the tasks according to the scheduling weight function , where IL n,m represents the importance of the task, wt n,m represents the time the task has waited after arriving at the edge server, represents the execution deadline of the task, λ n represents the preference degree of the edge server for the user to which the task belongs, and ns n,m is the normalized value of the resources required by the task;

[0028] S53: For the sorted tasks, if the memory resources of the edge server meet their execution requirements, they enter the allocation queue in sequence according to the sorting order and wait for the edge server to allocate computing resources for them. The tasks whose memory resources do not meet their execution requirements wait for the scheduling arrangement in the next cycle;

[0029] S54: For the tasks in the allocation queue, their computing resource allocation problem is described as a problem of solving the optimal value of a function. Based on the currently available CPU and GPU resources, the edge server uses the Lagrange multiplier method according to the KKT conditions to calculate the optimal resource allocation plan for the tasks.

[0030] S55: The edge server allocates CPU and GPU resources to the tasks according to the calculated optimal resource allocation result, and the tasks are executed.

[0031] The present invention also discloses a computer-readable storage medium with a computer program stored thereon. After the computer program runs, it executes the method described in any one of the above.

[0032] The present invention also discloses a computer system, including a processor and a storage medium. The storage medium stores a computer program, and the processor reads and runs the computer program from the storage medium to execute the method described in any one of the above.

[0033] The beneficial effects of the present invention are as follows:

[0034] In the present invention, computing tasks such as rendering and visual positioning are completed by the edge server, thereby solving the high demand for device hardware in the existing solutions, greatly improving the resource utilization rate of the server and the cross-platform performance of the system. It can not only provide a deep immersive 3D interaction experience, but also the device is very light and comfortable to wear. In addition, the present invention also proposes a VR task scheduling algorithm on the server side, which can adaptively schedule VR tasks, thereby increasing the upper limit of the number of users that the system can accommodate. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] In order to more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the following will briefly introduce the drawings required for the description of the embodiments or the prior art. Obviously, the drawings in the following description are some embodiments of the present invention. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative efforts.

[0036] Figure 1 It is a flowchart of the method of the present invention.

[0037] Figure 2 It is a schematic diagram of a task offloading process provided by the present invention.

[0038] Figure 3 It is a schematic diagram of an adaptive rendering task scheduling and computing resource allocation algorithm provided by the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0039] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are some, but not all, of the embodiments of the present invention. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without creative efforts shall fall within the protection scope of the present invention.

[0040] As Figure 1 , a multi-person VR task offloading method based on multi-path transmission includes the following steps:

[0041] S1. Multiple users wear lightweight VR glasses and provide a multi-person VR interaction application to join the interaction system, supporting interaction methods such as gestures or laser handles. Each user has a unique user ID.

[0042] S2. The multi-person VR interaction application establishes a communication link with the edge server based on the QUIC protocol and constructs a multi-path transmission network.

[0043] Specifically, the application uses the MPQUIC protocol to establish a communication link with the edge server and constructs a multi-path transmission network to optimize data transmission. The application initiates a connection with the edge server through the MPQUIC protocol and performs a P2P negotiation process through this connection. With the support of the MPQUIC protocol, two parallel transmission paths are established between the edge server and the client: WiFi and 5G, to achieve redundancy and load balancing. After a series of negotiations, the edge server and the client can establish a multi-path P2P connection based on MPQUIC. This connection is not only used for the edge server to transmit the rendered video stream and audio stream to the client, but also a DataChannel based on MPQUIC is established to transmit user interaction data, pose data, text information, etc.

[0044] S3. The user initializes his position and orientation in the VR environment. The VR headset tracks the user's head movement through the built-in sensor, records the corresponding data, generates a task and packages it, and sends it to the edge server according to the multi-path scheduling algorithm;

[0045] Specifically, step S3 includes the following steps:

[0046] S31. The user selects his virtual coordinates, orientation, and other initialization position information in the VR environment according to the current program prompt. The VR headset sends this data to the edge server according to the multi-path scheduling algorithm.

[0047] S32: The edge server constructs a completely virtual environment based on the user's initial position, or creates a copy of the virtual scene according to the real-world model and returns it to the user's VR headset for display. That is, the edge server retrieves and matches with the pre-constructed virtual scene and the position information provided by the user, and uses the pose estimation algorithm to obtain the pose of the space where the user is currently located. The system performs three-dimensional reconstruction on the virtual scene, and the generated map contains the pose information of each user in the virtual scene.

[0048] S33: When the user performs corresponding operations, the sensors built into the VR headset generate corresponding tasks in a package based on the body part transformation information generated by the user's operations and send a task upload request to the edge server.

[0049] S34: The edge server determines the user clients that can upload tasks currently according to the peak shaving strategy based on the current user task request scale, and notifies other user clients to wait. The peak shaving strategy is as follows:

[0050] After the edge server counts the users who have currently made upload requests, when the number of these users (request_number) is in the interval [10n + 1, 10(n + 1)], the proportion of users who need to wait is determined according to the following stepped waiting ratio formula:

[0051]

[0052] The number of users who need to wait (wait_number) can be obtained by request_number × wait_ratio. After determining the number of users who should wait, we will use the roulette method to select the specific users who need to wait. The probability of each user being selected in each round of the roulette method is 1 - λ, where λ is the preference degree of the edge server for the user. For the users selected to wait, their clients will wait for whether they can upload in the next cycle. For the users who can upload, their clients will use the LinUCB algorithm to select a suitable path to upload tasks. The feature vector in the LinUCB algorithm contains the current delay, rate, and jitter information of the two paths.

[0053] S35: The user clients that can upload tasks select a suitable path according to the multi-path scheduling algorithm to send the packaged tasks to the edge server, and other user clients wait for whether they can upload in the next cycle.

[0054] S4. For the 3D models required to be rendered in the scene, obtain the number of triangular faces and texture pixels of the models, and obtain the current number of users from the edge server;

[0055] S5. The edge server gives priority scheduling and optimization methods, performs adaptive task scheduling, computational resource allocation, and executes tasks.

[0056] Specifically, step S5 specifically includes the following steps:

[0057] S51. For the tasks arriving at the edge server in the current cycle, the edge server runs a priority-based scheduling algorithm to sort the tasks. The input parameters include the CPU cycles (w n,m ) and GPU cycles (y n,m ) required for the task to be completed, as well as the deadline and importance level of the task, and the user ID to which the task belongs.

[0058] S52: As Figure 3 shown, this scheduling algorithm sorts the tasks according to the scheduling weight function , where IL n,m represents the importance level of the task, wt n,m represents the time the task has waited after arriving at the edge server, represents the execution deadline of the task, λ n represents the preference level of the edge server for the user to which the task belongs, ns n,m is the normalized value of the resources required by the task.

[0059] S53. For the sorted tasks, if the memory resources of the edge server meet their execution requirements, they enter the allocation queue in sequence according to the sorting order and wait for the edge server to allocate computational resources to them. Tasks whose memory resources do not meet their execution requirements wait for the scheduling arrangement in the next cycle.

[0060] S54. For the tasks in the allocation queue, their computational resource allocation problem is described as a function optimal value solving problem. The edge server, based on the currently available CPU and GPU resources, uses the Lagrange multiplier method according to the KKT conditions to calculate the optimal resource allocation plan for the tasks.

[0061] The formula used for the function optimal value solving problem is as follows:

[0062] In the current cycle, where U is the set of users to which the tasks in the allocation queue belong, TK n is the set of tasks belonging to user n in the allocation queue, w n,m and y n,m are the CPU cycles and GPU cycles required for the m-th task of user n to be completed, c n,m and g n,m are the CPU frequency and GPU frequency allocated to the m-th task of user n. By optimizing the c n,m and g of this functionn,m Taking the second derivative respectively shows that the Hessian matrix of the function is strictly positive definite. Therefore, the optimal value can be solved by the KKT conditions according to the Lagrange multiplier method. The optimal CPU and GPU resource allocation results are as follows:

[0063]

[0064] Among them, C(t) and G(t) are the CPU resources and GPU resources available to the edge server in the current cycle respectively.

[0065] S55: The edge server allocates CPU and GPU resources to the task according to the calculated optimal resource allocation result, and the task is executed.

[0066] S6, the edge server renders the virtual image of the camera and returns it to the user in the form of a video stream; the VR headset merges and outputs the results of remote rendering and local rendering.

[0067] In the present invention, rendering and visual positioning and other computing tasks are completed by the edge server, so as to solve the high demand for device hardware in the existing solutions, greatly improve the resource utilization rate of the server and the cross-platform performance of the system. It can not only provide a deep immersive 3D interaction experience, but also the device is very light and comfortable to wear. In addition, the present invention also proposes a VR task scheduling algorithm on the server side, which can adaptively schedule VR tasks, thereby increasing the upper limit of the number of users that the system can accommodate.

[0068] The present invention also discloses a computer-readable storage medium and a computer system. Among them, a computer program is stored on a computer-readable medium. After the computer program runs, it executes the method described in any one of the above. A computer system includes a processor and a storage medium. A computer program is stored on the storage medium. The processor reads and runs the computer program from the storage medium to execute the method described in any one of the above.

[0069] Those skilled in the art will further appreciate that the various illustrative logical blocks, modules, circuits, and algorithm steps described in conjunction with the embodiments disclosed herein may be implemented as electronic hardware, computer software, or a combination of the two. To clearly illustrate this interchangeability of hardware and software, the various illustrative components, blocks, modules, circuits, and steps are described above in terms of their functional form. Whether such functionality is implemented as hardware or software depends upon the particular application and design constraints imposed on the overall system. Skilled artisans may implement the described functionality in varying ways for each particular application, but such implementation decisions should not be interpreted as causing a departure from the scope of the present invention.

[0070] The various illustrative logical blocks, modules, and circuits described in connection with the embodiments disclosed herein can be implemented or performed with a general purpose processor, a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic device, discrete gate or transistor logic, discrete hardware components, or any combination thereof designed to perform the functions described herein. A general purpose processor may be a microprocessor, but in the alternative, the processor may be any conventional processor, controller, microcontroller, or state machine. The processor may also be implemented as a combination of computing devices, such as a combination of a DSP and a microprocessor, multiple microprocessors, one or more microprocessors in conjunction with a DSP core, or any other such configuration.

[0071] The steps of a method or algorithm described in connection with the embodiments disclosed herein can be embodied directly in hardware, in a software module executed by a processor, or in a combination of the two. A software module may reside in RAM memory, flash memory, ROM memory, EPROM memory, EEPROM memory, registers, a hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art. An exemplary storage medium is coupled to the processor such that the processor can read from, and write to, the storage medium. In the alternative, the storage medium may be integral to the processor. The processor and the storage medium may reside in an ASIC. The ASIC may reside in a user terminal. In the alternative, the processor and the storage medium may reside as discrete components in a user terminal.

[0072] In one or more exemplary embodiments, the described functionality may be implemented in hardware, software, firmware, or any combination thereof. If implemented in software as a computer program product, the functions may be stored on or transmitted via a computer-readable medium as one or more instructions or code. The computer-readable medium includes both computer storage media and communication media including any medium that facilitates transfer of a computer program from one place to another. The storage media may be any available media that can be accessed by a computer. By way of example and not limitation, such computer-readable media can comprise RAM, ROM, EEPROM, CD-ROM or other optical disk storage, magnetic disk storage or other magnetic storage devices, or any other medium that can be used to carry or store desired program code in the form of instructions or data structures and that can be accessed by a computer. Any connection is properly termed a computer-readable medium. For example, if the software is transmitted from a web site, server, or other remote source using a coaxial cable, fiber optic cable, twisted pair, DSL, or wireless technologies such as infrared, radio, and microwave, then the coaxial cable, fiber optic cable, twisted pair, DSL, or wireless technologies such as infrared, radio, and microwave are included in the definition of medium. As used herein, disk and disc include compact disc (CD), laser disc, optical disc, digital versatile disc (DVD), floppy disk, and Blu-ray disc where disk generally reproduces data magnetically and disc uses lasers optically to reproduce data. Combinations of the above should also be included within the scope of computer-readable media.

[0073] The foregoing description of the disclosure has been provided to enable any person skilled in the art to make or use the disclosure. Various modifications to the disclosure will be readily apparent to those skilled in the art, and the generic principles defined herein may be applied to other variations without departing from the spirit or scope of the disclosure. Thus, the disclosure is not intended to be limited to the examples and designs described herein but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

[0074] Although the present invention has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that: they can still modify the technical solutions recorded in the foregoing embodiments, or perform equivalent replacements for some of the technical features; and these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A multi-person VR task offloading method based on multi-path transmission, characterized in that: The following steps are involved: S1, multiple users wear lightweight VR glasses, and provide multi-person VR interactive applications to join the interactive system, and each user has a unique user ID; S2, the multi-person VR interactive application establishes a communication link with the edge server based on the QUIC protocol to construct a multi-path transmission network; S3, the user initializes his position and orientation in the VR environment. The VR headset tracks the user's head movement through the built-in sensor and records the corresponding data. The tasks are generated and packaged from the data and sent to the edge server according to the multi-path scheduling algorithm; S4, for the 3D model to be rendered in the scene, obtain the number of triangles and texture pixels of the model, and obtain the current number of users from the edge server; S5, the edge server gives priority scheduling and optimization methods, performs adaptive task scheduling and computing resource allocation, and executes tasks; S6: The edge server renders the virtual image of the camera and returns it to the user in the form of a video stream; the VR headset merges the remote rendering result with the local rendering result and outputs it.

2. The method according to claim 1, characterized in that Step S3 includes the following steps: S31, the user selects his initial position information in the VR environment according to the current program prompt, and the VR headset sends this data to the edge server according to the multi-path scheduling algorithm; S32: The edge server builds a completely virtual environment based on the user's initial position, or creates a copy of the virtual scene based on a model of the real world and returns it to the user's VR headset for display; S33: When the user performs a corresponding operation, the VR headset generates a corresponding task package according to the body part transformation information generated by the user operation and sends a task upload request to the edge server; S34: The edge server determines the user client that can currently upload tasks based on the current user task request scale according to the peak avoidance strategy, and notifies other user clients to wait; S35: The user client that can upload the task selects a suitable path according to the multi-path scheduling algorithm to send the packaged task to the edge server, and other user clients wait to see whether they can upload in the next cycle.

3. The method according to claim 1, characterized in that Step S5 specifically includes the following steps: S51, for the tasks arriving at the edge server in the current cycle, the edge server runs a priority-based scheduling algorithm to sort the tasks, and the input parameters include the CPU cycles required to complete the task execution (w n,m ) and GPU cycles (y n,m ), as well as the deadline and importance of the task, and the user ID to which the task belongs; S52: The scheduling algorithm is based on the scheduling weight function To sort the tasks, where IL n,m Indicates the importance of the task, wt n,m Indicates the waiting time after the task arrives at the edge server. represents the execution deadline of the task, λ n Indicates the edge server’s preference for the user to whom the task belongs, ns n,m The normalized value of the resources required for the task; S53: For the sorted tasks, if the memory resources of the edge server meet their execution requirements, they enter the allocation queue in turn according to the sorting order and wait for the edge server to allocate computing resources to them. Tasks whose memory resources do not meet their execution requirements wait for the scheduling of the next cycle; S54: For the tasks in the allocation queue, their computing resource allocation problem is described as a function optimal value solution problem. The edge server uses the Lagrange multiplier method to calculate the optimal resource allocation solution for the task based on the currently available CPU and GPU resources according to the KKT condition; S55: The edge server allocates CPU and GPU resources to the task according to the calculated optimal resource allocation result, and the task is executed.

4. A computer-readable storage medium, characterized in that: A computer program is stored on the medium, and after the computer program is run, the method according to any one of claims 1 to 3 is executed.

5. A computer system, characterized in that: The method comprises a processor and a storage medium, wherein a computer program is stored in the storage medium, and the processor reads and runs the computer program from the storage medium to execute the method as claimed in any one of claims 1 to 3.

Images