Multi-user collaboration method and system for mobile virtual reality based on edge computing

Abstract

The present invention discloses a multi-user collaborative method and system for mobile virtual reality based on edge computing. In an edge computing system for collaboration among a cloud server, multiple edge servers connected to the cloud server, and multiple user terminals connected to the edge server, a first edge server receives an interaction request from a first user terminal, decomposes the interaction task corresponding to the interaction request into a first real-time task and a second non-real-time task; completes the first real-time task locally on the edge server, sends the second non-real-time task to the cloud server for completion, receives the second completion data of the second non-real-time task completed by the cloud server, synthesizes the first completion data of the first real-time task with the second completion data, and returns it to the first user terminal; when different user terminals perform collaborative processing, the edge servers to which they belong forward data to each other and synchronize it to the corresponding user terminals. The present invention reduces the computing pressure on the cloud server and reduces latency.

Description

Technical Field

[0001] The present invention relates to the field of edge computing technology, and in particular to a multi-user collaboration method and system for mobile virtual reality based on edge computing. Background Art

[0002] The emergence of virtual reality (VR) and augmented reality (AR) technologies has revolutionized how users interact with the virtual world, enhancing their immersion and improving their experience in a variety of interactive scenarios, including gaming, healthcare, and education. In recent years, VR hardware has continuously advanced towards mobility, high performance, and long battery life. Simultaneously, VR application scenarios and user needs have become increasingly diverse.

[0003] However, current mobile VR devices have relatively limited computing power, battery life, and wireless communication capabilities, failing to meet the two requirements for high-quality VR applications: responsiveness (low latency) and high-quality visuals (ensure seamless playback through high frame rates). A common solution involves remotely rendering VR images in the cloud and then transmitting them to a mobile HMD (head-mounted display). 5G communication technology offers high bandwidth, making this cloud-end collaborative VR service model possible. As the number of users increases, the highly interactive nature of user behavior and the limited wireless network bandwidth required for VR implementations make it difficult to meet the real-time requirements of VR services. This has led to the introduction of edge computing into existing VR service systems that collaborate between devices and the cloud. Edge computing aims to move computing, storage, and network resources from cloud centers to the edge of the network to better meet end users' low-latency service requirements. It has become one of the most promising new computing models in the 5G era. Therefore, offloading the high-computational cost of VR image rendering to a nearby edge device, and then wirelessly transmitting the rendered image data back to the mobile head-mounted display, can effectively reduce the average processing delay; because the computing power of edge devices is usually much lower than that of the cloud platform, the assistance of the cloud platform is also needed to jointly complete the VR computing tasks, thus forming a VR service model based on end-edge-cloud collaboration.

[0004] With the continuous development of VR technology, its application scenarios are becoming increasingly diverse. Many VR application scenarios require multi-user collaboration. For example, VR-based engineering education and teaching platforms in medicine, navigation, mining, and other fields often require users to perform tasks that require multi-party collaborative training. In addition, the recent Covid-19 pandemic has imposed many restrictions on people's travel, leading to an increase in working and living at home, which has accelerated the birth and development of virtual reality social networking. Currently, most VR multi-user collaborative systems are deployed locally, with a small number of them being deployed in cloud environments. Moreover, when data is transmitted, it usually needs to be forwarded multiple times through edge servers and cloud servers to achieve collaboration. Due to the multi-user collaborative environment, relying solely on cloud-edge collaborative computing will still cause a large amount of synchronous data to be forwarded to the cloud, resulting in long data links and high latency. This greatly affects the application of VR multi-user collaborative systems in actual work and life, and greatly hinders the development of VR multi-user collaborative technology. Summary of the Invention

[0005] The present invention provides a multi-user collaboration method and system for mobile virtual reality based on edge computing, which is used to solve the technical problems of large amount of cloud-side synchronization data forwarding and high latency during VR multi-user collaboration in the existing VR service model based on end-edge-cloud collaboration.

[0006] The present invention provides a multi-user collaboration method for mobile virtual reality based on edge computing, which is used in an edge computing system in which a cloud server, multiple edge servers connected to the cloud server, and multiple user terminals connected to the edge servers collaborate with each other, comprising the following steps:

[0007] The first user terminal sends an interaction request to the first edge server to which it belongs;

[0008] The first edge server receives an interaction request from a first user terminal, decomposes the interaction task corresponding to the interaction request into a first real-time task and a second non-real-time task; completes the first real-time task locally on the edge server, sends the second non-real-time task to the cloud server for completion, receives second completion data of the second non-real-time task completed by the cloud server, combines the first completion data of the first real-time task with the second completion data, and returns the result to the first user terminal;

[0009] When the first edge server decomposes the interaction task corresponding to the interaction request from the first user terminal, it extracts the collaborative data request that needs to be synchronized from the second user terminal. When the second user terminal and the first user terminal are connected to subordinates of different edge servers, the collaborative data request is directly sent to the second edge server to which the second user terminal belongs, and the collaborative data returned from the second edge server is received.

[0010] Preferably, when the user terminal sends an interaction request to the edge server, the edge server decomposes the interaction task and executes the first real-time task, while determining whether a collaborative processing task between edge servers is required;

[0011] If the edge server needs to perform collaborative processing tasks, it forwards collaborative requests and data between edge servers, receives collaborative data returned by other edge servers, and finally synchronizes the collaborative data to the user terminal;

[0012] If the edge server does not need to perform the collaborative processing task, determining whether the second completion data exists in the cache of the edge server;

[0013] If the second completion data exists in the edge server cache, it is synthesized with the first completion data rendered by the edge server and sent to the user terminal.

[0014] If the second completion data does not exist in the edge server cache, a second non-real-time task rendering request is sent to the cloud server; after receiving the request, the cloud server determines whether the second completion data exists in the cloud server cache. If so, the cloud server directly returns the second completion data to the edge server; if not, the cloud server executes the second non-real-time task and then returns the second completion data to the edge server, and the edge server combines the first completion data with the second completion data and sends it to the user terminal.

[0015] Preferably, when the interaction request is a VR rendering request, the first real-time task is a foreground interaction rendering task, and the second non-real-time task is a distant BE rendering task.

[0016] The present invention also provides a multi-user collaborative system for mobile virtual reality based on edge computing, comprising: a cloud server, multiple edge servers connected to the cloud server, and multiple user terminals connected to the edge servers; the multiple edge servers are connected to each other;

[0017] Multiple user terminals are used to perceive user actions and send interaction requests to the edge server and decode and display the received rendering data;

[0018] The plurality of edge servers are configured to decompose the interaction request sent by the user terminal, execute the first real-time task to complete the first completion data according to the interaction request, coordinate the processing task, send the second non-real-time task request to the cloud server, receive the second completion data, and perform data merging;

[0019] The cloud server is used to execute the second non-real-time task and return second completion data.

[0020] Preferably, the user terminal includes: a motion detector module, a cloud-request encoding module, and a graphics decoding module;

[0021] The motion detector module is used to sense user movements; the cloud encoding module is requested to encode the motion update request sensed by the motion detector module;

[0022] The graphics decoding module is used to decode the graphics encoding data sent by the edge server into frame data.

[0023] Preferably, the edge server includes: a motion request receiver module, a frame merging module, an edge cache module, a panoramic transmitter module, an edge decoding module, a collaborative processing module and a near-view synchronization module;

[0024] The motion request receiver module is used to receive and decompose the interaction request sent by the user terminal. After the decomposition is completed, the collaborative processing module determines whether a collaborative processing task is required. In the collaborative processing task, the near-view synchronization module is responsible for the interaction of collaborative data between edge servers. The edge cache module is used to store the cached data decoded and rendered in the edge server.

[0025] The panoramic transmitter module is used to send a rendering request to the cloud server, and the second completion data returned from the cloud server received by the edge server is decoded by the edge decoding module; the frame merging module is used to render the first real-time task and merge the first completion data and the second completion data.

[0026] Preferably, the cloud server includes: a cloud decoding module, a perspective rendering engine module, a cloud cache module and a cloud encoding module;

[0027] The cloud decoding module is used to decode the rendering request received by the cloud server, and the distant rendering engine module is used to perform the second non-real-time task rendering calculation; the cloud cache module is used to store the cached data decoded and rendered in the cloud server, and the data after rendering is completed is compressed and encoded by the cloud encoding module.

[0028] Preferably, the cloud decoding module includes decoding coordinate information for representing the three-dimensional coordinates of the rendering task; the cloud encoding module includes a panoramic extractor for extracting the second non-real-time task rendering data and a panoramic encoder for encoding the second non-real-time task rendering data.

[0029] The present invention has the following beneficial effects:

[0030] The multi-user collaborative method and system of mobile virtual reality based on edge computing of the present invention offloads tasks in user VR terminals to edge servers or cloud servers for rendering according to different timeliness and then synthesizes them, thereby optimizing the efficiency of task completion and reducing the computing requirements of user VR terminals. This makes the VR collaborative system easier to promote and the user VR terminals more lightweight, while reducing latency and improving user experience. In the multi-user collaborative edge computing scenario, data synchronization requests between multiple users are directly implemented based on communication forwarding between edge servers, so that collaborative tasks between user VR terminals do not need to be broadcast through cloud servers, shortening the data link, reducing the latency of collaborative services, and reducing the return traffic between user VR terminals and cloud servers, thereby reducing the network communication load and management complexity of the cloud server.

[0031] In addition to the above-described objects, features and advantages, the present invention has other objects, features and advantages. The present invention will be further described in detail below with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] The accompanying drawings, which constitute part of this application, are intended to provide a further understanding of the present invention. The exemplary embodiments of the present invention and their descriptions are intended to explain the present invention and do not constitute an undue limitation of the present invention. In the accompanying drawings:

[0033] Figure 1 This is a schematic diagram of the overall design of the preferred embodiment 1 of the present invention;

[0034] Figure 2 This is a schematic diagram of the system framework in the first preferred embodiment of the present invention;

[0035] Figure 3 is a schematic diagram of the format of a collaborative presentation request message in a preferred embodiment 1 of the present invention;

[0036] Figure 4 Schematic diagram of the format of a data packet returned by a collaborative presentation request in a preferred embodiment 1 of the present invention;

[0037] Figure 5 This is a schematic diagram of a multi-user remote collaboration workflow in a preferred embodiment 1 of the present invention;

[0038] Figure 6 This is an example diagram of an object data collaboration request packet in the first preferred embodiment of the present invention;

[0039] Figure 7 is a total delay comparison histogram in the first preferred embodiment of the present invention;

[0040] Figure 8 is a histogram comparing frame rate performance results in the first preferred embodiment of the present invention;

[0041] Figure 9 is a histogram comparing average rendering times in the first preferred embodiment of the present invention;

[0042] Figure 10 This is a comparative bar chart of CPU and GPU utilization under different rendering allocation conditions in the first preferred embodiment of the present invention;

[0043] Figure 11 It is a comparative bar chart of collaborative service delay in the first preferred embodiment of the present invention. DETAILED DESCRIPTION

[0044] The embodiments of the present invention are described in detail below with reference to the accompanying drawings. However, the present invention can be implemented in many different ways as defined and covered by the claims.

[0045] Example 1:

[0046] In a preferred embodiment of the present invention, a multi-user collaboration method for mobile virtual reality based on edge computing is provided, which is used in an edge computing system in which a cloud server, multiple edge servers connected to the cloud server, and multiple user terminals connected to the edge servers collaborate with each other, including the following steps:

[0047] The first user terminal sends an interaction request to the first edge server to which it belongs;

[0048] The first edge server receives an interaction request from a first user terminal, decomposes the interaction task corresponding to the interaction request into a first real-time task and a second non-real-time task; completes the first real-time task locally on the edge server, sends the second non-real-time task to the cloud server for completion, receives second completion data of the second non-real-time task completed by the cloud server, combines the first completion data of the first real-time task with the second completion data, and returns the result to the first user terminal;

[0049] When the first edge server decomposes the interaction task corresponding to the interaction request from the first user terminal, it extracts the collaborative data request that needs to be synchronized from the second user terminal. When the second user terminal and the first user terminal belong to subordinate connections of different edge servers, the collaborative data request is directly sent to the second edge server to which the second user terminal belongs, and the collaborative data returned from the second edge server is received.

[0050] See also Figure 1In a preferred embodiment of the present invention, the collaborative rendering of the cloud server and the edge server refers to rendering the dynamic foreground interactive content (real-time task) in the VR application on the edge side, and rendering the static BE (non-real-time task) on the cloud side, and then returning the BE frame data on the cloud side to the edge side and merging it with the foreground interactive content. The key problem to be solved is how the cloud side accepts and processes the collaborative rendering instruction request sent by the edge side, and returns the result correctly and completely to the edge side. To this end, it is necessary to establish a suitable communication and data transmission mechanism between the cloud side and the edge side. In the edge-cloud collaborative rendering mechanism, the edge server and the cloud server serve as the client and server side of the communication respectively. The client first needs to establish a connection with the server side, and the main thread is used on the server side to handle the client's connection request. In order to ensure reliable data transmission, the preferred embodiment of the present invention uses connection-oriented TCP (Transmission Control Protocol) as the transport layer protocol. At the same time, in order to meet concurrency requirements, a user connection pool is established to maintain a list of all users who have established connections from the client side. Then, the client sends a collaborative rendering request message to the server side, and its data packet format is shown in Figure 3 , where the description of each field is shown in Table 1. The server will open a child thread to register and register a single user's collaborative rendering request and put it into a shared message queue. It should be noted that the queue needs to be locked before the rendering request is placed in it. The main thread parses the rendering request and then processes the rendering task in the request. The processing flow is shown in Figure 2 A key step is how to fully present the rendered BE panorama cube map to the client.

[0051] Table 1: Field descriptions of the collaborative presence request message

[0052]

[0053] Cube map consists of six faces, which are represented by enumeration types. The server needs to send the images corresponding to the six faces in the cube map to the client in an orderly and complete manner, and these images are likely to have a large amount of data. However, under the usual communication protocol, when the data packet is too large, it will be automatically sub-packetized. This will cause the six faces contained in the cube map to generate data confusion during the transmission process, thereby affecting the final correct transmission. For this reason, in the preferred embodiment of the present invention, it is agreed that each data packet sent to the client only contains a face in the cube map, and the transmission of the entire cube map is completed in six times. Therefore, the format definition of the data packet returned by the collaborative rendering request is shown in Figure 4, where the description of each field is shown in Table 2. The client uses a child thread to asynchronously receive data from the server. The client opens a buffer register in the memory to store the asynchronously received data. When the data is larger than 4 bytes, it starts to parse the packet header to obtain the size of the data packet sent by the server. Only when the data packet is completely sent to the buffer register in the client, it is converted into a byte stream protocol and then registered in a message queue. The main thread parses the data packet and then searches the edge cache for the coordinates of the point. If not, it means that the relevant information of the coordinate has not been instantiated. The point needs to be instantiated and stored in the edge cache. If it exists, the instance of the corresponding information of the point is returned, and then the information of the cube map corresponding to the point is updated. When all six faces of the cube map of the coordinate exist in the edge cache, the merge screen operation is performed.

[0054] Table 2: Field descriptions of the packet returned by the collaborative presentation request

[0055]

[0056] The mobile virtual reality multi-user collaboration described in the preferred embodiment of the present invention refers to the interaction between multiple users in different network locations within the same VR scene, while ensuring consistency in the VR program state between users. For example, in a VR serious game for surgical training, multiple geographically distributed trainees collaborate to complete surgical training tasks in the same surgical scene according to their respective divisions of labor. During this process, users may engage in collaborative behaviors such as moving objects, manipulating scene objects, and changing postures. These changes and updates should be synchronized among participating users. Unity allows for the construction of multiplayer games. Using the HLAPI and MLAPI frameworks, developers can build multiplayer networked games with minimal effort. However, HLAPI and MLAPI utilize a client-server communication architecture, which can hinder VR multi-user collaboration by preventing the client from synchronizing scene objects with the server. In cloud-based VR applications, multi-user collaboration is typically achieved by broadcasting the interactive updates of participating users to other users via a cloud server. This collaborative approach can place excessive load on the cloud server and increase network transmission latency. To this end, the preferred embodiment of the present invention implements a novel VR multi-user remote collaboration mechanism based on collaboration between edge servers.

[0057] In a preferred embodiment of the present invention, VR applications run on edge servers, and multi-user collaboration involves dynamic interactions. Therefore, data required for synchronization between users only needs to be transmitted between edges, rather than being broadcasted through cloud servers. This data is divided into character data and object data. Character data primarily includes coordinates, rotations, and animation controller variables; object data includes coordinates, rotations, and event trigger semaphores. Figure 5 This is a multi-user remote collaborative workflow in a preferred embodiment of the present invention, where ACK stands for Acknowledge character. First, an area is selected as the multi-user collaborative range. The communication distance between edge nodes in the area is significantly shorter than that between users and the cloud. This is to meet the principle of edge computing for nearby services, which is conducive to reducing service latency. The edge nodes in the area form an autonomous multi-user collaborative organization, which does not require the intervention of a cloud server for collaborative processing. Then, data synchronization is performed using P2P with host server instead of native P2P communication. This is because native P2P needs to synchronize data between all nodes during collaboration to maintain consistency in the state of distributed VR games, which increases communication overhead. In a preferred embodiment of the present invention, an edge node is selected as a service node within the collaborative area. Figure 5 In this example, assume that edge node S is selected as the service node, while the other edge nodes are clients. Edge node S sends a Start signal to the other edge nodes, indicating that the collaborative service has been initiated. When a client needs to participate in multi-user collaboration, it first initiates a connection request to the service node. The service node then starts a dedicated thread to listen for and process connection requests from the client. After the connection is successful, if the client generates a collaborative action, the character and object data in that action are packaged into a collaborative request and sent to the service node. Figure 6 This is an example of an object data collaboration request packet. The format of this packet is similar to that of a collaborative rendering request packet, except that the message body contains object data-related fields, such as coordinates, rotation, and event trigger semaphores. It's important to note that service nodes also act as ordinary collaborative users. Based on the request information, service nodes synchronize the data required for collaboration and then distribute the collaborative data to all other participating edge nodes, causing them to perform data synchronization operations, ultimately enabling multi-user remote collaboration.

[0058] The multi-user collaborative method of mobile virtual reality based on edge computing of the present invention offloads tasks in user VR terminals to edge servers or cloud servers for rendering according to different timeliness and then synthesizes them, thereby optimizing the efficiency of task completion and reducing the computing requirements of user VR terminals. This makes the VR collaborative system easier to promote and the user VR terminals more lightweight, while reducing latency and improving user experience. In the multi-user collaborative edge computing scenario, data synchronization requests between multiple users are directly implemented based on communication forwarding between edge servers, so that collaborative tasks between user VR terminals do not need to be broadcast through cloud servers, shortening the data link, reducing the latency of collaborative services, and reducing the return traffic between user VR terminals and cloud servers, thereby reducing the network communication load and management complexity of the cloud server.

[0059] In a preferred embodiment of the present invention, when a user terminal sends an interaction request to an edge server, the edge server decomposes the interaction task and executes the first real-time task, while determining whether a collaborative processing task between edge servers is required;

[0060] If the edge server needs to perform collaborative processing tasks, it forwards collaborative requests and data between edge servers, receives collaborative data returned by other edge servers, and finally synchronizes the collaborative data to the user terminal;

[0061] If the edge server does not need to perform the collaborative processing task, determining whether the second completion data exists in the cache of the edge server;

[0062] If the second completion data exists in the edge server cache, it is synthesized with the first completion data rendered by the edge server and sent to the user terminal.

[0063] If the second completion data does not exist in the edge server cache, a second non-real-time task rendering request is sent to the cloud server; after receiving the request, the cloud server determines whether the second completion data exists in the cloud server cache. If so, the cloud server directly returns the second completion data to the edge server; if not, the cloud server executes the second non-real-time task and then returns the second completion data to the edge server, and the edge server combines the first completion data with the second completion data and sends it to the user terminal.

[0064] In a preferred embodiment of the present invention, the user terminal runs on an Android mobile VR device and performs VR image display. It mainly includes a motion detector module, a cloud-request encoding module, and a graphics decoding module. The motion detector module uses the VR device and its corresponding sensors to sense the user's movements. The cloud-request encoding module encodes the received motion updates. The graphics decoding module decodes the graphics encoding data sent by the edge end into frame data.

[0065] The VR device sends a motion update request to the edge server. If the requested VR object is non-head motion, the user terminal sends it to the motion request receiver module in the edge server for parsing. After parsing, the collaborative processing module determines whether collaborative processing is required. If multi-user collaborative processing is required, the near-view synchronization module synchronizes foreground interaction data between collaborating users. If collaborative processing is not required, the motion update request is sent to the edge cache module to determine whether the required BE (Background Environment) panorama frame is available in the edge cache. If not, the panoramic transmitter module sends a collaborative rendering request for the BE panorama to the cloud server. Upon receiving the rendering request, the cloud decoding module decodes the rendering instructions and immediately determines whether the required panorama is in the cloud cache. If so, it is encoded and returned to the edge server. If not, the distant rendering engine module performs the rendering calculations, creating a cube map. After rendering, the cloud encoding module compresses and encodes the image resources. The encoded graphic data is sent to the edge decoding module of the edge server for decoding. After decoding, it is transferred to the edge cache module for storage. Then, it is sent to the frame merging module to merge the BE panorama into the foreground interactive screen. The file is encoded into a binary file and transmitted to the user terminal. The user terminal VR device decodes the data and delivers the image display to the user.

[0066] The multi-user collaborative method of mobile virtual reality based on edge computing of the present invention, when the edge server receives a task request, it will first disassemble the task request and determine whether a collaborative task between edge servers is required. When the edge server processes the collaborative task, it only needs to exchange data between edge servers and then synchronize it to the corresponding user terminal without passing through the cloud server, which greatly shortens the data link and reduces data delay; before the edge server sends a second non-real-time task rendering request to the cloud server, it will first determine whether the required data is in the edge cache. If so, it will be directly used. When the cloud server renders the second non-real-time task, it will first determine whether the required data is in the cloud cache. If so, it will be directly used. These judgment behaviors can greatly save the rendering pressure of the edge server and the cloud server, reduce rendering time, and thus shorten the overall delay and improve user experience.

[0067] In a preferred embodiment of the present invention, when the interaction request is a VR rendering request, the first real-time task is a foreground interaction rendering task, and the second non-real-time task is a background BE rendering task.

[0068] In the preferred embodiment of the present invention, two groups of experiments using different methods are set up to implement the VR application process:

[0069] (1) Cloud Server and User Terminal Collaborative Remote Rendering (CERR): All rendering workloads are offloaded to the cloud server. Since running VR programs requires expensive GPU resources, a preferred embodiment of the present invention uses a PC with the same configuration as the edge server to simulate the cloud server. To realistically reflect the network environment of the cloud server, the PC and mobile head-mounted display are placed in an Internet simulation environment using the virtual networking tool Zerotier.

[0070] (2) The mobile virtual reality multi-user collaboration method based on edge computing (MUCVR for short) provided by the present invention: VR content is presented collaboratively using edge servers and cloud servers, and a multi-user collaboration mechanism is constructed using collaboration between edge servers.

[0071] To ensure comparability between MUCVR and CERR, the network environments of the two approaches were kept as consistent as possible during the experimental testing. Therefore, the cloud servers and edge servers were virtualized using Zerotier, similar to CERR. The experiments primarily compared the performance of MUCVR and CERR. Each experiment was repeated multiple times to ensure statistical convergence.

[0072] 1. User experience quality

[0073] The key indicators that affect the mobile VR user experience include service response time and frame rate. In the preferred embodiment of the present invention, the service response time of the VR application is defined as the delay or elapsed time from the terminal device sending a user motion request or user operation request, generating a motion update, transmitting it back to the terminal device, and displaying the updated frame. The latency in CERR is mainly composed of the latency of the terminal sending the rendering request to the cloud, the cloud rendering latency, and the latency of the cloud transmitting the frame data to the terminal. In the preferred embodiment of the present invention, the latency of MUCVR and CERR were tested and compared respectively, and the helmet (Viking) and the moving position (Corridor) were randomly rotated during the test. The comparison results of the total latency averages (Total latency) of the two are shown in Figure 7 .

[0074] The results show that regardless of whether running Corridor or Viking, CERR's total delay is significantly higher than MUCVR. Furthermore, 10 groups of total delay results (10 in each group) were randomly selected for variance analysis, and it was found that CERR's total delay fluctuated greatly, while MUCVR's total delay was relatively stable. In order to verify the impact of network transmission on delay, the network delay of MUCVR and CERR were tested and calculated respectively. The results showed that MUCVR's network delay was significantly lower than CERR, and the network delay accounted for a large proportion of their respective total delays. The terminal's frame rate (FPS) test results are shown in Figure 8 Where MUCVR-serverFPS is the frame rate of the MUCVR edge device, MUCVR-clientFPS is the frame rate of the MUCVR mobile device, CERR-serverFPS is the frame rate of the CERR cloud device, and CERR-clientFPS is the frame rate of the CERR mobile device. The average frame rate results from three randomly selected user tests show that the average frame rate of MUCVR on VR mobile devices is higher than that of CERR, and the frame rates on the edge and cloud are very close. Because the users participating in the experiment experienced significant motion sickness under CERR during the actual test, further analysis of the frame rate variance revealed significant fluctuations in frame rate both within and between groups under CERR, which is strongly correlated with jitter caused by network instability. MUCVR, on the other hand, maintains a frame rate above 60 FPS in most cases, thus ensuring a good user experience. In summary, MUCVR can effectively improve the user experience.

[0075] 2. Rendering performance

[0076] In the preferred embodiment of the present invention, the effectiveness of MUCVR in improving rendering performance is evaluated. First, the rendering time of MUCVR is benchmarked and compared with the local execution method of the mobile terminal. The comparison results are shown in Figure 9 , the rendering time of MUCVR in the cloud and edge is very close, and it should be pointed out that the rendering calculations on the cloud and edge are executed concurrently in most cases, so the actual rendering calculation time consumed by the system is generally only slightly longer than the longer of the two. Since the graphics processing performance of edge servers and cloud servers is generally better than that of mobile VR devices, the rendering time of MUCVR is significantly reduced compared to local execution of mobile VR; secondly, both the edge and cloud in MUCVR have a certain capacity of cache to store cached data. The experiment extracted the experimental data of 3 users for analysis. It should be noted that the experiment recorded the location information of these users' movements. During the interaction process, they will repeatedly move between some locations to ensure that the cache can be hit. The results show that in the case of cache hits, the rendering time is much lower than normal, thereby further ensuring the improvement of rendering performance. Finally, the CPU (central processing unit) and GPU (graphics processing unit) utilization of mobile VR devices were tested in two situations: MUCVR and local execution of mobile VR devices. The experimental results are shown in [1]. Figure 10, where MUCVR-Viking represents Viking execution within MUCVR, MUCVR-Corridor represents Corridor execution within MUCVR, Mobile-Viking represents Viking execution locally on the VR device, and Mobile-Corridor represents Corridor execution locally on the VR device. Compared to local execution, MUCVR reduces CPU resource usage and GPU resource consumption on mobile VR devices. This is because the terminal VR device does not need to perform rendering calculations. In summary, MUCVR offers significant improvements in rendering performance compared to existing technologies.

[0077] 3. Multi-user collaboration efficiency

[0078] In a preferred embodiment of the present invention, the multi-user collaborative mechanism based on collaboration between edge servers can mainly reduce the collaborative service delay and improve the user experience. In order to verify the effect of reducing the collaborative service delay, 13 cloud servers were used in the experiment. None of these cloud servers are equipped with independent GPUs. L is selected as the cloud center, and the rest are edge nodes (including G1, C1, H1, B2, and N mentioned in this article). In MUCVR, collaborative data needs to be transmitted between each node. The collaborative data is mainly coordinate data, and the space occupied is generally small. The transmission of multi-user collaborative data is simulated by sending data of a size basically equivalent to the size between nodes, and the corresponding delay is tested. Since MUCVR multi-user collaboration adopts the P2P with host server method, the edge node can be independently selected as the host server (main server) according to demand in the edge node. The corresponding service delay (Delay) was tested when different host servers were selected, and compared with the cloud-based collaborative mechanism. The experimental results are shown in Figure 11 , where G1-Host represents the selection of edge node G1 as the primary server; C1-Host represents the selection of edge node C1 as the primary server; H1-Host represents the selection of edge node H1 as the primary server; B2-Host represents the selection of edge node B2 as the primary server; and N-Host represents the selection of edge node N as the primary server. The system can autonomously select a host server in an edge node cluster closer to the user. Therefore, in most cases, the service latency of a multi-user collaborative mechanism based on collaboration between edge servers will be lower than that of a cloud-based collaborative mechanism. Of course, there are cases where improper host server selection can result in even greater service latency. For example, when N is selected as the host server, the service latency is even higher than that of a cloud-based collaborative mechanism. However, during actual application debugging, this can be avoided by filtering out the host servers.

[0079] In summary, the multi-user collaboration method of mobile virtual reality based on edge computing of the present invention is superior to the existing technology in terms of user experience quality, rendering performance and multi-user collaboration efficiency.

[0080] Example 2:

[0081] In a preferred embodiment of the present invention, a multi-user collaborative system for mobile virtual reality based on edge computing is provided, comprising: a cloud server, a plurality of edge servers connected to the cloud server, and a plurality of user terminals connected to the edge servers; the plurality of edge servers are interconnected;

[0082] Multiple user terminals are used to perceive user actions and send interaction requests to the edge server and decode and display the received rendering data;

[0083] The plurality of edge servers are configured to decompose the interaction request sent by the user terminal, execute the first real-time task to complete the first completion data according to the interaction request, coordinate the processing task, send the second non-real-time task request to the cloud server, receive the second completion data, and perform data merging;

[0084] The cloud server is used to execute the second non-real-time task and return second completion data.

[0085] The multi-user collaborative system of mobile virtual reality based on edge computing of the present invention adopts multiple edge servers connected to multiple user terminals, so that the collaborative tasks of the user terminals can be forwarded only between the edge servers without passing through the cloud server; since the connection relationship between the user terminals, edge servers and cloud servers is set up, the rendering tasks of the user terminals can be offloaded to the edge servers and cloud servers respectively for rendering, thereby improving the completion efficiency of the rendering tasks.

[0086] In the embodiment selected by the present invention, the user terminal includes: a motion detector module, a cloud-request encoding module, and a graphics decoding module;

[0087] The motion detector module is used to sense user movements; the cloud encoding module is requested to encode the motion update request sensed by the motion detector module;

[0088] The graphics decoding module is used to decode the graphics encoding data sent by the edge server into frame data.

[0089] In the multi-user collaborative system of mobile virtual reality based on edge computing of the present invention, a motion detector module is provided in the user terminal, so that the user's movement can be detected and a motion update request is generated; since a request cloud encoding module is provided, the motion update request can be encoded into data that is convenient for transmission; and since a graphics decoding module is provided, the graphics encoding data from the edge server can be decoded into frame data that can be used by the user terminal, thereby enabling the user terminal to display the content normally.

[0090] In a preferred embodiment of the present invention, the edge server includes: a motion request receiver module, a frame merging module, an edge buffer module, a panoramic transmitter module, an edge decoding module, a collaborative processing module and a near-view synchronization module;

[0091] The motion request receiver module is used to receive and decompose the interaction request sent by the user terminal. After the decomposition is completed, the collaborative processing module determines whether a collaborative processing task is required. In the collaborative processing task, the near-view synchronization module is responsible for the interaction of collaborative data between edge servers. The edge cache module is used to store the cached data decoded and rendered in the edge server.

[0092] The panoramic transmitter module is used to send a rendering request to the cloud server, and the second completion data returned from the cloud server received by the edge server is decoded by the edge decoding module; the frame merging module is used to render the first real-time task and merge the first completion data and the second completion data.

[0093] The multi-user collaborative system of mobile virtual reality based on edge computing of the present invention is capable of receiving and decomposing interaction requests from user terminals due to the provision of a motion request receiver module. Only after the decomposed interaction requests can it be determined whether the user terminal needs to perform collaborative processing tasks; due to the provision of a close-up synchronization module, data interaction between edge servers can be carried out to achieve the effect of low-latency collaboration of user terminals; due to the provision of an edge cache module, the edge server can store cached data. When the stored cached data is required for the current task, the rendering time can be effectively shortened and the delay can be reduced; due to the provision of an edge decoding module, the data from the cloud server can be decoded into second completed data that can be used by the edge server; due to the provision of a frame synchronization module, the edge server can render the first real-time task, generate the first completed data, and can merge the first completed data and the second completed data to make the data content complete.

[0094] In a preferred embodiment of the present invention, the cloud server includes: a cloud decoding module, a perspective rendering engine module, a cloud cache module and a cloud encoding module;

[0095] The cloud decoding module is used to decode the rendering request received by the cloud server, and the distant rendering engine module is used to perform the second non-real-time task rendering calculation; the cloud cache module is used to store the cached data decoded and rendered in the cloud server, and the data after rendering is completed is compressed and encoded by the cloud encoding module.

[0096] The multi-user collaborative system of mobile virtual reality based on edge computing of the present invention is provided with a cloud decoding module, which enables the cloud server to decode the received rendering request and clarify the specific content of the rendering task to be executed; the cloud server is provided with a distant rendering engine module, which enables the cloud server to complete the rendering of the second non-real-time task; the cloud cache module is provided, which enables the cloud server to store cached data. When the stored cached data is required for the current task, the rendering time can be effectively shortened and the delay can be reduced; the cloud encoding module is provided, which enables the data rendered by the cloud server to be encoded into data that is convenient for transmission.

[0097] In a preferred embodiment of the present invention, the cloud decoding module includes decoded coordinate information for representing the three-dimensional coordinates of the rendering task; the cloud encoding module includes a panoramic extractor for extracting the second non-real-time task rendering data and a panoramic encoder for encoding the second non-real-time task rendering data.

[0098] The foregoing description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Those skilled in the art will readily appreciate that various modifications and variations of the present invention are possible. Any modifications, equivalent substitutions, or improvements made within the spirit and principles of the present invention are intended to be within the scope of protection of the present invention.

Claims

1. A multi-user collaboration method for mobile virtual reality based on edge computing, used in an edge computing system in which a cloud server, multiple edge servers connected to the cloud server, and multiple user terminals connected to the edge servers collaborate with each other, characterized in that: The following steps are involved: The first user terminal sends an interaction request to the first edge server to which it belongs; The first edge server receives an interaction request from the first user terminal, and decomposes an interaction task corresponding to the interaction request into a first real-time task and a second non-real-time task; The first real-time task is completed locally on the edge server, the second non-real-time task is sent to the cloud server for completion, and the second completion data of the second non-real-time task completed by the cloud server is received. The first completion data of the first real-time task and the second completion data are combined and returned to the first user terminal; When the first edge server decomposes the interaction task corresponding to the interaction request from the first user terminal, it extracts the collaborative data request that needs to be synchronized from the second user terminal. When the second user terminal and the first user terminal are connected to subordinates of different edge servers, the collaborative data request is directly sent to the second edge server to which the second user terminal belongs, and the collaborative data returned from the second edge server is received. When the user terminal sends an interaction request to the edge server, the edge server decomposes the interaction task and executes the first real-time task, while determining whether collaborative processing tasks between edge servers are required; If the edge server needs to perform collaborative processing tasks, it forwards collaborative requests and data between edge servers, receives collaborative data returned by other edge servers, and finally synchronizes the collaborative data to the user terminal; If the edge server does not need to perform the collaborative processing task, determining whether the second completion data exists in the cache of the edge server; If the second completion data exists in the edge server cache, it is synthesized with the first completion data rendered by the edge server and sent to the user terminal; If the second completion data does not exist in the edge server cache, a second non-real-time task rendering request is sent to the cloud server; after receiving the request, the cloud server determines whether the second completion data exists in the cloud server cache. If so, the cloud server directly returns the second completion data to the edge server; if not, the cloud server executes the second non-real-time task and then returns the second completion data to the edge server, and the edge server combines the first completion data with the second completion data and sends it to the user terminal.

2. The multi-user collaborative method for mobile virtual reality based on edge computing according to claim 1 is characterized in that: When the interaction request is a VR rendering request, the first real-time task is a foreground interaction rendering task, and the second non-real-time task is a distant view BE rendering task.

3. A multi-user collaborative system for mobile virtual reality based on edge computing, used to execute the multi-user collaborative method for mobile virtual reality based on edge computing according to claim 1 or 2, characterized in that: include: A cloud server, multiple edge servers connected to the cloud server, and multiple user terminals connected to the edge servers; The multiple edge servers are connected to each other; The multiple user terminals are used to sense user actions and send interaction requests to the edge server and decode and display received rendering data; The multiple edge servers are used to decompose the interaction request sent by the user terminal, execute the first real-time task according to the interaction request to complete the first completion data, coordinate the processing task, send the second non-real-time task request to the cloud server, receive the second completion data and perform data merging; The cloud server is used to execute the second non-real-time task and return second completion data.

4. The multi-user collaborative system of mobile virtual reality based on edge computing according to claim 3 is characterized in that: The user terminal includes: a motion detector module, a cloud-request encoding module, and a graphics decoding module; The motion detector module is used to sense user movements; the request cloud encoding module encodes the motion update request sensed by the motion detector module; The graphics decoding module is used to decode the graphics encoding data sent by the edge server into frame data.

5. The multi-user collaborative system of mobile virtual reality based on edge computing according to claim 3 is characterized in that: The edge server includes: a motion request receiver module, a frame merging module, an edge cache module, a panoramic transmitter module, an edge decoding module, a collaborative processing module and a near-view synchronization module; The motion request receiver module is used to receive and decompose the interaction request sent by the user terminal. After the decomposition is completed, the collaborative processing module determines whether a collaborative processing task is required. In the collaborative processing task, the near-view synchronization module is responsible for the interaction of collaborative data between edge servers. The edge cache module is used to store cached data decoded and rendered in the edge server. The panoramic transmitter module is used to send a rendering request to the cloud server, and the second completion data returned from the cloud server and received by the edge server is decoded by the edge decoding module; the frame merging module is used to render the first real-time task and merge the first completion data and the second completion data.

6. The multi-user collaborative system of mobile virtual reality based on edge computing according to claim 3 is characterized in that: The cloud server includes: a cloud decoding module, a perspective rendering engine module, a cloud cache module and a cloud encoding module; The cloud decoding module is used to decode the rendering request received by the cloud server, and the distant rendering engine module is used to perform the second non-real-time task rendering calculation; the cloud cache module is used to store the cached data decoded and rendered in the cloud server, and the data after rendering is completed is compressed and encoded by the cloud encoding module.

7. The multi-user collaborative system of mobile virtual reality based on edge computing according to claim 6, characterized in that: The cloud decoding module includes decoded coordinate information for representing the three-dimensional coordinates of the rendering task; the cloud encoding module includes a panoramic extractor for extracting the second non-real-time task rendering data and a panoramic encoder for encoding the second non-real-time task rendering data.

Images